Sample Size and Uncertainty: Why 80% of 10 Trials Is Not Enough

Two people report what looks like the same result. The first says, “8 of my 10 trials worked.” The second says, “800 of my 1,000 trials worked.” Both say 80%.

Are the two reports equally strong? No. A percentage hides one crucial piece of information: how many trials are behind it. Eight out of ten can happen by chance alone. Eight hundred out of a thousand is very hard to explain by chance alone if the true chance is only 50%, the trials are independent, and every result was recorded.

This article gives a simple way to measure the difference: look at how wide the uncertainty is around a percentage. It is useful for judging any claim of the form “X percent success”, including testimonials, accuracy reports and numbers said to come from “live” results.

All examples use made-up numbers for learning. This article does not assess any game, provider or site, and it is not an invitation to bet.

Every “percent success” report actually contains three numbers. Make sure all three are visible.

• Number of successes (k). How many times the counted event really happened.
• Number of trials (n). How many times it was observed, including every failure.
• The percentage (k ÷ n). The quotient of the two. This is usually the only number shown.

A percentage without n cannot be judged. A percentage without a definition of “success” cannot be judged either. So ask two questions: what counts as a success, and how many trials were observed in total, not just the ones chosen for display.

Example: “80% accuracy” means nothing until it is clear what was predicted, when the prediction was recorded, and how many predictions were made in total.

Take an example you can recompute: flipping a fair coin 10 times. Each flip has a 50% chance of heads.

The chance of getting 8 or more heads in 10 flips is 56 out of 1,024 possibilities, or about 5.5%. That is roughly 1 in 18 attempts. It is not rare.

If 100 people each flip a fair coin 10 times, on average about 5 of them will report “80% heads or more”, even though the coin is ordinary. The lucky ones are the most likely to tell their story.

Now scale it up. The chance of 80 or more heads in 100 flips is only about 0.00000006%, roughly 1 in 1.8 billion. For 800 heads in 1,000 flips, the result is about 19 standard deviations from the average, so it is practically impossible with a fair coin.

The percentage is the same, 80%. Only the number of trials changed. The fewer the trials, the bigger the role of chance.

To say how certain a percentage is, we use a confidence interval. A good choice for a yes/no proportion is the Wilson interval. The basics of interval estimation for proportions are covered in the NIST Engineering Statistics Handbook.

Results for a few examples, rounded to one decimal:

Notice that 8 of 10 gives a very wide range, while 800 of 1,000 gives a narrow one. Notice also 0 of 10 and 10 of 10: the intervals are still wide. A perfect result from a few trials is not proof that the event is certain.

Try it yourself with the calculator below. Enter the number of successes and the number of trials, then look at the range.

The calculator runs in your browser and neither stores nor sends data. Its result is a calculation from the numbers you enter, not a prediction.

A 95% interval is easy to misread. Three things need to be kept apart.

• What it means. If the same procedure were used on a very large number of new samples, about 95% of the intervals it produces would contain the true value. The 95% describes the procedure.
• It is not the range of the next result. The interval 71.1–86.7% for 80 of 100 does not mean 95% of future results will fall in that range. A single future result can be much more variable.
• It is not a guarantee. There is always about a 5% chance that the procedure misses.

The width of the interval is roughly inversely proportional to the square root of the number of trials, when the proportions are similar, not close to 0% or 100%, and the number of trials is large enough. From the earlier table: a width of 45.3 points at 10 trials, 15.5 points at 100, and 5.0 points at 1,000. In this example, each time the number of trials is multiplied by 10, the width shrinks about threefold. To make the range roughly half as wide, you need about four times as much data. Near 0% or 100% the pattern differs.

That is why a claim based on a few trials cannot be “fixed” by adding one or two more.

The interval formula is valid only when certain conditions hold. Violations damage conclusions more often than sample size itself.

• The same chance of success. Every trial is assumed to have the same, stable chance of success. If the chance changes from one trial to another, a single percentage and a single interval no longer represent it.
• Independent trials. One result does not influence another. Linked results make the interval look narrower than it should be.
• The sample is not cherry-picked. If only good results are shown, the percentage is skewed before it is even computed. Stopping the record exactly when the numbers look good counts as selection.
• The definition is set before seeing results. A success criterion changed after the results are in makes any number look convincing.
• Every trial is counted. Failures, cancelled trials and “irrelevant” data must be explained.

Example: in section 11 of the (Indonesian) article Mitos Slot, five screenshots of wins circulate out of 10,000 players. Those five screenshots say nothing about the chance, because 9,995 other results are invisible. That is a selected-sample problem, and no interval, however narrow, fixes it.

An interval for a proportion applies only to yes/no events: win or not, right or wrong. A payout whose size varies is not a yes/no event.

Imagine two made-up models from the (Indonesian) Panduan Slot. In model A, 80% of spins pay 1.2 units. In model B, 8% of spins pay 12 units. The share of paying spins is very different, yet both have an RTP of 96%.

So “how often it pays” is not enough to know the RTP, and an interval for the win percentage is not an interval for RTP. Estimating RTP from observed results needs the average payout and how widely the payouts vary, and the spread is far wider when there are very large but rare payouts. The expected value article shows the relation between odds, probability and long-run averages.

A practical rule: before using an interval, make sure what is counted really is a yes/no event.

Imagine a claim: “My method is 80% accurate. 8 of my 10 guesses were right.” Let us check it with the tools we now have, without judging the method.

The 95% Wilson interval for 8 of 10 is 49.0–94.3%. The simplest comparison for a two-way guess that is equally likely either way is 50%. The 50% sits inside that interval, just a little above its lower end. So 8 of 10 is not enough to tell the method from random guessing.

If the result were 80 of 100, the interval would be 71.1–86.7%. The 50% is outside it. The evidence is far stronger, but other questions remain: were all guesses counted, were they recorded before the results were known, and does the past say anything about the next one?

For claims connected to prices, the comparison is not always 50%. At decimal odds of 1.90, the implied probability is about 52.6%, before anything else is considered. The 52.6% break-even is higher than 50% because the margin is paid for.

The lesson: a high percentage from few trials is the start of a question, not a conclusion.

Exercise without betting

Someone says “60% worked”. Compute the interval for two possible numbers of trials: 6 of 10, and 60 of 100. Does each one distinguish from 50%?

Five things to remember

1. A percentage without the number of trials cannot be judged.
2. Small samples often produce good-looking numbers by chance.
3. The 95% Wilson interval measures uncertainty, not the range of the next result.
4. For similar proportions not close to 0% or 100%, four times the data roughly halves the width.
5. An interval only means something if trials are independent, the chance of success is the same each time, the sample is not cherry-picked, and what is counted really is a yes/no event.

Laws differ in every country. In Indonesia, gambling, including online, is prohibited by law. This article is written for reading numbers critically, not as an invitation to bet.