Updated on September 18, 2026

Summary

2.37. This is the factor by which the required sample size must increase if you look at the results of your A/B test 20 times before it is finished rather than just once, according to a statistical analysis published on arXiv in 2024In other words: the dashboard you refresh every morning "to see where things stand" is not a neutral act. It's a statistical operation that, without correction, directly increases your risk of being wrong.

Statistical significance is arguably the most frequently cited—and most misunderstood—concept in the entire CRO vocabulary. This guide explains what it truly means, where the most common pitfalls lie, and, most importantly, why simply "taking a look" at an ongoing test can be enough to invalidate its result.

Statistical significance in a nutshell

Statistical significance measures the probability that the observed difference between two versions is not due to chance. A 95% confidence level—the industry standard—means that if the experiment were repeated many times with no real difference between the versions, we would observe a difference as pronounced as that measured in only 5% of cases.

So it's not the probability that your winning variant is actually better. It's a nuance that seems technical, but it concretely changes how you interpret a result—and it's precisely where most teams go wrong when reading their analytics dashboard.

Type I and Type II errors: the two ways to make a mistake

An A/B test can be wrong in two distinct ways, and confusing the two leads to opposing decisions:

  • Type I error (false positive) — you declare a variant a winner when it is not actually so. This is the risk that a 95% confidence threshold limits to 5% by default.
  • Type II error (false negative) — you conclude that no variant stands out, whereas a real gain exists but the test did not have enough statistical power to detect it.

A example documented by VWO The first case illustrates this well: a change in the position of a call-to-action button shows an apparent 45% increase in sign-ups over a week—a signal that disappears completely once the feature is deployed to 100% of traffic. The test had "successfully" been conducted on a sample size too small to be reliable.

The second case is more insidious because it doesn't trigger any alarm: the test simply concludes that "nothing is happening," whereas the real problem is a lack of statistical power—the probability of detecting a real effect if one exists. A power of 80%, the generally recommended standard, still means that a real gain has a 20% chance of going unnoticed.

P-value and confidence interval: what they really say

The p-value answers a specific question: if no real difference existed between A and B, what would be the probability of observing a difference at least as large as the one measured? A p-value of 0.03 means that such a difference would have a 3% chance of occurring by pure chance — not that your variant has a 97% chance of being the best, a very common misconception.

The confidence interval usefully complements this reading, and it's a point that many guides on the subject—including those from competitor Varify.io—overlook in favor of simply displaying the confidence percentage. Rather than a single figure ("+15% conversion"), the interval provides a realistic range (for example, between +4% and +26%) that reflects the actual uncertainty of the measurement. A wide range means the sample is still too small to draw definitive conclusions, even if the result is already marked as "significant."

The "peeking" trap: why looking at your results too often skews everything

This is the point that almost all mainstream CRO guides — both French and English — completely ignore, even though it is probably the most common mistake made by teams starting A/B testing.

The principle is simple to state but has far-reaching consequences: every time you check the results of a test in progress and ask yourself, "Can I stop now?", you create a new statistical opportunity to encounter a false positive. Researchers call this phenomenon "peeking"—literally, taking a quick look—and it's a temporal form of data dredging: the unintentional manipulation of data through the simple act of repeated checks.

An analysis published on arXiv in 2024 precisely quantifies this risk. For a test observed only once at its planned completion, a Z-score of approximately 1.96 is sufficient to reach a 95% confidence level. However, if this same test is checked 20 times before its completion—a rate that, in reality, corresponds to a daily check over three weeks of testing—the required Z-score climbs to approximately 3.02 to maintain the same reliability. This increase translates into a sample size requirement that is approximately 2.37 times greater to maintain an equivalent level of evidence.

In concrete terms, this means that a test that appears "significant" after four days, when it was designed to last three weeks, probably hasn't reached a real threshold of evidence—it simply got lucky on one of the multiple checks performed. The researchers propose a workaround: require that the result remain significant over several consecutive checks, not just one, before declaring a winner. This is exactly the type of safeguard that reputable tools on the market incorporate to prevent a hasty team from generalizing a premature result—and it's also one of the most concrete use cases of automation of test decisions : let a system apply the rule coldly rather than giving in to the urge to conclude at first favorable glance.

And this is not just an academic detail. A test that is stopped too early because it "seems to be a winner" can, once generalized to 100% of traffic, turn out to be neutral or even negative — exactly the false positive scenario mentioned above with the example of the call-to-action button.

Bayesian vs. Frequentist: Should we change our approach?

Two statistical schools of thought have clashed for decades over the best way to interpret an A/B test, and the CRO is no exception, as detailed in CXL's analysis of Bayesian vs. FrequentistThe frequentist approach — the most widespread, the one that uses p-value and confidence threshold — remains simpler to calculate and standardize from one tool to another.

The Bayesian approach, on the other hand, directly produces a probability that B beats A (for example, "92% chance that variant B is better"), which is more intuitively communicated to a non-technical audience. It also theoretically allows for a more natural handling of the peeking problem described above, by integrating repeated checks into the model rather than treating them as a problem to be corrected afterward.

But neither approach is universally superior—this is the conclusion reached by even the most in-depth analyses on the subject. The choice depends primarily on the tool you use and how you need to present your results internally. What matters more than which approach you choose is understanding the limitations of the method you're using rather than blindly trusting it.

Calculate the sample size before running a test

The best protection against all the pitfalls described above remains the simplest: design your test before launching it, not mid-way through. sample size calculation requires three pieces of information: the current page conversion rate, the minimum improvement you want to be able to detect (a 10% "effect" does not have the same traffic cost as a 30% effect), and the target level of trust.

A practical rule to remember: the smaller the effect you're trying to detect, the more disproportionately large the sample size you need—not linearly. Trying to detect a 5% gain often requires several times more traffic than detecting a 20% gain. This is one reason why it's better to prioritize high-potential hypotheses over micro-adjustments, as long as available traffic remains limited—a principle detailed in our guide on... goals and tracking of a testing program.

Once the sample size is calculated, set an end date in your test creation tool — and above all, stick to it, even if the dashboard already seems to be leaning to one side after a few days.

A "significant" test is not always a reliable test

Statistical significance does not protect against everything. A test can display a "significant" badge while still being biased by factors that statistical calculation alone cannot detect:

  • A sample that is too small despite a result that "seems" clear — the confidence interval, mentioned above, reveals this problem where the mere significance badge masks it.
  • A test stopped too early without correction for peeking, as explained above.
  • A test period that is too short to cover a complete cycle — a test started on a Monday and stopped on a Wednesday does not capture weekend behavior or end-of-month behavior, two times when shopping habits often differ significantly.
  • A technical failure in broadcasting introduces bias even before statistics are considered—a flicker effect between versions, for example, can silently distort the measured behavior. Our page on the Ultra-fast diffusion details this often ignored risk.

A rigorous A/B test is therefore judged on all of these criteria, not on the single number displayed at the top of the report.

Mistakes that deceive even experienced teams

Three errors keep recurring, even among teams that have been practicing A/B testing for a long time:

  • Stop a test as soon as the significance level crosses 95%, without verifying that the minimum planned duration is reached.
  • Ignoring statistical power and confusing "the test showed nothing" with "there is no difference" — two very different statements.
  • Generalize a result tested on a single traffic segment (for example, only advertising traffic) to all visitors, whereas behavior can differ greatly from one source to another.

These errors are costly precisely because they are not seen at the time — they reveal themselves weeks later, when the announced gain never materializes in the company's overall results.

If you are just starting your testing program, our CRO glossary lays the groundwork for the concepts used here, and our A/B testing guide for e-commerce proposes initial hypotheses that are less exposed to these pitfalls, because they are simpler to interpret.

Frequently Asked Questions

What is a 95% confidence level in A/B testing?

A 95% threshold means that if the experiment were repeated many times with no real difference between the versions, the result would be as pronounced as that obtained in only 5% of cases. This is not the probability that the winning variant is actually better.

Why shouldn't you check the results of an A/B test every day?

Each check of the results along the way presents a new opportunity for a false positive. Without statistical correction, a test observed 20 times before its planned completion requires a much higher confidence score (Z of approximately 3.02 versus 1.96 for a single final check) to maintain 95% reliability.

What is the difference between a Type I error and a Type II error?

A Type I error involves declaring a variant as winning when it is not actually so (false positive). A Type II error involves failing to detect a real gain and incorrectly concluding that no variant is better (false negative).

Should we use Bayesian statistics rather than frequentist statistics?

Neither approach is universally superior. The Bayesian approach yields results that are more intuitive to communicate (a direct probability that B beats A); the frequentist approach remains simpler to calculate and standardize. The choice depends primarily on the tool used and the audience to whom the results are presented.

Is a statistically significant A/B test always reliable?

No. A test can be statistically significant while being biased by too small a sample, premature termination without correction, unbalanced traffic segmentation, or a test duration too short to cover a complete usage cycle.

Key takeaways

Statistical significance is not a binary badge to be monitored until it turns green. It is a measure of confidence that depends directly on how the test is conducted — its duration, its sample size, and above all, the discipline with which one resists the urge to draw a conclusion at the first favorable signal.

The peeking trap, documented with precise figures by recent research, deserves special attention: a factor of 2.37 on the required sample size is not a theoretical detail; it's the difference between a reliable test and a result that won't survive full deployment. Set a timeframe before launching the test, stick to it, and let the confidence interval—not just the displayed percentage—guide your final decision.

It is precisely to avoid this type of error that the dashboard BlastOptimize displays the estimated remaining time before reliable significance is reached, rather than a simple percentage that might lead to premature conclusions. If you are currently testing another platform, our comparison with Kameleoon also details how each tool manages this safeguard — and our entry offer allows you to verify it directly on your own data.

Similar publications:

News Methodology & Statistics
Share:

Published by Tom BlastOptimize

Tom leads BlastOptimize, Blast Group's no-code A/B testing tool: a visual editor, flicker-free server-side testing, and statistical analysis of results. He has over 16 years of experience in digital acquisition and conversion optimization with Edilead.

Comments are closed.