False positives in A/B testing: the causes we don't see

Updated on September 25, 2026 — understand false positives in A/B testing before they distort your decisions.

Summary

• A false positive, beyond the purely statistical definition
• Sample Ratio Mismatch: the false positive that dare not speak its name
• Bot contamination: an underestimated problem
• Calendar effects: day of the week, seasonality, promotions
• A reminder about peeking, in short
• How to detect false positives in A/B testing after the fact
• The checklist to avoid false positives in A/B testing before generalizing
• Frequently Asked Questions
• Key takeaways

False positives in A/B testing don't always come from where you expect them: a 1% imbalance in the distribution of your traffic between the two tested versions can be enough to cause a 2% variation in your metrics, according to the figures published by the DoorDash engineering teamNot a 1% discrepancy in the final result—a 2% variation generated simply by the fact that the two groups are no longer truly comparable. And yet, most teams never check this point before reading their conversion rate.

This guide goes beyond the classic false positive — that which stems from a misinterpreted confidence threshold or repeated peering on the results. Specifically, he explores three technical causes of false positives in A/B testing. These causes invalidate a test even before statistics come into play: sample imbalance, bot contamination, and timing effects.

A false positive, beyond the purely statistical definition

A false positive is when a variant is declared the winner when it isn't actually the winner. The most frequently cited cause is statistical—a confidence level reached by chance, a sample size that's too small, or a test stopped too early. But that's only part of the problem.

Another set of causes, more technical and harder to detect, produces the exact same symptom: a result that seems clear, but which is based on data already flawed before the statistical calculation. Even a perfectly designed test conducted for the recommended duration can still produce a false positive. The cause: traffic that has never been comparable between the two groups.

Sample Ratio Mismatch: the false positive that dare not speak its name

Here is the concept most absent from French-language guides on the subject, even though it is among the routine checks of the largest experimentation platforms: the Sample Ratio Mismatch, or SRM.

The principle is simple to understand. An A/B test configured to split traffic equally (50/50) should, on a sufficient sample size, produce a distribution close to this balance. When the actual observed distribution deviates significantly from the expected distribution—for example, 55/45 instead of 50/50—this indicates that a technical problem is affecting one of the two groups, regardless of any conversion results.

DoorDash documents several common causes of SRM in its own system: employees included by default in a group via early access to a feature, bug fixes applied during testing that don't "reset" the history of users already exposed, or simply misconfigured data filtering. In each of these cases, the test result—even if statistically significant—no longer measures what it is supposed to measure.

How to detect and correct a Sample Ratio Mismatch

Detection is usually done with a chi-square test applied to the observed distribution between the groupsThis is a check that DoorDash systematically performs even before looking at conversion metrics. The company reports having reduced its SRM incidents by 70% in six months after automating this check—proof that the problem, once actively monitored, is largely corrected before it skews product decisions.

In practical terms: before celebrating a winning result, verify that the number of visitors in each group matches the initial distribution. A seemingly minor discrepancy—a few percentage points—can be enough to invalidate the entire analysis.

And this verification isn't just for platforms that handle tens of billions of events per day like DoorDash. The principle applies equally to a smaller e-commerce site: the only difference is the volume of traffic needed for the chi-square test to reliably detect the anomaly. With a small volume, an imbalance can remain statistically invisible while still skewing the result—all the more reason to visually check the distribution before relying solely on automated calculations.

Bot contamination: an underestimated problem

A second, often overlooked technical reason is that a significant portion of the tested traffic is sometimes not human-generated. analysis published by Peakhour, a web security specialist, cites the case of a client who discovered that 40% of their test traffic came from bots — a figure that makes you dizzy when you know that this traffic is counted exactly as human traffic in most A/B testing tools.

The problem is exacerbated by the democratization of residential proxies, which allow automated traffic to impersonate individual visitors spread across millions of different IP addresses — rendering traditional detection methods (IP blocking, bandwidth limiting) largely ineffective.

A bot obviously doesn't behave like a human visitor: it doesn't convert in the same way, doesn't stay on the page for the same amount of time, and can even be distributed asymmetrically between your two versions if its browsing behavior differs (a bot that systematically follows the first link on a page, for example). The result: a measured conversion rate that no longer reflects the actual behavior of your audience, without any alerts being triggered if you don't specifically filter this traffic.

This risk particularly affects e-commerce sites with high advertising traffic, where the volume of clicks also attracts more automated traffic for price checking or competitor scraping. For this type of profile, filtering non-human traffic before analysis is no longer a secondary option but a step as essential as the significance calculation itself—a point we elaborate on in our guide dedicated to...A/B testing e-commerce.

Calendar effects: day of the week, seasonality, promotions

A third, more well-known but still underestimated cause in practice is that purchasing behavior varies significantly depending on the day of the week, the time of month, and current sales events. A test launched on a Monday and stopped on Wednesday because it "already seems significant" actually captures a very specific segment of behavior, not a representative trend.

This risk is all the more real given that purchasing behavior in France exhibits strong seasonality—sales periods, Black Friday, holiday periods—during which the usual conversion rate can deviate significantly from normal, in either direction. Concluding a test during one of these periods, without explicitly mentioning it in the analysis, amounts to generalizing a result that will not be repeated the rest of the year.

The simplest practical rule remains to always run a test over complete multiples of seven days, never an arbitrary number of days — this ensures that each day of the week is represented an equal number of times in the final sample, regardless of the total duration of the test.

A 10-day test, for example, contains more Mondays and Tuesdays than Saturdays and Sundays—an imbalance that can be enough to skew the results toward weekday shopping behavior, which differs from weekend behavior on most mainstream websites. Extending the test to 14 or 21 days, rather than settling on a round number that doesn't respect this constraint, corrects this bias without adding complexity.

A reminder about peeking, in short

A fourth cause, this one statistical, deserves a brief mention without being elaborated on again here: "peeking," which consists of looking at the results of a test as it progresses and deciding to stop as soon as a favorable signal appears. We detail this mechanism, with the precise figures published on the subject, in our article dedicated to the statistical significance — essential supplementary reading to cover all the causes of false positives, both technical and statistical.

How to detect false positives in A/B testing after the fact

You can still retroactively verify a result that has already become widespread. Three warning signs should raise concern:

• The announced gain is not reflected in the company's overall metrics. a few weeks after the generalization, when it should have been visible on this scale.
• An A/A test (comparing two strictly identical versions) carried out on the same traffic segment reveals a significant difference where statistically there should be none — a sign that the test infrastructure itself is producing noise.
• The distribution of traffic between the two groups, verified a posteriori, does not correspond to what was configured at the start of the test.

This last point is precisely what an automated Sample Ratio Mismatch check can detect even before having to conduct this investigation afterwards — one of the most concrete arguments in favor of a tool that monitors this type of anomaly continuously rather than on demand.

And it's not just a matter of convenience: an investigation conducted after a change has been implemented is always more expensive than one done beforehand. The time spent understanding why a projected gain never materializes in the company's overall figures is time that could have been used to test the next hypothesis in the backlog.

The checklist to avoid false positives in A/B testing before generalizing

Before deploying a winning variant to 100% of traffic, four quick checks limit the risk of widespread false positives:

• Does the observed distribution between the groups correspond to the expected configuration (no Sample Ratio Mismatch)?
• Did the test cover a full multiple of seven days, without overlapping an atypical promotional period?
• Have you filtered the traffic to exclude bots and identifiable non-human traffic?
• Did the test reach its planned duration, without you stopping it prematurely based on a favorable signal observed along the way?

These checks take a few minutes and prevent weeks of decisions based on a result that will never be replicated in production. A poorly distributed test can also introduce this type of bias even before the analysis phase—see our page on the ultra-fast diffusion details how to avoid side effects related to poor technical implementation of the test. The best time to integrate this checklist remains the configuration phase itself, even before launch — this is what we detail in our guide on the goals and tracking of an A/B test, to avoid having to diagnose a problem after the fact rather than preventing it.

Frequently Asked Questions

What is a Sample Ratio Mismatch (SRM)?

This is an unexpected imbalance in the traffic distribution between test groups—for example, a measured 55/45 split when 50/50 was expected. An SRM almost always signals a technical issue (redirection bug, bot, incorrect filtering) that invalidates the test result, regardless of its apparent statistical significance.

How to detect a Sample Ratio Mismatch

A chi-square test applied to the observed distribution between groups can detect an abnormal imbalance. Some experimental platforms run this test automatically for each test and provide an alert even before viewing the conversion results.

Can bots skew an A/B test?

Yes, and we often underestimate the scale of the problem: some sites have discovered that up to 40% of their test traffic came from bots, notably via residential proxies that are difficult to distinguish from classic human traffic.

Does a test started on a Monday give a reliable result by Friday?

No. Purchasing behavior often varies between weekdays and weekends. A test that doesn't cover at least a full seven-day cycle risks capturing a calendar effect rather than a true effect of the tested variant.

Is peeking (looking at the results too often) the only cause of false positives?

No. It is a well-documented statistical cause, but false positives also come from technical causes independent of the statistical method: sample imbalance, contamination by bots, or uncontrolled timing effects.

Key takeaways

Detecting false positives in A/B testing isn't just about statistical significance—that's only half the work. The other half happens even before calculating the confidence level: balanced traffic distribution, traffic free from bot contamination, and a test duration that covers a full cycle rather than a calendar-biased snapshot.

DoorDash's figure is worth remembering: a 1% imbalance can be enough to generate a 2% variation in your metrics. At this scale, a test that "seems" successful may very well be measuring only a technical artifact—not a real conversion gain.

To limit this risk without having to manually verify each test, our visual editor incorporates a distribution control at each test launch, and the CRO autopilot can automatically flag an anomaly before it becomes a product decision. While the concepts discussed here may still be unclear, our CRO glossary explains them in simple language — and our offer is accessible from our entry-level formulas to apply this rigor from the very first test. If you are currently evaluating another platform, our comparison with AB Tasty It also details how each tool handles this type of quality control — a criterion as important as the price displayed on the page reception of each solution.