Bandit algorithms vs. classic A/B testing: when to use which?

Updated on October 15, 2026

12.75%. That's the false winner rate we obtained by applying a standard statistical test at the end of tests driven by a bandit algorithm, on 2,000 simulations where the two versions were strictly identical. With a fixed 50/50 split, this rate remains at 4.65%, consistent with the expected 5% threshold. The bandit algorithm isn't a bad tool. It simply addresses a different question than A/B testing, and interpreting it as an A/B test leads to errors.

This guide compares the two approaches from a statistical point of view (exploration, Thompson sampling, regret, power), then by use case, with supporting simulation figures.

Multi-armed bandit or A/B testing: the short answer

A traditional A/B test distributes traffic in a fixed way to precisely measure the difference between two versions; a multi-armed bandit test shifts traffic to the version that appears to be winning in order to lose as few conversions as possible during the test. Choose A/B testing when you need to make a long-term decision and quantify its impact, and the bandit test when the timeframe is short and only the cumulative result matters.

The two methods optimize two different quantities:

• A/B testing minimizes estimation error. It tells you if B beats A, by how much, and with what confidence interval. You pay for this certainty by sending half the traffic to the worse version until the end.
• The bandit downplays the regret, That is, the conversions lost compared to a scenario where you would know the winner from day one. You pay for this gain in accuracy: the losing version receives too few visitors to be measured properly.

Neither method wins on both counts.

Exploration versus exploitation: the dilemma of the one-armed bandit

The name comes from slot machines, the "one-armed bandits": faced with several machines whose payouts you don't know, you must both try several (explore) and play as often as possible on the best one (exploit).

In CRO, each variant is an "arm," each visitor a part, and conversion the gain. A classic A/B test involves 100% exploration during the test, then 100% exploitation once the winning variant is deployed. A bandit continuously mixes the two.

The problem is an old one: William R. Thompson published the method that bears his name as early as 1933, for medical trials.

If the terms "arm," "regret," or "allowance" are new to you, the glossary of A/B testing and CRO defines each one in one line.

Epsilon-greedy, UCB, Thompson sampling: three ways to arbitrate

The three families of bandit algorithms differ in the way they measure exploration: by chance (epsilon-greedy), by calculated optimism (UCB) or by probabilistic sampling (Thompson sampling), and it is the latter that dominates today in testing tools.

Epsilon-greedy: simple, but wasteful

The algorithm sends a fixed portion of traffic, for example 10%, randomly among all variants, and the rest to the best one at any given time. Simple as that, it continues to feed a clearly poor variant until the end. Allocations like "80% based on performance, 20% random" stem from this logic.

UCB: Optimism in the face of uncertainty

Upper Confidence Bound calculates an upper bound for each variant's potential return and selects the highest one. A poorly tested variant has a wide bound, so it will be tested. A well-tested but mediocre variant will no longer be tested. It is deterministic, therefore less suitable for batch updates.

Thompson sampling: betting based on the probability of being the best

For each variant, the algorithm maintains a probability distribution of its conversion rate. For each visitor, it selects a value from each distribution and displays the variant with the highest value. As a result, a variant with a 70% probability of being the best receives approximately 70% of the traffic. The reference tutorial by Russo et al. on Thompson sampling explains why this rule approaches the theoretical minimum regret without any special adjustment.

This is the method used by most platforms. The dynamic allocation documentation from AB Tasty indicates that it relies on Thompson, that it only affects new visitors (a visitor already exposed retains their variant), and that it waits for at least one conversion per variant on the chosen indicator before starting. The comparison between BlastOptimize and AB Tasty discusses the differences in settings between the two tools.

The downside: the cost of A/B testing in conversions.

Regret is the number of conversions lost during a test compared to the ideal scenario where the entire audience would have seen the best version from the start; a 50/50 has regret that grows in proportion to the traffic, a bandit has regret that slows down over the course of the test.

Let's take the case already detailed in our autopilot guide BlastOptimize With 40,000 visitors, A converts at 3%, B at 3.6%. In a 50/50 split, A receives 20,000 visitors and B 20,000, resulting in 600 + 720 = 1,320 conversions. If everyone had seen B, the conversion rate would have been 1,440. The regret associated with the A/B test is 120 conversions, or 8.3% of the potential conversions.

The work of Elea Feit (Drexel) and Ron Berman (Wharton), titled "Test & Roll", This is clearly demonstrated. For a site with 100,000 visitors, the test that maximizes total profit requires only 2,284 visitors per group, compared to 18,468 for a classic hypothesis test designed to detect a 2% increase. The large, "clean" test costs 476 more conversions. The authors also show that the regret associated with a hypothesis test increases with the audience size, while the regret associated with Thompson sampling increases only with its square root.

Remember this idea: for a two-week campaign, almost all the traffic is "in the test" and the regret is significant. For a homepage that will be around for two years, the test represents only a fraction of that traffic.

Our simulation: bandit vs. 50/50 based on 2,000 tests

We replayed each scenario 2,000 times, comparing a fixed distribution to a Thompson sampling updated in batches of 500 visitors: the bandit recovers three-quarters of the regret when the gap is real, but detects the winner less often and almost triples the false positives when there is no gap.

Method: a priori Beta(1,1), batch update, 5% two-sided z-test calculated at the end, as a team would do when reading its results.

Four lessons learned:

• The gain is there. Thompson loses only 29 conversions against 120, consistent with the autopilot guide calculation (approximately 1,408 conversions).
• The situation is deteriorating. The losing variant receives an average of only 4,370 visitors. As a result, the test becomes significant in only 66% of cases, compared to 92% in the 50/50 scenario.
• Without any real deviation, the bandit creates certainties. It continues the randomness of the first batches, unbalances the distribution, and a classic test then concludes wrongly in almost one out of eight cases.
• In small-time trafficking, the bandit saves nothing. With 4,000 visitors, the two methods identified the wrong variant in 15 to 17% of cases, and the regret difference was limited to 4 conversions.

The price of the bandit: power, bias and false positives

A bandit sacrifices three things that A/B testing protects: statistical power, the absence of bias on the measured difference, and the validity of classic significance tests.

Less power

Power, the probability of detecting a real discrepancy, depends primarily on the smallest group. A bandit deliberately starves this group: good for your conversions, bad for your measurement.

A poorly estimated gap

An unlucky early performer quickly receives less traffic: its valuation remains fixed on its poor start, while the favorite corrects itself. The displayed spread is therefore skewed, often overestimated. Before presenting a rise from a troublemaker to your management, plan a confirmation test.

P-values ​​that no longer mean the same thing

Significance formulas assume a predetermined distribution. When the algorithm decides the distribution based on the results, the assumption breaks down. This is related to the "peeking" problem, which involves continuously observing the results and stopping as soon as they appear favorable. Our article on the false positives in A/B testing explains why this practice causes the error rate to skyrocket, and the list of common A/B testing errors the place among the most common.

Operational blind spots

DoorDash's experimental team, which built its own bandit platform based on Thompson sampling, lists in its feedback published in December 2025 several concrete limitations exist: the difficulty of analyzing indicators that are not part of the optimized reward, the inconsistent experience of a user who switches variants from one visit to the next, and the lag between action and result. To limit bias when customer behavior changes over time, the team chose to model the difference from the control version rather than the raw rate of each variant.

When to choose a bandit algorithm

Choose a bandit when the cost of showing the wrong version during the test outweighs the value of knowing precisely why the other wins, which happens especially on short windows and decisions without a future.

• Limited-time campaigns Sales, Black Friday operation, product launch. On a promotion of 20,000 visitors with four headlines, our simulation recovers 82 conversions out of 136, i.e. 60% of the cost of the test, and there will be no "deployment" afterwards.
• Highly rotating elements Titles, email subjects, banners, advertising visuals, replaced every month.
• Many variations, few preconceived notions Six hooks are proposed by the team or generated by AI. The bandit quickly eliminates the weakest ones. A fixed-distribution test would pay a heavy price for each one.
• Continuous optimization : a block that is left to run indefinitely, sometimes adding a variation.

Common condition: the optimized indicator must be measured quickly (click, add to cart, form submission) and accurately reflect the value. On an e-commerce site, our page on the CRO for e-commerce details the pages where these conditions are met.

When to keep a classic A/B test

Keep a fixed-split A/B test for any long-term decision that is costly to reverse or will need to be justified with supporting data, and for any test where conversion comes late.

• Redesign, pricing, payment funnel : a few weeks of regret weigh little against the risk of deploying a false winner for years.
• Need for an impact figure : revenue forecasting, budget arbitration, report to management. Only a fixed allocation provides an unbiased variance with a reliable confidence interval.
• Deferred conversions In B2B or lead generation, form submissions happen quickly, but the real value (qualified lead, appointment, contract signing) arrives days later. A scammer who optimizes the submission process might favor the variant that attracts the least qualified leads. Before launching, define the objective and the tracking which truly reflect this value.
• Several indicators to monitor : a bandit optimizes a single reward and misjudges the rest, including the average basket.
• Different effects depending on the segments : a variant that wins on desktop and loses on mobile. The bandit converges towards the average and stops sending the affected segment to the version that suited it.
• Low traffic As the simulation with 4,000 visitors shows, the bandit adds almost nothing when the signal is too weak. A simple test on a significant change is better.

The same logic applies to testing several elements at once: our comparison between A/B testing and multivariate testing shows that each added combination requires additional traffic.

Combining the two: three hybrid strategies

The two methods follow each other well: rapid exploration first, proper measurement afterwards, depending on the stakes.

1. Sort by bandit, confirm in A/B

Run five or six bandit variants for one to two weeks to eliminate the weakest. Then compare the two best in a predetermined A/B test.

2. Test briefly, then deploy ("test and roll")

This is Feit and Berman's recommendation: to size the test according to expected profit rather than a significance threshold. The test is shorter, the distribution remains fixed and unbiased, and the regret remains close to that of a bandit.

3. Validate using A/B testing, then exploit using bandit testing.

For a permanent campaign (banners, recommendations), an initial A/B test confirms that the campaign delivers a benefit. Then, a bandit manages the content rotation within this campaign. Maintain a small, permanent control group, for example, 5% of the traffic, to verify that the overall benefit is sustained over time.

Whatever the strategy, the fast variant delivery note: a blink of the original page invalidates both methods in the same way. And to read the actual difference between variants, with its intervals, the analytics dashboard BlastOptimize displays the results without jargon.

Checklist: Choosing the right method in 7 questions

Answer these seven questions before each test: at least four "bandit" answers out of the first six indicate automatic allocation; otherwise, remain with fixed allocation. The seventh question verifies that the test has a chance of succeeding.

1. Lifespan of the decision less than a month (bandit) or several months or more (A/B)?
2. Need for an impact figure to present or reuse: no (bandit), yes (A/B)?
3. Conversion time : measurable within the session (bandit) or several days later (A/B)?
4. Number of variants : four or more without a clear favorite (bandit), two or three (A/B)?
5. Secondary indicators things to monitor (average basket size, refund rate, lead quality): none (bandit), at least one (A/B)?
6. Segments likely to react differently: no (bandit), yes (A/B or contextual bandit with a control group)?
7. Traffic : at least a few hundred conversions per variant expected over time (both are possible) or less (neither will decide: test a more marked change)?

Fewer than ten tests per year? Avoid paying for an rarely used option: compare with the prices BlastOptimize.

Key takeaways

Bandit and A/B testing answer two different questions.

• A/B testing minimizes measurement error; the multi-armed bandit minimizes regret, i.e., conversions lost during the test.
• Thompson sampling is the reference algorithm: each variant receives a share of traffic proportional to its probability of being the best.
• In our simulation, the bandit recovers about 75% of the regret of a 50/50, but the detection of the winner goes from 92% to 66% and the false winners from 4.65% to 12.75% without any real difference.
• Bandit for short campaigns, fast-rotating content and numerous variations; A/B testing for lasting decisions, delayed conversions and price testing.
• For small traffic volumes, neither method works miracles.

Frequently Asked Questions

Is a bandit test faster than an A/B test?

It's faster to exploit the data, but not to draw conclusions. Since the losing variant receives few visitors, more traffic is often needed to demonstrate the difference. For a statistical conclusion, a fixed-distribution A/B test remains the shortest path.

Can we calculate significance on a bandit test?

Not with traditional formulas, which assume a predetermined distribution. In our simulation, applying them to a bandit produces 12.75% false winners instead of 5%. Instead, use the probability of being the best variant provided by the algorithm, or confirm the result with a short A/B test.

How much trafficking does it take to use a bandit?

The bandit doesn't need less traffic than an A/B test to learn. It needs conversions on each variant to get started, at least one according to the documentation.AB Tasty, And in practice, a few dozen to get past the noise of the first batches. With several thousand visitors in total, both methods often fail.

Thompson sampling or epsilon-greedy: which to choose?

Thompson sampling is the preferred method in most cases: it adapts the search to the uncertainty and doesn't waste traffic on a beaten variant. Epsilon-greedy sampling is only justified if you want a fixed and easily explainable share of the search.

Does a bandit work for B2B lead generation?

For tactical elements (landing page headline, ad visual, button label), yes, if the optimized metric is form submission. For strategic decisions, no: lead quality is measured too late to guide the algorithm, which risks favoring forms that attract unqualified contacts.

What is a contextual bandit?

It's a bandit that chooses the variant based on the visitor's profile: device, traffic source, page views. It serves to personalize rather than designate a single winner, requires much more traffic, and needs to maintain a control group to measure its impact.