Bayesian A/B Test Calculator: Chance Your Variant Wins

Enter the visitors and conversions for both sides of your test and this returns the probability that the variant is genuinely better, the amount you would expect to give up by calling it today, and a credible interval on the uplift. The probability is the part people misread a p-value as giving them, and the part that actually answers the question a stakeholder asks. The expected loss is the number that tells you the test can stop.

What the probability actually means

Run the example in the calculator and it reports a 96.6% chance that the variant is better. That sentence means exactly what it says: given the data you have entered and no other assumptions, there is a 96.6% chance the variant's true conversion rate is higher than the control's. It is a statement about the two rates, which is what everyone in the room thought a p-value was giving them.

A p-value says something narrower and much less useful in a meeting. It answers: if the variant were truly identical to the control, how often would random chance produce a gap at least this large? A p-value of 0.034 does not mean a 3.4% chance the variant is worse. Plenty of experienced people say it out loud anyway, because the Bayesian sentence is the one the brain wants. This calculator gives you that sentence honestly, without having to mistranslate the frequentist output into it.

The price is that you have to state a prior, a starting belief about the rates before the test ran. This tool uses a flat prior, Beta(1, 1), which treats every conversion rate between 0% and 100% as equally plausible before a single visitor arrives. That is deliberately uninformative, so the output is driven by your data rather than by an assumption you did not choose. It also means the tool agrees closely with a classical analysis of the same numbers, which is a useful property when somebody wants both.

Expected loss is what ends the test

The probability is the headline, but it is a poor stopping rule on its own. A test can sit at 95% confident that the variant is better by an amount nobody will ever notice, and it can sit at 70% while the upside is worth six figures. The number that resolves this is expected loss, sometimes called the risk or the potential loss.

Expected loss answers a different question: if I make the call now and I am wrong, how much do I lose on average? It weighs every possible size of mistake by how likely that mistake is. In the example, shipping the variant risks 0.005 percentage points of conversion rate, around 0.09% of the control rate. Keeping the control instead risks 0.655 percentage points, more than a hundred times as much. The decision is not close, and it is not close for a reason that a probability alone never shows you: the downside of being wrong about B is tiny, while the downside of being wrong about A is the whole uplift.

That is why the calculator asks for the smallest uplift worth having. It converts that into a threshold and compares it with the expected loss of whichever option is ahead. Once the remaining risk is smaller than the improvement you said you would care about, more traffic buys you nothing you were ever going to act on. This is the decision rule behind the Bayesian engines in commercial tools, including VWO's SmartStats, which is worth knowing if you are comparing platforms on their statistics.

Reading the credible interval

The interval on the uplift is the part most teams skip and then regret. In the worked example the variant is 96.6% likely to be better, and the 95% credible interval on the uplift runs from about minus 1% to plus 29%. Both of those are true at once. The result is convincingly positive and you still do not know its size to within an order of magnitude, which matters enormously if a forecast is about to be built on it.

A credible interval means what it sounds like: there is a 95% chance the true uplift is inside it. Again, this is the interpretation people wrongly attach to a frequentist confidence interval, where the correct reading involves imagining repeated experiments. The same logic applied to a single rate rather than a difference is in the conversion rate confidence interval calculator.

Intervals narrow roughly with the square root of your sample, so halving the width of that interval takes four times the traffic. If your finance team needs the size of the win and not just its direction, that cost has to be planned in at the start with a sample size calculation, not discovered at the end.

How the calculation works

Each conversion rate is modelled as a Beta distribution updated by your data. Start with Beta(1, 1), add c conversions and the n - c visitors who did not convert, and the posterior is Beta(c + 1, n - c + 1). This conjugacy is the reason the whole thing runs instantly in your browser with no simulation and no server call.

The probability that the variant wins is the chance that a draw from the variant's posterior exceeds a draw from the control's. For integer counts this has an exact closed form, a finite sum evaluated in log space here so it stays stable with large numbers. That sum can be written four equivalent ways, and the tool takes whichever is shortest, so the work is set by the smallest of the four counts rather than the largest. Only when all four pass sixty thousand, the point at which both posteriors are indistinguishable from normal curves, does it fall back to a normal approximation, and there the two agree to six decimal places.

Expected loss uses the same machinery. The loss from shipping the variant is the average of the shortfall in the cases where the control is genuinely better, and zero where it is not, which reduces to the difference of two of those same probability terms scaled by the posterior means. The credible interval on the relative uplift is found by integrating the control's posterior against the variant's cumulative distribution, then solving for the two bounds, so it is a genuine posterior interval rather than a normal approximation of a ratio.

The per rate intervals under the result come from inverting the incomplete beta function, which is why they are slightly asymmetric: a conversion rate near zero has more room above it than below, and the interval reflects that instead of pretending the posterior is a bell curve.

Where Bayesian analysis still bites

The common claim is that Bayesian testing lets you peek whenever you like. The truth is narrower. The probability statement stays valid at every point in the test, so nothing breaks by looking. What is not safe is a habit of looking every morning and stopping the moment the probability crosses 95%. Watch a coin flip closely enough for long enough and it will cross 95% at some point, and you will be there to see it. Peeking is safer with this method, not free.

The expected loss rule is the reason it is safer. Stopping when the remaining risk falls below a threshold you set in advance caps the average damage no matter when you look, because you are not stopping on a probability that wanders, you are stopping on an amount of money you decided you could shrug off. Set that threshold before the test starts, the same way you would write the hypothesis before the test starts.

Four things this calculator cannot see, and any of them will beat the arithmetic:

• A broken split. If the two sides did not receive the traffic you intended, the comparison is contaminated before the maths begins. Check it with the sample ratio mismatch calculator, which is the first thing to run on any surprising result.
• Too short a run. Weekday and weekend buyers behave differently, and so do the people who arrive during a payday week. Cover whole business cycles rather than stopping on a Tuesday afternoon, using the duration calculator to set the floor.
• Novelty and disruption. Returning visitors react to change itself, in both directions. The first days of a test on a familiar interface are rarely representative of the third week.
• Selective segments. Slicing a flat result until one segment looks good reintroduces every multiple comparisons problem the Bayesian framing is often assumed to solve. Decide the segments in advance or treat them as the next hypothesis rather than this test's finding.

Setting the smallest uplift worth having

The threshold input does real work, so it deserves a minute of thought rather than the default. It should be the improvement below which you would not have bothered building the change, expressed relative to your current rate. For a high traffic checkout, 1% relative can be worth a great deal of money and is reachable in a week. For a low traffic B2B enquiry form, anything under 10% is unmeasurable in any sensible timeframe, and setting the threshold low just guarantees the test never ends.

If you are unsure, work backwards from what the change is worth. The revenue uplift calculator turns a percentage into money at your volumes, and the figure that is too small to fund another week of traffic is your threshold. The mirror image of this question, how big an effect your traffic can detect at all, is the minimum detectable effect calculator.

One last habit worth keeping. Record the probability, the expected loss and the interval at the moment you stopped, alongside the hypothesis. Bayesian results are easy to remember flatteringly, because "96% likely to win" sounds like "we won by 13%" a month later, when the interval said the honest range was anywhere from nothing to a quarter. New to the method underneath all this, or need to explain it to somebody else? Start with what A/B testing is and come back.