Revenue Per Visitor Significance Calculator (Welch's t-test)

This calculator tests whether the difference in revenue per visitor (or average order value, or any other money metric) between your control and variant is statistically significant, using Welch's t-test. A conversion rate calculator cannot do this job, because revenue is not a yes-or-no outcome: most visitors spend nothing and a few spend a lot. That spread is what decides significance, so you need each group's standard deviation as well as its average. With the example figures loaded below, a 14.6% lift in revenue per visitor across 24,000 visitors is still not significant at 95%.

How the calculation works

The calculator runs a two-sided Welch's t-test, the version of the t-test that does not assume both groups have the same variance. That matters for revenue, because a variant that changes buying behaviour often changes the spread of spend as well as the average.

• Standard error of the difference = √(SD_A² / n_A + SD_B² / n_B)
• t statistic = (mean_B − mean_A) / standard error
• Degrees of freedom come from the Welch-Satterthwaite formula, so they can be a fraction.
• p-value is read from the t distribution with those degrees of freedom, and the confidence interval is the difference plus or minus the critical t value times the standard error.

The relative lift range is the absolute interval divided by the control mean. It treats the control mean as fixed, so read it as a close approximation rather than an exact interval.

Where to find the standard deviation

Most testing tools report revenue per visitor but hide its standard deviation. The reliable route is to export one row per visitor (or per user) with their revenue in the test window, putting 0 for everyone who did not buy, then use =STDEV.S() in Excel or Google Sheets on each group. If you leave the zeros out you are measuring spend per buyer, and the answer will be wrong for revenue per visitor. For a quick check you can paste a few thousand values into the raw-values box above and it will fill the mean and standard deviation for you.

Why revenue tests need more traffic than conversion tests

Revenue per visitor combines two sources of noise: whether someone buys, and how much they spend when they do. When most visitors spend nothing, the standard deviation is usually several times the mean, as in the example above, and the standard error shrinks only with the square root of the sample. A few very large orders can also swing the result on their own. Two common, defensible fixes, both agreed before the test starts:

• Cap extreme values (winsorising): replace every order above, say, the 99th percentile with that percentile value, in both groups, then recompute the mean and standard deviation.
• Run the test for whole weeks so weekday and weekend spending patterns appear in both groups equally. Our A/B test duration calculator helps you plan that.

If your main metric is a yes-or-no outcome such as purchase or sign-up, use the A/B test significance calculator instead. To turn a confirmed lift into money, try the CRO revenue uplift calculator.