Bonferroni Correction Calculator for A/B/n Tests

Paste the p-values from a test with several variants or several metrics, and this shows which results still count once you correct for multiple comparisons. Run four challengers against a control at a 5% significance level and, if none of them truly works, the chance of at least one false winner is about 18.5%, not 5%. Every extra comparison you check is another chance to be fooled by noise.

Which correction should you use?

Bonferroni multiplies each p-value by the number of comparisons. It is the simplest and the strictest, and it controls the family-wise error rate: the chance of even one false winner across the whole test. Šidák does the same job with a slightly less harsh adjustment, assuming the comparisons are independent.

Holm (Holm-Bonferroni) also controls the family-wise error rate, but it works down the sorted p-values step by step, so it never finds fewer winners than Bonferroni and often finds more. If you want family-wise protection, there is little reason to pick plain Bonferroni over Holm.

Benjamini-Hochberg controls the false discovery rate instead: the expected share of your declared winners that are false. It is the most forgiving of the four, and it suits exploratory work such as scanning many secondary metrics for leads to retest. For a ship-or-don't-ship decision on several challengers, Holm is the safer default.

Corrections cost statistical power, so plan for them before launch. Feed the corrected alpha (alpha divided by the number of comparisons, for Bonferroni) into the A/B test sample size calculator and the test duration calculator, and you will know whether an A/B/n test is worth running on your traffic at all. Fewer variants and one primary metric is usually the better trade.