Capping Outliers in A/B Tests Cuts Error by a Third

The most useful thing published for testing teams in the last fortnight is a careful look at a setting most people switch on without thinking: capping outliers in revenue metrics. Alongside it, LaunchDarkly changed what you can do to a flag mid-rollout, Headspace explained how it writes stop rules before a test starts, and Amplitude made a case for fixing checkout bugs before testing anything new this Christmas.

Capping outliers: a third less noise, and a known bias

On 29 October GrowthBook published a simulation study of winsorization, which replaces every value above a percentile, such as the 99th percentile of revenue per buyer, with that percentile. The set-up is 300,000 customers, one in ten of whom buys, and the top 1% of buyers account for about 13% of revenue. Capping at the 99th percentile of buyers shrank the standard error of the estimate by about a third, which GrowthBook says is roughly the same as doubling the sample size. Where the top 1% hold close to a third of revenue, the gain was worth fourteen times the sample.

The cost depends on who the change affects. If every buyer spends the same extra amount, the capped estimate is essentially unbiased. If every buyer spends the same extra percentage, the cap erases what the biggest buyers added and the estimate comes out about an eighth too low, matching their 13% share of revenue. GrowthBook’s advice: if you are deciding whether to ship, cap the top 1% or so unless you expect your biggest buyers to respond much more strongly than everyone else. If you need the number itself for a forecast or business case, treat it as a trade between bias and noise, and remember the confidence interval is for the capped effect, not the raw one. To check where an effect comes from, compare spend between the arms at the 90th and 99th percentiles. Our guide to what A/B testing is covers the basics of reading a result. Read the post on the GrowthBook blog.

LaunchDarkly locks rule order during guarded rollouts

LaunchDarkly’s documentation changelog records that on 28 October it clarified that you cannot reorder a flag’s targeting rules while a guarded or progressive rollout is running on it. On 21 October it documented choosing or overriding the value column when creating warehouse-native numeric metrics, with an optional event value mapping for metric data sources. And on 29 October it published a guide to connecting AWS DevOps Agent, GitHub Actions and Kiro to automate experiments and guarded rollouts.

The rule-order lock matters because targeting rules are evaluated from the top, so moving one can change who sees a variation halfway through a rollout. If your team tidies up flags as it goes, plan the rule order before a rollout starts, not during it. The warehouse metric change helps teams whose event tables keep the revenue figure in a column with a different name. Our A/B testing tools comparison covers which platforms work warehouse-native. See LaunchDarkly’s changelog.

Headspace: write the stop rule before the test

On 20 October GrowthBook published a write-up of a conversation with Mary Alfheim, VP of data and analytics at Headspace, about testing Ebb, its AI companion. Every Ebb experiment is organised in three layers: inputs (the prompt, safety rule or infrastructure being changed), outputs (more sessions, longer sessions, more messages, user feedback) and guardrails, with clinicians in the loop on every test doing clinical QA. Headspace was running little experimentation a year ago.

The part that travels is how stop rules work. Her example has nothing to do with AI: if a test lifts engagement and does no harm to a cost guardrail, the team can roll it out; if the guardrail is harmed, there are rules to roll back or send it for review. With those rules written down first, product teams do not need the data team for every decision, which is how self-serve testing stays safe. That is the same discipline as writing the success measure into a CRO hypothesis before launch. Read it on the GrowthBook blog.

Amplitude: fix broken checkouts now, test new ideas later

Amplitude published a pre-Christmas checklist on 26 October with one clear rule: “A broken payment form needs a fix; a new way of presenting shipping options needs a test.” It suggests starting with the steps where shoppers have shown clear intent, from adding to cart through payment to a completed purchase, broken down by device, channel and new or returning shoppers, then watching session recordings of people who left at the worst step. Its example is Fender, which found a redundant cart review step with funnel analysis, removed it and reported a 27% increase in overall conversions.

With a few weeks to peak trading, the point is to spend scarce traffic on tests only where the answer is genuinely unknown, and to ship the obvious fixes now. Our guides to conversion funnel optimisation and CRO best practices cover how to find the leaks. Read it on the Amplitude blog.

Also this week

PostHog’s changelog lists a new Java provider, dated 23 October, that lets JVM services evaluate PostHog feature flags through the standard OpenFeature API, and on 22 October it added feature flag setup to its install wizard, which creates inactive example flags. See PostHog’s changelog.