GrowthBook Interleaving Tests Rankers on 2% of the Traffic

GrowthBook 5.2, released on 3 November, adds interleaving, a way of comparing two ranking algorithms that companies such as Airbnb report needs about 2% of the traffic of a standard A/B test. GrowthBook says it is the first experimentation platform to offer it; most teams that use it have built it in-house. The same release adds approval controls for AI agents, LaunchDarkly brought outlier capping to its guarded rollouts, and Preply explained why about 80% of its roughly 1,000 tests a year never ship.

GrowthBook 5.2: interleaving for search and recommendation tests

Interleaving is for teams that change the order of a list: search results, recommendations, a feed. Instead of showing ranker A to one group and ranker B to another, it merges both rankers’ picks into one list, and every user sees it. A click on an item credits the ranker that picked it. GrowthBook uses team-draft interleaving, the method Netflix, Airbnb and DoorDash also use: in each round a coin flip decides which ranker picks first, so neither gets the higher slot more often. Items both rankers want in the same round count for neither.

According to GrowthBook’s data scientists, companies running interleaving report needing under 1% of a standard experiment’s traffic at Netflix, about 1.5% at DoorDash, about 1.7% at Amazon and about 2% at Airbnb. When Airbnb compared 29 interleaving results with standard experiments on the same rankers, 82% picked the same winner. GrowthBook itself describes the speed-up as 50 to 100 times.

There are two catches. Interleaving tells you which ranker’s items people picked more often, not how much revenue or retention would change if you shipped the winner. And most of the gain comes from people scanning from the top and stopping once they find something good; on a page where users read the whole list or compare every option, it gains little. The feature is in private beta for Enterprise customers. If you are testing anything other than a ranked list, the standard method and our sample size calculator still apply. Read the 5.2 release notes and the interleaving deep dive on the GrowthBook blog.

GrowthBook 5.2: agents propose, people approve

The other half of the release is about AI agents working inside the platform. A new Human in the Loop mode for API keys lets an agent propose a risky or destructive change but holds it until a person approves it in GrowthBook. It works with the MCP OAuth server and personal access tokens. Admins can now switch off a member’s tokens, set expiry rules, and give a token fewer permissions than its owner. When a Slack bot acts through an API key, revision history can show which teammate asked for the change.

Two smaller additions are useful whether or not you use agents. Remote saved groups let you upload a CSV of millions of IDs, such as a loyalty scheme’s members, to target a flag or experiment without making the SDK payload bigger; they work with the GrowthBook Proxy and remote evaluation for now. A new Diagnostics section on an experiment’s Health tab shows the raw exposure events, so you can see exactly who is in a test when results look wrong. The release also adds 30 API endpoints and updates more than 60. Our A/B testing tools comparison covers where GrowthBook fits against Statsig and PostHog.

LaunchDarkly caps outliers in guarded rollouts

LaunchDarkly’s documentation changelog records that on 4 November it documented winsorization, metric measurement windows and clustered analysis for LaunchDarkly hosted metrics in guarded rollouts. It replaces values beyond a chosen percentile with that percentile, at the top, the bottom or both, and works only on count and numeric metrics, not occurrence or count distinct. The percentile is worked out across all variations of the rollout, and within the measurement window if you set one.

This matters for rollouts guarded on revenue or latency, where one extreme user can drag the average far enough to trigger a false regression, or hide a real one. Last week we covered GrowthBook’s simulations of the same technique: capping at the 99th percentile cut the noise by about a third, at the cost of some bias when the biggest buyers respond most.

Two other LaunchDarkly changes landed in the window. From 2 November you can schedule changes to a segment’s rules or members in advance, such as adding a beta group on launch day; each schedule applies to one environment, synced segments are excluded, and the feature is on select plans only. On 4 November LaunchDarkly moved its hosted MCP server to mcp.launchdarkly.com/mcp, which it now runs directly; it accepts OAuth sign-in only and groups tools into toolsets you can restrict. See LaunchDarkly’s changelog and the metric winsorization topic.

Preply: about 80% of tests never ship, and that is the plan

On 3 November GrowthBook published an interview with Sebastien Polis, senior experimentation manager at Preply, the language tutoring marketplace. Its 25 to 30 product teams run close to 1,000 experiments a year, and quarterly objectives can only be met by claiming results from experiments on an approved target metric. He estimates about 80% never scale to production, and 30 to 40% of each year’s tests are new versions of an earlier idea.

Two lessons are worth copying. A global control group went sharply negative within a month, and the cause turned out to be bad luck in allocation: one side drew more weaker users by chance, so Preply now works on balance before high-stakes tests. And a wording change in the tutor-matching flow, “budget” to “price”, showed a lift in new subscribers that vanished after scaling and came back flat on a rerun. His rule is Twyman’s law: if a simple change shows a big win, run it again before you believe it. Preply also randomises tutors rather than students, because one tutor teaches many students and would otherwise get a different product in each lesson. For setting up the targets and scaling rules, see our guide to building a CRO programme. Read the Preply interview on the GrowthBook blog.