Experimentation News: July 2026

This fortnight leans toward the parts of experimentation that decide whether a result is worth trusting. GrowthBook put out a plain guide to running trustworthy tests, Optimizely shipped a way to isolate a set of experiments so their combined effect is measurable, and its Web editor picked up tooling that makes metrics quicker to define. Here is what shipped and whether it matters for a small programme.

GrowthBook writes down Ronny Kohavi’s rules for trustworthy tests

On 17 July GrowthBook published a set of lessons from Ronny Kohavi, co-author of Trustworthy Online Controlled Experiments and the person who built experimentation at Amazon, Microsoft and Airbnb, alongside Luke Sonnet, GrowthBook’s head of experimentation. The through line is that getting numbers is easy and getting numbers you can trust is hard, and most of the trust is won or lost before a single user is bucketed. The practical advice is concrete: do a power analysis up front so you know the minimum sample you need, set a realistic minimum detectable effect because real effects are small (Bing’s averaged well under one percent), and count only the users actually exposed to the change rather than all your traffic, since a test that looks like 54,000 users but really touched 300 has no power at all. Afterwards, run a sample ratio mismatch check and treat any surprisingly large result as probably wrong until proven otherwise. None of this needs a bigger tool, just discipline, and it maps directly onto our minimum detectable effect calculator, our sample ratio mismatch calculator and the habits in our CRO research methods guide. The full write-up is on GrowthBook’s blog.

Optimizely adds local holdouts to Feature Experimentation

On 22 July Optimizely released local holdouts in Feature Experimentation, which hold traffic back from specific flag rules rather than from an entire project. The point is measurement: if you keep a slice of users out of a chosen set of experiments, you can read the cumulative impact of that whole set instead of guessing at it from a pile of separate readouts, while the rest of your programme keeps running normally. For a small team this is the honest way to answer “did all this testing actually move the number”, because individual wins rarely add up as cleanly as a stack of green results suggests. Deciding what goes in the holdout and why is exactly the kind of upfront call a good hypothesis and QA process should force you to make. The change is logged in Optimizely’s Feature Experimentation release notes.

Optimizely’s Web editor turns any element into a metric

The same week, on 21 July, Optimizely’s Web Experimentation shipped three editor changes. It moved every code editor to Monaco, so custom JavaScript and CSS now get line numbers, syntax highlighting and find and replace. It added click event creation to the new Visual Editor, letting you select any element on your live page and turn it into a reusable metric without hand-writing a tracking call. And it added a Filter User Agents and IPs setting under Settings then Advanced that blocks bots and unwanted visitors from being tracked. The metric-from-a-click piece is the useful one for most teams, since a mislabelled or missing goal is a common way an otherwise clean test ends up unreadable, and keeping bot traffic out of the count protects the same data. If you are still deciding where Optimizely fits against the field, our CRO best practices guide covers the workflow it slots into. The detail is in Optimizely’s Web Experimentation release notes.