Relevancy Analytics relevancyanalytics.com

Incrementality test sizer

Before you hold out a slice of your audience to prove a channel works, check whether the test can see the answer. Put in what you convert in a week, the share you'd hold out, and the lift you'd need to see. It tells you how long the test takes, what the holdout costs, and when a holdout is the wrong tool.

Your numbers

Conversions per week Bookings, orders, leads. Whatever the channel is meant to cause, counted the same way in both groups.

Holdout share, % The random share of people who won't see the channel while the test runs.

Lift you need to see, % The smallest effect that would change a budget decision. If a 3% lift wouldn't move money, don't size the test to find it.

Value per conversion, optional Turns the holdout's cost into money. Any currency.

Measurable in a quarter.

weeks the test needs

1,091

conversions in the holdout by the end

109

conversions the holdout forgoes

6.5%

smallest lift you could see in 12 weeks

Weeks by holdout share

Holdout	Weeks

Smallest lift by test length

Length	Lift

What this assumes

People are assigned to the holdout at random and stay there for the whole test. If the channel can still reach them, the test is measuring something else.
Conversions arrive roughly like counts do (a Poisson process). Seasonality that hits both groups equally cancels out; a promotion that reaches only one group does not.
Two-sided significance at 95% and power at 80%, the usual defaults. A one-sided test with the same inputs runs about a fifth shorter.
The lift is measured as the exposed group against the holdout, so the holdout's conversions are what you'd see with the channel off.

When this isn't your test

If the weeks run past a quarter, a bigger holdout or a bigger target lift are the only two levers here, and both have a price. Past that, change the method. A geo-lift test pools whole regions and borrows strength from the ones you don't touch. Marketing mix modelling reads the lift out of the history you already have, with wider error bars and no holdout at all. Matching the method to the size of the channel is most of the job.

Built by Charlie Ballard, Relevancy Analytics. A sizing estimate, not a test design.