← All playbooks·Digital attribution
Playbooks · Digital attribution
Isolate incremental conversions with a holdout test
This job isolates the conversions a campaign actually caused from the ones that would have happened anyway, using a randomized holdout instead of attribution modeling. It produces an incremental conversion count and a cost-per-incremental-conversion figure that feeds the go/no-go and budget-reallocation decision for the channel or campaign.
What you need first
- Eligible audience size and identifiers, from the ad platform or CDP audience export
- Target holdout percentage and minimum sample size for the expected lift, from a power calculation or the platform's built-in incrementality calculator
- Campaign spend and flight dates, from the media plan
- Standard conversion rate or CPA benchmark, from historical campaign reporting
- Conversion value or AOV, from the client's analytics or finance data
The procedure
- Define the eligible population and randomly split it into holdout and exposed groups before launch, producing two comparable groups sized for the expected lift
- Suppress campaign exposure to the holdout group across every channel touching that population, producing a clean control free of cross-channel leakage
- Run the campaign against the exposed group for the full flight, producing conversion counts in both groups over the same window
- Calculate the conversion rate in each group and test the gap for significance with a chi-square or z-test, producing a lift estimate with a confidence level
- Multiply the lift rate by the exposed group size, producing an incremental conversion count and, at the given AOV, an incremental revenue figure
- Divide campaign spend by incremental conversions, producing a cost-per-incremental-conversion figure to set against the standard CPA
Worked through with numbers
Poland flight, eligible audience 1,500,000. Holdout set at 10% = 150,000 users; exposed group = 1,350,000. Holdout converts at its baseline rate of 1.6% = 2,400 conversions. Exposed group converts at 2.4% = 32,400 conversions. Expected conversions in the exposed group at the holdout's rate would be 1,350,000 x 1.6% = 21,600, so incremental conversions = 32,400 - 21,600 = 10,800, a 50% relative lift (10,800/21,600) and comfortably significant at this sample size. Campaign spend was €180,000, giving a cost per incremental conversion of €180,000/10,800 = €16.67. At an AOV of €45, incremental revenue = 10,800 x €45 = €486,000, so incremental ROAS = €486,000/€180,000 = 2.7. Read it as: for every euro spent, €2.70 in revenue would not have arrived without the campaign; compare the €16.67 cost-per-incremental-conversion against the client's standard CPA to see if the channel is worth its current budget.
Where it goes wrong
- Size the holdout to the expected lift with a power calculation, not to a round number like 10% - a sub-1pt lift on a small holdout won't clear significance
- Suppress the holdout across every channel in the plan, not just the primary platform - leakage through retargeting or a second platform shows up as false negative lift
- Compare conversion rate between groups, not raw conversion count - unequal group sizes make raw counts overstate or understate the real gap
- Commit to the pre-set flight duration and sample size instead of reading interim results - stopping early on a favorable peek inflates false positives
How to know it is right
The result holds if the lift clears your pre-set confidence threshold (typically p<0.05) and the holdout group's own conversion rate tracks its pre-campaign baseline, confirming the split was clean before you attribute the gap to the campaign.
Terms used