Measuring Incrementality with Control Groups: A Guide

Every offers dashboard can show redemptions. Far fewer can answer the question a finance director actually asks: how much of this would have happened anyway?

The reliable way to answer it is a control group, sometimes called a holdout. This guide explains how control groups work for card-linked offers, how to design them, and how to avoid the mistakes that make results misleading.

The idea in one paragraph

Take the customers eligible for a campaign. Randomly set aside a portion who will not receive it. Run the campaign for everyone else. At the end, compare the two groups. Because the split was random, the groups were alike at the start, so any difference in behaviour afterwards can be attributed to the campaign. That difference is the incremental effect.

Why before-and-after comparisons mislead

Without a control group, teams often compare spend during a campaign with spend before it. The problem is that many other things change at the same time: salary timing, seasons, holidays, school terms, competitor campaigns and even the weather. A rise during Ramadan or after payday may have nothing to do with your offer.

A control group experiences the same season, the same salary cycle and the same competitive environment. It absorbs all of those effects, leaving only the offer's impact.

Designing the groups

Randomise properly

Assignment must be random, not chosen. Do not hold back "less valuable" customers or those who "wouldn't respond anyway". That breaks the comparison. Use a random split within the eligible audience.

Choose the holdout size

The holdout must be large enough for differences to be meaningful, but not so large that you give up too much of the campaign's benefit. Common choices range from a small share of the audience for large portfolios to a larger share for small campaigns. As a rule of thumb, smaller audiences need proportionally larger holdouts.

Your analytics team can estimate the size needed based on how variable spend is and how large an effect you expect to detect.

Keep the groups stable

A customer in the holdout should stay in it for the whole campaign, and should not receive the same offer through another channel. If holdout customers see the offer anyway, the measured effect shrinks.

Consider persistent holdouts

Some banks keep a small, long-running holdout that receives no offers at all for a period. This measures the total effect of the programme, not just individual campaigns. It needs careful governance, since those customers miss out on benefits, so keep it small and rotate it.

What to compare

Decide your metrics before the campaign starts. Typical choices include:

• spend on your cards in the offer's category, per customer;
• transactions per customer in that category;
• active card rate, for reactivation campaigns;
• new-to-merchant visits, for merchant acquisition goals;
• spend after the campaign, to test whether behaviour persisted.

Compare averages per customer across the whole group, not just among those who redeemed. Comparing redeemers with non-redeemers is a common mistake: redeemers were often more engaged to begin with.

Reading the results

When the campaign ends, calculate:

1. The difference in the metric between the exposed and holdout groups, per customer.
2. Total incremental effect: that difference multiplied by the number of exposed customers.
3. Confidence: whether the difference is large enough, given the natural variation, to be unlikely to be chance. Your analytics team can calculate confidence intervals.

Then set this against the campaign's cost, including any bank-funded rewards and running costs, to estimate return.

A hypothetical walkthrough

Suppose a bank targets 50,000 eligible cardholders with a grocery offer and randomly holds back 5,000 of them. Over four weeks, in our hypothetical example, the exposed group spends on average slightly more on groceries with the bank's card than the holdout. Multiplying the per-customer difference by 45,000 exposed customers gives the campaign's incremental spend. If the difference is within normal variation, the honest conclusion is that the campaign had no clear effect, and the offer design should change.

Common pitfalls

• Contaminated holdouts: customers in the control group see the offer on the merchant's own channels or through a friend.
• Changing the rules mid-campaign: adjusting audiences after launch breaks randomisation.
• Measuring too short: some effects, such as habit formation, appear only after the campaign ends.
• Cherry-picking metrics: choosing whichever metric looks best after the fact. Decide in advance.
• Ignoring small effects: a small per-customer lift across a large base can still be valuable.

Making it routine

Control groups work best when they are standard practice, not special projects. Ideally, your offers platform should:

• create holdouts automatically for every campaign;
• keep holdout customers from seeing the offer in any channel;
• report exposed and holdout results side by side;
• show confidence clearly, so teams do not over-read noise.

Whatever platform you use, ask whether it can do these four things. Until it can, a random split of customer IDs and a simple spreadsheet still get you most of the way.

The takeaway

Redemptions measure activity. Control groups measure impact. With random, stable holdouts, pre-agreed metrics and honest reading of results, banks can show exactly which offers create value, and invest with confidence.