If you haven’t been using incrementality to measure the performance of your advertising campaigns, you should be.
Incrementality is a measurement tool that calculates incremental outcomes of your campaigns—everything from conversions and sales to brand awareness and more. One commonly used methodology is building an experiment where you identify test and control audience groups during a campaign and then run an outcome comparison between the two groups. The difference in the outcome is the incremental results.
While any simple test-vs.-control experiment can enable marketers to understand incremental lift, a strong, statistically validated experiment can help you not only understand the lift but also make marketing decisions with high confidence.
Albertsons Media Collective creates these types of experiments for client campaigns using two advanced statistical methodologies. The first is setting up a randomized controlled experiment before each campaign begins. The second uses time series data to estimate the causal inference and calculate incremental lift results that are proven to solely contribute to an ad’s performance.
Set up a randomized experiment
When it comes to applying incrementality to their campaigns, many marketers use a synthetic control methodology where ads are shown to the entire audience segment. Then after the campaign ends, a look-alike—or synthetic—group, who did not see the ad, is recruited from the initial target audience pool to perform a retrospective analysis against the group that did see the ad. While these experiments are easy to implement, they could be compromised by sampling biases.
To avoid this pitfall, you should use a randomized controlled trial. Before a campaign starts, the entire audience segment is randomized into test and control groups. A selected ad is then targeted to the test group while being withheld from the control group. The randomization ensures both the test and control groups are comparable. Having this audience control group at the start of a campaign helps to eliminate the biases mentioned above.
Dig deeper into randomization methodology
Take the example of Albertsons Companies, which sells products under 24 different food and drug retail store brand names, or “banners,” in 34 states. The nature of products, promotional events, seasonality trends and customer base for each banner and state can vary, meaning the average basket size per customer can also fluctuate.
A strong, statistically validated experiment can help you not only understand [incremental] lift but also make marketing decisions with high confidence.
You can use stratified random sampling based on historical transactions to ensure when splitting the audience, each has an equal chance of being selected and assigned to either the test or control group. It also ensures that both groups receive an equal representation of transactions in each banner and state where the company operates. This reduces sampling bias and creates test and control groups that still represent the entire audience population.
Analyze lift
In statistics, there is a common belief that correlation does not equal causation. This is important because when a campaign shows a lift, one might assume that the campaign caused the lift automatically.
However, there could be other factors—promotions, price changes, seasonality, etc.—that affected the campaign or the lift. Without controlling these variables in the analysis, one can only assume the campaign played a role in driving the lift; but it may not necessarily be the sole contributor.
That’s why it’s helpful to use Bayesian Structural Time Series (BSTS) to measure the causal effect between the lift and the campaign. This is achieved by estimating the counterfactual outcome of the campaign and understanding what the outcome would have been like if the campaign had never run.
BSTS takes in daily historical data, such as sales and orders of the test and control groups, which helps the model learn the relationship between the two datasets that can be used as a predictor in the model. It then applies this data during the campaign to create a statistically validated counterfactual outcome. The statistically significant difference between the counterfactual and actual outcomes during the campaign yields the causal effect which is used to calculate the incremental lift.
And just like that, you have statistically significant metrics to inform your post-mortem meeting and to help optimize your next ad campaign.
