Stop Chasing Data and Start Discovering Insights

My first job out of college was at Google. I didn’t know much about marketing at the time, let alone what marketing meant at Google, but it was the best offer I had by a mile. At the end of our first orientation day, after getting branded lava lamps and sitting in on a Q&A with then-CEO Eric Schmidt, we were told which product we’d be working on. My day-old, newly graduated friends were put on the marketing teams for Google Maps, Search, Docs and other equally sexy consumer products. I, however, got the Google Content Network. At the time, we were told that marketing managers selected which new grads they wanted on their teams based on the skills that were reflected on our CVs—we had no say in the matter. Being assigned to the Content Network was like finding out you’d been placed into a fifth Hogwarts house, one you’d never heard of that also happened to be on the outskirts of campus. As my friends worked on big, visible marketing campaigns, I begrudgingly tried my best to convince advertisers that running online display ads was as effective as paid search, all at a time when the “long tail” of websites was perceived to be the dark web. It wasn’t glamorous. Back then, I didn’t yet understand the difference between data and insights. While others shared blog posts, videos and landing pages of cool new things they were doing, I spent my time thinking about the recipe for effective advertising: If we could demonstrate that Content Network ads (now known as the Google Display Network) could consistently produce similar results to search ads—or even work in tandem with search to increase ROI—we’d be off to the races. This was, at its core, a data and measurement challenge, and it radically transformed me into a data evangelist and obsessive. Once I got the hang of it, I wanted to measure everything. Because I could. Because it felt good to feel like I had the answers. And because back then, I didn’t yet understand the difference between data and insights. There’s a common saying that the definition of insanity is doing the same thing multiple times and expecting different results. But I think a different type of insanity is more pervasive in marketing organizations: doing the same thing multiple times without ever validating if it’s having the intended result. This stems from easy access to metrics without the right tools to differentiate them—the signal vs. noise problem. This type of insanity is caused by a lack of understanding of statistical significance. The mere mention of statistical significance may cause some eyes to glaze over—push through it, I beg of you—but having a comfortable familiarity with this concept is like wearing a pair of glasses that puts into focus which data to pay attention to. Just because something can be measured doesn’t mean the datapoint is an insight or should weigh into decision-making. The simplest way I’ve found to get my head around statistical significance is repeatability. When we find that something is statistically significant, it means it wasn’t random luck. It means if you did that same thing again, you’re likely to get the same result. It’s the necessary ingredient to building a repeatable playbook that systematically improves the effectiveness of your marketing. Lots of companies prey on the fact that marketers don’t understand statistical significance. This has become especially prevalent in the creative measurement industry, as computer vision technology has increased our ability to recognize objects in ads. Companies will look at 10 ads, note that the one with a higher clickthrough rate had lemons in it, and claim that lemons increase clickthrough—ignoring the thousands of other variables that influenced this outcome. It’s misleading, but worst of all, it’s irresponsible. Here's how to protect yourself from faux data: Start with a simple question. When presented with data and insights, especially around marketing effectiveness and performance, ask: “Is this statistically significant?” If the person doesn’t know what you’re talking about, or dodges the question, hold off on getting too excited and ask them to increase the sample size or simply rerun the data to determine if it’s a significant finding.Go beyond campaign-level data. Campaign measurement is important, but it rarely leads to replicable findings that will work across your future campaigns due to low sample sizes that don't pass muster for statistical significance. Push your data partners to deliver cross-campaign insights over a longer timeframe—data variety can improve the replicability strength of your insight.Be wary of those selling you the answer. The only true way to know why something worked is through causal testing. Create the same ad, one with lemons and one without, and see if lemons are indeed the performance holy grail you’ve been promised. But this is only worth investing resources into if you establish a statistically significant correlation first, which tells you there’s a pattern worth paying attention to. Marketers don’t need to become data scientists, but we do need to hone our ability to separate anecdotes from insights we can systematically layer, brick by brick, to strengthen the foundation of what we’re doing. Data carries tremendous power, and as an industry we have a duty to ensure those wielding it use it responsibly, accurately and representatively. 14 Ways Brands Can Put One-to-One Marketing Back to Work