Editor’s note: Industry consultant Shelly Palmer is taking his popular newsletter and turning it into an Adweek article once per week in an ongoing column titled “Think About This.”
We’ve been running a data science experiment over the past few months. Our first goal was to compare and contrast the amount of data we could actively gather using a link to an online survey (please click here to take it) versus the amount of data we could passively gather using our cookies and pixel-monitoring tools. Our second goal was to compare and contrast the value of self-reported data versus observed behavioral data. Our final goal was to turn both data sets into actionable insights and analyze the results.
We were shocked (but not surprised) by what we learned.
What counts?
There are hundreds of data points that can be measured during an average visit to a website. Most are meaningless for the immediate task, although they may have immense value in aggregate over time. But the data doesn’t tell you everything. Clearly, as the adage says, “Not everything that can be counted counts, and not everything that counts can be counted.”
Data rich versus data poor
Data-rich organizations such as Google, Facebook, Apple and other tier-one tech companies are purpose-built to use every data point that users generate. But data-poor organizations (basically everyone else) can’t even buy a seat at their poker table. The data poor must find creative ways to accumulate the information they need to compete.
Regardless, it’s nice to have data
In part one of our experiment, we examined cookie and pixel data that we have been collecting for years to determine which readers gravitate toward which articles. This required no effort on our part. We tag every article we create. All of our articles are classified and categorized using a long-established proprietary taxonomy. We also dynamically classify and categorize our readers by exhibited behaviors.
If you tend to read a lot of articles I write about AI, for instance, you are scored for the likelihood that you will respond to articles about AI and related topics. We have pretty good feedback loops set up, so the system is constantly attempting to improve your experience. We may be data poor, but with regard to the behaviors of our readers, we have some useful information that helps us on our journey of continuous improvement.
What about the data impoverished?
We create quite a bit of content, and we use it to help us learn how to create better content. We also use it to help us spot trends and to understand what products and services to offer our clients. But what about organizations that don’t have a steady stream of content to measure? Or what if you sell through channels you don’t fully control or that don’t have a way to gather first-party data? How can you possibly compete?
One easy way to get answers to what counts is to simply ask, which is what we did. This survey asks an open-ended question and a few multiple choice follow-ups. While not rigorously scientific from a researcher’s POV, the survey questions are in order of Boolean complexity, and therefore, even forms abandoned mid-survey have remarkable value for our purposes.
What we learned
After several versions of this test, we are ready to more rigorously test the thesis that by asking the right questions in the right order, a small but statistically significant set of respondents will reward you with actionable insights that closely match insights obtained from analyzing much larger, passively gathered data sets.
If you ask the right question…
“What is the right question?” is the right question. Information is not knowledge; a pile of data is just that. In order to make data actionable, you need to learn from it and transform it into knowledge that you can act upon. While massive amounts of quantitative data can be analyzed for correlations, smaller data sets custom-crafted to solve specific business problems can be exceptionally effective. But to take advantage of any data set, you need to start with the right question(s).
What do you need to know? What assumptions do you need to be confirmed? What myths do you need to dispel? What sacred cows need to be desecrated? Which radical approach do you want feedback on? Be specific, and be sure that your target respondents are representative of your target audience. Armed with the right question(s), you are on your way to competing with the best of them.
Data privacy regulation
GDPR is being enforced, CCPA is next and further global and domestic regulation is on the way. These regulations will dramatically impact what data can be passively monitored and analyzed. But data privacy regulation will not impact opt-in research. That will always be between you and those willing to interact with you. So, while you technologically prepare for a cookie-less, highly data-regulated world, spend some time thinking about how to ask the right questions. You’ll be in good company.
