Election Polls and the Price of Being Wrong

The thing about predictive analytics is that the quality of a prediction is eventually exposed — clearly cut as right or wrong. There are casually incorrect outcomes, like a weather report failing to accurately declare the time it will start raining, and then there are total shockers, like the outcome of the 2016 presidential election. In my opinion, the biggest losers in this election cycle are pollsters, analysts, statisticians and, most of all, so-called pundits. I am saying this from a concerned analyst's point of view. We are talking about colossal and utter failure of prediction on every level here. Except for one or two publications, practically every source missed the mark by more than a mile — not just a couple points off here and there. Even the ones who achieved “guru” status by predicting the 2012 election outcome perfectly called for the wrong winner this time, boldly posting a confidence level of more than 70 percent just a few days before the election. What Went Wrong? The losing party, pollsters and analysts must be in the middle of some deep soul-searching now. In all fairness, let’s keep in mind that no prediction can overcome serious sampling errors and data collection problems. Especially when we deal with sparsely populated areas, where the winner was decisively determined in the end, we must be really careful with the raw numbers of respondents, as errors easily get magnified by incomplete data. Some of us saw that type of over- or under-projection when the Census Bureau cut the sampling size for budgetary reasons during the last survey cycle. For example, in a sparsely populated area, a few migrants from Asia may affect simple projections like “percent Asians” rather drastically. In large cities, conversely, the size of such errors are generally within more manageable ranges, thanks to large sample sizes. Then there are human inconsistency elements that many pundits are talking about. Basically everyone got so sick of all of these survey calls about the election, many started to ignore them completely. I think pollsters must learn that at times, less is more. I don’t even live in a swing state, and I started to hang up on unknown callers long before Election Day. Can you imagine what the folks in swing states must have gone through? Many are also claiming that respondents were not honest about how they were going to vote. But if that were the case, there are other techniques that surveyors and analysts could have used to project the answer based on “indirect” questions. Instead of simply asking “Whom are you voting for?”, how about asking what their major concerns were? Combined with modeling techniques, a few innocuous probing questions regarding specific issues — such as environment, gun control, immigration, foreign policy, entitlement programs, etc. — could have led us to much more accurate predictions, reducing the shock factor. In the middle of all this, I’ve read that artificial intelligence without any human intervention predicted the election outcome correctly, by using abundant data coming out of social media. That means machines are already outperforming human analysts. It helps that machines have no opinions or feelings about the outcome one way or another. Dystopian Future? Maybe machine learning will start replacing human analysts and other decision-making professions sooner than expected. That means a disenfranchised population will grow even further, dipping into highly educated demographics. The future, regardless of politics, doesn't look all that bright for the human collective, if that trend continues. In the predictive business, there is a price to pay for being wrong. Maybe that is why in some countries, there are complete bans on posting poll numbers and result projections days — sometimes weeks — before the election. Sometimes observation and prediction change behaviors of human subjects, as anthropologists have been documenting for years. Marketers Get It Wrong, Too Nonetheless, the consequence of a wrong prediction is definitely not limited to large, highly-publicized cases like this election outcome. Marketers, some of the most casual users of predictive analytics, often pay dearly for erroneous predictions. For instance, if the prediction of unit sales were off by a mere 5 percent to 6 percent in a billion-dollar company, the error may result in some dire consequences. The company would be in trouble even if its sales projection is underestimated, as such a conservative estimate will lead to under-production, affecting everyone in the supply chain. Conversely, over-projection may lead to some serious inventory problems. Often, the predictive analytics business is not just about prediction; it must include all of the investigative work in case the projections are off, even by a few percentage points. When there are decent model algorithms in place (i.e., model-based forecasting with a clear list of predictive variables), marketers will have a good place to start the investigation — variable-by-variable, if necessary. This is one of the benefits of predictive analytics that is rarely discussed. Simply maintaining a list of predictors helps decision-makers narrow down the search, even when the prediction turns out to be less than ideal. Conversely, I’ve seen many marketers going through a series of wild goose chases based on gut feelings and guesswork, when encountering failure of their rudimentary sales projections. Such not-so-scientific investigation methods often lead to blame games, without yielding usable answers. In 1:1 marketing, which is much more personal than mass marketing, marketers must consider the price of wrong predictions at all times. In a simple case of response modeling, marketers are generally happy with a 1 percent response rate, and may be ecstatic with a 2 percent conversion rate in a campaign. But even in such cases, we should never forget that they were wrong for about 98 percent of contacts. That is why I always consider the price of being wrong when faced with predicting seemingly sensitive subjects or products. For instance, ailment data is notoriously difficult to obtain. And even with the hard data, pushers of marketing campaigns must be really careful, as they may face questions like “Hey, how did you know that I am diabetic?” That could be the case even when dealing with known, explicit data. Imagine someone actually built a propensity model, such as “Likely to be diabetic,” and ended up using such a prediction to promote medical products. Going further, how about “Likely to suffer from erectile dysfunction”? Users of such a sensitive-subject projection — even based on sound mathematics — shouldn’t just be counting on the incremental revenue that such predictive modeling would bring to them; they must also think about the kind of backlash from the inevitable wrong prediction. Modeling is about increasing the probability of being right, not having a mirror ball that guarantees 100 percent accuracy. Yes, it is the analyst’s job to minimize the errors using only available datasets. But predictions can go wrong for all kinds of reasons — inadequate input data often being the No. 1 reason for inaccurate outcomes. Predictions can be wrong; at times, quite wrong, as we’ve just witnessed. That awareness may save the users from total shock, while some disappointments may not be completely avoidable. Nonetheless, we should not give up on predictive analytics altogether, as less-than-perfect estimates are still far better than complete obliviousness. We all get soaked in rain thanks to inaccurate weather forecasts once in a while, but would you ever give up on them entirely?

Data

Election Polls and the Price of Being Wrong

Stephen H. Yu

About

Subscriptions

Events

Publications

WORK SMARTER - LEARN, GROW AND BE INSPIRED.

Subscribe today!

Stephen H. Yu