Can I tell you a secret? The average person really has no idea how to make sense of election polls. Okay, in all honesty, you probably already knew that. The real secret? Many of the so-called experts and talking heads really have no idea either.
And that’s the problem. It’s so easy to get caught up in your own personal cycle of bad information and even worse predictions. In fact, this entire cycle is perpetuated by our unwillingness to take the long view and stand up to tell polls and predictors that they’re wrong. And they are so often wrong, as anyone following the 2016 presidential race can attest.
In fact, if you look back to my previous article on little-known facts about political betting, you’ll see how so-called experts aren’t really any better at making major predictions than flipping a coin. So what’s stopping you from improving on them? Simple: understanding how polling really works.
The Basics of Presidential Polls in 2016
How do polls work? It seems simple enough: The organization conducting the poll contacts people, asks them what they think and records the results. But this process actually contains an enormous number of variables. Understanding just what those variables are is the key to really understanding polling and how to use it.
Sample Size
Imagine you’re trying to determine which of the 2016 Republican presidential candidates is going to win in California. The population of that state was 38.8 million in 2014. But is that the population the polls are really sampling? No, because no pollster is interested in who an eight-year-old supports in a presidential campaign. There are two possible groups they will be sampling: registered voters and likely voters.
What’s the difference between the two? Registered voters are pretty straightforward, and in California’s case, that group is 17.2 million people, as of 2016. That’s less than half of the total population, so knowing that’s the group you’re targeting makes a huge difference. If you’re only looking at likely voters (those voters determined to be most likely to vote, as plenty of peolpe are registered but don’t often comeregistered voters never make it to the polls), that number is closer to 6 million (, using data from the California Elections Board and Public Policy Institute of California).
Generally, you want to give more weight to polls that target likely voters, for two reasons. Firstly, because that sample size is smaller, you can get more accurate estimates with a similar sample size. Secondly, you’re generally getting a more accurate picture of the opinions of people who are likely to get out and vote on election day, instead of people with opinions who will stay home.
For example, let’s say you want to poll likely voters in California to see which candidateso they support in the 2016 campaign. If your target group is 6 million people, let’s say you want your poll to be 95% certain, with a confidence interval,  (also commonly known as the margin of error), of 3 (meaning that you’re 95% sure the true numbers will be within 3% plus or minusof your number). To do that, you need a sample size of 1067. If you want to play around with these numbers you can check out this calculator.
So when you’re looking at a poll, take a look at the sample size, investigate their target group, and see for yourself how confident they should be in their numbers based on that information. You may not see all of this kind of information inIf you’re seeing poll results in a news article, but if you may have to go to the source of the poll, it’s usually available. to find this info.
Recency
This one is pretty simple. The more recent a poll is, the more relevant it is and the greater weight you should give it. Importance declines exponentially in most advanced polling outfits starting from the day the poll is released.
Polling Method
Here things get complicated again. First, how did the pollsters reach their audience? Was it by phone? In that case, you have to ask (as all pollsters do these days) who even has a landline anymore? It’s important to note here that polling isn’t permitted with cell phone numbers.
What these methodological restrictions mean is that younger people and the poor, to take two examples, tend to be underrepresented in polls. Why? Simple: They’re less likely to have a landline phone. For the same reason, the elderly tend to be overrepresented. This can be very important when evaluating specific candidates, as I’ll explain later.
Even more tricky is the wording of the questions. Some polls may be done with an agenda, in which case they can word their questions to encourage or discourage a particular answer. You can either take a look at the questions asked in the polls, or try relying on polling firms with good reputations.
Election Poll Biases: Lies, Damned Lies, and Statistics
So how do you determine whether a particular polling firm has a good reputation? Sure, you could try to analyze who owns it, what their motivations are, look at the wording of their questions – but there’s a much simpler answer. Just look at their track record. This method cuts through all the other potential issues to get to the heart of how effective they are at making accurate predictions.
Many poll aggregation websites do this for you. That’s why one of the most effective ways to get reliable polling information is to refer to several trusted sites and familiarize yourself with their methodology.
Why rely on aggregators? Well, they rely on the same thinking as polls themselves. Think of them as a poll of the polls. By combining information from many polls, in effect, they’re increase the sample size and reducing the effect of any specific errors in any single poll. The trick is aggregating those polls in the right way. Here are some examples:
RealClearPolitics
One of the most popular polling aggregation and political analysis websites out there, RCP provides clear and transparent lists of polls as well as high quality interactive aggregations. Bear in mind, it’s only updated when the latest polls come in, so the averages may be as much as a week behind at any given point.
FiveThirtyEight
This site is considered the gold standard within the world of US politics and sports. The brainchild of master wonk Nate Silver, the site began its life as a blog under the aegis of the New York Times in March of 2008. Almost immediately after he began blogging about polls, Silver made a name for himself with his astonishingly accurate predictions during the 2008 presidential primaries.
Since then, the reputation of FiveThirtyEight has only grown, particularly when it accurately predicted the winner of all 50 states in the 2012 presidential race. So how do Silver and his team do it? What makes their predictions so much more accurate than other poll aggregators
Throwing Weight Behind Your Poll Aggregations
This is the technique that separates the amateurs from the professionals when it comes to election polls. It brings together all of the elements of polling discussed so far, using weighted poll numbers. Before, I mentioned the polling date as an important factor for how important a poll is when determining an aggregate number.
Weighing polls is another way of doing this, except that this weighting is based on other factors. Namely: the pollster’s historic track record, in addition to the recency and sample size. However, there’s one more metric which gets thrown into the mix and which is becoming increasingly important: demographics.
Polling Like a Demographer
The classic example of the importance of taking demographics into account when looking at 2016 presidential polls is Bernie Sanders’ historic upset in the Michigan primary. Sanders was behind Clinton by around 20 points leading up to the vote, and yet he won by almost 2%. So how on earth did that happen?
There’s been much speculation, but demographics undoubtedly played a role. The economic, racial, and other demographic factors in Michigan matched other states in which Sanders had performed well. But those changes just weren’t picked up in the polling. So far, this has been the main explanation for the huge discrepancy.
But on that note, FiveThirtyEight offers both sStraight-up poll aggregations and estimations, like those offered by FiveThirtyEight, as well as estimations that combine those e numbers with demographic predictors based on past performance. As polling becomes more and more difficult, many professional poll takersrespondents and poll watchers see an increasingly important role for this kind of demographic-assisted analysis in the future.
Putting it All Together: How to Evaluate Polls Like a Pro
This is a lot to digest, but there are some key takeaways.
1. Always be skeptical.
One of the biggest mistakes made in using polls is taking them at face value. Turn on any major news channel and you’ll hear anchors and talking heads quoting the latest poll like it’s gospel. And those folks are usually wrong, remember?
2. Use reliable poll aggregation sites.
These sites are full of people who are passionate about data, numbers, and accuracy – and not about a particular political party. But familiarize yourself with their methodologies too; don’t just take them at their word either.
3. Check your answers.
When the real numbers come out, see how good your best predictions were. If they were wrong, dig into why. This is what helps aggregation sites decide which polling firms to trust, and it should help you decide which instincts and techniques of your own you should put faith in.
Want more analysis on the latest polling data? Check out our political news section for more insights.
