Recent polls in the seven core swing states show an astonishingly tight presidential race: 124 out of the last 321 polls conducted in those states — almost 39% — show margins of 1 percentage point or less.
In fact, the state polls are showing not just an astonishingly tight race, but also an improbably tight race. Even in a truly tied election, the randomness inherent in polling would generate more varied and less clustered results — unless the state polls and the polling averages are artificially close because of decisions pollsters are making.
The results of a poll depend on the opinions of the voters and the decisions of pollsters. Decisions about how to weight polls to match the expected composition of the electorate can move the results of a poll up to 8 points. This is true even if pollsters are making perfectly reasonable decisions on how to weight their survey data, as survey researchers have been forced to consider new methods and ideas for weighting and addressing falling response rates following polling misses in 2016 and 2020.
But the fact that so many polls are reporting the exact same margins and results raises a troubling possibility: that some pollsters are making adjustments in such similar ways that those choices are causing the results to bunch together, creating a potential illusion of certainty — or that some pollsters are even looking to others’ results to guide their own (i.e., “herding”).
If so, the artificial similarity of polls may be creating a false impression that may not play out on Election Day. We could well be in for a very close election. But there’s also a significant chance one candidate or the other could sweep every swing state and win the presidency somewhat comfortably, at least compared to the evenly balanced picture in the polls.
What should we see in a perfect polling world because of randomness?
In a world perfect for polling — a researcher’s paradise in which every voter can be contacted and every contacted voter responds — we can use math to calculate how much variation there should be, because of the fact that voters are randomly selected to take a poll.
If a race in this world was truly tied 50%-50%, the polls would not all produce results that split 50%-50%. Imagine if pollsters in this world conducted 100 identical surveys of 863 randomly selected voters (that’s the average sample size of this year’s swing state polls). The results in 95 of those polls would show candidates getting support somewhere in a range of 46.7% to 53.3% — even though we know in this imaginary world that the race is actually tied at 50%. The other five polls would show the candidates earning something even larger or smaller outside that range.
This variation is known as the “margin of error” in a poll — i.e., how much randomly selecting voters who always respond may affect a poll’s estimate for a candidate.
Because each candidate’s support varies randomly, these polls predict a margin in a tied race that ranges from -6.6 to +6.6 for 95 out of 100 polls (with even larger margins for the other five).
It is important to highlight that the range of margins we can expect in a tied race (and in a perfect world for polling) is much larger than the margins in the swing states in 2020. Even in ideal circumstances for polling, it is difficult, if not impossible, for a poll to be very informative about who is leading a tight race. And this is arguably a lower bound for what we should observe in the messier real world, where polls vary in how respondents are selected, contacted and weighted to match the electorate pollsters believe will turn out in 2024.
It is important to highlight that the range of margins we can expect in a tied race (and in a perfect world for polling) is much larger than the margins in the swing states in 2020. Even in ideal circumstances for polling, it is difficult, if not impossible, for a poll to be very informative about who is leading a tight race. And this is arguably a lower bound for what we should observe in the messier real world, where polls vary in how respondents are selected, contacted and weighted to match the electorate pollsters believe will turn out in 2024.
We can also calculate what share of 863-person polls we should expect to show various margins in a truly tied race. Rounded to the nearest percentage point, roughly 11% of polls in a tied race should show a tie.
About 32% of polls should have a 1-point margin or closer, 55% should have a 2-point margin or closer, and 69% should have a 3-point margin or closer. Even in a 50-50 race, roughly 10% of the polls should have more than a 5-point margin because of inherent randomness — nearly the same percentage that show a (rounded) tie!
With enough polls, the predicted margin should also resemble a “bell curve” normal distribution — with a similar number of polls showing either candidate leading.
What do we see in swing state polls?
Actual swing state polls show far less variation than the benchmarks we would expect in a perfect polling world. Across the 321 polls in the seven swing states, only 9 polls (3%) report a margin greater than 5 points. Even if every race was tied — which they are not — we would still expect to see around 32 of the 321 polls with more than a 5-point margin due to randomness.
Visualizing how the reported polling margins compare to what we would expect in a perfect polling world strongly suggests “herding” of swing state polling margins around the statewide polling averages. In these 321 state polls, 69 of them (21%) report an exact tie and 124 polls (39%) report a margin that is 1 percentage point or less. Both of those figures are roughly double what we would expect in a perfect polling world where the only source of variation is the random selection of voters who respond.
Pennsylvania is perhaps the most troubling state. Fully 20 out of 59 polls there (34%) show an exact tie and 26 (44%) show a 1-point margin or less And even though there is a 15% chance that a truly tied race could produce a poll with more than a 5-point margin due to randomness, we see only 2 of 59 Pennsylvania polls (3.3%) with a margin greater than 5 points.
Even where the polling results are not so tightly clustered, such as in Arizona, Michigan and Wisconsin, there are still far more polls than we would expect around the polling average and too few polls with large margins.
What is going on?
The concentrated margins we see in swing state polls likely reflect one of two possibilities.
One possibility is that pollsters may sometimes adjust a poll result that looks “weird” to them by choosing a weighting scheme that produces results closer to the results of other polls. There seem to be strong incentives for risk-averse pollsters to do so. Unless a pollster is conducting a lot of polls and they can be sure that the impact of randomness averages out, they may fear reputational and financial costs for getting a result wrong due to randomness, since pollsters are graded on their polling accuracy.
A risk-averse pollster who gets a 5-point margin in a race they think is tied may choose to “adjust” the results to something closer to what other polls are showing, lest their outlier poll adversely affects their reputation relative to competitors.
Another more likely possibility is that some of the tools pollsters are using in 2024 to address the polling problems of 2020, such as weighting by partisanship, past vote or other factors, may be flattening out the differences and reducing the variation in reported poll results. The effect of such decisions is subtle, but important, because it means that the similarity of polls is being driven by the decisions of pollsters rather than voters.
And if these assumptions are wrong, something that is unknowable until after the election, then the risk of a potentially sizable polling error increases as the variation in different polls decreases.
Why this matters
The fact that so many swing state polls are reporting similar close margins is a problem because it raises questions as to whether the polls are tied in these races because of voters or pollsters. Is 2024 going to be as close as 2020 because our politics are stable, or do the polls in 2024 only look like the results of 2020 because of the decisions that state pollsters are making? The fact that the polls seem more tightly bunched than what we would expect in a perfect polling world raises serious questions about the second scenario.
The reported polls and polling averages are creating a consensus that the race is going to be very tight and we are likely to see an outcome similar to 2020. Perhaps that is true. It would be wonderful for polling to successfully address the concerns of 2016 and 2020 in 2024.
But the fact that the polls are all reporting such similar margins doesn’t necessarily make it more likely that those margins represent the final outcome. In fact, it raises the possibility that the results of the election could be unexpectedly different than the razor-close narrative the cluster of state polls and the polling averages suggest.
www.nbcnews.com
