Polls can't predict who will be our next president

/ Carlos Carmonamedina for NPR Public Editor

/

Carlos Carmonamedina for NPR Public Editor

Pollsters cannot tell us who is likely to win the race for the next American presidency. It's just too close and the polls are too flawed. Journalists cannot bring clarity to murky poll results. So why report on polls at all?

Answering that question is complicated because news stories about poll results are not the same as news stories that mention poll results. The first are low-value horse-race stories. You can easily spot these stories because the headline calls out news of a new poll result, a novel analysis or some shift in voter preferences.

Stories about poll results have a this-just-in quality that makes news consumers feel good or bad, depending on how their preferred candidate fares.

NPR hasn't done many news stories based solely on polls this season. But the rare exception attracted a critique from an audience member who felt she had to work hard to uncover the methodology behind the polls referenced in the story. We contacted the journalist who reported the story, who also happens to be NPR's senior political editor, to discuss journalistic standards related to political polls. You can find his insights below, in our response to this audience member's critique.

The profession of political polling has had some serious challenges in the last few presidential cycles. Pollsters believe that when former President Donald Trump is on the ballot, he draws in some voters who are not captured in polls. While the misses have been exacerbated by what is called the Trump effect, they are not unique to his candidacy. In 2012, polls understated support for President Barack Obama's bid for a second term. Back in 1980, polls underestimated the Ronald Reagan landslide.

In 2016, state-level polls were significantly flawed, and did not accurately reflect the possibility of a Trump electoral college win. In 2020, it was even worse. The polls were off even more, overstating the margin of support for Joe Biden by more than 4%, which was largely outside the margin of error. It was the worst miss for polling since 1980.

Every pollster calculates a margin of error, which is meant to account for flaws in the math as pollsters try to weight their group of respondents in ways that will better mirror the general voting population, Courtney Kennedy, vice president of methods and innovation at Pew Research Center, told me. (She's helped craft advice for journalists reporting on polls.)

"The margin of error is a really flawed metric. It's too low. It gives people a falsely optimistic sense of accuracy," she said. "The margin of error only addresses sampling errors. But there are other error issues, like the fact that Trump turns out supporters that are less likely to participate in polling."

While pollsters wrestle with the shortcomings of their work, journalists have had a similar reckoning. Kennedy reminded me of the prevalence of probable forecasts, which were all over the media up through the 2016 presidential election.

This was primarily an innovation of Nate Silver's FiveThirtyEight blog. Journalists mushed all the polls together to extract more meaning and then presented news consumers with handy graphics. They still exist, but they aren't nearly as prominent in the coverage these days. And NPR stays away from them.

"I would say there's a lot more humility in 2024 than there was in 2016" among journalists, Kennedy said.

Of course, there's only one poll that matters. That's the vote count that starts on Tuesday, Nov. 5. It will probably take several days to complete and even more to clear legal challenges.

In addition to responding to the audience question on the recent poll story, we highlight a useful story that teaches people how to avoid sharing misinformation as we enter the presidential transition.

Also, a quick note: I'll be live on Reddit for an Ask Me Anything on Wed., Nov. 6, 2024, at noon ET. I'll be answering questions about our work as NPR's Public Editor, so please join me. We've also revived our social media presence, so follow us on Instagram, Threads and Facebook.

/

/

Illustration by Carlos Carmonamedina

Searching for basic information about political polls

NPR senior political editor and correspondent Domenico Montanaro told us that this Oct. 16 article was a rare exception for NPR. He told us he relied on polls for this analysis to demonstrate how the small changes could impact the electoral map.

Because the polls are so close and historically flawed, he and his colleagues have steered away from reporting many stories based only on polls. However, after he noticed slight movement, not just in single polls but in the aggregators as well, he wanted to take a closer look. His resulting analysis told readers and listeners that Vice President Kamala Harris was losing momentum and Trump was showing small gains. It was accompanied by an extensive set of graphics that illustrated the electoral map, the slight changes in the swing states and where each candidate stands on their quest to get the 270 electoral votes they need to be president.

A small credit tells readers on the website that Montanaro's analysis was rooted in the work of aggregator sites FiveThirtyEight and Decision Desk HQ/The Hill. There were links to both news sites at the bottom of the story, below the last graphic. There was a short explanation describing how NPR determined which states were leaning Democratic or Republican. But there was no note describing why NPR selected these two poll aggregators.

The goal of this story was to let listeners know that the polls are so close that they will not provide any insight into the likely winner.

"First of all, our listeners should know that the race is close. And that's really all you need to know. Right?" he said. "Like refreshing FiveThirtyEight or (Decision Desk HQ) or any other aggregator is not going to change the outcome of this election."

Montanaro told me that, after years of experience, he has finely tuned his personal trust of pollsters and political journalists. He first looks for two big signals. He wants to see the margin of error and he shies away when pollsters use decimal points in their reports. Decimal points imply a level of statistical confidence that is just not possible, he said. He also looks at reputation and tracks reliability.

I wish he'd included this information in his stories. An editor could have suggested adding an explanation of how he picked his sources, and what their methodologies are.

Given the shaky recent history of political polling, NPR audience members deserve that information. It could have been provided in the text of his story on NPR's website, in the credits to the story, or in a small note. He could have even explained why NPR doesn't do many stories on poll results.

On the air, the host could have asked him how he selects reliable polls.

Montanaro disagrees with me on this point. "I have explained many times on the air how to read and interpret polls," he said. "But it's not realistic, I don't think, to include that level of detail every time we do a story related to polling."

These small signals of transparency build trust between newsrooms and news consumers. When they are missing and news consumers have to hunt them down, trust is eroded.

Even though polls aren't very helpful in predicting who will win the presidency, they are an important tool in a democracy. In fact, public opinion polling is banned in many countries where leaders don't want people to have evidence of popular beliefs.

"Polls are an independent check on people in power," Kennedy from Pew Research told me. "That's easily overlooked when polls don't get the election right."

But it doesn't make it less true. If both polls and journalism are to continue as a check on power, the public has to have faith in how they work. That will happen only with transparency.

Illustration by Carlos Carmonamedina /

Educating consumers on voting misinformation

NPR recently published a story to help consumers understand how to spot election week misinformation, so they don't inadvertently share it with others. The article explains how misinformation is often disguised by emotionally charged language. Readers can find tangible tips on how to pause, ask questions and fact check what they are listening to, reading or watching. The helpful suggestions also cover issues with edited photos, videos and audio. Simple acts of service journalism play an essential role in educating consumers. Yet they are often overlooked by outlets. With this piece, NPR zeroed in on an important topic at exactly the moment when people need it. They also distilled the article into a handy Instagram post. — Nicole Slaughter Graham

The Office of the Public Editor is a team. Reporters Amaris Castillo and Nicole Slaughter Graham and copy editor Merrill Perlman make this newsletter possible. Illustrations are by Carlos Carmonamedina. We are still reading all of your messages on Facebook, X and from our inbox. As always, keep them coming.

Kelly McBride
NPR Public Editor
Chair, Craig Newmark Center for Ethics & Leadership at the Poynter Institute

Copyright 2024 NPR