Elections have consequences for parties—and now, for polling.
An industry accustomed to unquestioned respect that had struggled quietly against its mounting demons for the previous few election cycles is facing an intervention post-2012. A decades-old method of gauging a person’s likeliness to cast a vote for president failed. The resulting gap between some pre-election ballot tests and the actual outcome shook those pollsters including the oldest brand in the business, Gallup.
A robopoll—an automated survey involving no person-to-person contact—mirrored the final results as closely as any set of live interviews.
And by offering a shortcut through the glut, Nate Silver and other poll aggregators became what pollsters once were, our national tea-leaf readers, while diminishing the value of accurate individual surveys.
Pollsters, meet Jesus.
But the wake-up call doesn’t end there. The industry is undergoing only its second methodological overhaul in 50 years, the last one being a luxuriously years-long switch from in-person interviews to landlines. The upheaval has spotlighted cultural and technological shifts that are challenging the very premises of our national public affairs polling in a way the first overhaul didn’t: that people’s opinions are best collected by having conversations with them, and that it’s possible to create truly random samples that are methodologically sound.
Within a span of five years, response rates—the percentage of people contacted by a pollster that actually completes the survey—have plummeted. Pollsters have had to adjust to rates that have dropped from 25 percent, to 10, to single digits.
Roughly one-third of American adults now live in cell-only households. The accepted percentage for cellphone interviews in a credible survey is about 34 percent and rising fast.
Pollsters have not been caught flat-footed—except on the failure of the likely voter model—by the changes that have buffeted their industry over the past few years. But America’s business of national public affairs polling was built by individuals with their own particular academic training and experiences that guide how they each pursue their work. Beyond following certain standards meant to ensure sound polling, they don’t exactly trade notes. “Someone tends to be proven right, someone tends to be proven wrong,” says NBC News/Wall Street Journal pollster Bill McInturff of the GOP firm Public Opinion Strategies. The only way to know is to “start doing more work.”
Polling the professionals on how the industry will respond to its challenges by 2014 or the next presidential cycle is an interesting survey in and of itself.
In “sampling”—creating my sample—for this exercise, I screened for pollsters who do significant publicly available work. Campaign pollsters face many of the same challenges, but the cause of most of the heartburn has been the public polling on which the news media and Americans rely.
That said, this didn’t preclude my speaking with some of the most experienced political pollsters thanks to the patchwork of sources of our body of public affairs research:
- Global market research firms that do it for the PR boost (GfK for the Associated Press, Opinion Research Corp. for CNN, Ipsos for Reuters);
- Academic institutions that do it for PR and scholarly work (e.g., Quinnipiac College);
- Small nonpartisan firms that specialize in public affairs research (e.g., Princeton Survey Research, which polled for the late Newsweek; Selzer & Company for Bloomberg News);
- Bipartisan pairs of political pollsters that serve as a check on each other (e.g., Hart/McInturff for NBC News and The Wall Street Journal);
- In-house operations (Pew Research Center, ABC News and the Washington Post, CBS News and The New York Times); and,
- Robocall firms.
The Response Rate/Cellphone/Likely Voter Knot
When was the last time you were sitting at home after dinner, answered an unknown caller and stayed on the phone for 20 minutes? Yet as YouGov pollster Thom Riehle points out, that’s the Ward Cleaver-esque scenario on which 21st-century polling still depends. “We are in single-digit territory now—a lot of it, low single digits,” warns Pollster.com founding editor Mark Blumenthal of the percentage of people reached by pollsters who are willing to complete a survey.
The lower the response rate, the more weighting is required to create a sample that’s representative of registered or likely voters nationwide. Reaching a male adult under the age of 30 today is nearly impossible. Riehle says the rare one who does answer his cellphone and actually completes a survey is basically being given the proxy of five, six or even seven others.
People who rely solely or heavily on cellphones are harder to find; when found, are harder to keep on the line; and are more expensive to poll. More people have unlimited calling plans today, but pollsters still sometimes repay respondents for their minutes used. Federal law forbids auto-dialing cellphone numbers so every cellphone must be dialed by hand, which adds to labor costs.
The likely voter model faltered in 2012, McInturff finds in a new study, because voters who turned out to support President Obama in unexpectedly high numbers were the same types of voters who are most likely to use cellphones and thus are most difficult to poll: young adults age 18-29 and Latinos.
The cellphone genie is out of the bottle for good. The floor for the accepted percentage of a sample that is polled by cellphone can only rise.
12 Steps to Recovery
“The death of polling is being greatly exaggerated,” asserts Charles Franklin, director of the Marquette Law School poll and political science professor at the University of Wisconsin. It’s easy to feel bogged down by challenges that seem so clear in contrast to a future that’s anything but. I put the question to the pollsters: how to get people to take and finish surveys, and how to create samples that are representative? And with no time to waste, preferably all at once?
1. Cellphones are a problem taking care of itself, Franklin thinks. Pollsters just have to wait it out, if uncomfortably, while more people adopt the cell as their only phone. If America goes 100-percent cell, he suggests, the only difference from the days of landlines will be where you are if you take the call, and do your activities or your coverage interfere with your ability to complete a survey.
2. McInturff sees phones retaining their primacy in the near future of public polling, assuming pollsters keep pace with the growing reliance on cellphones, but thinks phones will be augmented with web-based polling to look more closely at key audiences. After a string of polarizing national elections with “not a lot of people left in the middle,” he suggests, “it may make sense to use the Internet” to help track where they’re headed.
3. Other pollsters agree that web-based polling will soon be let in from the cold. My colleague Michelle Harrison, head of political and social research for Kantar in the UK and a BBC-registered pollster, points out that “online polling for voting intention work has been very accurate” in recent UK elections. “If panels are being well maintained and looked after, and if previous voting behavior is known, [web-based] is coming through as a very accurate way of polling.”
4. Public affairs polling could become device-neutral, offers Fred Yang of NBC/Journal Democratic co-pollster Hart Research. “At some point, we have to stop using these terms of ‘online, etc.’ and just call them ‘polls.’” People will participate orally by landline or cellphone, whether by responding to a live or computerized interviewer or a robopoll, or by texting, emailing or visiting a website.
5. Polling will become 24/7, says Riehle. The more platforms you offer to potential respondents, the easier you make it for them to respond on their own time. Even just for phone banks, interviewing hours need to change, says Peter Hart of Hart Research. “We continue to have the same hours of interviewing we did 20 years ago, 5 PM to 9 PM, and the weekends,” he said in remarks at a Roper dinner last November.
6. To support this, pollsters will be able to create samples that come with not just phone numbers but email addresses, Twitter handles, etc., Franklin adds. Professional sample providers can now create such lists for large blocs of the population such as registered voters. Having multiple means of contact may improve pollsters’ chances of reaching potential respondents and provide people with choices for how to respond. Certain privacy and regulatory concerns currently stand in the way, Franklin cautions, and it still doesn’t guarantee responses. But it’s a potential path to more representative samples.
7. Peter Hart also thinks the trend toward close-ended poll questions will shift toward more open-ended questions because of the storytelling power of people speaking for themselves. “Answers to complex and important problems have been reduced to choosing between Option A and Option B,” he said last November.
Changes on the sponsoring news media side may drive changes in polling, too. Tightening budgets and the horse race’s total dominance in driving viewers and traffic may prevail in making some polls shorter. By this point, the popularity of their horse-race numbers are “probably what’s justifying the expenditure [on polls] by news organizations,” per Blumenthal.
8. But maybe polls won’t just get shorter—maybe they’ll be rebuilt. If polls are going to shrink, why not split the traditional politics-and-policy questionnaires? When was the last time public opinion on a tax policy issue was the focus of a breaking news alert? In the UK, not only are candidate trial heats conducted online, but a lot of “social research” polling is still done face to face. Per Kantar’s Harrison, the response rate for in-person social research polling in the UK can be between 60 and 75 percent.
9. On the other hand, 2012 highlighted demographic shifts that national news brands, not just politicians, need to heed for the sake of diversifying their audiences. At FOX, Roger Ailes is talking openly about courting Latino viewers. NBC now owns Telemundo and African-American news site theGrio.com. ABC just made President George W. Bush’s polling director Matt Dowd a special correspondent “to explore uniquely American cultural trends.” For these news divisions, a better way to cut the polling budget may be to continue with longer surveys but do what campaigns do: spend the money closer to the election.
10. As the universe of people willing to respond to polls shrinks, and the number of polls grows, brand and identity may be increasingly key to producing quality surveys. Pollsters keep their response rates close to the vest, but Public Opinion Strategies found “significantly higher” rates for the NBC/Journal polls it co-conducts than for other national surveys the firm has done. At the same time, people are generally thought to be more likely to complete long live interviews than long automated ones.
11. In the pipedream category, national polls may integrate Big Data. Campaigns already merge their polling with market research (see Obama, Barack). As global consumer research companies play a bigger role in subsidizing America’s public affairs polling, why not get them to see it as more than just a pro bono obligation? Have them bring their wealth of experience in examining how people make choices about politics and policy to bear on our national survey research.
12. In short, our national public polls of 2014 and 2016 seem likely to include data derived from two or more methodologies: live phone interviews, web-based interviews and automated interviews. The growing array of options means the industry will become even more eclectic, with pollsters devising their own methodologies to produce either a single set of results or sets of multiple scenarios. Which means, for better or worse, that poll aggregators will be valued even more for averaging polls of varying methodologies to produce a rolling consensus.
Anyone can become a pollster. That sounds like a bad joke and also happens to be true. As Hart has noted, unlike lawyers, accountants, realtors and even roofers, polling requires no credential. It has no accreditation group. And for every bad poll, says ABC News polling director Gary Langer, there’s a platform and an audience. (And multiple aggregators.)
But dedicated public pollsters who sweat out the challenges and keep the profession current are what will make the difference between a country that is increasingly diverse yet self-aware and one afflicted by proliferating numbers but a paucity of insights.
Elizabeth Wilner, VP Kantar Media Campaign Media Analysis Group (CMAG), is Contributing Editor at the Cook Political Report
February 5, 2013The 113th Congress: Brought to You by 800+ Advertisers
January 29, 2013DC TV’s Challenge: Moving Beyond Alphabet Advertising
January 11, 2013Ads About Guns: The Silence in the Air