Opinion polling's identity crisis
As the midterm elections loom, the political world is doing what it does best: pointing to the polling numbers with either pride or alarm. In any election year, pundits can pick and choose polls on either side of the aisle to boost their argument. But why have so many polls been so far off the mark lately? It’s because there’s a new wild card that impacts even the best pollsters: cell phone penetration. As Cliff Zukin, a Rutgers University professor of public policy and past president of the American Association for Public Opinion Research, wrote in the New York Times in 2015, “Election Polling is in near crises, and we pollsters know.” The problem is twofold, attributed to both the growing popularity of cell phones and the decline in people’s willingness to answer surveys, according to the article. These trends have ramped up the cost of polling, and the situation has only grown worse in the past few years.
In 2005, the U.S. government estimated that only about 6% of Americans used cell phones in place of landlines. But by early in 2014, the estimated number had grown to 43 percent. By the spring of 2017, it was 50.8 percent. Zukin concluded: “In other words, a landline only sample conducted for the 2014 elections would miss about three-fifths of the American public, almost three times as many as it would have missed in 2008.”
In the 21st-century environment, those who end up being polled are older Americans and those in rural areas with poor or no cell service. Polling companies recognize this and have taken steps to remedy it by ramping up cell phone polling and using the Internet to conduct polls. The Pew Research Center states: “We sample landline and cellphone numbers to yield a combined sample with approximately 25% of the interviews conducted by landline and 75% by cellphone.” Pollsters generally strive to complete 1000 to 1500 interviews of registered voters to achieve an accuracy within 3%.
Complicating factors further drive up the cost. One must consider how many cellphone “hits” pollsters must get to find a registered voter age 18 or older who is willing to be surveyed, as opposed to getting confused 13-year-olds. That leads to a growing decline in the response rate, which Zukin says has dropped from 80% in the 1970s to 8% in 2014. Pew currently reports response rates of 5% to 15%.
In the good old days, pollsters could buy landline numbers sorted by criteria such as zip codes, ethnicity, etc. They often used automated “robocalls” to do the dialing, and the interviewer got on the line only if someone answered. But the Federal Communications Commission has interpreted the 1991 Telephone Consumer Protection Act to prohibit autodialing of cell phones, meaning they must be dialed manually.
Zukin says “To complete a 1,000-person survey, it’s not unusual to have to dial more than 20,000 random numbers, most of which do not go to actual working telephone numbers.” Manual dialing costs pile up, especially if pollsters offer respondents $10 for their lost minutes. Some pollsters admit that interviewing costs have more than doubled between 2008 and 2016.
Another issue is that respondents tend to overstate their plans to vote, especially in midterm elections. Pollsters are grappling with this by “weighting” the respondent’s answers by data like household size, which can be gathered from theU.S. Census. Dan Cassino, an associate professor of political science at Fairleigh Dickinson University, says in a 2016 Harvard Business Review Article “In the simplest form, a pollster might find that only 6% of the sample is African-American, compared to 12% of the American public.” To weight, a pollster might count the response twice towards the overall results.
Internet polling has also come on the scene but presents similar problems. About 16% of Americans do not use the Internet, most often members of the older population. And attempts are being made to reduce phone costs by using automated Interactive Voice Response, but response rates trend lower.
One surprise in the world of polling is how little has been written about these issues by mass media since 2016. After all, if the pollsters and pundits were on target in 2016, the president of the United States would be named Hillary. Polling remains at least as much of an art as a science.