At The New York Times, Cliff Zukin explains that election polls have become
increasingly unreliable because of rising cellphone usage and declining response rates.
In terms of speed, the growth of cellphones is like few innovations in our history. About 10 years ago, opinion researchers began taking seriously the threat that the advent of cellphones posed to our established practice of polling people by calling landline phone numbers generated at random. At that time, the National Health Interview Survey, a high-quality government survey conducted through in-home interviews, estimated that about 6 percent of the public used only cellphones. The N.H.I.S. estimate for the first half of 2014 found that this had grown to 43 percent, with another 17 percent “mostly” using cellphones. In other words, a landline-only sample conducted for the 2014 elections would miss about three-fifths of the American public, almost three times as many as it would have missed in 2008.
Since cellphones generally have separate exchanges from landlines, statisticians have solved the problem of finding them for our samples by using what we call “dual sampling frames” — separate random samples of cell and landline exchanges. The problem is that the 1991 Telephone Consumer Protection Act has been interpreted by the Federal Communications Commission to prohibit the calling of cellphones through automatic dialers, in which calls are passed to live interviewers only after a person picks up the phone. To complete a 1,000-person survey, it’s not unusual to have to dial more than 20,000 random numbers, most of which do not go to actual working telephone numbers. Dialing manually for cellphones takes a great deal of paid interviewer time, and pollsters also compensate cellphone respondents with as much as $10 for their lost minutes.
THE best survey organizations, like the Pew Research Center, complete about two of the more expensive cellphone interviews for every one on a landline. For many organizations, this is a budget buster that leads to compromises in sampling and interviewing.
The second unsettling trend is the rapidly declining response rate. When I first started doing telephone surveys in New Jersey in the late 1970s, we considered an 80 percent response rate acceptable, and even then we worried if the 20 percent we missed were different in attitudes and behaviors than the 80 percent we got. Enter answering machines and other technologies. By 1997, Pew’s response rate was 36 percent, and the decline has accelerated. By 2014 the response rate had fallen to 8 percent. As Nate Silver of fivethirtyeight.com recently observed, “The problem is simple but daunting. The foundation of opinion research has historically been the ability to draw a random sample of the population. That’s become much harder to do.”
