One question that some clients have been concerned about in recent years is whether we can trust the results from quantitative studies.
It’s a good question. In the early days of Market Street Research, virtually every household had a landline. We identified households to contact using random digit dialing, in which telephone numbers in a specific geographic area were generated at random, so we included both listed and unlisted telephone numbers. Thus we had a scientific sample of households with landlines, meaning that every household with a landline had an equal chance of participating in the survey. In addition, response rates (the proportion of households we attempted to contact who responded to the survey) were very high—over 70%.
Those days are long gone. Caller ID made it possible for people to screen out calls from people they didn’t know, and the plethora of calls from sales people, fundraisers, political candidates, and pollsters made people hesitant to answer the phone, and if they did, they were much less likely to participate in a survey. Then, as the use of cell phones proliferated, the proportion of households with landlines plummeted, and is currently under 50%.
On the positive side, about 90% of the U.S. population now uses the internet, which approaches the proportion of the population that used to have landlines. And many companies have developed panels of internet users who have agreed to participate in marketing research, providing researchers with a much faster and less expensive means of collecting data.
But is it accurate? I was skeptical myself when those panels were first developed. Were the characteristics of people who agreed to be on panels similar enough to those of people who hadn’t to be confident in the findings? Fortunately, we had many clients who were interested in exploring that question to see if they could move their tracking surveys from the phone to online. We developed a split methodology for those clients, collecting half the responses by phone and half online. We found very few significant differences between the responses of people who completed the questionnaire by phone vs. online, and the comparison of the results from prior surveys was also consistent.
Despite that experience, there has been more skepticism about survey findings in the last couple of years, with many people believing that the polling for the 2016 Presidential election was inaccurate. The surveys, however, were highly accurate: The polls indicated that Hillary Clinton would win the popular vote by 3% and she won by 2%. Pew Research Center, which is a highly respected research organization, has an interesting video about the accuracy of surveys, looking at the accuracy of survey findings from a number of different perspectives: Can we still trust polls?
So while it’s highly unlikely we will ever return to the days where researchers have access to nearly every household in the country, the methods we have can be used with confidence to make decisions.