The Setting
When conducting market research surveys, reaching enough respondents to capture statistically significant data and craft meaningful insights can be challenging, particularly when it comes to niche audience segments or limited geographic areas. Incentives for participants (including those provided by market research panels) are common and used to motivate participation and help avoid non-response bias. But they also provide a compelling reason for people to try and cheat the system.
Unfortunately, cheaters are very good – and getting better. The reality is, software designed to catch cheaters is slow to advance, while those doing the cheating are working quickly to improve their methods to go undetected. Some simply learn to answer screening questions in a way that allows them undeserved access to more surveys and the accompanying incentives. Others rely on bots to do the dirty work for them, and once the bot gains access to a survey, it submits large numbers of responses to acquire as many incentives as possible, creating a systematic bias in the resulting data. As cheaters improve their techniques, they are also growing in number, with many sharing their strategies and techniques on social media. This can be disastrous for surveys, especially those aimed at small target populations (Ploskonka, 2022).
The Client
One of Market Street Research’s clients is a health system that works with us on a regular basis to conduct a system-level brand health and market intelligence tracking study.
In 2022, the client’s survey was pretty badly hit by bots, resulting in a significant—though sadly not unusual—proportion of bad data. As always, we systematically and thoroughly cleaned out the bad data, ensuring our client would have reliable, accurate market intelligence. Then it occurred to us to consider what the results would look like if we hadn’t identified and removed the bad data, and what (false) conclusions our client might have drawn from it.
What Could Have Been
The results based on the uncleaned data didn’t look pretty. Our client was losing significant ground throughout the consumer journey, with lower top-of-mind awareness, lower levels of consumer familiarity, and lower advertising recall. They were less likely to be seen as having the best reputation for serious or complex care, compared to prior years, and fewer people preferred them for their key service lines. And their net promoter score suggested a possible patient experience challenge as well.
While different from prior years, this picture looked frighteningly believable; especially given their competitive market area, and the lingering effects of COVID. It also suggested a major course reversal was needed, immediately. Only, it wasn’t. Because the problem wasn’t their performance, it was their data.
The Real Story
The real story could hardly be more different. Our client was holding their own well, with the only loss being some of familiarity that had spiked during the COVID-19 pandemic, thanks to the key role they played in supporting their community during that time. They were even gaining ground in a few specific high-value service lines, and their net promoter score was an enviable score of 60.
Our conclusion, and the one we presented, was that their strategies were in fact effective at holding their market position. Because we knew there wasn’t a problem at a high-level, we worked with the client to recognize the parts of their market area where they have both the greatest opportunity and the greatest threat.
The Impact
We saw this project as an opportunity to draw attention to the significant impact bad data can have on a study’s conclusions. Some clients and researchers remain relatively unconcerned about the prospect of cheaters making their way into the sample, reasoning that cheaters are a very small proportion of the results. Hopefully this data demonstrates the danger of doing so.
If our client had acted on the unfiltered results, they would have spent tremendous amounts of money, time, and effort to fix a problem that didn’t exist.
At Market Street Research, we believe bad data is something to take seriously, and we constantly strive to stay a step ahead with our strategies to prevent, identify, and remove cheaters, ensuring our clients can feel good about their research.
To learn more about our approach to data collection and management, contact us for a consultation today.