Consistency of Consumer Responses in Qualitative and Quantitative Research Methodologies
As a market research firm that’s been in business for over 40 years, one of the questions we often hear from clients is how much they can rely on the information they get through focus groups to make important decisions. They know they want the level of depth and the quality of insight focus groups provide, but they can’t help feeling a little anxious about the results—can they really trust what’s being said in a room among a small group of consumers?
It’s true that focus groups are a qualitative research method and that the findings will not necessarily be representative of the total population. Yet, in our many years’ experience conducting market research projects in which an initial qualitative study is followed up with a quantitative study, the findings we get from focus groups are highly consistent with the findings we get from quantitative surveys of the same population.
In 2013, we conducted a study for NYU Langone Medical Center that vividly illustrated how much our clients can trust what happens within a focus group. NYU Langone Medical Center was in the process of branding a service line and to this end, it wanted to understand perceptions of its brand and test consumer response to a variety of branding materials, including a brand description, brand messages, and imagery, and potential names for the service line.
We began the study with 6 focus groups of area residents. The groups included patients of NYU Langone and competing hospitals who had used this service line as well as residents who had not. We then conducted a survey of 1,000 area residents with the same overall characteristics. In both methodologies, the focus was on reviewing the branding materials, including identifying residents’ favorite and least favorite materials.
This project provided us with a unique opportunity to assess the consistency between focus groups and a large-scale survey. Typically, the information we get from focus groups is non-numeric: people talk about their impressions, emotions, and experiences in a way that does not fit into a Likert scale. For this project, however, in addition to the in-depth conversation about the materials, focus group participants were asked to complete worksheets in which they identified their favorite and least favorite of a set of materials. At the conclusion of the groups, these worksheets were tallied and included in our analysis. These exercises provided us with numeric information similar to that which we obtained through the online survey.
The final analysis revealed an extraordinarily high level of consistency between responses from focus group participants and the 1,000 survey respondents. For example, among four brand messages reviewed, in both the focus groups and survey residents preferred Message A, with 52% of survey respondents and 54% of focus group participants selecting this as the most appealing of the four messages. Message D was considered the least appealing brand message by 41% of survey respondents and 43% of focus group participants.
The results were particularly interesting when we looked at the naming exercise. We asked residents to identify their favorite of nine possible names for the service line. [1] Not only did focus group participants and online survey respondents again select the same preferred name within a few percentage points of each other—37% of survey respondents and 33% of focus group participants preferred Name A—but their order of preference was remarkably consistent, down to their least preferred names.
The results of this multi-phase project were very useful for NYU Langone. They had strong survey data, a good understanding of the relative appeal of the materials they were working with, and could feel confident about the feedback they were getting. Importantly, they had the depth and insight from focus group participants as well. The online survey told them how many people found the particular names and messages appealing, and the focus groups helped them understand why certain materials were more and less appealing, and how they could strengthen them to make them even better.
Ultimately, this research provided us with powerful evidence of what we’ve been telling our clients for years. The two methodologies will provide you with different types of information; both are excellent tools that can be used separately or together to answer the important questions you have about your business. In our experience, focus group findings are highly consistent with the findings from representative surveys. Yes, you can trust what’s being said in the room.
[1] The names have been blinded for the purposes of client confidentiality.
To download this article as a PDF, click here.