FAQs

It depends. If your organization needs to make a major decision or there is a great deal of disagreement around the decision, it is critical to include key decision-makers early in the research process. If key decision-makers agree on the methodology and the research instrument (e.g., questionnaire or moderator guide), they are more likely to accept the research results and less likely to attack the process when the research is complete.

We’re happy to talk with you about who you would want to include in the research process, given the factors at play in your organization.

The best way to survey consumers is online, using companies with panels of consumers who have agreed to participate in surveys. The panels we use have millions of consumers on them, and it’s possible to target the respondents by age, income, or other characteristics.

While online surveys are ideal, they aren’t always feasible, particularly when surveying market areas with relatively small populations. In those cases, we may do a combination online-phone survey or a survey entirely by phone. We have consistently found that when conducting a combination study, the results from the phone and online components rarely differ significantly from each other, which gives us confidence in these approaches.

We have experimented with surveying people on their cell phones but have found that most people are unlikely to answer a call from an unknown caller on their cell phones. When they do answer, they could be anywhere—driving in their cars, on vacation, or shopping. The number of questionnaires we can complete by cell phone is generally not worth the added cost.

The first consideration in deciding on a sample size depends on how you define your market area. If you make decisions based on your entire market area, you only need to ensure your sample size accurately reflects the whole. If you have multiple areas or want to divide your total area into regions, you would need to have a sufficient sample size in each region in order to be able to draw statistically accurate conclusions by region.

The second consideration relates to the error rate. The error rate is a calculation based on the sample size, the size of the population you are sampling from, and the degree of confidence you set. For our studies, we set the confidence level at 95% (we are 95% confident that the error rate is what we say it is.)

In terms of the size of the population, because of a statistical property called the law of large numbers, the error rate for a specific sample size is stable once the size of the underlying population is greater than 99,999. For example, the error rate for a population base of 100,000 is exactly the same as that for a population base of 1 million.

In terms of recommending a specific sample size, larger the sample size the lower the error rate, but the more the study will cost. We work with our clients to decide how much error they can tolerate relative to the constraints of their budget. Typically, we recommend a sample size of 300, which has an error rate of ±3.4-5.7 percentage points. We believe this size is the best combination for highly accurate results while keeping costs low.

Another aspect of deciding on a sample size is the credibility of the data to key stakeholders. There are situations in which clients want a larger sample size to give more confidence that they are moving in the right direction. If you are a political candidate, the difference between getting 50.1% of the vote and 49.9% of the vote is huge, so you might want a larger sample size to reduce your error rate. For most of our clients, that extra percentage point or two in accuracy is not worth the additional investment. The decisions they’d make would be no different whether their name recognition, for example, was 25% vs. 28%. Finally, budget is always a consideration. We work with our clients to design the best study we can within the constraints of their budgets.

We recommend limiting questionnaire that can be completed in 15 minutes or less. Research that shows that people can be kept fully engaged in a survey for up to 20 minutes, but that the quality of people’s responses deteriorates after that point.

In general, people can complete 4 to 5 questions in a minute, which means we can ask about 60 to 75 questions in 15 minutes. The number of questions that can be completed in a minute varies, though, depending on the length and complexity of the question and the answer categories.

Deciding between doing focus groups in-person vs. online depends on a variety of factors. The first factor relates to the characteristics of the people you want to participate in the focus group. People with complex or busy schedules, such as physicians, have an easier time committing to an online focus group. We generally hold our online focus groups over 3 days, with participants committing to participating at least 30 minutes a day, but can participate at any time that is convenient for them based on their schedule.

Also, if the potential participants live in a wide geographic area, we’d recommend an online group. People don’t like to travel more than 15 to 20 minutes to attend a focus group, which limits your potential participants for an in-person group.

On the other hand, if the goal of the focus group is to explore a new concept or to talk about a subject that participants don’t know much about, an in-person group would be more effective, because the moderator can create a conversation by probing people’s responses and participants can gain insight into their own perceptions and attitudes by hearing from other people in the group. In addition, the moderator has the flexibility in an in-person group to rearrange the guide or go in a different direction based on the responses. With an online group, the moderator can post follow-up probes to people’s responses, but their responses are not always immediate. In addition, the moderator guide is generally fixed in advance, and while some modifications can be made to the questions asked in Day 2 or 3, an online group has less flexibility than an in-person group.

We regularly include the question about recommending our clients and their key competitors that we use to calculate the Net Promoter Score (NPS). The NPS question is, “One a scale from 0 to 10, how likely are you to recommend COMPANY to a friend?” People who give a rating of 0 to 6 are Detractors, those who give a 7 to 8 are Passives, and those who give a 9 or 10 are Promoters. The NPS is calculated by subtracting the percentage of Detractors from the percentage of Promoters.

Including the NPS question and analysis does not affect the budget for a study.

In general, a qualitative study is appropriate when you are trying to get in-depth information about how people make decisions and/or why they have the attitudes or perceptions they have. Qualitative research is also valuable when you want information about a new concept, product, or service—particularly one that’s unique, is unlike something people have experience with, or that is complex and challenging to understand.

A quantitative study is appropriate when you need to know what proportion of your target market has specific attitudes, beliefs, and preferences and when you need to be able to track changes in those attitudes, beliefs, and preferences over time.

There are many cases in which the ideal methodology would be to conduct both qualitative and quantitative research. The understanding we’d gain about the topic from the qualitative research can help inform the questions we ask in the quantitative research. Sometimes we start with the quantitative research and follow-up with the qualitative research to get at the “whys” behind the quantitative findings.

If both types of research would be helpful, but there isn’t room in the budget, we will work with the client to determine which approach would be most useful If qualitative research as most appropriate, we are confident that clients can act on the basis of qualitative research alone. With over 40 years of experience in conducting qualitative followed by quantitative research, we’ve found that the themes from the qualitative research are confirmed by the quantitative research. We wrote a white paper comparing the finding from qualitative and quantitative research, which demonstrated the comparability of the findings from the two methodologies: http://marketstreetresearch.com/wp-content/uploads/2014/09/Trusting-The-Room.pdf

Market Street regularly conducts surveys using lists provided by our clients, such as lists of students, donors, board members, parents, and alumni. We have developed protocols employ those protocols with every list provided to us by a client.

We’d be happy to provide you with the specific protocols we follow in order to protect the confidentiality of any list you provide us.