Tips & Techniques
Here are some methodological tips to help guide the research process. By sharing best practices, we aim to improve the quality and relevance of the insights you generate.
Before launching into a methodological debate (survey vs. focus groups), get clear on what you need to achieve and how your study will contribute to this goal.
- What business problem are we aiming to address?
- How can market research help us solve this problem?
Put this in writing and communicate it with your team and vendors. This way, everyone will be grounded in a shared understanding of the business challenge and research objectives.
Since the dawn of the Internet, many new technologies and data collection platforms have emerged to revolutionize the market research industry.
Yet the fundamental distinction between qualitative and quantitative research remains unchanged.
Sure, okay, there are ways to meld the two: You can add open-ended questions to your quantitative survey. Or include rating sheets to your qualitative discussions. But the fundamental issues of sample size, “representativeness,” and structured vs. unstructured data will always distinguish the two.
Why use each one? Which one is right for your study? Consider the differences …
There are many ways to conduct qualitative research and pretty much any method that once could only be done in-person can now be done online.
Your study may demand that your qualitative study be conducted in person. Are you testing highly technical or complex messaging? Do you want to see and feel respondents’ reactions? Will you miss the subtle nuances that only come through when you’re in the same room? Would you and other stakeholders benefit from actually being there?
Don’t be surprised if you find yourself doing things the old fashioned way.
Allow for Flexibility
Focus groups can be unpredictable. Moderators need to be flexible, while keeping a sharp eye on the clock.
A core benefit of qualitative research is to unearth insights you hadn’t thought of. So think of your discussion guide as a basic skeleton you can move through in a fluid way. Don’t load it with too many details and questions. Allow time for respondents to raise and discuss topics they bring to the table. Then bring the conversation back to the next topic at hand.
Most major cities have research facilities with modern amenities and equipment. They may be smaller and less attractive, but your basic needs will be met.
Lean on your local partner for the kinds of details only they would know: Are there any national holidays you should avoid? What time during the day/evening are focus groups usually scheduled? Are weekends better than weekdays? Is the facility in a convenient location for the type of respondents you’re aiming to recruit?
Online Survey Platforms
Your questionnaire is ready to go. Great! Now it’s time to think about programming and hosting.
There are many platforms available for programming and hosting online surveys. Survey Monkey and Google Forms are very user-friendly and do not require any programming expertise. For simple, straightforward surveys, these are good choices.
For more complicated surveys, platforms like Decipher and Qualtrix are good options. Programming teams are available to build complex programming and assignment logic. They also offer advanced tools for analyzing data on the backend.
PII + GDPR
A critical issue you may face in the field process involves privacy or the protection of personal information. Data privacy has become an increasing concern in the market research industry.
Under GDPR (General Data Protection Regulation), researchers need to be extra careful when collecting personal data. Similarly, researchers need to protect user privacy and store any PII (Personal Identifiable Information) securely. If a respondent requests to be removed from the survey or opt-out from future communication due to privacy reasons, researchers need to erase those data to comply with the respondent.
Open Ended Questions
OE questions reveal what respondents have to say about your question without any prompt or canned responses. You can learn what comes to mind, off the top of their heads. This can be very powerful if you want to “hear” respondents voices and/or have an issue you want to explore without dictating precise answer options.
OE questions take longer to answer so you must limit the number you ask. OE data is also raw, random, unstructured. If you want to quantify the data, they need to be coded and processed, which can take a lot of time.
Avoid “Drowning in Data”
Ever feel like you’re drowning in data? It is common to feel adrift and overwhelmed in an ocean of information.
How to stay afloat?
Start by going back to your objectives: What strategic issue(s) drove the design of your research in the first place? What hypotheses did the research set-out to test?
Focus on those issues first. Formulate your conclusions, using the variables / responses that best inform those conclusions. Stay high-level. See the forest through the trees.
It is critical to identify the right stakeholders for your project. Too many “cooks in the kitchen” will derail the process. Too few and you may not get the buy-in you need to ensure the research is ultimately valued and relevant.
- Who should be involved in — or at least made aware of — the study?
- What are their interests? What objectives will they want met?
- How much involvement should they have in the process?
- What is the best way to gather input without diluting the focus?
Qualitative research is defined by small sample sizes; in-depth and (somewhat) free-flowing discussions; nuance and subtlety; an opportunity for unexpected insights to arise.
Qualitative is best when you are new to a topic and need to explore themes or generate hypotheses. It is good for complex subjects that require careful thought and deliberating. It is great for getting feedback that goes beyond Agree vs. Disagree, Like vs. Dislike, Choice A vs. Choice B.
Is your target audience extremely small/niche? Qualitative may be required.
A critical step in research design is deciding who should be eligible to participate in a study. Certain criteria will be obvious, while others inevitably require careful deliberation and compromise.
How wide (or narrow) a “net” should you cast?
For niche professionals (B2B), start with your ideal profile. What would that look like? Then round-out that profile to make it feasible. You may need to add another segment or broaden a certain definition.
For general consumers (B2C), it is best to err on the side of constraint: Stick to your ‘core’ and ‘low hanging fruit’. Add extra questions to make sure respondents are genuinely engaged and can bring relevant perspectives to the table.
Before you begin recruiting, consider which of your eligibility requirements are most versus less important.
You often don’t know how challenging a recruit will be until the clock is ticking. So be ready to make some adjustments. For example, if the objectives of a study demand that a certain quota be hit, make that quota a priority. But perhaps age or income criteria can be relaxed.
Where can you be flexible without jeopardizing results? Keep this in your mind so that, should recruiting start to slow, you can adjust quickly and hit your quotas on schedule.
Do your groups or interviews need to be conducted in a foreign language? If so, you’ll need to find a native speaker who is a skilled moderator and knowledgeable of the topic.
This can be challenging if your topic is highly specialized or technical. Build time into your schedule to train the moderator. You must bring them up-to-speed on the market landscape, strategic objectives, key issues you want to explore, the discussion guide, etc. Make sure they have a chance to digest the materials and discuss any questions he/she may have.
Sample Size (n)
Since it is typically impossible to have everyone in a target population answer a survey, we rely on the power of statistics to sample a portion of the population and draw conclusions from their data. This raises an important question:
How many people must we sample to gain sufficient statistical power to prove (or disprove) our hypothesis?
To determine sample size, here are a few factors to take into account:
Total number of people in the population you aim to understand. Sometimes it’s hard to know this number, but it’s always good to have a rough estimate.
Degree of confidence that your sample mean accurately reflects the total population mean. 95% is typically the target CL, while 80% is considered “directional”.
Determines the uncertainty that can be attributed to sampling error. The bigger the sample, the lower the margin-of-error, the more trustworthy the data.
An understanding of these three factors will help guide your decision.
Once the scope and objectives of your quantitative study have been defined, it’s time to design your questionnaire.
Most survey questions are “closed-ended”, though it is often useful to include a few “open-ended” questions as well:
Respondents must select from the answer choices offered in the survey. Examples of CE questions include Single Select, Multi-Select, Ranking, Likert scales, Total Sum, etc.
Respondents are given the opportunity to respond in their own words.
Ah! Your report is almost ready for distribution. Hold on! Have you double-checked the data?
Small data errors are bound to occur at some point in the production process. You need to set-up a process for catching them. We recommend making it a “two-person job” in which every data point is compared against the original data run or banner book. It is entirely worth the extra time.
When errors are found, team members should not only correct the numbers, but also dig in to discover why they happened so mistakes can be avoided in the future.
Dive Deep to Support Conclusions
Once you’ve drawn your preliminary conclusions, it’s time to go deep … without drowning.
Now it’s time to go deep into your data set and examine the details to support your conclusions. You may need to refine your conclusions based on new insights. That’s good. Your analysis is stronger for it. You’re also apt to go off on tangents and fall down rabbit holes. That’s fine — it’s part of the process. But don’t stay too long. Turn your focus back to your core objectives and press on from there.
Define Target Outcomes
If you can articulate and envision what you want to get out of the research — when all is said and done — the easier it will be to design a study that gets you there:
- How will research findings be used?
- What decisions will they inform?
- What do I need/expect in a deliverable?
- What specific data points will I want to have?
Be sure to communicate this with your research team and vendors so everyone is on the same page.
If your end-goal is to gain statistically reliable data that conveys authority and certitude, then quantitative is the way to go.
For example, you would use quant to measure product usage and brand perceptions and then track those metrics, statistically, over time. Or you may want to determine the optimal price point for a new product you plan to market. You would want qualitative data for something so precise.
Numbers are great when you want definitive data.
But then wouldn’t you always go with quantitative research? Qualitative data is so “messy” and subjective. Who wouldn’t want to generate authoritative and certain results? It’s a fair question. But quant is not always right. It only works well when you’re ready to ask the hard questions and, even then, it only works for some strategic issues.
It is common to want to squeeze every last drop out of your focus groups. “Can’t we can add just one more topic? Just one more question?”
While it’s tempting to try to cram multiple topics and objectives into your groups, remember the old adage: “less is more.” The true value of focus groups come when you focus on a single, cohesive topic. Otherwise, you’re apt to end up with vague and inconclusive data.
Should you take your qualitative study abroad? Understanding regional differences will help you position your products and marketing campaigns appropriately for foreign audiences.
So where should you go? Ask yourself:
- In what markets are we going to spend the most on marketing?
- In what markets are we lagging behind or facing the most competition?
- Where haven’t we done research in the past and thus could use some insights?
- Do we have any international business trips coming up? Should we do research while we’re there?
Time and budget constraints will also dictate where you go. Of course, you can always consider doing everything virtually (online), but the time- and cost-savings are not always worth it.
Materials will need to be translated into the local language, so be sure to build enough time in the schedule for that step.
Find a translator that knows the topic and note any specialized “terms of art”. Review these materials with your moderator to make sure the meaning and intention of each topic is not lost in translation.
If any non-native speakers are going to be in the back-room, you’ll need a simultaneous translator to translate the discussion in real-time. Send your SimTranslator all research materials in both languages so they can prepare.
Fake Data is Useful
Unlike fake news, fake data is extremely valuable. Why? It’s your last opportunity to do a rigorous check on programming.
Now it’s time to get your questionnaire out again. List all of the questions in one column and give each question three values. We like to call them the “Should be” base number; the “Actual” base number, and the “If the first two match or not” value.
Using the designed logic and the actual data run, fill out all the values for each question. If all the “should be” bases match with the “actual” base numbers, you’re good to go! Otherwise, you need to dig into the data to figure out and fix the issue.
Closed-ended (CE) questions are an efficient way to gather responses because the terms and attributes are pre-defined. Since they are relatively quick and easy to answer, your questionnaires can be longer — which leads to deeper, richer datasets and insights. Data processing is also efficient because the data are automatically quantified.
CE questions must be designed well. They need to be clear, mutually-exclusive and easy-to-think. Otherwise, respondents may get confused and enter junk answers or abort the survey altogether. “Bad data in. Bad data out”. Pre-test your survey prior to programming and definitely avoid changing any CE questions mid-field.
Base + Sample Size Notations
As with many things in life, it is common to ignore the fine print. We’re referring here to the tiny footnotes that appear under graphs and data tables.
Don’t ignore the fine print!
These footnotes help define the context and reliability of the data. For example:
- Sample size: Before looking at the percents in a table or graph, first check the sample size or “n”. If the sample size is small, the data should be considered “directional” (i.e., taken with a grain of salt).
- Base: Who do the data represent? It might be “all respondents” or it might be a subset of respondents. You need to know this in order to interpret the data correctly.
The “So-What” Question
The final step requires you to go beyond your conclusions. Ask yourself: “So what?”
What do your insights mean? What implications for product or marketing strategies can be drawn from your conclusions? You want you research to be actionable, so you need to figure out what action(s) should be taken based on the data. Come up with three recommendations and explain why they make sense. It may only be a kernel of a plan or strategy. That’s fine. You’ve at least given others a starting point to seize and develop.
And there’s always the option of conducting a follow-up study to test your kernel of a plan or strategy. Yep … now you’ve got a few new hypotheses (recommendations) to test.