The Power and Art of Cognitive Interviews: A Personal Perspective

Firstly, for those unfamiliar with the term, a cognitive interview evaluates and refines survey questions by examining how respondents interpret and process them. Additionally, we assess respondents’ willingness and cooperation by essentially questioning the questions. These interviews are conducted by a highly trained moderator, whose expertise lies in knowing when to probe for clarification. A hesitation, pause, a roll of the eye or uncertain response from the participant signals the need for further exploration to understand any doubts or confusion. By identifying misunderstandings or ambiguities, cognitive interviews ensure that questions elicit reliable and meaningful responses.

These interviews go way beyond mere data collection—they truly unlock a deeper understanding of how people think and perceive the world. While AI-generated surveys are becoming more common, I strongly believe that cognitive interviews should be a staple part of any research process and in some cases, mandatory for larger and long-term studies as a course of best practice in research. Their ability to minimise bias, explore the logic behind responses and clarify meaning is unparalleled.

Why Cognitive Interviews Matter

Their power to unveil the feelings and thought processes behind people’s answers is what captivates me about cognitive interviews. It’s fascinating to reveal the reasoning, assumptions and context that shape a response. They help us go beyond what someone said to really uncover why they said it. Pre-designed questions may appear flawless on paper but often fail to capture this depth. Even well-crafted surveys can overlook the complex ways in which individuals interpret information. Understanding this underlying logic is crucial for gathering high-quality, actionable insights.

Reducing Bias in Research

Research can easily be skewed by poorly designed questions or preconceived notions. Advanced AI tools often rely on algorithms that can carry the biases of their developers or training data or poorly designed historical surveys. This can affect how questions are framed and how respondents interpret them, leading to unreliable results.

For example, in a recent health and wellness project, we asked participants about their views on beans as a healthy food. Surprisingly, many respondents considered beans unhealthy. Through cognitive interviews, we discovered that participants were picturing Heinz baked beans, high in sugar, rather than healthy green beans or legumes. Had we relied solely on the original version of the question, this critical distinction would have been missed, resulting in misleading data. Cognitive interviews enabled us to refine the question, ensuring that the data accurately reflected perceptions of nutrition.

Understanding Audience Logic

Cognitive interviews have taught me the importance of understanding how people interpret questions differently. The same question can be processed in various ways, depending on someone’s mood, background, culture, or personal experiences. These interviews help identify potential misunderstandings and ensure that questions are universally understood as intended.

In the health and wellness project, the confusion over beans stemmed not from a lack of nutritional knowledge but from the way the question was framed. If we had stuck to the original version, we would have gathered inaccurate data, suggesting participants misunderstood the health benefits of beans. Instead, cognitive interviews revealed the issue, allowing us to adjust the question to better align with our audience’s thinking.

The Art of Questionnaire Design

Many companies and industries today are conducting internal research, but often fall into the trap of asking the wrong or leading questions, resulting in one-sided, skewed and noisy data. Designing a questionnaire is a complex process, it’s an art that should not be trivialised. Poorly designed questions can distort data and lead to misguided conclusions. Cognitive interviews prevent this by testing and refining each question, ensuring they capture the intended information without ambiguity or bias.

For example, asking a question like, "Do you agree that productivity has decreased due to working from home?" subtly leads participants towards a particular answer. A cognitive interview would highlight this bias, prompting the researcher to rephrase the question for neutrality, thus gathering more reliable data.  “In your opinion, do you think productivity has increased, decreased or remained the same due to working from home?”  would give the respondent the freedom to choose an answer more meaningful to them.

The Limits of AI in Question Design

While AI is useful for generating large volumes of questions or processing data quickly, it lacks the nuanced understanding necessary for effective question design. AI-generated questions may seem adequate but often fail to account for differences in interpretation or context. Cognitive interviews, by providing real-time feedback from participants, allow researchers to adapt questions based on actual responses and ensure clarity.

These are just some of the reasons why I believe cognitive interviews should be a fundamental part of any research process. They ensure that the data collected is not only accurate but truly meaningful, reflecting participants’ true thoughts and experiences. As AI plays an increasing role in research, cognitive interviews remind us of the irreplaceable value of human insight. Understanding how people think and feel leads to better questions, more reliable data, and ultimately smarter decisions.

Daniel J McGinn, Associate Director – Qual CIF Research 1 November 2024