Key Characteristics of a Quality Research Partner: Sourcing Breadth and Depth

Digital technologies have dramatically lowered the cost of research, ushering in important new opportunities to gain and apply research insights. Unfortunately, these cost structures can also lead to compromises in quality, which in turn can compromise the accuracy and reliability of research insights, and ultimately the business decisions they support. This article will explore the first key indicator of quality – sourcing breadth and depth.

Bias in a sample produces results that are not representative. Imagine a group of dart players trying to hit the bullseye. A sample with high bias would produce a tight grouping of darts, but they would all be off to one side of the dartboard. In survey research, high bias equates to results that are consistent, but consistently non-representative.

Bias is a potentially serious issue for online surveys. Online research is a form of nonprobability sampling, in that some elements of the population have no chance of being selected – in this case, people who aren’t online. As a result, providers must be rigorous in sourcing to ensure that the sample frame is accurately representative, drawing from a wide population reach and recruiting data sources from multiple channels.

• Open recruitment through mass-market digital advertising draws sample from all over the internet and broadens representativeness nationally and globally.
• Loyalty programs, often associated with travel, tend to yield high-worth individuals and are a good source for professionals for B2B sample.
• Affiliate networks utilise partnerships with web-centric companies to build traffic and reach innovators, early adopters and hard-to-reach segments.
• Mobile apps are a valuable source for younger sample and data sharers – and increasingly important to representativeness as mobile internet use surges past desktop access.

Beyond diverse recruiting, providers who are committed to quality incentivise research samples with a variety of rewards geared to different kinds of studies. For example, in consumer research, gratuities with a value calibrated to the complexity of the survey are appropriate; participants in B2B studies are likely to be more motivated by access to the study results.

Quality Issue: Variance. Variance is a measure of representativeness over time, or consistency. To return to the dart players, a high-variance sample will scatter darts in random patterns across the dartboard. In research, high-variance samples deliver results that change unpredictably, which is especially problematic for tracker studies.

To control variance, providers need to ensure consistency in their sample and in the research experience. Achieving the first means establishing a stable, consistent recruiting effort to prevent short-term recruiting imbalances, and nurturing long-term relationships by treating sample with respect, consideration and transparency to support longevity.

To deliver a consistent, high-quality research experience, providers should ensure that survey designs are easy to navigate and appropriate for their purpose; that their technology and connectivity is robust and redundant to give sample reliable accessibility; and that their delivery team has the expertise to assist clients as needed in creating and administering well designed studies.

Evaluating providers. There are relatively reliable and objective ways to judge a provider’s commitment to responsible sourcing that minimises these two issues.

For bias, ask for measures of representativeness. How does their sample frame compare to established, objective benchmarks in the general population such as home ownership, TV viewing, employment, illness and other important characteristics?

To evaluate variance, find out how their sample compares to the same benchmarks over time.

This article is a part of an eight-blog series by Dynata. To read the second article on how sound methodologies in a survey is crucial to the quality please visit https://bit.ly/2JNOJjr.