The market research industry is obsessed with blocking bad respondents, looking for fraud, duplicate participants, suspicious devices, poor open ends, speeding and dozens of other signals that might justify removing someone from a study. Given the data quality problems facing the industry, there is good reason for that attention.

Most of the tools built to address data quality reflect that focus. Fraud scores, exclusion lists, attention checks and other measures are designed to tell researchers when something looks wrong. Respondent history gives us a chance to learn something about the people who consistently do a good job, too.

There has long been some suspicion around frequent survey takers. A study published in the Journal of the Royal Statistical Society examined previous web survey experience across several measures of response quality and found little evidence overall that greater previous survey experience was associated with poorer responses.

That makes sense to me. Frequency by itself doesn't tell us much about quality. Someone may take a lot of surveys and consistently provide thoughtful, reliable data, while someone who participates less frequently may repeatedly exhibit quality problems. If we have a history for those respondents, we can see the difference.

We can identify the good, too

We are starting to see this in our own data. As respondent histories have grown, we can identify a group of people with a consistent record of good participation. We call them Gold Standard respondents: people with an established history who consistently perform well across DQC client surveys. Among known respondents with sufficient history, 22% currently meet that standard, accounting for 29.5% of survey starts.

I don't want to read more into those numbers than they tell us. We can't conclude from them that taking more surveys makes someone a better respondent, and they don't explain why Gold Standard respondents account for a larger share of starts. But we know that nearly three in 10 starts among respondents are coming from people who have already demonstrated that they can provide good data.

Until recently, there wasn't enough respondent history available across suppliers and studies to identify that group.

A quality signal doesn't have to be negative

I recently wrote about an academic study that evaluated 31 fraud detection methods and six combinations, or ensembles, of those methods. The researchers found that combining different sources of evidence generally worked better than relying on an individual quality check. In that piece, I looked at respondent history as another source of evidence researchers could add to the ensemble. A history of consistently good participation gives us another way to use the same history.

Suppose someone triggers a speeding flag. If that is all we know about the person, it may carry considerable weight in a quality decision. If the same person has participated in a number of unrelated studies and consistently performed well, the speeding flag still matters, but there is more evidence available to evaluate it.

Buyers already take past performance into account when evaluating sample suppliers. Respondent history gives researchers the ability to do something similar with the people participating in their studies.

Using positive history

Buyers could be more proactive about where their data comes from. Maybe that means telling suppliers which respondents they want to see or making it easier for people with a long record of good participation to get through screening. These individuals may not need to go through as many annoying quality checks every time they enter a survey.

Suppliers have something to think about here as well. Good respondents are the people producing successful completes and, in turn, driving revenue and profit. If suppliers can identify the people who consistently provide good data, there is a business case for keeping them around. That might mean better incentives, better opportunities or fewer of the frustrating checks that make participating in surveys harder than it needs to be.

Over time, that could change the relationship with participants. People who have demonstrated that they provide good data would have a reason to keep participating, while suppliers would retain more of the respondents they know they want, creating a positive feedback loop.

Most buyers and suppliers aren't operating this way today and still primarily use quality information to find and block problems. Changing that model would require participation from buyers, suppliers and others across the industry. We now have data that can help make it possible.

I don't expect the industry’s focus on fraud to go away. Nor should it. But I do think we have spent so much time figuring out who we don't want in our research that we haven't spent nearly enough time figuring out what to do when we know we've found someone good.