The feedback loop is a project devised by MRS in partnership with SampleCon and is part of the organisation’s contribution to the Global Data Quality (GDQ) initiative and MRS Campaign for Better Data.

The GDQ Feedback Loop is a mechanism that allows buyers and sample suppliers to share structured feedback on data quality issues. The feedback loop uses a common code frame to categorise the reasons why participants are removed from a study for quality reasons.

The code frame is a multi-coded variable which is to be programmed into all surveys that uses a universal code frame to record reversal reasons. There are 18 codes in the code frame that are designed to cover any reasons why survey participants might be rejected, with the codes linked to the GDQ glossary to help maintain consistency.

Speaking in the MRS webinar introducing the feedback loop, Rebecca Cole, chief executive at Cobalt Sky and MRS chair designate, said that the feedback loop would provide a “strong starting point” to understand the reasons for removing research participants from online quantitative studies.

Cole said that initial analysis of the feedback loop showed that fewer projects than expected were showing abnormally high removal rates of above 20%.

“If you had asked me before, I would have estimated a much higher number of projects being affected significantly by poor data quality,” Cole added. “I think what that tells us is how easy it is to overestimate negative issues when all we have to rely on is perception rather than evidence.

“Obviously, that’s not me saying that data quality and fraud are not concerns – they absolutely are – but the feedback loop and code frame gives us objective data, and that means we can quantify the scale of the issues with confidence, we can identify where the problems are most concentrated and it reduces reliance on anecdotal evidence that can be misleading. I was pleasantly surprised that maybe there weren’t as many projects that were being highly impacted.”

Cole said the feedback loop could also help to show clients that data quality issues in the industry were under control: “Trust from end clients is so important, and end clients are a bit nervous and a bit worried. I genuinely believe this is a way we can reassure them.”

Joanna Price, research methods consultant at Kantar, said that the feedback loop would categorise the data quality issues presenting in survey data and help buyers and suppliers to discuss those issues in “a transparent but precise way, so we look for ways to collectively raise the standard”.

She added that the feedback loop is “bridging the gap in the middle; it is a defined quality connection between buyer and supplier to give us a holistic picture of a particular respondent”.

This means using the feedback loop as widely as possible. “The benefit comes from when everyone is using it,” Price explained.

“It [the feedback loop] is about being accountable when you are buying, and it gives us more rigour – it means that we are looking at data quality in a very consistent and ethical way, and we are making good judgments on the data we have.”

Price added: “At the moment, we are still firefighting, we are still reactive, and this sets us up to approach things in a much more controlled, proactive way. We know things will evolve and we will see new challenges, but if you have a structure in place, you can adapt that structure to the new issues. It’s a strong foundational piece, if we can get it adopted.”

Rachel Alltmont, president at SampleCon, said that the feedback loop “gives hope in terms of where we go as an industry”.

“We can move past this ‘blurriness’ we have been living in around what is data quality and what is fraud, because we haven’t had a way to talk about that,” explained Alltmont. “We can be better, but without that information sharing, it was really hard to work out where to go.”

Alltmont added: “Quality should become a conversation about partnership and industry-wide collaboration, versus quality being a point of differentiation.”

Bob Fawson, co-founder at the Data Quality Co-Op, told the webinar that the feedback loop could be “absolutely critical for addressing that systemic risk that exists across the industry”.

He added: “The time is now for us to move into the future. The world needs more primary data – even the biggest AI companies in the world are buying more and more primary data, and that’s our opportunity.

“For me, success looks like continued trust in the integrity of what we do, and the ability to provide insights and foresights that are specific and help our clients grow their businesses.”