Meeting the data quality challenge

The Global Data Quality (GDQ) initiative is a coordinated partnership between MRS, Association for Qualitative Research (AQR), The Canadian Research Insights Council (Cric), Esomar, Insights Association, the QRCA, The Research Society (TRS), SampleCon, and The Association of Market Research Austria (VMÖ).

Data quality is an industry-wide issue, and as such, collaboration has been at the centre of the GDQ’s efforts. One of the steps MRS has taken is to establish a group of around 30 practitioners from across the industry, including full-service agencies, panel companies, sample providers, qualitative agencies and data operations businesses to support MRS in developing best practice guidance, tools and techniques.

Debrah Harding, managing director at MRS, said: “Ensuring the quality of our data will be a defining challenge in securing the future of research. While AI and other technologies offer potential benefits for researchers, including the improvement of data integrity, we know they also pose the risk of being used by bad actors to exploit the research sector through data fraud. 

“This is an issue that affects everyone within the research supply chain and transcends competitive advantage. That’s why MRS formed the Global Data Quality (GDQ) initiative, bringing together practitioners from across the world to share their unique insights to develop tools, guidance, and approaches that help the whole sector to fight back.”

The MRS activities as part of the GDQ are split into three pillars: developing internal tools and guidance; identifying new external tools and adopting or adapting these for research; and harnessing existing initiatives.

Developing internal tools and guidance

This week, MRS issued its new guidance, Improving Data Quality and Integrity: Some Practical Approaches, developed in partnership with the VMÖ. It identifies and classifies data quality approaches being used across the sector to combat data fraud and uphold data integrity, including a review of legal considerations, particularly GDPR, for each of the approaches detailed.

This is the first step in a broader process, with MRS aiming to offer more detailed guidance in future on how to act upon the types of issues identified through the techniques outlined in the document. 

In October 2023, MRS and the Insights Association worked together to launch a data quality glossary to define the terms used to describe bot and fraud technologies, which is being expanded as new terms emerge.

The glossary was developed after the associations found that phrases such as ‘data fraud’ were being used liberally to describe a range of data quality issues, according to Harding.

Harding said: “Often, when practitioners were discussing ‘fraud’ they were in fact describing challenges such as inattentive and disengaged participants. While these behaviours can create problems, they are not the same as ‘data fraud’ – which refers to malicious or deceptive behaviour that misrepresents the identity of participants and their information – and therefore they need to be addressed differently.

“The creation of the data quality glossary was important to make sure we talk about these issues with precision and create the right solutions to them.”

MRS plans to continue expanding the data quality glossary and further develop the guidance document, offering more operational guidance on how to use the recommended techniques effectively. It has also developed a new training course, Mastering Data Quality in Online Research, set to launch on 7th May.

In tackling an issue as wide-ranging as data quality, research has been key to identifying some of the factors at play. One project undertaken involved auditing a survey, which found that over 50% of completes were rejected, highlighting the complexity of the research sample ecosystem.

The results from that study will be published at the Association for Survey Computing (ASC) International Conference in London on 19th and 20th June.

The group involved in this project is in the process of producing a visual interpretation of the sample ecosystem and developing a sample buyers’ guide to help buyers better understand the challenges of the ecosystem, and what questions they should be asking when buying sample. Additionally, MRS is working with SampleCon to investigate whether a feedback loop could be developed for when problematic records are identified.

Issues are not confined to quantitative studies, however. Another research project from MRS, alongside AQR and QRCA, is examining how fraud and technology is impacting qualitative research, with the aim of developing new guidance to help practitioners.

Identifying new external tools and adopting or adapting these for research

Another piece of research has involved documenting and classifying third-party data quality and security solutions and how they address various aspects of the fraud challenge. MRS has undertaken a survey of these solutions, with the results due to be released later in the year.

Continuing the theme of collaboration running through the GDQ initiative, MRS has also been holding discussions with representatives of other sectors – including financial, gambling, retail, fintech, advertising and academia – to identify approaches that might be suitable for research.

MRS has also had discussions with the Home Office to ask for research fraud to be designated as a crime, which would allow the sector to utilise government schemes such as the Online Fraud Charter.

Harnessing existing initiatives

Lastly, the GDQ is drawing on existing industry efforts as it looks to see how previous findings could be applied to data quality issues.

The MRS Mobile Optimisation Project, for example, is an ongoing partnership between MRS, Dynata, Kantar, Cint and Toluna. While the initiative launched in 2018 and pre-dates GDQ, it has highlighted similar issues, including the impact of poor mobile research design and the lack of mobile optimisation on research participation rates.  

Next steps

The key current priority for MRS is to complete the development of the new MRS tools and guidance, and to share these with the sector for adoption.

In the short term, the next step for the GDQ initiative will be to finalise some standard metrics for measuring fraud, a project which is being led by the Insights Association. MRS will use the results of this project to inform its guidance.

Discussing the most challenging aspects of the GDQ initiative to date, Harding said: “Obtaining a complete picture of the sample ecosystem has been difficult, as it is complex and multi-tiered.  It is however essential that we understand this complexity to develop practical and workable tools and guidance.”

Harding added: “While the GDQ initiative has brought its challenges, the willingness of practitioners to volunteer their time and expertise and work with other research associations from around the world has been wonderful to see. Data quality issues are borderless and unrelenting and only by working together will we be able to address them effectively.”