Programmatic 2.0: the future of sample

Some are calling it the beginning of a Fourth Industrial Revolution. Automation is changing the way we work at a fundamental level. While it may not mean a robot will take your job, it will still change the way business is done in nearly every industry.

In market research, its impact is now visible across every phase of a project, from programming to sampling to reporting. In the sample space, the first wave of automation – let’s call it Programmatic Sampling 1.0 – involves the automation of basic tasks like the communication of project specs and quotas. I believe that Programmatic 2.0 is fast on its heels.

Programmatic 1.0 did two things very well: it has made us quicker and more cost effective. Using APIs, machines exchanged basic information about our projects, our questionnaires and our respondents. But, as we implemented this technology, we became aware that companies stored and coded this information in vastly different ways.

More perversely, the immediacy and low cost of the exchange made it even easier to abuse respondents who, instead of bouncing around in one router, might bounce around in several. In short, in Programmatic 1.0 we got faster and cheaper. But we didn’t really get better.

This next generation of sampling, which I’ve dubbed ‘Programmatic 2.0’ promises to do what Web 2.0 did for the internet – improve reach, overall usability and interoperability. Put differently, Programmatic 2.0 can vastly improve the accuracy and reliability of our data and our operational dependability.

As the industry continues to bemoan sample quality, the greatest irony may be that improvement lies in faster adoption of automation. Below are ways that the next wave of automation is set to transform online sampling.

Respondent engagement is co-created and managed automatically

Respondents are not shy about sharing bad experiences. As an industry, we just don’t always choose to listen to them. The consequences are known and not subject to debate: greater attrition (and so higher costs) and bad data (and so lower revenue).

The next wave of automation will mean that treating respondents respectfully becomes a truism and not empty words. Respondent data, in the form of field statistics such as dropout rates or direct feedback like survey ratings, will power algorithms that push poor survey experiences out of field.

Platforms will use these self-healing mechanisms to preserve the long-term health of their business.

Automation extends across all project phases

Machine management of two areas of the research process will lead to dramatically better operational outcomes. One is the bidding, pricing and feasibility phase of the project. Some are critical of this and claim it simply signals a ‘race to the bottom’ in terms of price. I disagree. I believe it will make the purchase decision more nuanced.

Automation during bidding will allow companies to develop their own demand and supply algorithms that leverage other data points in real-time, such as operating reliability, respondent engagement metrics and so on. In effect, cost will become one of many variables that companies may weigh and optimise differently.

The second place automation will be a boon is in field monitoring. This is still largely a manual process, and painfully so. Filled quotas that are left open, poorly-estimated incidence that is not caught until late in field… these situations arise all too frequently. Automation and AI can help proactively spot and even solve problems in the blink of an eye.

AI and machine learning emerge to tackle fraud

Nobody likes to talk about it, but fraud is a significant and growing menace across the entire digital ecosystem, from advertising to market research. The industry’s current techniques like digital fingerprinting are no match for bad actors who have the motive and know-how to interfere.

As fraudsters change tactics, traditional detection tools fall down because they only know how to stop yesterday’s fraud. Advanced AI and machine learning is the only way to stay on top of evolving fraud trends and will be enabled by the billions of digital behavioral biometric data points that sample companies collect.

Sample acquisition truly enters the digital age

Automation and APIs enable the acquisition/recruitment of panellists at scale. Algorithms that combine anti-fraud, respondent engagement and actuarial approaches will allow new respondents, those who would not ordinarily join a panel, to participate while ensuring they remain real and engaged people and profitable for the business.

The consequences of Programmatic 2.0

The ways in which we use this next wave automation will certainly have a positive impact on the product and process of online research.

• Speed will mature into agility. Automation made us faster. Now, thanks to advanced techniques, sample firms with flexible technology and real-time supply can handle even niche targets or innovative data collection.
• Quality goes back to the forefront. Systems take on the real-time responsibility for ensuring data quality and combating fraud through a combination of proprietary methods and commodity tools.
• Dependability takes giant steps forward. Machines increase certainty about feasibility and monitoring, giving greater reliability, efficiency and significant error reduction.
• Greater capabilities propel us to growth. Sample firms take a giant step closer to helping end clients improve their execution through relationships in the broader adtech/martech ecosystem.

JD Deitch is chief revenue officer at P2Sample