Why are we still asking about previous participation?

Every now and then, I come across a range of questions at the beginning of a survey that I call the ‘old-school opening.’ It usually includes a range of demographics, perhaps some survey-specific filters, and then two classics of the research world: ‘Have you participated in research in the last x number of weeks?’ and ‘Do you work in any of the following industries?’ The latter is certainly worth its own discussion in another article, but today, I want to address the former.

We happily disqualify respondents based on their previous research participation, but how recently have we asked ourselves ‘why?’ At a time when the research industry is competing for participants’ time and attention—whether that’s online, face-to-face, CATI, or through another method — can we still (literally) afford to disqualify respondents because of recent participation in another survey?

Respondent bias

The brain is like a muscle – the more that certain areas are used, the more easily accessible they become. This means areas of the brain that recall certain information (such as brands) may be more stimulated if they were recently activated through participation in a questionnaire or focus group. As a result, certain information may be more easily accessible to these respondents, which can produce, for example, a higher brand awareness or more thought-through answers than a ‘fresh’ respondent would have given.

However, this would only be an issue if a respondent had recently participated in a survey on a similar topic, and if the survey was aimed at collecting unprompted information, such as unprompted brand awareness or spontaneous opinions on a topic. Questions aimed at recalling certain behaviours may even benefit from previous exposure to a similar topic, as that area of the brain would already be activated, and recall would arguably be better.

Over-participation

Secondly, there’s the topic of over-participation. By asking about previous participation, the thought is that you can identify respondents who are trying to participate in research at a high frequency for financial gain — and who may, therefore, be misrepresenting their actual views. I’m sorry to burst that bubble, but these respondents are savvy enough to tick or say ‘no’ when asked whether they’ve recently participated in research.

In instances when online panels are used for recruiting, only advanced tracing technology (such as digital fingerprinting) and daily limits on survey participation will properly tackle this challenge. In instances when sample is recruited offline, this is even more complicated — the most effective way to identify inauthentic responses here is to identify inconsistencies in respondents’ behaviours. Whichever method you choose for data collection, the point remains that asking people whether they’ve recently participated in research is an ineffective — and perhaps even slightly naïve — way of trying to tackle inauthentic over-participants.

To summarise, like any question in a good survey, we should only ask about previous participation if it serves a purpose and if it serves that purpose well. If we’re asking it to tackle over-participation, then we may as well not ask it. If a survey’s data may be skewed by a respondent having recently thought about the topic, then let’s very specifically disqualify respondents who have recently participated in research on that subject.

In a research landscape in which respondents are a finite resource and every question counts in keeping them engaged, we need to ask ourselves: why are we still asking about previous participation?

Marie Hense is vice-president of data quality at Toluna