FEATURE15 April 2014

Plenty more fish in the sea?

In 1883, English biologist Thomas Huxley stated that overfishing of sea fisheries was “scientifically impossible” and dismissed calls to address the fishing techniques being utilised.


Though he couldn’t have been more wrong about the fish stocks, this didn’t stop this school of thought continuing well into the 20th century.

A fishy subject

You may ask why I am writing about fish. The reason is that this is the best possible comparison I have found for how we currently consider our research respondents. It is important to remember that a large proportion of our industry is based upon the willingness of the public to provide us with their opinions and habits, yet these people are consistently forgotten when we consider the research we are planning to conduct. We simply consider that there will always be more fish in the sea for our research even if it happens to be too long, too repetitive or hard to navigate.

In the 90s and 2000s, we witnessed a significant decline in willingness of respondents to participate in telephone research – a decline from 36% in 1997, to 9% in 2012, according to the Pew Research Centre – and our reaction as an industry was not to focus on the drivers of the decline but instead to switch data collection to online methodologies. As online yielded not only novelty for respondents but also both speed and price advantages for the industry, the transition was, as we all know, exceptionally fast.

This meant that we never really considered how to adapt to, and optimise, this medium and we continued to recreate the errors of the past, forgetting the respondent at the other end of this data collection technique. What made this even worse in online was that we no longer had an interviewer to encourage our respondents through the toughest parts of the surveys. We lost our personal link to the respondent and this has led to them feeling an even greater disconnect with our purpose. Unsurprisingly, we have now seen the online response rates decline just as sharply as telephone did, with a drop in the US from over 40% in the early 2000s to less than 10% in most panel companies I consulted in recent weeks.

Unfortunately, the pace of change and pricing pressures in the industry mean that we have focused on becoming experts in how to catch our respondents but we still know very little about them. For an industry that helps its clients understand their consumers and drive loyalty, we still know very little about what drives our respondents to come to us, and hopefully stay with us. Why is this so important? Well, we need to go back to the fishing analogy and ask ourselves the fundamental question: “In the future, will we have enough willing respondents to answer our research?”

“We have focused on becoming experts in how to catch our respondents but we still know very little about them”

Weary respondents

3 years ago, I presented a paper at the MRS Online Methods conference that demonstrated the clear link between negative survey experiences and the decreased likeliness of a respondent to answer the next invite they received. This showed that panels are continuously eroded by respondents facing unsatisfactory survey experiences, and much of this responsibility rests with the questionnaire they are being asked to complete.

While online panels have evolved, and mostly kept pace with technology, the survey itself is the one part of our process that we have not significantly changed – and the part that respondents are, in my view, genuinely becoming tired of:

  • In a world where multitasking is prevalent, we persist with 25-minute surveys
  • In a world where micro blogging is king, we persist with question texts that run into multiple paragraphs
  • In a world where the most successful web properties are simple to navigate and get you to the objective in as few clicks as possible, we continue to create mundane, complex, repetitive online forms

Prize catch

We should also consider the anonymity of the online respondent. While it has advantages from a data collection angle, it has actually been detrimental to the way we manage the research process. It is a lot easier to be apathetic if we can’t put a face to someone.

I am regularly struck by the significant difference in the way a respondent is treated when we do actually meet them.In viewing facilities, they are welcomed with coffee, treated like we need them, talked to like we value them, made to feel they have been a key part of a research project and they often leave happy with their contribution and reward. In online quant, how often can we truthfully say that our respondents have left a survey satisfied and content with their contribution?

These are all experiences that are sorely missed in the lonely world of self-administered online surveys, but that we can clearly learn from and apply, bringing the principle of a two-way respondent relationship back into data collection. This means that, regardless of the sample source, we simply have to ensure we keep those we have, re-engage those we have lost and engage better with those we have yet to meet. This can only happen through a fundamental re-think of how we question them. Think about how we, as an industry, advise brands to regain lost customers – you have to convince them something has changed, and follow through on that promise.

So are there really more fish in the sea? Well, only if we start to take better care of them, rethink the research we put in front of them and, ultimately, treat the respondents as if they were the prize catch.

James Sallows is MD Europe for Schlesinger Interactive.


10 years ago

I don't think panels are a fair comparison to RDD response rates as panels are opt-in and coddled. I wonder what response rates would be for an online survey emailed to random email addresses. Would 1% be an exaggeration?

Like Report

10 years ago

James nice article. I'd love to say timely but as you know many of us (yourself included) on our side of the online fence have been saying this for years. We simply have to start treating people like they were our mother - you never know, one of them might just be your mother. It's not hard, it just takes a bit of work. And I mean more work than just saying "Thanks!" at the end.

Like Report

10 years ago

Broadcast campaigns via email to non-panel samples would likely be in line with the direct marketing response rates, so yes definitely sub 5%. However this makes the dramatic decline amongst the 'coddled', as you put it, even more concerning as these are the ones who actually want to contribute.

Like Report

10 years ago

I agree with James and Pete, though it is important to distinguish between Access Panels and Custom Panels. At EasyInsites, we use both extensively and there are several differences which result in custom panel members feeling much more engaged in the experience. These include shorter surveys -- there is really no reason to ever run a survey longer than 10 minutes with a custom panel, as it is easy and cost effective to recontact; more variety in the types of research, including quick polls and forum discussions hosted on the custom panel website, and more involved qualitative focus group or bulletin board discussion group sessions, in addition to the surveys; consistency in the survey design template used always, as this is branded for the custom panel to which they belong; and accessing the panel website (if designed with responsive design) and participating in the surveys (if optimised for mobile) means a consistent and high quality experience throughout as a member of a custom panel. In contrast, we know that access panels are used by a wide variety of researchers with different survey design themes, varying quality in questionnaire design, and most of the time are still not optimised and working properly on mobile. This is not to suggest that custom panels are the answer to everything, but simply to point to the stark contrasts between custom panels and access panels and suggest that a lot can be learned from the way in which we manage custom panels that could be applied back to the access panel world.

Like Report