Slave to technology, or master of it?

Casro president Diane Bowers opened the event focusing on the increased adoption across the USA of ISO 20252 – the quality standard for market research processes which Casro actively promotes. Casro’s Institute for Research Quality (CIRQ) is now an accrediting body for the standard in the US. Bowers also acknowledged the role technology plays in the research process, stating that CIRQ has a technology director, in order to be able to validate companies making extensive use of technology.

From standards, we quickly moved onto ethics as Jeffrey Henning, founder of Vovici and a prolific blogger on research practice, took us through some of the dilemmas of applying received wisdom on market research ethics and codes of conduct when eavesdropping on social media spaces for the purposes of research (his talk was entitled “Um, we didn’t know you were listening”). Henning presented some results from a survey among social media site users on their attitudes to being researched without their awareness.

“People are concerned about privacy but they are not really changing their behaviour,” he said. “The question is, do we honour their understanding of privacy?”

Only 30% surveyed said they wanted to be contacted by a market researcher, and 85% expected researchers to seek their permission before using their comments. A real challenge is that only 7% wanted their identity to be revealed yet, as Henning pointed out, there is no such thing as an anonymous quote on the web – anyone can enter the quote into a search engine and get right back to the page where it first appeared, often complete with user profile information.

He concluded by proposing a four-point ethical framework for social media analysis which respects the privacy of participants while ensuring the accountability of the research.

Another highlight of the event was Bernie Malinoff’s look at the perils of using Flash and other Web 2.0 artefacts in survey construction (“Sexy questions, dangerous results?”). Malinoff, CEO of Element 54, presented findings from two research-on-research studies he had carried out, with some arresting observations that overturn the assumption that more engagement is necessarily a good thing.

Eye-tracking revealed the extent to which respondents simply ignore the question when graphical answers are shown, and anticipate the task from the graphics. Fancy slider questions consistently got higher scores; others understated things. He also showed the embarrassingly hilarious efforts of respondents trying to fill in graphical surveys that were poorly constructed, to emphasise the point that survey designers now need to be competent interface designers and usability experts too. “The survey interface is identifying one of those skill gap areas”, he cautioned. “If you are developing innovative ways, practise responsible innovation and make sure people know when and how to use innovation.”

Chuck Miller of DMS provided practical insights into the use of routers for web surveys, not only those based on the river-sample principle but also for panels, where they are increasingly being used by panel providers to ensure that participants are not always screened out. He’d been looking into the effect of different optimisation models for handing respondents from one survey to a second, when they screened out of the first.

Miller demonstrated that any kind of prioritised eligibility queueing method, and even methods that weighted certain studies tended to produce skewed results across a broad range of questions. He urged transparency among panel providers about the methods they are using.

There was much more on offer – the event was a true smörgåsbord of what is happening in research technology today. The sheer array of what was on offer demonstrates how difficult it has become for research firms to be master of their technology, and not slave to it and its unintended consequences.