NEWS20 September 2010
AUSTRIA — Rushing through long or difficult questions in online surveys harms data quality more than any other type of respondent behaviour, a new study suggests.
Psychologists from the University of Vienna and the University of Deusto in Bilbao, Spain, tested a Javascript tool which tracked respondents’ activity while they completed a 23-question online survey about their use of instant messaging (IM).
The researchers assessed the quality of the data collected by checking results for consistency, and by comparing demographic information from respondents who shared their IM user names with information found in online address books.
As well as looking at plausibility of answers and non-response, the Javascript tool allowed the researchers to study behaviour such as excessive clicking or mouse movements, time spent considering questions and whether respondents changed their mind before settling on an answer.
The biggest problem, they found, was respondents answering without reading questions properly. “Introduction texts were rarely read thoroughly,” they said, “and semantic differentials [matrix questions] showed higher levels of clicking through than other questions.”
On the basis of their findings the authors recommend that questionnaire designers
- keep introductory texts as short as possible
- only use matrix questions if absolutely necessary
- avoid questions that require calculations or other difficult tasks
- avoid putting more than one question on each page
The researchers said the project had shown that their tool, the UserActionTracer, can be implemented easily to detect problematic items in surveys. Further reserach, they said, might include looking at the positioning of radio buttons, and analysing the speed and duration of mouse movements.
The study was conducted by Stefan Stieger of the University of Vienna and Ulf-Dietrich Reips of the University of Deusto in Bilbao, Spain and is published in the journal Computers in Human Behavior.