In search of an answer

One common element of research applications today and earlier applications, going back to DOS, BASIC and even punch cards, is that programs were tools – the researcher did the work using the tool, telling the software what analyses he or she wanted to conduct.

Google Consumer Surveys (GCS) turns that – and many other things – on its head. It will gather the data, and then it will tell you what’s important about that data. For instance, to explore regional differences in the term Americans use for “a carbonated cola beverage”, GCS took a survey of 1,492 people and generated and analysed 154 “insights”, cross-tabulating answers by inferred demographics. In this case, it identified seven insights that it thought were significant, though it cheerfully confessed to “3.1 false discoveries expected on average (p value: 0.02 )”. The big insights: the US midwest picked “pop” three times more than the US west, and the US north-east picked “soda” two times more than the midwest.

The findings aren’t revolutionary, but the approach is. Just about every other research tool written to date requires you to run the crosstabs and investigate them yourself to determine what is significant. But then Google didn’t build an application for researchers – nor did it set out to. The driving goal was to help traditional publishers better monetise their websites.Initially they tried micro-tasks similar to those that Amazon Mechanical Turk fields. It turns out that for all our lamenting as an industry about the lack of interest in surveys, web users preferred answering a survey 3-4 times more than completing a micro-task.

New thinking

Google may have entered the research world reluctantly, but it is not reluctant to reinvent it. Clearly it was eavesdropping on all our conversations (or at least indexing all our whiny blog posts) and, as a new entrant, it is happy to turn established wisdom on its ear. For instance, established wisdom says:

You can’t conduct representative research online, unless you build a probability panel. Paul McDonald, Matt Mohebbi and Brett Slatkin of Google write in their methodology white paper that GCS results were found to be “more accurate than both probability- and non-probability-based internet panels in three separate measures: average absolute error (distance from the benchmark), largest absolute error and percent of responses within 3.5 percentage points of the benchmark”.

This claim needs to be validated by external researchers – it makes sense that random sampling of visitors to publishers’ web sites would produce greater accuracy than panels, which are just convenience samples, but it is unclear why it would be superior to carefully constructed probability panels using address-based sampling – and it is unclear why we should assume users of these sites are representative of consumers in general. However, even if external research shows that GCS is somewhat worse than probability panels, it will always be more scalable and affordable.

We can’t convince clients to field shorter surveys. We’ve always known shorter surveys had greater response rates and higher respondent satisfaction rates, yet even getting clients to buy off on 10-question surveys has been difficult. If you want to field a 10-question survey with the first question a screener, Google will field that as nine independent two-question surveys (everyone sees the screener). Google enforces the discipline that we have been unwilling to impose on ourselves. At a cost for analysis of course – you can’t cross-tabulate answers on anything other than the inferred demographics.

Respondents may hate long questions, but they’re essential. Google limits question length to 100 characters, answer length to 36 characters and the number of choices shown at one time to five textual choices or two image choices. I thought learning how to write for Twitter was tough, but these limits are brutal. I found it difficult and time consuming to write and rewrite my questions to fit this format. You can have longer choice lists – but Google will show a random subset of the choices up to its imposed limit, then weight the choices based on “wins and losses against competing answer choices”.

I ran my first survey the week GCS came out and was surprised it took a week to gather 200 responses. Since then, Google has added more publishing partners, new screening options, bold and italic formatting to questions and some other options, a 10% discount if you are willing to share access to your research results and lots of other minor changes – and at lease one bug fix that I noticed (screened questions seem to have been reported using the viewing data of the screener question). Expect additional question types and analytics in the future, and a roll-out to other countries, as GCS is currently only available in the US.

But what does it all mean for the research industry? First, Google’s values and goals are different from ours. Based on the comments of Monica Plaza, head of business development for GCS, at IIR’s The Market Research Technology Event, I believe their priorities are these:

1. Readers – Google wants readers to continue to enjoy free content, and they want to offer “micro-surveys” that readers will enjoy taking. I don’t believe GCS will ever offer longer surveys nor do I believe the screener questions will ever be used for “selling under the guise of research” by branching people to calls to action. Showing every reader two-question surveys will train respondents not to try to bias their first answer to skip the second question.
2. Publishers – Google wants to help their publisher partners make more money from their content, but not at the expense of sacrificing the reader’s experience. While that might be in a publisher’s short-term interest, it is in neither the publisher’s nor Google’s long-term interest.
3. Marketers – Google wants its paying clients – marketing people without research expertise – to be able to conduct quality research even without understanding market research. The automated identification of insights is to spare business people from needing to have an understanding of statistics; the large error bars shown when diving into the data are there to remind them not to over-generalise from the results.
4. Market researchers – Yes, Google would like us as customers, where appropriate. But Google Consumer Surveys is not fundamentally for us. I think it will be a great tool for finding the incidence of attitudes or behaviors, which can then be calibrated against a long-form survey – a format GCS itself uses in having customers assess their satisfaction with its own tool,
   I should add.