OPINION10 June 2014

Up And Down The Validity Ladder

Does social media have a bias towards invalid data?

Simon Kendrick of the Curiously Persistent blog put up a good post the other day on the way research gets into the media – often via PR companies, and often not in especially good or reliable shape. Simon’s whole post (linked below) is worth reading, but I particularly liked his “ladder of invalidity”:

  • Made up data
  • Straw poll of close friends
  • Accumulating opinion from one source (e.g. comments from one news story)
  • PR survey
  • Hypothesis testing survey
  • Exploratory survey
  • Census

Now, we can argue over positionings, and where qualitative research techniques might or might not fit in. But the basic principle is sound. What’s striking to me as someone who reads a lot of social media blogs and news stories is that it seems a lot of people prefer material near the shaky top of the ladder, not the more robust stuff at the bottom. “Straw poll of close friends” is a particularly common tactic – “three friends make one trend”, as one writer put it to me.

This has always been an issue with some areas of journalism, and to be honest we wouldn’t have it any other way – style magazines and supplements would be dull fare if everything in them had to be statistically robust. Flimsy evidence can turn into received wisdom very quickly, but that very effect contains the seeds of its own debunking: a finding that goes against what people have come to assume becomes more of a story.

What about social media? Well, you see the same thing happening – a tech writer posts about his frustration with the iPhone, say, and it’s talked into a crisis by mid-afternoon (and forgotten two mornings later). But the thing with social media is that its whole ethos rests on trusting – and being interested in – what your friends are saying. So a “straw poll of friends” starts to feel truer than a real poll of strangers. Since I started working full time in social media I’ve noticed that the middle of the ladder gets cut out – friends’ experiences count, and so does large-scale data crunching. But our bread and butter – surveys based on representative samples – are often mistrusted.

But then again, that’s just my experience. And that of a few friends, you understand.