They've got your number

“It’s time to curb media hyperbole when reporting on statistics,” says the New Scientist.

The magazine dedicated its editorial last week to having a go at dodgy stats. But the interesting thing is, they didn’t use this as an opportunity to rail against those who take scientific stats and run with them. Instead they say: “Journalists and policymakers can only work with what they are given. Scientists need to bear in mind how their statistics can be abused – or perhaps unintentionally read the wrong way.”

Rather than pointing the finger of blame the New Scientist highlights an unhealthy set of relationships. “The interests of researchers in search of funding, scientific journals in search of wider media exposure, and journalists in search of compelling stories often coincide to create a military-industrial complex for the production and propagation of dodgy statistics.”

Vigilance against the misrepresentation of stats “must start at the source”, they conclude, “cutting off questionable figures before they receive a bad press”.

The examples of statistics abuse cited by New Scientist are mostly from clinical trials, but the same issues apply to survey research results. In this month’s edition of Research, Outlook’s Adam Curtis berates advertisers about their use of statistics from surveys – including grand claims based on tiny samples and ill-defined statements about “longer-looking lashes” and “shinier-looking hair” presented as if they were objective fact.

The comments we’ve received in response to Curtis’s article suggest that many in the industry feel the same – but there are some who believe research needs to get its own house in order before complaining about how survey data is misused in other fields.

Philip Graves, whose book Consumer.ology sets out to explode the “myth” of market research, commented: “At least the advertising brands know what they’re doing when they conduct and publicise them. This, it strikes me, is somewhat more justifiable than the rest of the quantitative survey industry pretending that its results represent some kind of objective measure of something meaningful; when in fact the vast majority of survey responses are a by-product of a wide variety of influences brought on by the process of asking questions and the timing and location in which they are asked.”

Another commenter took the opportunity to criticise agencies that continue to describe samples taken from access panels as ‘representative’.

When it comes to communicating research results, we mustn’t forget the human factor: once statistics are out in the world, people are going to share them, talk about them, try to understand them and maybe use them to try to achieve something. Researchers are right to criticise those who misuse their data. But if they want to be taken seriously, they also need to think about their own role and relationships with the people whose hands their work might end up in.

We hope you enjoyed this article.
Research Live is published by MRS.

The Market Research Society (MRS) exists to promote and protect the research sector, showcasing how research delivers impact for businesses and government.

Members of MRS enjoy many benefits including tailoured policy guidance, discounts on training and conferences, and access to member-only content.

For example, there's an archive of winning case studies from over a decade of MRS Awards.

Find out more about the benefits of joining MRS here.

0 Comments


Display name

Email

Join the discussion

Newsletter
Stay connected with the latest insights and trends...
Sign Up
Latest From MRS

Our latest training courses

Our new 2025 training programme is now launched as part of the development offered within the MRS Global Insight Academy

See all training

Specialist conferences

Our one-day conferences cover topics including CX and UX, Semiotics, B2B, Finance, AI and Leaders' Forums.

See all conferences

MRS reports on AI

MRS has published a three-part series on how generative AI is impacting the research sector, including synthetic respondents and challenges to adoption.

See the reports

Progress faster...
with MRS 
membership

Mentoring

CPD/recognition

Webinars

Codeline

Discounts