All posts from: September 2009
The University of Westminster’s Steven Barnett criticises the government’s “blatantly one-sided questionnaire” (commissioned from BMRB, where Barnett himself once worked) and the use of the results in a speech by culture secretary Ben Bradshaw.
The way questions were presented meant that the 65% of respondents who Bradshaw says agreed with the government’s proposal “had virtually no choice”, Barnett said.
This sort of analysis of survey data is all too often lacking in the press. Often the best they can manage is fluff like this. And The Guardian gets extra points for illustrating the online version of the story with a clip from Yes Minister.
David Brock’s Mediamatters.org has spotted some shameless survey abuse by Fox News in the increasingly bewildering healthcare reform “debate”.
Stats from a poll of doctors by Investor’s Business Daily (which has been slated by poll analyst Nate Silver) were cited repeatedly by Fox’s news anchors. And in a display of audacity that we can’t help but grudgingly admire, the figures were in one case presented with the caveat “not scientific” in the graphic at the bottom of the screen.
The facts (or not) pretty much speak for themselves on this one, we think.
There were some great examples of how (and how not) to influence people’s behaviour on Danny Finkelstein’s programme ‘Persuading us to be good’ on Radio 4 last night. (NB the audio is only available online for a limited time, and only in the UK)
Looking at the theory and practice of “the new science of persuasion”, Finkelstein spoke to thinkers including Robert Cialdini, Richard Thaler (co-author of Nudge) and politicians including shadow chancellor George Osborne.
It turns out that much of the government’s efforts to get people to do things like not smoke, recycle more and so on, are doomed to failure. And I have to say, having sat through countless spectacularly misjudged public information campaigns, it was nice to hear these ideas articulated. One particular pitfall (which Cialdini calls “the great mistake”) is to highlight widespread bad behaviour – an approach that falls into the trap of reinforcing an apparent social norm, which people then tend to conform with. D’oh!
Finkelstein introduced the slightly scary new phrase ‘person shaping’, which some local councils are using to describe their efforts to influence behaviour. “Politicians will run a mile from those words,” he said. “But it is in effect what they will be doing. There will be people who will be uncomfortable about government learning how to manipulate us, but that naively assumes that they’re not trying to manipulate us now. They are. They’re just doing it badly.”
One of the persuasive techniques being tried out by Barnet Council in north London sounded a lot like the social science version of a push poll – door-to-door interviewers asking people what they’re doing to go green (turning thermostats down, walking the kids to school etc), ticking the actions off on a list, then asking people to make a pledge to do more. The focus was on making a personal commitment in front of another person, but the act of being taken through one’s bad behaviour point by point in a survey must have some impact too.
Persuasive techniques based on simple, effective communication rather than financial incentives or expensive campaigns, have a chance to gain even more traction at a time when government doesn’t have a great deal of money to spend on such things. The Conservatives are apparently already showing a keen interest.
In the future, expect to be nudged rather than nagged.
Let’s have a look at how the media covered the launch last week of the Coalition for Innovative Media Measurement (CIMM).
The official line from the group, which will fund pilot studies into new ways of measuring media, focusing on set-top box data and cross-media measurement, was that it is ‘not about Nielsen’.
That’s their story and they’re sticking to it, but it hasn’t stopped everyone else from coming to the conclusion that this is indeed very much about Nielsen.
The New York Times’ Brian Stelter said the fourteen companies involved were “tacitly displaying their frustration with the country’s chief source for television ratings”. Some might say it wasn’t even very tacit.
Claire Atkinson of Broadcasting & Cable said the move might be “less about bringing in new competition and more about bringing the monopoly provider of TV ratings to the negotiating table”.
Joe Flint of the LA Times suggested that the results of the planned studies “could prod Nielsen to change its methodology”, and the AP’s David Bauder agreed, saying: “Although attempts to compete with Nielsen have failed, networks have been convinced in the past that complaints or other research initiatives prod Nielsen into updating its methods.”
Kenneth Li of the Financial Times saw it as “a move that could upset Nielsen’s control on media tracking”, following “decades of frustration” with the firm.
Beyond that, not much is clear. Joe Flint’s response in a blog post to the coalitions’ own attempts to explain its plans was: “Confused? So are we.”
That was certainly how we felt here at Research – James Verrinder commented that what we have so far been told about the coalition’s plans has raised more questions than it has answered.
In response to a request from “Paul in Nottinghamshire”, the BBC’s More or Less has been looking into the claim by the UK government that there are 7 million illegal filesharers in the country.
The widely reported figure turns out to have a rather convoluted back story. The government got it from an advisory body specialising in intellectual property, which got it from a report done for them by a team of academics at University College London, who attributed it to a study by Forrester.
But the BBC’s Oliver Hawkins discovered that it actually came from a different study by Forrester subsidiary Jupiter Research, which was commissioned privately by none other than the British Phonographic Industry, which spends much of its time lobbying for the government to clamp down on filesharing.
Delving deeper into the research behind the claim, they discovered that 7 million was actually a rounding up from 6.7 million, and that that figure was based on 11.6% of a sample of 1,176 households in a survey who admitted to using filesharing. The percentage was hiked up to 16.3% to account for people lying about their dodgy behaviour, and this was then applied to an estimate of 40 million people using the internet – which is in itself significantly higher than the 33.9 million figure that the government would have got from its own source of stats on such things, the Central Office of Information.
Change the population estimate and remove the adjustment for people misreporting their behaviour, and you might come up with a figure of 5.6 million illegal filesharers, or just 3.9 million.
None of this means that the 7 million figure is “wrong”, but the government seems to have been very quick to accept a number that was rounded generously upwards, was based on evidence that it clearly hadn’t checked very thoroughly, was calculated from an estimate of the total population that contradicted its own figures, and was originally commissioned by a partisan body.
Hawkins said: “The number of offenders varies enormously depending on the assumptions you make about consumer behaviour and about the size of the online population.” In this case, the government has swallowed the somebody else’s assumptions (somebody with a vested interest), without asking any questions. And it’s not the first time that a body representing media owners has been accused of overstating the scale of online piracy.
The government can argue that its estimate remains valid. But it might take more than that to win back the trust of Paul in Nottinghamshire.
NB More or Less has its own section on the BBC site, which I highly recommend for anyone interested in the use and abuse of numbers.
South London rapper Speech Debelle has won the Mercury Music Prize (that’s the award for the record that music snobs think you “should” be listening to this autumn) and guess what? She’s a former market researcher!
In an interview with The Mirror, Speech tells of her troubled teenage years, and how she finally pulled herself together when she stopped smoking weed and got a job in market research.
She’s only the latest in a long line of celebs who’ve used research as a stepping stone to stardom, of course. Last week it was England’s contestant for Miss Earth.
Keep the dream alive.
There’s an interesting article by Chuck Chakrapani and David Scholz in this month’s Vue (the magazine of Canada’s MRIA) about social relationships between companies and consumers.
Business transactions tend to be based on market norms, but the only way to build loyalty is to start basing them on social norms too. However, the authors warn that if you then break those social norms and return to your old ways, (in other words, you display disloyalty to your customer), then the customer is highly likely to be disloyal back to you – and leave you for a competitor.
Social norms imply treating someone more like you would a person – recognising and reciprocating honesty, trust and loyalty.
But the really interesting bit is that when the relationship is based purely on market norms, “the intention to defect is substantially lower than when social norms are adopted and then abandoned”. [my emphasis]
“Reciprocity is the price a firm should be prepared to pay when it adopts social norms,” they write. “If a firm is not willing to pay or cannot afford the price, it is in the firm’s interest not to adopt social norms. There is no free lunch for companies any more than there is for its customers.”
The conclusions are based on an online survey of 1,600 people in Canada, which asked respondents for their reactions to hypothetical situations in which firms either acknowledged or ignored social norms.
I’d like to see this research followed up with something based on real behaviour rather than hypotheticals, but it certainly makes a strong argument for authenticity in what brands do and say. The study’s about loyalty marketing, really, and doesn’t touch on the use of social media for engagement and research, but the parallels are clear.
The rewards of developing a ‘social’ relationship with consumers make it a tempting strategy, but if you don’t walk the talk, you risk doing yourself more harm than good.
Unfortunately, when marketers hear the message that they need to start being authentic and sincere, they are trained to interpret it as, “I need to start saying that I’m authentic and sincere.” Which, of course, isn’t quite enough.
The argument in that article, which cited marketing cock-ups by the likes of Tropicana and Motrin, was that calling on a sample of consumers in the “contrived environment of a focus group” is not a patch on listening to their views through more direct methods.
For MediaPost, simply mentioning the dirty words ‘focus group’ was apparently sufficient to make the point that these clients went about things all wrong.
Kendrick defends the technique, saying that it is only when focus groups are misused that such problems arise. Only bad workers blame their tools, he writes.
But from the client’s point of view, arguing that it would have worked if it had been done properly doesn’t cut much ice. For whatever reason, focus groups didn’t get these brands where they wanted, and that has allowed another dent to be put in the reputation of the technique at a time when it is already challenged by shiny, new, and often cheaper methods.
Focus groups may not be dead, but more of this sort of thing poses a serious health risk for them.
A slightly irritating report on the BBC website this weekend headlined: Contraception myths ‘widespread’
According to a survey conducted by Opinion Health (and paid for by Bayer Schering Phara) one in five women “said they had heard of kitchen items being used” as alternative barrier methods. Others (we aren’t told how many) had “heard of” food items being used as oral contraceptives.
But aren’t we overlooking a rather important distinction here, between people hearing rumours and believing them, or spreading them, or acting on them? I, for example, have “heard” that babies are brought in baskets by storks. I’ve “heard” that there’s a sea monster living in Loch Ness. But I’m not going to be contacting the BBC to warn them of the dangerous prevalence of either of these myths.
A similar thing occured a few years back with a widely reported survey on identity fraud, which said that a quarter of people had been a victim of ID fraud… or knew somebody else who had. How did the “or knew somebody” part get reduced to an afterthought? It’s like saying: “Thousands of people in America have been to the moon… or know somebody who has.”
These reports are like the end of a long game of Chinese whispers: a survey saying what somebody said somebody else said, filtered through a client that wants publicity, a PR that sees a way to get it, and a journo who goes along with it.
The Economist is predicting the death of the landline, suggesting that the last cord in the US could be cut as soon as 2025.
We know already that this threatens to be problematic for research, since calling mobiles for phone surveys is fraught with difficulty, and techniques to harness the potential of mobile devices are proving slow to catch on.
The article also highlights how the transition will affect other industries, not to mention the emergency services. And the more people ditch their landlines, the more difficult and expensive it will become to maintain the ageing system and infrastructure.
Still, as far as research is concerned, I can’t help thinking these challenges are get-over-able. The basic idea of speaking to someone using your mouth over some kind of long distance connection seems pretty sound, so it would be a shame for the research world to get hung up on landlines. No-one else is going to miss them.
It’s so easy to knock ‘traditional’ research in these heady days of social media. Augustine Fou of Omnicom offers a prime example in this ClickZ article on metrics. He kicks off with a sentence beginning: “The problem with traditional market research is…” then goes on to talk about “flawed and inaccurate techniques like surveys or focus groups”.
Fou has some good points to make, of course, about the wealth of data out there in social media and the value of looking at what people do rather than what they say they do. But when he talks about the ‘traditional’ alternatives, there’s less meat to his argument. ‘Research’ is just shorthand for a lack of guts or imagination. By all means let’s get excited about the opportunities that social media offers, but let’s not make ‘research’ the scapegoat for everything that’s done wrongly.