FEATURE16 June 2014

On the right tracks?


With mobile tracking proving such a hot topic in the industry, we asked TNS’s chief research officer, Jan Hofmeyr, to share his thoughts on the truth behind the methodologies ahead of the IIeX Forum in Atlanta


Why do you think large trackers are flawed? Where is the market research industry going wrong?

There are a great many problems. First questions that are invalid (in other words, that don’t relate, or relate very poorly to what people actually do). Examples include: aided awareness, customer satisfaction, purchase intention, recommendation, net promoter score, memory based usage questions (e.g. ‘what brands of laundry detergent have you bought in the past three months?). Second questions that are too long and boring — and that therefore lead to poor quality responses — which in turn lead to noisy statistical models. The best examples are long satisfaction rating lists; long brand attribute association grids. And third, questions that are completely irrelevant to the particular respondent and that create statistical noise as a result e.g. questions about brands in which the respondent has no interest. These are just some of the problems. I’ve written a more complete critique in a white paper called ’The Trouble with Tracking’.

Where is the research industry going wrong? I don’t know how to answer that; it’s just plain wrong. What stuns me is the fact that people just continue to do these things even when they know they’re wrong. People are more afraid of discontinuity in their data, than of falsehoods! It’s mind-boggling. I think one of the biggest problem is the inclusion of very poor quality marketing research data in people’s KPI’s. Perhaps that’s the main way we’ve gone wrong.

Is mobile tracking currently over-hyped?

The promise and importance of mobile tracking isn’t over-hyped. What’s over-hyped are some of the methodologically shallow survey results. So we hear about surveys in Africa that involve huge numbers of respondents — but there’s nothing about the quality of the sample. Also, some mobile tracking simply perpetuates the mistakes of conventional tracking.

What are the pitfalls of mobile tracking?

Rather than pitfalls I’d like to talk about difficulties. The main one is representative sample. Far too many companies are taking the easy way out and only offering surveys on smartphones or web-enable phones. That’s in spite of the fact that 30%+ of respondents in developed countries like the USA, Germany, Japan don’t have phones that connect to the web. The situation’s far worse in the developing countries.

There are obvious things that you have to get right, like not having long lists because order effects are much bigger in mobile than online or face to face; or like keeping the survey short. But those are just ordinary matters of avoiding bad practices.

Jan Hofmeyr

Jan Hofmeyr, chief research officer, TNS

How can the industry learn to manage the data mobile tracking generates?

Great question. Not that long ago researchers and clients were overwhelmed by the data-stream generated by passive listening apps. At TNS we’ve made a lot of progress since then. The main challenge is to organise and clean the data. But once it’s clean one can do great things with it. Two of TNS’s R&D initiatives involve generating valid brand health measures from the data; and a data taxonomy called mobile chapters.

When it comes to handling the data generated by surveys — that’s just a matter of standard data management except with a real-time element. Many companies, including TNS, have, or are, developing the capability to go straight from the phone to a client portal. We should be ready with a sophisticated live tracking offer in the next month or two.

You’re a proponent of ‘in the moment’ mobile tracking – why is that and why is the data from these more valuable/reliable?

The main benefit is that the data are more accurate because you don’t have to rely on a person’s memory. Their answer is based on what they’re actually doing, using, or seeing (or whatever). We should not underestimate the value of that. As a result we get much more accurate models of brand equity, drivers of equity, ‘in the moment’ feedback as to the challenges that a service or brand may be facing in its market.

The second benefit is the ability to turn surveys around really quickly to deliver close to real-time feedback to marketers. Although marketers tell us they want greater speed (and the old days of a 4-6 week cycle just have to end), I’ve been a bit surprised that marketers are also telling us they don’t want information in real-time when we offer it.

How can mobile tracking support predictive analytics to show future behavioural trends?

Again I think this is just a matter of much greater immediacy of feedback. For instance, we may segment someone in a way that predicts they will be one brand rather than another; or that they’ll respond to one kind of offer rather than another. With mobile we can get in the moment feedback that validates our predictions. Self-reported behaviour after the fact in traditional surveys isn’t as accurate.


8 years ago

Well said again Jan! Lets see how long it takes for the industry to change... still waiting! :-0

Like Report

8 years ago

Liking the idea that we push away from online trackers in some respects. You ask why do brands continue to use them? I think because if they don't they have nothing else to work with. Many brands have also been successful using online brand trackers, so they will also need convincing to move away from something that 'does the job' to something they see could be better but is untried and would require a lot of change internally within their business to adopt. I am sure there are many products we all use day to day which do the job but do we switch them all to the better solution? How many of use change our energy supplier to one that is better or cheaper? Personally I love the way mobile research is going, but for me its not about just having respondents answering on the phone, its about engaging them at the right time to do it. So when they are about to buy something, how do we know? How do we get the survey to them to understand at that exact moment when they are reaching for the product on the shelf, we get them to answer a survey as to why they are making that choice. Asking them on a phone even 1 hour later is not going to really pick all the nuances of the decision.

Like Report