Saturday, 02 August 2014

Tim Macer's research technology blog

Big data? Big problem

Fri, 25 May 2012

The Economist recently ran an article on “Big Data” in its special report on International Banking. Its assessment of banking is of an industry surprisingly resistant to embracing the internet as an agent of change in banking practice. It reveals, counter-intuitively, that the number of bank branches has actually risen by 10-20% in most developed economies – yet most customers pass through their doors once a year rather than once a week, as used to be the case.

The newspaper explains this paradox thus: banks with a denser branch network tend to do better, so adding more branches is rewarded by more business. But it’s business on the bank’s terms, not necessarily the customer’s. It does not increase efficiency – it increases cost. And, as The Economist points out, banks’ response in general to customers using mobile phones for banking has been lacklustre, even though customers love it and tend to use it to keep in daily contact with their accounts. It’s a level of engagement that most panel providers would envy.

All of which is to say that there are parallels here with our own industry. Here at Meaning, we’re just about to release the findings of the latest annual MR Software Survey, sponsored by Confirmit. In a sneak peek, Confirmit blogger Ole Andresen focuses on an alarming finding about the lack of smartphone preparedness among most research companies.

But what interests me is the Big Data – both in The Economist’s report and our own. The former offered a fascinating glimpse into the way banks were using technology to read unstructured text and extract meaning, profiling some of the players involved and the relative strengths of different methods. This is technology which is improving rapidly and can already do a better job than humans.

In our survey we asked a series of questions on unstructured text. Research companies – in embracing social media, “socialising” their online panels and designing online surveys with more open, exploratory question – are opening the floodgates to a deluge of words that need analysing: at least, that was what we suspected.

It turns out that half of the 230 companies surveyed see an increase in the amount of unstructured text they handle from online quant surveys, and slightly more (55%) from online qual and social media work. Yet the kinds of text analytic technologies that banks and other industry sectors now rely on are barely making an impact in MR.

Most research companies are barely scratching the surface of this problem. It’s not the only area where market research looks as if technology has moved on, and opened a gap between what is possible and what is practised. There’s much more on this in our report, which will be publishing in full on 30 May. Highlights will also be appearing in the June issue of Research magazine.

Comments (1)

SODA adds a dash of mobile fizz to Confirmit's platform

Mon, 12 Sep 2011

Not one but two technology acquisitions in the same week: Confirmit, providers of one of the most widely used web interviewing platforms for research agencies, announces it is buying mobile survey specialist Techneos, while Kantar Group acquires panel and technology provider GMI.

Robert Bain covered the GMI acquisition in an interview with David Day. My focus is on Confirmit and what it plans to do with Techneos SODA, the company’s flagship software product for both self-completion and interviewer-administered surveys on smartphones and handheld devices.

Confirmit’s chief technology officer Pat Molloy tells me: “The basic plan is the same as with Pulse Train… to bring the functions of Techneos SODA and all the best features into our Confirmit Horizons platform.” Techneos will be rebranded as Confirmit in the New Year, and Confirmit will be adding to Techneos’ development resources in Vancouver to start work on building an integrated platform. Existing SODA users will see development continue on the existing SODA platform in the short term, but eventually it will mean a switch to Confirmit for them.

“We hope we’ll do a good enough job that existing SODA users will want to move over. We are not going to mothball SODA for a considerable time,” says Molloy. “But eventually we will want to roll those customers over at no charge on to the new platform.”

Confirmit has recently added pretty advanced support for interviewing on smartphones into its flagship Horizon software. What it lacked was a dedicated app for mobile that could be run natively on the mobile device – surveys were still delivered via the device’s web browser and required a stable internet connection. Using an app offers many different benefits. I’m presenting a paper at the forthcoming ASC Conference in Bristol on the very subject (and will share the link to the paper after the event). There are pros and cons in each approach, but an app allows for much greater sophistication in survey design.

Techneos has been providing mobile apps before we even understood the term – starting with its interviewer-only Entryware product, originally designed for Palm Pilots. Techneos had appeared to lag behind on mobile self-completion until it brought out SODA at the end of 2008, which was a big leap forward.

Entryware, however, will be left to wither on the vine, on the premise that SODA provides most of its functions. This will be unwelcome news for any Entryware customers that have not yet switched to SODA, as they will be faced with two migrations.

Comments (5)

Mobile research growing up fast

Thu, 21 Apr 2011

Globalpark, organisers of the 2011 Mobile Research Conference asked me to chair day two of the event. I decided, rather ambitiously, to close the conference with a round-up of all the presentations that day. Here, in prose, is what I verbalised at the end of long day of very interesting presentations.

Don’t be surprised if you don’t recognise many of the names here. It is true to say that the early adopters of this method are not necessarily the usual suspects – there were some familiar firms present – but as the industry as a whole continues to see only problems with mobile research, it was illuminating to hear from those who are not only convinced of the value of mobile research, but are developing expertise, best practice and clients hungry for more.

In a few sentences for each session, here is what came up:

Bruce Hoang (Orange Advertising Network) presented a multi-country study of mobile media consumption by mobile data users in the UK, France, Spain and Poland – countries with marked differences in adoption and usage. He has concluded that web-optimised sites are more popular with consumers in mature markets than using apps to access content. “The web browser in the mobile device is the killer app” according to Bruce. He advocates sticking to web browser-based surveys on the mobile as it most closely aligns with respondents’ preferences and experiences.

Guy Rolfe (Kantar Operations) asserted that mobile apps for surveys definitely have their place. Kantar finds participants are willing to download survey apps, which can enrich their survey experience. In parallel with this, many consumer product manufacturers and retailers are now creating lifestyle apps that capture a lot of useful data which are proving to be very popular with consumers – they don’t have a research purpose at their heart but the data they collect they could be very useful to researchers if they can get their hands on it.

Jeroen de Rooij (ValueWait) presented a lifestyle case study that proved it’s possible to use mobile research to ask 60-question surveys with modest incentives if you do it with care. The survey also asked respondents to email in pictures after completing the survey and a very high proportion were willing to go to this effort.

Peter Lynn (University of Essex) explained that from a social science and statistical perspective, the focus of scientific survey literature has tended to emphasise the negative – seeing mobile samples as a problem. This needs to be questioned. If you take a Total Quality perspective, there are many areas in which mobile samples are no better or worse than others – coverage may be better, in fact. There is also a one-to-one relationship between respondent and phone unlike landlines. Other sources of error are reduced, e.g. people are more willing to answer some kinds of questions, it avoids recall error by taking place ‘in the moment’, and overall, the responses are not otherwise fundamentally different from other modes. It’s strength surely lies in complementing other modes.

Michael Bosnjak (Free University of Bozen-Bolzano) and Sven Scherrer (Globalpark) took us through some early results of a study on how useful voice capture and voice recognition might be in overcoming the Achilles’ Heel of mobile research – capturing lengthy verbatim responses. Low response and high drop-off is often observed in mobile surveys when these questions get asked. The study pitched standard touch-screen entry with voice capture and voice recognition. From the preliminary results presented, voice did not come out well from a respondent perception point of view. Touch screen entry was preferred over voice entry – voice recognition was the least preferred and the spread of responses indicated a divergence of opinions here. Interestingly, respondents seemed to warm to those methods, particularly voice capture, when asked about it five days later. The actual effect on the data has not been measured yet: those results are due out soon.

Justin Bailey (The Nielsen Company) presented a case study on using BlackBerry Curve devices with a recruited panel of South Africans during the period of the World Cup which showed the extent to which low response really does not need to be a feature of survey-based research. The study was monitoring media and advertising consumption and some brand recognition over the period of the World Cup. Very high response rates were sustained throughout an extended survey. Pictures were collected too, and Nielsen ended with a library of 60,000 submitted pictures. The case study offered a real feel-good moment for mobile research.

Thaddeus Fulford Jones (Locately) has created a panel of mobile phone users in the US who are willing to allow the firm to capture location data and use this to model actual behavior. You can learn about the extent to which consumers go to some outlets and often will drive past rivals in order to reach them. Raw location data is used to identify locations such as retailers, leisure destinations and other important consumer touchpoints. It tends to be most powerful when combined with other data to provide context. Location data also reveals useful temporal data – e.g. how long people have really spent in a store or waiting at the checkout.

Hannu Verkasalo (Zokem) spoke of  “on-device measurement” or using the mobile phone for passive data gathering. What came out was just how much you can measure passively – free from the response bias of a survey – when using a mobile device, from sites accessed, search terms entered and time spent on different activities to location data: what was accessed at home, at work or on the road. He also revealed the very different ways that people consume mobile content on mobile devices compared to the web, and again the different profile of apps versus browsers in the content that people access. Hannu’s prognosis is that the mobile app is in the ascendant – which contradicted Bruce Hoang’s earlier analysis.

AJ. Johnson (Ipsos Mori) chaired a panel session entitled “Debunking the myths of mobile research” and asserted that research needs to treat mobile very differently. People will be engaged if you approach them via mobile research, but as researchers we have to be very transparent, open and honest with respondents.

Paul Berney (Mobile Marketing Association) challenged research to take a greater interest in mobile research. Mobile is the natural home of the digital native – the under 30s who have grown up knowing nothing other than the internet and the mobile phone. It’s already changing the way that retailers are working and it fundamentally changes the engagement model for brands. It is a mistake to think that mobile is about the technology – it’s about people. Mobile is a two-way channel and if we don’t go there with our research, then others will.

To round up, a few common themes emerged:

  1. Mobile surveys can be a bit longer than we may have first thought. 8-12 questions is a common-sense length, but examples were presented of 30 and 60 questions, and much longer, when carried out over an extended period. But is trying to push up the limit the start of the same slippery slope that has led to the downfall of online research?
  2. The experience in emerging markets and less mature markets is very different. The penetration of mobile is so high in emerging markets that it far exceeds every other channel except face-to-face – it is the natural equivalent of online research.
  3. In developed economies, there is an assumption that mobile research is a replacement for online. In reality, it seems to supplement it, and it is more of a replacement for telephone and face-to-face.
  4. Mobile research is not one thing – it’s a multimodal channel in its own right, embracing self completion, interview-administered, quantitative or qualitative, visual, textual, voice and image, or passive observational, which can be augmented with location or temporal data.
  5. The sphere of mobile research is changing fast and it is continuing to evolve. It is not something that research can afford to ignore.

Comments (1)

MR's twin-track approach to social media research

Wed, 16 Mar 2011

Social media research continues to be one of the hottest topics in research. I’ve just been reviewing the abstracts for this year’s CASRO Technology Conference in New York in June, which I will be co-chairing, and of all the topics, its the one with the longest string of submissions. Not only that, but there is some diversity of opinion into what it is, how to do it, and whether it adds anything at all to the existing researchers’ toolkit. Closer to home, it’s a topic that will be debated in next week’s Research conference in London too.

Social media research is also one of the new topics we focused on in the 2010 annual software survey, sponsored by Globalpark, which my firm, meaning ltd, carries out each year. The results of the 2010 study are published today. There are some curious findings – and some predictable ones too – that add perspective to the current debate on social media research.

Our survey of over 200 research companies of all size around the world, shows social media research is still at the early-adopter stage,  accounting for revenue-generating activity in just 17% of the firms surveyed. Close to the same number – 19% – say they are unlikely to offer social media research, and of the remaining 63% who gave an answer, 31% say they are either experimenting with it and 32% are considering it for the future. Small firms and research companies in Europe are the least likely to be doing social media research and are also the most likely to have ruled it out, whereas large firms are the ones that are most active. The actual volumes of work are still low – we also asked how much revenue social media research accounted for. It is 5% or under for  two-thirds of the agencies that do it and tails off beyond that – but there appear to be some specialists emerging, with a handful of firms deriving more than 20% of their income from it.

Many firms are bullish about the future, though, with 20% predicting strong growth, and a further 52% anticipating some growth, with North America, and again the larger firms, most optimistic about its future.

As a technologist, I was most interested to see what technology firms were applying to what is, after all, something born out of technology. Were the tech-savvy gaining the upper hand, or were researchers taking the conventional, low-tech approach beloved of qualitative researchers. Again, it’s a bit of both. Of all the software-based or statistical methods we suggested for data analysis, the one that came top, was “manual methods”, used by 57%. For analysis, this followed by 54% citing “text mining” (as correspondents could pick all that they used). Text mining, though it uses some computing power, is also very much a hands-on method – but it’s good to see more than half turning to this method. Other methods make much less of an appearance, and the method that I consider shows most promise for dealing with the deluge of data, machine learning-based text classification, was bottom of the list, cited by one in six practitioners.

For data collection, technology was much more apparent – although it is hard to avoid here. We were still intrigued by the massive 54% who say they are using manual methods to harvest their social media data from the web; 57% were using web technologies to collect the data, and the more exotic methods were also fairly abundant, including using bots (43%), crowdsourcing (41%) and avatars (24%).

I’ll pick up on some of the other intriguing findings from the study later. But as the report is out now, you can pick up your own copy by visiting this webpage – and there will be a full report in the May issue of Research magazine.

Research firms reveal nine month technology lag (and a secret affair with the Mac)

Tue, 14 Dec 2010

 

It's not the first time I've postulated that MR firms can be laggards with their technology. An interesting early finding to emerge from the 2010 Globalpark MR Software Survey, a survey among research agencies worldwide carried out annually by meaning, provides some supporting evidence for this by looking at the actual technology being used to access the survey.

We've been able to analyse the browser string returned by the 550-some participants who responded to our survey invitations, which were targeted exclusively at MR companies across the globe. The browser string - which many MR survey packages capture automatically - reveals exactly what browser and also what operating system the survey participant used. It's hard data, free from any response bias, as it is picked up from the routine chattering that goes on in the background between web server and web browser.

We thought it would be interesting to compare this with the current state of worldwide usage to see if MR differs, and if so, how. Overall, the figures are very close with respect to operating systems in use. We compared our figures with those from GlobalStats Stat Counter who measure usage in the same way, only on a somewhat grander scale: typically 15 billion page impressions from 3 million websites per month across the world.

The headline figure for Windows, at 93.3% among our MR participants, against the GlobalStat's worldwide figure of 92.0% is within a whisker. Perhaps surprisingly too is the 5.9% figure for Mac OS X usage - a squeak away from the global figure of of 6.2%. Most MR software providers produce only Windows versions of their software, and even web-based software - such survey authoring tools or online analysis programs, which could be platform independent - are often locked down to Windows-only browsers.

Chart showing OS usage 2009 to 2010

The trend, which we can see by comparing browser string data from our 2010 survey with the equivalent from our 2009 survey, shows Mac usage has surged from 2% to amost 6% in the last year. We cannot tell how many of these users were taking the survey at home on their own Macs (which is quite likely) but it's clear they were responding to a work-oriented email on a platform that most MR software managers choose to ignore. If this is a continuing trend, this minority will be increasingly hard for software developers to dismiss as undeserving of their attention.

We can also see some interesting things happening with the flavours of Windows being used. It is well known that most corporate IT departments gave Windows Vista a miss when it came out in 2007, choosing to stick with the the ageing but more reliable XP. Even now, as XP approaches its tenth birthday, it is to be found on 70% of the PCs in our study that have Windows installed on them (65% overall). Among our 2009 participants, it was found on 74% of the Windows PCs. In the meantime, what little share Vista had accumulated has now mostly been ceded to the newest Windows 7, which launched in October last year, just before our fieldwork period. It has now grabbed nearly a quarter (23% among Windows PCs, 21% overall).

Mobile devices barely made an appearance - less than 1% of our sample. There again, we'd expect most researchers to realise that taking our survey on a handheld device was unlikely to be a joyous experience. The same is not true for iPad users. We tested it on iPad and it looked good: but only two showed up among 554 research professionals who clicked the invitation link.

Chart showing web browser usage in MR firms vs global usage

Browser usage was also remarkably close to the global figures overall. Between last year and this, the major trends are that Internet Explorer has been losing share to both Firefox and Google's own Chrome browsers - and our MR sample show signs of the same trend. However, when we compared the trend with the global data, our November 2010 sample showed a remarkably close fit with what the rest of the world looked like around March and April this year. Worldwide, Internet Explorer dipped below the 50% mark in August and by November it had eased down further to 48%. Our November 2010 snapshot shows IE as the browser of choice among 54% of research users, down from 62% last year, while Firefox has gained strongly, and even Chrome and Safari have picked up users.

We've just started to look at the actual questions in the 2010 MR Software survey - we will have the results out in March 2011.

 

Online samples - where lighting a few fires might help

Mon, 2 Aug 2010

 

Struck match and flame

In Greek mythology, it was Prometheus who stole the secret of fire from Zeus and gave it to humankind. The new non-profit organisation being launched by rival research giants GfK and Kantar to address industry-wide concerns about the online survey quality, seems to make a nod to the myth in its chosen name, the Promedius Group.

The industry’s concerns about online research are many and various, but a common complaint is the lack of transparency of sample providers in the composition of their samples and the extent to which these overlap. It’s worrying enough that, as response rates dwindle, research firms are probably already relying on less than 20% of the population to answer more than 80% of their surveys. But what if it is the hyperactive 0.1% of the population that turn out to be answering 10%, or 20%, as some fear, turning survey data into junk? Without the vantage point of the gods, no-one can really tell what is happening.

Good research is always a balance of art, craft and science. The risk is that if survey results are no longer generally reproduceable, any claims to scientific validity are lost. Those that spend the big money in research, like General Mills and P&G, have noticed this, and are highly likely to start putting their money in consumer intelligence gathering elsewhere unless research can be made more accountable again.

The solution is staring at us from the problem. There is a vast trail of data that can be associated with every web-based interaction – put it all together and it becomes possible to pinpoint individuals and identify, within reasonable probabilities, that they do seem to be taking 20 surveys a day, or that they are very unlikely to be a gynaecologist because the digital wake emanating from the same PC speaks more of college student. Getting at this data, however, is much more difficult. If you are a big player, with a large volume of interactions, you can do this – but even the industry’s own demi-gods face a major hindrance, in that most of the panel providers don’t reveal the key information you need to start putting this information together, like respondent IDs or IP addresses.

Promedius will, it appears, be making use of a lot of technology to match data and perform checks on data, and they will be making this technology available for other research companies to use. This is welcome news, as the problem has been proving too big for anyone to solve on their own. There are already commercial services – MarketTools’ Truesample and Peanut Lab’s Optimus to name two – and these have gained some traction. They also add cost, and are restricted to some extent by only ever showing part of the picture – from those samples and providers that have opted in.

With three major players backing this initiative (Ipsos were involved in the development of the technology behind Promedius) it is likely that it will have the critical mass that is needed for it to become established. What the technology does, and how affordable and convenient that is (the announcements do not say that this will be offered to the industry for free) remains to be seen. I’ll be looking to secure a software review as soon as it becomes available. But there is a good chance that Promedius will be putting fire into the hands of the people, as far as panel and online survey transparency is concerned.

Hopefully Promedius will enjoy a better fate than its near namesake, who after several other skirmishes, found himself bound to a rock by the vengeful Zeus, with an eagle visiting him every day into eternity to peck away his liver.

Her Majesty's Cross-tabs

Tue, 13 Jul 2010

I suspect market research is not often mentioned at Buckingham Palace. So it was pleasing to be summoned to the presence of the Queen’s representative in London, the Deputy Lord Lieutenant, to a bit of modest pomp at a business part in North London, on the cleared carpet-tiled floor of  E-Tabs Ltd. We were here to witness a little bit of MR software history being made. This MR software developer  has just won the Queen’s Award for Innovation. It’s an award in one of three categories among the annual Queen’s Awards for Enterprise: essentially honours for firms, and an award not easily come by.

Queen

Benjamin Rietti, MD of E-Tabs, receiving the Queen’s Award for Innovation from Martin Russell, Deputy Lord Lieutenant of Greater London

As the DLL said “this year there was a total of 414 applications nationally for the Innovation category, of which only 38 were awarded.” In fact, just three other firms in London received the 2010 Innovation award, making this quite an achievement, as E-Tabs was competing against every imaginable industry sector.

Clearly, more than one thing turned the judges’ heads in E-Tabs. The Sovereign’s representative was not that specific. As with the honours list, we never get to learn what really clinched it, but he mentioned sustained development since the first E-Tabs browser emerged in 1993 (then the ITE Electronic Fiche) and the product’s contribution to ‘significant improvement in business performance and commercial success’ and also the extent to which E-Tabs had probably saved a few forests ‘through the paper you save your clients’. I’d add to that the extent to which these guys get it about what matters in market research at the delivery end.

While others have been building palaces at the data collection end, E-Tabs has focused on finding better ways to create and publish research data, and in an admirably democratic way. They don’t dictate how you collect your data or what platform you use to process it. They are masters of collaboration and rigorously agnostic about data formats. They have gone from one product for reading static reports to a whole family of products to publish reports, churn out PowerPoints or create web portals. Their latest offering - and boy do we need this - software for automatically checking tables and detecting inconsistencies through a number of probability-based deduction methods.

So, hats off to E-Tabs! Congratulations on an award deservedly won. But don’t let it go to your heads, guys,  there’s still room for more inspiration on the results delivery side of our industry, and that innovation needs to continue to flow.

Comments (1)

How will iPad affect MR?

Tue, 1 Jun 2010

 

iPad displaying a website

iPad displays content-rich websites like research live with astonishing clarity, given its modest screen size.

Apple’s new IPad launched in the UK on Friday and one has happened to find it’s way into my hands pretty early in the day - in the interests of research, of course. It is a truly impressive device every bit as iconoclastic as the iPhone and possibly more so. iPhone could always be seen as a next generation smartphone delivered with Apple’s customary flair for designing what everyone else was trying to design - but didn’t.

With iPad they have come up with something no-one was even thinking about. It isn’t really like anything else and this explains some of the criticism and nay-saying there has been about it. E.g “Kindle does a better job at being a e-book reader”, “If you want to do any serious work, you’re better off with a tablet PC”, “do we really need another device alongside the desktop, the laptop  rand the smartphone?”

If I were to sum up what it is, it is a personal device for using the Internet, free from the overhead and constraints of using a personal computer. It is a content consumption tool with some exceedingly convenient capabilities for interacting and responding too.

iPad is superb at displaying web pages - not in any way restricted for size, as many assume due to it’s paperback book-sized dimensions. In fact, full-size 1024 pixel wide pages are crystal clear and readable.  I was relieved to see all the text on our own website and on others, like the Research Live site are legible without any zooming in required. The sound is good and rich without resorting to earphones and video is astonishing: is on a par with HDTV.

It’s do-it-nowability is its strength

The on-screen keyboard has come in for criticism, but it really isn’t a bad way of entering text. Maybe not for writing a report but for email, entering searches, filling in forms for online shopping and the like it is entirely up to the job. It helps if you are used to the soft keyboard on iPhone as it is simply a bigger version of that. This blog entry, to prove the point, comes to you from an iPad. Did it take longer to write? Yes and no. If I had used my laptop it would have been written faster. But probably not until Monday. It is the convenience, the do-it-nowability of iPad which is it’s strength.

iPad standing up on a desk

Apple’s carrying case for the iPad also acts as a handy desk stand, when flipped open.

It is heavier than some might like but the metal case and glass screen actually give it resilience and a quality feel. It would make an extremely convenient tool for face-to-face interviewing. Much more portable then a tablet and not pinched for size like a PDA.  A whole day’s interviewing on a single battery charge. Where a wireless network is available or cellular G3 coverage, surveys could be set up and run using any unmodified web survey tool, provided it did not use flash. I am sure we will see some extensions to existing CAPI products to provide specific support for iPad which would then allow interviewing to be done off-line too. The quality of the multimedia support means there would be no reservations or technical hangups to showing TV quality stimulus materials for ad testing, for example.

iPad will be the must-have device for 2010 I have no doubt. According to some new research from Intersperience, online consumers do see a need for getting one, and there could be 7 million of them in the UK in 5 years time. That means it is inevitable that people will be taking surveys on iPads.

Researchers need to start thinking of how their surveys might be ‘consumed’

All the more important then that researchers consider where and how their surveys are going to be “consumed”, and shed a few assumptions of the past. Prime among these must be to ensure the survey experience delivered to the participant does not jolt them back a couple of decades from the rich, colourful and conversational world of the Internet today to one more like an exam paper on screen. What iPad does is raise the stakes in survey design.

Comments (3)

The curse of seeing everything

Mon, 29 Mar 2010

 

Model of a brain showing neural activity

A major issue with post-modern research methods, or ‘new MR’ as it is sometimes called - a recurrent theme at Research 2010 - is the amount of data and consequent effort that goes into extracting any meaning from this data. This came home in the new technology session, chaired by Robert Bain and billed as ‘Research Unlimited’.  Not that any of the technology being presented was essentially new - naming the session “incremental developments in technologies based around memory and newly applied to market research” may have added precision, but not made the message any clearer.

Neurometrics

The pursuit of clarity should be at the heart of any new methods – and that is a challenge with two of the methods showcased  based on neurometrics: from Nunwood’s head of R&D Ian Addie and Millward Brown’s new head of ‘Consumer Neuroscience’, Graham Page. Page is probably the first MR staffer to have the N-word in their job title.

Improvements in EEG measurement and analysis technology  make the approach more affordable and slightly more applicable to surveys in the real world, but they still have a long way to go. The electrode caps and camera-rigged spectacles modelled on stage by Addie, and even the slimmed down version shown by Page, are still pretty clunky and intrusive. Addie also cautioned that ‘noise’ in the data collection meant that 30 per cent of the data they had collected had to be discarded.

Positivism with a big P

Both speakers showed that this kind of data can aid understanding, and can usefully cast a new light on some deeply held assumptions about consumer behaviour, which is no bad thing. Nunwood respondents who had been wired up with electrodes for supermarket visits had revealed that a significant amount of time in selecting products seemed to be spent in rejecting other projects - not something that is much questioned in conventional recall studies. As research was busy going po-mo in other sessions, this looked like a rallying call for Positivism with a big P.

Page cautioned: “Hype means it is very easy to get carried away with exaggerated claims [for neuroscience]. The results don’t stand on their own: you have to combine this with something else.”

Not only that, but you quickly accumulate a vast amount of data that takes time and effort to process. Furthermore, to give any meaning to it, you must be applying the qualitative judgements of the researcher or neuroscientist.

This additional burden was also true of the other novel method in the session. Here, Bob Cook from Firefly presented an interesting extension to diary research - particularly those studies that lean towards the auto-ethnographic - with a methodology based on Life blogging, or ‘glogging’ using a small fish-eye camera worn by the participant around their neck. This can take a shot and capture everything the respondent sees, paced out at minute intervals throughout the day. Cook reckons it can overcome the usual problems of incomplete recall that can arise over the more mundane and automatic activities respondents may be asked about.

Making sense of the data

The problem, in trying to move such techniques into the mainstream, comes at the analysis stage. To get meaning from these techniques takes extraordinary effort - and they are not amenable to the analytical methods conventionally applied to either qual or quant. We’re not usually short of data these days, but we are short of tools to make sense of these new streams of data. Without them, analysis is inordinately time-consuming. Technology makes it easy to add precision in volumes, but with all these new methods, it falls heavily on the researcher to bring out the message.

Comments (1)

Mobile fallout - to be ignored at your peril

Tue, 16 Mar 2010

Mobile research as a method may still be in its infancy, but researchers already need to be aware of the fallout from the growing phenomenon of mobile communications, both in telephony and in data communications and the mobile web. These effects cannot be avoided and need to be better understood. It was clear from last week’s Mobile Research Conference, organised by software firm Globalpark, that respondents are already taking online surveys designed for conventional PCs and laptops on their web-enabled smartphones in small but significant numbers. The response to simply exclude them by closing the survey when an iPhone or other smartphone is detected is not a neutral decision from the sampling perspective.

Hands holding a range of mobile phones

But neither is it smart to exclude them simply because the survey then behaves in a way that does not let them continue, or which makes it difficult for them to select some responses. It is no longer safe to assume survey participants are using a conventional browser as their preferred means of accessing the Internet, and that trend will accelerate as other portable devices, such as Apple’s iPad and the imitators it will spawn, start to emerge.

The same is the case with mobiles replacing landlines - the figures I found were that 20% of households in the USA were mobile only last year, and that is likely to be 25% now - so a quarter of the population will now fall outside any RDD sampling frame in the USA. Marek Fuchs from the Technical University of Darmstadt, in one of the sessions at the event that I was chairing, presented some astonishing figures on the extent to which people were giving up their landlines in many other European countries at an even faster rate. He presented a Europe-wide average from Eurobarometer data in late 2009 of 30%. It is even higher in some counties, notably 66% of people in Finland and 75% in the Czech Republic who have only a cellphone to answer.

Is this a problem? Including mobiles in sample frames is not impossible, but is expensive, and in many cases means going the panel route. We also learned, from Peter Lynn & Olena Kaminska at the Institute of Social and Economic Research at Essex University, that the differences in responses to the same questions in telephone and mobile surveys were mostly benign, though there are differences - and in their experiments mobile phone interviews tended to last slightly longer, for reasons not fully understood yet. Yet I come across research companies whose blanket policy with CATI studies is simply “we do not include mobiles”. This policy is unlikely to be sustainable even in the UK, where 15% have gone mobile only, for very much longer.

Mobile web may not quite be mothholing sampling frames to the extent mobile voice is to CATI, but the greater cause for concern here is just what these respondents do who do participate online. Mick Couper, who knows more about interview effects than anyone, warned that the effect on completion is barely understood yet, but one thing is clear - making assumptions from web surveys would be very risky. Even if survey tools are set up to convert full screen surveys for gracefully to the smaller format, as  Google’s Mario Callegaro said, a concern for him would be to know on what basis this conversion was being done and what lay behind the decision-making process adopted by web survey developers or software providers.

The uncomfortable truth is that we just don’t know the answers to many of these questions yet.

Bad data can mean more than bad research

Wed, 27 Jan 2010

Bending the rules a bit on who meets the quota should never be seen as a ‘technical’ breach of any research code of conduct, no matter what pressure we find we are under to meet the target. Karen Forcade’s admission that she passed off research findings in the USA that had been manipulated to meet the client’s sampling criteria  (see “Youth Research boss pleads guilty to federal fraud charges”), and in some cases, simply fabricated interviews, is all the more reprehensible because the study in question was for child safety relating to lighters. Forcade, along with another researcher at this now-defunct agency are due to be sentenced: it could mean jail for them.

It’s an extreme case, but it serves to remind us that our combined labours are about delivering the truth.  Often, what emerges from our inquiries is indistinct and contradictory and getting to the truth may legitimately involve editing both data and findings to bring it out. We also know that some respondents provide false data – and it is not a problem that only afflicts online surveys. Discretion is called for in choosing what to edit and how to edit it, and wherever discretion and professional judgement enters a process, so too does the potential for abuse. Respondents fake interviews because they’ve been promised trinkets and tokens. Forcade faked interviews because her firm gained around $15,000 for each safety study they did: higher stakes and also greater consequences, though quite why she did this, is still hard to comprehend.

Yet most errors are more mundane in origin.  From my own observations, and conversations I’ve had with those who work with data on the technical side, data quality is an issue that large areas of the industry have become complacent about. Execs and project directors look far less now at actual respondent data than they used to.  And while eyeballing the data will only uncover some problems: error detection is really best done using automated methods. Yet few research firms seem to be putting effort into innovating in this area. Manual processes for error checking seem to abound, focused on checking tables while other parts of the research process that will introduce error (scripting, editing, data processing, coding and report preparation) are largely left to their own devices.

Yet every time I’ve been involved in the migration of a tracker from one supplier or one technology to another, errors have  emerged where published findings have been found retrospectively to be just plain wrong. Only yesterday, I was talking with a technology provider about automated error detection, and we both found we had several anecdotes that illustrated this very situation. In one case, it was simply that the labels for one regional report had been inverted – the more the manager tried to get his low satisfaction score up, the lower the scores went. He was about to lose his job when someone spotted the error. It seems the unfortunate manager had actually been doing rather well.

Research does not need to be wilfully misrepresentative to do real damage to lives and reputations.

I’m curious to know if others have observed this problem too, or how they police their data.

Comments (1)

Double standards, triple costs

Mon, 11 Jan 2010

According to a recent white paper produced by Oracle’s BI Consulting Group (“The Great Debate: buy versus build”) it costs between two-and-a-half and three-and-a-half times as much to build your own business intelligence and analytics system than buy and adapt an off-the-shelf solution. It must be admitted, the white paper concerned is a marketing piece to support this firm’s services in - you guessed it - adapting standard Oracle systems to provide BI solutions. But their figures are based on experiences with some 250 large-scale implementations, which is a decent number to conjure with.

They also found that such projects when built from scratch by system developers averaged 34 weeks from start to finish and consumed 925 person days, whereas implementing then adapting a generic platform took half the time - 17 weeks - and less than a third of the effort, with an average of 290 days required. When you consider the amount of effort involved in managing a project twice as long, and the drain that this has internally on the stakeholders involved with prototyping and early adoption, there are also likely to be sizable hidden internal costs that can be avoided. MR applications may differ in some respects, and the development projects around them may be smaller in scale, but the experience is unlikely to be radically different from this scenario.

There is a surprisingly large amount of custom-built MR software around, developed in-house from scratch. It’s not all legacy stuff either. I continue to hear of firms developing their own tools across the range of research applications, from panel management through to dashboards and portals. As the article from Oracle points out, their experience is that the out-of-the-box solution tends to meet between 70% and 80% of needs, so development effort is concentrated instead on the 20% to 30% that needs to be accommodated.

Of course, it is always possible that, lurking in that 20-30% are some very nasty problems that require almost infinite resource to resolve, but that remains true whatever route you take. Perhaps decision-makers are not comparing like-with-like when opting for own-grown. An off-the-shelf solution has known limitations - these limits are usually cited as reason for rejection. Despite this, they are limitations that may often be overcome through customisation. Open systems now make it readily feasible to customise rather than bribe your software vendor into changing their core software product or live with the limitations. Yet it is a choice that is often neglected.

The build-your-own route is seductive – theoretically, it appears to have no limits in what is achievable. The danger lurks in those obscure user requirements where effort starts to spiral towards infinity in achieving them, and where hard decisions are needed – always assuming you’ve noticed in time.

If you can find something where 70% of the work was already done without having to think about it, surely that’s a good thing? Yet, “it only met 70% of our requirements” is often the siren call that lures project teams into the unfathomed depths of what astute system developers call “the vanity project”. It’s flattering to be told your requirements are so special that a system must be built from scratch to meet them. It’s also very very unlikely. But to counter vanity with vanity, who likes finding out they paid more than three times over the odds?

Comments (2)

The long tail for tech firms

Thu, 10 Dec 2009

We will only know whether we were at the turning point of an L, a W or a U when looking back. Nevertheless, my own unscientific poll of firms I’ve spoken to in the past few weeks confirms that, although the recession has been acutely felt by MR technology providers, things seem to have been looking up slightly since people got back to work after the summer. Some tech firms have been busy, even very busy, and some have continued to grow despite the downturn. Being inexpensive or on a short track to adoption seems to help here. Another factor seems to be the needs-led solution: an agency client needs a custom panel,  a web-based analysis tool, a dashboard – reactive rather than strategic purchases.

At the same time, others have been putting on a brave face, weathering out the storm and continuing to develop their products. Exhibition organisers are going to have a tough of it time next year. Tech providers are wincing at the costs of going to the big shows at a time when very few are buying. For one, just the charge for electricity levied by the venue was sufficient to wipe out all profit.

Even the firms that have remained busy are reporting that it is taking much longer to close the deal. People will talk for 18 months or longer about a £25K order but it never seems to materialise. Others find the orders they do land have been scaled back considerably from what they were asked to bid for.

It makes me feel that MR firms are still not approaching their technology from a strategic point of view. As I reported in June, several research companies at CASRO were seeing a slowdown in work as being the opportunity they needed to get their processes and tools in order for when life got busy again. Is this opportunity being squandered?

There are many tales out there of firms never coming to a decision, seeing almost everyone and rejecting them all, or having virtually an annual review and still sticking with the same set of ageing or complicated tools that require high levels of skill and effort to operate them. This is symptomatic of technology decisions being delegated down to those in the organisation who are perceived to understand them: unfortunately it is often those who have the greatest investment in being indispensable masters of the dark arts who hold most sway. Perhaps they did reach the right decision, though I’m often unconvinced. But even if they did, I’m often unconvinced it’s for the right reasons.

Where are the tools to enable Web 2.0 research?

Wed, 18 Nov 2009

Researchers cannot afford to ignore Web 2.0 approaches to research, as Forrester’s analyst Tamara Barber makes clear in a persuasive article on Research Live, in which she settles on market research online communities (MROCs) as being the most effective way to achieve this. How to do Web 2.0 research, from a methodological point of view, is engaging a great deal of discussion at MR events this year.

In her piece, Ms Barber has focused on social or participatory characteristics of Web 2.0, where there is obvious value to research. But the other characteristics of Web 2.0 lie in the technological changes that have emerged from its 1.0 antecedents - that the Internet becomes a platform for software, rather than a delivery channel for information. Indeed it is technology - using Ajax, Web services, content integration and powerful server-side applications - that are as much the hallmarks of Web 2.0 as the outward manifestations of the social web. It’s on the technology side that there is a lot of catching up to do, in the world of market research, and until this gets sorted out, Web 2.0 research will remain an activity for the few - for patient clients with deep pockets.

The specialist tools we use in research are starting to incorporate some Web 2.0 features, but nowhere does this yet approach a fully integrated platform for Research 2.0 - far from it. Panel management software is morphing into community management software, but the Web survey tools they link to don’t make it easy yet to create the kind of fluid and interactive surveys the Web 2.0 researcher dreams of. Neither are the tools to analyse all of the rich textual data that come out of these new kinds of research truly optimised for all forms of Web 2.0 research data. There are pockets of innovation, but multi-channel content integration - a key feature of Web 2.0 sites - is still difficult, so researchers are still drowning in data and left running to catch up on the analytical side.

Another problem arises too as more ambitious interactive activities and research methods emerge: the demands on both the respondent and the respondent’s technology increase, and some are getting left behind. Participants find themselves excluded because their PC at home or at work won’t let them run the Java or other components needed to complete the activity - whether it’s a survey, a trip into virtual reality or a co-creation exercise, and their PC won’t let them upload what you are asking them to upload. Even relatively modest innovations such as presenting an interactive sort board in the context of an online survey or focus group will exclude some participants because their browser or their bandwidth won’t handle it. Others simply get lost because they don’t understand the exercise - there is a growing body of studies emerging into the extent to which respondents fail to understand the research activities they are being asked to engage in.

New Scientist recently reported on innovations taking place in gaming technology where the game learns from the level of competence demonstrated by the player and uses this to adjust the game’s behaviour. It’s the kind of approach that could help considerably in research. Unlike online gamers, we can’t ask participants to spend more than a few seconds in learning a new task and we can’t afford to lose respondents because of the obvious bias that introduces into our samples.

For Web 2.0 research to move beyond its current early-adopter phase, not only do researchers need to take on these new methods, but research software developers also need to be encouraged to take a Web 2.0-centric approach to their developments too.

Predicting an outcome for Dimensions under IBM

Thu, 1 Oct 2009

Now that the dispute with SPSS founder Norman Nye has been settled, it looks very likely that the IBM takeover of SPSS will go ahead. Though SPSS spokesperson Heena Jethwa put up a spirited defence in an interview here last month (Commitment to MR ‘as strong as ever’, says SPSS), the question remains for the hundreds of users of SPSS MR-specific software worldwide, including Dimensions and the legacy Quancept and Quantum software: just what is IBM going to do with these product lines?

I happen not to share Ms Jethwa’s upbeat assessment that MR customers are likely to be even better off with IBM. The evidence of the recent past, even under the SPSS régime, does not bear this out. SPSS has not had the zeal for MR that was evident when it had its own SPSS MR division and the core Dimensions range of products have not, in my view, kept up with the recent pace of development or innovation from rivals often much smaller than SPSS - but wholly focused on MR and survey-based activities. Make no mistake, there is some excellent technology for MR in the Dimensions range, but the firm’s once mighty grip on the specific and changing needs of research has been allowed to slip.

Whether this is the cause or a symptom is hard to say, but over the past four or five years, SPSS has allowed almost all of its specialists in MR to leach away - most have been snapped up by other MR software firms or research companies. There is no longer a strong force of internal champions for MR within the firm to drive the innovation that is needed.

So what might happen under IBM? IBM has re-invented itself as a software and information management provider, and it went for SPSS for its predictive analytics software to strengthen its range in the business information area of its activities. However, predictive analytics will still only appeal to a minority of actual users, and in relation to that, all the complicated MR stuff it will be acquiring too sits inside a niche within a niche. Even if there were still a strong team of internal champions within SPSS, they would have their work cut out to convince IBM to invest in specialised products with such limited mass market appeal.

Perhaps the best outcome would be for IBM to spin off its MR products. The worrying thing is that, for a firm the size of IBM, simply collecting the rent from existing Dimensions tenants and letting the house fall down over time would have virtually zero impact on its business overall: the MR software market is such a tiny vertical market in global terms. If shareholders vote to accept IBM’s offer on Friday, it would be a good time for the SPSS MR Users’ Group to convene an emergency meeting.

Comments (1)

Measures of happiness

Wed, 16 Sep 2009

Yesterday’s FT had two lengthy stories on the inadequacies of the GDP as a narrow measure of success. “GDP branded a poor gauge of progress” examined the report from the Commission on Economic Performance Measurement, headed by Joseph Stiglitz, and another “France to count happiness in GDP” reported President Nicolas Sarkozy’s attempts to persuade other countries adopt these reporting reforms as France is doing. It formed part of my reading on the long flight back from South Korea.

Happiness is a word that crops up frequently in South Korea – and it appears surprisingly often in official literature and publicity. It appeared in the highly polished video presentation at the start of the IWIS09 workshop, organised by Statistics Korea. Compiling official statistics was, amongst other things, ‘helping to increase happiness in Korea’.

It came again after the conference was over, and as one of the group of international delegates, I was was treated to very generous hospitality by Statistics Korea, as they laid on a ‘Cultural Tour’ of Daejeon and Seoul. The first stop was the Statistics Museum - yes, there actually is such a place - at the Government Complex in Daejeon.

 

View of exhibits in the Statistics Museum in South Korea

The Statistics Museum in Daejeon, South Korea

 

Our visit to the museum started with another video enthusing about the value of statistics to the public. ‘Increasing happiness’ was again presented as one of the outcomes. We asked how they saw happiness. “Happiness is about building a better future, about how things become better than they were in the past,” one of the people accompanying us from Statistic Korea explained.

This is an advanced country which not only suffered brutal occupation by the Japanese from 1910 to the end of the World War II but then the bitter and extremely destructive Korean War with North Korea between 1950 and 1953. South Korea founded its governmental statistics service in 1948, and the museum was full of early paraphanalia from census and survey gathering activities. For me, the most interesting glass case contained an 80-column punched card counter sorter from the 1950s and then a 9-track half-inch  mag tape and tape drive from the 1960s. There were also glass cases containing long and complicated paper questionnaires – which all in all seemed the best place for them. I could now see how online surveys were contributing another increment to national happiness.

 

Yesterday

Yesterday’s survey technology - an 80-col card counter-sorter and 9-track mag tape drive and media

 

South Korea is about as capitalist a nation as you can get - yet embedded in their psyche is that one of the most important outcomes of this progress is happiness, though improvement in personal circumstances and a better way of life for all. It feels as if Europe and North America are running to catch up on this one. Still, it’s a good omen for survey-based research. After all, how else do you measure happiness?

Tricks to improve response online (and offline)

Wed, 16 Sep 2009

The second day of the IWIS09 Internet Workshop in South Korea focused on practical measures and finding in improving response in online surveys (following on from Some Korean insights into MR and Online is the future for national statistics).

Jan Zajac (University of Warsaw) overviewed factors which can drive participation rates in online surveys, both to boost them and, in some cases, diminish them too. His own experiments, carried out in Poland, optimising email survey invitations to boost response found that including a picture of ‘the researcher’ made a surprisingly large improvement to response. Less surprisingly, pretty, young and female researchers seem best in pulling in the respondents – though not only from males but females too.

Pat Converse (Florida Institute of Technology) revisited Dillman’s Tailored Design Method to see the differences in response rates to in mixed-mode paper and web surveys, and the extent combining both best improves response. It seems paper is far from dead. His analysis across a wide range of published survey results results seem to show that a 34% response rate is about middling for Internet only surveys whereas mail surveys still typically achieve a 45% response. In his experiment, he looked at how effective using a second mode to follow up non-response at the first mode can be - and clearly it will improve response. Surprisingly, the greatest improvement was in following up a web survey invitation that had got nowhere, with an approach by mail: almost 50% of those approached responded, taking overall response to 72%.  The best response came from mail first with web as the fall-back, though this is likely to be the most costly, per interview. Web first, with a switch to mail could hit the sweet spot in terms of cost, when a high response really matters - such as for a low incidence sample.

As an event, these two days have effectively provided a cross-section of the state of current knowledge and inquiry into Internet research. There was talk of making the papers and presentations available, and if so, I’ll provide a link here.

Comments (1)

Some Korean insights into MR

Thu, 10 Sep 2009

More insights into market and social research in Korea emerged in day two of the Internet Survey International Workshop, hosted by Statistics Korea.

South Korea is one of the most technically advanced nations in the world, with a young and growing population. Virtually 100% of those aged under 40 are Internet users and across the board, South Korea ranks eighth globally for Internet penetration: higher than both the USA and the UK. Using Internet panels is therefore very appealing for national statisticians and social researchers - if only ways could be found to overcome coverage and non-response bias.

Sunghill Lee (UCLA) proposed an advance on Harris Interactive style of propensity weighting, to nudge panels towards national representativeness by supplementing propensity weights with a stage of calibration against a reference dataset which nationally representative, or from a true random probabilty sample. Her model was capable of halving the observed discrepency, but at a cost, as the sample variability tended to increase.

A time for everything

Prof. Cho Sung Kyum (Chungnam National University, Korea) had noticed others’ attempts to weight their panels in the direction national representivity tended to use demographic data, including some measures that were hard to calibrate, such as occupation. There is often frustration in being able to get hold of robust reference data. Prof. Cho had noticed that many national statistics offices around the world conduct a Time Use study among the general population. These meet most criteria for good reference data - large, robust, random probability samples that are representative of the population, and they are highly associated with lifestyle. They also cover Internet-specific information, as one use of time which is tracked in these studies, in some detail.

In his test online surveys, he asked respondents some time characteristics that could be cross-matched, such as the typical time home from work, typical bedtime and time spent online. Matching by six measures, his model provided near perfect adjustments for questions relating to leisure, education or media consumption; but it offered no improvement for income or work-related questions. However, his work is ongoing, and he hopes to identify other variables that could narrow the gap in future.

Online on a slow burn

In MR, online research has only a ten per cent share in Korea, an astonishingly low figure given the very high Internet penetration in Korea, stated Choi In Su, CEO of Embrain, an Asian panel provider. Face-to-face still tends to dominate, as telephone is not particularly useful either with less than 70% of Koreans having a fixed phone line. However, he predicted quite rapid change, expecting the share to reach 20% or more.

The reluctance among MR firms also stems from the same concerns that the statisticians had been airing - coverage and non-response error, and low quality in particiation. Mr Choi outlined a number of unusual characteristics of the Embrain panels designed to combat these limitations - which include a combination of online and offline recruitment, rigorous verification of new recruits against registers or other trusted sources, a range of fraud detection measures, and good conduct training for panel members. A key measure of success is the consistent 60% response rate from survey invitations.

It felt as if the social statisticians were ahead of the game. Kim Ui Young from the Social Surveys division of Statistics Korea spoke of two successful online introductions of large-scale regular surveys. A key driver had been to reduce measurement error and respondent burden, and one diary study of household economic activity provided a good example of this. In fact, Kostat had gone as far as to work with online banking portals to allow respondents to access their bank statements securely, and then import specific transactions directly into the online survey, which a lot of respondents found much easier to do.

In my concluding blog entry, tomorrow, I will cover the highlights from international case studies and new research on research, which were also presented today.

Online is the future for national statistics

Wed, 9 Sep 2009

 

I’m at the First International Workshop on Internet Survey at Daejeon, Korea. It is hosted by Statistics Korea (or Kostat) which has put together an impressive roster of presentations on leading edge thinking in using online research for public policy research and other nationally representative surveys: eighteen speakers, fourteen from around the world, and a nice fat 320-page book of scholarly papers to accompany the event.

My own talk was on software and technology (what else?) and how appropriate technology can help control measurement and non-response error: but unlike many of these events, I did not find myself the pariah for speaking technology. There has been explicit acknowledgment throughout the first day of this two-day event for the need for researchers to be more discriminating and more demanding of the technology being used, in order to improve response, reduce respondent burden and control error more effectively — as well as reducing cost.

The event started with Yi Insill, the Commissioner of Statistics Korea, who predicted “a significant increase in demand for Internet Surveys” in National Statistics work in Korea. “We are expecting them to reduce non-participation and make them engaging for participants,” she stated. She also acknowledged that national statisticians had been reluctant to use online surveys because they were not based on random probability samples and “have been criticised for poor quality”, but that was now changing as the methodology was being understood and tested. Preparations were well advanced for the 2010 e-Survey in Korea, and we heard more of this later on.

One good paper followed another - but I will pull out a few highlights. Frederik Funke (Tübingen University) showed how Visual Analog Scales (VAS), when applied to online surveys, can dramatically reduce measurement error, while showing that conventional 5-point scales, applied to online surveys by convention (and possibly for no better reason) can enforce measurement error on participants by restricting their options - to the extent that different results will arise from a VAS which appear to be more accurate.

Surveys that leak cash

Lars Kaczmirek (GESIS, Mannheim) followed through with three practical changes to survey design that would improve response and reduce error. He showed the results of some experiments that showed how, compared to the effect of providing an incentive on a survey or not, some simple changes to survey design were actually more effective. In other words, you could chop the incentive, improve the design, and still be slightly better off in terms of response.

Kaczmirek was also critical of the way in which new technology was sometimes applied to surveys uncritically, even though it would increase non-response. Another example was the automatic progress bar - inaccurate or misleading progress bars, particularly those that jump due to routing, are such a turn-off to respondents that actually removing them altogether will often improve response. Accurate bars, or bars where jumps are smoothed and averaged out, do better than no bar, though.

Boxes for Goldilocks

Marek Fuchs (University of Kassel) gave us the latest thinking on verbatim response box size and design in online surveys: getting the size right can mean more characters and potentially, more concepts - like Goldilocks and the porridge, they should not be too small or too large. Adding in a Twitter-style count of how many characters remain can also boost response length, provided the starting number is realistic (a couple of hundred, not a thousand characters). However, too much trickery, such as dynamically appearing or extending boxes will also send any gains into reverse. As with the wonky progress bars, the point is that any feedback must be realistic and honest for it to act as a positive motivator.

Questionnaires with added AJAX

Peter Clark (Australian Census Bureau) talked us through the 10 per cent uptake of an online census option in Australia for the 2006 Census, and the plans being made to increase this now to 25% for the 2011 Census. ACB had appointed IBM as its technology partner for 2006 and again for 2011. IBM had pioneered adding browser-based processing in AJAX (a Web 2.0 technology) to the 2011 e-Census form, to cut down server load. It has saved them a fortune in hardware requirements, as the server load is now a third of what it was. For the many participants on slower dial-up connections, the form took longer to load, but once loaded, was actually faster, as all further traffic to the server was minimal and therefore very fast to the user.

Australia, along with other speakers describing their e-census strategies in Singapore and Estonia, had added an online option to the national census as a means of reducing cost. For the obvious coverage reasons, e-census is offered as an option to back up self-completion by mail, and as a last resort, face-to-face for non-responders.

Pritt Potter (Webmedia, Estonia) spoke of the work he had done in providing novel respondent validation methods to the forthcoming e-census in Estonia, which included using trusted third parties such as banks to offer verification through their normal online banking security, and then pass on to the census bureau key identification data. Another method offered to respondents is mobile phone verification (provided that the phone is registered). Both methods have the advantage that the public can respond to ads in the media, visit a website and self-verify, instead of the census bureau having to send out numerous unique passcodes.

And there is more in store tomorrow…

Translation on the fly (or on the sly?)

Tue, 8 Sep 2009

World Wide Lexicon Toolbar is a new plug-in to the Firefox web browser that promises to take webpages in any unfamiliar language and, as you browse, simply present the pages in English (or for non-English speakers, the language of their choice).  My preparations for the trip I am about to make to Korea have focussed my mind on the frustrations of being unable read webpages. But I was also curious to see how useful this would be to Web 2.0 researchers that are analysing social media content and the like.

It is a very smart add-on: if you browse a page, and it isn’t in the language you understand, the page will be machine-translated and presented to you. If a human translation has been made, it will show this instead. It surpasses the Google option to machine-translate pages in a couple of other ways, too: more languages are covered and the translated version is presented in the format and style of the original page. There is even an option to double up the text so you can see the original and the translation. Of course, the translated text may still disrupt the layout, but it gives you a much better idea of the context of the text,  which aids understanding considerably.

Human or machine translations

The software is currently in beta, and can be installed free-of-charge from the  Mozilla Firefox add-ons page. Reports from early adopters are that it is extremely useful, provided that you are willing to put up with the limitations of machine translations. The human translations it shows are those that have been entered by volunteer contributors to the World Wide Lexicon community. It’s a fantastic idea and is another example of the wisdom of the crowd at work on the Web. Yet the reality for any social Web researcher is that the blogs and community forums you are likely to visit will not have attracted the attention of a community-minded translator, and you will still need to endure the inadequacies of the machine translation.

Machine translations are not bad with well-constructed texts that have been written in a stylistically neutral way, but the more colloquial and idiomatic the text is, the more bizarre and worthless the translation becomes. I don’t have the means to try this out, but I suspect this tool may be more useful when doing Web-based desk research into more authoritative sources than the general Web 2.0 free-for-all. For that, we need machine translations to get smarter.

A catch?

Why on the sly? You need to login and register to use the service, and the server must, by definition, be aware of all of the pages you visit - so you are giving to the plug-in owner a complete trail of all your browsing activity. This is not made clear when you sign up. If it bothers you, you could only use Firefox when you wish to translate something, and another browser for what you wish to keep private.

Tim is at the First International Workshop on the Internet Survey this week, organised by Kostat, the Korean National Statistics service, and will be posting highlights from the event.

  • If you have tried out this plug in or any other comments concerning machine translations, please log in and leave a comment.

Comments (1)

View results 10 per page | 20 per page | 50 per page