The Great Quality Control Debate

The creation, maintenance and enforcement of research standards are having a bold impact on the international research community. We bring together five expert voices to debate some of the chewier issues surrounding the quest for quality

Research brought together five influential and experienced figures in the MR industry: Rowland Lloyd (RL), the chair of the Market Research Standards Board and managing director of operations at Ipsos Mori; Phyllis Macfarlane (PM), managing director of GfK NOP; from the US Diane Bowers (DB), the president of the Council of American Survey Research Organizations (Casro); from Europe, Dr Lex Olivier (LO), the director general of Efamro; and from the clientside, Crispin Beale (CB), the director of insight, intelligence and analysis at the Royal Mail Group.

Our topic: the role of professional and processing standards in the industry. Today we have a global market for research where regulation and standards regimes are not consistent. We are faced with declining response rates and declining quality of research in some areas. We have a market where clients seek to drive down costs, but where research has never been more in demand. For all these reasons standards have a vital role to play. Yet the industry is certainly not unanimous as to what that role will be, as our small sample demonstrates.


Research Magazine (RM): Are clients willing to pay extra for good standards in research?
CB As a user of research, high quality data is absolutely fundamental. If you have an insight that's not based on robust data and then you decide to launch based on that, you could waste millions of pounds. So for me standards are fundamental, almost priceless. I would always pay more for robust conclusions. If you are researching your pricing and you get it wrong, your competitors will wipe the floor with you. Low-quality research is worse than a waste of money; it's dangerous for the business. The question for all of us on the clientside is, "What risk do we want to take?"


PM I would say it's a complete hygiene factor for our clients. If you're a big agency, the clients assume you know what you are doing, and that you deliver to a standard. But they don't always know what the standards are; it's a complete position of trust, so they don't actually end up paying for external standards.

I was at a conference recently and a guy from a very large client was there, and we were talking about access panel standards. He told me that he had no idea there was a problem until he was asked to speak – he just assumed we all knew what we were doing.


RL I believe there is a willingness to pay with some clients, and that spans both commercial and social. If we help them understand that good standards ensure more accurate results, they are worth the price. Some clients will naturally come to this conclusion, and some will need an explanation. My question is, are we as an industry doing well enough in explaining this? I'm not always sure that we do.


LO My experience is more that clients heavily rely on their own experience and pay for what they think is quality: they judge the quality of an agency, and at the moment they are satisfied with that. The moment they see lousy quality, they take that agency off the list. That's how they tend to think.


DB Clients support good standards, and quality, but the responsibility for providing good standards is with the research provider. How much clients are willing to pay relates to the overall service given to them. So there's a reality check that providers need to make. Too often we accede to client desires for faster, cheaper research, instead of pushing back. We need to make sure they understand that applying standards in this age, with all the difficulties of collecting data, means that the cost of good research has in some cases gone up.


RL But if the suppliers explain what standards do, we're not pricing ourselves out of the market by adhering to them. They have a cost, but if we implement them pragmatically and efficiently, they don't have a disproportionate cost. The argument I would use strongly is that good standards mean we get things right first time more often, and in that case it makes our operations more efficient. So there is a cost to standards, but that can be minimised with a pragmatic approach to ensuring that our people are trained and supported to follow them.


PM Up to a point. There is no way we agencies could go to clients and say "we are ISO 20252-compliant, so we're putting an extra two per cent on the bill".


RM If there is a willingness to pay for standards for high-value research, are these standards appropriate for internet research, for example?
CB If you just want to get a feel for the market quickly, it is possible to whip through a quick online survey that isn't as robust. If I want to launch a product and there are three possible colours to test, it doesn't matter if my margin of error is 20 percent. Even if I get it wrong, it doesn't cost much to put it right later.


LO At the cheaper end of the market, standards like ISO 20252 are also an agency issue to level the playing field. We must see to it that no one enters market with low prices and lousy quality. The client will find out his mistake one day, but it will take too long, and it can ruin the price levels in the market, so we need those prices to sustain quality levels.


DB In the US, standards and quality are presumed among legit professional researchers. The Casro code is truly an industry standard – whether or not the provider belongs to Casro. It's not so much that standards need to be impressed on people, but that they are presumed when clients buy research, so we need to make sure we maintain our diligence and see they are applied across the board.


PM We all need to be trustworthy, and that's not always the case. In some online methodologies for example, we've got to a position where almost anything goes. We have to be clear that if we are delivering samples that are not as representative as in the good old days, we are transparent about what we are delivering. It is about clarity.


RM: In that case, can an equivalent standards regime be applied in developing economies?
CB It would certainly help us if the international regime was tougher and more consistent. We operate in a world where there are no physical boundaries. We need to understand that the quality of the data in Taiwan or Australia is as good the data in the UK. My experience is that you can rely on data from the UK to a far greater extent than elsewhere, and I don't want to stand up in front of the board and tell them my results and find out that the data from the rest of the world doesn't stand up.


DB This is problematic. From our perspective, there isn't one standard but there are principles of ethics, professionalism and process that need to apply across the globe. You need to be very careful about how onerous and bureaucratic the standards become for developing markets – will it tie their hands? We do want to help emerging countries comply to a common standard, but you have to be careful not to put all this into a single standards regime; instead we can make sure that research processes and practices have good guidelines that can be shared and taught. We have an obligation to provide help and education to help them with their research.


RL When you are looking at how you weight data for example, that's a process issue – it has nothing to do with location. Statistical rules decide what you should do. In other areas, you have to be very careful – but as Diane says, you can agree the principles without over-proscribing them.

That allows a standards committee to decide whether principles are upheld or not. I think this is being recognised pretty well. ISO 20252 has been constructed by international teams; they have taken account of different conditions in markets very well.

LO When you look at subcontracting fieldwork-only agencies – they report to me that ISO 20252 is a real blessing. They say it's good because normally they have to make service-level agreements with Argentina, Colombia, Peru, wherever – and they have to negotiate all the agreements, or fully describe the procedures in place, and rely on good faith of partners internationally. For them a standard makes it very transparent. This is the quality level, this is the price.


RL I think ISO 20252 is extremely important to set a good level of process quality internationally. Clients like The Royal Mail are trying to look at results in different markets across the world, and if the standards are substantially different, and that is impacting results, it's a major problem. Bringing consistency is a good thing but it will take time. ISO 20252 is a brilliant tool to be driving forward, but we need increased coordination and cooperation between the MRA, Esomar, Efamro, Casro and equivalents around the world to try to encourage consistency between the professional codes. If we do that alongside the process standard, we are headed in the right direction.


PM I think it depends on whether you are a cynic. The clients who buy the global projects assume the global research companies are providing a worldwide standard, and I'd be surprised if many clients trouble their heads about it. They assume it's in place. The days when we used to sit down with clients and have big methodological discussions about problems like this are long gone.


RM: Even if we have a consistent regime, how can we enforce international standards meaningfully?
LO Applying for 20252 is appropriately tough. Look at interviewer training for example: in ISO 20252, you have six hours of training, three hours face-to-face. The standard is spreading over the world from Japan to France, Spain and Italy, and that has to be good.


RL In MRS, almost all breaches of the MRS code will be made public, so we can't be accused of self-regulation behind closed doors, and so there's a consistent approach. These sorts of sanctions need strengthening at an international level, and not just a country level. If you have equivalent standards but not equivalent sanctions, it undermines those standards. We need auditing companies to agree on the standards or audit internationally. There are good signs, but a lot more work to be done.


LO There will be a moment in all of these countries where they say, "We must improve our quality". One of my clients is responsible for research in Africa for example, and conducting robust research there is a real problem. Lots of bad practice happens in Africa – fraud, copying interview lists, bad processing, bad punching – so there is lots of work to be done. It's a similar situation in South America. The alternative – and something that we have done before in the Caribbean – is that you get the questionnaires, put them in a box and say, "send them to me" so that all the processing is done in the UK for example. For employment reasons it is better to do the processing locally; and that also means the local economies will benefit if they have a good quality system. When they realise this, ISO 20252 gives them a point of reference, they don't have to think too hard about what those improvements might be. Instead they buy a copy of the ISO 20252 document, and work to implement it.


DB In the UK you have a terrific relationship with the government, and there's a strong identity for research which in the US isn't as strong. But in the US we're the only country in the world that has laws against selling under the guise of research. I think those legal constraints would be wonderful in other countries. It wouldn't solve all our problems, but it would go a long way to solving one problem.


RM: Do we make enough of standards? Should we market ourselves more to clients based on our high standards?
PM We have to. Research can be done so easily now by anyone: internal research departments, someone who set up a business last night – there are few barriers to entry. If the traditional industry doesn't say that there are standards that we adhere to, we'll be in trouble.


RL I feel strongly that if we present the benefits, and we are careful not to over-engineer – so that the standards are relevant to the way we work – clients will pay for it. But we need to do a good job in convincing them first.


DB Absolutely. But first we have to fundamentally change our mindset toward what we are doing. The problem is not so much with our standards, but our resistance to recognising that the individual respondent, the information-sharing participant, wants to have more information and control of the research process and is insisting on it. We have to ensure that we acknowledge their needs. At the same time, the clients have an increasingly strong need for fast, reliable information that is actionable. Research agencies need to look both ways and sometimes push back when client demands affect the standard of our work.


RM: Does a tough standards regime mean that UK- or US-based research is more expensive? Potentially implementing onerous process standards could price agencies out of the market when they compete globally.
DB The negative view is, there are a lot of cheap schlock operators out there, and we have to say, "Be careful". In the UK you have a strong ability to use standards to go after those who are hurting the industry this way. It's more difficult in the US to do that.


RL There's always the risk of higher prices. I believe it's fairly minimal. My argument is that standards help organisations work efficiently, so they will get payback. Also, if you are a fieldwork-only company, you can still be assessed to the fieldwork-only IQCS standards; so it's a case of horses-for-courses.


DB We really need to push back at clients. Truly the quality of research has little to do with the ability and talent of the agency, often it is more to do with the difficulty of collecting the data, and meeting the immediacy of the needs of the client. I don't think UK and US agencies are pricing themselves out of business, but we do need to press very hard on low cost providers overseas who are building 'unawareness' among clients. Perhaps we should do more industry benchmarking ourselves, so we can say to clients "you get what you pay for", and back that with evidence.


PM I don't think we price ourselves out of the market. In the UK we've all been in ISO or BS for a while. It's part of the process. It doesn't result in any additional cost.


LO No, I've never noticed the UK is particularly expensive. I would say that if the UK is expensive, it's about high staff costs and the currency rate, not standards. Compare the UK with France or Germany, and there's not a big difference. In fact, France used to be very expensive - but that was the social cost of employing the interviewers, not the cost of implementing standards.


CB I agree. When I've conducted international research, the UK hasn't been at a price disadvantage. Some agencies that work outside the MRS code of conduct have a habit of promising the earth and not delivering much. If the clients want to save money this way, they might as well act without any research at all and save time.


RM: Some small market research agencies allege that a standards regime is a way for the large, established providers, who can absorb the cost, to grab market share. Is this fair?
CB I don't think it is. I've worked with a lot of one- and two-man bands who can offer research that conforms to the MRS code of conduct. Process standards mean something: there's no point in offering an innovative method and then letting clients discover there are big holes in it.


PM I don't think a standards regime shuts anyone out, even if they are small agencies. Most of the standards aren't that onerous. If anyone says "we don't want to check a proportion of our phone interviews" or who won't back-check personal interviews, how do they know what quality they have? The standards protect us all from rogue respondents or bad interviewers.


LO And so far the big research agencies have not generally dominated their local markets. I don't see it happening. If there is a battle for market share, it will be commercial rather than about quality.


DB In the US the laws around unfair business practices are very strong, so we have to be careful we don't break the law by excluding anyone. But that is why all our associations are pressing for voluntary best practice guidelines. But saying, "you can only do research this way" to new agencies would not work in the US.


RL I don't think small companies should be exempted from standards. I don't think it would be a benefit for them. I think standards help them play on a level playing field. It's important we have standards for small companies – whether a small company or a large one lets the industry down, it still affects the industry. I think the standards as they are designed – codes of conduct as well as the process standard – are relevant to big and small companies alike. And all good companies should want to be following those standards.


PM OK, so there's a cost to being registered, but if I was setting up a small agency tomorrow and wanted to avoid that cost, I would simply say to my clients "we work to those standards - we're not registered,but we can demonstrate that we follow them".


ISO 20252: A definition
ISO 20252 is the first-ever set of agreed global minimum quality standards for the market, opinion and social research industries. Launched in March 2006 after two years development involving 22 participating countries. The new standard is designed to put market research firms across the world on a more equal footing, overcome barriers to trade and reassure both internal and external clients that agencies can do basic research.


• Respond to this article at marcb@researchmagazine.co.uk


April | 2007