Should norms be the norm for making decisions?

I recently saw a client brief for creative testing and a key criterion for agency selection was the availability of norms. This struck me as strange; not the fact that the client was asking whether agencies have a normative database (fair enough) but the fact that the availability of norms seemed to be the main factor used to decide which agency to work with.

In my view this is wrong, because while norms can be useful to inform a decision, they shouldn’t be used to make a decision and an over-reliance on norms can lead to wrong decisions being made.

Unfortunately, I see a risk of more wrong decisions being made as research becomes increasingly automated and ‘do it yourself’. This isn’t due to research automation itself, which should be an enabler of better decisions because the reduced price and increased speed mean that more decisions, such as which creative or innovation to progress, can be informed by consumer insight.

My concern is the ‘DIY’ aspect – someone who isn’t an insights specialist will probably make decisions on the basis of ‘traffic lights’ or similar short-cuts which show whether a concept or creative copy performs better than a norm. There are many reasons why this can lead to the wrong decision being taken.

Being precise on the key questions the business needs answers for should determine the research design, the analysis that needs to be conducted and any action standards or norms that should be used to inform decisions on whether or not to progress the idea.

This may mean that the best research design either doesn’t require norms, or relevant norms don’t exist. In some cases, using a norm could cloud the decision or lead to the wrong decision being taken.

For example, a campaign might focus on building salience or it could be intended to build advocacy and loyalty among existing users. Given these very different objectives, it would be more relevant to look at the action the campaign delivers against relevant groups at different stages of the marketing funnel rather than look at creative performance versus norms among a broader group.

In food and drink categories, many of the most successful innovations of recent years have not (at least initially) been mainstream but have started at the category edges. Such innovations may have created a new category, targeted a growing sector or launched at a premium.

Brands such as Oatly, Halo Top and Fever Tree have each done at least two of these three things. These products have become very successful, but if they had been concept-tested against a category norm, I suspect they wouldn’t have been progressed, especially if tested priced.

Rather than just comparing to a category norm, the decision for whether to progress these types of ideas would be best made using analysis for which norms are unlikely to exist. For example, what is the appeal among vegans and the environmentally conscious, among people who want low calorie treats or people who buy premium gin?

An experienced researcher will be able to advise if scores are high enough among these groups to merit progression, even without norms. None of this means that in these cases norms wouldn’t provide value in informing the decision to take, alongside other data points. However, in such cases, norms should not be the focus for decision-making as it could very likely lead to wrong decisions being made.

When norms are used, we need to consider what they do well. In my experience, they are great for helping you to identify the real stars or the absolute duds among any stimulus you are testing. If your concept or creative is in the top 20% it’s highly likely you’ve got a winner among the group the norm is based on and if it’s in the bottom third you can usually throw it away.

However, if you can only be confident with using norms to identify the stars and the duds, that leaves a big proportion of the ideas you will test, almost 50%, in what I’d call the fuzzy middle. Here, there are many traps that you could fall into when using norms and a more nuanced approach is needed to decide whether to progress the creative or innovation – factors that require more detailed analysis and knowledge of how innovation or advertising works. 

Additionally, it is important to consider the make-up of the records in the normative database to identify how relevant and robust are the norms being used to guide the decision that needs to be made.

To summarise, a clear focus on the questions the business needs to answer should inform the research design, analysis and consequently the decision about whether to progress creative content or innovation. The availability of norms is a factor to consider but shouldn’t be paramount in deciding which tool or agency to work with.

Phil Sutcliffe is founder of triple i