The devil is in the detail

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

In the past few years, the field of psychology has suffered several blows to its reputation, as many of its well-known, established findings have not been replicated in recent studies. This has led to big, wide-sweeping headlines that miss several important points:

• Many studies for specific behavioural science concepts, such as framing and anchoring, have been successfully replicated, sometimes finding even stronger effects. These positive results are seldom reported, however, because good news doesn’t sell
  
• Headlines often highlight the findings of large-scale replication studies that analyse a wide range of psychological findings – most of which are unrelated to applied behavioural science
  
• Our understanding of behavioural science can be deepened and strengthened, rather than weakened, by replication failures. This is often when we learn that tiny, contextual differences or nuances can affect the outcome in unexpected ways, frequently overlooked by the headlines.

So let’s look more closely at the replication studies conducted recently. The first Many Labs replication project, led by Brian Nosek in 2013, was a much larger and more robust study than the originals. It found that effects such as anchoring, gain versus loss framing, reciprocity and sunk costs could be replicated. In fact, its findings on anchoring revealed even stronger effects than the original studies by Amos Tversky and Daniel Kahneman.

In the second Many Labs replication project, Nosek and his team completed an even larger study, attempting to replicate 28 classic psychology studies. The way these were written about illustrates my second point. Headlines focused on the fact that ‘only 14’ of the 28 studies could be replicated. However, the majority of the studies tested were looking at much broader findings from psychology, such as morals, personality, trolley dilemmas and fate, not relevant to behavioural science.

The third point can be illustrated by looking at a ‘snail trail’ of linked studies over time, allowing us to consider the impact of contextual differences on behaviour.

Fritz Strack’s original 1988 study on ‘facial feedback’ (smiling your way to feeling better) involved asking participants to rate how funny a set of Far Side cartoons was while holding a pen in their mouth. Half the participants held the pen with their teeth, forcing their mouth and face into a smile. The other half held the pen with their lips, forcing their face into a frown. The ‘smilers’ rated the cartoons marginally funnier than the frowners.

In a 2016 replication of the Strack research, Dutch researchers Wagenmakers et al ran a study across 17 locations and involving almost 2,000 participants. On average, researchers found no facial-feedback effect, but Strack argued that the replication had altered important aspects of the experiment design and context. In the replication study (unlike the original) a camera had been directed at the participants. Strack believed that being filmed may have interfered with their emotional responses.

In 2018, another replication of Strack’s study was carried out by Noah, Schul and Mayo. This version tested whether or not the presence of a camera made a difference. Researchers were able to replicate Strack’s facial-feedback findings when no camera was present, but unable to do so when it was present. This illustrated that small changes in context can make a difference and that a single study does not ‘prove’ or ‘disprove’ anything.

Implications

• Science is an evolving, dynamic process, and findings are never black and white. Scientific knowledge is indicative and provisional; we can never say anything is ‘proven’ – in fact, scientists try to disprove a hypothesis rather than prove it
• It is valuable to take the time to delve into the detail – small contextual differences and changes – to get an accurate picture of human behaviour
• Replication studies, particularly ‘failed’ ones, can tell us more and enhance our understanding of the science – not the opposite
  
• The replication studies in psychology are quietly strengthening many of the most important findings in behavioural science, especially those that regularly feature in a good practitioner’s toolbox.