Economist and author Tim Harford urged researchers to embrace failure in the first session of Research 2011 this morning.

The world we live in, Harford argued, is just way more complex than we realise.

“Almost anything we do in the world is an interaction with an almost impossibly complex economy we’ve created around ourselves,” he said.

As a result, so-called experts in all sorts of fields are actually very poor at making predictions about the future — not because they’re stupid, but because “the world is just too complicated for single human minds to get their heads around. It can’t be done. It’s too difficult.”

The answer, Harford believes, is trial and error — an approach that frequently yields success, but only after failure, failure and more failure.

“You solve impossible problems through trial and error,” he said. “The difficult thing is dealing with the politics and psychology of when things go wrong — as they will.”

“We hate the idea of trial and error,” said Harford. Governments are unwilling to experiment because they can’t face the prospect of failure, and voters shun “flip-floppers” in favour of those who convey an air of certainty.

Harford said what many researchers (particularly in the field of new product development) will have been thinking, but don’t want to say: that failure is just part of life. People love to take potshots at research by quoting statistics on the proportion of new products that fail, but Harford reminded us to look at the bigger picture: most new products fail because most things fail, full stop.