Tim Harford on the pros and cons of AB testing

Tim Harford, Financial Times columnist, author and Undercover Economist, has a love-hate relationship with AB testing.

“I wish it was more widely used,” he said, after recounting the story of Matt Parker, head of marginal gains for the British Olympic Cycling Team.

‘Marginal gains’ might not sound exciting, but in competitive sport, they can make all the difference.

Harford tells how Parker’s invention of heated hotpants helped keep Chris Hoy’s thigh muscles primed for the wins that led him into the history books as the most successful British Olympian, with six gold medals to his name.

Parker also came up with the idea of rubbing alcohol on cycle tires, to remove a thin film of dust – shaving fractions of seconds off race times.

As Harford tells it, Parker’s innovation process involves examining, in detail, every aspect of the sport to identify and test small ways of making improvements.

“This has become a very popular model of innovation,” says Harford – but it is not right for every occasion. Despite his belief in the value of AB testing, Harford says: “Something makes me uncomfortable about innovating through marginal improvements.”

“It’s a seductive idea,” he says. “But it’s not the only idea. We are in danger of thinking that it’s the only way to innovate.”

Here, Harford gives the example of the birth of the Spitfire – a “most interesting experiment” that was far from guaranteed to succeed, but which was pursued anyway. The end result helped turn the tide of the Second World War and made Hitler rethink his plans for an invasion of Britain.

“There was no reason to expect it would work,” says Harford, “but it was worth a try because the possible gains were so great.”

The Spitfire was a long shot that paid off. “The Matt Parkers of the world will give you results every time,” he said. “But there is a role for more risky research.

“Marginal improvements are good for exploring new territory, but you need those long shots to reach new territory,” said Harford.