Don’t go chasing waterfalls

The research process is a seamless funnel-like argument, cycling through objectives, methodology, design, analysis, conclusions and recommendations. It requires curiosity and perspective to land the right answers. It also needs a military-like co-ordination among engaged stakeholders to ensure timely impact. What it does not need is traditional project management, the vice of which are Gantt charts mandating that each phase flows into the next, like a series of irreversible waterfalls, cascading toward a known finish.

A waterfall removes impetus to learn. The fiercer the current, the more research aspires to be a series of data verifications ending in death by PowerPoint. It financially incentivises a wilful ignorance of the novel and unexpected. Research surges on, despite management already having taken the decisions weeks before. Fresh information or solutions that do not fit the original plan, tossed onto shore.

Experience tells us that we can’t learn anything new when we know what we are looking for. Our most celebrated research programmes are not linear and their most commercially valuable returns impossible to premeditate. Brilliance visits us serendipitously and only when we challenge assumptions, revaluate priorities and rapidly adjust course. For too long we have prized tightly defined objectives, success criteria, pre-specified deliverables, and anything for that matter, which discourages ambiguity. Listen and you will hear insight professionals across the board screaming for agile.

Agile is not a framework or methodology, nor should it be confused with organisational tools, structure, meeting formats and job titles. Rather it is a set of beliefs and principles fanatical about working smarter. Software houses and their clients adopted agile because it encourages strategies and behaviours suited to a volatile, complex and dynamic market. Project management is deemed counterproductive and expensive. I stop short at suggesting that it is sensible or even possible to apply the same set of management practices across research, data science and software development as their economics differ. However, as these three worlds continue to converge, traditional project management must take a hike.

We are fortunate, then, that agile already reflects the reality of how researchers and data science teams try to work. We iterate and experiment wherever possible, traversing projects with multiple stakeholders and data sources. We already trust that reacting to change is better than following a blueprint, and we downplay e-mail chains, form filling and documentation in favour of face-to-face conversations where collaboration eclipses negotiation. We make infinitely more impact when everything we do is directed at the most commercially valuable, tangible outcome. Agile is a door we are all pushing on, both agency- and client-side and beyond.

Sceptics peer through the keyhole and see chaos, devoid of scope, costings and timelines. They find agile naïve, lacking in intent, transparency and accountability – a sneaky way to shift risk, to call for blank cheques. Stalwart project managers stand ready to pay lip service to agile’s jargon. Worse still, some doff their caps to ‘flexible and adaptable insight’, unaware how increasingly vapid anything short of agile sounds. As was the case in the software industry, this window dressing, alongside any debate on the relative merits of agile will be shortlived. Unable to take advantage of change quickly and efficiently enough, they await their extinction, trapped in waterfalls. There is no middle road.

Meanwhile, this industry is busy forging stronger, fluid partnerships with tiny feedback loops – the kind required to cycle through the research process more frequently in smaller, shorter, sharper bursts. Unbeholden to outdated priorities, we are increasingly encouraged to follow the data wherever it may lead; without the stigma of hitting dead ends or switching hypotheses, we will learn more and build more. May we be free to explore and capitalise on every opportunity, no matter what a Gantt chart says.

Ryan Howard is director advanced analytics at Simpson Carpenter