A question of evidence: Policy and research

x Sponsored content on Research Live and in Impact magazine is editorially independent.
Find out more about advertising and sponsorship.

At the height of Covid-19, the mantra of ‘follow the science’ became a justification for reams of government policy. It carried anti-Covid-19 policy through lockdowns, social distancing, masks, tiers, travel restrictions and, eventually, to the end of restrictions altogether. Throughout, the pandemic rumbled on.

The question remains: to what extent was there an evidence base behind much of the Covid-19 policy?

And it is a theme throughout government. From education to immigration to transportation, it can be hard to separate the policies that derive from political values and ideology from those that come from an evidence-based approach.

Evidence-based policymaking is broadly the idea that public policies and decisions should be based on or informed by rigorously established, objective evidence. Most UK government departments have a policy formulation framework, which guides civil servants through evidence-based policy development.

The general approach taken by central government in the UK incorporates working out the rationale for research and objectives, appraising and analysing the current situation, deciding on a preferred policy and delivery, implementation, and finally evaluation.

The pandemic, given the prominent public role of the Scientific Advisory Group for Emergencies (Sage), was a golden opportunity to showcase the benefits of evidence-based policy making. The problem, however, was that despite best efforts, the nature of a newly discovered – and highly contagious – virus was that often policy needed to be developed and implemented ahead of sufficient rigorous evidence which was coming to the fore.

Tracey Brown, director at Sense About Science, a campaign organisation calling for more scientific approaches to policy-making, says that the pandemic underlined a growing scientific literacy among the general public.

“Politicians believe the public is intolerant of uncertainty and evidence,” she says. “What we saw at the beginning of the pandemic was an amazing amount of tolerance for the government being in a difficult decision-making role with very little evidence.

“They could have done more to share the emerging evidence base.”

For Stian Westlake, chief executive at the Royal Statistical Society (RSS), Covid-19 provided a considerable boost to the status of evidence-based policymaking. “We’ve seen data brought into government decision-making in a range of fields, from public health to economics,” he says. “We’ve also seen the government take much more care in how it uses evidence to explain and justify its actions. Data bodies such as the Office for National Statistics have taken a leading role and I think statisticians and data scientists have done a good job in engaging with government, both behind the scenes and publicly to improve communication and understanding of data.”

Strength or weakness?

The benefits of introducing evidence-based policymaking are multitudinous, says Dr Cosmina Dorobantu, co-director and policy fellow on the public policy programme at the Alan Turing Institute. “The most tangible benefit of evidence-based policymaking is that the world that we live in – and the people who inhabit it – are not the subject of a policymaker’s experiment,” she explains. “Without evidence-based policymaking, those who govern us have no other option but to try various measures in the hope that they find one that works. But trial and error, in a policymaking context, can have long-lasting consequences on people’s lives.

“If you get it wrong in a pandemic, many people will die unnecessarily. If you get it wrong in welfare, you are exposing some of the most vulnerable people to even more hardship. If you get it wrong in education, a cohort of children will not have the skills they need in their adult lives. When people’s lives and livelihoods are on the line, taking some of the guesswork out of what might or might not benefit them is a duty we all have.”

The problem is that while there appears to be plenty of research informing government policy, it is hard to be sure to what extent evidence-based policy is being implemented.

According to Brown, work by Sense About Science with the Institute for Government showed that there was a lack of transparency about what evidence central government departments used in their policymaking decisions, which has led to the formation of the Institute for Government’s Policy Transparency Framework. The framework, published in 2015, offers a rapid assessment to rate government departments on how transparent they are in the use of evidence in policy decisions. This includes examining the ‘diagnosis’ – the issue that will be addressed – and then the policy proposal, implementation and evaluation. Sense about Science took the lead in testing and refining this framework in 2016, in collaboration with the Institute for Government and the Alliance for Useful Evidence.

A ‘transparency of evidence’ review by Sense About Science made several observations, while also noting that a large amount of good work was being carried out at government level. The review, among other proposals, suggested better sharing of work done within departments, better referencing, sharing modelling and assumptions behind models and better descriptions for plans to test and evaluate policies once they have been implemented.

The ability to share information is crucial to evidence-based policymaking, but government – as shown with Covid-19 – has been poor in actually sharing the evidence, says Brown, which effectively renders examining the success of a policy much more difficult. Then there is the research that never enters the public realm. “What tends to happen is that government does work on a subject, and often with really good subject specialists within government – sometimes their work is actually better than anything in the academic environment – and what happens is it doesn’t get published properly,” she adds. “It gets chucked into an archive somewhere, and the next generation come through the department and end up repeating themselves.

“It is really frustrating that there is a lot of work that goes on – both reviewing and commissioning external work and internally government generating its own – which never sees the light of day. There is a lack of collective memory and understanding across government on what it has done before. There is also a huge democratic issue of us understanding the basis on which a policy has been proposed.”

For Dorobantu, there also remains the issue of an inherent incompatibility between hard science and policymaking. “Research takes time and needs resources: the two things that policymakers who want to change things straight away don’t have,” Dorobantu says. “Policy is fast-paced: ministers want answers on the same day they ask the questions. And they often struggle to see why they should invest some of the resources that they can put towards solving a problem in building an evidence base.

“This inherent incompatibility, however, does not mean that evidence-based policy making is impossible. But it does mean that it needs forethought, planning and commitment. You have to invest today to build the infrastructure and research capabilities that you will need to answer tomorrow’s questions.”

Putting it into practice

This culture clash is also evident in the relationship between data and analytics teams or academics more used to evaluation, and communications teams and political strategists who feel publishing information makes the government hostage to fortune. “It is far better to own the evidence and its inconsistencies and uncertainties yourself,” says Brown.

It is also important to remember that often there are values implicit in an evidence-based response. Take tax breaks for marriage, for example – some would argue marriage helps with social cohesion, but these days views on the policy often come down to agreeing with the principle or not.
“Some people in certain fields would like to see a situation where the politics were taken out of some things a bit more,” Brown says.

“I am not sure that is a feasible approach or the right answer. The crucial step we want is to know where evidence has played a role and where it hasn’t. It is much better to ask: ‘which parts of this are testable’ and ‘what are the good questions’, such as ‘is there a good evidence base we could use?’. If not, what are the questions that need to be explored – how should they be framed to get us closer to a clear picture? That’s the conversation we need to aspire to.”

Recently, government research has become more open, with the Treasury Green Book – the guide to policymaking in government – now including transparency of the evidence behind policy proposals as a requirement for all government departments.

But more is needed. Brown says there are benefits to research synthesis – broadly the process of combining the results of numerous research studies on the same subject to make the findings more widely applicable.

The RSS has been working on improving civil servants’ and MPs’ knowledge of data science and statistics, with the pandemic having provided a once-in-a-lifetime opportunity to build stronger relationships between statisticians and researchers and policymakers.

In its Covid-19 Lessons learned memo, the RSS recommended building up the statistical skills of decision-makers through training for politicians and civil servants, with a course due to launch in 2023 that will help non-specialist civil servants improve their statistical skills.

Westlake says that work is still required to build on the hard-fought gains evidence-based policy has made over the past two years. “Knowledge of statistics really varies within government and between governments,” he adds. “As a result of the pandemic, when statistics and data have really taken centre stage, more politicians now see the value of them, but whether they have the knowledge is a different issue. We also need to see more people with statistical skills in leadership positions. We can no longer see government statisticians as in a back-office role – they need to be in the room when crucial policy decisions are made and have their voices heard.”

Making improvements

As well as addressing the transparency gap, Brown would like to see a greater focus on ‘testable questions’ in policy – for example, Birmingham hosting the Commonwealth Games could be deemed a good or bad idea depending on the person but is not a testable statement, as it is based on a person’s values, whereas you can test and measure its impact on areas such as child obesity or involvement in sports. “People get hung up around a war of who should be in power – the scientists or the politicians?” she adds. “The issue is whether we have those testable questions framed and answered effectively, and if I can see the answers they come up with.”

Dorobantu says that artificial intelligence (AI) could help boost government effectiveness. In 2015, only seven government publications mentioned AI, she notes, but this number is now into the hundreds, including a national strategy for AI. Since 2018, the Alan Turing Institute has been working alongside government through its programme to improve policymaking with data science and AI.

Four years on, the institute has more than 50 researchers working on building data science models and AI systems to help policymakers take advantage of the latest data-intensive technologies.

“Despite the hype around AI, there is a limited understanding of how the latest generation of data-intensive technologies might help the public sector,” Dorobantu adds.

“The past decades have seen tremendous investments in AI, but they have been mostly driven by the private sector and targeted at solving private sector problems.

“The public sector is home to some of humanity’s most consequential questions – from protecting the environment to nurturing children, advancing healthcare, and tackling systemic biases and injustices.

“AI has enormous potential in helping policymakers tackle some of these questions and improve the lives of millions of people. But for that potential to turn into reality, policymakers and researchers must turn their attention to building AI technologies.”

Arguably the biggest issue, says Brown, is the lack of resources to help MPs and parliament properly scrutinise the policy decisions taken and evidence provided by government.

“Parliament is too weak in the sense that it does not have the resources to scrutinise the evidence government is using,” she explains.
“Between elections, the only true scrutiny and accountability about whether new and emerging evidence being used is parliament asking questions and scrutinising legislation. You have to ask whether our parliament has 21st century tools to do that. When you have government departments using algorithmic tools, does parliament have the capability?”

MPs are often generalists with a small, often non-specialised staff tasked with researching a broad range of subjects. The daily life of an MP can shift from scrutinising policy on food standards, followed by health policy, foreign relations and the government’s economic performance.
Following Brexit, powers that were previously exercised by the European Union have returned to parliament, and so this lack of capacity is set to worsen.

Ultimately, there will probably always be tension between policymakers and academics and researchers.

“Politicians often have to make decisions in the absence of complete evidence,” says Westlake. “Ultimately, we live in a democracy, not a technocracy, so conflicts between evidence and politics are not going to go away. A further challenge is that many people within government lack the technical and quantitative skills to interpret evidence in an effective and timely way.

“We don’t need to be defeatist about this though: clear communication of statistics to the public and to decision-makers makes it easier to make evidence-based decisions that have public buy-in.”

This article was first published in the October 2022 issue of Impact.