Andrew Leigh

This is about drawing together the best evidence about what works, as the evidence commission report recommended: making better use of systematic reviews. We’ll be carrying out more randomized trials across government and ensuring that the big administrative datasets are used so that we don’t run a bespoke follow-up survey and can build a better feedback loop.

Previously, Andrew Leigh shared the following comment as part of the report launch.

Participating in the preparation of this report and in the discussions among commissioners has shifted my thinking about what I can do personally, what countries like my own need to do, and what I’d like to see multilateral organizations do.

On a personal level, section 4.8 – best evidence versus other things – is my favourite section. There is so much wise advice here about how to get more from the ‘other things’ that elected officials like me are regularly presented with, such as a single preprint, an expert with an opinion, a panel of experts offering recommendations, and a jurisdictional scan. A few years ago, I wrote a book on randomized trials. Now, after working on this report, I’m even more passionate about the need for randomized policy evaluations. One of the strengths of trials is that they’re easy to explain to citizens. They help us get around citizens’ concerns about ‘technocracy,’ in which regular people feel they’re being scammed through decision-making processes they don’t understand. Trust in government isn’t just about making the right decisions; it’s about making decisions that citizens perceive to be right.

Evaluation isn’t an elite issue. Evidence is for everyone. Our report offers suggestions to individuals, governments, and non-governmental organizations. If you’re an individual looking at the evidence on quitting smoking or losing weight, you should look at evidence syntheses, not single studies. If you’re a journalist writing about health, become a regular visitor to Cochrane, where you’ll find the distilled evidence on thousands of topics. For media outlets reporting on social policy, the Campbell Collaboration offers the same service. Our report proposes that governments become better at using evidence in their decisions, and build the evidence base through rigorous evaluations. International organizations should place greater reliance on evidence, and the World Bank should prepare a landmark report on best-practice use of evidence.

International organizations differ markedly in their use of evidence. Reports from the Intergovernmental Panel on Climate Change use a highly rigorous approach to selecting and grading evidence on global warming and its consequences. Other global bodies are less systematic in their use of evidence, frequently relying on single studies, citing only expert opinion when a substantial body of peer-reviewed literature exists, or extrapolating evidence across very different contexts. This is not a matter of international bodies wanting to misrepresent the science – these organizations are keen to improve, and outside experts can help them do so by assessing reports against each body’s published policy on how to use evidence. As described in section 5.5, ‘naming and shaming’ had a tremendously positive impact on the World Health Organization’s use of evidence, starting in 2007. Other parts of the UN system need to follow WHO’s lead.

Among philanthropic organizations, there is a growing recognition that high-quality evaluation can create a virtuous cycle: allowing ineffective programs to be wound down and effective programs to be scaled up. The fast-growing effective-altruism movement is demanding that charities produce rigorous evidence of their impact. For example, GiveWell.org estimates that two of its top-rated charities – the Against Malaria Foundation and the Malaria Consortium – each save a life for every additional US$4,500 that they spend on their programs. This is a powerful incentive for donors to support these charities. More evidence of direct impact from other charities could help to spur a philanthropic race to the top.