Education Endowment Foundation:EEF Blog: How to use evidence when you’re time-poor

How best strike the balance between rigour and urgency when putting evidence at the heart of your work, whether you’re a policy-maker or a school leader? That’s the question our deputy chief executive Stephen Fraser explores here…

For the evidence-informed policymaker, time can seem the arch-villain. Too often the world conspires to rob us of the time we need to define the policy problem, build buy-in with citizens and stakeholders, gather the evidence, test the counterfactual, monitor the impact, replicate the findings — all that might be required to build the confidence to advise and act.

But perhaps the problem isn’t time but the linear model described above. After all, it’s unlikely that governments will often, if ever, have the time to plan to answer questions they’ve not yet anticipated asking

So, how can the responsive and responsible public servant best strike the balance between rigour and urgency? Is there a time-turner for evidence?

Step 1: Be humble and learn from others

Humility first

Perhaps the first step is to act with humility and be willing to learn from others. Just as the advent of evidence-based medicine over the past half-century has given rise to resources such as the Cochrane Library, so too has a diverse range of fields, including international development and social science begun to benefit from investment in accessible evidence repositories that allow us to start with what we know.

Jack Martin from the Early Intervention Foundation (EIF) has succinctly outlined the range of evidence repositories available through the UK’s What Works Network, emphasising the need to factor questions of evidence quality, relevance and cost-effectiveness into the decision-making process.

These resources summarise vast quantities of research findings and help us determine whether a programme or approach is likely to be a good bet for solving our specific policy or practice challenge. They simultaneously help us to avoid the risk of overreliance on a single study, allow us to understand the extent to which the feasibility of an approach has been tested in real-world conditions, and help us bypass the time usually required to generate and synthesise complex research findings.

The Education Endowment Foundation’s Teaching and Learning Toolkit, for instance, can give policymakers a ready-made answer to the question of whether pupils who do not reach a given standard of learning at the end of each year should be required to repeat that year (there is moderately good evidence it’s likely to lead to negative impact on learning and be very expensive to implement).

Ask ​‘how’ rather than ​‘what’

But of course, using these repositories offers no guarantee of success. They describe what has worked, rather than what will work. And so the question quickly becomes one of generalisability. If the existing evidence offers promise without guarantee, how can we maximise the chance of success? How do we remain faithful to adopting the evidence, whilst intelligently adapting it to fit our context?

Mary Ann Bates and Rachel Glennerster offer a useful four-step framework for determining the likelihood that the existing research findings about a programme or approach can be generalised to a new context. They ask:

1. What is the disaggregated theory behind the programme? 
2. Do the local conditions hold for that theory to apply? 
3. How strong is the evidence for the required general behavioural change? 
4. What is the evidence that the implementation process can be carried out well?

These questions take us beyond ​“what works?”, and even beyond ​“what works, for whom, and in what circumstances?” and move us to questions of how an approach seems to work or not work, connecting context, mechanism and outcome

It is not the act of repeating a year that is inherently harmful to students’ learning. Indeed, some studies have shown benefits. But when we ask how repeating a year typically leads to negative impacts, we begin to see that repeating a year can affect pupils’ self-esteem, can risk social isolation and disengagement, risks repeated exposure to the same curriculum and a lowering of expectations that the pace of learning progress will match that of peers

Equipped with an understanding not only of what works but how, we can begin to engage with the behavioural, cognitive or physical mechanisms through which learning is affected, and begin to ask whether it applies in our circumstances.

Come prepared

It’s a rare Minister who’s happy, or able, to wait more than a few days for a policy response; as the Covid-19 crisis has underscored with tragic consequences, a few weeks are sometimes the best we can hope for and seldom will we get a second chance to answer the same question. And so the imperative is to be prepared.

Build expectations for evidence into the fabric of public decision making – into recruitment, induction and ongoing professional learning for policy teams, into templates for cabinet briefs and budget decisions, into the public communication of policy rationale. Cheat time by investing in long-term evidence infrastructure (funding the equivalent of a What Works approach is a great place to start), and ultimately, change the cultural expectations for how evidence is applied

It’s one thing to convince a Minister of the need for evidence; quite another if the Minister expects evidence-informed advice from the outset. The more public sector leaders can hardwire the use of evidence into culture and practice, the greater our chances to meet the demands of ever-complex, ever-changing and ever-urgent public policy questions

And finally, we can learn from the example of Pedro Cunho, the former Portuguese Deputy Director General of Education, who literally carried the evidence with him – a printed copy of the EEF’s Teaching and Learning Toolkit, ready to table the moment the discussion called for evidence. Pedro kept his evidence time-turner in his pocket.