Education Endowment Foundation:EEF Blog: When does an EEF-funded project become promising?

EEF Blog: When does an EEF-funded project become promising?

Author
Emily Yeomans
Emily Yeomans
Director of Research
Blog •5 minutes •

Promising Projects are those EEF-funded projects – 16 to date – which have shown initial promise when trialled. This means they have demonstrated the potential to improve attainment for young people cost-effectively when independently and robustly evaluated. Here, our head of programme strategy Emily Yeomans explores how we designate Promising Projects and why we update the list. 

Building a secure body of evidence

The EEF has funded 190 projects to date, across all four stages of our programme pipeline – pilot, efficacy, effectiveness and scale-up: 

  • at the pilot stage we’re seeing if ideas are feasible; 
  • during efficacy trials we’re testing programmes under best possible’ conditions; 
  • effectiveness trials test scalable programmes under real world’ conditions; and finally
  • we provide scale-up support to expand well-evidenced programmes to more schools.

Each stage in the pipeline is important, adding to the evidence base and refining our understanding about whether, or how best, to take the programme to more schools and settings

We want to signpost clearly those programmes we’ve tested which have the most encouraging evidence of promise

As we build our understanding of what works’ to improve teaching and learning, we become more confident in what we are able to say about the effectiveness of different approaches. This can be seen most clearly in our Teaching and Learning Toolkit, an accessible summary of research based on large, cumulative bodies of evidence, which is updated as new findings become available.

As well as building this secure and wide-ranging evidence base we also want to keep teachers and senior leaders informed about the latest results from the independent evaluations of the projects we trial and to make their findings as useful as possible to schools

In particular, we want to signpost clearly those programmes we’ve tested which have the most encouraging evidence of promise. This is the purpose of our menu of Promising Projects

Helping schools to select programmes

The Promising Projects list is a menu of best bets’ for schools: all of them have demonstrated the potential to improve attainment for young people cost-effectively when independently and robustly evaluated.

This means that, for example, if a primary school is interested in investing in a programme to improve the quality of classroom talk we would highlight Dialogic Teaching as a good option to explore because our trial in 76 schools found consistent, positive effects in English, science and maths for all children in Year 5, equivalent to +2 months’ additional progress.

For a project to be included as an EEF Promising Project it will normally need to meet three key criteria in its independent evaluation: 

  1. To have secured at least one month’s additional progress for participating young people (ie, an effect size greater than 0.05 standard deviations);
  2. To have delivered this positive impact cost effectively (which we classify as meaning the project costs less than £80 per pupil for each additional month’s progress per pupil); and
  3. The EEF trial needs to have achieved an EEF security rating of at least 3 padlocks’ out of 5, meaning it has at least moderate security.

EEF Promising Projects normally need to meet three key criteria

That cost-effectiveness is one of these criteria means that programmes which have delivered relatively modest positive impact can be considered to be promising, as long as they are cheap and straightforward to implement.

For example, while the independent evaluation of Texting Parents estimated an impact of only +1 month’s addition progress in English and maths, it is estimated to cost just £6 per pupil per year. This, together with the ready availability of the technology to schools, mean that communicating with parents through text messaging is an approach the EEF is comfortable recommending.

Responding to new evidence

A few days ago, we published independent evaluation reports of three EEF-funded projects which had delivered encouraging results when we first tested their impact under best possible conditions (ie, as efficacy trials) and were therefore listed as Promising Projects. They were Catch Up® Numeracy, Catch Up® Literacy, and Grammar for Writing

All three progressed to the next stage of our pipeline, to be tested under everyday conditions in large-scale effectiveness trials. However, none repeated the earlier, positive impacts. As a result, we have removed them from our list of Promising Projects. While naturally disappointing, that is the nature of sticking firm to a rigorous, evidence-informed approach, ensuring teachers and senior leaders know the EEF will always report transparently and in full what the evidence says.

There are lots of possible explanations for differences in the results of two trials of the same programme. Some of these are explored in my previous blogs, Testing, testing, testing! How do we respond when trials produce different results?’ and Scaling education interventions – what are the challenges?’.

And, as I set out then, a no impact’ outcome in a second trial won’t automatically mean the EEF removes a programme from our Promising Projects list. Each case will be judged on its own merits, particularly if the evaluation report from the large-scale effectiveness trial suggest that changing the delivery model to make a programme scalable meant its implementation quality suffered. Such a finding does not necessarily invalidate the evidence from the first trial showing the programme’s potential to improve teaching and learning; it may instead indicate we need to work with the grantee to find a better way of ensuring the programme can be properly delivered at scale

Promising Projects are a useful place to start for schools interested in purchasing evidence-based programmes

Identifying new Promising Projects

Last week, we also added two new programmes to the Promising Projects list:

  • Embedding Formative Assessment is a professional development package which embeds formative assessment as a tool to improve teacher practice and ultimately improve student learning. It is particularly interesting because schools already prioritise formative assessment, but often report that it can be challenging to implement – this programme enables them to run training for their staff themselves, using detailed resource packs, making the approach both highly relevant to current practice, as well as scalable.

    The impact in the EEF trial in 140 schools was measured using Attainment 8 GCSE scores, an outcome that we know directly affects the life chances of pupils and which is hard to shift. The independent evaluator detected an impact equivalent to +2 months’ additional progress. This positive result, achieved at very low cost, means the programme is very cost-effective: just £1.20 per pupil for each additional month’s progress. The trial finding has a very high security rating: a maximum 5 out of 5 padlocks’ on the EEF rating scale

    As can be seen, Embedding Formative Assessment comfortably meets our criteria for inclusion as a Promising Project. For senior leaders looking to improve formative assessment in their secondary school we therefore think that this programme holds good promise

  • Tutor Trust provides tuition to primary and secondary schools by recruiting and training university students as paid tutors. There is good evidence that one-to-one and small-group tuition can have a positive impact on attainment. However, the cost can be high. We therefore funded this trial to test if Tutor Trust, which offers tuition at a competitive rate, could boost pupil outcomes, with our trial in 105 schools measuring Key Stage 2 maths scores, an outcome which correlates with GCSE results.

    The independent evaluator found that children who received tutoring from Tutor Trust made +3 months’ additional progress. This equates to a cost-effectiveness of approximately £60 per pupil for each additional month’s progress. The trial finding has a high security rating: 4 out of 5 padlocks’ on the EEF rating scale

    Therefore, Tutor Trust meets the criteria for inclusion as a Promising Project, and we are actively exploring with the grantee how to expand the areas it serves while maintaining the quality of tutoring. 

Conclusion

As can be seen, EEF Promising Projects is a dynamic list, updated as new evidence becomes available – with programmes removed and added depending on the findings of our independent evaluations

For schools interested in purchasing a programme to help raise attainment and close their gaps, our Promising Projects are a useful place to start, providing up-to-date recommendations based on the best available high-quality evidence