EEF Blog: Making use of the evidence on educational programmes
Stephen Fraser, Deputy Chief Executive of the Education Endowment Foundation, highlights the importance of using evidence to guide decision making around the implementation of educational programmes.
The sheer number of programmes available in the English education system is astonishing. The EEF has funded the evaluation of over 140 different programmes to date, and every week schools are inundated with information about the latest innovations and initiatives. As schools plan to support long term COVID-recovery, this is only likely to increase.
On the one hand, this is a sign of system strength – it is full of people with ideas wanting to improve pupils’ learning. On the other, it presents a challenge. Schools cannot do everything, and they face real resource constraints. Indeed, in EEF trials, one of the most frequent reasons reported for programmes not being implemented successfully is lack of time.
With evidence suggesting that school closures will have widened the attainment gap, alongside providing high quality teaching and making use of the National Tutoring Programme schools will likely look for programmes to mitigate the impact on disadvantaged pupils.
Educational research isn’t just useful for helping schools decide which programmes to start. It is also crucial in thinking through which programmes to stop, in order to release school time and money for programmes and approaches with greater promise.
Only 1 in 5 EEF-funded trials finds a meaningful positive impact. In other words, the vast majority of programmes are no better than what schools are already doing. At first sight, that might seem disappointing. However, not only does it highlight the effective job our schools are doing, it also allows us to focus our resources on the few programmes which do appear to make a difference over-and-above current practice. Approaches our trials have found to be no more effective than usual practice include teacher observation, pupil handheld devices and financial incentives. Trials have even shown that some very popular programmes may negatively impact pupils’ academic outcomes.
A case in point is the independent evaluation of Achievement for All (AfA), a programme that has been delivered in over 4,000 English primary schools. This programme provides schools with an AfA coach, who works with a designated member of staff in school to devise a bespoke action plan which aims to improve the outcomes of children. This plan is then used to inform monthly coaching and training sessions delivered by the AfA coach, while schools are also provided with online learning resources.
The evaluation, conducted by the University of Manchester and published last year, found that pupils in participating Achievement for All schools, who received the programme in year 5 and 6 from 2016-2018, made the equivalent of 2 months’ less progress in reading and maths compared to children in control schools, where usual practice continued. The same negative impact was found for children eligible for free school meals.
This week the EEF has published the follow up study for cohort 2 children, who began this study in year 4. After 2 years of AfA, cohort 2 children in AfA schools also made 2 months’ less progress in reading and maths compared to children in control schools.
Contrast this with the independent evaluation the EEF published in May 2020 of a large-scale trial of the Nuffield Early Language Intervention (NELI). NELI is a programme that aims to improve the language skills of reception-age pupils. Developed by academics at Oxford University, in this trial training was delivered by Elklan with resources from Oxford University Press. It involves scripted individual and small-group language teaching sessions delivered by trained teaching assistants for 20 weeks.
Our report found that children who received the NELI programme made the equivalent of 3 additional months’ progress in language skills, on average, compared to similar children who did not receive NELI. The Department for Education has worked with the EEF, Nuffield Foundation and partners to scale it up to over 6,500 schools this year, and has subsequently announced that NELI will play a key role in the COVID-recovery plan, providing an additional £8m in funding to support reception classes to receive NELI in the coming year.
Every EEF-funded trial – whether positive, negative or no-impact – has been delivered by a team that is open and willing to find out about the impact of their programme. They take the risk of doing so to learn and improve their programme and out of a commitment to providing schools with high-quality information. This merits great credit, regardless of the evaluation findings – especially as there are many programme developers who avoid the challenge of rigorous or independent evaluation and so deprive schools of the evidence they need to inform their decisions.
It is crucial that schools use this evidence, avoiding programmes which may negatively impact pupil outcomes, and identifying and stopping programmes that may not be having an impact. Even approaches that are not having a negative impact have a cost. It might be in teacher workload or not having time to implement a more positive approach; or it might simply be the financial cost to the school.
When planning how to deliver long term COVID-recovery, and as the offers of programmes or training come flooding in, schools should all be asking “where is the evidence?”