EEF statement: re-publication of the evaluation of school breakfast clubs
Today the Endowment Foundation (EEF) is publishing a revised report of its evaluation of school breakfast clubs, following the notification of an error by the Institute for Fiscal Studies (IFS) in its evaluation. A statement from IFS on this error can be found here.
Many schools in England offer breakfast provision for their pupils, and there are many reasons for doing so. Most importantly, breakfast provision helps ensure that no child has to learn when they’re hungry.
In November 2016, the Education Endowment Foundation (EEF) published an independent evaluation of school breakfast clubs, which found that a model of free, universal, before-school breakfast clubs was a cost-effective way of raising pupil attainment in primary schools.
The EEF had appointed the IFS to undertake this evaluation as a randomised controlled trial (RCT). The evaluators’ results were rated by the EEF and the independent peer reviewers as ‘high security’, reflecting the quality, size and design of the study.
In 2019, the IFS reported an error in its evaluation process to the EEF. The mistake resulted from human error in the randomisation code used to allocate schools to intervention and control groups. Because of the error, the evaluation design can no longer be classified as an RCT and has been reclassified as a control group study.
There are two consequences for the results of the study:
- For Key Stage 1 outcomes, the impact rating as originally reported, of +2 months’ additional progress, is unchanged. The security of the findings has been reduced from 'high security' to 'moderate-to-low security'.
- For Key Stage 2 outcomes, the impact rating as originally reported, of +2 months, has been reduced to 0 months. The security of the findings has been reduced from 'high security' to 'moderate-to-high security'.
The IFS carried out robustness checks on the KS1 outcome result and found that the impact remained, independent of analysis choice. As in the original study, the revised report finds that pupil behaviour, as measured by a teacher survey, improved in breakfast club schools. Attendance at school also improved for children in breakfast club schools, resulting in about 26 fewer half-days of absence per year for a class of 30.
Today the EEF is publishing a revised evaluation report, reflecting the changes to the study. The EEF publishes all results, regardless of outcomes, to prevent publication bias.
The updated report has been peer reviewed by two independent reviewers, Professor John Jerrim and Dr Alex Sutherland. Their statement is available here.
All EEF reports are independently peer-reviewed and the analysis replicated by the EEF’s overarching evaluators at Durham University. In this instance, the EEF, peer-reviewers and Durham colleagues were unable to identify the error in the original report because the randomisation code that was included in the report did not match the code used by the IFS in the randomisation.
The revised report still finds a positive impact on KS1 attainment in maths, reading and writing. However, due to the study’s revised security rating, the programme no longer meets the EEF’s Promising Projects criteria, which requires at least 'moderate-to-high security' (3/5 padlock rating). The EEF remains interested in school breakfast clubs and will look for ways to add to the evidence base about them in the future.
This study is still the largest and most robust study of breakfasts in the UK and the findings remain important. Schools considering implementing breakfast clubs should also consider the multiple positive impacts of the approach, including improved pupil behaviour and improved attendance.
As always, the EEF recommends that schools carefully consider how to introduce any new programme and monitor its delivery and outcomes to check it is achieving the desired impact.
Sir Kevan Collins, CEO of the Education Endowment Foundation, said:
Publishing this updated report from the Institute for Fiscal Studies today provides an opportunity to emphasise the principle of transparency in the EEF’s approach.
My message to schools is this. The model of pre-school breakfast clubs the EEF trialled – free and universal – appears to have clear benefits to pupils. In addition to the positive attainment impact found for pupils in Year 2, the independent evaluation also found both improved attendance and behaviour in schools. Most importantly, breakfast clubs help ensure that no child has to learn when they’re hungry
That the revised report no longer concludes Year 6 pupils’ attainment improved does, however, highlight that closing the gap is complex. As always, we recommend that schools consider carefully how they introduce any new programme and carefully evaluate for themselves that it is achieving the intended impact.