Today the EEF has published an evaluation of Achievement for All (AfA), a programme that has been delivered in over 4,000 English schools. We funded an independent evaluation of the programme, conducting a randomised controlled trial in 134 primary schools to test its impact on children in Key Stage 2.
In this trial, conducted between 2016 and 2018, AfA resulted in negative impacts on academic outcomes for pupils, who received the programme during five terms of Years 5 and 6 (ages 9 – 11). Children in the schools which received AfA made 2 months’ less progress in reading and maths, compared to similar children in the control group of schools which continued with their usual practice. The same negative impact was found for children eligible for free school meals. ‘Target children’ – those the intervention specifically aimed to support: the lowest attaining 20% or those considered ‘vulnerable to underachievement’ – also made 2 months’ less progress in reading, and 3 months’ less progress in maths. The co-primary outcome finding (whole-group reading, and ‘target children’ reading) had a very high security rating, a maximum 5 out of 5 on the EEF’s padlock scale.
Given the size of the effects and the consistency of negative findings, these results are noteworthy. Of particular importance is the impact that the programme had on target children, and children eligible for free school meals
As so many schools have delivered the programme previously, and 1,500 schools are continuing to deliver the programme at a cost of up to £5,950 per year, this result is likely to raise a number of questions. Here we offer answers to some of the key questions
Why is the EEF publishing this now?
The EEF recognises that the results of this report, and the implications of them, may not be the immediate priority for schools given that they are currently adapting to the Covid-19 crisis.
However, we also appreciate that schools will be planning for next year and will be considering how to mitigate the impact of school closures. With this in mind, we believe it is important not to delay publication of the results of the trial, and to ensure schools have access to information on a programme that they may be considering delivering.
Should schools still purchase and deliver Achievement for All?
The evaluation concluded that AfA did not improve KS2 pupils’ academic outcomes and had a detrimental effect on learning. On the basis of these highly secure findings, we recommend that schools think carefully before purchasing and using AfA. Schools currently delivering AfA should carefully monitor and evaluate whether it is having the intended impact.
Of course, this is the result from one trial, and AfA continue to maintain that their intervention supports schools. We also recognise that AfA is an organisation with substantial levels of experience, and they certainly intend to design programmes that benefit schools. The programme may also have undergone some changes since delivery in this trial.
On the basis of the evidence gathered, and from the findings of other EEF-funded evaluations, the EEF’s view is that there are other interventions which are more likely to improve pupils’ attainment outcomes. This could change if the AfA programme is substantially changed and if future robust evaluations demonstrate a positive impact.
On the basis of these highly secure findings, we recommend that schools think carefully before purchasing and using AfA
Should schools still purchase and deliver other Achievement for All programmes?
AfA deliver a variety of programmes, spanning early years to post-16. This trial tested the Achieving Schools programme in a primary context, collecting KS2 results. Elements of this programme may have changed since. When deciding whether to purchase and use a different AfA programme, schools and providers should consider how different the model is to that which has been tested here, and whether they would expect to see different outcomes from those seen in this trial
What was the impact of Achievement for All on children eligible for free school meals?
In this trial, AfA resulted in similarly negative impacts for FSM children, and ‘target children’. FSM children made 2 months’ less progress in reading and maths, while ‘target children’ made 2 months’ less progress in reading and 3 months’ less progress in maths than similar pupils in the control group of schools
How confident can I be in the EEF’s evaluation result?
The independent evaluation of this trial achieved a maximum 5 out of 5 ‘padlocks’ on the EEF’s security rating scale for the co-primary outcome measures of whole-group reading and reading for the ‘target group’ of pupils. This means we have very high confidence in its findings.
Previous evaluations of AfA produced contradictory findings and had substantial limitations which limited their security. This EEF trial is therefore the first robust evaluation of Achievement for All
What may have caused the negative impact in this trial of Achievement for All?
It is difficult to identify one reason why the programme failed to have its desired impact, and detrimentally impacted pupils’ attainment outcomes. Rather, it is likely to be the result of a variety of factors:
- One factor that may have contributed to the negative findings is the flexibility of the programme. While some case study schools commented that they found this beneficial, others reported that the flexibility made it difficult to grasp what the programme was. Several teachers within the case studies also could not identify how the programme could lead to a direct impact on pupil learning.
- It may have been the case that other school priorities began to compete with the priorities set by the Achievement for All action plan at the outset of the programme. School responses to Ofsted actions or other emerging issues may have overshadowed their action plans.
- In this context, and given these weaknesses, the negative findings may have, in part, been caused by the resources that schools expended on the programme, at the expense of other activities. Champions and teachers in case studies commented that certain aspects of the programme came with a number of time and logistical demands, which may have had financial consequences. The implementation of termly, ‘structured conversations’, which are up to an hour long, with the parents of all target children was particularly resource intensive. It may be the case that control schools used this time and resource more effectively.
While these are all plausible explanations, ultimately, we cannot be sure of exactly what caused the negative impact
Does this mean that children in Achievement for All schools went backwards?
No. Pupils in AfA schools may still have made progress. However, they made less progress than pupils in control schools
It is unlikely that the negative findings were a result of not delivering AfA properly.
Was the negative impact caused by schools not delivering Achievement for All properly?
There was variable implementation of the programme, and schools failed to deliver key elements of the intervention (as is often the case in EEF trials). For instance, few schools delivered the expected number of mandatory termly ‘structured conversations’ that teachers were supposed to deliver to the parents of target children. There were also very few instances of staff in case study schools using the online resources. However, it is unlikely that the negative findings were a result of not delivering AfA properly. 71% of schools did receive 20 or more coaching sessions (out of an intended 24), while the analysis of implementation suggested that higher levels of implementation were not associated with improved outcomes. More negative impacts may even have been associated with higher fidelity implementation of the ‘Leadership for Inclusion’ strand of the programme.
What was the impact on cohort 2 children (those who started the trial in year 4)?
The addendum report, published in April 2021, shows that AfA had broadly the same negative impact on a second cohort of pupils who began the trial in year 4, whose school received the programme for 6 terms (January 2017-December 2018), and whose results were taken 7 ½ terms after AfA delivery began. Cohort 2 children in AfA schools made 2 months’ less progress in reading on average, compared to children in schools that did not receive the programme. Cohort 2 target children also made two months’ less progress in reading, while cohort 2 children eligible for FSM made one month less progress in reading. All children, AfA target children, and FSM-eligible children in this second cohort, who were in the Achievement for All schools, made two months’ less progress in maths, on average, compared to equivalent children in schools which did not receive the programme. The evaluation of cohort 2 found that the programme did not improve pupils’ self-esteem, goals and aspirations, perceptions of how supportive their families were, perceptions of how supportive their schools were, or target children’s attendance. However, cohort 2 FSM-eligible children in Achievement for All schools were more likely to report higher levels of goals and aspirations.
Where can I learn more about the EEF’s trial of Achievement for All?
The full independent evaluation published by the EEF is available here