Education Endowment Foundation:Fresh Start – trial

Fresh Start – trial

Harlow Education Consortium, Ercall Wood Technol
Implementation costThe cost estimates in the Toolkits are based on the average cost of delivering the intervention.
Evidence strengthThis rating provides an overall estimate of the robustness of the evidence, to help support professional decision-making in schools.
Impact (months)The impact measure shows the number of additional months of progress made, on average, by children and young people who received the intervention, compared to similar children and young people who did not.
+3
months
Project info

Independent Evaluator

Durham University logo
Durham University
A popular phonics programme for pupils in Years 5 – 8
Schools: 10 Grant: £350,000
Key Stage: 2, 3 Duration: 2 year(s) Type of Trial: Efficacy level evidence
Completed January 2014

The project

This page covers the first (efficacy) trial of Fresh Start, which tested whether it could work in schools under best possible conditions. To read about the second (effectiveness) trial – testing a scalable model under everyday conditions in a large number of schools – click here.

Fresh Start (FS) is a catch-up literacy intervention for pupils at risk of falling behind their peers in early secondary schooling. It provides systematic and rigorous practice in phonics so that pupils are at an appropriate level to join the mainstream group after completion of the intervention. Pupils are assessed and then grouped according to their levels of reading ability. Teaching in these groups begins with recognition, practice and blending of sounds and graphemes, based on a set of module booklets.

This evaluation involved 433 pupils in Year 7, in ten secondary schools, who had failed to achieve a ​‘secure’ National Curriculum Level 4 (4b and above) in their primary KS2 results for English. Of these, 212 within all schools were randomly allocated to receive the intervention for 22 weeks during their first year at secondary school. This shorter time period was partly justified because the pupils are at secondary school. The other 221 pupils were randomly allocated to a control group in phase one, and received the intervention the following year. The intervention was organised by taking pupils out of regular English lessons for one hour, three times per week.

This was a different kind of study to the trials usually funded by the EEF. In 2013, three secondary school clusters, consisting of 10 schools in total, applied independently to EEF for funding to set up a programme for FS, and simultaneously evaluate its impact in their own schools. Each application was deemed too small in scale to run a successful evaluation of the programme, but if the schools involved were to cooperate then the scale would be sufficient for an ​‘aggregated’ efficacy trial. Efficacy trials seek to test evaluations in the best possible conditions to see if they hold promise.

Each school ran a small trial of FS in isolation, and made all of the relevant evaluation decisions such as randomly allocating pupils to the control or intervention group. Researchers at Durham University were assigned as independent evaluators for this trial. Their roles were to advise the school leads on the process of conducting research, randomisation and testing, and to aggregate the eventual results from all schools. There was no direct involvement from FS in the initial funding proposal to the EEF, but the developers were used by schools to provide training.

Outcome/​Group
ImpactThe size of the difference between pupils in this trial and other pupils
SecurityHow confident are we in this result?
Intervention vs control (all pupil)
+3
Months' progress
Intervention vs control (FSM)
+3
Months' progress
N/A

Key Conclusions

Key Conclusions
  1. Fresh Start shows considerable promise as an effective catch-up intervention for low-attaining readers at the transition phase from primary to secondary school.
  2. Pupils must be grouped according to the reading scores obtained from a Fresh Start screening test. Each group has a homogeneous level in terms of pupil reading abilities
  3. Continuous feedback to teachers and support from trainers assists successful implementation.
  4. It is feasible for schools to conduct evaluations of their own planned interventions, under favourable circumstances, and with some advice and oversight from independent expert evaluators.

What is the impact?

The FS pupils in the intervention group made more progress in literacy than the control group after 22 weeks. The overall effect size was +0.24 in terms of the gain between the scores in the test before the intervention (the pre-test) and the test after the intervention (the post-test). This is equivalent to approximately 3 months of additional progress in reading age. Progress was assessed as the gain in scores between a pre-test (GL Assessment’s New Group Reading Test A) and a post-test (New Group Reading Test B). The impact evaluation also suggests positive progress results for FSM-eligible pupils, and for all pupil sub-groups regardless of age, sex, first language, ethnicity, or special education need. However, as the numbers of pupils in the sub-groups are smaller, these findings are less secure than the overall finding.

The intervention was observed by the evaluation team as being generally well conducted, and attractive to teachers and pupils. The intervention was well received by the schools. All participating schools have plans to continue Fresh Start for future cohorts.

How secure is the finding?

The existing literature suggests considerable promise from using phonics in general, and Fresh Start more specifically. However, the impact of FS has not been demonstrated in the UK, and so this efficacy trial is the first of its kind in England.

The primary outcome measure was reading comprehension as measured by an independent test. The results were analysed with gains in overall reading scores as well as with standardised age scores, and both gave similar effect sizes. A pre-test indicated there was considerable imbalance between the groups (with the treatment group having lower prior attainment). Attrition was very low ¬ only 2%. Only 14 pupils have missing test scores. The control group activity was ​‘business as usual’ and these pupils were not given access to the programme. The intervention was regularly administered, the process was closely monitored and there was no indication of pupils or teachers becoming demoralised after the randomisation occurred. Therefore, the evidence achieved can be interpreted as having a security rating of three padlocks.

Participation in the Fresh Start intervention was at the instigation of the school leaders and cluster heads. They were already enthusiastic about the programme. This may limit the generalisability of results to other schools who may be less enthusiastic about its potential and therefore not deliver the intervention as effectively. Group No. of pupils Effect size Estimated months’ progress Evidence strength Cost rating Intervention vs control (all pupil) 419 (10 schools) +0.24 +3 months ££ Intervention vs control (FSM) 104 +0.24 +3 months ££ *For more information about evidence ratings, see Appendix 1 in the main evaluation report. Evidence ratings are not provided for sub-group analyses, which will always be less secure than overall findings. **For more information about cost ratings, see Appendix 2 in the main evaluation report.

The process demonstrated that schools are capable of running trials, but only with appropriate expert help. The main area of concern was in allowing schools to do the randomisation. Following the randomisation process, there was a noted imbalance in the pupils’ pre-test scores in the control and intervention groups. A number of statistical techniques were applied to control for the imbalance in the two groups in the final analysis. The results suggest that the outcomes of the trials are unlikely to be caused by bias or dropout of the 14 cases.

How much does it cost?

The direct cost to schools is for the Fresh Start training and material resources required in the intervention. For each pupil the cost of the material is £95.90. The cost of staffing to assist teachers depends largely on each school’s initial position. So, for example, a school with four relevant teachers to be trained and 20 pupils per year group, would have to pay £2,316 or around £116 per pupil.

Downloads