Butterfly Phonics aims to improve the reading of struggling pupils through phonics instruction and a formal teaching style where pupils sit at desks in rows facing the teacher. The teacher directs questions to the pupils throughout the lesson in order to check their understanding. It is based on a course book created by Irina Tyk, and was delivered in this evaluation by Real Action, a charity based in London.
Real Action staff recruited and trained practitioners to deliver the intervention. These practitioners worked with trained teaching assistants to teach classes of six to eight pupils, although some groups were larger. Pupils were eligible for participation in the trial if they did not reach level 4 in their Key Stage 2 SATs or their reading skills were at least a year behind their chronological age. In most schools, lessons were taught over a period of ten to twelve weeks, typically with two one-hour lessons each week. One school delivered the intervention over just four weeks, and the implications of this variation are discussed in the main body of the report.
The evaluation was set up as a randomised controlled trial, which compared the progress of pupils who received Butterfly Phonics to a “business-as-usual” control group. It did not test the delivery model at scale, and should therefore be considered an efficacy trial. The study was funded by the Education Endowment Foundation as one of 23 projects focused on literacy catch-up at the primary-secondary transition. It is one of seven literacy catch-up projects with a focus on phonics.
Testing the impact of an out-of-school-hours phonics programme.
Staff deployment & development
Language and literacy
The following conclusions summarise the project outcome
This evaluation provided evidence of promise; there was a positive, statistically significant effect on the primary outcome measure of reading comprehension. However, this effect size was lower than the minimum detectable effect size of the trial, so we cannot confidently conclude that the effect was due to the intervention and did not occur by chance.
The secondary outcome measures indicated positive impacts on children’s literacy skills, but these were not statistically significant.
This intervention is recommended to take place during the school day, when it is easier to secure sustained co-operation and support from school staff. Where that support was present, the intervention was able to progress more satisfactorily than in schools where it was lacking.
Schools should ensure that people delivering the intervention receive training in the Butterfly method so that it is implemented as intended.
Further research could investigate the intervention’s impact on early readers. Its emphasis on larger word units and comprehension skills might enable a more rapid progression in early reading than a pure phonics course.
What is the impact?
The evaluation found that, on average, the reading comprehension skills of pupils who received the intervention improved at a faster rate than those in the control group. This improvement is equivalent to the pupils who received the intervention making an additional five months’ progress over the course of the school year. This estimate is statistically significant, but the observed effect size is lower than the minimum detectable effect size that was specified at the beginning of the study. This means that, although this evaluation provides evidence of promise, we are unable to confidently conclude that the observed effect is real and did not occur by chance.
The evaluation also considered the impact of the intervention on two secondary outcome measures of literacy skills, but did not find statistically significant impacts.
|GROUP||NO. OF PUPILS||EFFECT SIZE (95% CONFIDENCE INTERVAL)||ESTIMATED MONTHS' PROGRESS||EVIDENCE STRENGTH||COST|
|Intervention vs control (all pupils)||310||0.43 (0.03 to 0.84)||+5 months|
|Intervention vs control (FSM)||140||0.16 (-0.18 to 0.49)||+2 months||N/A|
How secure is the finding?
The primary analysis in this evaluation is judged to be of weak security and was awarded a security rating of 0 padlocks. The main cause of the low security rating awarded to this evaluation is that the differences between schools’ post-test results were greater than envisaged at the beginning of the trial. This meant that the trial was not large enough to confidently detect an effect as small as the one that was ultimately observed. We cannot therefore confidently conclude that the observed effect is real and did not occur by chance.
A further limitation of the study is that the test administrators reported that the post-tests in two schools were disrupted by the poor behaviour of the pupils involved. It was judged reasonable for the pupils to re-sit the test in one of these schools and the primary analysis, which was performed on an ‘intention to treat’ basis, used this re-sit data. It was not judged reasonable for the pupils in the other school to re-sit the post-test and therefore the data from the disrupted post-test session was included in the primary analysis. A report about the compromised administration conditions of the post-test in this school is included in Appendix 2. These disrupted post-tests and their inclusion in the final analysis should be considered when interpreting the security of the findings.
It is possible that the positive impact was caused by a ‘confounding’ factor. Most of the participating schools scheduled the intervention to take place outside English lessons and some of the effect was possibly due to the pupils in the intervention group receiving additional English teaching, not the nature of the Butterfly Phonics intervention itself. Also, pupils received the intervention in small groups, while pupils in the control group continued with normal classroom teaching.
The schools involved were located within a small geographical area of London, and care should be taken when applying these findings to schools in different contexts.
To view the project's evaluation protocol click here.
How much does it cost?
The cost of Butterfly Phonics as it was delivered in this evaluation is estimated at £108.50 per pupil. This estimate is based on the assumption that there are eight pupils in each Butterfly Phonics class, and includes the salaries of the Butterfly Phonics teaching staff, training costs and the cost of course books (about £10 each). The cost of the Butterfly teaching staff for a one hour class was £35-40 and the specialist training from Irina Tyk cost £600 for half a day.