1. This evaluation cannot conclude with certainty what impact the LIT programme had on reading ability for those pupils who received the intervention.
2. This evaluation did not find evidence of a significant impact on reading ability at the year group level (comparing all Year 7 pupils in treatment schools with similar Year 7 pupils in control schools).
3. Teachers felt that the programme facilitated ​‘healthy debate’ within the classroom, increased confidence in pupils who struggled with core literacy skills, and promoted independent learning. However, they did not feel it worked well with children with underlying cognitive issues requiring intensive vocabulary support.
4. Feedback from those who used the programme suggested groups of 5 – 6 children led by qualified teachers or teaching assistants with experience of delivering literacy interventions worked best.

The primary objective of the independent evaluation was to test whether the LIT programme had a significant impact on the literacy skills of the approximately 15% of Year 7 children who were eligible to take part (those with the poorest literacy skills). Unfortunately, it was not possible to do this because the characteristics of pupils in treatment and control schools were too different to yield an unbiased estimate of the impact of the programme. However, it was possible to compare all Year 7 pupils in treatment schools with similar Year 7 pupils in control schools, regardless of whether they received the LIT programme. This captures the effect of the LIT programme on the whole year group.

These results suggest that the reading scores of Year 7 pupils in schools running the LIT programme rose 0.09 standard deviations more over the course of the year than those of similar Year 7 pupils in control schools. This effect size of 0.09 is equivalent to 1 month’s additional progress in reading. However, statistical tests indicate that this estimate is not ​‘significant’, i.e., we cannot be sure that the difference in scores is not due to chance. It is important to note that this is the average effect across all pupils in Year 7, i.e., across the 15% of pupils who received the LIT programme as well as the 85% of pupils who did not. It is possible that the effect on those who received it was much higher.

Recommendations for further study

In addition to a further study to estimate the impact of the LIT programme on pupils who received it, it would be useful to explore in more detail which elements of the programme are vital to its success. The LIT programme was delivered in a variety of ways across schools in the evaluation, with variation in programme length, number of pupils per group, qualifications and experience of staff delivering the programme and whether they were specialists in English, and whether the LIT programme was delivered in addition to or instead of English or literacy lessons. Unfortunately it was not possible to separately identify the contributions of these different elements to the overall programme impact.

What did schools think of the programme?

The reciprocal teaching approach on which the LIT programme is based was seen as having a number of strengths: it encourages teamwork and debate, gives lower attaining pupils confidence, and is fun. But it was not felt to be suitable for the lowest ability pupils who required intensive support. Teachers felt that the positive benefits of using the LIT programme arose from the reciprocal teaching approach, the fact that teaching occurred in small specialised groups, the engaging programme content and quality of resources, and the assessment and pupil feedback element. Several schools reported that they would continue to use the programme at the end of the intervention.

The intention was to estimate the impact of the LIT programme by comparing the subset of pupils in treatment and control schools who were eligible to receive the programme with similar pupils in control schools who were not. However, the randomization was undertaken at school level and hence does not guarantee that the characteristics of a subset of pupils in treatment and control schools are similar. In addition, a small number of schools dropped out of the programme, and not all pupils sat the ART post-test, which meant that the original randomization was compromised. Consequently, we had to try to ​‘balance’ the characteristics of pupils in treatment and control schools, to ensure we were comparing like with like.

This was a challenging exercise, as the characteristics of pupils in treatment and control schools were very different. We identified pupils in treatment and control schools who were as similar as possible using a number of characteristics we deemed important for pupil progress such as prior attainment, gender, ethnicity, eligibility for free school meals, school type and composition.

Unfortunately it was not possible to find a group of pupils in control schools who ​‘looked’ similar enough to pupils who were eligible to receive the LIT intervention across all of the dimensions that we thought mattered for progress. This meant that the original intention of estimating the impact of the LIT programme on the approximately 15% of Year 7 children in treatment schools who were eligible to receive it was not possible. Instead, the final analysis compared the reading scores of the 2,889 Year 7 pupils in 19 treatment schools – 660 of whom were eligible to receive the LIT programme and 2,229 of whom were not – with the 1,524 Year 7 pupils across 15 control schools who provided useable data at the start and end of the trial.

If we instead thought that only prior attainment (and specifically Key Stage 2 scores) mattered for pupil progress then we could estimate the impact of the LIT programme on pupils who were eligible to receive it, by comparing those pupils in treatment and control schools with similar Key Stage 2 scores. Doing so resulted in an estimate of the impact of the LIT programme on those who are eligible to receive it (but not necessarily receiving it) of 0.13 standard deviations. However, this estimate is not statistically different from zero, and is highly likely to be biased by the fact that the pupils that we are comparing differ in other ways that matter for their progress. We would therefore strongly caution against regarding this as a robust estimate of the effectiveness of the LIT programme on those who received it.