Mind the Gap

{editor} {/the-completed-project}

Key Conclusions

The following conclusions summarise the project outcome

  1. The headline findings provide no evidence of a statistically significant impact of Mind the Gap on attainment. There is not sufficient evidence to conclude that any observed effect was caused by the programme rather than occurring by chance.

  2. The estimate of the programme’s impact on pupils’ metacognition was positive and statistically significant. This improvement in metacognition may in time lead to an impact on academic attainment.

  3. Participating families and staff felt the intervention enhanced home–school relationships and strengthened the learning relationship between children and parents.

  4. To increase the proportion of families who sign up to the animation course, it is important that schools clearly communicate its aims and promote the potential benefits of participation.

  5. The difficulty in recruiting schools suggests that introducing Mind the Gap more generally may be difficult where schools are not committed to parental engagement or where they have difficulty delivering activities out of hours.

What is the impact?

The impact analysis considered the effect of Mind the Gap as a whole, as well as the separate effects of the teacher training component and eligibility for the parental engagement intervention. InCAS tests of reading, general maths and mental arithmetic were administered six months after the intervention. The intervention had no statistically significant effect on the primary outcome, a combined reading and numeracy score, so we do not have sufficient evidence to conclude that any observed effect was caused by the programme rather than occurring by chance. This was also true for the subgroup of pupils eligible for free school meals.

A number of secondary outcomes were also considered. These included reading, general maths and mental arithmetic scores as well as a measure of how children felt about the relationship with their parents (constructed using their responses to questions taken from the Self-Description Questionnaire). These estimates were consistent in showing no statistically significant effect. However, measures of metacognition (elicited from data collected using Pupil Views Templates) provided evidence that eligibility for the animation course increased pupils’ ‘productive thinking’ and ‘metacognitive skilfulness’.

GroupNo. of pupilsEffect size (95 confidence intervals)Estimated months’ progressEvidence strengthCost of approach
Mind the Gap overall effect (all pupils) 492 -0.141 (-0.697, 0.414)-2
Teacher training alone2780.009 (-1.119, 1.137)
Parental engagement offer alone186 -0.252 (-0.576, 0.071)-3
Mind the Gap overall effect (FSM pupils)123-0.265 (-1.332, 0.801) -4

How secure is the finding?

Overall the evaluation findings are judged to be of low security, largely due to the high degree of attrition.

Mind the gap was evaluated using a randomised controlled trial. Participating schools were randomised to a ‘business-as-usual’ control group or an intervention group, which received the teacher training element. Within the intervention group, Year 4 classes were then randomised into a further two groups: one that were eligible to participate in the animation course and a group that were not. This design allowed three effects to be estimated:

  • the overall effect of Mind the Gap
  • the effect of teacher training alone
  • the effect of eligibility for the animation course.

It was eligibility for participation in the animation course that was randomised. Participation in the course was very low: 72% of pupils eligible for the animation course did not attend any sessions. The results must therefore be regarded as estimating the effect of eligibility for the animation course (alone or in combination with teacher training, as appropriate) rather than participation in it.

In practice, implementation difficulties raised concerns about the security of the results. The main problem was the high level of dropout and the fact that this was concentrated among control schools. There was no means of including such schools in the analysis since they provided no outcome data. A comparison of baseline characteristics suggests that there may be some degree of imbalance across experimental arms in the sample used for analysis, but these comparisons themselves are made difficult by the high incidence of missing baseline data. Furthermore, among schools that did not drop out, there were missing outcome data, compounding the problem caused by dropout.

Another limitation is that, in practice, the treatment status of classes within schools did not agree in all cases with the randomised status. Such non-compliance is a common feature of randomised controlled trials and generally one can still adhere to the intent to treat (ITT) principle. However, in several cases the randomised status became meaningless in practice, as the teachers who were randomised (in most cases, classes were identified by their teacher) had left the school. The final analysis used de facto treatment status. This is a deviation from ITT and further reduces the extent to which the impact estimates for the components of Mind the Gap can be viewed as experimental.

Existing evidence suggests that metacognition and parental engagement interventions have a high average impact on attainment. However, these interventions can take many forms and the intervention considered in this project has no directly comparable precedent.

How much does it cost?

For schools that already have the required IT hardware, delivery of the animation course for up to 15 families costs in the region of £1,950. This translates into a per child cost of roughly £130. The cost of teacher training was typically an additional £195 per delegate.