Education Endowment Foundation:SHINE on Manchester – trial

SHINE on Manchester – trial

SHINE Trust
Implementation costThe cost estimates in the Toolkits are based on the average cost of delivering the intervention.
Evidence strengthThis rating provides an overall estimate of the robustness of the evidence, to help support professional decision-making in schools.
Impact (months)The impact measure shows the number of additional months of progress made, on average, by children and young people who received the intervention, compared to similar children and young people who did not.
+1
months
Project info

Independent Evaluator

Durham University logo
Durham University

A Saturday school programme for primary school pupils who are falling behind.

Schools: 32 Grant: £510,175
Key Stage: 2 Duration: 4 year(s) 5 month(s) Type of Trial: Effectiveness level evidence
Completed July 2016

The project

The Hallé SHINE on Manchester (HSoM) programme is a Saturday school educational programme designed to increase the reading and maths attainment, as well as engagement with school, of underachieving and disadvantaged pupils at Key Stage 2.

Developed in collaboration between the SHINE Trust and Hallé Orchestra, the intervention provides additional school-based literacy and numeracy lessons, based on musical themes, as well as visits toHallé rehearsals, performances and other theme-based activities. Twenty-five Saturday sessions, each lasting five hours, were planned for the intervention over the course of an academic year, delivered by qualified teachers, teaching assistants, peer mentors, and professional musicians.

The evaluation consisted of two randomised controlled trials (RCTs) — a pilot trial and a main trial — and a process evaluation conducted with primary schools in the Manchester area between July 2012 and July 2015. The pilot trial involved 361 Year 5 and 6 pupils in 18 schools; the main trial involved 2,306 Year 4, 5 and 6 pupils in 38 schools. The second year involved both the pilot and main trial running concurrently. The third year involved main trial schools only.

Outcome/​Group
ImpactThe size of the difference between pupils in this trial and other pupils
SecurityHow confident are we in this result?
SHINE vs. control (Year 1)
+1
Months' progress
(Effect size 0.03)
SHINE vs. control (Year 2)
-2
Months' progress
(Effect size 0.1)
SHINE vs. control (Year 3)
+2
Months' progress
(Effect size 0.1)

Key Conclusions

The following conclusions summarise the project outcome
  1. There is no evidence that the Hallé Shine on Manchester programme had an impact on the reading attainment of children in the trial. This finding was consistent across all age groups, for pupils eligible for free school meals, and for all three years of the trials.
  2. There is no evidence that the Hallé Shine on Manchester programme had an impact on the attainment of children in mathematics, or attitudes to reading, maths, music, and school of the children in the trial.
  3. Attendance of eligible pupils was often low and considered as a barrier to successful implementation. Reasons for low attendance included pupils’ lack of availability to attend the Saturday sessions, variable parental engagement with the programme, and limited time at the beginning of the programme for schools to engage children and parents.
  4. The process evaluation revealed a positive picture of involvement and engagement for those pupils who attended the Saturday school activities. Evaluators observed good working relationships between the teachers and pupils, and positive and purposeful learning environments in lessons. All stakeholders felt pupils were making noticeable improvements in behaviour, confidence, and development of social skills.

What is the impact?

The findings demonstrated no evidence that the Hallé Shine on Manchester programme had an impact on the primary outcome of reading attainment of children in the trial. This outcome was consistent across all age groups, for both the pilot and main trial. This effect also applied to pupils eligible for free school meals.

No positive effects were observed for the secondary outcome measures of attainment in maths and attitudes to reading, maths, and music. Further exploratory analysis of Key Stage 2 results also showed no signs of positive impacts in reading, maths and writing for Year 5 and 6 pupils in the first two years of the trial.

Pupil recruitment and attendance was often low and a barrier to successful implementation. Reasons for low attendance included pupils’ lack of availability to attend the Saturday sessions, variable parental engagement with the programme, and limited time at the beginning of the programme for schools to engage children and parents. Nevertheless, further analysis that took attendance into account did not alter the overall finding.

In contrast, the process evaluation revealed a positive picture of involvement and engagement for those pupils who attended Saturday school activities. Good working relationships between the teachers and pupils were observed, with positive and purposeful learning environments demonstrated in lessons. Teachers, parents, and musicians all felt pupils were making noticeable improvements in behaviour, confidence, and the development of social skills. Although pupil engagement and participation in lessons were generally very good, more challenging feedback may have helped pupils’ learning.

These results are generally consistent with existing evaluations of out-of-school programmes, showing small or no effects on literacy or maths achievement when robust research methods are adopted. Small positive effects on academic outcomes have previously been observed in other research when qualified teachers were used (as is the case in this evaluation); however, this study did not confirm this finding. Another EEF-funded pilot evaluation of a similar SHINE programme with pupils in Year 7 also found only very small non-statistically significant effects of

How secure is the finding?

Findings from this study have moderate security. The study used randomised controlled trial designs (RCTs) with random allocation at the level of the year group (i.e. schools identified eligible pupils from two year groups, one of which was randomly selected to participate in the intervention). The trials were efficacy trials, which aim to test whether the intervention can succeed under ideal conditions.

The trials were smaller in size than expected because not as many pupils were recruited as planned and because a significant proportion of pupils did not complete all the tests at the end of the project (15% in Year 1, 22% in Year 2, and 25% in Year 3).

How much does it cost?

The costs of the intervention were calculated by dividing the grant provided to each school to run the programme for a year by the number of pupils the intervention could accommodate. This cost estimate is £727 per pupil per year.

EEF commentary

The EEF funded the Hallé SHINE on Manchester programme because it built on existing evidence about how out-of-school programmes can be most effective. This evidence suggests that programmes which use qualified teachers to deliver high quality provision might have greater impact.

However, despite using qualified teachers to deliver intensive Saturday school activity over three terms, our project provided no evidence that SHINE improved attainment. Instead, the results are broadly consistent with evaluations of other out-of-school programmes, which tend to showin small or no effects on literacy or maths achievement when robust research methods are used. Another EEF-funded pilot evaluation of a similar SHINE programme, with pupils in Year 7, showed signs of very small effects on numeracy and literacy.

It should be noted that the existing evidence typically focuses on short-term effects and further research is needed to explore longer-term impacts of this and other Saturday school programmes.

This project experienced difficulties in recruiting and retaining pupils, which has been a feature of all four EEF-funded Saturday and Summer School programmes. Whilst recruitment may be influenced by the constraints of running a trial (e.g. limited time to recruit pupils), consistent barriers are reported in terms of engaging pupils and parents with the other programmes.

Downloads