1. Impact of the intervention: Very small effect sizes on the primary and secondary outcomes of literacy and numeracy were detected. As this was an exploratory evaluation design and the number of schools involved in the project was small, this made it unlikely that any impact of SHINE could be detected, should the true estimate of effect be small.
2. Implementation and organisation issues: Feedback from the teachers and pupils suggested that the intervention was perceived to be valuable in providing an opportunity to consolidate literacy and numeracy skills.
3. Feasibility of evaluation design: The purpose in piloting this design was to explore the feasibility of using this approach when randomly allocating pupils to an intervention group was not possible or desirable. The regression discontinuity design is an appropriate evaluation method for education interventions. However, because it is less powerful than a randomised controlled trial, the number of pupils involved must be two and a half times greater if a small effect is to be detected.

This pilot was designed to explore the impact of a Saturday school for secondary pupils on literacy and numeracy, as well as the organisational and implementation issues of the intervention. The feasibility of using a non-randomised evaluation approach using a regression discontinuity design (RDD) to evaluate the impact was also tested in this pilot.

The evaluation showed a very small impact on literacy outcomes across all four schools using the Progress in English and Progress in Maths tests. These findings are limited evidence of promise of the intervention. A larger RDD study or randomised controlled trial would be required to investigate the impact of the intervention and the size of the effect (if any) with confidence.

A subgroup analysis of children eligible for free school meals (FSM) was conducted. This showed no evidence of the intervention differentially benefiting pupils eligible for free school meals in any of the schools. These findings should be treated with caution because, due to the small sample size, the study was unable to detect the small effects with confidence.

The pilot demonstrated that it was feasible to design, implement and analyse the RDD successfully; and the design could be used again to evaluate similar interventions where randomisation is not possible or desirable.

The process evaluation involved observations of activities, interviews and focus groups with pupils and teachers. However, due to the very small samples in the evaluation, all findings should be treated with caution.

All teachers interviewed agreed that pupils liked the familiarity and the relaxing atmosphere that SHINE in Secondaries encouraged and that they learnt while being engaged in an enjoyable way. An important outcome reported by the teachers in two schools observed was pupils’ improved social skills, better relationships at all levels and increased confidence through their participation in practical and team-building activities.

The five pupils who were interviewed spoke enthusiastically about their SHINE in Secondaries experiences. All pupils mentioned that they had made good progress in their learning while having fun.

The SHINE project managers, who were also teachers in the school, were considered to be integral to the successful running of the project. All eight teachers interviewed agreed that SHINE in Secondaries gave them the opportunity to think and plan ​‘outside of the box’ and they enjoyed teaching around a particular theme, which helped reinforce links with ​‘real life’.

It was also felt that the continuity with the school week that the SHINE in Secondaries programme provided for pupils in Year 7 was an excellent feature of the intervention. From our visits, it seemed that all SHINE in Secondaries teachers see all or at least some of the pupils during the week. This allowed the teaching content of the SHINE in Secondaries programme to better target pupil needs and also to build on the pupils’ familiarity with the teaching staff.

This evaluation was a pilot using a regression discontinuity (RD) design. Unlike a randomised controlled trial, where pupils are randomly allocated to an intervention or control group, this approach assigns pupils to intervention and control groups according to their results on a pre-test. We began by looking at the range of pupil outcomes on a Progress in English test. Pupils were assigned to the intervention if they fell below a certain mark (the first ​‘cut point’). Pupils were assigned to the control group if they fell above a second ​‘cut point’ on the test. Pupils falling between the two cut points were randomly allocated to the intervention or control group, thereby creating a ​‘mini’ randomised controlled trial (RCT) within the RDD.

Although a randomised controlled trial design is stronger for inferring causation, for this evaluation it was not deemed to be suitable. The RD design guaranteed all low-attaining pupils would be offered the opportunity to attend SHINE in Secondaries, which was a very important consideration for schools.

Uptake of places at the intervention was 73% of those who were assigned to the intervention group which means that 27% of those counted in the analysis did not attend the intervention. The intention-to-treat analysis provides average impacts of the intervention for those who were offered the intervention, whether or not they actually participated in the intervention. This created a ​‘fuzzy’ regression discontinuity and reduces the robustness of the evaluation. However, average individual attendance of those children who accepted places was 68%. This includes any pupil that is considered as taking up their place by attending a minimum of one session. The average attendance is based on the numbers of sessions possible in each school.