The intervention evaluated here is one of two ‘youth social action’ projects jointly funded by the Education Endowment Foundation, the U.K. Cabinet Office, the Pears Foundation and the Stone Family Foundation. It was delivered by the Youth United Foundation (YUF) and involved uniformed youth organisations being established in schools in six areas in the north of England. YUF helped to set up new units of The Scout Association, Fire Cadets, Sea Cadets or St John Ambulance in participating schools. The number, duration, and frequency of sessions varied: most groups met weekly, sessions lasted two hours on average, and the average number of sessions in the academic year was 24. Activities were delivered by trained staff from the uniformed youth organisations and in some cases also involved adult volunteers, including teachers.
This project assessed the impact of participation on pupils’ academic attainment, and on wider outcomes such as self-confidence and teamwork, using a randomized controlled trial design. The wider outcomes are of particular relevance because the participating organisationsshare core aims of inspiring young people to do community work and volunteer, to learn new skills, and to be active citizens. Seventy-one schools were randomly allocated to either receive the intervention or not. Of 7,781 Year 9 students, 3,377 reported in the initial survey that they would like to take part in the kinds of activities offered, and 663 took part in uniformed youth group activities during the 2014/2015 academic year. A process evaluation was also conducted to collect information about the mode of delivery and programme implementation, and feedback from teachers, pupils and parents.
Testing the impact of social action on young people’s engagement and attainment.
Character & essential skills
The following conclusions summarise the project outcome
There is no evidence that the intervention had any benefit in terms of pupils’ academic performance. Although the attainment data suggests a small negative impact, the quality of this data is too low to draw this conclusion with confidence. The data quality was compromised due to changes in national testing.
Participation in the intervention is associated with a small improvement in self-reported non-attainment outcomes including self-confidence and teamwork. It is possible that these small effects are an underestimate due to technical issues regarding the groups of children that were compared in the analysis.
For pupils eligible for free schools meals, there is no evidence that the intervention had a positive impact on academic attainment or self-reported character attributes. Again, the attainment data suggests a negative impact, but the quality of this data is too low to draw this conclusion with confidence.
Almost a quarter of schools did not deliver the intervention due to issues such as lack of teacher volunteers, and other schools did not deliver a full programme of activity. Support from senior leaders, dedicated space, school staff time, and a dedicated slot in the school day or after school were all identified as necessary conditions for successful implementation.
Study participants were extremely positive about the intervention and many felt it had a positive effect on the behaviour and skills of participating pupils.
What is the impact?
In general, the analysis to assess the impact of the intervention reveals small ‘effect’ sizes. In the case of non-attainment outcomes these are small and positive. This is the case for almost all of the non-attainment outcomes measures, as well as the two identified as primary outcomes (teamwork and self-confidence). In the case of attainment outcomes they are small and negative. However, for these attainment outcomes, the reliability of the finding is undermined somewhat by the quality of the data. For reasons beyond the control of the evaluator, the KS3 attainment data collected at the end of the trial may not have been consistent across schools. Further evaluation would be required to understand whether the intervention did indeed have a negative impact on attainment. Future research could also assess the impact of full exposure to the programme over two to three years, to give adequate time for the uniformed organisations to establish themselves.
The process evaluation revealed that pupils, teachers, and parents all had very positive views about the intervention, and in general believed that it had a positive impact on the pupils involved. It also highlighted a number of factors that had prevented the intervention from being delivered as planned in some schools, including a lack of dedicated space and time, a lack of adult volunteers (including teachers) to support the uniformed youth group delivery staff, and sometimes a lack of support from school senior leaders. Schools should consider these factors when deciding whether to implement a similar intervention.
|Outcome||Effect size||Estimated months’ progress||Security rating||Cost rating|
|KS2–KS3 progress in English||-0.09||-1|
|KS2–KS3 progress in Maths||-0.09||-1|
|Gain in self-confidence||0.10||N/A|
|Gain in teamwork||0.04||N/A|
How secure is the finding?
The findings related to non-attainment outcomes have medium to high security. The trial was a large, well-designed, randomised controlled trial, with schools randomly allocated to either receive the intervention or not. The pupils from the intervention schools were similar to those from the comparison schools who received no intervention, and relatively few pupils were lost to the analysis due to issues such as moving school. However, the non-attainment outcomes were measured using a bespoke survey. This survey was well-designed and based on well-established test items, but as it was developed in its current form for the purposes of this particular evaluation, it may not be able to provide a standardised measure of non-attainment outcomes.
The findings related to attainment outcomes have low to medium security. Although the trial was well-conducted as described above, it transpired that the Key Stage 3 data which was to be used as an outcome measure would not be available. The alternative outcome measures that had to be used were based on scores from internal school records and were very weakly related to previous standardised test scores. Different schools also used different tests and marking schemes to derive the scores. This makes it harder to estimate reliably the size of the intervention’s attainment impact. A more reliable estimate of the impact on attainment will be available after these pupils reach the end of Key Stage 4 and additional national test data is available.
How much does it cost?
The costs vary depending on the number of outdoor activities and the size of the groups. Assuming 20 pupils per school, the cost is estimated at £18 per pupil per year for St John Ambulance, £26 for the Sea Cadets, £245 for The Scout Association, and £420 for the Fire Cadets—an average of about £180.
Around 1.5 million children and young people in the UK take part in uniformed youth groups such as those involved in this project. Although there is widespread belief that this kind of activity has a range of benefits, this is the first large scale evaluation in the UK which has robustly assessed the impact of participation on attainment and non-cognitive skills.
The project found a small positive impact on outcomes such as self-confidence and teamwork. This is high quality evidence which supports claims that there are non-cognitive benefits to taking part. However, we did not find a positive impact on attainment. In fact the data indicated a possible small negative effect, although the security of this result is very low. We will therefore look at attainment outcomes again after the children in the trial sit their next national test. This will provide better data about the impact on attainment.
Currently, the existing evidence from this project and other sources suggests that schools should not expect uniformed youth group activities to raise pupil attainment. However, there are many other reasons why schools may want to offer these activities, including the growing evidence that they can positively affect non-cognitive skills.