ThinkForward is a coaching programme, developed by Impetus Private Equity Foundation, which is designed to support secondary school pupils. ThinkForward is targeted at pupils who have been identified as being at high risk of not being in education, employment or training (NEET) following the completion of compulsory education. Coaches are assigned to schools and work with selected pupils as they progress through Key Stage 4 (GCSEs), with the aim of supporting them to make a successful transition into adulthood. The programme provides targeted support tailored to pupils’ needs through one-to-one and group work. The programme usually works with young people aged 14 - 19, though this pilot focused only on the intervention up to age 16, prior to pupils leaving school. The pilot involved Year 10 and 11 pupils in four London secondary schools, beginning in January 2014. The Y11 pupils received up to six months of ThinkForward until their GCSEs in summer 2014. The Y10 pupils received up to 18 months of ThinkForward up until their GCSEs in summer 2015.
The pilot was evaluated using a range of qualitative methods and a small Randomised Controlled Trial (RCT). The aims of the evaluation were to determine whether a future large-scale evaluation was possible, and to gain an initial estimate of the programme’s impact on GCSE scores, the likelihood of continuing into post-compulsory education, and decreasing pupil absences. Randomisation took place at both school and pupil level. Within the two randomly assigned intervention schools, eligible pupils were randomly allocated either to an intervention group or a within-school control group. Across the four schools, there were 181 pupils in the Y11 cohort, 40 of whom received the intervention, and 160 pupils in the Y10 cohort, 37 of whom received the intervention.
Testing the impact of one to one coaching for at-risk 14-16 year olds.
Sheffield Hallam University, University of Essex
The following conclusions summarise the project outcome
The approach is not ready for a large school level randomised controlled trial.
Before a further trial, it would be necessary to solve data collection issues and to develop an approach that allows the intervention to be targeted at pupils at risk of becoming NEET, while conducting a rigorous randomised controlled trial.
Future trials should not use pupil-level randomisation because it is likely that the intervention has ‘spillover’ effects to other pupils.
The trial did not find evidence that the programme had an impact on GCSE scores, absences, or attitudes. This was a small pilot RCT so these findings have low security.
Teachers reported benefits, particularly during the later months of the pilot, where they observed impacts on the intervention pupils’ behaviour.
What is the impact?
The study found a number of issues would need to be addressed before a future RCT is conducted: for example, further work is needed so that the intervention is able to accurately target pupils at risk of becoming NEET, while at the same time meeting the requirements of an RCT research design that pupils are randomly allocated to either receive the intervention or not. Another risk that would need to be mitigated is that some pupils who are not allocated to the intervention might benefit from the intervention, for example, if ThinkForward Coaches are asked to teach regular classes, as was reported in the process evaluation. This could result in the impact of the programme being underestimated and future RCTs would need to be designed to minimise this risk. Future trials should also employ approaches that might increase response rates to survey data, while making sure that any data is collected independently by researchers not involved in the delivery of the intervention.
The pilot found no positive impact on unauthorised absences or GCSE scores amongst students that took part in the programme.
Attitudes to higher education, and the ‘ThinkForward Mindsets’ of ‘aspiration’ and ‘determination’ were measured using a survey. Responses showed a sharp decline in aspiration to attend higher education, measured in terms of expectations of applying to university, getting a place and graduating, and limited evidence of increases in ‘determination’ and ‘aspiration’, measured using survey questions. The survey results should however be treated with caution due to the low response rate within the pilot.
Evidence from the process evaluation suggests that ThinkForward is feasible to deliver in schools. School leads, coaches themselves and young people from intervention cohorts across both schools reported that they believed the programme was beneficial. Coaches and teachers felt that the support of a trusted adult help pupils when they were considering GCSE options in Year 9 and once they left school, as well as during Years 10 and 11. However, the complexity of options beyond school, and the numerous policies and procedures in place to support young people with behavioural and motivational issues, means it is difficult to evaluate interventions like ThinkForward in isolation from other influences.
The costs of ThinkForward are high in comparison to other interventions, with a cost of £2426.50 per pupil per year during the pilot. Although, the costs of the intervention have since been reduced, and schools often do not pay for the entire intervention themselves, the high cost could make recruiting schools to a large school-level trial expensive.
This evaluation was undertaken when ThinkForward was in a relatively early stage of its development, and the findings should be considered in this context. ThinkForward is committed to developing the programme and has already instituted many of the improvements that this report recommends.
ThinkForward is a coaching programme starting in Year 9 and designed to reduce the risk of students not being in education, employment or training after leaving school. EEF funded a pilot evaluation to investigate whether this could also be a promising approach for improving attainment, because of the intensive support students receive.
Coaching programmes are difficult to evaluate using the high quality trials EEF usually run, because of how participants are selected. This pilot therefore investigated whether ThinkForward was ready for a large trial, and concluded it was not.
The pilot also provided initial evidence on whether ThinkForward was likely to be effective at raising attainment. It found no evidence of promising results for improving Key Stage 4 attainment, reducing absences or changing attitudes towards further and higher education.
Because of the low impact estimates and relatively high cost, EEF is unlikely to pursue ThinkForward as a cost effective way to improve attainment. However, it may still be effective at achieving its wider aim of helping students to stay in education, employment or training.
|Is there evidence of promise?||No / Unclear||Evidence of impact was not found across any of the four outcome areas.|
|Was the approach feasible?||Yes||The process evaluation found positive comments from the students and staff involved in the approach.|
|Is the approach ready to be evaluated in a trial?||Not yet||The pilot found methodological challenges around data collection, and spillover, which make a within-school trial unsuitable. The approach is not ready for a large school-level RCT.|