Education Endowment Foundation:EEF blog: How we’re supporting school leaders with evidence to inform their decision-making

EEF blog: How we’re supporting school leaders with evidence to inform their decision-making

An introduction to our our research on School Choices evaluations
Author
EEF
EEF

Every day, school leaders make a number of decisions which could have important implications for pupil learning. So how can they ensure they’re on the right path? In this blog, Christine Kelly, Methodological Innovation Lead, and Guillermo Romero, Senior Evaluation Manager, explore how we’re expanding our work on School Choices evaluations, to better equip school leaders with the evidence they need to support decision-making. 

Blog •7 minutes •

School leaders make choices about school-wide practices and approaches – such as how to organise the school day, or how to communicate with parents – that aim to produce positive outcomes for pupils. But, many school-level practices have limited or no evidence behind them, which means leaders must make decisions using other information.

Through our School Choices research, we want to produce causal evidence about the impact of different school-level approaches and policies on outcomes of interest, with particular attention to impact on pupils from socio-economically disadvantaged backgrounds.

Since 2011, we’ve funded more than 200 impact evaluations to find out whether educational programmes aimed at improving children and young people’s attainment work (or not) and why they work (or don’t work). These programmes typically involve a package of training and resources that are delivered as whole-school, whole-class, or targeted interventions. However, we know that programmes are not the only tool at school leaders’ disposal to influence pupil learning. So it’s important we understand the impact of other choices that school leaders make.

What are School Choices evaluations?

As an example, think about the different ways schools go about managing and improving pupil attendance. There are lots of things that schools could do (have a look at EEF’s Rapid Evidence Review to learn more), but a common approach in England is to hire an Attendance and Family Liaison Officer (which we’ll refer to as attendance officer’ for simplicity). In most instances, this is a full-time school staff member who monitors pupils’ attendance, engages with families and provides support and guidance to help navigate the challenges that may be leading to pupil absenteeism.

Although this sounds like a promising approach, there isn’t much evidence on whether hiring a full-time staff member for this purpose is an effective strategy for improving attendance.

Improving attendance is clearly a high-priority issue for schools, so how can we help schools and investigate the impact of hiring an attendance officer on pupil absenteeism?

Methodologically speaking, there are two ways we could go about this. If the School Choice can be randomly allocated across schools, we could conduct a randomised controlled trial (RCT). Previous EEF blogs highlight the merits of RCTs, which provide a robust way of comparing outcomes between a random group of schools that are selected to implement a certain practice – such as hiring an attendance officer – and another random group of schools that do not implement the practice.

However, it may not always be feasible to randomly allocate schools to certain practices. In the case of attendance officers, school hiring decisions are determined by personnel budgets, the availability of candidates in the local area, and strategic decisions about how best to deploy funding. So, it would be very difficult to design a trial that relied on randomising full-time working individuals into schools to act as attendance officers. As an alternative to the RCT model, we can leverage what are known as quasi-experimental designs (QEDs), which include a range of estimation techniques that capitalise on existing variation in practice to construct comparable groups.

QEDs provide a framework for comparing schools that already employ an attendance officer to similar schools that do not – for example, by applying matching or weighting of school and pupil characteristics to ensure the two groups are as alike as possible except for their use of attendance officers. So, we could take a sample of schools with an attendance officer and find a comparison group of schools that are as similar as possible to that group in terms of certain characteristics, such as: past pupil attendance, school size, funding, location, number of disadvantaged pupils. This allows the evaluation to make an apples-to-apples comparison and say something meaningful about the use of attendance officers.

In collaboration with the Youth Endowment Fund (YEF), we’re funding an evaluation that looks at the impact of Attendance and Family Liaison Officers as part of our co-funded round on attendance and exclusions.

What have we learned about these types of evaluations?

We commissioned our first School Choices evaluations in 2019, although the early projects were interrupted by the COVID-19 pandemic. Here are all the School Choices studies we’ve funded to date:

One big lesson from this existing work is, not surprisingly, that schools do a lot of different things, which can make it tricky to pin down distinct practices that are comparable and feasible to evaluate.

To carry out a robust evaluation, it’s important to establish:

  • If each practice being compared is well-defined.
  • If we have a way of identifying what practice schools are using, either from sourcing administrative data or asking schools directly.
  • If among the groups of schools implementing each practice, there are enough schools with similar characteristics to enable a meaningful apples-to-apples’ comparison.
  • If an impact evaluation can be feasibly designed, as either an RCT or QED, considering factors such as the ability to recruit enough schools to detect a meaningful impact and ensuring that outcomes are measurable and relevant to the study’s objectives.

As a first step, our latest School Choices evaluations include a scoping phase where researchers use methods such as surveys, interviews, and observations to build a clear picture of the school-level practices they want to compare and a better understanding of how they might evaluate them.

To further support these evaluations, we also pair each School Choices evaluation with an independent study advisory board. This group of experts provides valuable insights, guidance, and feedback throughout the evaluation process. Their expertise has been key in quality assuring outputs and informing progress.

And finally, there is always a degree of uncertainty regarding practice, so flexibility is an inherent aspect of our School Choices evaluations. That said, while adaptability is important, having a clear plan at the outset is just as crucial so as not to lose sight of the aim of the evaluation.

What’s next? Evaluating School Choices on teacher recruitment and retention

We know that high-quality teaching is important for pupil outcomes. However, teacher supply is a persistent challenge in the English education system. The hurdle isn’t just attracting quality candidates, but ensuring their continuity is also a massive challenge.

What’s more, schools serving disadvantaged communities tend to face even greater challenges in recruiting and retaining teachers, meaning that teacher turnover and teacher vacancies have a particular impact on socio-economically disadvantaged pupils (Allen et al., 2019; Worth et al., 2022; Tereshchenko et al., 2020).

To address this challenge and better understand how we can make a difference, the EEF commissioned four comprehensive reviews on evidence and practice. These focused on:

*Note: For a short summary on these reviews and their findings, check out one of our recent blogs – Appetite amongst teachers for flexible working approaches, but limited research into their impact.

To build on this work, we’re about to launch a School Choices commissioning round on December 20th, 2023 (keep an eye out!). The aim is to further explore what practices English schools adopt to enhance the working lives of teachers and evaluate the efficacy of such strategies for both recruitment and retention. To learn more about the round or if you are part of a research team interested in applying, please don’t hesitate to contact Guillermo Romero (guillermo.​romero@​eefoundation.​org.​uk).

References

Allen & McInerney (2019) The Recruitment Gap: Attracting teachers to schools serving disadvantaged communities. Available at: https://www.suttontrust.com/wp-content/uploads/2019/12/The-Recruitment-Gap.pdf.

Tereshchenko, A., Mills, M. & Bradbury, A. (2020) Making progress? Employment and retention of BAME teachers in England. UCL Institute of Education: London. Available at: https://discovery.ucl.ac.uk/id/eprint/10117331/

Worth, J. and Faulkner-Ellis, H. (2022) Teacher labour market in England – Annual report 2022, NFER. Available at: https://www.nfer.ac.uk/teacher-labour-market-in-england-annual-report-2022/.