EEF Blog: Testing, testing, testing! How do we respond when trials produce different results?
Today the EEF publishes the independent evaluation of the first programme that we've re-granted following a promising initial trial. Here, Emily Yeomans explores why re-granting and re-testing is important - and how we can respond to results which challenge previous findings.
The EEF pipeline
The EEF project pipeline has re-testing built in to it. Projects that haven’t been delivered in English schools before are piloted, whilst those that we’re confident can be feasibly delivered are tested as either an efficacy or effectiveness trial. Testing projects at the efficacy stage allows us to assess whether the core idea, when delivered as intended and under ideal conditions, has promise; while effectiveness trials allow us to test a scalable version of the programme that can reach a greater number of schools.
EEF-funded projects can enter the pipeline at different stages, but the majority will initially be tested as an efficacy trial, with those that show positive results at this stage moving on to an effectiveness trial. 14 projects have made this transition to date and the results of Switch-on, the first of these, are published today.
Re-testing can lead to results that challenge previous findings and this can be difficult. However, it’s also very important. It allows us to test programmes under different conditions, either strengthening our resolve in the effectiveness of a programme, or guiding us to instigate further innovation to improve a programme so that consistent results can be seen, including when delivered at scale.
What the two Switch-on trials tell us
The efficacy trial of Switch-on, published in 2014, indicated that it had a positive impact equivalent to pupils making three months’ additional progress in reading. In this trial, Switch-on training was delivered by the original programme developers. The security of this trial was assessed as being ‘3 padlocks’ on the EEF’s 5-point scale, meaning we are reasonably confident in its findings.
In the effectiveness trial, published today, participating children made no additional progress in reading compared to children in the control schools. The training in this case was delivered by a network of trainers who were trained by the original developers; the Switch-on developers had no direct contact with any of the schools involved. The security of this trial was assessed as being ‘4 padlocks’ on the EEF’s 5-point scale, meaning we have high confidence in its findings.
Why we might expect to sometimes see different results
Replications of studies often lead to different results, even if everything is kept consistent (which wasn’t the case in Switch-on, as the two trials tested different models). This is a difficult message, particularly when we are keen to identify attainment-raising programmes which we can feel confident can be delivered at scale, consistently and reliably.
This isn’t just an issue in education. A study published in 2015 which sought to reproduce 100 psychology studies that had been published in 2008 found considerable differences between the original and the repeated studies, with 97% of the original studies having significant results, but only 36% of the replications showing similar results. The authors suggested a number of reasons why these differences were seen; however, we have to acknowledge that no single result can ever tell us that something will definitely work (or definitely won’t work) in all situations.
The reality is that differences in impact between EEF efficacy and effectiveness trials will sometimes be seen.
This is why our Teaching and Learning Toolkit synthesises the results of multiple trials, so that we can highlight which approaches are generally more effective. And it is why we encourage teachers and senior leaders to test the impact of new programmes through our DIY Evaluation Guide, enabling schools to evaluate impact in their own context. Crucially, it is also why we bring together what we have learned from multiple programmes, as well as what the wider evidence tells us, in accessible, practitioner-facing Guidance Reports – two on 'Improving Literacy' have just been launched, and there will be many more over the next 18 months.
In addition, EEF efficacy-to-effectiveness trials are not simply replications, as what we are attempting to find out is whether an idea that has been shown can work for a few schools will also work when delivered using a model capable of being delivered to many schools.
In the case of Switch-on, the report of the independent evaluator outlines a range of plausible reasons why the results might differ, including changes made by schools to the programme’s intended format and content; that the trials were done with pupils of different ages; and the more active literacy support offered by schools in the effectiveness trial to those pupils in the control group not participating in Switch-on.
What do the new results mean?
The findings from this new effectiveness trial do not erase the findings from the earlier efficacy trial. There is reasonable evidence – from the wider research about effective one-to-one support, as well as the first trial – to suggest that Switch-on does have promise. That is why we advise in our commentary, published alongside the evaluation report today, that “schools considering using Switch-on now should aim to make the conditions as similar as possible to those in that first trial. Involvement from the original developers could be beneficial”. The challenge now is to find an effective way to scale the model so that it can be delivered to large numbers of schools with similarly positive impacts to those seen in the first trial.