WhatWorked Teachers is an online platform that helps teachers run small-scale randomised controlled trialsAn RCT is used evaluate an educational programme by assigning settings to one of two groups: the intervention group, who receive the programme or the control group, who continue with business as usual. This ensures that any differences in outcomes can be confidently attributed to the programme, providing a robust estimate of the impact and contributing to the evidence for what works in improving educational outcomes. as part of everyday classroom practice. It automates key stages of the evaluation process.
In this study, teachers used the platform to test a six-week spelling intervention, choosing between two spelling approach options and whether to randomise pupils at a whole-class or individual level. The platform assigned pupils to groups, and teachers administered a pre-test and entered the results. After delivering the intervention, teachers administered a post-test and entered those results. The platform used the data to generate an impact report
This study drew on platform usage data, teacher surveys and interviews to assess whether the platform could feasibly automate small-scale randomised controlled trialsAn RCT is used evaluate an educational programme by assigning settings to one of two groups: the intervention group, who receive the programme or the control group, who continue with business as usual. This ensures that any differences in outcomes can be confidently attributed to the programme, providing a robust estimate of the impact and contributing to the evidence for what works in improving educational outcomes.. It also explored whether the resulting data provided meaningful findings.
This methodological innovation project was funded to explore the barriers, risks, and potential of using the WhatWorked Teachers platform to support a new approach to classroom-based evaluation.
- Some teachers reported initial uncertainty when selecting from the options available on the platform (the combination of spelling approach and whole-class or individual randomisation), suggesting clearer guidance at this stage would be helpful.
- More teachers selected the whole-class than the individual randomisation option, which provides a useful insight into the type of small-scale trial that teachers prefer.
- Teachers who started a small-scale trial were able to complete the trial procedures, with 100% of teachers who completed pre-tests with their pupils going on to complete post-tests.
- Teachers reported a high level of confidence with using the platform.
- Implementation was more consistent for whole-class randomisation than it was for individual randomisation (small groups). For individual randomisation, staffing pressures, timetabling and maintaining separation between groups were reported to be challenges.
- Teachers had some difficulties with interpreting the impact reports, especially the effect sizes, confidence intervals, and the meaning of statistical uncertainty.
- Some teachers reported discussing findings with colleagues and senior leaders, suggesting the potential for the platform to stimulate professional dialogue.
- The platform was able to collect cumulative data and an exploratory meta-analysis of some of the data was conducted. For one of the study conditions, the pooled effect size was 0.27, this is larger typically observed in educational RCTs. However, despite pooling the data, the confidence interval remains wide (0.03 to 0.51), suggesting that there are still questions around the precision of the estimate.
- Further research is needed to identify the best ways to analyse evidence collected across multiple classroom evaluations. Clear safeguards are also needed to help teachers interpret results from individual classrooms appropriately.