Sci-napse aims to improve pupils’ science knowledge using quizzes in lessons. Teachers use an online quizzing tool called Q‑fire to deliver quiz questions interspersed with usual classroom activity. This trial evaluation tested two versions of the programme – a ‘test-based’ version, in which fixed points were allocated for correct answers; and a second, ‘game-based’ version, in which pupils were allocated varying awards.
There is some evidence from neuroscience that the motivation associated with rewards can influence the rate at which pupils learn – and that this impact might be increased through the use of uncertain rewards. The EEF and Wellcome funded this evaluation to see if these neuroscientific findings would translate to improved pupil outcomes in the classroom.
The independent evaluation did not find a positive impact on science test scores for Key Stage 3 pupils. However, this result compares all classes of teachers that were offered the programme to a control group of classes where teachers were not offered the programme – meaning that teachers who did not fully implement the approach contribute to the overall result. This is standard in all EEF evaluations, as it is critical to find out whether a programme is likely to be used by teachers in their classrooms. This is important to bear in mind in the case of Sci-napse because only 54% of teachers in the test-based approach and 29% of the teachers in the game-based classes met the minimum requirements of the intervention, wherein those classes which did meet the minimum requirements there was some tentative evidence of a positive impact. Some of the reasons given by teachers for this poor programme implementation include difficulties with the Q‑fire quizzing tool and the challenge of fitting six quiz questions into a lesson
This trial shows the importance of implementation, both for schools and programme developers. Despite the theoretical evidence to support the intervention, proper implementation is crucial if programmes are to have a positive impact on pupil outcomes. The EEF is focused on improving the implementation of approaches in schools, and has published a guidance report to support best practice. The EEF has no further plans to trial the Sci-napse programme
- Children in both test- and game-based Sci-napse classes made less progress than pupils in the comparison classes. The difference in scores is not “statistically significant”. This means that the statistical evidence does not meet the threshold set by the evaluator to conclude that the true impact was not zero. This finding has moderate to high security.
- Only 54% of teachers in the test-based Sci-napse classes and 29% of in the game-based classes met the minimum requirements of the interventions. This meant using questions in some, most or all of their lessons – and, for the game-based intervention, making use of escalating points and the wheel of fortune.
- Exploratory analysis found that the test-based Sci-napse classes that met the minimum requirements of the intervention made an average of three additional months progress. A small positive impact was found in the game-based classes that met the minimum requirements. The small number of pupils included in these analyses means that they should be treated with caution.
- Teachers and pupils reported that Sci-napse was valuable as a revision tool but indicated that it was difficult to fit six questions into each lesson and that there were challenges with the technology – particularly around adding their own questions.