Do all EEF-funded projects need to be independently evaluated?
Who evaluates EEF-funded projects?
How will you measure impact on attainment?
What counts as a valid comparison group?
How big does the project need to be for a robust evaluation?
How long will the evaluation last?
Should I include evaluation costs in my budget?
Will the control group in the project receive the intervention?
How will you use the results from EEF evaluations?
Yes, every project that the EEF funds will be evaluated by a member of our panel of independent evaluators. This will include a quantitative evaluation to assess the impact of the project on students' attainment outcomes, and an implementation and process evaluation to understand the elements of effective practice, as well as to identify the challenges and solutions to scaling it up. The results and full report will be published on the EEF website.
The EEF has recruited through open competition a panel of independent evaluators who are experts in education evaluation. The panel members will be matched with projects based on their knowledge and expertise.
Where a university is involved in delivering a project and evaluation is built in to the application, the minimum role of the independent evaluator will be to validate the evaluation done by the university and author the report published by the EEF. A list of the current panel members can be found here.
Ultimately, the EEF wants to measure the impact that projects have on children and young people's attainment in national tests such as Key Stage 2 assessments and GCSEs, as these are highly correlated with later life outcomes. The performance of all students in EEF-funded projects in these assessments will be tracked using the National Pupil Database. In addition, the EEF may commission additional robust tests of attainment, such as standardised tests of literacy and numeracy, to understand the impact of projects impact on attainment before these milestones.
The EEF is not prescriptive about which test providers evaluators should use, nor do we set a minimum standard for tests. Instead we aim to select the most appropriate test of attainment for the group and intervention being measured. Tests should be reliable, valid and free from bias. The EEF's approach to using tests includes a set of criteria which is intended to provide a framework with which to evaluate tests of attainment and ensure that all the relevant issues are addressed.
A project’s impact is the difference between what actually happened to the students who participated and what would have happened to them had they not been part of the work. To estimate this we need to identify and track a comparison group of students who do not participate in the project, but are similar to the students who do participate in all other ways.
The best way of doing this is by designing the evaluation as a randomised controlled trials (RCT). This involves randomly assigning, from a group of potential participants, which children and young people receive the treatment and which do not (though in some cases the control group will be ‘wait-listed’ and receive the intervention after the trial has been completed). By randomly assigning a large group of students, we aim to get an unbiased measure of the impact of the project.
A good explanation of RCTs, written by Laura Haynes, Owain Service, David Torgerson and Ben Goldacre, can be found here.
In some cases it may be impossible to conduct an RCT. In such cases, the EEF will aim to use a comparison group that is closely matched to those receiving the intervention, for example by controlling for characteristics such as baseline attainment, age, sex and the number of students eligible for free school meals.
Sample size is an important issue in evaluation. We need to include enough participants (e.g. pupils, schools) in our projects to ensure we can be confident that we are observing real, rather than random, differences. The bigger the sample, the more confident we can be with our estimates.
There is no minimum size for EEF-funded projects and we might fund a small project in one school if the research design was good enough, for example involving random assignment of individual students. However, in practice, many education interventions are conducted in whole classes or schools/nurseries/colleges. Where this is the case, the project will need to be run in many schools in order that we can be confident that the differences are caused by the intervention rather than the teachers or schools/nurseries/colleges involved.
The length of the evaluation will depend upon the length of the project. The independent evaluation should, as a minimum, demonstrate the immediate impact of the project on attainment, and ideally also the impact on attainment a year after the project finishes.
In addition, Durham University are leading the over-arching evaluation of EEF-funded projects and will be tracking all pupils using the National Pupil Database to observe long-term impact on attainment.
You can include the costs of some aspects of the evaluation in your budget if you wish. However, this is not necessary because all EEF-funded projects will be evaluated by one of our panel of independent evaluators.
Where evaluation is built into the application, for example because a university is involved in the project, the role of the independent evaluator will be to validate the evaluation done by the university and to author the report the EEF publishes.
Many of our projects will involve a group of schools/nurseries/colleges, classes or students being randomly assigned to either receive the intervention, or not. This is the best way to get an unbiased measure of the impact of the project on attainment.
In all cases, the purposes of the research for understanding 'what works' in raising attainment will be explained to the schools/nurseries/colleges and they will be incentivised to take part in the project. In many projects, they may be ‘wait-listed’ to receive the intervention after the trial is completed, or will receive a different intervention. Where standardised tests are used to measure attainment the results will be given to schools/nurseries/colleges so that they can use them in planning.
All results of EEF evaluations will be published on our website. In addition, we encourage universities involved in our evaluations to publish papers in peer-reviewed journals.
All the results of our evaluations will be integrated with the summary of evidence in our Teaching and Learning Toolkit and its Early Years companion. These Toolkits are resources for teachers and senior leaders on how best to improve the attainment of children and young people, particularly from disadvantaged backgrounds.