Evaluation FAQs

Do all EEF projects need to be independently evaluated?
Who evaluates EEF projects?
How will you measure impact on attainment?
What counts as a valid comparison group?
How big does the project need to be for a robust evaluation?
How long will the evaluation last?
Should I include evaluation costs in my budget?
Will control schools involved in the project receive the intervention?
How will you use the results from your evaluations?

Do all EEF projects need to be independently evaluated?

Yes, every project that the EEF funds will be evaluated by a member of our panel of independent evaluators. This will include a quantitative evaluation to understand the impact of the project on children’s attainment and a qualitative evaluation to understand how it worked in practice. The results will be published on the EEF’s website.

Who evaluates EEF projects?

The EEF has recruited a panel of independent evaluators who are experts in education evaluation through a process of open competition. The panel members will be matched with projects based on their knowledge and expertise.

Where a university is involved in a project and evaluation is built in to the application, the minimum role of the independent evaluator will be to validate the evaluation done by the university and author the EEF report. A list of the current panel members can be found  here.

How will you measure impact on attainment?

Ultimately, the EEF wants to measure the impact that projects have on children’s attainment in national high stakes tests such as Key Stage 2 assessments and GCSEs. The performance of all children in EEF projects in these assessments will be tracked using the National Pupil Database. In addition, the EEF may commission additional robust tests of attainment, such as standardised tests of literacy and numeracy, to understand the impact of projects impact on attainment before these milestones.

The EEF is not prescriptive about which test providers evaluators should use, nor do we set a minimum standard for tests. Instead we aim to select the most appropriate test of attainment for the group and intervention being measured. Tests should be reliable, valid and free from bias. The  EEF's approach to using tests includes a set of criteria which is intended to provide a framework with which to evaluate tests of attainment and ensure that all the relevant issues are addressed.

What counts as a valid comparison group?

A project’s impact is the difference between what  actually happened to the children who participated and what would have happened to them had they not been part of the work. To estimate this we need to identify and track a comparison group of children who do not participate in the project, but are similar to the children who do participate in all other ways.

The best way of doing this is to randomly assign, from a group of potential participants, which pupils receive the treatment and which do not (though in some cases the control group will be ‘wait-listed’ and receive the intervention after the trial has been completed). By randomly assigning a large group of children, we hope to get an unbiased measure of the impact of the programme. Therefore the EEF will seek to do this wherever possible.  

A good explanation of randomised trials written by Laura Haynes, Owain Service, David Torgerson and Ben Goldacre can be found  here.

In some cases it may be impossible to randomly assign. Where this is the case the EEF will aim to use a comparison group that is closely matched to the pupils or schools receiving the intervention, for example by controlling for characteristics such as baseline attainment, age, sex and the number of pupils receiving free school meals.

How big does the project need to be for a robust evaluation?

Sample size is an important issue in evaluation. We need to include enough children and schools in our projects to ensure we can be confident that we are observing real rather than random differences. The bigger the sample the more confident we can be with our estimates.

There is no minimum size for EEF’s projects and we might fund a small project in one school if the research design was good enough, for example involving random assignment of individual children. However, in practice many education interventions are conducted in whole classes or schools. Where this is the case the project will need to be run in many schools in order that we can be confident that the differences are caused by the intervention rather than the teachers or schools involved.

How long will the evaluation last?

The length of the evaluation will depend upon the length of the project. The independent evaluation should as a minimum demonstrate the immediate impact of the project on attainment, and ideally also the impact on attainment a year after the project finishes.

In addition, Durham University are leading the overarching evaluation of EEF projects and will be tracking all pupils using the National Pupil Database to demonstrate long term impact on attainment.

Should I include evaluation costs in my budget?

You can include the costs of some aspects of the evaluation in your budget if you wish. However, this is not necessary because all EEF projects will be evaluated by one of our panel of evaluators.

Where evaluation is built in to the application, for example because a university is involved in the project, the role of the independent evaluator will be to validate the evaluation done by the university and author the EEF report.

Will control schools involved in the project receive the intervention?

Many of our projects will involve a group of schools, classes or pupils being randomly assigned to either get the programme or not. This is the best way to get an unbiased measure of the impact of the programme on attainment.

In all cases the purposes of the research for understanding what works in raising attainment will be explained to the schools and they will be incentivised to take part in the project. In many projects schools may be ‘wait-listed’ to receive the intervention after the trial is completed or will receive a different intervention. Where standardised tests are used to measure attainment the results will be given to schools so that they can use them in planning.

How will you use the results from your evaluations?

All results of EEF evaluations will be published on our website using a standard EEF report. In addition, we encourage universities involved in our evaluations to publish papers in peer-reviewed journals.

All the results of our evaluations will be integrated with the summary of evidence in the  Teaching and Learning Toolkit. This is a resource for teachers and schools on how best to improve the attainment of disadvantaged pupils. It summarises the average improvement in children’s attainment for different learning strategies, and in the future will include case studies from EEF projects that have worked.