The EEF ‘padlocks’ security rating

This rating is a measure of the security of the trial result. 

A detailed explanation of the EEF's 'padlock' security rating for each project is provided in the first section of the evaluation report, which can be downloaded at the bottom of the project page.

'Padlock' security ratingLevel of security
Very high security
High security
Moderate to high security
Moderate to low security
Low security
Very low security

You should always consider the EEF Months’ Progress measure and cost rating as well as the EEF ‘padlocks’ security rating. For some projects, the security rating is high, but the impact is low or negative or the cost is very high. For more information about the EEF’s Months’ Progress measure see here.

Why we have a security rating

RCTs are one of the most rigorous way of evaluating the impact of an intervention. But even the result of an RCT is always an estimate: there will always be uncertainty around the precise size of the educational impact. This is because factors such as unintended biases, dropout rates among schools or pupils, and random sampling, can sometimes affect the results. 

That is why the EEF presents a ‘padlock’ security rating for every trial that it funds. It provides a measure of the extent to which such factors may have affected the result of the evaluation. 

The security rating is awarded based on the assessment of two external peer reviewers. More detail on how the security rating is allocated can be found here.

The EEF security rating and statistical significance 

Many evaluators use a concept called statistical significance to assess whether RCT results are sufficiently secure. Statistical significance focuses on the role of ‘statistical uncertainty’ in the result of the evaluation. Statistical significance is a widely used concept but there is a lot of disagreement about exactly what it means.

When summarising results, the EEF uses the 'padlock' rating to assess their security because it is simple, and it takes account of a range of factors which can affect the result of the evaluation. 

The evaluation reports published by the EEF also present findings on statistical significance wherever the independent evaluator has provided them. However, the EEF does not require the reporting of statistical significance. Where statistical significance is reported, results can be found in the first section of the project report, which can be downloaded at the bottom of the relevant project page. For more information on the EEF’s use of padlock rating and statistical significance you can read our blog on the topic, or see our more detailed statement here.