Education Endowment Foundation:Guest Blog: Why digging a little deeper into educational research matters

Being genuinely evidence-informed means going beyond the quick fix or the top tip, writes Mark Miller, Head of Bradford Research School, as he explains how a little digging beneath the surface can reveal important nuances…

The EEF Toolkit

The Teaching and Learning Toolkit, and its counterpart the Early Years Toolkit, are heavily visited by teachers and school leaders. They are both incredibly useful, but there are pitfalls to avoid, and you will get much more from it if you scratch beneath the surface and read the full summaries. In our blog here, we set out 5 questions that can you can ask:

• How effective is this intervention? 
• Which specific approaches are most effective?
• Who benefits the most from this kind of approach?
• What are the problems/​gaps/​areas of concern?
• Any other factors worth considering?

When dealing with a meta-analysis such as this, the overall rating can hide good and bad studies. Take ​‘Setting or Streaming’ for example. The EEF Toolkit states that the impact of this is negative (-1 month’s progress). But in the summary is the following passage:

So the average is negative, but there are examples where the impact is positive. And you see this idea of variability in other summaries, such as that for Digital Technology:

And if you want to explore this even further, the summary links to the studies that fed into the meta-analysis.

Similar nuances can be seen in evaluations of EEF-funded projects. It’s easy to skim the surface of an evaluation and conclude that it worked or it didn’t. But the reality is that everything is more complicated than that. It’s always worth reading further

For example, following a successful efficacy trial of Catch Up Literacy, where the EEF noted an effect size of +3 months, the scaled up version showed less promise. One of the conclusions of the evaluation report was that implementation was an important factor, a nuance that is lost without reading the report:

Beyond the headlines

Interesting new research can often be published with an attention-grabbing headline in the press

When the Improving Behaviour in Schools guidance report launched, one such headline was ​‘Greeting pupils at the door improves behaviour’. When our research lead Luke Swift looked a little further into the original study (Cook et al, 2018), he saw that ​‘positive greeting at the door was only one aspect of the intervention evaluated’. Four of the researchers were kind enough to discuss the paper with him, and while the headline is not exactly wrong, it certainly hides the nuance. Andrew Thayer said:

While you may not go so far as the intrepid Luke Swift, the nuance is always there to be found. And you’ll find it any time complex evidence is reduced to a soundbite or a quick summary.

Read Part 1 and Part 2 of Luke’s blogs.

Following the evidence

Sometimes a lot of the how and why can be lost by the time a message around evidence is communicated. One example of this that we encountered recently was in the EEF’s Metacognition and Self-Regulated Learning guidance report

Recommendation 7 is that ​‘Schools should support teachers to develop their knowledge of these approaches and expect them to be applied appropriately’, and within that a bullet point suggests that ​‘Teachers can use tools such as ​‘traces’ and observation to assess pupils’ use of self-regulated learning skills’.

This reference to ​‘traces’ is merely the tip of an evidence iceberg, and to understand some of the nuance it’s worth exploring. This reference took me to the Metacognition and Self-Regulation: Evidence Review, where the following appeared:

Not only did this reveal some of the nuances and limitations of ​‘traces’, but it pointed me to additional research (Dent and Koenka, 2015) which pointed me to additional research (Winne and Perry, 2000). The latter revealed much finer detail into the concept of traces, and broadened my understanding beyond that simple bullet point in the guidance.

We have used so many metaphors in this post: digging deep; scratching the surface; icebergs. Whatever the metaphor, look for the nuances.