Effect Sizes in Education Trials in England

Abstract

Sample size, or power calculations are an integral part of conducting randomised controlled trials. Rules of thumb can be useful in these calculations, but can have negative consequences if the rules do not match up with reality. We review a large database of effect sizes in education, and find that average effect sizes are roughly one third the size of those described as ‘small’ in a widely used rule of thumb, that less than 5% of all effect sizes meet the threshold for ‘small’ and that effect sizes are fairly stable over time and the age of children involved. We find that interventions that are clustered for randomisation typically produce smaller effect sizes, and that researchers’ intuition about the ordinal ranking of findings is fairly accurate. We suggest a revised rule of thumb for education randomised trials.

Keywords: Education, RCTs, Experiments, Power, Sample Size, Methods

JEL Classification: C93