Designing and Analyzing Studies that Randomize Schools to Estimate Intervention Effects on Student Academic Outcomes Without Classroom-Level Information

Abstract

This paper provides practical guidance for researchers who are designing and analyzing studies that randomize schools - which comprise three levels of clustering (students in classrooms in schools) - to measure intervention effects on student academic outcomes when information on the middle level (classrooms) is missing. This situation arises frequently in practice because many available data sets identify the schools that students attend but not the classrooms in which they are taught. Do studies conducted under these circumstances yield results that are substantially different from what they would have been if this information had been available? The paper first considers this problem in the context of planning a school-randomized study based on preexisting two-level information about how academic outcomes for students vary across schools and across students within schools (but not across classrooms in schools). The paper next considers this issue in the context of estimating intervention effects from school-randomized studies. Findings are based on empirical analyses of four multisite data sets using academic outcomes for students within classrooms within schools. The results indicate that in almost all situations one will obtain nearly identical results whether or not the classroom or middle level is omitted when designing or analyzing studies.

Keywords: group randomized experiment, multi-level model, evaluation research and missing middle-level variance component

JEL Classification: C90, I21