The Unvarnished Report of Conducting an RCT of a Statewide STEM Program in the Process of Scaling Up

Abstract

We were responsible for a large scale randomized control trial to evaluate the effectiveness of the Alabama Math Science and Technology Initiative (AMSTI). The official report released by IES after considerable technical review, editing, and formatting, is what we call the varnished version. This chapter delves into a number of the nuts and bolts issues and lessons-learned in conducting this study and provides an unvarnished version of the story. Our study has provided a strong example of taking advantage of a planned scale-up of a program by an education agency to conduct an RCT. We provide five lessons we learned from processes that arose behind the scenes and highlight the importance of exploratory findings. These include how the research questions evolved, how the approach to randomization can help in working with a local agency, reasons for difficulty in maintaining a control group embargo when working in the context of a locally-funded scale-up, issues in avoiding bias in the impact model selection, and what the local agency should make of the results of an RCT.

Keywords: AMSTI, RCT, Alabama Math Science and Technology Initiative, research questions, impact model selection, IES, education agency, locally-funded, scale-up, control group, randomization, exploratory findings