The Unvarnished Report of Conducting an RCT of a Statewide STEM Program in the Process of Scaling Up

22 Pages Posted: 9 Nov 2014  

Denis Newman

Empirical Education Inc.

Andrew P Jaciw

Empirical Education Inc.

Date Written: October 31, 2014

Abstract

We were responsible for a large scale randomized control trial to evaluate the effectiveness of the Alabama Math Science and Technology Initiative (AMSTI). The official report released by IES after considerable technical review, editing, and formatting, is what we call the varnished version. This chapter delves into a number of the nuts and bolts issues and lessons-learned in conducting this study and provides an unvarnished version of the story. Our study has provided a strong example of taking advantage of a planned scale-up of a program by an education agency to conduct an RCT. We provide five lessons we learned from processes that arose behind the scenes and highlight the importance of exploratory findings. These include how the research questions evolved, how the approach to randomization can help in working with a local agency, reasons for difficulty in maintaining a control group embargo when working in the context of a locally-funded scale-up, issues in avoiding bias in the impact model selection, and what the local agency should make of the results of an RCT.

Keywords: AMSTI, RCT, Alabama Math Science and Technology Initiative, research questions, impact model selection, IES, education agency, locally-funded, scale-up, control group, randomization, exploratory findings

Suggested Citation

Newman, Denis and Jaciw, Andrew P, The Unvarnished Report of Conducting an RCT of a Statewide STEM Program in the Process of Scaling Up (October 31, 2014). Available at SSRN: https://ssrn.com/abstract=2519677 or http://dx.doi.org/10.2139/ssrn.2519677

Denis Newman (Contact Author)

Empirical Education Inc. ( email )

425 Sherman Ave, #210
Palo Alto, CA 94306
United States

HOME PAGE: http://empiricaleducation.com/

Andrew P Jaciw

Empirical Education Inc. ( email )

425 Sherman Ave, #210
Palo Alto, CA 94306
United States
6503281734 (Phone)
6503281734 (Fax)

HOME PAGE: http://empiricaleducation.com/

Paper statistics

Downloads
17
Abstract Views
247