The Unvarnished Report of Conducting an RCT of a Statewide STEM Program in the Process of Scaling Up

22 Pages Posted: 9 Nov 2014

See all articles by Denis Newman

Denis Newman

Empirical Education Inc.

Andrew Jaciw

Empirical Education

Date Written: October 31, 2014


We were responsible for a large scale randomized control trial to evaluate the effectiveness of the Alabama Math Science and Technology Initiative (AMSTI). The official report released by IES after considerable technical review, editing, and formatting, is what we call the varnished version. This chapter delves into a number of the nuts and bolts issues and lessons-learned in conducting this study and provides an unvarnished version of the story. Our study has provided a strong example of taking advantage of a planned scale-up of a program by an education agency to conduct an RCT. We provide five lessons we learned from processes that arose behind the scenes and highlight the importance of exploratory findings. These include how the research questions evolved, how the approach to randomization can help in working with a local agency, reasons for difficulty in maintaining a control group embargo when working in the context of a locally-funded scale-up, issues in avoiding bias in the impact model selection, and what the local agency should make of the results of an RCT.

Keywords: AMSTI, RCT, Alabama Math Science and Technology Initiative, research questions, impact model selection, IES, education agency, locally-funded, scale-up, control group, randomization, exploratory findings

Suggested Citation

Newman, Denis and Jaciw, Andrew, The Unvarnished Report of Conducting an RCT of a Statewide STEM Program in the Process of Scaling Up (October 31, 2014). Available at SSRN: or

Denis Newman (Contact Author)

Empirical Education Inc. ( email )

425 Sherman Ave, #210
Palo Alto, CA 94306
United States


Andrew Jaciw

Empirical Education ( email )

2955 Campus Dr
San Mateo, CA 94403
United States

HOME PAGE: http://

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
PlumX Metrics