Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI)

Abstract

The Alabama State Department of Education (ALSDE) developed a statewide initiative to improve mathematics and science teaching and student achievement in kindergarten through grade 12 (K-12). The Alabama Math, Science, and Technology Initiative (AMSTI) is a two-year intervention intended to better align classroom practices with national and statewide teaching standards — and ultimately to improve student achievement — by providing PD, access to materials and technology, and in-school support for teachers.

AMSTI, a schoolwide intervention, was introduced in 20 schools in 2002. Each year since then, the state has rolled out the program to additional schools within its 11 regions. By 2009, about 40% of the state’s 1,518 schools were designated as AMSTI schools. Funding for the program from the state legislature was $46 million in 2009.

Given the policy relevance and level of investment in AMSTI, the Regional Educational Laboratory Southeast mounted a longitudinal, cluster randomized controlled trial to determine the effectiveness of AMSTI in grades 4-8, as implemented in five regions in the state. Previous evaluations of the program’s effects on students in K-12 did not use randomized controlled trials. The most recent evaluation (Miron & Maxwell 2007) reported that students in grade 5 in AMSTI schools outperformed students in non-AMSTI schools in math, science, and reading and students in grade 4 in AMSTI schools outperformed their counterparts in non-AMSTI schools in reading only. These evaluations used a study design that compared school-level test scores of AMSTI schools with non-AMSTI schools in the same district but did not establish preintervention comparability. This study’s RCT design improves on previous evaluations because it eliminates selection bias and establishes the preintervention comparability of the two groups.

The effect of AMSTI on student achievement in math after one year, as measured by end-of-the-year scores on the Stanford Achievement Test Tenth Edition (SAT 10) math problem solving assessment of students in grades 4–8, was 2.06 scale score units. The difference of 0.05 standard deviation in favor of AMSTI schools is equivalent to a gain of 2 percentile points on the SAT 10 math problem solving assessment for the average control group student had the student received AMSTI. The 0.05 standard deviation is statistically significant but smaller than the effect the research team believed would be detectable by the experiment as designed. Whether this size effect is educationally important is an open question. It may be useful to convert this effect into a more policy-relevant metric — additional student progress measured in days of instruction. In these terms, the average estimated effect of AMSTI was equivalent to 28 days of additional student progress over students receiving conventional math instruction. The effect of AMSTI on student achievement in science, as measured by end-of-the-year scores on the SAT 10 science assessment, required only in grades 5 and 7, was not statistically significant after one year.

AMSTI had a positive and statistically significant effect on classroom practices in math and science after one year. Based on multiple surveys in which teachers reported the number of minutes of active learning strategies used during the previous 10-day period, AMSTI math teachers averaged 49.83 more minutes, and AMSTI science teachers averaged 40.07 more minutes than control teachers. These estimated effects are equivalent to 0.47 standard deviation in math and 0.32 standard deviation in science. Although teachers in both the AMSTI and control groups reported using active learning instructional strategies, teachers in AMSTI schools reported spending more time engaged in this type of instruction.

The exploratory investigation of the two-year effect of AMSTI on student achievement in science also found a statistically significant result, with a magnitude of 4.00 scale score units. This effect represents a difference of 0.13 standard deviation in favor of AMSTI schools, equivalent to a gain of 5 percentile points for the average control group student had the student received AMSTI for two years.

The effect of AMSTI on teacher-reported content knowledge after one year was not statistically significant in either math or science. AMSTI did have a positive and statistically significant effect on student engagement after one year, measured on a 5-point scale ranging from “not engaged” to “fully engaged.” AMSTI teachers were more likely than control teachers to rate their students as achieving higher levels of engagement.

An exploration of the differential effects of AMSTI on student achievement for subgroups of students found no statistically significant differential effects on student achievement in mathematics or science based on racial/ethnic minority status, eligibility for free or reduced-price lunch, gender, or pretest level. In reading, however, AMSTI had a statistically significant differential effect for minority and White students of 3.04 scale score points (p < .001). This difference can be translated into days of student progress, where progress is measured as the average gain in test scores over the course of the school year by the control group using conventional reading instruction. In this metric, White students in AMSTI made an estimated 52 more days of progress than minority students in AMSTI. The effect of AMSTI on reading achievement for minority students was not statistically significant (p = .294); for White students, there was a statistically significant positive effect of AMSTI on reading achievement (p < .001).

Keywords: Empirical Education, AMSTI, ALSDE, Regional Educational Laboratory Southeast, student achievement, subexperiments, RCT, math problem solving, cluster randomized controlled trial, exploratory investigation, statistically significant positive effect