Measuring Test Measurement Error: A General Approach

59 Pages Posted: 21 Apr 2012

See all articles by Donald Boyd

Donald Boyd

SUNY University at Albany

Hamilton Lankford

SUNY at Albany - College of Arts and Sciences

Susanna Loeb

Stanford University; National Bureau of Economic Research (NBER)

James Wyckoff

University of Virginia

Date Written: April 2012

Abstract

Test-based accountability including value-added assessments and experimental and quasi-experimental research in education rely on achievement tests to measure student skills and knowledge. Yet we know little regarding important properties of these tests, an important example being the extent of test measurement error and its implications for educational policy and practice. While test vendors provide estimates of split-test reliability, these measures do not account for potentially important day-to-day differences in student performance.We show there is a credible, low-cost approach for estimating the total test measurement error that can be applied when one or more cohorts of students take three or more tests in the subject of interest (e.g., state assessments in three consecutive grades). Our method generalizes the test-retest framework allowing for either growth or decay in knowledge and skills between tests as well as variation in the degree of measurement error across tests. The approach maintains relatively unrestrictive, testable assumptions regarding the structure of student achievement growth. Estimation only requires descriptive statistics (e.g., correlations) for the tests. When student-level test-score data are available, the extent and pattern of measurement error heteroskedasticity also can be estimated. Utilizing math and ELA test data from New York City, we estimate the overall extent of test measurement error is more than twice as large as that reported by the test vendor and demonstrate how using estimates of the total measurement error and the degree of heteroskedasticity along with observed scores can yield meaningful improvements in the precision of student achievement and achievement-gain estimates.

Suggested Citation

Boyd, Donald and Lankford, Hamilton and Loeb, Susanna and Wyckoff, James, Measuring Test Measurement Error: A General Approach (April 2012). NBER Working Paper No. w18010, Available at SSRN: https://ssrn.com/abstract=2043468

Donald Boyd

SUNY University at Albany ( email )

135 Western Ave
Building, Room 109
Albany, NY 12222
United States

Hamilton Lankford

SUNY at Albany - College of Arts and Sciences ( email )

1400 Washington Avenue
Albany, NY 12222
United States

Susanna Loeb

Stanford University ( email )

School of Education 402P CERAS, 520 Galvez Mall
Stanford, CA 94305
United States
650-725-4262 (Phone)

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

James Wyckoff

University of Virginia ( email )

1400 University Ave
Charlottesville, VA 22903
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
14
Abstract Views
296
PlumX Metrics