On Guessing: An Alternative Adjusted Positive Learning Estimator and Comparing Probability Misspecification with Monte Carlo Simulations

Abstract

Instructors and researchers have used the ‘flow’ of knowledge (post-test score minus pre-test score) to measure learning in the classroom for the past fifty years. Walstad and Wagner (2016) and Smith and Wagner (2018) move this practice forward by disag- gregating the flow of knowledge and accounting for student guessing. These estimates are sensitive to misspecification of the probability of guessing correct. This work provides guidance to practitioners and researchers facing this problem. We introduce a transformed measure of true positive learning that under some knowable conditions performs better when students’ ability to guess correctly is misspecified. This measure converges to Hake’s (1998) under certain conditions. We then use simulations to compare the accuracy of two estimation techniques under various violations of the assumptions of those techniques. Using recursive partitioning trees fitted to our simulation results, we provide the practitioner concrete guidance based on a set of yes/no questions.

Keywords: Disaggregated Learning, Gain Measurement, Value-Added Learning, Monte Carlo Simulation

JEL Classification: C63, I21, A20, A22, A23