On Guessing: An Alternative Adjusted Positive Learning Estimator and Comparing Probability Misspecification with Monte Carlo Simulations

Applied Psychological Measurement, https://doi.org/10.1177/01466216211013905

49 Pages Posted: 1 May 2018 Last revised: 6 Aug 2021

See all articles by Ben Smith

Ben Smith

University of Nebraska at Omaha - Department of Economics

Dustin White

University of Nebraska at Omaha - Department of Economics

Date Written: February 22, 2021

Abstract

Practitioners in the sciences have used the ‘flow’ of knowledge (posttest score minus pretest score) to measure learning in the classroom for the past fifty years. Walstad and Wagner (2016) and Smith and Wagner (2018) moved this practice forward by disaggregating the flow of knowledge and accounting for student guessing. These estimates are sensitive to misspecification of the probability of guessing correct. This work provides guidance to practitioners and researchers facing this problem. We introduce a transformed measure of true positive learning that under some knowable conditions performs better when students’ ability to guess correctly is misspecified and converges to Hake’s (1998) normalized learning gain estimator under certain conditions. We then use simulations to compare the accuracy of two estimation techniques under various violations of the assumptions of those techniques. Using recursive partitioning trees fitted to our simulation results, we provide the practitioner concrete guidance based on a set of yes/no questions.

Keywords: Disaggregated Learning, Gain Measurement, Value-Added Learning, Monte Carlo Simulation

JEL Classification: C63, I21, A20, A22, A23

Suggested Citation

Smith, Ben and White, Dustin, On Guessing: An Alternative Adjusted Positive Learning Estimator and Comparing Probability Misspecification with Monte Carlo Simulations (February 22, 2021). Applied Psychological Measurement, https://doi.org/10.1177/01466216211013905, Available at SSRN: https://ssrn.com/abstract=3168663 or http://dx.doi.org/10.2139/ssrn.3168663

Ben Smith (Contact Author)

University of Nebraska at Omaha - Department of Economics ( email )

College of Business Administration
60th and Dodge Streets
Omaha, NE 68182
United States

HOME PAGE: http://bensresearch.com

Dustin White

University of Nebraska at Omaha - Department of Economics ( email )

College of Business Administration
60th and Dodge Streets
Omaha, NE 68182
United States

HOME PAGE: http://www.dustinrwhite.com

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
100
Abstract Views
1,287
Rank
479,249
PlumX Metrics