An Empirical Method for Harmless Error
D. Alex Winkelman
University of Arizona - James E. Rogers College of Law
David V. Yokum
University of Arizona - James E. Rogers College of Law; University of Arizona - College of Science
Lisette C. Cole
second year law student at Arizona State University Sandra Day O'Connor College of Law
Shelby C. Thompson
Christopher T. Robertson
University of Arizona - James E. Rogers College of Law; Harvard University - Petrie-Flom Center for Health Law Policy, Biotechnology, and Bioethics
November 5, 2015
46 Arizona State Law Journal 1405 (2014)
Arizona Legal Studies Discussion Paper No. 14-19
USC CLASS Research Paper No. CLASS14-31
Trials are often imperfect. When inadmissible evidence is introduced or the jury is incorrectly instructed, judges must determine whether the error was prejudicial or merely harmless. In making that assessment, judges resort to speculation about the counterfactual question of whether the error changed the outcome, compared to the decision of a properly informed and instructed jury. These decisions are likely colored by confirmation and status quo biases, along with “mental contamination” of the error itself. Even when appellate judges perform these analyses accurately, their decisions appear conclusory. Scholars and judges have roundly criticized this doctrine, but no solution has emerged.
We developed and piloted an unbiased and transparent method for making harmless error determinations, using randomized experiments with simulated jurors. To pilot this method on three real cases, we recruited 489 human subjects to participate as mock jurors reviewing trial vignettes that we manipulated into conditions with and without the errors. Subjects were blinded to the purpose of the study and to the first trial’s outcome. By comparing verdict rates in the error and no-error conditions, we estimated whether the error was harmful.
We found a high degree of correspondence between the assessments of real judges and our experimental method, which could be taken as a validation of the method and reassurance that it would not cause a radical change in the rates at which new trials are granted. Still, across the thousands of cases in which harmless error determinations are made each year, the empirical method may be more reliable since it avoids known biases. The transparency of our method may also lend greater legitimacy to harmlessness determinations. If such a method is used as a tool for litigants in real cases, courts will be called upon to establish procedures for taking such evidence and then draw lines to specify how much prejudice is too much, while also being sensitive to the limitations of statistical power. Our study is most useful as proof of concept for a new method to improve harmless error analyses.
Number of Pages in PDF File: 37
Keywords: harmless error, randomized experimentation, confirmation bias, appellate review, harmless error analyses
Date posted: August 7, 2014 ; Last revised: November 6, 2015