Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli

Perspectives on Psychological Science, Forthcoming

28 Pages Posted: 14 Jun 2014 Last revised: 22 Sep 2014

See all articles by Jacob Westfall

Jacob Westfall

University of Colorado Boulder

Charles M. Judd

University of Colorado at Boulder - Department of Psychology

David A Kenny

University of Connecticut

Date Written: September 21, 2014

Abstract

In a direct replication, the typical goal is to reproduce a prior experimental result with a new, but comparable sample of participants in a high-powered replication study. Often in psychology, research to be replicated involves a sample of participants responding to a sample of stimuli. In replicating such studies, we argue that the same criteria should be used in sampling stimuli as are used in sampling participants. Namely, a new, but comparable sample of stimuli should be used to ensure that the original results are not due to idiosyncrasies of the original stimulus sample, and the stimulus sample must often be enlarged to ensure high statistical power. In support of the latter point, we discuss the fact that in experiments involving samples of stimuli, statistical power typically does not approach unity as the number of participants goes to infinity. As an example of the importance of sampling new stimuli, we discuss the bygone literature on the risky shift phenomenon, which was almost entirely based on a single stimulus sample that was later discovered to be highly unrepresentative. We discuss the use of both resampled and expanded stimulus sets, that is, stimulus samples that include the original stimuli plus new stimuli.

Keywords: replication, stimulus sampling, statistical power

Suggested Citation

Westfall, Jacob and Judd, Charles M. and Kenny, David A, Replicating Studies in Which Samples of Participants Respond to Samples of Stimuli (September 21, 2014). Perspectives on Psychological Science, Forthcoming. Available at SSRN: https://ssrn.com/abstract=2449567 or http://dx.doi.org/10.2139/ssrn.2449567

Jacob Westfall (Contact Author)

University of Colorado Boulder ( email )

1070 Edinboro Drive
Boulder, CO 80309
United States

Charles M. Judd

University of Colorado at Boulder - Department of Psychology ( email )

Boulder, 80309
United States

David A Kenny

University of Connecticut ( email )

Storrs, CT 06269-1063
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
137
rank
204,473
Abstract Views
1,192
PlumX Metrics