Too Big to Fail: Larger Samples and False Discoveries

37 Pages Posted: 3 Feb 2009 Last revised: 16 Jun 2011

See all articles by Mingfeng Lin

Mingfeng Lin

Georgia Institute of Technology - Scheller College of Business

Henry C. Lucas

University of Maryland - Robert H. Smith School of Business

Galit Shmueli

Institute of Service Science, National Tsing Hua University, Taiwan

Date Written: June 15, 2011

Abstract

The Internet presents great opportunities for research about information technology, allowing IS researchers to collect very large and rich datasets. It is common to see research papers with tens or even hundreds of thousands of data points, especially when reading about electronic commerce. Large samples are better than smaller samples in that they provide greater statistical power and produce more precise estimates. However, statistical inference using p-values does not scale up to large samples and often leads to erroneous conclusions. We find evidence of an over-reliance on p-values in large sample IS studies in top IS journals and conferences. In this commentary, we focus on interpreting effects of individual independent variables on a dependent variable in regression-type models. We discuss how p-values become deflated with a large sample and illustrate this deflation in analyzing data from over 340,000 digital camera auctions on eBay. The commentary recommends that IS researchers be more conservative in interpreting statistical significance in large sample studies, and instead, interpret results in terms of practical significance. In particular, we suggest that authors of large-sample IS studies report and discuss confidence intervals for independent variables of interest rather than coefficient signs and p-values. We also suggest taking advantage of a large dataset for examining how coefficients and p-values change as sample size increases, and for estimating models on multiple subsamples to further test robustness.

Keywords: large samples, p-values, statistical significance, practical significance

Suggested Citation

Lin, Mingfeng and Lucas, Henry C. and Shmueli, Galit, Too Big to Fail: Larger Samples and False Discoveries (June 15, 2011). Robert H. Smith School Research Paper No. RHS 06-068, Available at SSRN: https://ssrn.com/abstract=1336700 or http://dx.doi.org/10.2139/ssrn.1336700

Mingfeng Lin (Contact Author)

Georgia Institute of Technology - Scheller College of Business ( email )

800 West Peachtree St.
Atlanta, GA 30308
United States

HOME PAGE: http://https://mlin.scheller.gatech.edu/

Henry C. Lucas

University of Maryland - Robert H. Smith School of Business ( email )

College Park, MD 20742-1815
United States

Galit Shmueli

Institute of Service Science, National Tsing Hua University, Taiwan ( email )

Hsinchu, 30013
Taiwan

HOME PAGE: http://www.iss.nthu.edu.tw

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
922
Abstract Views
5,383
Rank
51,705
PlumX Metrics