Large-Scale Global and Simultaneous Inference: Estimation and Testing in Very High Dimensions

Posted: 14 Aug 2017

See all articles by Tony Cai

Tony Cai

University of Pennsylvania - Statistics Department

Wenguang Sun

University of Southern California - Marshall School of Business

Date Written: August 2017

Abstract

Due to rapid technological advances, researchers are now able to collect and analyze ever larger data sets. Statistical inference for big data often requires solving thousands or even millions of parallel inference problems simultaneously. This poses significant challenges and calls for new principles, theories, and methodologies. This review provides a selective survey of some recently developed methods and results for large-scale statistical inference, including detection, estimation, and multiple testing. We begin with the global testing problem, where the goal is to detect the existence of sparse signals in a data set, and then move to the problem of estimating the proportion of nonnull effects. Finally, we focus on multiple testing with false discovery rate (FDR) control. The FDR provides a powerful and practical approach to large-scale multiple testing and has been successfully used in a wide range of applications. We discuss several effective data-driven procedures and also present efficient strategies to handle various grouping, hierarchical, and dependency structures in the data.

Suggested Citation

Cai, Tony and Sun, Wenguang, Large-Scale Global and Simultaneous Inference: Estimation and Testing in Very High Dimensions (August 2017). Annual Review of Economics, Vol. 9, pp. 411-439, 2017. Available at SSRN: https://ssrn.com/abstract=3017739 or http://dx.doi.org/10.1146/annurev-economics-063016-104355

Tony Cai (Contact Author)

University of Pennsylvania - Statistics Department ( email )

Wharton School
Philadelphia, PA 19104
United States

Wenguang Sun

University of Southern California - Marshall School of Business ( email )

701 Exposition Blvd
Los Angeles, CA 90089
United States

Register to save articles to
your library

Register

Paper statistics

Abstract Views
74
PlumX Metrics