p-Hacking: Evidence from Two Million Trading Strategies

51 Pages Posted: 14 Aug 2017 Last revised: 16 Aug 2017

Tarun Chordia

Emory University - Department of Finance

Amit Goyal

University of Lausanne; Swiss Finance Institute

Alessio Saretto

University of Texas at Dallas - School of Management - Department of Finance & Managerial Economics

Date Written: August 12, 2017

Abstract

We implement a data mining approach to generate about 2.1 million trading strategies. This large set of strategies serves as a laboratory to evaluate the seriousness of p-hacking and data snooping in finance. We apply multiple hypothesis testing techniques that account for cross-correlations in signals and returns to produce t-statistic thresholds that control the proportion of false discoveries. We find that the difference in rejections rates produced by single and multiple hypothesis testing is such that most rejections of the null of no outperformance under single hypothesis testing are likely false (i.e., we find a very high rate of type I errors). Combining statistical criteria with economic considerations, we find that a remarkably small number of strategies survive our thorough vetting procedure. Even these surviving strategies have no theoretical underpinnings. Overall, p-hacking is a serious problem and, correcting for it, outperforming trading strategies are rare.

Keywords: Hypothesis testing, False discoveries, Trading strategies

JEL Classification: G10, G11, G12

Suggested Citation

Chordia, Tarun and Goyal, Amit and Saretto, Alessio, p-Hacking: Evidence from Two Million Trading Strategies (August 12, 2017). Available at SSRN: https://ssrn.com/abstract=3017677

Tarun Chordia

Emory University - Department of Finance ( email )

Atlanta, GA 30322-2710
United States
404-727-1620 (Phone)
404-727-5238 (Fax)

Amit Goyal (Contact Author)

University of Lausanne ( email )

Lausanne, Vaud CH-1015
Switzerland

Swiss Finance Institute ( email )

c/o University of Geneve
40, Bd du Pont-d'Arve
1211 Geneva, CH-6900
Switzerland

Alessio Saretto

University of Texas at Dallas - School of Management - Department of Finance & Managerial Economics ( email )

800 Campbell Road
SM 31
Richardson, TX 75080
United States
972-883-5907 (Phone)

HOME PAGE: http://www.utdallas.edu/~axs125732

Paper statistics

Downloads
136
Rank
177,206
Abstract Views
495