The Forest or the Trees? Tackling Simpson's Paradox with Classi fication and Regression Trees

36 Pages Posted: 11 Feb 2014

See all articles by Galit Shmueli

Galit Shmueli

Institute of Service Science, National Tsing Hua University, Taiwan

Inbal Yahav

Bar-Ilan University - Graduate School of Business Administration

Date Written: February 9, 2014

Abstract

Prediction and variable selection are major uses of data mining algorithms but they are rarely the focus in social science research, where the main objective is causal explanation. Ideal causal modeling is based on randomized experiments, but because experiments are often impossible, unethical or expensive to perform, social science research often relies on observational data for studying causality. A major challenge is to infer causality from such data. This paper uses the predictive tool of Classification and Regression Trees for detecting Simpson's paradox, which is related to causal inference. We introduce a new tree approach for detecting potential paradoxes in data that have either a few or a large number of potential confounding variables. The approach relies on the tree structure and the location of the cause vs. the confounders in the tree. We discuss theoretical and computational aspects of the approach and illustrate it using several real applications

Keywords: CART, causality, data mining, conditional-inference trees, decision making, aggregation, variable selection

Suggested Citation

Shmueli, Galit and Yahav, Inbal, The Forest or the Trees? Tackling Simpson's Paradox with Classi fication and Regression Trees (February 9, 2014). Indian School of Business Research Paper Series. Available at SSRN: https://ssrn.com/abstract=2392953 or http://dx.doi.org/10.2139/ssrn.2392953

Galit Shmueli (Contact Author)

Institute of Service Science, National Tsing Hua University, Taiwan ( email )

Hsinchu, 30013
Taiwan

HOME PAGE: http://www.iss.nthu.edu.tw

Inbal Yahav

Bar-Ilan University - Graduate School of Business Administration ( email )

Ramat Gan
Israel
97235318913 (Phone)

HOME PAGE: http://faculty.biu.ac.il/~yahavi1

Register to save articles to
your library

Register

Paper statistics

Downloads
288
Abstract Views
1,051
rank
108,837
PlumX Metrics