Causal Inference in Conjoint Analysis: Logit Models vs. Potential Outcomes

Abstract

Estimating average marginal effects (AME) using logit models is a common approach for assessing the relative importance of attributes in conjoint analysis, but it relies on strong parametric and behavioural assumptions. To address these limitations, recent work in political science has reformulated conjoint analysis within Rubin's potential outcomes framework, introducing the Average Marginal Component Effect (AMCE) as a nonparametric causal estimand. However, AMCE depends on assumptions like preference stability. The recently proposed Conditional Randomization Test (CRT) relaxes these assumptions when testing for causal effects and provides diagnostics to assess the validity of AMCE assumptions. This research note presents the first comprehensive empirical comparison of AME, AMCE, and CRT using four public datasets on pizza and holiday preferences, which vary in task complexity and attribute count. Our findings show that designs with many attributes and tasks are more prone to violating AMCE assumptions, while simpler designs (e.g., 8 attributes and 16 tasks) tend to be more robust, making AMCE a preferred causal estimand in such settings. When AMCE assumptions are not satisfied, AME estimates—if consistent with CRT results—are more credible than AMCE. If AME is insignificant but CRT suggests a significant effect, interaction terms should be explored in the logit model to reconcile the results. Thus, CRT can help test AMCE assumptions and improve MNL specifications when those assumptions are violated, but its reliance on Lasso-based test statistics can sometimes yield implausible results, such as an insignificant CRT effect despite a significant AME.

Keywords: Conjoint AnalysisAverage Marginal Component EffectAverage Marginal EffectPotential OutcomesConditional Randomization Test