Trade-off between validity and efficiency of merging p-values under arbitrary dependence

Abstract

Various methods of combining individual p-values into one p-value are widely used in many areas of statistical applications. We say that a combining method is valid for arbitrary dependence (VAD) if it does not require any assumption on the dependence structure of the p-values, whereas it is valid for some dependence (VSD) if it requires some specific, perhaps realistic but unjustifiable, dependence structures.The trade-off between validity and efficiency of these methods is studied via analyzing the choices of critical values under different dependence assumptions.
We introduce the notions of independence-comonotonicity balance (IC-balance)
and the price for validity. In particular, IC-balanced methods always produce an identical critical value for independent and perfectly positively dependent p-values, a specific type of insensitivity to a family of dependence assumptions. We show that, among two very general classes of merging methods commonly used in practice, the Cauchy combination method and the Simes method are the only IC-balanced ones. Simulation studies and a real data analysis are conducted to analyze the sizes and powers of various combining methods in the presence of weak and strong dependence.

Keywords: Hypothesis testing; multiple hypothesis testing; validity; efficiency