Communicating the Robustness of Findings of COVID-19 Studies

The COVID-19 pandemic is forcing researchers, clinicians, and policymakers to accelerate the evaluation of treatments and vaccines. Critical to these evaluations is the ability to characterize the uncertainty of inferences in clear terms accessible to a broad set of stakeholders with varying statistical backgrounds. Here we quantify the uncertainty about inferences from Randomized Controlled Trials (RCTs) by quantifying how many patients would have to have experienced different outcomes to change the inference. For example, the inference of a positive effect of Hydroxychloroquine (HCQ) on pneumonia from an open label RCT would be overturned if one of the treatment cases characterized as improved had instead been characterized as unchanged or exacerbated. We generalize the technique to apply to thresholds defined by any effect size. We also apply the analysis to an inference of no effect of Remdesivir on mortality and an historical example of anti-hypertensive treatments on stroke. Quantifying the robustness of inferences in terms of patient outcomes supports a more precise dialogue among clinicians, researchers, policymakers, and the general public.

Note: Funding: None to declare

Declaration of Interest: None to declare

Keywords: COVID-19 treatments, causal inference, robustness, policy

JEL Classification: C12, C19

Suggested Citation: Suggested Citation

Frank, Ken and Lin, Qinyun and Maroulis, Spiro and Mueller, Anna S. and Xu, Ran and Rosenberg, Joshua M. and Hayter, Christopher S. and Mahmoud, Ramy A. and Kolak, Maryina and Dietz, Thomas and Zhang, Lixin, Communicating the Robustness of Findings of COVID-19 Studies (May 22, 2020). Available at SSRN: https://ssrn.com/abstract=3607967 or http://dx.doi.org/10.2139/ssrn.3607967