Human Forest vs. Random Forest in Time-Sensitive COVID-19 Clinical Trial Prediction

Abstract

How do we combine historical data and human insights to predict complex outcomes, such as the timely advancement of clinical trials? We report the methods and results of the first study comparing the new Human Forest (HF) method with a control crowdsourcing method and a machine model, a time-specific random survival forest (RSF) model. We provide the first description of the Human Forest method, which enables forecasters to define custom reference classes, query a historical database and review base rates specific to their selections. These base rates, and adjusted probabilistic estimates, are then aggregated. Forecasters receive proper scoring feedback and accuracy incentives. HF works in tandem with a new algorithm, Most Popular Selections, which provides a collective intelligence approach for addressing the long-standing reference class problem, by crowdsourcing and aggregating reference class selections. The empirical validation spans two 6-month tournaments that focus on trial phase transition for vaccines and treatments for COVID-19 and other infectious diseases. The tournaments include 60 forecasting questions. Results show that HF significantly outperforms the RSF model, registering mean Brier scores between 36% and 52% lower than those earned by the RSF model. HF and Control Polls exhibit approximately equivalent performance. Including Human Forest- derived base rate estimates at the aggregation stage improves overall performance. MPS-generated base-rate estimates exhibit performance between that of RSF and human crowdsourcing methods. Our results show that human forecaster crowds with appropriate elicitation and aggregation tools can outperform statistical models. Interactive access to data through HF appears either beneficial or neutral to forecasting performance, even in a setting where new developments deviate from historical patterns.

Keywords: Forecasting, Crowdsourcing, Machine Learning, Clinical Development

JEL Classification: C45, I10, C44