lancet-header

Preprints with The Lancet is part of SSRN´s First Look, a place where journals identify content of interest prior to publication. Authors have opted in at submission to The Lancet family of journals to post their preprints on Preprints with The Lancet. The usual SSRN checks and a Lancet-specific check for appropriateness and transparency have been applied. Preprints available here are not Lancet publications or necessarily under review with a Lancet journal. These preprints are early stage research papers that have not been peer-reviewed. The findings should not be used for clinical or public health decision making and should not be presented to a lay audience without highlighting that they are preliminary and have not been peer-reviewed. For more information on this collaboration, see the comments published in The Lancet about the trial period, and our decision to make this a permanent offering, or visit The Lancet´s FAQ page, and for any feedback please contact preprints@lancet.com.

Development and Validation of an Early Warning Model for Hospitalized COVID-19 Patients: A Multi­Center Retrospective Cohort Study

29 Pages Posted: 10 Feb 2022

See all articles by Jim Smit

Jim Smit

Erasmus University Rotterdam (EUR) - Department of Intensive Care

Jesse H. Krijthe

Pakistan Institute of Engineering and Applied Sciences (PIEAS) - Pattern Recognition Lab

Andrei N. Tintu

Erasmus University Rotterdam (EUR) - Department of Biochemistry

Hendrik Endeman

Erasmus University Rotterdam (EUR) - Department of Intensive Care

Jeroen Ludikhuize

Hospital de Porto Alegre - Prone Teaching and Research Group

Michel E. Van Genderen

Erasmus University Rotterdam (EUR) - Department of Intensive Care

Shermarke Hassan

Leiden University - Department of Clinical Epidemiology

Rachida el Moussaoui

Maasstad Hospital - Department of Internal Medicine

Peter E. Westerweel

Albert Schweitzer Hospital

Robbert J. Goekoop

Leiden University - Department of Rheumatology

G. Waverijn

Maasstad Hospital

Tim Verheijen

Leiden University - Department of Rheumatology

J. G. den Hollander

Maasstad Hospital - Department of Internal Medicine

Mark De Boer

Leiden University - Department of Infectious Diseases

Diederik Gommers

Erasmus University Rotterdam (EUR) - Department of Intensive Care

R. van der Vlies

Albert Schweitzer Hospital

Mark Schellings

Maasstad Hospital

Regina A. Carels

Komfo Anokye Teaching Hospital - Department of Internal Medicine

Cees van Nieuwkoop

Komfo Anokye Teaching Hospital - Department of Internal Medicine

M.S. Arbous

Leiden University - Department of Intensive Care

Jasper van Bommel

Erasmus University Rotterdam (EUR) - Department of Intensive Care

Rachel Knevel

Leiden University - Department of Rheumatology

Yolanda B. de Rijke

Erasmus University Rotterdam (EUR) - Department of Biochemistry

Marcel Reinders

Leiden University - Leiden Computational Biology Center

More...

Abstract

Background: Timely identification of deteriorating COVID­-19 patients is needed to guide changes in clinical management and admission to intensive care units (ICUs). There is significant concern that widely used early warning scores (EWSs) underestimate illness severity in COVID­19 patients and therefore, we developed an early warning model specifically for COVID­-19 patients.

Methods: We collected electronic medical record data to extract predictors and used these to fit a random forest model. To simulate the situation in which the model would have been developed after the first COVID­-19 ‘wave’ in the Netherlands and implemented during the second wave, we performed a temporal validation by splitting all included patients into groups admitted before and after August 1, 2020. Furthermore, we propose a method for dynamic model updating to retain the model’s predictive performance over time. We evaluated model discrimination and calibration, performed a decision curve analysis, and quantified the importance of predictors using SHapley Additive exPlanations values.

Findings: We included 3,514 COVID­-19 patient admissions from six Dutch hospitals between February 2020 until May 2021, and included a total of 18 predictors for model fitting. The model showed a higher discriminative performance in terms of partial area under the receiver operating characteristic curve (0∙82 [0∙80 to 0∙84]) compared to the National Early Warning Score (0∙72 [0∙69 to 0∙74]) and the Modified Early Warning Score (0∙67 [0∙65 to 0∙69]), a greater net benefit over a range of clinically relevant model thresholds, and good calibration (intercept=0∙03 [­0∙09 to 0∙14], slope=0∙79 [0∙73 to 0∙86]).

Interpretation: This study shows the potential benefit of moving from early warning models for the general inpatient population to models for specific patient groups. The COVID­-19 ­specific early warning model and is available online at https://github.com/jimmsmit/COVID­19_EWS and we encourage others to further validate this model independently.

Funding Information: No external funding.

Declaration of Interests: The authors declare that they have no conflict of interest.

Ethics Approval Statement: Ethical approval was given by the local research ethics committee.

Keywords: Early warning, COVID-19, machine learning, medical prediction model, artificial intelligence

Suggested Citation

Smit, Jim and Krijthe, Jesse H. and Tintu, Andrei N. and Endeman, Hendrik and Ludikhuize, Jeroen and Van Genderen, Michel E. and Hassan, Shermarke and el Moussaoui, Rachida and Westerweel, Peter E. and Goekoop, Robbert J. and Waverijn, G. and Verheijen, Tim and den Hollander, J. G. and De Boer, Mark and Gommers, Diederik and van der Vlies, R. and Schellings, Mark and Carels, Regina A. and van Nieuwkoop, Cees and Arbous, M.S. and van Bommel, Jasper and Knevel, Rachel and de Rijke, Yolanda B. and Reinders, Marcel, Development and Validation of an Early Warning Model for Hospitalized COVID-19 Patients: A Multi­Center Retrospective Cohort Study. Available at SSRN: https://ssrn.com/abstract=4031569 or http://dx.doi.org/10.2139/ssrn.4031569

Jim Smit (Contact Author)

Erasmus University Rotterdam (EUR) - Department of Intensive Care ( email )

Rotterdam
Netherlands

Jesse H. Krijthe

Pakistan Institute of Engineering and Applied Sciences (PIEAS) - Pattern Recognition Lab ( email )

Islamabad
Pakistan

Andrei N. Tintu

Erasmus University Rotterdam (EUR) - Department of Biochemistry ( email )

United States

Hendrik Endeman

Erasmus University Rotterdam (EUR) - Department of Intensive Care ( email )

Rotterdam
Netherlands

Jeroen Ludikhuize

Hospital de Porto Alegre - Prone Teaching and Research Group ( email )

Brazil

Michel E. Van Genderen

Erasmus University Rotterdam (EUR) - Department of Intensive Care ( email )

Rotterdam
Netherlands

Shermarke Hassan

Leiden University - Department of Clinical Epidemiology ( email )

Netherlands

Rachida El Moussaoui

Maasstad Hospital - Department of Internal Medicine ( email )

Rotterdam
Netherlands

Peter E. Westerweel

Albert Schweitzer Hospital ( email )

Netherlands

Robbert J. Goekoop

Leiden University - Department of Rheumatology ( email )

Leiden, 2333 ZA
Netherlands

G. Waverijn

Maasstad Hospital ( email )

Maasstadweg 21
Rotterdam, South Holland 3079 DZ
Netherlands

Tim Verheijen

Leiden University - Department of Rheumatology ( email )

Leiden, 2333 ZA
Netherlands

J. G. Den Hollander

Maasstad Hospital - Department of Internal Medicine ( email )

Rotterdam
Netherlands

Mark De Boer

Leiden University - Department of Infectious Diseases ( email )

Netherlands

Diederik Gommers

Erasmus University Rotterdam (EUR) - Department of Intensive Care ( email )

Rotterdam
Netherlands

R. Van der Vlies

Albert Schweitzer Hospital ( email )

Netherlands

Mark Schellings

Maasstad Hospital ( email )

Maasstadweg 21
Rotterdam, South Holland 3079 DZ
Netherlands

Regina A. Carels

Komfo Anokye Teaching Hospital - Department of Internal Medicine ( email )

Kumasi
Ghana

Cees Van Nieuwkoop

Komfo Anokye Teaching Hospital - Department of Internal Medicine ( email )

Kumasi
Ghana

M.S. Arbous

Leiden University - Department of Intensive Care ( email )

Postbus 9500
Leiden, Zuid Holland 2300 RA
Netherlands

Jasper Van Bommel

Erasmus University Rotterdam (EUR) - Department of Intensive Care ( email )

Rotterdam
Netherlands

Rachel Knevel

Leiden University - Department of Rheumatology ( email )

Leiden, 2333 ZA
Netherlands

Yolanda B. De Rijke

Erasmus University Rotterdam (EUR) - Department of Biochemistry ( email )

United States

Marcel Reinders

Leiden University - Leiden Computational Biology Center ( email )

Netherlands