lancet-header
Preprints with The Lancet is part of SSRN´s First Look, a place where journals and other research experts identify content of interest prior to publication. These preprint papers are not peer-reviewed and are posted here as part of a 12-month trial. Authors have either opted in at submission to The Lancet family of journals to post their preprints on Preprints with The Lancet, or submitted directly via SSRN. The usual SSRN checks and a Lancet-specific check for appropriateness and transparency have been applied. These papers should not be used for clinical decision making or reporting of research to a lay audience without indicating that this is preliminary research that has not been peer-reviewed. For more information see the Comment published in The Lancet, or visit The Lancet´s FAQ page, and for any feedback please contact preprints@lancet.com

Deep Learning Under Scrutiny: Performance Against Health Care Professionals in Detecting Diseases from Medical Imaging - Systematic Review and Meta-Analysis

33 Pages Posted: 13 May 2019

See all articles by Livia Faes

Livia Faes

Cantonal Hospital of Lucerne

Xiaoxuan Liu

Government of the United Kingdom - Moorfields Eye Hospital

Aditya Kale

University Hospitals Birmingham, Department of Ophthalmology

Alice Bruynseels

University Hospitals Birmingham, Department of Ophthalmology

Mohith Shamdas

University of Birmingham - Academic Unit of Ophthalmology

Gabriella Moraes

Government of the United Kingdom - Moorfields Eye Hospital

Dun Jack Fu

Government of the United Kingdom - Moorfields Eye Hospital

Siegfried K. Wagner

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

Christoph Kern

Government of the United Kingdom - Moorfields Eye Hospital

Joseph R. E. Ledsam

Google DeepMind

Martin K. Schmid

Cantonal Hospital of Lucerne

Eric J. Topol

The Scripps Research Institute - The Scripps Research Institute California

Konstantinos Balaskas

Government of the United Kingdom - Moorfields Eye Hospital

Lucas M. Bachmann

Medignition, Inc

Pearse A. Keane

Government of the United Kingdom - Moorfields Eye Hospital

Alastair Denniston

University Hospitals Birmingham, Department of Ophthalmology; University of Birmingham - Academic Unit of Ophthalmology; University of Birmingham - Centre for Patient Reported Outcome Research; Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

More...

Abstract

Background: Deep learning offers considerable promise for medical diagnostics. In this review, we evaluated the diagnostic accuracy of deep learning (DL) algorithms versus health care professionals (HCPs) in classifying diseases from medical imaging.

Methods: We searched (Pre-)Medline, Embase, Science Citation Index, Conference Proceedings Citation Index, and arXiv from 01 January 2012 until 31 May 2018. Studies comparing the diagnostic performance of DL models and HCPs, for any pre-specified condition based on medical imaging material, were included. We extracted binary diagnostic accuracy data and constructed contingency tables at the reported thresholds to derive the outcomes of interest: sensitivity and specificity. Studies undertaking an out-of-sample validation were included in a meta-analysis.

Results: 24 studies, from a starting number of 19889, compared DL models with HCPs. 22 studies provided enough data to construct contingency tables, enabling calculation of test accuracy. The mean sensitivity for DL models was 78% (range 13 - 100%), and mean specificity was 86% (range 51 - 100%). An out-of-sample external validation was performed by 5 studies and were therefore included in the meta-analysis. We found a pooled sensitivity of 86% (95% CI: 84 - 88%) for DL models and 93% (95% CI: 87 - 97%) for HCPs, and a pooled specificity of 88% (95% CI: 84 - 92%) for DL models and 87% (95% CI: 84 - 89%) for HCPs.

Conclusion: Our review found the diagnostic performance of deep learning models to be similar to health care professionals. A major finding was the poor reporting and potential biases arising from study design that limited reliable interpretation of the reported diagnostic accuracy. New reporting standards which address specific challenges of deep learning could improve future studies, enabling greater confidence in the results of future evaluations of this promising technology.

Funding Statement: The authors state: "None"

Declaration of Interests: All authors have completed the ICMJE uniform disclosure form online (available on request from the corresponding author) and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Ethics Approval Statement: The authors state: "Not required." The authors utilized PRISMA and MOOSE protocols.

Keywords: Artificial intelligence, machine learning, deep learning, medical imaging, diagnosis, classification, diagnostic accuracy, sensitivity, specificity, systematic review, meta-analysis

Suggested Citation

Faes, Livia and Liu, Xiaoxuan and Kale, Aditya and Bruynseels, Alice and Shamdas, Mohith and Moraes, Gabriella and Fu, Dun Jack and Wagner, Siegfried K. and Kern, Christoph and Ledsam, Joseph R. E. and Schmid, Martin K. and Topol, Eric J. and Balaskas, Konstantinos and Bachmann, Lucas M. and Keane, Pearse A. and Denniston, Alastair, Deep Learning Under Scrutiny: Performance Against Health Care Professionals in Detecting Diseases from Medical Imaging - Systematic Review and Meta-Analysis (May 8, 2019). Available at SSRN: https://ssrn.com/abstract=3384923

Livia Faes

Cantonal Hospital of Lucerne

Lucerne
Switzerland

Xiaoxuan Liu

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Aditya Kale

University Hospitals Birmingham, Department of Ophthalmology

Birmingham
United Kingdom

Alice Bruynseels

University Hospitals Birmingham, Department of Ophthalmology

Birmingham
United Kingdom

Mohith Shamdas

University of Birmingham - Academic Unit of Ophthalmology

Birmingham
United Kingdom

Gabriella Moraes

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Dun Jack Fu

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Siegfried K. Wagner

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

London
United Kingdom

Christoph Kern

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Joseph R. E. Ledsam

Google DeepMind

Martin K. Schmid

Cantonal Hospital of Lucerne

Lucerne
Switzerland

Eric J. Topol

The Scripps Research Institute - The Scripps Research Institute California

La Jolla, CA 92037
United States

Konstantinos Balaskas

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Lucas M. Bachmann

Medignition, Inc

Zurich
Switzerland

Pearse A. Keane

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Alastair Denniston (Contact Author)

University Hospitals Birmingham, Department of Ophthalmology ( email )

Birmingham
United Kingdom

University of Birmingham - Academic Unit of Ophthalmology ( email )

Birmingham
United Kingdom

University of Birmingham - Centre for Patient Reported Outcome Research ( email )

Birmingham
United Kingdom

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology ( email )

London
United Kingdom

Click here to go to TheLancet.com

Go to TheLancet.com

Paper statistics

Abstract Views
1,588
Downloads
272