lancet-header
Preprints with The Lancet is part of SSRN´s First Look, a place where journals and other research experts identify content of interest prior to publication. These preprint papers are not peer-reviewed and are posted here as part of a 12-month trial. Authors have either opted in at submission to The Lancet family of journals to post their preprints on Preprints with The Lancet, or submitted directly via SSRN. The usual SSRN checks and a Lancet-specific check for appropriateness and transparency have been applied. These papers should not be used for clinical decision making or reporting of research to a lay audience without indicating that this is preliminary research that has not been peer-reviewed. For more information see the Comment published in The Lancet, or visit The Lancet´s FAQ page, and for any feedback please contact preprints@lancet.com

Feasibility of Automated Deep Learning Design for Medical Image Classification by Healthcare Professionals with Limited Coding Experience

37 Pages Posted: 14 Jun 2019

See all articles by Livia Faes

Livia Faes

Cantonal Hospital of Lucerne

Siegfried K. Wagner

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

Dun Jack Fu

Government of the United Kingdom - Moorfields Eye Hospital

Xiaoxuan Liu

Government of the United Kingdom - Moorfields Eye Hospital

Edward Korot

Beaumont Eye Institute

Joseph R. E. Ledsam

Google DeepMind

Trevor Back

Google DeepMind

Reena Chopra

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

Nikolas Pontikos

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

Christoph Kern

Government of the United Kingdom - Moorfields Eye Hospital

Gabriella Moraes

Government of the United Kingdom - Moorfields Eye Hospital

Martin K. Schmid

Cantonal Hospital of Lucerne

Dawn Sim

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

Konstantinos Balaskas

Government of the United Kingdom - Moorfields Eye Hospital

Lucas M. Bachmann

Medignition, Inc

Alastair Denniston

University Hospitals Birmingham, Department of Ophthalmology; University of Birmingham - Academic Unit of Ophthalmology; University of Birmingham - Centre for Patient Reported Outcome Research; Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

Pearse Keane

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology; Government of the United Kingdom - Medical Retina Department

More...

Abstract

Background: Deep learning has huge potential to transform healthcare however significant expertise is required to train such models. In this study, we therefore sought to evaluate the use of automated deep learning software to develop medical image diagnostic classifiers by healthcare professionals with limited coding - and no deep learning - expertise.

Methods: We used five publicly available open-source datasets: (i) retinal fundus images (MESSIDOR); (ii) optical coherence tomography (OCT) images (Guangzhou Medical University/Shiley Eye Institute, Version 3); (iii) images of skin lesions (Human against Machine (HAM)10000) and (iv) both paediatric and adult chest X-ray (CXR) images (Guangzhou Medical University/Shiley Eye Institute, Version 3 and the National Institute of Health (NIH)14 dataset respectively) to separately feed into a neural architecture search framework that automatically developed a deep learning architecture to classify common diseases. Sensitivity (recall), specificity and positive predictive value (precision) were used to evaluate the diagnostic properties of the models. The discriminative performance was assessed using the area under the precision recall curve (AUPRC). In the case of the deep learning model developed on a subset of the HAM10000 dataset, we performed external validation using the Edinburgh Dermofit Library dataset.

Findings: Diagnostic properties and discriminative performance from internal validations were high in the binary classification tasks (range: sensitivity of 73·3-97·0%, specificity of 67-100% and AUPRC of 0·87-1). In the multiple classification tasks, the diagnostic properties ranged from 38-100% for sensitivity and 67-100% for specificity. The discriminative performance in terms of AUPRC ranged from 0·57 to 1 in the five automated deep learning models. In an external validation using the Edinburgh Dermofit Library dataset, the automated deep learning model showed an AUPRC of 0·47, with a sensitivity of 49% and a positive predictive value of 52%. The quality of the open-access datasets used in this study (including the lack of information about patient flow and demographics) and the absence of measurement for precision, such as confidence intervals, constituted the major limitation of this study.

Interpretation: All models, except for the automated deep learning model trained on the multi-label classification task of the NIH CXR14 dataset, showed comparable discriminative performance and diagnostic properties to state-of-the-art performing deep learning algorithms. The performance in the external validation study was low. The availability of automated deep learning may become a cornerstone for the democratization of sophisticated algorithmic modelling in healthcare as it allows the derivation of classification models without requiring a deep understanding of the mathematical, statistical and programming principles. Future studies should compare several application programming interfaces on thoroughly curated datasets.

Funding Statement: National Institute for Health Research, United Kingdom. PAK is supported by an NIHR Clinician Scientist Award (NIHR-CS--2014-14-023). The research was also supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology.

Declaration of Interests: The authors have no conflict of interest to declare. JL and TB are employees of DeepMind Technologies, a subsidiary of Alphabet Inc. RC is an intern at DeepMind. PK is an external consultant for DeepMind.

Ethics Approval Statement: Not required.

Keywords: Deep learning, Automated deep learning, Neural architecture search

Suggested Citation

Faes, Livia and Wagner, Siegfried K. and Fu, Dun Jack and Liu, Xiaoxuan and Korot, Edward and Ledsam, Joseph R. E. and Back, Trevor and Chopra, Reena and Pontikos, Nikolas and Kern, Christoph and Moraes, Gabriella and Schmid, Martin K. and Sim, Dawn and Balaskas, Konstantinos and Bachmann, Lucas M. and Denniston, Alastair and Keane, Pearse, Feasibility of Automated Deep Learning Design for Medical Image Classification by Healthcare Professionals with Limited Coding Experience (June 10, 2019). Available at SSRN: https://ssrn.com/abstract=3402015

Livia Faes

Cantonal Hospital of Lucerne

Lucerne
Switzerland

Siegfried K. Wagner

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

London
United Kingdom

Dun Jack Fu

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Xiaoxuan Liu

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Edward Korot

Beaumont Eye Institute

United States

Joseph R. E. Ledsam

Google DeepMind

Trevor Back

Google DeepMind

Reena Chopra

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

London
United Kingdom

Nikolas Pontikos

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

London
United Kingdom

Christoph Kern

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Gabriella Moraes

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Martin K. Schmid

Cantonal Hospital of Lucerne

Lucerne
Switzerland

Dawn Sim

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology

London
United Kingdom

Konstantinos Balaskas

Government of the United Kingdom - Moorfields Eye Hospital

London
United Kingdom

Lucas M. Bachmann

Medignition, Inc

Zurich
Switzerland

Alastair Denniston

University Hospitals Birmingham, Department of Ophthalmology ( email )

Birmingham
United Kingdom

University of Birmingham - Academic Unit of Ophthalmology ( email )

Birmingham
United Kingdom

University of Birmingham - Centre for Patient Reported Outcome Research ( email )

Birmingham
United Kingdom

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology ( email )

London
United Kingdom

Pearse Keane (Contact Author)

Government of the United Kingdom - NIHR Biomedical Research Centre for Ophthalmology ( email )

London
United Kingdom

Government of the United Kingdom - Medical Retina Department ( email )

United Kingdom

Click here to go to TheLancet.com

Go to TheLancet.com

Paper statistics

Abstract Views
1,211
Downloads
63