lancet-header

Preprints with The Lancet is part of SSRN´s First Look, a place where journals identify content of interest prior to publication. Authors have opted in at submission to The Lancet family of journals to post their preprints on Preprints with The Lancet. The usual SSRN checks and a Lancet-specific check for appropriateness and transparency have been applied. Preprints available here are not Lancet publications or necessarily under review with a Lancet journal. These preprints are early stage research papers that have not been peer-reviewed. The findings should not be used for clinical or public health decision making and should not be presented to a lay audience without highlighting that they are preliminary and have not been peer-reviewed. For more information on this collaboration, see the comments published in The Lancet about the trial period, and our decision to make this a permanent offering, or visit The Lancet´s FAQ page, and for any feedback please contact preprints@lancet.com.

Evaluating Diagnostic Accuracy of a New Artificial-Intelligence Driven Diagnostic Support Tool

17 Pages Posted: 1 Jul 2021

See all articles by Niv Ben-Shabat

Niv Ben-Shabat

Tel Aviv University

Ariel Sloma

Tel-Aviv University - Sackler Faculty of Medicine

Tomer Weizman

Technion-Israel Institute of Technology - Faculty of Medicine

David Kiderman

Hebrew University of Jerusalem - Hadassah Medical Center

Howard Amital

Tel-Aviv University - Sackler Faculty of Medicine

More...

Abstract

Background: Diagnostic decision support systems (DDSS) are computer programs aimed to improve healthcare by supporting the clinician in the process of diagnostic decision making. Previous studies demonstrated their ability to enhance clinicians’ diagnostic skills, prevent diagnostic errors, and reduce hospitalization costs. Despite their potential benefits, their utilization in clinical practice is limited, emphasizing the need for new and improved products. Therefore, in this study we aimed to conduct a primary evaluation of diagnostic performance for “Kahun”, a new artificial intelligence driven diagnostic tool.

Methods: Diagnostic performance was evaluated based on the program ability to “solve” clinical cases from the USMLE®-step-2-clinical-skills board-exams simulations. Cases were entered to Kahun by three blinded physicians, unexperienced with the platform. The generated differential-diagnoses (DDX) were recorded and compared to the expected one. The cases were drawn from the case-banks of three leading preparation companies: UWorld, Amboss and FirstAid. Each case included 3“correct” differential-diagnoses. Diagnostic performance was measured in two ways. First, as sensitivity, calculated as the total number of expected DDX appropriately suggested by Kahun divided by the total number of expected diagnoses in all cases. Second, as case specific success rates, calculated as the number of cases with 1/3,2/3 and 3/3 of expected DDX appropriately suggested by Kahun divided by the total number of cases.

Findings: 91 clinical cases were included in the study with 78 different chief complaints, and 174 different DDX. Kahun correctly suggested 231 diagnoses, resulting in an overall sensitivity rate of 84.9%which was stable across different disciplines. In 63.8%of the cases Kahun correctly suggested 3/3 of expected DDX within the topmost likely diagnoses, in 89%at least 2/3, and in 97.8%at least 1/3.

Interpretation: Kahun demonstrates an acceptable diagnostic accuracy and comprehensiveness.

Funding: None to declare.

Declaration of Interest: NBS, AS and TW were employed by Kahun Medical Ltd as medical advisors. All other authors have nothing to declare.

Suggested Citation

Ben-Shabat, Niv and Sloma, Ariel and Weizman, Tomer and Kiderman, David and Amital, Howard, Evaluating Diagnostic Accuracy of a New Artificial-Intelligence Driven Diagnostic Support Tool. Available at SSRN: https://ssrn.com/abstract=3878093 or http://dx.doi.org/10.2139/ssrn.3878093

Niv Ben-Shabat (Contact Author)

Tel Aviv University

Ramat Aviv
Tel-Aviv, 6997801
Israel

Ariel Sloma

Tel-Aviv University - Sackler Faculty of Medicine ( email )

Tel-Aviv
Israel

Tomer Weizman

Technion-Israel Institute of Technology - Faculty of Medicine ( email )

Israel

David Kiderman

Hebrew University of Jerusalem - Hadassah Medical Center

Jerusalem
Israel

Howard Amital

Tel-Aviv University - Sackler Faculty of Medicine ( email )

Tel-Aviv
Israel

Click here to go to TheLancet.com

Paper statistics

Abstract Views
257
Downloads
28
PlumX Metrics