Automatic Quantitative Stroke Severity Assessment Based on Chinese Clinical Named Entity Recognition with Domain-Adaptive Pre-Trained Large Language Model

Abstract

Background: Stroke is a prevalent disease with significant global impact. Effective assessment of stroke severity is vital for an accurate diagnosis, appropriate treatment, and optimal clinical outcomes. The National Institutes of Health Stroke Scale (NIHSS) is a widely used scale for quantitatively assessing stroke severity. However, the current manual scoring of NIHSS is labor-intensive, time-consuming and sometimes unreliable. Applying artificial intelligence (AI) techniques to automate the quantitative assessment of stroke on vast amount of electronic health records (EHRs) has attracted much interest.

Objective: This study aims to develop an automatic, quantitative stroke severity assessment framework through automating the entire NIHSS scoring process on Chinese clinical EHRs.

Methods: Our approach consists of two major parts: Chinese clinical named entity recognition (CNER) with a domain-adaptive pre-trained large language model (LLM) and automated NIHSS scoring. To build a high-performing CNER model, we first constructed a stroke-specific, densely annotated dataset “Chinese Stroke Clinical Records” (CSCR) from EHRs provided by our partner hospitals, based on a stroke ontology that defined intensive, semantically related entities for stroke assessment. We then pre-trained a Chinese clinical LLM coined “CliRoberta” through domain-adaptive transfer learning and constructed a deep learning-based CNER model that can accurately extract entities directly from Chinese EHRs. Finally, an automated, end-to-end NIHSS scoring pipeline is proposed by mapping the extracted entities to relevant NIHSS items and values, to quantitatively assessing the stroke severity.

Results: Results obtained on a benchmark dataset CCKS2019 and our newly created CSCR dataset demonstrate the superior performance of our domain-adaptive pre-trained LLM and the CNER model. The high accuracy of 0.993 F1 score ensures the reliability of our model in accurately extracting the entities for the subsequent automatic NIHSS scoring. Subsequently, our automated, end-to-end NIHSS scoring approach achieved excellent inter-rater agreement (0.823) and intraclass consistency (0.986) with the ground truth and significantly reduced the processing time from minutes to a few seconds.

Conclusion: Our proposed automatic and quantitative framework for assessing stroke severity demonstrates exceptional accuracy and reliability through directly scoring the NIHSS from diagnostic notes in Chinese clinical EHRs. Moreover, this study also contributes a new clinical dataset, a pre-trained clinical LLM and an effective deep learning-based CNER model. The deployment of these advanced algorithms can improve the accuracy and efficiency of clinical assessment, and help improve the quality, affordability and productivity of healthcare services.

Note:
Funding Declaration: This work was supported by the international research project RES17-1120 of the University of Technology Sydney (UTS).

Conflicts of Interest: None

Ethical Approval: This study was approved by both the University of Technology Sydney (UTS) Human Research Ethics Committee (HREC) (EC00146) under ETH23-8230, and The Third Affiliated Hospital of Sun Yat-sen University Medical Ethics Committee under A2019-007-01. The data in this study is de-identified and secondarily used, allowing for waiver of informed consent as detailed in A2019-007-01. This study is undertaken strictly in compliance with the Australia National Statement on Ethical Conduct in Human Research (Chapter 2.1) and UTS Research Policy.

Keywords: Automatic stroke severity assessmentChinese electronic health recordsClinical named entity recognitionDomain-adaptive pre-trainingLarge language model