A Critical Assessment of Large Language Models for Systematic Reviews: Utilizing ChatGPT for Complex Data Extraction

Abstract

Objectives: To assess the effectiveness of large language models (LLMs) for systematic reviews, focusing on measures from basic details to complex criteria requiring nuanced evaluations.

Methods: Screening the full text of 10 COVID-19 modeling studies, we analyzed three basic measures of study settings, e.g., analysis location, and three complex measures of behavioral components in models, e.g., risk perception. To extract data on these measures, we conducted 60 manual codings and compared them with 420 queries spanning seven iterations in ChatGPT.

Results: ChatGPT demonstrated 72% overall accuracy in extracting 60 data elements, performing better in extracting explicitly stated study settings (93%) than subjective behavioral components (50%).

Discussion: While ChatGPT’s accuracy improved as prompts were refined, varying accuracy across measures highlights its limitations.

Conclusion: We underscore LLMs’ utility in systematic reviews for basic, explicit data extraction but reveal significant limitations in handling nuanced, subjective criteria, emphasizing the current necessity for human oversight.