Multibcd: A Multimodal Model that Simulates the Human Diagnostic Process for Automated Breast Cancer Detection

Abstract

To enhance the accuracy of breast cancer detection, our study introduces MultiBCD, a multimodal model that emulates the human diagnostic process for breast cancer detection. Integrating an image classifier with GPT-4, it evaluates mammographic images alongside patient complaints. The model’s dual-head autoencoder efficiently processes image data, eliminating the need for manual lesion delineation, while GPT-4 extracts critical information from patient narratives.MultiBCD demonstrates superior diagnostic accuracy and efficiency, achieving an F1 score of 86.49% and a recall rate of 94.12%, which marks an improvement over traditional methods. Furthermore, its design, emphasizing interpretability, aligns with the intuitive experience of medical consultations. The encouraging results of MultiBCD in breast cancer detection indicate its potential for application in similar diagnostic contexts.The MultiBCD model is characterized by its compact structure, flexible and efficient coupling, and the open-sourcing of its code(\href{https://github.com/zhangzihan-is-good/AI-breast-cancer}{https://github.com/zhangzihan-is-good/AI-breast-cancer}), thereby enhancing the model's practical utility.

Note:
Funding Declaration: We thank the reviewers for their constructive comments and gratefully acknowledge the support of the First Affiliated Hospital of Harbin Medical University Fund for Distinguished Young Medical Scholars (2021J17) and the BEIJING MEDICAL AWARD FOUNDATION (YXJL-2021-0302-0287).

Conflict of Interests: None.

Keywords: Artificial IntelligenceMultimodalityBreast CancerMammogramCNNGPT-4