Classification Performance Thresholds for BERT-Based Models on COVID-19 Twitter Misinformation

6 Pages Posted: 14 Jul 2023 Last revised: 8 Jan 2024

See all articles by Johnattan Ontiveros

Johnattan Ontiveros

Boston Children's Hospital - Computational Health Informatics Program

Robyn Correll Carlyle

Vaccinate Your Family

Anika Puri

Boston Children's Hospital - Computational Health Informatics Program

Sagar Kumar

Northeastern University - Network Science Institute

Alexander Tregub

Kent State University - Department of Mathematical Sciences

Caroline Nitirahardjo

Kent State University - Department of Biological Science

Evelynne Morgan

Kent State University - Department of Biological Science

Brendan C Lawler

Boston Children's Hospital - Computational Health Informatics Program

Eliza Aimone

Kent State University - Department of Biological Science

Helen Piontkivska

Kent State University - Department of Biological Science

Maimuna S. Majumder

Boston Children's Hospital - Computational Health Informatics Program; Harvard University - Harvard Medical School

Date Written: June 23, 2023

Abstract

The use of artificial intelligence and machine learning methodologies like natural language processing (NLP) have allowed researchers to analyze large amounts of text to identify misinformation and evaluate public attitudes, especially on Twitter. Large language models like Bi-directional Encoder Representations from Transformers (BERT) can be fine-tuned with subject-specific, hand-labeled text to improve accuracy for niche classification tasks. However, it remains unclear to what extent keyword selection and training data size might impact model performance. To examine these effects, two separate Twitter datasets relating to sentiments regarding COVID-19 vaccination — one using a more specific set of keywords and another using more generalized keywords — were used to train two BERT models with increasing quantities of hand-labeled data. The model trained on a more specific set of keywords quickly achieved high accuracy and F-1 scores, even with <1000 tweets of training data, whereas the model trained on more generalized keywords consistently had lower performance across all quantities of training data. Our findings demonstrate a quantifiable tradeoff between accuracy and generalizability of BERT-based models within the context of vaccine-related sentiment classification.

Note:
Funding Information: JO, AP, CN, EM, EA, and HP were supported in part by grant SES2230083 from the National Science Foundation. SK was supported in part by grant SES2200228 from the National Science Foundation. The CompEpi Dispersed Volunteer Research Network is sponsored in part by grant R35GM146974 from the National Institute of General Medical Sciences, National Institutes of Health. The funding sources had no involvement in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the paper for publication

Declaration of Interests: We know of no conflicts of interest associated with this publication, and there has been no financial support associated with this research that could have affected its outcome

Keywords: Natural language processing,Sentiment classification,Large language models,Machine learning,Twitter,COVID-19 Vaccines,Vaccine misinformation,Anti-vaccine sentiment

Suggested Citation

Ontiveros, Johnattan and Carlyle, Robyn and Puri, Anika and Kumar, Sagar and Tregub, Alexander and Nitirahardjo, Caroline and Morgan, Evelynne and Lawler, Brendan C and Aimone, Eliza and Piontkivska, Helen and Majumder, Maimuna, Classification Performance Thresholds for BERT-Based Models on COVID-19 Twitter Misinformation (June 23, 2023). Available at SSRN: https://ssrn.com/abstract=4489865 or http://dx.doi.org/10.2139/ssrn.4489865

Johnattan Ontiveros

Boston Children's Hospital - Computational Health Informatics Program

Robyn Carlyle

Vaccinate Your Family ( email )

Washington, DC 20002
United States

Anika Puri

Boston Children's Hospital - Computational Health Informatics Program

Sagar Kumar

Northeastern University - Network Science Institute

Alexander Tregub

Kent State University - Department of Mathematical Sciences

Caroline Nitirahardjo

Kent State University - Department of Biological Science

Evelynne Morgan

Kent State University - Department of Biological Science

Brendan C Lawler

Boston Children's Hospital - Computational Health Informatics Program

Eliza Aimone

Kent State University - Department of Biological Science

Helen Piontkivska

Kent State University - Department of Biological Science

Maimuna Majumder (Contact Author)

Boston Children's Hospital - Computational Health Informatics Program ( email )

United States

Harvard University - Harvard Medical School ( email )

25 Shattuck St
Boston, MA 02115
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
116
Abstract Views
458
Rank
435,447
PlumX Metrics