Evaluating Criminal Justice Reform During COVID-19: The Need for a Novel Sentiment Analysis Package

Abstract

Existing natural language processing lexicons that underlie current sentiment analysis (SA) algorithms may not perform adequately in certain academic disciplines depending on contextual complexities. The health and safety of incarcerated persons and correctional personnel have been prominent in news media discourse during the COVID-19 pandemic, potentially highlighting the need for a novel SA lexicon and algorithm that is tailored for the examination of public health policy in the context of the criminal justice system. We utilized a text corpus consisting of news articles at the intersection of COVID-19 and criminal justice to analyze the performance of existing lexicons collected across state-level outlets between January and May 2020. Our results demonstrated that sentence sentiment scores provided by three popular SA packages differ considerably from manually-curated ratings. This dissimilarity was especially pronounced when the text was more polarized, whether negatively or positively. A randomly selected set of 1,000 manually scored sentences, and the corresponding binary document term matrices, were used to train two new sentiment prediction algorithms (i.e., linear regression and random forest regression) to verify the performance of the manually-curated ratings. By better accounting for the unique context in which incarceration-related terminologies are used in news media, both of our proposed models outperformed all existing SA packages considered for comparison. Our findings suggest that there is a need to develop a novel lexicon, and potentially an accompanying algorithm, for analysis of text related to public health within the criminal justice system, as well as criminal justice more broadly.

Keywords: COVID-19, criminal justice, sentiment analysis, text analysis, public health, health, safety, public policy, lexicon, algorithm, NLP