Can Base ChatGPT be Used for Forecasting without Additional Optimization?

77 Pages Posted: 16 Apr 2024

See all articles by Van H. Pham

Van H. Pham

Baylor University - Department of Economics

Scott Cunningham

Baylor University

Date Written: July 03, 2024

Abstract

This study investigates whether OpenAI's ChatGPT-3.5 and ChatGPT-4 can forecast future events. To evaluate the accuracy of the predictions, we take advantage of the fact that the training data at the time of our experiments (mid 2023) stopped at September 2021, and ask about events that happened in 2022. We employed two prompting strategies: direct prediction and what we call future narratives which ask ChatGPT to tell fictional stories set in the future with characters retelling events that happened in the past, but after ChatGPT's training data had been collected. We prompted ChatGPT to engage in storytelling, particularly within economic contexts. After analyzing 100 trials, we find that future narrative prompts significantly enhanced ChatGPT-4's forecasting accuracy. This was especially evident in its predictions of major Academy Award winners as well as economic trends, the latter inferred from scenarios where the model impersonated public figures like the Federal Reserve Chair, Jerome Powell. As a falsification exercise, we repeated our experiments in May 2024 at which time the models included more recent training data. ChatGPT-4's accuracy significantly improved when the training window included the events being prompted for, achieving 100% accuracy in many instances. The poorer accuracy for events outside of the training window suggests that in the 2023 prediction experiments, ChatGPT-4 was forming predictions based solely on its training data. Narrative prompting also consistently outperformed direct prompting. These findings indicate that narrative prompts leverage the models' capacity for hallucinatory narrative construction, facilitating more effective data synthesis and extrapolation than straightforward predictions. Our research reveals new aspects of LLMs' predictive capabilities and suggests potential future applications in analytical contexts.

Keywords: ChatGPT, Natural Language Processing (NLP), Large Language Models

JEL Classification: C53, E52, C00

Suggested Citation

Pham, Van H. and Cunningham, Scott, Can Base ChatGPT be Used for Forecasting without Additional Optimization? (July 03, 2024). Available at SSRN: https://ssrn.com/abstract=4792918 or http://dx.doi.org/10.2139/ssrn.4792918

Van H. Pham (Contact Author)

Baylor University - Department of Economics ( email )

One Bear Place #98003
Waco, TX 76798
United States
(254) 710-3521 (Phone)
(254) 710-6142 (Fax)

HOME PAGE: http://www.baylor.edu/van_pham

Scott Cunningham

Baylor University ( email )

One Bear Place #98003
Waco, TX 76798
United States
254-710-4753 (Phone)
254-710-6142 (Fax)

HOME PAGE: http://www.scunning.com

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
55
Abstract Views
362
Rank
697,116
PlumX Metrics