Challenging ChatGPT ' Intelligence' with Human Tools: A Neuropsychological Investigation on Prefrontal Functioning of a Large Language Model

33 Pages Posted: 6 Mar 2023

See all articles by Riccardo Loconte

Riccardo Loconte

IMT School for Advanced Studies Lucca - Molecular Mind Lab

Graziella Orrù

University of Pisa

Mirco Tribastone

IMT School for Advanced Studies Lucca - Molecular Mind Lab

Pietro Pietrini

IMT School for Advanced Studies Lucca - Molecular Mind Lab

Giuseppe Sartori

Università di Padova - Department of General Psychology

Abstract

The Artificial Intelligence (AI) research community has used ad-hoc benchmarks to measure the 'intelligence' level of Large Language Models (LLMs). Previous research has found that LLMs struggle with cognitive tasks that required the integrity of the human prefrontal lobes, known as 'prefrontal functions.' In December 2022, OpenAI released ChatGPT, a new chatbot that quickly gained popularity for its ability to understand and respond to human instructions. To investigate ChatGPT’s level of 'intelligence,' we conducted a neuropsychological evaluation with the same tests routinely used to evaluate prefrontal functioning in humans, since human 'intelligence' requires the functional integrity of the frontal lobes. While ChatGPT is well-known to exhibit outstanding performance on generative linguistic tasks, its performance on prefrontal tests was inhomogeneous, with some tests well above average, others in the lower range, and others frankly impaired. Specifically, we have identified poor planning abilities and difficulty in recognising semantic absurdities and understanding others’ intentions and mental states. This inconsistent profile highlights how LLMs’ emergent abilities do not yet mimic human cognitive functioning. In addition, our results indicate that standardised neuropsychological batteries developed to assess human cognitive functions may be suitable for challenging ChatGPT performance.

Keywords: artificial intelligence, Large Language Models, ChatGPT, Prefrontal Functioning, Neuropsychological Evaluation

Suggested Citation

Loconte, Riccardo and Orrù, Graziella and Tribastone, Mirco and Pietrini, Pietro and Sartori, Giuseppe, Challenging ChatGPT ' Intelligence' with Human Tools: A Neuropsychological Investigation on Prefrontal Functioning of a Large Language Model. Available at SSRN: https://ssrn.com/abstract=4377371 or http://dx.doi.org/10.2139/ssrn.4377371

Riccardo Loconte (Contact Author)

IMT School for Advanced Studies Lucca - Molecular Mind Lab ( email )

Graziella Orrù

University of Pisa ( email )

Lungarno Pacinotti, 43
Pisa PI, 56126
Italy

Mirco Tribastone

IMT School for Advanced Studies Lucca - Molecular Mind Lab ( email )

Pietro Pietrini

IMT School for Advanced Studies Lucca - Molecular Mind Lab ( email )

Lucca
Italy

Giuseppe Sartori

Università di Padova - Department of General Psychology ( email )

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
928
Abstract Views
3,459
Rank
48,043
PlumX Metrics