Unraveling Generative AI from A Human Intelligence Perspective: A Battery of Experiments

Posted: 18 Aug 2023

See all articles by Wen Wang

Wen Wang

University of Maryland - Robert H. Smith School of Business

Siqi Pei

Independent

Tianshu Sun

Cheung Kong Graduate School of Business; University of Southern California - Marshall School of Business

Date Written: August 17, 2023

Abstract

Generative AI and large language models (LLMs) have made remarkable progress recently. In this study, we propose a novel comprehensive evaluation framework that leverages human intelligence as a benchmark to assess LLMs’ intellectual capabilities. Our proposed framework builds on behavioral theory and can identify four key intelligence including cognitive, emotional, social, and creative intelligence with 18 sub-abilities using a battery of canonical experiments from the literature. Three studies are conducted to demonstrate the superiority of our framework and fully understand the intellectual capabilities of LLMs. Firstly, comprehensive evaluation results show that our proposed framework uncovers key properties of LLMs from an intelligence perspective and offers meaningful insights on the practical adoption of LLMs for organizations. For example, GPT-4 surpass human subjects not just in cognitive intelligence but also in social, emotional, and creative intelligence as well as in the stability of performance across all dimensions, indicating LLMs have attained a respectable degree of general human intelligence. Moreover, we uncover a distinctive tradeoff between emotional intelligence and other facets of intelligence, which provides a strategic guidance for organizations to tailor their LLMs selection to their specific needs. Secondly, we further open-up the black box of LLM intelligence by testing various open source models and investigate how back-end model training strategies influence front-end intelligence. Interestingly, prescriptive human guidance with RLHF strategy may reduce the creativity of the model. Thirdly, we demonstrate the close connection between the proposed human intelligence framework of LLM and the human society through a representative labor market showcase. Our framework displays superior predictive power for occupational skills in large scale job postings compared to existing frameworks. Taken together, the findings from three studies can evaluate the full breadth of LLMs’ intellectual capacities, and provide implications to firms and governments on IT adoption within organizations, and how they should prepare for labor market transformation in the era of LLMs.

Keywords: Generative AI, Large Language Models, Human Intelligence, Behavioral Theory, Comprehensive Evaluation, Organization IT Adoption, Labor Market Transformation

Suggested Citation

Wang, Wen and Pei, Siqi and Sun, Tianshu, Unraveling Generative AI from A Human Intelligence Perspective: A Battery of Experiments (August 17, 2023). Available at SSRN: https://ssrn.com/abstract=4543351

Wen Wang (Contact Author)

University of Maryland - Robert H. Smith School of Business

Siqi Pei

Independent

Tianshu Sun

Cheung Kong Graduate School of Business ( email )

1017, Oriental Plaza 1
No.1 Dong Chang'an Street
Beijing
China

University of Southern California - Marshall School of Business ( email )

3670 Trousdale Parkway
Bridge Hall 310B
Los Angeles, CA 90089
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Abstract Views
875
PlumX Metrics