Construction of a Japanese Financial Benchmark for Large Language Models

9 Pages Posted: 15 Apr 2024

Date Written: March 22, 2024

Abstract

With the recent development of large language models (LLMs), models that focus on certain domains and languages have been discussed for their necessity.
There is also a growing need for benchmarks to evaluate the performance of current LLMs in each domain.
Therefore, in this study, we constructed a benchmark comprising multiple tasks specific to the Japanese and financial domains and performed benchmark measurements on some models.
Consequently, we confirmed that GPT-4 is currently outstanding, and that the constructed benchmarks function effectively.
According to our analysis, our benchmark can differentiate benchmark scores among models in all performance ranges by combining tasks with different difficulties.

Keywords: Learg Language Model; Benchmark; Finance; Japanese

Suggested Citation

Hirano, Masanori, Construction of a Japanese Financial Benchmark for Large Language Models (March 22, 2024). Available at SSRN: https://ssrn.com/abstract=4769124 or http://dx.doi.org/10.2139/ssrn.4769124

Masanori Hirano (Contact Author)

Preferred Networks, Inc. ( email )

Otemachi Bldg., 1-6-1 Otemachi
Chiyoda-ku, Tokyo 1000004
Japan

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
11
Abstract Views
62
PlumX Metrics