Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training

7 Pages Posted: 16 Apr 2024

See all articles by Masanori Hirano

Masanori Hirano

Preferred Networks, Inc.

Kentaro Imajo

Preferred Networks, Inc.

Date Written: April 16, 2024

Abstract

Large language models (LLMs) are now widely used in various fields, including finance.
However, Japanese financial-specific LLMs have not been proposed yet.
Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training.
Before tuning, we constructed Japanese financial-focused datasets for continual pre-training.
As a base model, we employed a Japanese LLM that achieved state-of-the-art performance on Japanese financial benchmarks among the 10-billion-class parameter models.
After continual pre-training using the datasets and the base model, the tuned model performed better than the original model on the Japanese financial benchmarks.
Moreover, the outputs comparison results reveal that the tuned model's outputs tend to be better than the original model's outputs in terms of the quality and length of the answers.
These findings indicate that domain-specific continual pre-training is also effective for LLMs.
The tuned model is publicly available on Hugging Face.

Keywords: large language model, continual pre-training, domain-specific tuning, Japanese, finance

Suggested Citation

Hirano, Masanori and Imajo, Kentaro, Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training (April 16, 2024). Available at SSRN: https://ssrn.com/abstract=4796245 or http://dx.doi.org/10.2139/ssrn.4796245

Masanori Hirano (Contact Author)

Preferred Networks, Inc. ( email )

Otemachi Bldg., 1-6-1 Otemachi
Chiyoda-ku, Tokyo 1000004
Japan

Kentaro Imajo

Preferred Networks, Inc.

Otemachi Bldg., 1-6-1 Otemachi
Chiyoda-ku, Tokyo 1000004
Japan

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
6
Abstract Views
138
PlumX Metrics