From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models

10 Pages Posted: 26 Sep 2023

See all articles by Masahiro Suzuki

Masahiro Suzuki

University of Tokyo; Sumitomo Mitsui Trust Bank, Limited - Nikko Asset Management Co., Ltd.

Masanori HIRANO

Preferred Networks, Inc.

Hiroki Sakaji

The University of Tokyo

Date Written: September 7, 2023

Abstract

Instruction tuning is essential for large language models (LLMs) to become interactive. While many instruction tuning datasets exist in English, there is a noticeable lack in other languages. Also, their effectiveness has not been well verified in non-English languages. We construct a Japanese instruction dataset by expanding and filtering existing datasets and apply the dataset to a Japanese pre-trained base model. We performed Low-Rank Adaptation (LoRA) tuning on both Japanese and English existing models using our instruction dataset. We evaluated these models from both quantitative and qualitative perspectives. As a result, the effectiveness of Japanese instruction datasets is confirmed. The results also indicate that even with relatively small LLMs, performances in downstream tasks would be improved through instruction tuning. Our instruction dataset, tuned models, and implementation are publicly available online.

Keywords: Large Language Model (LLM), Instruction Dataset, Instruction Tuning, Japanese

Suggested Citation

Suzuki, Masahiro and HIRANO, Masanori and Sakaji, Hiroki, From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models (September 7, 2023). Available at SSRN: https://ssrn.com/abstract=4564308 or http://dx.doi.org/10.2139/ssrn.4564308

Masahiro Suzuki (Contact Author)

University of Tokyo ( email )

Hongo 7-3-1
Bunkyo-ku
Tokyo, Tokyo 113-8657
Japan

Sumitomo Mitsui Trust Bank, Limited - Nikko Asset Management Co., Ltd. ( email )

Midtown Tower
9-7-1 Akasaka
Minato-ku, Tokyo 107-6242
Japan

Masanori HIRANO

Preferred Networks, Inc. ( email )

Otemachi Bldg., 1-6-1 Otemachi
Chiyoda-ku, Tokyo 1000004
Japan

Hiroki Sakaji

The University of Tokyo ( email )

7-3-1 Hongo
Bunkyo-ku
Tokyo, 113-0033
Japan

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
34
Abstract Views
193
PlumX Metrics