Geoprospect: A Domain-Specific Geological Large Language Model with Enhanced Continual Learning

Abstract

Developing domain-specific large language models (LLMs) in highly specialized fields such as geology presents three challenges: the discrepancy between general LLMs’ pretrained knowledge and the specialized knowledge structures of geology, the lack of domain-specific corpora, and the risk of catastrophic forgetting during continual learning. To tackle these challenges, we introduce GeoProspect, the first LLM designed specifically for the geological domain. Our methodology includes the creation of a high-quality geological corpus and the adaptation of the general-purpose Qwen2-7B model through continued pretraining and supervised instruction fine-tuning. We present two variants, GeoProspect-LoRA, a model fine-tuned using low-rank adaption (LoRA) as an initial phase, and GeoProspect, which alleviates catastrophic forgetting during continual learning via a novel task-driven grouped mixture of LoRA experts (TG-MoLE) architecture. TG-MoLE integrates a dual-layer grouped routing strategy and a task-driven cross-group routing adjustment mechanism to optimize weight distribution and expert collaboration. To assess model performance, we develop GeoEval, the first benchmark tailored to geological LLMs. Experimental results demonstrate that our model significantly improves performance in geological knowledge understanding and mineral exploration question answering. Moreover, GeoProspect, equipped with TG-MoLE, maintains general task performance while achieving superior results in geology-specific applications.

Keywords: Domain-Specific LLMs, Geology, Continual Learning, Mixture of Experts