Gov-RAG: A Retrieval-Augmented Generation Framework for Enhancing E-Government Services

Abstract

The rapid evolution of Large Language Models (LLMs) has opened new opportunities for applying AI in high-stakes domains such as government and legal services. However, deploying LLMs involves challenges like outdated training data, limited explainability, and reliability issues due to hallucinations. This study introduces Gov-RAG, a novel Retrieval-Augmented Generation (RAG) framework, specifically tailored to address the unique demands of e-government applications.  Grounded in trust in automation theory and domain-specific practical adaptation needs, the Gov-RAG framework achieves five key goals: (1) In-depth domain knowledge, (2) Factual accuracy in sensitive domains, (3) Human-centered explainability, (4) Low-cost real-time updates, and (5) Hallucination reduction. The framework incorporates a real-time updated comprehensive dataset, aggregating national laws, regulations, and government policies. To enhance retrieval precision, Gov-RAG combines dense and sparse vector retrieval with inverted index full-text retrieval and applies domain-specific hierarchical slicing. We also develop a fact-based evaluation framework to measure hallucinations across different LLM architectures, an aspect that was not properly measured in prior work. Rigorous computational experiments show that while fine-tuning improves stylistic alignment with standard answers, it also increases hallucination. Gov-RAG can improve factual consistency and reduce contradictions, consistently outperforming base LLMs and alternative frameworks in governmental applications. Furthermore, RAG with fine-tuned LLMs exhibits degraded performance in answering questions that require historical knowledge. Gov-RAG exhibits balanced capabilities across different test datasets. Online experiments with real citizens further validate the feasibility and impact of Gov-RAG on user performance, user perceptions, and user behaviors. Results indicate significant improvements in task performance, user-perceived explainability, efficiency, and user sentiment during engagement compared to base LLMs. Additionally, we delve into human-centered perceptions and adopt explainable machine learning approaches to identify what drives user performance improvement. 

Keywords: AI-assisted Governmental Consultation, Large Language Models, Retrieval Augmented Generation, E-Government, Design Science