Entropy Regularization for Mean Field Games with Learning

26 Pages Posted: 19 Nov 2020

See all articles by Xin Guo

Xin Guo

University of California, Berkeley - Department of Industrial Engineering and Operations Research

Renyuan Xu

Mathematical Institute, University of Oxford

Thaleia Zariphopoulou

University of Texas at Austin - Red McCombs School of Business

Date Written: October 1, 2020

Abstract

Entropy regularization has been extensively adopted to improve the efficiency, the stability, and the convergence of algorithms in reinforcement learning. This paper analyzes both quantitatively and qualitatively the impact of entropy regularization for Mean Field Game (MFG) with learning in a finite time horizon. Our study provides a theoretical justification that entropy regularization yields time-dependent policies and, furthermore, helps stabilizing and accelerating convergence to the game equilibrium. In addition, this study leads to a policy-gradient algorithm for exploration in MFG. Under this algorithm, agents are able to learn the optimal exploration scheduling, with stable and fast convergence to the game equilibrium.

Keywords: Mean field game; Multi-agent reinforcement learning; Entropy regularization; Linear-quadratic games

Suggested Citation

Guo, Xin and Xu, Renyuan and Zariphopoulou, Thaleia, Entropy Regularization for Mean Field Games with Learning (October 1, 2020). Available at SSRN: https://ssrn.com/abstract=3702956 or http://dx.doi.org/10.2139/ssrn.3702956

Xin Guo

University of California, Berkeley - Department of Industrial Engineering and Operations Research ( email )

4141 Etcheverry Hall
Berkeley, CA 94720-1777
United States

Renyuan Xu (Contact Author)

Mathematical Institute, University of Oxford ( email )

Andrew Wiles Building
Radcliffe Observatory Quarter (550)
Oxford, OX2 6GG
United Kingdom

Thaleia Zariphopoulou

University of Texas at Austin - Red McCombs School of Business ( email )

Austin, TX 78712
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
79
Abstract Views
322
rank
377,208
PlumX Metrics