Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach

33 Pages Posted: 9 Aug 2021 Last revised: 31 Mar 2022

See all articles by Haotian Gu

Haotian Gu

affiliation not provided to SSRN

Xin Guo

University of California, Berkeley - Department of Industrial Engineering and Operations Research

Xiaoli Wei

affiliation not provided to SSRN

Renyuan Xu

University of Southern California - Epstein Department of Industrial & Systems Engineering

Date Written: February 22, 2022

Abstract

One of the challenges for multi-agent reinforcement learning (MARL) is designing efficient learning algorithms for a large system in which each agent has only limited or partial information of the entire system. In this system, it is desirable to learn policies of a decentralized type. A recent and promising paradigm to analyze such decentralized MARL is to take network structures into consideration. While exciting progress has been made to analyze decentralized MARL with the network of agents, often found in social networks and team video games, little is known theoretically for decentralized MARL with the network of states, frequently used for modeling self-driving vehicles, ride-sharing, and data and traffic routing.

This paper proposes a framework called localized training and decentralized execution to study MARL with network of states, with homogeneous (a.k.a. mean-field type) agents. Localized training means that agents only need to collect local information in their neighboring states during the training phase; decentralized execution implies that, after the training stage, agents can execute the learned decentralized policies, which only requires knowledge of the agents’ current states. The key idea is to utilize the homogeneity of agents and regroup them according to their states, thus the formulation of a networked Markov decision process with teams of agents, enabling the update of the Q-function in a localized fashion. In order to design an efficient and scalable reinforcement learning algorithm under such a framework, we adopt the actor-critic approach with over-parameterized neural networks, and establish the convergence and sample complexity for our algorithm, shown to be scalable with respect to the size of both agents and states.

Keywords: Mean field game; Multi-agent Reinforcement learning;Graph network; Deep Learning

Suggested Citation

Gu, Haotian and Guo, Xin and Wei, Xiaoli and Xu, Renyuan, Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach (February 22, 2022). Available at SSRN: https://ssrn.com/abstract=3900139 or http://dx.doi.org/10.2139/ssrn.3900139

Haotian Gu

affiliation not provided to SSRN

Xin Guo

University of California, Berkeley - Department of Industrial Engineering and Operations Research ( email )

4141 Etcheverry Hall
Berkeley, CA 94720-1777
United States

Xiaoli Wei

affiliation not provided to SSRN

Renyuan Xu (Contact Author)

University of Southern California - Epstein Department of Industrial & Systems Engineering ( email )

United States

HOME PAGE: http://renyuanxu.github.io

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
211
Abstract Views
579
rank
204,471
PlumX Metrics