Heterogeneous Multi-Agent Reinforcement Learning for Large-Scale Cooperative Control of Connected Vehicles and Traffic Lights in Mixed Traffic Environments

Abstract

With the rapid development of vehicle networking and autonomous driving technology, mixed traffic scenarios—where human-driven vehicles (HDVs) coexist with connected and autonomous vehicles (CAVs) and interact with traffic lights (TLs)—have become increasingly prevalent. However, the heterogeneity of CAVs and TLs poses significant challenges to promotion of overall traffic efficiency through traditional control methods. To address this challenge, this study proposes a heterogeneous multi-agent reinforcement learning (HMARL) algorithm based on potential game (PG). Specifically, an information-sharing framework that leverages a heterogeneous graph structure and an encoder is employed to effectively integrate diverse data from CAVs and TLs. Secondly, the CAVs and TLs are integrated into a PG structure, wherein all agents share the same goal, which enables theoretically validated reward convergence and individual policy adjustments to be uniformly quantified, and a computationally efficient safety projection mechanism is introduced in PG to enhance motion safety. Extensive simulations on the city-scale demonstrate the preferable performance of the algorithm. The average speed is promoted by [[EQUATION]] and the traffic density is reduced by [[EQUATION]] during peak hours compared to baselines. In a custom urban network, the HMARL increases the average vehicle speed by approximately [[EQUATION]], exhibiting more stable synergistic effects in heterogeneous multi-agent strategy convergence.

Keywords: Heterogeneous multi-agent reinforcement learning (HMARL), connected and automated vehicles (CAVs), mixed traffic, potential game (PG), safety projection