Deep Reinforcement Learning Framework to Optimize Long-Range Transportation Plans

Abstract

Existing decision support tools for long-range transportation planning evaluate and forecast, rather than prescribe or recommend, projects and policies. Based on Reinforcement Learning (RL), this paper proposes a sequential decision optimization framework for guiding planning decisions and their implementation timings in a metropolitan area where social and economic activities interact with transportation infrastructure and land use at different timescales. To systemically address the challenge of sequential decisions in composing urban transportation plans, we propose (1) customizing a Deep Q-Learning approach for maximizing the cumulative multi-criteria objective function across planning updates and (2) adopting Bayesian Optimization at each planning interval to maximize the non-myopic values of state-action pairs (Q-values) given the state of the urban system. We apply the RL approach to a case study of the San Diego region of California, where the RL agent represents the corresponding Metropolitan Planning Organization, and a partially observable Markov Decision Process model represents the environment. We show that the proposed RL framework is suitable for recommending a portfolio and sequence of transportation and land use policies over the long but finite planning horizon. It outperforms alternative decision policies for the long-range transportation planning problem. The case study results also suggest that coordinating the timings of infrastructure expansion, environmental impact fees for new development, and congestion pricing enables economic growth and limits induced vehicle travel.

Keywords: Reinforcement learning, Bayesian Optimization, Deep Q-Learning, Smart Planning, Multiobjective Optimization, Land Use-Transportation Interactions, Markov Decision Process