Traffic light control using deep policy-gradient and value-function-based reinforcement learning

• Published in: IET intelligent transport systems Vol. 11; no. 7; pp. 417 - 423
• Main Authors: Mousavi, Seyed Sajad, Schukat, Michael, Howley, Enda
• Format: Journal Article
• Language: English
• Published: The Institution of Engineering and Technology 01.09.2017
• Subjects: action spaces; adaptive control; adaptive traffic light control agents; complex control problems; control engineering computing; control signals; deep neural network architectures; deep policy‐gradient RL algorithm; digital simulation; gradient methods; graphical traffic simulator; high‐dimensional state space; learning (artificial intelligence); optimal control; PG‐based agent maps; Research Article; road traffic control; traffic engineering computing; traffic intersection; traffic light control; traffic signal; training process; urban mobility traffic simulator; value‐function‐based agent RL algorithms; value‐function‐based reinforcement learning; high-dimensional state space; complex control problems; urban mobility traffic simulator; road traffic control; adaptive traffic light control agents; traffic intersection; traffic signal; training process; adaptive control; graphical traffic simulator; optimal control; digital simulation; traffic engineering computing; deep neural network architectures; traffic light control; control signals; PG-based agent maps; action spaces; deep policy-gradient RL algorithm; value-function-based reinforcement learning; learning (artificial intelligence); gradient methods; control engineering computing; value-function-based agent RL algorithms
• ISSN: 1751-956X; 1751-9578; 1751-9578
• DOI: 10.1049/iet-its.2017.0153

Recent advances in combining deep neural network architectures with reinforcement learning (RL) techniques have shown promising potential results in solving complex control problems with high-dimensional state and action spaces. Inspired by these successes, in this study, the authors built two kinds of RL algorithms: deep policy-gradient (PG) and value-function-based agents which can predict the best possible traffic signal for a traffic intersection. At each time step, these adaptive traffic light control agents receive a snapshot of the current state of a graphical traffic simulator and produce control signals. The PG-based agent maps its observation directly to the control signal; however, the value-function-based agent first estimates values for all legal control signals. The agent then selects the optimal control action with the highest value. Their methods show promising results in a traffic network simulated in the simulation of urban mobility traffic simulator, without suffering from instability issues during the training process.