# Multiobjective Reinforcement Learning for Traffic Signal Control Using Vehicular Ad Hoc Network

- Duan Houli
^{1}Email author, - Li Zhiheng
^{1}and - Zhang Yi
^{1}

**2010**:724035

https://doi.org/10.1155/2010/724035

© Duan Houli et al. 2010

**Received: **1 December 2009

**Accepted: **5 September 2010

**Published: **16 September 2010

## Abstract

We propose a new multiobjective control algorithm based on reinforcement learning for urban traffic signal control, named multi-RL. A multiagent structure is used to describe the traffic system. A vehicular ad hoc network is used for the data exchange among agents. A reinforcement learning algorithm is applied to predict the overall value of the optimization objective given vehicles' states. The policy which minimizes the cumulative value of the optimization objective is regarded as the optimal one. In order to make the method adaptive to various traffic conditions, we also introduce a multiobjective control scheme in which the optimization objective is selected adaptively to real-time traffic states. The optimization objectives include the vehicle stops, the average waiting time, and the maximum queue length of the next intersection. In addition, we also accommodate a priority control to the buses and the emergency vehicles through our model. The simulation results indicated that our algorithm could perform more efficiently than traditional traffic light control methods.

## Keywords

## 1. Introduction

Increasing traffic congestion over the road networks makes the development of more intelligent and efficient traffic control systems an urgent and important requirement. However, traffic systems are typically complex large-scale systems consisting of a great number of interacting participants. It is very difficult to use traditional control algorithms to get satisfied control effect. Thus, various intelligent algorithms have been used in attempts to build an efficient traffic control system, such as fuzzy control technologies [1, 2], artificial neural networks [3, 4], and genetic algorithms [5, 6], which greatly improve the efficiency of urban traffic signal control systems.

Reinforcement learning is a category of machine learning algorithms including Q learning, temporal difference, and SARSA algorithm [7–9]. Reinforcement learning is to learn the optimal policy by a trial-and-error process including perceiving states from the environment, choosing an action according to current states and receiving rewards from the environment. The policy which maximizes the expected long-term cumulative reward is considered as the optimal one. Reinforcement learning is a self-learning algorithm which does not need an explicit model of the environment. Thus, it can be applied in traffic signal control effectively to respond to the constant changes of traffic flow and outperform traditional traffic control algorithms. Thorpe studied reinforcement learning for traffic light control in 1997. He used a neural network to predict the waiting time for all cars standing at the intersection and selected the best control policy using the SARSA algorithm [10]. Abdulhai et al. presented a basic framework of applying Q-learning to traffic signal control and got encouraging results while applying it to an isolated intersection [11]. Mikami and Kakazu combined the evolutionary algorithm and reinforcement learning for coordination traffic signal control [12]. However, the above methods use traffic-light-based value functions, which means that the state space is too large to handle. Therefore, these methods suffer from the "dimension curse" and achieve limited success when applied to large-scale road networks. Wiering et al. utilized a car-based value function to solve this problem [13, 14]. They predicted each car's total expected waiting time until it arrived its destination given possible choices of related traffic lights using reinforcement learning, and chose the action which minimized the summed waiting time of all cars in the network. This method effectively reduces the state space and thus can be applied to large-network control. Experiments in a network with 12 edge nodes and 16 junctions proved the effectiveness of this method.

However, Wiering's method uses the total waiting time as the optimization goal which is mainly suitable for the medium traffic condition. In practical traffic systems, we should consider different optimization objectives adaptive to different traffic situations, called the multiobjective control scheme in this paper. Under the free traffic condition, the average vehicle speed is high and the average waiting time is short, so the waiting time is not the focal point, while the vehicle stops will increase the vehicle emission and oil consumption. Therefore, we should try to minimize the overall vehicle stops in the network. Under the medium traffic condition, the overall waiting time is regarded as the optimization goal because most drivers want to arrive at their destinations as soon as possible. Under the congested traffic situation, queue spillovers must be avoided to keep the network from large-scale congestion, thus, the queue length must be regarded as the control goal [15]. Since the multiobjective control scheme can adapt to various traffic conditions and make a more intelligent control system, we propose a multiobjective control strategy based on Wiering's model. In our model, data exchanges among vehicles and roadside equipments are necessary. Thus, a vehicular ad hoc network is utilized to build a wireless traffic information system.

This paper is organized as follows: in Section 2, we will introduce how to model the road network with an agent-based structure; Section 3 describes how to exchange traffic data using the ad hoc network; in Section 4, a multiagent traffic control strategy using reinforcement learning is proposed; in Section 5, the proposed method is applied to a road network with 7 intersections to prove its effectiveness; finally, in Section 6, we draw the conclusion of this paper.

## 2. Agent-Based Model of Traffic System

## 3. Traffic Information Exchange System Using Vehicular Ad Hoc Network

- (a)
traffic flow through each intersection within each time step;

- (b)
queue length at each traffic light within each time step;

- (c)
type of each vehicle (car, bus, or emergent vehicle);

- (d)
destination of each vehicle;

- (e)
node where each vehicle stands at;

- (f)
direction each vehicle moving towards;

- (g)
position in the queue where each vehicle stands at;

- (h)
total waiting time each vehicle used to pass through the network;

- (i)
total number of stops each vehicle used to pass through the network.

## 4. Multiobjective Control Algorithm Based on Reinforcement Learning (Multi-RL)

We extend Wiering's algorithm to a multiobjective scheme by selecting the optimization objective according to the real-time traffic condition. In addition, it is assumed that some special vehicles such as buses and ambulances need a priority control, and thus they should be considered separately.

The multiobjective control algorithm considers three types of traffic conditions as follows. The method to estimate traffic conditions should be defined carefully according to the actual situation of the road network.

### 4.1. Free Traffic Condition

Under this condition, we aim to minimize the number of stops, in other words, we expect to have the vehicles pass through the network with the fewest stops. Thus, the cumulative number of stops is selected as the optimization objective.

*∣*[node, dir, pos, des], , gives the probability that the traffic light turns at the next time step given the current state and the next state of this vehicle; , is a reward function as follows: if , which means the vehicle moving to a green light at the current time step meets a red light at the next time step, then the number of vehicle stops will increase, ; otherwise, ; is the discount factor ( ) which ensures that the -values are bounded. The probability that a traffic light turns red is calculated as follows:

where , , means the number of times a car in the state of transiting to the state of and the transiting light is , , , is the number of times the light turns after such a transiting procedure.

### 4.2. Medium Traffic Condition

where
is the traffic light state (red or green),
*∣* [node, dir, pos, des]
is calculated in the same way as (3),
is defined as follows: if a car stays at the same place, then
, otherwise,
(the car can move forward).

### 4.3. Congested Traffic Condition

where and have the same meanings as under the medium traffic condition. Compared (6) with (5), another reward function , is added to indicate the influence from traffic condition at the next light. , is the reward of vehicles' waiting time while , indicates the reward from the queue length increasing at the next traffic light. The parameter is an adjusting factor.

is defined as follows: if a car stays at the same place, then , otherwise, (the car can move forward).

is defined as follows: if a car passes through the current intersection to the next traffic light, which means that the queue length at the next traffic light will increase by 1 in a short time, then , otherwise, .

Through the definition we can find that will increase sharply when the queue length approaches the capacity of the lane, which means that queue spillovers would like to happen. Thus, under such a situation, will increase sharply and make the gain of this policy decrease. Therefore, the green phase length and the number of vehicles allowed to pass through will be decreased until the queue at the next light has been dispersed. The largest value of is set to 2 in this paper, but you can adjust its value according to the practical traffic condition.

### 4.4. Priority Control for Buses and Emergency Vehicles

## 5. Case Studies

**N1–N7)**and 8 OD zones (

**Zone1–Zone8**). Intersections

**N1–N7**correspond to the real intersections Xiaoweihutong, Dongdansantiao, Jingyuhutong, Dengshidongkou, Dengshikou, Wangfujingbeikou, and Taiwanfandian.

The simulation ran for 10000 time steps, the first 4000 steps made up the learning process, and the latter 6000 steps was used to collect the simulation results. Factor is set to be 0.9 and is set to be 3. The lanes in the network are divided into cells with length of 7.5 m. The capacity of the lanes equals to the number of the cells.

We compared our method with the fixed control, the actuated control and also Wiering's method. The setting of fixed control is as follows, the cycle is 2 minutes and the green time is equally assigned to all phases. In the actuated control strategy, the minimum green time is 10 s, the maximum green time is 50 s, and the extension of green time is set to 4 s. Parameters of Wiering's method are the same as our model under the medium traffic condition.

We wanted to estimate the effectiveness of the multiobjective scheme, thus, we estimated the control effects of these four algorithms under different traffic conditions. We changed the traffic volume entering the network every minute from 30 to 270 and estimated the average waiting time, the number of stops, and maximum queue length of these four methods.

In our model, when the traffic volume entering the network in a minute is less than 90, it is regarded as the free traffic; when the volume is larger than 90 but less than 180, it is regarded as the medium traffic; when the traffic volume is larger than 180, it is regarded as the congested traffic condition.

### 5.1. Comparison of the Number of Stops

It is obvious that when the traffic volume is less than 90, which means that the traffic state is free. The number of stops under the multi-RL control is less than those under other control strategies. This is because the multi-RL is the only one that aims to minimize the number of stops. However, with the increase of traffic volume, the multi-RL method changes its objective, and the actuated control gets the minimum stops.

### 5.2. Comparison of the Average Waiting Time

### 5.3. Comparison of Maximum Queue Length

## 6. Conclusion

In this paper, a multiobjective control algorithm based on reinforcement learning is proposed. The simulation results indicate that the multi-RL gets the minimum stops under the free traffic, though not the minimum waiting time; the multi-RL has almost the same performance with the RL method under the medium traffic, which is better than the fixed control and the actuated control; under congested condition, the multi-RL can effectively prevent the queue spillovers to avoid large-scale traffic jams. It should be also noticed that multi-RL is a car-based algorithm. Therefore, it is less time consuming than the light-based reinforcement learning algorithms [13].

However, there are still some system parameters that should be carefully determined by hand, for example, the adjusting factor indicating the influence of the queue at next traffic light to the waiting time of vehicles at current light under the congested traffic condition. This is a very important parameter, which we should further research its determining way based on the traffic flow theory. In addition, some phenomena in real traffic system such as the lane changing and overtaking of cars will influence their travel time. The assumption that all vehicles run at the same speed is also not so reasonable. We would take these into consideration and build a model closer to the real traffic system in future work. Besides, the communications between traffic signal controllers will help to observe the network-wide traffic states and predict future traffic conditions, which will improve the traffic control effect and should be further researched in the future.

## Declarations

### Acknowledgments

This work is supported by the National High Technology Research and Development Program ("863" Program) of China, Contract no.s 2006AA11Z229, 2007AA11Z215; by the Key Project of Chinese National Programs for Fundamental Research and Development (973 program), Contract no. 2006CB705506; by Chinese National Natural Science Foundation, Contract nos. 60834001, 60774034.

## Authors’ Affiliations

## References

- Pappis CP, Mamdani EH: Fuzzy logic controller for a traffic junction.
*IEEE Transactions on Systems, Man and Cybernetics*1977, 7(10):707-717.View ArticleMATHGoogle Scholar - Trabia MB, Kaseko MS, Ande M: A two-stage fuzzy logic controller for traffic signals.
*Transportation Research Part C*1999, 7(6):353-367. 10.1016/S0968-090X(99)00026-1View ArticleGoogle Scholar - Spall JC, Chin DC: Traffic-responsive signal timing for system-wide traffic control.
*Transportation Research Part C*1997, 5(3-4):153-163. 10.1016/S0968-090X(97)00012-0View ArticleGoogle Scholar - Liu Z: Hierarchical fuzzy neural network control for large scale urban traffic systems.
*Information and Control*1997, 26(6):441-448.Google Scholar - Foy MD,
*et al*.: Signal timing determination using genetic algorithms. In*Transportation Research Record*. National Research Council, Washington, DC, USA; 1992.Google Scholar - Park B,
*et al*.: Enhanced genetic algorithm for signal timing optimization of oversaturated intersections. In*Transportation Research Record*. National Research Council, Washington, DC, USA; 2000.Google Scholar - Sutton RS: Learning to predict by the methods of temporal differences.
*Machine Learning*1988, 3(1):9-44.Google Scholar - Watkins C:
*Learning from delayed rewards, Ph.D. thesis*. King's College, Cambridge, UK; 1989.Google Scholar - Kaelbling LP, Littman ML, Moore AW: Reinforcement learning: a survey.
*Journal of Artificial Intelligence Research*1996, 4: 237-285.Google Scholar - Thorpe T:
*Vehicle traffic light control using SARSA, M.S. thesis*. Colorado State University; 1997.Google Scholar - Abdulhai B, Pringle R, Karakoulas GJ: Reinforcement learning for true adaptive traffic signal control.
*Journal of Transportation Engineering*2003, 129(3):278-285. 10.1061/(ASCE)0733-947X(2003)129:3(278)View ArticleGoogle Scholar - Mikami S, Kakazu Y: Genetic reinforcement learning for cooperative traffic signal control.
*Proceedings of the 1st IEEE Conference on Evolutionary Computation, June 1994, Orlando, Fla, USA*1: 223-228.Google Scholar - Wiering M,
*et al*.:*Intelligent traffic light control.*University Utrecht; 2004.Google Scholar - Wiering M: Multi-agent reinforcement learning for traffic light control. Proceedings of the 17th International Conference on Machine Learning (ICML' 2000), 2000 1151-1158.Google Scholar
- Daganzo CF: Queue spillovers in transportation networks with a route choice.
*Transportation Science*1998, 32(1):3-11. 10.1287/trsc.32.1.3MathSciNetView ArticleMATHGoogle Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.