# Design and implementation of a real time and train less eye state recognition system

- Mohammad Dehnavi
^{1}Email author and - Mohammad Eshghi
^{1}

**2012**:30

https://doi.org/10.1186/1687-6180-2012-30

© Dehnavi and Eshghi; licensee Springer. 2012

**Received: **22 May 2011

**Accepted: **15 February 2012

**Published: **15 February 2012

## Abstract

Eye state recognition is one of the main stages of many image processing systems such as driver drowsiness detection system and closed-eye photo correction. Driver drowsiness is one of the main causes in the road accidents around the world. In these circumstances, a fast and accurate driver drowsiness detection system can prevent these accidents. In this article, we proposed a fast algorithm for determining the state of an eye, based on the difference between iris/pupil color and white area of the eye. In the proposed method, vertical projection is used to determine the eye state. This method is suitable for hardware implementation to be used in a fast and online drowsiness detection system. The proposed method, along with other needed preprocessing stages, is implemented on Field Programmable Gate Array chips. The results show that the proposed low-complex algorithm has sufficient speed and accuracy, to be used in real-world conditions.

## Keywords

## Introduction

All over the world and every day, driver's fatigue and drowsiness have caused many car accidents. In fact, drowsiness is the case of about 20% of all car accidents in the world [1, 2]. As a result, an electronic device to control the driver's awareness is needed. This device should monitor and detect the driver's drowsiness online and activate an alarm system immediately.

In recent years, many researches on these systems have been done and their results are reported [3–12]. One of these methods is to monitor the movement of the vehicle to detect drowsiness of the driver [3]. This method depends very much to the type of vehicle and the condition of road. Another method is to process the electrocardiogram (ECG) signals of driver [4]. In this system, some ECG probes are needed to be connected to the driver, which are disturbing the driver. There are other methods based on processing of the image of driver's face and eye. Some of methods in this category are to process the image of driver and to monitor his/her eye blinking [5–11]. In these systems, the face process, eye region detection process, and eye state recognition process are performed.

In order to determine state of an eye, authors of [5] propose a method based on combination of projection and the geometry feature of iris and pupil. Authors of [6, 7] use the fact that the iris and pupil are darker than skin and white part of the eye. Authors of [11] proposed an algorithm based on the cascade AdaBoost classifier. In [12], a gray level image of an eye is converted to a binary image, using a predetermined threshold. Then, based on the number of black and white pixels of this binary image, state of the eye is determined.

The algorithm presented in [8] used the Hough Transform to detect the iris and to determine openness of the eye. Authors of [13] used three steps to recognize the eyes' state. In the first step, the circular Hough transform is used to detect the circle of an iris in the image of an open eye. If this circle is not found then in the second step, the direction of the image of upper eyelid is obtained to determine whether it is below of the line between two corners of an eye, to detect a closed eye. If a closed or open eye is not determined in the first two steps, then in the third step, the standard deviation of distance between upper and lower eyelids is obtained and is compared to a threshold to determine the eye state.

Some researches are based on the projection of the image, to determine the state of an eye. In [9], the vertical projection of the image of both eyes is used. In [10], horizontal projection image of an eye is used to determine the interval between eyebrows and eyelids and to recognize the state of an eye. In [14], the horizontal projection of the image of a face is calculated to determine state of an eye.

Some works also are based on "Supported Vector Machine" (SVM) classifier. In [15], the SVM classifier is used to detect state of the eye. Authors of [16] used SVM classifier and Gabor filter to extract eye characteristic.

In the above methods, the authors used some conditions which make some difficulties in the eye state recognition. For example, the algorithm presented in [5] has many stages which make it slow. As a result, this method cannot be used in a real-time system. Conditions such as light from different angles, dark eyelashes, eyebrows image located in eye block, and glasses decrease the accuracy of algorithms presented in [6, 7].

Since Hough transform has a massive calculation, the algorithm proposed in [8] is also slow. Algorithm in [9], in addition to higher computation, has a high sensitivity to light radiation. Some factors such as difference interval between eyebrow and eyelid, changes of environment light, and the color of eyebrow and eyelash highly affect the accuracy of the proposed algorithm in [10]. Algorithms in [11, 15, 16] have a training phase and also their hardware implementation is complicated. Determining the threshold of algorithm presented in [12] is difficult and this algorithm is very sensitive to the light condition. Algorithm in [13] has also many stages and hardly can be implemented on a hardware platform. The accuracy of the algorithm proposed in [14] is not enough to be used in driver's situation.

In this article, a new algorithm to recognize the state of an eye, without constraints of the previous methods, is proposed. This algorithm has less sensitivity to the light conditions than other algorithms, with no need to a training phase. In order to verify the correctness of the proposed algorithm, a computer simulation is developed. In order to check and compare the speed of the proposed algorithm, we implemented it on a Field Programmable Gate Arrays (FPGA) hardware platform. The results show a fast performance and acceptable accuracy for the proposed train less eye state recognition algorithm.

The rest of this article is organized as follows. In Section 2, our real-time algorithm to determine an open or closed eye is described. The computer simulation results of the proposed algorithm are provided in Section 3. In Section 4, the design of hardware implementation of the proposed algorithm is presented. The result of this hardware implementation is provided in Section 5. Comparisons between our algorithm and others are presented in Section 6. In Section 7, the conclusion of article is presented.

### Real-time eye state recognition

In a driver drowsiness detection system based on image processing, first the location of the face in the image is determined. Then, place of eyes are determined and finally the image of an eye is processed to recognize the state of the eye. The overall driver drowsiness detection system is shown in Figure 1. In this article, it is assumed that the face detection and localization of eyes are performed with one of the methods presented in [17–22]. Our proposed algorithm recognizes the state of eyes to determine the driver drowsiness.

*m*×

*n*image the vertical projection vector, PV, is calculated using below equation.

*i*is the row number,

*j*is the column number, and PVlen

*= n*is the size of this projection vector. For example, the original vertical projection of an image of an eye shown in Figure 2a is depicted in Figure 2b.

As shown in Figure 2a, the image of an open human eye has three different areas, pupil/iris in the middle and two white parts in the left and right sides. However, in the image of a closed eye, these areas are not discriminated. The effect of this observation in the projection vector is the basis of our proposed algorithm to determine the state of the eye. As shown in Figure 2b, the projection vector of an eye in the open state has three areas. The projection vector of the pupil and iris area has less gray values than two other white areas. As a result, the projection vector has a local minimum in its middle, belongs to pupil area, and two local maximums, belong to two white parts of the eye.

The method which searches for these local minimum and maximums in the projection vector is as follows. First, we add AFlen/2 zeros to left and right sides of projection vector, to generate the zero padded projection vector (ZPPV), with a length of ZPlen = PVlen + AFlen. Then, the local maximums and minimums of this vector are obtained. The local minimums that are located between the two maximums are classified in different groups. Each minimum and two adjacent maximums form a group. In each group, the minimum is occurred at *X* min of zero padded projection vector with a value of *Y* min. In each group, also the smallest maximum is at *X* smax of zero padded projection vector with a value of *Y* smax. If at least one group in the ZPPV of the image satisfies the both following conditions then the eye is open, otherwise it is closed.

*Condition-1:*The ratio of difference between

*Y*smax and

*Y*min to

*Y*smax is greater than a threshold value,

*θ*,

*Condition-2:*The minimum that satisfies

*condition-1*is located almost in the middle of ZPPV. That is, location of this minimum,

*X*min is between 0.4

*ZPlen*to 0.6

*ZPlen*.

The ratio stated in *condition-1* is based on the difference between the color of the pupil (black) and the white area of an eye. This difference varies when the state of an eye is changing from an open state to a closed state. That is, when an open eye is going to be closed it passes different steps. C*ondition-1* verifies that an eye is open when this ratio is greater than a threshold value. In the other hand, in the relaxed and open eye, such as driving situation, the pupil almost in the center of eye, Condition-2 checked this condition to validate the openness of the eye.

As an example, consider image of an open eye, as shown in Figure 2a. Figure 2b is its projection vector and Figure 2c shows the smoothed and ZPPV of this image. Based on our experiments the threshold of Condition-1 is considered to be *θ* = 0.05. Figure 2c is satisfied both conditions of an open eye, therefore it belongs to an open eye.

### Simulation results

Results of computer simulation in The Caltech Frontal Face Database

Number of corrects | Number of incorrect | Accuracy (%) | |
---|---|---|---|

Open eye | 393 | 46 | 89.5 |

Close eye | 9 | 2 | 81.8 |

Total | 402 | 48 | 89.3 |

Results of computer simulation in SBU database [25]

Number of corrects | Number of incorrect | Accuracy (%) | |
---|---|---|---|

Open eye | 32 | 4 | 88.8 |

Close eye | 30 | 4 | 88.2 |

Total | 62 | 8 | 88.7 |

### Hardware implementation design

The address of IMAGE_RAM is generated through IMAGE_RAM_ADRESS register/counter, a ring counter which counts from 0 to 11151 (Figure 6b). The address of PV_RAM is also generated by PV_RAM_ADRESS1, a ring counter which counts from 0 to 135.

When the projection vector is completed, PV_C flag is set; and then smoothing unit starts its process, as shown in Figure 6b. For implementation of smoothing unit, we use the following procedure:

In smoothing procedure, we need to divide the summation of projection vector data which are overlap with smoothing filter to the length of the smoothing filter, AFlen. To simplify the hardware implementation of this procedure we approximate the AFlen with the nearest number of 2^{
k
}, less than AFlen. In the other word, if 2^{
k
}< AFlen < 2^{k+1}then the denominator consider to be 2^{
k
}. Dividing a number by 2^{
k
}is a *k*-bit shift to the right. In our simulation, for example, since AFlen = 19, we considered *k* = 4 and instead of division in averaging procedure, we shift the result 4 bits to the right.

In Max/Min searching unit, the local maximum and minimum of smoothed vector, generated through smoothing unit, are obtained. The input to this unit is the arrays of smoothed vector which is generated at each step. In order to find the local minimum and maximums of the smoothed projection vector, we used the following method.

In this method, each element of smoothed vector, *d*_{
i
} , is compared to the previous element, *d*_{i-1}. If *d*_{
i
} ≥ *d*_{i-1}then a '1' is shifted into a 9-bit shift register. This shift register always contains the result of the last nine comparisons. If this shift register has a pattern equal to "000001111" then it indicates a maximum in the smoothed vector. If this shift register has a pattern equal to "111100000" then it indicate a minimum in the smoothed vector. Otherwise, there is neither a minimum nor a maximum in this part of the vector.

*d*

_{i-4}, are stored. If a maximum occurred then a '1' is stored in the type register, otherwise, a '0' is stored. The index of the location is stored in the step counter. The value of the smoothed data of this minimum or maximum,

*d*

_{i-4}, is obtained from R4 of the register bank. Figure 9 shows the architecture of this process.

Last unit of this system is the condition checking unit. In this unit, conditions of maximums and minimums are checked, concurrently. In each step of type checking, three successive bits of type register is selected. If these three bits have a pattern of "101" then a minimum exist between two maximums. We called this minimum in this group as *Y* min. The index of this minimum in ZPPV is called *X* min. Also in each group one of the maximum is less than the other which we called it *Y* smax.

If *X* min in Equation (4) is between (54,81), then *Condition-2* is satisfied.

*Condition-1*is checked as follows: first, rewrite Equation (3) as Equation (5).

*θ*= 0.05; hence, Equation (5) is rewritten as Equation (6).

^{8}to obtain Equation (8).

*Y*smax. To calculate the left side of Equation (8), the

*Y*min and

*Y*smax are subtracted and the result is shifted to left by 8-bits. Concurrently,

*Y*smax is shifted 2-and 3-bits to the left and they are added to obtain right side of Equation (8). The comparator unit compares these two sides to check the threshold condition. Figure 10 shows the architecture of this unit.

This procedure is repeated for all frames and PV_RAM is cleared when the process of one frame is completed. Therefore, data are overwritten on PV_RAM, when IMAGE_RAM_ADDRESS points to the first column of IMAGE_RAM. Control circuit controls the flow of data. When the first row of IMAGE_RAM is read then the inputs of the adder are connected to IMAGE_RAM and '0'. When the other rows of IMAGE_RAM are read then the inputs of the adder are connected to IMAGE_RAM and port B of PV_RAM. Figure 6b shows this operation.

The total needed clock cycles to complete an eye state recognition in an 11152 pixels image is 11310 clocks, 11153 clocks to obtain projection vector, 137 clocks to obtain smoothed data; and 20 clocks to check the conditions.

### Hardware implementation result

FPGA is a platform to implement hardware in the gate level abstraction. Since the designs are executed in parallel, speed of the FPGA implementation of any system is higher than speed of its software implementation. Furthermore, FPGAs are reconfigurable device, minimizing time-to-market, and simplifying verification and debugging [17]. So, FPGA is one of the best platforms to implement a real time such as driver drowsiness system.

FPGA resource requirements to implementation proposed algorithm result

Logic utilization | Used | Available | Utilization (%) |
---|---|---|---|

Number of slices | 1136 | 23872 | 4 |

Number of slice flip flops | 836 | 47744 | 1 |

Number of 4 input LUTs | 2071 | 47744 | 4 |

Number of BRAMs | 1 | 126 | 0 |

### Comparison

A computer simulation is developed to evaluate the proposed algorithm. The results show that a good balance between speed and accuracy obtained in our research, compared to other articles. An FPGA implementation also is presented to obtain a high speed. The FPGA implementation result shows that the proposed algorithm and its implementation method have a high speed and accuracy. Therefore, our proposed system can be used in real-time applications.

## Conclusion

In this article, an algorithm to determine the state of an eye by using its image was presented. In our algorithm, we used the fact that the pupil and iris are darker than white part of the eye. We used the vertical projection to distinguish the state of the eye. The proposed method performed well in different light and eye color conditions. A computer simulation showed that the proposed algorithm has 89% accuracy.

Hardware design and implementation of the proposed algorithm were also presented. In this design, the algorithm is partitioned into five units and in each unit, the parallel and pipeline architectures were used. This implementation used 4% of the XC3SD3400A FPGA of Spartan 3-A DSP family. Result of this hardware implementation showed a fast architecture, i.e., the total time to process the image of an eye with 136 × 82 pixels in this system is 136 μs. This is a hardware implementation of an eye state recognition for real time.

## Declarations

## Authors’ Affiliations

## References

- Drowsy Driving[http://dimartinilaw.com/motor_vehicle_accidents/car_accidents/drowsy_driving/]
- Newsinferno news site[http://www.newsinferno.com/accident/drowsy-driving-behind-1-in-6-fatal-traffic-accidents]
- Boyraz P, Hansen JHL: Active accident avoidance case study: integrating drowsiness monitoring system with lateral control and speed regulation in passenger vehicles.
*IEEE International Conference on Vehicular Electronics and Safety, ICVES 2008*2008, 293-298.View ArticleGoogle Scholar - Yun Seong K, Haet Bit L, Jung Soo K, Hyun Jae B, Myung Suk R, Kwang Suk P: ECG, EOG detection from helmet based system.
*6th International Special Topic Conference on Information Technology Applications in Biomedicine, 2007. ITAB 2007*2007, 191-193.Google Scholar - Tabrizi PR, Zoroofi RA: Drowsiness detection based on brightness and numeral features of eye image.
*Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, 2009. IIH-MSP'09*2009, 1310-1313.Google Scholar - Horng WB, Chen CY: Improved driver fatigue detection system based on eye tracking and dynamic template matching ICS. 2008, 11-2.Google Scholar
- Wen-Bing H, Chih-Yuan C, Yi C, Chun-Hai F: Driver fatigue detection based on eye tracking and dynamic template matching. In
*Proceeding of the 2004 IEEE International Conference on Networking, Sensing & Control*. Taipei, Taiwan; 2004.Google Scholar - D'Orazio T, Leo M, Guaragnella C, Distante A: A visual approach for driver inattention detection.
*Pattern Recogn*2007, 40: 2341-2355. 10.1016/j.patcog.2007.01.018View ArticleGoogle Scholar - Zhang Z, Zhang J-S: Driver fatigue detection based intelligent vehicle control.
*Paper presented at The 18th IEEE International Conference on Pattern Recognition*2006.Google Scholar - Devi MS, Bajaj PR: Driver fatigue detection based on eye tracking.
*First International Conference on Emerging Trends in Engineering and Technology*2008, 649-652.View ArticleGoogle Scholar - Liu Z, Ai H: Automatic eye state recognition and closed-eye photo correction.
*19th International Conference on Pattern Recognition*2008, 1-4.Google Scholar - Wu J, Chen T: Development of a drowsiness warning system based on the fuzzy logic images analysis.
*Expert Syst Appl*2008, 34(2):1556-1561. 10.1016/j.eswa.2007.01.019View ArticleGoogle Scholar - Lei Y, Yuan M, Song X, Liu X, Ouyang J: Recognition of eye states in real time video.
*International Conference on Computer Engineering and Technology, ICCET'09*2009, 554-559.Google Scholar - Hong T, Qin H, Sun Q: An improved real time eye state identification system in driver drowsiness detection.
*IEEE International Conference on Control and Automation*2007, 0: 1449-1453.Google Scholar - Wu Y-S, Lee T-W, Wu Q-Z, Liu H-S: An eye state recognition method for drowsiness detection.
*IEEE 71st Vehicular Technology Conference*2010, 1-5.Google Scholar - Flores MJ, Armingol JM, de la Escalera A: Driver drowsiness warning system using visual information for both diurnal and nocturnal illumination conditions.
*EURASIP J Adv Signal Process*2010, 2010: 1-20.View ArticleGoogle Scholar - Viola P, Jones M: Rapid object detection using a boosted cascade of simple features.
*Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001*2001, 1: I-511-I-518.Google Scholar - Smach F, Atri M, Mitéran J, Abid M: Design of a neural networks classifier for face detection.
*Eng Technol*2005, 124-127.Google Scholar - Huang C, Ai H, Li Y, Lao S: High-performance rotation invariant multiview face detection.
*IEEE Trans Pattern Anal Mach Intell*2007, 29(4):671-686.View ArticleGoogle Scholar - Jin Z, Lou Z, Yang J, Sun Q: Face detection using template matching and skin-color information.
*Neurocomputing*2006, 636-645.Google Scholar - Jie Y, Xufeng L, Yitan Z, Zhonglong Z: A face detection and recognition system in color image series.
*Math Comput Simulat*2008, 77(5-6):531-539. 10.1016/j.matcom.2007.11.020View ArticleGoogle Scholar - Zhang P: A video-based face detection and recognition system using cascade face verification modules. In
*Proc of the 2008 37th IEEE Applied Imagery Pattern Recognition Workshop*. Washington DC; 2008:1-8.View ArticleGoogle Scholar - Vezhnevets V, Degtiareva A: Robust and accurate eye contour extraction. In
*Proc Graphicon-2003*. Moscow, Russia; 2003:81-84.Google Scholar - Caltech vision lab[http://www.vision.caltech.edu/html-files/archive.html]. Accessed 20 July 2011
- Eye SBU database[http://faculties.sbu.ac.ir/~eshghi/sbu-database-9008/one_eye.rar]
- Xilinx co[http://www.xilinx.com]. Accessed 2 May 2011

## Copyright

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.