A 2D eye gaze estimation system with low-resolution webcam images

Gaze information plays an important role in point of regard (PoR) which is a proxy for handling the users' attention or intention as an interaction between human and the computer where it can be used as a form of input instead of keyboard and the mouse [1–3]. Every single eye gaze tracking system has different visual angle accuracies which are important for the size of selection targets such as buttons, icons, pictures, and texts [4]. The systems with lower visual angle accuracy are more robust in terms of selection of targets and visual angle accuracy is generally used for benchmarking of the eye gaze systems [5].

Various systems have been described in the literature. Deng et al. [6] present a local deformable template method for locating the eye and extracting eye features. A system based on a dual-state model for eye tracking is proposed in [7]. Both of these methods require manual initialization of eye location. Witzner et al. [8] model the iris as an ellipse, but the ellipse is locally fitted to the image through an EM and RANSAC optimization process. A system based on feature detection using one camera is suggested by Smith et al. [9]. Noureddin et al. [10] suggest a two-camera solution where a fixed wide-angled camera with rotating mirror directs the orientation of the narrow-angled camera. A practical gaze point detecting system with dual cameras is also proposed by Park et al. [11]. They employ IR-LED illuminators for wide and narrow view camera to overcome the problem of specula reflection on glasses. They attain the 2.89 cm of RMS error between the computed and the real positions. A non-intrusive eye gaze estimation system is proposed by Yoo et al. [12]. Their system allows ample head movements and yields quite accurate results using cross-ratio under large head motion. Artificial neural network (ANN) is also applied into eye gaze estimation in the literature [13]. This method employs cropped images of eyes for training the activation functions. It requires calibration and training process which is not applicable in everyday life.

Guestrin et al. [14] claim that the point of gaze (POG) can be estimated only if the head is completely stationary when the system employs one camera and one light source. Therefore, head-movements are the major problem of low-cost 2D eye gaze trackers. This article neglects the head-movements and estimates the approximate POG using shape- and intensity-based methods.

The remaining part of the article is organized as follows. In Section 2, insights of the face and eyes socket detection are presented. In Section 3, deformable angular integral search by minimum intensity (DAISMI) algorithm is introduced. In Section 4, deformable template-based 2D gaze estimation (DTBGE) algorithm with mathematical explanation is described. In Section 5, a dwell-time-based virtual keyboard is employed to measure the visual angle accuracy of the system. Experimental evaluation and benchmarking of the system are given at the end. Section 6 gives some discussion and concludes the paper.

The fundamental requirement of a simultaneous eye tracking and gaze detection system is to accurately detect the eye sockets, which can easily be achieved by Haar-like object detectors which is formerly proposed by Paul Viola [15] and latterly extended by Rainer Lienhart [16]. It allows a classifier trained with sample views of a particular object to be detected in a whole image. The advantages of this method are that when implemented correctly it is fast, efficient, and accurate. It is also effective in detecting objects which are either partially occluded or if the video frame is noisy. By applying it in a multistep detection approach, it is possible to detect precisely the eye region by first recognizing the face and then by setting a region of interest on it. It needs the correct Haar-cascade descriptors for front face and two-eye region. Santana et al. [17] present fully trained Haar-cascade descriptors which are employed in our system and classified by means of Intel OpenCV library. Figure 1 shows a successful screenshot for face and eye socket detection.

The detected eye sockets contain eyeball, eye brow, and eyelashes together. However; in most of the intensity-based trackers, eye brows and eyelashes are described as an obstacle for robust eyeball detection. It is physically known that pupil is the darkest contour in an eye-socket; however, eye brows and eyelashes are also dark and have similar intensity values with pupil. This makes everything difficult and prevents developing robust eye tracking systems.

The mass-center of an eye-socket is considered as the most probable pupil location in terms of the less weighted intensities [18]. If the mass-center of an eye-socket is consecutively updated, then the resulting center of mass will be much closer to the pupil-center in each loop-step. From this point of view, the mass-center of an eye-socket is derived in x number of steps in terms of size and position. Gaussian blurring is employed before the processing. Size of the socket is reduced by p percentage rate in each step. Our system employs x as 2 and p as 15%. Finally, eye-socket is reduced in size which mostly excludes the eye-brows. Equation 1 describes the center of mass estimation as follows:

(1)

where c is the position of mass-center, P is the position, and I is the intensity of each pixel within the eye-socket which contains n number of pixels. Figure 2 shows screenshots from the socket reduction process.

The sockets derived from the original ones show more precise location of the eyeball. However, center of mass mostly overlaps on the eyelashes which are still very close to the eyeball. Finally, the derived sockets are employed as eye estimates in which DAISMI is employed to detect the stable approximate pupil-center position by averaging the positions of eyeball border vertices which is necessary for initial template orientation of DTBGE and the final movement displacement vector.

In this section, a binary deformable eyeball template is modeled and 2D gaze estimation is performed depending on the displacement in eye movements. At any horizontal and vertical eye movement, the displacement vector is estimated with respect to the previous frame.

The algorithm consists of four steps as follows:

4.1 The basic configuration

The algorithm aims to decide on the direction of the eyeball movement using a deformable template as shown in Figure 3. In other words; DTBGE is a kind of noise filter which avoids incorrect decisions caused by noisy pupil-center coordinates. DAISMI finds the approximate window size which is basically considered as the Euclidean distance between template origin and eyeball border vertices. DTBGE creates another template for gaze direction detection using the window size estimated by DAISMI and converts the source image into binary using p-tile thresholding algorithm which automates the thresholding ratio frame-by-frame depending on the varying density ratio between black and white pixels over the detected eyeball. Figure 6 shows the orientation of windows on binary source image.

Template is created with eight windows though it may vary depending on the desired accuracy rate. Considering the memory workload and speed of the system, the best results are taken with eight windows though higher number of windows is more precise for decision making.

In this regard; the internal energy of a single window is defined as follows:

(2)

According to Equation 2, black pixels with '0' intensity has a negative impact on the internal energy of a single window while the white pixels with '255' intensity affect the result positively.

4.2 Adjustment of template size

According to the definition of internal energy of a single window in Equation 2, the size of the template varies depending on the angular integral of window energies with angle θ.

(3)

According to the Equation 3, the size of the template increases when the angular integral of window energies is less than '0' and decreases when it is greater than '0'.

Figure 7 shows a deforming eyeball model in two cases: shrinking and expanding. Decision is made according to Equation 3. In both cases; eyeball size is updated and a new template is created in each frame using the updated window size.

4.3 Horizontal movement of the object

Decision on the horizontal movement of deformable template is made by cosine functions (Figure 8). Cosine functions produce negative values when the angle θ is between 90° and 270°. Using Equation 4, the direction of the horizontal movement is determined.

(4)

According to Equation 4, the decision on the horizontal movement is made as to the right-hand side when the angular integral of cosine functions of each window's internal energy is less than '0', conversely (Figure 9); it is to the left-hand side when it is greater than '0'.

E_{move (x )} > 0, the decision is made to (←) direction.

E_{move (x )} < 0, the decision is made to (→) direction.

Finally, horizontal movement is pumped.

4.4 Vertical movement of the object

Decision on the vertical movement of deformable template is made by sine functions (Figure 10). Sine functions produce positive values when the angle θ is between 0° and 180°. Using Equation 5, the direction of the vertical movement is determined.

(5)

According to Equation 5, the decision on the vertical movement (Figure 11) is made as to up-direction when the angular integral of sine functions of each window's internal energy is less than '0'; conversely, it is to down-direction when it is greater than '0'.

E_{move (y)} > 0, the decision is made to (↓) direction.

E_{move (y)} < 0, the decision is made to (↑) direction.

Finally, vertical movement is pumped.

The calibration procedure is performed as a promise between users and system. All the users are asked to look at the center of screen before the tracking starts. After the tracking starts, displacement vector in terms of stable approximate pupil-center position is created by DAISMI. DTBGE decides on the reliable movements in case of noisy displacement vectors. When the users move their eyes, the amount of pixel-wise pupil displacement filtered by means of DTBGE is oriented to the screen relative to the initial gaze point and screen size. The system assumes that the initial position of gaze is the center of screen.

The system is primarily developed and tested on a Windows XP PC with an Intel Pentium Dual CPU (T2390) with 1.86 GHz processor and 1.75 GB RAM. The monitor size is 12.1'' (307 mm) with 9.72'' (247 mm) width and 7.28'' (185 mm) height with 4:3 aspect ratios. Video is captured with a Logitech Quickcam Pro 4000 webcam at 30 frames per second. Video is processed as binary images of 160 × 120 pixels using various utilities from the Intel OpenCV library.

5.1 Eye tracking experiment

The experiment is conducted for testing the eye tracking system whether it can successfully locate the pupils of the users under different conditions. A dataset consisting of 300 images taken under varying lighting conditions, head positions, and with complex backgrounds is collected. The experimental results show that the performance rate can reach 100% at frame per second (FPS) = 15 while the camera input image size was 160 × 120. Figure 12 illustrates some of the successful examples.

5.2. 2D gaze estimation and mouse control experiment

The second experiment is performed by asking six users to move cursor on virtual keyboard buttons (Figure 13) which are generated relative to the screen size. Each user is given a brief introduction about how the system works and starts the experiment without any practice. Each user conducts the experiment five times (Table 2).

Button ID	EG	RG	UED	Duration(s)
User 1
1st Button	L	B	1	38.7
2nd Button	2	2	0	23.4
3rd Button	4	V	√2	12.9
4th Button	P	P	0	54.2
5th Button	X	Z	1	43.5
User 2
1st Button	F	F	0	23.3
2nd Button	O	Z	1	24.7
3rd Button	4	4	0	46.1
4th Button	SBAR	SBAR	0	14.3
5th Button	3	←	1	25.8
User 3
1st Button	R	G	√2	49.1
2nd Button	V	A	√5	34.7
3rd Button	9	SBAR	1	15.3
4th Button	4	W	√5	21.9
5th Button	D	D	0	19.3
User 4
1st Button	8	7	1	57.6
2nd Button	M	W	1	38.8
3rd Button	5	5	0	42.2
4th Button	T	1	√5	19.5
5th Button	E	F	1	26.7
User 5
1st Button	X	X	0	35.1
2nd Button	F	F	0	24.0
3rd Button	I	R	√2	34.6
4th Button	Y	N	1	33.9
5th Button	6	7	1	25.3
User 6
1st Button	C	M	1	31.1
2nd Button	R	Q	1	28.6
3rd Button	S	H	√2	13.8
4th Button	W	V	1	18.7
5th Button	O	P	1	47.7

EG, expected gaze; RG, resulting gaze; UED. unit error distance; SBAR, spacebar.

When the users are asked to gaze expected buttons, they might look at any pixel over the button. Therefore, it is impossible to measure the gaze accuracy in pixel-wise paradigm. Besides, the accuracy rate depends on the size of the selection targets which are buttons in this case. If the size of buttons is increased, then it is easier to hit the button, and then the accuracy of the system increases. In order to overcome this challenge, the relative unit error distance values which are calculated with respect to the true hits on buttons are used. If the unit error distance is '0', then it means that user hits the expected button successfully. If it is '1', then it means that user hits the neighbour button of the expected button. Then, the gaze error is assumed as 1 button. And if the unit error distance is ' ', then it means that user hits the button which is at the corner of expected button (horizontally: 1 and vertically: 1 button far from the expected button). Lastly, if the unit error distance is ' ', then it means that user hits the button which is also at the corner of expected button but in this case it is horizontally: 1 and vertically: 2 (or the reverse) button far from the expected button. According to the experimental results in Table 2 users can hit the expected button in different durations. Durations do not show the elapsed time between initial gaze button and expected gaze button, but rather they show the total elapsed time starting from the tracking system, move the cursor to initial gaze, and move to expected button. In fact, the elapsed time between initial gaze button and expected button is in millisecond which is ignored. Users are observed that they are in difficulty to bring the cursor to initial gaze button. However, even they cannot succeed in the true hit, their resulting gaze is so close to the expected gaze in terms of unit error distance. Therefore, unit error distance values are not employed in terms of gaze accuracy evaluation. That's why most of the gaze trackers are evaluated with respect to their visual angle accuracy. The visual angle is the angle a viewed object subtends at the eye, usually stated in degrees of arc [20]. Figure 14 shows how to measure the visual angle of a subject's eye looking at an object has a certain distance from the eye.

The visual angle V is measured directly using a theodolite placed at point O and is calculated using the formula [20]:

(6)

In order to measure the visual angle accuracy for the system, the minimum true saccadic displacement depending on users' eye movements is needed. For our system, according to the experimental results in Table 2, minimum saccadic displacement is 1 unit button.

In our virtual keyboard application, 1 unit button's size is 75 (px) × 75 (px). However, this size is relative to the screen size and the resolution. In order to find the real size in millimetre, the pixel unit is converted to millimetre unit, firstly.

According to the system settings in Table 3 and the formulas in Equations 6-8, visual angle is estimated. It cannot be calculated precisely because it is assumed that users' distance to the monitor is 400 mm and the minimum saccadic displacement is 1 unit button. These inputs are not precise but very close to the reality.

Screen resolution
Width	1280 pixels
Height	800 pixels
Screen dimensions (1 inch = 2.54 cm)
Width	247 mm
Height	185 mm
Subject
Distance to the monitor	400 mm

Finally, horizontal and vertical visual angle are estimated. According to Table 4, the horizontal visual angle is 2.07° while the vertical visual angle is 2.48°.

What is the angle for minimum saccade displacement?
Horizontal	Saccade width	75 pixels
	Angle	2.072829°
Vertical	Saccade height	75 pixels
	Angle	2.48392°

(7)

(8)

Finally, the visual angle accuracy of the system is considered as around 2°.

In this article, a shape- and intensity-based deformable eye pupil-center detection and movement decision algorithms are introduced in terms of developing a low-cost 2D gaze estimation and cursor control system regardless of the real position of pupil-center. In many other algorithms which are based on PoR information, real pupil-center position is required to get rid of noisy eye movements. However, unlikely, DTBGE employs a new deformable template model which decides on the eye movements' directions in binary source images regardless of knowing the real pupil-center. DAISMI finds the stable approximate pupil-center location by searching deformable eyeball models which includes low-intensity pixels within the template windows. It finds that the position of the eyeball with border vertices and the mass-center of vertices are considered as the stable approximate pupil-center position. Even the origin of the template is vibrating and noisy, the border vertices are almost stable and less vibrating. The stable approximate position of pupil is enough for getting robust eye movement detection and corresponding pupil-center displacement. This system is very useful for those who want to build 2D gaze estimation and mouse control systems with low-resolution source images such as simple webcams without infra-red filter. Experimental results show very encouraging results such that the visual angle accuracy is around 2° while the speed is 15 fps.

The comparison of the system with other systems in the same category is as follows: