Adding Image Constraints to Inverse Kinematics for Human Motion Capture

Abstract

In order to study human motion in biomechanical applications, a critical component is to accurately obtain the 3D joint positions of the user's body. Computer vision and inverse kinematics are used to achieve this objective without markers or special devices attached to the body. The problem of these systems is that the inverse kinematics is "blinded" with respect to the projection of body segments into the images used by the computer vision algorithms. In this paper, we present how to add image constraints to inverse kinematics in order to estimate human motion. Specifically, we explain how to define a criterion to use images in order to guide the posture reconstruction of the articulated chain. Tests with synthetic images show how the scheme performs well in an ideal situation. In order to test its potential in real situations, more experiments with task specific image sequences are also presented. By means of a quantitative study of different sequences, the results obtained show how this approach improves the performance of inverse kinematics in this application.

1. Introduction

In biomechanical applications that aim to study human motion, a critical component is to accurately obtain the 3D joints' positions of the user's body. Usually, the most common methods to obtain the joints' positions require a laboratory environment and the attachment of markers to the body. Modern biomechanical and clinical applications require the accurate capture of normal and pathological human movement without the artifacts associated with standard marker-based motion capture techniques such as soft tissue artifacts and the risk of artificial stimulus of taped on or strapped on markers [1]. Emerging techniques and research in computer vision are leading to the rapid development of the markerless approach to motion capture [2].

In computer vision, algorithms are designed to allow the system to analyze one or multiple image streams in order to recover human motion. However, the images are 2D and the human body representation is in 3D. This fact leads to the presence of ambiguities; there are a number of possible 3D configurations of the human body that could explain a single image. In addition, these images can be noisy or incomplete (some joints or limbs are not visible). Therefore, we can only estimate the users posture. Inverse kinematics approaches can solve the body posture from their 3D position if we can clearly locate visible body parts such as face and hands. For example, in the work of Zou et al. [3], the angles of joints are estimated by inverse kinematics based on human skeleton constraints, and the coordinates of pixels in the body segments in the scene are determined by forward kinematics. Finally the human motion pose can be reconstructed by histogram matching. Their main drawback is that the algorithm does not handle human motion in the direction perpendicular to the image plane displacement. In the case of multiple cameras, ambiguities appear to be less significant. For example, by using two cameras to recover the user's posture in order to recognize the user's gestures for Human-Computer Interaction applications [4]. This work is also based on computer vision and inverse kinematics in order to recover human body posture.

However, previous works based on the combination of computer vision and inverse kinematics tend to simplify the approach by combining the results of both techniques that are applied by separate. As it is shown in Figure 1, current approaches are based on the detection of certain joints into the images by using Computer Vision algorithms and, therefore, estimate a plausible body posture by using Inverse Kinematics algorithms. In this way, the Inverse Kinematics algorithms are "blinded" with respect to the projection of body segments into the images. They only use biomechanical constraints to estimate the best position of the non-detected joints in the images.

Figure 1

Current approach to combine computer vision and inverse kinematics.

In this paper we present a new approach where the objective is to include the image information directly onto the inverse kinematics scheme, see Figure 2. The idea is to use image constraints to solve the redundancy of kinematics solvers. In addition, we also explain that it is possible to use a preprocessed image, in other words, the computer vision algorithms could process the input images in order to make the problem more tractable or to enhance a desired image feature for specific applications. Finally, this scheme of posture reconstruction can be used with one o more views, that is, it can work using only one view but the results improve if more views of the performer are applied.

Figure 2

Our approach: Adding image constraints to inverse kinematics.

In order to show the viability of this scheme of 3D human posture recognition, different experiments have been conducted. First, using synthetic images in order to show how the scheme works in an ideal situation. This simple case shows how theoretically the scheme performs correctly. Next, in order to test its potential in real situations, experiments with real sequences are also presented. In these experiments an annotated sequence and a known database of human motions that contains motion capture data are used to make a quantitative study of different sequences in order to evaluate the performance of the presented approach.

This paper is organized as follows. In next section, current inverse kinematics approach is reviewed in order to introduce in Section 3 the image constraints. Section 4 analyzes the obtained results in order to demonstrate the viability of this approach. Finally, conclusions are presented in the last section.

2. Inverse Kinematics

In order to capture human motion, the human body is usually modeled as an articulated chain, which consists of a set of rigid objects, called links, joined together by joints. To control the movement of an articulated chain it is common to use inverse kinematics (IK). IK is exploited to reconstruct an anatomically correct posture of the user (i.e., its joint state) considering the 3D locations of selected end-effectors which are used to constrain the posture.

For the moment, let us consider only a single frame of motion. Write the vector of joint angles as . Assume that we would like to meet a set of constraints on joints positions as as functions of the joints degrees of freedom . The problem of inverse kinematics is to obtain a such that . Closed forms solutions are available for at least some parameters, when the limbs of the articulated chain are considered independently [5]. More often, one must see this as a numerical root finding problem based on the linearization of the set of constraints on joint positions, , considering small displacements about the current configuration, ,

(1)

where is the Jacobian matrix

(2)

The resulting Jacobian matrix is inverted to map the desired constraint variation to a corresponding posture variation . Using the pseudoinverse, noted by , the norm of the solution mapped by is minimal, that is, it is the smallest posture variation realizing the desired constraint variation:

(3)

Since there is an infinite number of solutions. For the positioning and animation of articulated figures in computer graphics, the weighting strategy [6] is frequently employed. In the field of robotics however, the strategy is to solve this inverse kinematics redundancy adding a secondary term (usually defined as secondary task) to (3) in order to minimize a criterion . In this formulation, redundancy solution is accomplished by moving the joints such that the end-effectors are moved in the desired way and the criterion is always kept at a minimum. This was first exploited by Liégeois [7] who added a secondary task by projecting the negative gradient of into the null space of , see (4),

(4)

where is the identity matrix, and is a positive gain factor which is configuration dependent. Definition of the secondary task by means of the criterion depends on the application. In following section, a criterion based on images will be defined in order to capture human motion.

3. The Image-Based constraint

As explained in previous section, it is possible to constrain the solutions of inverse kinematics by adding a scalar criterion . Next, we explain how to define this criterion by using the images in order to guide the posture reconstruction of the articulated chain for human motion capture applications.

The definition of the image-based constraint is inspired in the works of Visual Servo Control [8]. Specifically, Marchand and Courty define different secondary tasks for controlling a camera in virtual environments [9]. For motion capture purposes, it should be taken into account that the human structure is highly redundant and, therefore, a large solution space exists. A solution is to generalize (4) to include more tasks by using the priority strategy [10]. In this case, the solution guarantees that a task associated with a high priority will be achieved as much as possible, while a low-priority constraint will be optimized only on the reduced solution space that does not disturb all higher priority tasks. However, for the sake of clarity, we only consider two tasks. It is straightforward to extend to more tasks when the image constraint has low priority. In addition, it is possible to use the Extended Jacobian method [11] in order to give a high priority to the image constraint.

For motion capture applications, we define in order to maximize the overlap between the projection of the articulated chain into the images and the human body. Consider the case of the Figure 3(a), where is shown the initial configuration of the articulated chain and the objective is to estimate the elbow's position with the 3D position of the hand as end-effector. By applying IK we obtain the result of Figure 3(b), where the elbow's estimation lies outside of the body due its "blind" nature by using only the desired position of the end-effector. In order to solve this problem, we propose a criterion that tries to guide the articulated chain to the body projection into the image, Figure 3(c). Formally, let us to define as follows

Figure 3

(a) Initial configuration of the articulated chain, (b) IK estimation, (c) result by adding the image constraint.

(5)

where represents the intensity of the 2D point () of the image (), which corresponds to a different view of the user, and is the number of points that belong to the desired image support . Applying a background subtraction algorithm [12], it is possible to directly use the silhouette as the image support, , of the articulated chain, see Figure 4(a). However, in order to get a smooth surface we apply the euclidean distance transform [13] to the silhouette image, see Figure 4(b). Both operations are fast and do not introduce a significative delay in the algorithm.

Figure 4

Image supports: (a) silhouette; (b) euclidean distance transform.

In order to complete the definition of (5), let us to define the function , which is the projection of the articulated chain into the image . If are the coordinates of the i th joint in the 3D-space, and assuming knowing the calibration data in order to project the 3D coordinates into the 2D images, we define such as the 2D image coordinates of the projected i th joint into the image . Assuming that the joints are ordered in a consecutive way, the function is defined as follows:

(6)

where is the segment between the 3D joints' projection into the image. Figure 5 shows the function for the example of Figure 4 and its partial derivatives. Concluding, the image contraint is then given by the gradient of the criterion of(7)

Figure 5

The function and its partial derivatives for the case of Figure 3.

(7)

where the partial derivative of the joint is defined in(8)

(8)

4. Performance Evaluation

The proposed approach is evaluated using three different tests. The first test uses a virtual environment to show how the presented approach runs well in an ideal situation. The second test applies the proposed approach on a sequence of user's motions to show how the presented approach performs well using real images. Finally, the third test compares the evaluation of the inverse kinematics approach, with and without image constraints, by using HumanEva dataset [14]. This dataset comprises four subjects performing six different types of actions recorded in seven calibrated video sequences from different viewpoints. Additionally, the video sequences are synchronized with their corresponding motion captured 3D pose parameters.

In addition, the complete algorithm (with and without image constraint) has been implemented in Visual C++ using the OpenCV libraries [15] and it has been tested in a realtime interaction context on an Intel Core2 QUAD Q6600 under Windows Vista. First, without the image constraint, we have obtained a performance of 21 frames per second (with 15 steps of convergence). Second, with the image constraint, we have obtained a performance of 19 frames per second (with 15 steps of convergence). Therefore, its use in human-computer interaction applications is also possible. For other uses in non real-time applications the accuracy could be improved adding more steps of convergence.

4.1. Virtual Environment

First, we test the system in a virtual environment to show how the presented approach works in an ideal situation. We define an articulated chain in 2D space, composed by 4 segments, with a total of 4 rotational joints of 1 DOF each one (i.e., a rotational joint for each segment). To test the system, we generate an initial configuration and an objective configuration of the articulated chain. Next, we apply the inverse kinematics approach, from initial configuration with and without image constraint, to estimate the objective configuration of the articulated chain.

In the first experiment, displayed in Figure 6, we generate an initial configuration and an objective configuration of the articulated chain, we apply the inverse kinematics approach, with and without image constraint. Without the image constraint, when the articulated chain reaches the end-effector the estimation of the objective configurations stops. On the other hand, by using image constraint it continues to try inserting the articulated chain inside the projection of the objective configuration, even if the articulated chain reaches the end-effector. In Figure 7 we display a second experiment where we generate an initial configuration and an objective configuration of the articulated chain, with the same end-effector. We apply the inverse kinematics approach, from the initial configuration, with and without image constraint. The results show that the inverse kinematics without the image constraint estimation does not change the initial configuration because the end-effector is reached. On the other hand, the inverse kinematics with image constraint enforces the chain to reach the image projection of the objective configuration. Figure 8 shows the last experiment in virtual environment, where the articulated chain tries to avoid an object projected in an image. We define the initial configuration of the articulated chain, the objective end-effector, and a squared object. In this case, we use the image constraint to avoid the object. The experiments show that adding the image constraint outperforms inverse kinematics, in order to achieve the desired articulated chain configuration.

Figure 6

Experiment  1. From the initial configuration of the articulated chain (gray), both approaches try to reach the objective configuration (black). Left column, sequence applying inverse kinematics without image constraint. Right column, sequence applying inverse kinematics with image constraint.

Figure 7

Experiment  2. Initial (gray) and objective (black) configuration with the same end effector position. Left column, sequence applying inverse kinematics without image constraint. Right column, sequence applying inverse kinematics with image constraint.

Figure 8

Experiment  3. From initial configuration of the chain (gray) the experiment try to avoid an object (black). Left column, sequence applying inverse kinematics without image constraint. Right column, sequence applying inverse kinematics with image constraint.

4.2. Using Real Images

In this test we apply the inverse kinematics with image constraint approach on a real stereoscopic sequence of human motions. Besides, the 3D joints' positions of the sequence are manually annotated for a quantitative comparison. The sequence has 450 frames corresponding to 15 seconds in real-time. The main objective of this test is to show that the proposed approach performs well with real images. In addition, this experiment shows that using this approach it is possible to solve the problem using only one image.

In Figure 9 is shown a frame of the stereoscopic sequence while applying the inverse kinematics approach with and without image constraint. We define an articulated chain in 3D space, with 2 rotational joints of 3 DOF each one. In the case of using image constraint, we apply the approach firstly by using the left camera () and then, by using both cameras (). The results show that when the inverse kinematics approach loses the elbow position, the inverse kinematics image constraint approach estimates the position of the elbow inside the silhouette. Using this stereoscopic sequence, there are not any significant differences between using one or both cameras.

Figure 9

Working with real images. (a) Initial configuration of the articulated chain. (b) Arm estimation applying inverse kinematics approach. (c) And (d) arm estimation applying inverse kinematics with image constraints using one and two views respectively.

In order to perform a quantitative evaluation the mean squared error is used. Formally, the error between an estimated 3D joint and the truly performed one from ground truth data is computed as

(9)

where is the number of frames. Specifically, we compare the manually annotated elbows positions and the estimated elbows positions using the inverse kinematics with image constraints in the case of one view and in the case of two views. In addition, we also can compare with the results of the priority inverse kinematics (PIK) approach [16] because the same test sequence is used to evaluate its performance. Table 1 summarizes the results that shows how the inverse kinematics with image constraints has less error. It can also be observed how using the second view does not significantly improve the results.

Table 1 Comparison by using the manually annotated sequence.

4.3. HumanEva Test

This test evaluates our system using two views of two sequences of real motions, walking and box, of the subject 1 of HumanEva dataset. These sequences have a total of 3050 frames per view. Due to the fact that this database also contains the 3D positions of the joints by using markers, the objective of these experiments is to make a quantitative evaluation of our approach. We define an articulated chain in 3D space, with 2 rotational joints of 3 DOF each one, in order to estimate the configurations of the arm (for the box sequence) and the leg (for the walking sequence).

By using the mean square distance of (9), Table 2 shows the obtained results applying inverse kinematics with and without the image-based constraints.

Table 2 Overall error of the estimation of the 3D positions of the internal joints for the two sequences (the elbow in the case of the arm and the knee in the case of the leg) for the two sequences.

Visual results for the box sequence are shown in Figures 10 and 11, where it is possible to see how the inverse kinematics approach loses the elbow position, and how, by adding the image-based constraints, the elbow's estimation lies inside the silhouette. This fact can also be observed in the graphic of Figure 12 where the elbow's estimation error by frame of the two approaches is displayed.

Figure 10

HumanEva box sequence for the subject 1, using Inverse kinematics.

Figure 11

HumanEva box sequence for the subject 1, applying Inverse kinematics using image-based constraints.

Figure 12

Elbow's estimation error by frame of the box sequence.

Figures 13, 14 and 15 show the results of the walking sequence, where a similar performance than previous sequence can also be observed . In this case, by adding the image-based contraint the recovered motion of the leg is more natural, avoiding the artifacts caused by the inverse kinematics approach in the knee's estimation.

Figure 13

Knee's estimation error by frame of the box sequence.

Figure 14

HumanEva walking sequence for the subject 1, applying Inverse kinematics estimation.

Figure 15

HumanEva walking sequence for the subject 1, applying Inverse kinematics using image-based constraints.

5. Conclusion

The invasiveness of the sensor system, the high dimension of the posture space, and the modeling approximations in the mechanical model of the human body are sources of errors that accumulate and result in an approximate posture that could not be sufficient in biomechanical applications that study human motion precisely. Specifically, applications based on computer vision and inverse kinematics approaches presents the problem that no information was available to locate the internal joints and this forced the IK approach to make a somewhat arbitrary decision about what was the optimal angle for these joints.

In this paper, we present how to add image constraints to the inverse kinematics formulation in order to solve this problem. We have proposed a criterion that tries to guide the articulated chain to the body projection into the image. In this way, impossible chain configurations are avoided. Experiments using synthetic images show how this approximation performs correctly and, how to solve difficult situations that occur when there are motions that do not imply to the end-effectors. Besides, we have evaluated our approach using real images, including sequences of a known human motion database in order to compute quantitative results. The computed error, about 2 centimeters, can be considered as sufficiently small to permit its use in motion capture applications. Moreover, adding the image constraint implies that the solution of the kinematic chain is more independent on initial configuration.

As future work, we plan to generalize this approach to include more tasks by using the priority strategy. In this way, it would be possible to use more complex models of the human body in to order to achieve better estimations.