Craniofacial reconstruction based on a hierarchical dense deformable model

Abstract

Craniofacial reconstruction from skull has deeply been investigated by computer scientists in the past two decades because it is important for identification. The dominant methods construct facial surface from the soft tissue thickness measured at a set of skull landmarks. The quantity and position of the landmarks are very vital for craniofacial reconstruction, but there is no standard. In addition, it is difficult to accurately locate the landmarks on dense mesh without manual assistance. In this article, we propose an automatic craniofacial reconstruction method based on a hierarchical dense deformable model. To construct the model, we collect more than 100 head samples by computerized tomography scanner. The samples are represented as dense triangle mesh to model face and skull shape. As the deformable model demands all samples in uniform form, a non-rigid registration algorithm is presented to align the samples in point-to-point correspondence. Based on the aligned samples, a global deformable model is constructed, and three local models are constructed from the segmented patches of the eye, nose, and mouth. For a given skull, the global and local deformable models are matched with it, and the reconstructed facial surface is obtained by fusing the global and local reconstruction results. To validate our method, a face deformable model is constructed and the reconstruction results are evaluated in its coefficient domain. The experimental results indicate that the proposed method has good performance for craniofacial reconstruction.

Introduction

Craniofacial reconstruction is an efficient method to get a visual outlook of an individual in the case of only skull and bone remaining. The traditional plastic methods[1–3] depend on the time-consuming manual work of artists. The reconstruction result is generally determined by the experience of practitioners. To reduce reconstruction time and eliminate subjective biases, different computer-aid craniofacial reconstruction methods have been proposed[4–17]. The state-of-the-art of the computer-aid craniofacial reconstruction have comprehensively been reviewed in the surveys[18–21]. The soft tissue thickness measured on skull is the foundation for craniofacial reconstruction. To get complete tissue thickness, the head samples are usually measured by different equipments such as computerized tomography (CT), magnetic resonance imaging and ultrasound scanner. Most computer-aid craniofacial reconstruction methods fit a selected facial template to the target skull according to the average soft tissue thickness at the skull landmarks[4–8]. Others deform a reference skull to match the remaining skull according to the skull feature such as anthropologic points[9], lines[10], and other features[11]. Applying an extrapolation of the skull deformation to the face template, the reconstructed face will be achieved.

The selection of the template or reference is vital for accurate craniofacial reconstruction. In general, a generic or a specific craniofacial template with similar shape attributes is chosen. But it is difficult to get suitable reference for every dry skull because of the diversity of skull and face modality. In addition, as the complex deformation between the reference and the target skull, the warping methods should intensively be studied to get accurate reconstruction result. So many deformation methods are proposed to model the non-rigid shape deformation of skull and face, such as radial basis functions (RBF)[22, 23], or more exactly, a thin plate spline (TPS)-based deformation[12, 24, 25] for its smoothness. Instead of using fixed template, the recently proposed statistical craniofacial reconstruction methods[12–17] construct a type of deformable model from a set of 3D heads by the principle component analysis (PCA) technique. The statistic deformable model can be regarded as a dynamic template for the given skull. The template deformation is a model fitting procedure driven by the difference between the input skull and the template, in which the model parameters are adjusted by optimization method. The reconstruction result of the deformable model depends on the diversity of samples in the 3D heads database. If there are sufficient samples, good reconstruction results will be achieved. So the statistic method is regarded as the dominant method with great potential application in practice.

Essentially, the craniofacial reconstruction is to figure out the face of unknown skull by the knowledge of skull and face dependency, which is concretely represented as the distribution of the tissue thickness on skull. Most current methods utilize the soft tissue thickness of a set of skull landmarks for craniofacial reconstruction, but it is considered not an ideal approach to model the relationship between face and skull. One reason is that the statistical soft tissue thickness at a set of sparse landmarks is far less than enough to reflect the whole distribution of tissue depth. The other reason is that the quantity and position of the landmarks are indefinite. Different landmark sets have been proposed for craniofacial reconstruction[26–31], though there are definite anatomical points in biometrics[32, 33]. Moreover, it is difficult to detect the landmarks accurately on the complex surface of skull without manual interactive work. In order to reflect the complete tissue thickness distribution and eliminate the disadvantages of the sparse representations, the methods which measure tissue depth at all points have been proposed. In these methods, the face and skull are generally represented in dense form. For examples, Tu et al.[34] constructed a face space for craniofacial reconstruction from the dense skull and face surfaces extracted from head CT images. Vandermeulen et al.[35] also used dense representations (implicit surfaces) for both skin and skull in craniofacial reconstruction. Pei et al.[22] presented a dense tissue depth image representation for craniofacial reconstruction, namely tissue-map. The dense tissue depth methods utilize more information of the relationship between skull and face, it generally has better craniofacial reconstruction results. To represent the dense tissue depth exactly, the dense point registration of skull or face is usually demanded. Although many registration methods[12, 24, 25, 36] have been proposed to construct correspondence between surfaces and point sets, it is still a challenging problem for further investigation because of the complex skull mesh with gross errors or outliers.

To the complex skull and face surfaces, the modality variety is composed of global shape and local detail. However, most current craniofacial reconstruction systems generally take the whole face or skull for shape analysis, while the local feature of skull and face is not emphasized. The recent research reveals that the local shape model is better than the global model to represent local shape variety[37, 38]. Inspired by this point, we propose a hierarchical craniofacial reconstruction model which integrates the global model with several local models to improve craniofacial reconstruction result. To construct the model, the face and skull samples are represented as dense mesh and aligned in point-to-point form by a proposed automatic dense registration algorithm, which contributes to a fully automatic craniofacial reconstruction method. In addition, to get valid evaluation for the craniofacial reconstruction results, we transform the reconstructed face into the coefficient domain of a face deformable model and the distance in the coefficient space is used as the similarity measurement. Comparing with the current measurement methods, such as the mean correspondence point distance or the Euclidean distance matrix[12], the proposed measurement is more suitable for face recognition.

The proposed hierarchical craniofacial reconstruction system is composed of four components, namely, the data acquisition and preprocessing, the global deformable model, the local deformable model, and the result evaluation (the dashed rectangles shown in Figure1). In the data acquisition and preprocessing component, skull and face data are acquired by CT scanner and the prototypic data are preprocessed to construct 3D skull and face surfaces. To construct the model, all samples are aligned by a proposed non-rigid dense mesh registration algorithm. In the global/local deformable model components, the global/local deformable models are constructed and the global/local face reconstruction results are obtained by model-matching procedure. Fusing the global and local reconstruction results gives the final craniofacial reconstruction result. In the result evaluation component, the reconstruction result is evaluated and the validation of the proposed method is verified. To integrate the four components as a whole system, some key problems, such as the dense mesh registration, the model matching procedure, the mesh segment, and fusion (the small rectangles shown in Figure1) should be solved. In the rest of the article, we will give the detail of the main components and the solutions to these key problems.

Figure 1

The framework of the hierarchical craniofacial reconstruction.

Data acquisition and preprocessing

In order to construct the craniofacial reconstruction model, we have constructed a head database from CT images. The CT images were obtained by a clinical multi-slice CT scanner (Siemens SOMATOM Sensation 16) in the affiliated hospital of Shaanxi University of Chinese Medicine located in western China. More than 100 patients planned for preoperative osteotomy surgery gave informed consent to scan the whole head for scientific research. The images of each subject are stored in DICOM 3.0 with 512×512 resolution. To get complete head data, 250 to 320 slices are captured for different persons. Most of the patients belong to the Han ethnic group in northern China. In this article, 110 samples are used for craniofacial reconstruction experiments. There are 48 female and 62 male subjects in the collection. The age distribution ranges from 20 to 60.

Each prototypic sample in our database consists of a skull surface and its face surface, which are extracted from CT images. The point clouds composing of the outer surfaces of skull and skin are extracted slice by slice (Figure2a). For the skull, a three-step method is performed on each slice of CT images. The first step is to find the contour points of the skull through the Sobel operator model after filtering out noise. In general, the skull contour has inter and outer sides (Figure2). In the second step, a circular scanning is implemented to get a rough outer contour. The scanning line is radiated from the center of the image, toward the four edges of the image, to find the farthest contour point it encounters. The rough contour usually contains some points not belonging to the outer surface, as shown in Figure2c. In the last step, these pseudo contour sections are removed and the missing parts are mended. By setting the max length threshold L, the section will be deleted if its length is smaller than L pixels. In our experiment, L is set to be 10. Because the skull is non-convex, the rough contour may disrupt in some regions where it should be connected. We adopt an 8-neighborhood boundary tracing approach to connect each point of the rough contour if they are disrupted, and obtain the final contour, as shown in Figure2d. It is easier for skin to find the outer contour, as the skin contour is simple and generally close for all CT images (Figure2e). So we only need to find a point in the above second step to get the outer skin contour, as shown in Figure2f. After retrieving the outer contours for skull and skin from all CT images, the skull and face surfaces can be represented as triangle meshes by the marching cube algorithm[39]. Usually, the raw skin surface consists of about 220,000 points with 450,000 triangles, while the skull surface contains about 150,000 points with 320,000 triangles, as shown in Figure2g–j. It is dense enough to describe the rich details of skull and face shapes.

Figure 2

Data acquisition and preprocessing. (a) The head slice captured by CT scanner. (b–d) The skull contour extracting procedure. (e, f) The skin contour extraction. (g–j) The reconstructed 3D skull and face in point cloud and triangle mesh formation. (k, l) The uniform coordinate system of the skull and face samples.

To eliminate the inconsistence of position, pose, and scale caused by data acquisition, all samples are transformed into a uniform coordinate system. The uniform coordinate system is determined by four skull feature points, the left and right porion, the left (or right) orbitale and the glabella, denoted by L _p , R _p , L _o , G. From three points, L _p , R _p , and L _o , the Frankfurt plane[40] is determined. The coordinate origin (denoted by O) is produced from the intersection of the line L _p R _p and the plane which contains point G and orthogonally intersects with L _p R _p . We take the line O R _p as x-axis. The line contains point O and has the same direction as the normal of the Frankfurt plane is set as z-axis. y-axis is obtained by the cross product of z- and x-axis. The scale of the samples is standardized by setting the distance between L _p and R _p to unit, i.e., every vertex (x y z) of the skull and face is replaced by$(\frac{x}{\left|L_p{R}_p\right|},\frac{y}{\left|L_p{R}_p\right|},\frac{z}{\left|L_p{R}_p\right|})$. The uniform coordinate system of skull and face is shown in Figure2k,l.

Dense registration for skull and face

The original skull and face meshes have different connectivity with different number of vertices. To investigate the modality variety of skull and face, the original skull and face samples must be registered before model construction. For the samples in dense triangle mesh representation, the registration is to build a point-to-point correspondence according to the shape features, such as the tip of nose, the corner of mouth, and the center of eyes. It is a challenging problem to get accurate registration for dense skull and face meshes because there exists non-rigid deformations and big-block outliers on the complex surfaces. To solve the problem of point or surface registration with non-rigid deformation, many methods and algorithms have been presented. The TPS-RPM method[41, 42] incorporated TPS into the framework of iterative closest point (ICP) and adopted a soft-assign and deterministic annealing optimization to compute correspondence between two point sets. As all points of the aligning objects are used to determine the TPS deformation and a correspondent matrix with dimension in quadratic cardinality of the point sets is used to eliminate outliers, this method is not applicable to the dense skull and face with big-block outliers. Hutton et al.[43] proposed an automatic 3D faces registration based on a dense face model. But in the model construction procedure, a set of landmarks should be picked up by manual work. The coherent point drift method[44] formulated the alignment of two point sets as a probability density estimation problem. The reference point set is represented as gaussian mixture model centroid and fitted to the target point set by the expectation maximization algorithm. This method does not need landmarks, but it converges slowly for the dense objects with large data volume and easily fails for big-block outliers. In this article, we present a two-step method to solve the dense registration of skull and face. The first step is to align two samples by a non-rigid registration method based on the TPS transformation[45]. The second step is to improve the registration of all samples by a group registration method based on a linear combination model. The proposed method is implemented automatically and has good performance for big-block outliers. The whole procedure of our registration method is shown in Figure3. The detail is given in the following sections.

Figure 3

The procedure of dense mesh registration based on TPS and linear combination model.

TPS-based non-rigid registration

As the skull and face have complex shape modality with non-rigid deformation, the traditional rigid registration methods, for example, the widely used ICP algorithm[46], are not suitable for this problem. So we adopt non-rigid registration method to solve the dense registration of skull and face. The basic idea of the method is shown in Figure4, not constructing registration between the reference and the target directly, we construct an approaching template for the target skull or face from the non-rigid deformation of the reference. As the deformed reference is closer to the target than the reference, it will produce more accurate alignment. Considering the advantages of TPS deformation, such as good smoothness constraint, simple calculation, and the ability to be decomposed into affine transformation component and non-affine transformation component, we adopt TPS to represent the non-rigid deformation. There are two steps to make the registration between the reference and the target. First, the reference is transformed to the target by TPS deformation. Then, the point-to-point correspondence is achieved by the closest point searching procedure of ICP. The detail of the TPS-based non-rigid registration method is given in the following.

Figure 4

The TPS-based non-rigid registration and the traditional rigid ICP registration.

To begin the registration, the reference skull and face must be selected. In general, the head with common shape feature is used as the reference. We select a sample with complete data as the reference. Considering that the craniofacial reconstruction is mainly determined by the front part of head, the occipital parts of the reference are manually removed to reduce data volume. The back-removed reference skull and face are shown in Figure5a,b. There are 36,000 and 40,969 vertices with 70,827 and 81,458 triangles on the reference skull and face, respectively. The other preparing work for the non-rigid registration is to get the controlling points for TPS transformation. As TPS is type of interpolation method, generally the TPS deformation depends on a set of correspondent controlling points on the reference and the target. But it is difficult to get plenty of correspondent feature points on face and skull automatically. It generally demands time-consuming manual work to locate the feature points on skull and face. To get automatic point registration, we use a random method to produce the controlling points for TPS, in which a mount of controlling points are generated randomly in the reference and its correspondent points on the target are obtained by ICP closest points searching method. To get uniform distribution on the 3D surface, the random controlling points are computed by the farthest point sampling method[47]. The random points on the reference skull and face are shown in Figure5c,d. As the correspondence obtained by the closest point searching method for the random points is not exactly the true correspondence according to skull and face feature, instead of making one-step transformation from the reference to target, we do the TPS transformation in a stepwise procedure. At the same time, to eliminate the influence of inconsistent correspondence of some points, the random points on the reference are updated timely. Integrating this controlling points generating procedure with the TPS deformation, the selected reference skull and face will gradually be aligned to the target skull and face.

Figure 5

The selected reference, the random controlling points for TPS, and the segmented patches for local model. (a,b) The back-removed reference colored in blue with its whole entities. (c,d) The random controlling points on the reference skull and face. (e–h) The segmented eye, nose, and mouth patches on the reference and its views in separating form.

For convenience, the selected reference skull or face is denoted by S_ref = {P _rp |P _rp = (x _rp ,y _rp ,z _rp ),p = 1,…,N₁}, and the target, the i th sample, is denoted by S _i = {P _iq |P _iq = (x _iq ,y _iq ,z _iq ),q = 1,…,N₂}, where N₁ and N₂ are the points number of S_ref and S _i such that N₁ ≤ N₂. Then the TPS transformation can be regarded as a map from S_refto S _i , denoted by f(.). The correspondent random controlling point sets of S_refand S _i are denoted by$M_r=\left\{L_{\mathit{\text{rj}}}\right|L_{\mathit{\text{rj}}}=(x_{\mathit{\text{rj}}}^{\ast },y_{\mathit{\text{rj}}}^{\ast },z_{\mathit{\text{rj}}}^{\ast }),j=1,\dots ,M\}$,$M_i=\left\{L_{\mathit{\text{ij}}}\right|L_{\mathit{\text{ij}}}=(x_{\mathit{\text{ij}}}^{\ast },y_{\mathit{\text{ij}}}^{\ast },z_{\mathit{\text{ij}}}^{\ast }),j=1,\dots ,M\}$, where M is the count of the controlling points. From the definition of TPS, f(.) will satisfy the following interpolation conditions:

$$f\left(L_{\mathit{\text{rj}}}\right)=L_{\mathit{\text{ij}}},j=1,\dots ,M$$

(1)

By TPS theory, the deformation of other non-controlling points is restricted by the blending energy function in the following form:

$$E\left(f\right)=\int \underset{R^3}{\int }\int I_3^T{F}_X{F}_X^T{I}_3\mathit{\text{dxdydz}}$$

(2)

where X=(x,y,z)^T,$F_X={(\frac{\partial f}{\partial x},\frac{\partial f}{\partial y},\frac{\partial f}{\partial z})}^T$ and I₃ = (1,1,1)^T. It is proved that TPS can be decomposed into affine and non-affine components[45]. This fact is generally represented as the following formula:

$$f\left(P\right)=\mathit{\text{Pd}}+\mathit{\text{Kw}}$$

(3)

where P∈S_ref with the homogeneous coordinate (1,x,y,z). d is a 4×4 affine transformation matrix. K is the TPS kernel, a 1×M vector in the form of K = (K₁(P),…,K _M (P)), where K _j (P) = ∥P−L _rj ∥,j = 1,…,M. w is a M×4 warping coefficient matrix representing the non-affine deformation.

To solve TPS transformation, the matrix d and w must be determined. There are two solutions to this problem, namely, the interpolating and non-interpolating methods. In the interpolating case, formula (1) is satisfied. Putting formula (3) into (1) and confining w to non-affine transformation, i.e.,$M_1^{\prime T}w=0$, it leads to a direct solution for d and w formed by the following matrix relation:

$$\left(\begin{array}{c}w\\ d\end{array}\right)=\left(\begin{array}{cc}{K}^{\prime }& M_r^{\prime }\\ M_r^{\prime T}& 0\end{array}\right)\left(\begin{array}{c}{M}_i^{\prime }\\ 0\end{array}\right)$$

(4)

where$M_r^{\prime }$ and$M_i^{\prime }$ are M×4 matrixes corresponding to the controlling points sets M _r and M _i in homogeneous coordinate form. K^′ is a M×M symmetry matrix representing the spatial relation of M _r with the element k _uv = ∥L _ru −L _rv ∥,u = 1,…,M,v = 1,…,M. In the non-interpolating case, formula (1) is not strictly satisfied. The following energy function can be minimized to find the optimized answer.

$$E^{\prime }(\lambda ,w,d)=\frac{1}{M}\sum _{j=1}^M\parallel L_{\mathit{\text{ij}}}-f\left(L_{\mathit{\text{rj}}}\right)\parallel +\lambda ·E\left(f\right)$$

(5)

where λ is the weight to control the blending component, and given a fixed λ there is a unique minimum for the energy function. It is conducted that the non-interpolating solution has a parallel form replacing K^′ in formula (4) by K^′ + λI. As the correspondent controlling points on the reference and target acquired by ICP closest points searching is not exactly correct, the condition in formula (1) is not satisfied. So the non-interpolating method is adopted in this article.

Having determined the TPS transformation f(.), the reference S_refcan be deformed by the formula (3). The deformed reference is denoted by$S_i^{\prime }$. Then the correspondence between$S_i^{\prime }$ and S _i can be obtained using ICP closest point searching. But the closest point searching procedure of ICP is a time consuming procedure with computation in O(N₁×N₂). To get high efficiency, we adopt a K-dimensional binary search tree (KD-tree)[48] to model the target, which is proved having a complexity with O(N₁×logN₂) for the pairwise closest point searching. Considering that the closest point matching based on the initial TPS deformation is more inaccurate and the beginning alignment refers to the global correspondence while the later to the local area, a deterministic annealing strategy is applied in the stepwise TPS-based registration procedure. At the beginning of the registration, the points move a little to its deformed points, and the step size increases gradually when TPS deformation result becomes desirable. At the same time, the number of the random points increase from a small initial number for enhancing the holistic deformation at the beginning, and the blending weight of TPS in (5) decreases to relax the global constrains. The following gives the implementation of the proposed non-rigid TPS-based registration method.

• 1. Create KD-tree for the i th sample S _i , denote it by T _i ;

• 2. Apply ICP alignment between S _ref, S _i , then transform S _refby the rigid transformation of ICP, the transformed sample is denoted by $S_i^{\prime }$;

• 3. Produce random controlling point set $M_i^{\ast }$ with cardinality of M for $S_i^{\prime }$;

• 4. For each point in $M_i^{\ast }$, search its correspondent point on S _i by querying on T _i , the correspondent point set is denoted by M _i ;

• 5. Determine the TPS transformation f from $M_i^{\ast }$, M _i with blending weight λ;

• 6. Apply the TPS transformation f on $S_i^{\prime }$, the deformed $S_i^{\prime }$ is denoted by $S_i^{\mathrm{\prime \prime }}$;

• 7. Update $S_i^{\prime }$ by adding a movement to each point $P^{\prime }\in S_i^{\prime }$:P ^′= P ^′ + δ(f(P ^′)−P ^′), where $f\left(P^{\prime }\right)\in S_i^{\mathrm{\prime \prime }}$ and δ is the step size;

• 8. For each $P^{\prime }\in S_i^{\prime }$, search its correspondent point P ^′′∈S _i by querying on T _i ;

• 9. Update the parameters:M = M + △M, δ = δ + △δ, λ = λ−△λ, where △M, △δ and △λ are the pre-assigned increments;

• 10. If the iterations l < l ₀ and $\frac{1}{N_1}\sum _{P^{\prime }\in S_i^{\prime }}\parallel P^{\prime }-P^{\mathrm{\prime \prime }}\parallel >{\epsilon }_0$, where l ₀ is the given maximum loops and ε ₀ is the given threshold, goto 3;

• 11. The final correspondence of S _ref and S _i is achieved from the equivalent correspondence of $S^{\prime }$ and S _i , denote it by $S_i^0$.

In our experiments, M ranges from$\frac{1}{500}$ to$\frac{1}{80}$ of the point number of the reference, l₀ = 30, ε₀ = 10⁻⁶, the initial δ = 0 with$△\delta =\frac{1}{l_0}$, and the initial λ = 0.01 with △λ = λ∗0.05.

Group registration by linear combination

It is important to select a closest reference for all samples to get good alignment, but the fixed reference may greatly differ with some samples as there is much variety in skull and face modality. Considering there are enough samples in our database, we try to improve the above registration by a group registration method based on a linear combination model. Instead of using a fixed reference, we utilize the combination of the above aligned samples to generate dynamic reference for every samples. As the dynamic reference is closer to the given sample, aligning the dynamic reference to the target sample will give better result. Based on the new correspondence result, we can construct new dynamic reference by linear combination, which will get more accurate aligning result. By this iterative procedure, the registration precise will be improved gradually. In the following, the iterative registration procedure by linear combination is described in detail.

If we regard the reference S_refas a N₁×1 vector in form of${(x_{r1},y_{r1},z_{r1},\dots ,x_{\mathit{\text{rp}}},y_{\mathit{\text{rp}}},z_{\mathit{\text{rp}}},\dots ,x_{r{N}_1},y_{r{N}_1},z_{r{N}_1})}^T$, then from the point-to-point correspondence, the N aligned samples$\left\{S_i^0\right|i=1,\dots ,N\}$ in the first-step can be formatted as vectors with the same form as S_ref, i.e.,$S_i^0={(x_{i1},y_{i1},z_{i1},\dots ,x_{\mathit{\text{ip}}},y_{\mathit{\text{ip}}},z_{\mathit{\text{ip}}},\dots ,x_{i{N}_1},y_{i{N}_1},z_{i{N}_1})}^T$, where the point$(x_{\mathit{\text{ip}}},y_{\mathit{\text{ip}}},z_{\mathit{\text{ip}}})\in S_i^0$ is the correspondent point of (x _rp ,y _rp ,z _rp ). By this representation, we can get a new object by the following linear combination:

$$S_{\text{new}}\left(\mathbf{a}\right)=\sum _{i=1}^N{a}_i{S}_i^0\text{s.t.}\sum _{i=1}^N{a}_i=1$$

(6)

where a = (a₁,…,a _N ) is the linear combination coefficient vector. For each original sample S _i , a dynamic reference$S_i^{\ast }$ can be determined by the following minimizing formula:

$$S_i^{\ast }=\sum _{i=1}^N{a}_i^{\ast }{S}_i^0\text{s.t.}{\mathbf{a}}^{\ast }=\text{arg}\underset{\mathbf{a}}{min}\parallel S_i-S_{\mathit{\text{new}}}\left(\mathbf{a}\right)\parallel$$

(7)

where ∥.∥ defined as the vector module representing the difference between two samples. The overall difference for all samples is defined as$\mathit{\text{Eg}}=\frac{1}{N}\sum _{i=1}^N\parallel S_i-S_i^{\ast }\parallel$. The following is the detail of the registration method.

• 1. Align S _refto each sample S _i by TPS-based method and get the primary aligning result $S_i^0$;

• 2. Produce the dynamic reference $S_i^{\ast }$ for each S _i by linear combination;

• 3. Align the dynamic reference $S_i^{\ast }$ to S _i by TPS-based method and get the aligning result $S_i^1$;

• 4. If the iterations are less than the given maximum loops and the global difference Eg is great than the given threshold, update $S_i^0$ by $S_i^0=S_i^1$ and goto 2;

• 5. Get the final aligning result from $\left\{S_i^1\right\}$.

The construction of the hierarchical deformable model

After computing dense correspondence, we can model all skulls and faces as the formation of the reference vectors by the point-to-point correspondence. If the triangle meshes of the reference skull and face are also applied on the correspondence points of the target, then all skulls and faces will have same mesh structure with same number of vertices. For convenience, we represent the i th head sample as a high dimension vector composed of skull and face vectors in the following form:

$$H_i=\left(\begin{array}{c}{S}_i\\ F_i\end{array}\right)i=1,\dots ,N$$

(8)

where$S_i={(x_{i1}^S,y_{i1}^S,z_{i1}^S,\dots ,x_{\mathit{\text{im}}}^S,y_{\mathit{\text{im}}}^S,z_{\mathit{\text{im}}}^S)}^T$ and$F_i={(x_{i1}^F,y_{i1}^F,z_{i1}^F,\dots ,x_{\mathit{\text{in}}}^F,y_{\mathit{\text{in}}}^F,z_{\mathit{\text{in}}}^F)}^T$ are the vectors of the i th skull and face with dimensions of 3m and 3n, respectively.

Similar to the dynamic reference construction in the above section, the linear combination of the aligned head samples {H _i |i = 1,…,N} will produce new skull and face. Given an unknown skull, the closest combination skull can be achieved by the model matching procedure. Extrapolating the combination of the skull vectors to the face vectors in the model will get a reconstructed face for the given skull. The detail of the model matching will be given in the next section. As the prototypic skull and face samples have high dimension data with large redundance, PCA is applied to construct the following deformable model:

$$H_{\text{model}}\left(\alpha \right)=\overline{H}+\sum _{i=1}^{N^{\prime }}{\alpha }_i{h}_i$$

(9)

where$\overline{H}=\frac{1}{N}\sum _{i=1}^N{H}_i$, {h _i |i=1,…,N^′} are the former N^′ components corresponding to the eigenvalues {σ _i |i=1,…,N^′} of the covariance matrix of the subtracting vectors$\{H_i-\overline{H}|i=1,\dots ,N\}$ in descending order. N^′is determined by 98% of the cumulative eigenvalues of the variance. The combination coefficient$\alpha =({\alpha }_1,\dots ,{\alpha }_{N^{\prime }})$ is the parameter for the deformable model. To generate a plausible face, the probability of α is constrained by the following formula:

$$P\left(\alpha \right)\sim \exp \left[-\frac{1}{2}\sum _{i=1}^{N^{\prime }}{\left(\frac{{\alpha }_i}{{\sigma }_i}\right)}^2\right]$$

(10)

The model in (9) is the global model referring to the modality of whole skull and face. To characterize the local shape variety, we construct several local deformable models with respect to the main organs of face, the eye, nose, and mouth. The first step for constructing the local models is to segment the organs. It is difficult to get an ideal automatic segment for different skull and face. As our samples have been aligned, getting the segments of the reference, the segments of other samples can be obtained from the correspondence of points. So we segment the local patches of the reference by hand. The segmented local patches of the reference are shown in Figure4e–h. Based on the segmented data, the local models can be constructed by the similar method of the global model. The hierarchical deformable model is constructed by integrating the local models with the global model.

Craniofacial reconstruction

For a given skull, the craniofacial reconstruction is a model matching procedure, in which the coefficients of the deformable model are adjusted iteratively and the model combination skull approaches to the given skull gradually. To measure the difference between the model combination skull S _md (α) and the given skull, after assigning the initial values for the combination coefficients, we align S _md (α) to the given skull by the TPS-based registration. From the obtained correspondence, we format the given skull as a vector S _gv in the same form as S _md (α). So the difference between the given skull S _gv and the model combination skull S _md (α) can be represented as the square module of the subtracting vector as following form:

$$\begin{array}{l}E\left(\alpha \right)=\parallel S_{\mathit{\text{md}}}\left(\alpha \right)-S_{\mathit{\text{gv}}}{\parallel }^2\\ ={\left(S_{\mathit{\text{md}}}\right(\alpha )-S_{\mathit{\text{gv}}})}^T\left(S_{\mathit{\text{md}}}\right(\alpha )-S_{\mathit{\text{gv}}})\end{array}$$

(11)

Regarding this definition as the cost function, the reconstruction problem can be solved by an optimization method. It is denoted that the model combination skull S _md (α) will change as the combination coefficients updating in the optimization procedure. So the registration between S _md (α) and S _gv is implemented every 20 loops to ensure the error in 11 computed in correct correspondence with the updated S _gv . To solve the optimization, we adopt a gradient descent algorithm to resolve the optimization problem. The core of the method is to find the gradient descent direction of E(α) about α, which is equal to the negative derivative of E(α). From 11, the partial derivative of E(α) can be formed as follows:

$$\begin{array}{ll}\frac{E^{\partial }\left(\alpha \right)}{\partial \alpha }=& {\left(\frac{S_{\mathit{\text{md}}}^{\partial }\left(\alpha \right)}{\partial \alpha }\right)}^T\left(S_{\mathit{\text{md}}}\right(\alpha )-S_{\mathit{\text{gv}}})+{\left(S_{\mathit{\text{md}}}\right(\alpha )-S_{\mathit{\text{gv}}})}^T\\ \times \left(\frac{S_{\mathit{\text{md}}}^{\partial }\left(\alpha \right)}{\partial \alpha }\right)\end{array}$$

(12)

From formula 12, the partial derivative of E(α) depends on$\frac{S_{\mathit{\text{md}}}^{\partial }\left(\alpha \right)}{\partial \alpha }$, which can be deduced from formula 9 as the form$\frac{S_{\mathit{\text{md}}}^{\partial }\left(\alpha \right)}{\partial \alpha }=\frac{\partial (\overline{S}+\mathbf{s}{\alpha }^T)}{\partial \alpha }=\frac{\partial \left(\mathbf{s}{\alpha }^T\right)}{\partial \alpha }$, where$\overline{S}$ is the skull part of$\overline{H}$,$\mathbf{s}=(s_1\dots s_i\dots s_{N^{\prime }})$ and s _i is the skull part of h _i . For a α _i ,$\frac{\partial \left(\mathbf{s}{\alpha }^T\right)}{\partial {\alpha }_i}=\frac{\partial \left(\sum _{i=1}^{N^{\prime }}{\alpha }_i{s}_i\right)}{\partial {\alpha }_i}=s_i$. Putting it into 12 we get the following partial derivative computation for α _i :

$$\begin{array}{l}\frac{E^{\partial }\left(\alpha \right)}{\partial {\alpha }_i}=s_i^T\left(S_{\mathit{\text{md}}}\right(\alpha )-S_{\mathit{\text{gv}}})+{\left(S_{\mathit{\text{md}}}\right(\alpha )-S_{\mathit{\text{gv}}})}^T{s}_i\end{array}$$

(13)

As the vectors s _i ,$\overline{S}$, and S _gv on the above formulas have the dimension same as the dimension of the reference skull vector, it is time-consuming to implement the optimization of the gradient descent algorithm with the high dimension in 3m. To reduce computation, we extract a random sub-vectors of these vectors to replace them on the above equations. That is a subset indices$\{i_0,\dots ,i_{m^{\prime }}\}$ are selected randomly from the continuous indices {1,…,m} in each gradient descent iteration. As m^′≪m, the computation is greatly reduced. As the indices of the vector element is correspondent to the points on the skull, this method is equivalent to select a subset of random points, representing the model and given skull for similarity error computation. So the similar random point selection approach used in the TPS-based non-rigid registration can be applied here to get the random sub-vector. Considering that the model deformation scale is smaller than the deformation between the reference and the target in data registration, and the local deformation dominates in the model matching, we use more quantity of random points in the model matching procedure. In our experiment, a subset with$\frac{1}{20}$ indices of the model skull vector is selected to implement the model matching computation. The maximal iteration is set to 500 for the global model and 1000 for the local models. To avoid the influences of noise and make use of the contribution of every points, this random subset is updated at each iteration. However, this will lead instability for the error in 11 at the beginning tens of iterations, but it behaves steadily in the later iterations and converges to a minimal value. We have tested different sizes of random subsets in the model matching experiment, smaller size than the assigned number generally cannot get satisfied precision and even not convergent. While more points added in the subset, the improvement for the model matching is insignificant. By this model matching procedure, the best matched model skull will be obtained. Then the reconstructed face can be calculated by the combination of the face part of h _i in 9 with the same coefficients as the model skull.

Similar to the matching procedure of the global model, the local deformable models are also matched to the given skull. In general, the local deformable models have better reconstruction results in the local areas than the global model. But the matching procedure of the local models is independent of the global model. As a result, the local reconstruction results generally are not consistent with the global result, especially at the boundary (shown in Figure6a). To get a whole smooth reconstruction result, the fusion problem of the local and global results must be solved. We take two steps to integrate the global and local reconstruction meshes. For the global mesh F and the sub-mesh F_sub, the local meshes are set onto the global mesh in proper position using translation and rotation transformation R^∗, T^∗, where R^∗, T^∗ are determined by minimizing the average distance of the correspondent points of the sub-mesh and the global mesh, i.e.,$(R^{\ast },T^{\ast })=\text{arg}\underset{R,T}{\text{min}}\sum _{P_0\in F_{\text{sub}}}\parallel R{P}_0+T-P_1\parallel$, where P₁∈F is the correspondent points of P₀. The first step fusion result is shown in Figure6b. Second, the inconsistence at the boundary is removed by a mesh stitching algorithm, in which both the points on the sub-mesh and the global mesh near the boundary are deformed to an interspaced position by interpolation method. The detail of the mesh interpolation is shown in Figure6d. For the sub-mesh boundary B₀ and a point P₀∈B₀, we can get the correspondent contour (denoted by B₁) on the global mesh and the correspondent point P₁ ∈ B₁ of P₀ by the above segmentation of the reference. Then the interpolating point P₂ is calculated by$P_2=\frac{(P_0+P_1)}{2}$. Given a scale l₀, the boundary B₀ will shrink into interior with l₀ step and get a contour$B_0^{\prime }$ which is indicated by a point$Q_0\in B_0^{\prime }$ in Figure6d, while the counter B₁ shrink oppositely on the global mesh and get a contour$B_1^{\prime }$ which is indicated by a point$Q_1\in B_1^{\prime }$ in Figure6d. The stitching method is to find a pair of interpolation functions f₀,f₁ have the following conditions:

$$\begin{array}{ll}{f}_0\left(Q_0\right)& =Q_0,Q_0\in B_0^{\prime }{f}_0\left(P_0\right)=P_2,P_0\in B_0\\ f_1\left(Q_1\right)& =Q_1,Q_1\in B_1^{\prime }{f}_1\left(P_1\right)=P_2,Q_1\in B_1\end{array}$$

(14)

Figure 6

The mesh fusion of the global and local reconstruction results. (a) Before fusion. (b) After the first fusion step. (c) The final fusion result. (d) The mesh interpolation procedure.

There are many interpolation methods can be used to meet the above conditions, such as RBF function. For convenience, we adopt the above TPS to solve the interpolation. Having determined the interpolation functions, the final fusion result (Figure6c) can be achieved by applying f₀ to the points between the contours$B_0,B_0^{\prime }$ on the sub-mesh and f₁ to the points between the contours$B_1,B_1^{\prime }$ on the global mesh.

Experimental results and discussion

Based on the dense aligned face and skull samples, a hierarchical deformable model is constructed for craniofacial reconstruction, which includes a global model and three local models, namely, the eye, nose, and mouth model. To validate the craniofacial reconstruction method, we implement a leave-one-out craniofacial reconstruction experiment, in which each skull is used as the test skull for craniofacial reconstruction, and the rest skulls and faces are used as the samples to build the hierarchical deformable model. Both the global and the local models are matched to all tests. The final facial surfaces of the tests are reconstructed by the model matching and mesh stitching procedure. Some craniofacial reconstruction results and its actual faces are shown in Figure7.

Figure 7

Some craniofacial reconstruction results. The actual faces are shown in columns 1, 3; the reconstructed faces are shown in columns 2, 4.

The reconstructed faces of the hierarchical model are evaluated by comparing with their actual faces. First, the traditional measurement method is adopted for the evaluation, which adopts the average distance of correspondent points as the similarity measurement between 3D faces. We denote this method by ADCP. In ADCP method, the average reconstruction error of 110 tests are 0.01101 and 0.00998 for the global model and the hierarchical model with standard deviation 0.00297 and 0.00353, respectively. As the actual scale of our samples and the model has lost in the data preprocessing and coordinates uniforming procedure, we set the average distance between left and right porions to 15 cm according to the Chinese National Standard for Human Dimensions of Chinese Adult[49]. Then the average absolute error of correspondent points for the global and hierarchical model is 1.65 and 1.50 mm. Comparing with the results in[12, 15], the result is acceptable considering that some important properties, such as Body Mass Index (BMI), age, and gender, are not integrated into the model. The distribution of the average reconstruction error for every tests are shown as histogram in Figure8. From the histogram, it is figured out that there are more tests with error less than 1.5 mm for the hierarchical model. The distribution of the average reconstruction error for every points of the 110 tests is also computed and the results are displayed on the reference face in Figure9. The figure shows that the hierarchical model have better reconstruction results than the single global model, especially in the areas related to the local models. It is concluded that using the local models is beneficial to the improvement of the reconstruction accuracy.

Figure 8

The histogram of the average reconstruction error of the global and hierarchical model for 110 tests.

Figure 9

The distribution of the average reconstruction error for every points on the reference face of the 110 tests. (a) The result of the global model. (b) The result of the hierarchical model.

Although ADCP is the dominant method for craniofacial reconstruction evaluation, it is not an ideal method considering that the aim of the craniofacial reconstruction is identification through face recognition. In general, the dense faces have very large quantity of points, the distance changing on local area usually brings little changes to the average. As a result, the ADCP method is not sensitive to local shape change. But the local feature is important for face recognition, such as the width of mouth and the height of nose. The other drawback of ADCP is that every points have same weightiness in the distance computation, though the points on different facial position have different effect for face recognition. To get proper evaluation for the craniofacial reconstruction result, we define a similarity measurement for reconstructed face inspired the ideas of face recognition, which uses the distance in the coefficient domain of a face deformable model as the measurement. The method is denoted by DCD. Similar to the above deformable model, the face deformable model is constructed from the aligned faces as follows:

$$F_{\text{model}}\left(\beta \right)=\overline{F}+\sum _{i=1}^k{\beta }_i{f}_i$$

(15)

where$\overline{F}$ is the average face, {f _i |i=1,…,k} are the former k components. When two faces F₁, F₂ are compared, their model coefficients β₁, β₂ are computed by the model-matching procedure. Then the difference between F₁ and F₂ is measured by the distance between β₁, β₂ in the coefficients space.

To get DCD measurement, 110 aligned faces are used to construct the face deformable model. Then the actual faces and the reconstructed faces of the global and hierarchical models are represented by the coefficients of the face model. The DCD distance between the reconstructed faces and its actual faces are calculated. The average reconstruction error of 110 tests are 0.1906 and 0.1837 for the global model and the hierarchical model with standard deviation 0.0516 and 0.0539, respectively. It is shown that the hierarchical model has better reconstruction results in DCD measurement method. To some extent, the distance in the coefficients domain does not correlate to the distance of correspondent points. To explore the relationship between the presented DCD method and ADCP, we do a face recognition experiment, in which the closest real face to the reconstructed faces of the global and hierarchical model is calculated in DCD and ADCP measurements, respectively. The cumulative recognition curves of the reconstruction results are shown in Figure10. It shows that the face recognition results is great different for the two measurements. The cumulative recognition rate in DCD method is greater than that in ADCP method. And the hierarchical model generally has better recognition rate than the global model. These observations do not assuredly support that DCD is more suitable for evaluating the reconstructed faces than ADCP. But it is concluded that DCD has good tolerance to the reconstruction error in face recognition application. As DCD is a distance measurement in the facial shape space which is constructed from a set of original samples. Gathering sufficient face samples, the DCD method may reflect the real measurement of the face space.

Figure 10

Cumulative recognition curves of the reconstruction results of the global and hierarchical model in ADCP and DCD methods.

Conclusion

We proposed a hierarchical dense deformable model for automatic craniofacial reconstruction. The feature of proposed model is that the skull and face are represented as dense mesh without landmarks. The advantage of this representation is that the dense meshes contain more meta-data for exploring the intrinsic relation between skull and face. In addition, the presented non-rigid dense meshes registration and the model matching procedure can be implemented automatically, which contributes to the fully automatic craniofacial reconstruction method. The craniofacial reconstruction experiments show that the hierarchical model has better reconstruction results than the single global model. The craniofacial reconstruction evaluation problem is also explored in this article. We present an evaluation method based on a deformable facial model. By comparing with the average distance of correspondent points method in face recognition experiment, the evaluation method may be the potential method for identification in the application of craniofacial reconstruction. In the future work, we plan to capture more head scans to increase the plenty of the samples, which is important for the model deformable capacity. Based on the abundant samples, the personal properties, such as gender, age, and BMI, are considered integrating with the hierarchical dense deformable model. The reconstruction result will be improved if these properties information are properly utilized. In addition, it is worthy of exploring ideal evaluation methods for the results of craniofacial reconstruction.