Pavement Crack Classiﬁcation via Spatial Distribution Features

. Pavement crack types provide important information for making pavement maintenance strategies. This paper proposes an automatic pavement crack classiﬁcation approach, exploiting the spatial distribution features (i.e., direction feature and density feature) of the cracks under a neural network model. In this approach, a direction coding (D-Coding) algorithm is presented to encode the crack subsections and extract the direction features, and a Delaunay Triangulation technique is employed to analyze the crack region structure and extract the density features. As regarding skeletonized crack sections rather than crack pixels, the spatial distribution features hold considerable feature signiﬁcance for each type of cracks. Empirical study indicates a classiﬁcation precision of over 98% of the proposed approach.


Introduction
Pavement crack types are important for pavement dilapidation analysis and pavement maintenance decision-making. For asphalt pavements, the pavement cracks can generally be classified into four types-the transverse crack, the longitudinal crack, the block crack, and the alligator crack [1] (see Figure 1). Each type of crack holds its own weight in the pavement maintenance evaluation. Therefore, the exploration of a robust and reliable approach for pavement crack classification has great significance.
Over the past several decades, with the development of high-speed cameras and large storage hardware, a realtime collection of pavement images has been realized. While along with the progress of image processing and pattern recognition techniques, the image-based crack recognition method gradually replaces the traditional manual method and becomes a common way for pavement crack detection [2][3][4][5][6][7]. Pavement crack recognition includes two stagesthe crack detection and the crack classification. This paper mostly focuses on the later.
Though a variety of approaches for pavement crack classification have been proposed in the last two decades, most of them cannot meet the requirements in practice due to their inadequate consideration on spatial distribution features of the cracks. For example, the projection histogram methods [8][9][10] can be qualified to identify the directional difference between cracks, but it may not be capable of distinguishing the density difference. In a pavement image, typically, a crack has a linear or curvilinear structure, the spatial distribution of the crack points determines which type of crack it is. Therefore, analyzing the crack's spatial distribution features, that is, the direction feature and density feature, is the key point to crack classification. In this study, a novel pavement crack classification approach is proposed by using spatial distribution features in a neural network. Under this approach, the problem of crack feature extraction is formulated as the problem of direction and density feature extraction on a binary skeletonized crack section. Generally, the transverse and longitudinal cracks hold much more direction features than the block and alligator cracks, while the block and alligator cracks have more density features. Moreover, the block cracks own less density features than the alligator cracks. According to these characteristics of the different crack types, we present a direction coding algorithm (D-Coding) stemming from Freeman coding [11] to acquire the direction features from skeletonized crack sections, meanwhile we adopt the Delaunay Triangulation [12] technique to analyze the structure of the crack regions and gain the density features of the crack. Experiments in our research indicate the reliability of the extracted features. The contributions of this paper are twofold: (1) a D-Coding algorithm stemming from Freeman coding is presented for encoding the direction information of the linear structures, and (2) the Delaunay Triangulation technique is innovatively applied to analyze the crack region structure and extract the crack density features.
The rest of this paper is organized as follows. Section 2 briefly gives the literature review of the related work in pavement crack classification. Section 3 presents the architecture of the proposed approach. Section 4 describes our methodology in detail, which contains the singularity points detection, linear subsection extraction, density feature expression, feature vector construction, and network structure design. Section 5 gives the experimental results and analysis, and Section 6 concludes our work by pointing out future directions.

Previous Work
In recent years, a number of approaches for pavement crack classification have been proposed which generally fall into two categories-the supervised and the unsupervised. The former includes a series of neural network-based approaches [8][9][10][13][14][15][16][17][18][19], while the later are rule-based approaches [1,[20][21][22]. Among the neural network-based approaches, Kaseko et al. [13] exploited a two-stage piecewise linear neural network for crack classification and proved that it outperforms the Bayes classifier and the k-nearest neighbor (k-NN) classifier. In their study, five features were selected to construct the class feature space: the number of crack pixels in an image tile, the number of distressed pixels per line in the transverse and longitudinal directions, and the number of distressed pixels per line in the two diagonal directions. Through crack primitives (i.e., crack sections) analysis, Koutsopoulos et al. [14] extracted crack features using the discriminant analysis, k-NN, and discrete choice models. Some other approaches exploited the moment features [15,16]. Chou et al. [15] classified the pavement distress based on Hu moments, Zemike moments, and Bamieh moments, with a reported one-hundred percent classification accuracy. Hsu et al. [16] used moment features to classify real airport pavement distresses and gained an accuracy of 85%. On the basis of extracting the geometric and textural features, Sinha and Karray [17] constructed a fuzzy neural classifier, and the declared precision is above 92.7%. Lee [8,18] exploited three kinds of neural networksthe image-based neural network (INN), the histogram-based neural network (HNN), and the proximity-based neural network (PNN). They divided the segmented image into subimages firstly, tagged each sub-image into a crack tile "1" or a noncrack tile "0", thus forming a two-dimensional Boolean crack matrix. After that, they summed this matrix along the X and Y axes, forming two histogram vectors. Then they tested three different feature extraction strategies on these two vectors and demonstrated a best performance of PNN of a classification accuracy of 95%. Considering the density distribution difference between linear and regional pavement distress, Xiao el al. [19] presented a densitybased neural network (DNN) classifier, and the claimed precisions is above 99% to the simulated data and above 97% to the real pavement images. Rababaah   studied the projection features and Hough represented features over three classifiers-the genetic algorithm (GA), multilayer perceptrons (MLP), and self-organizing maps (SOM). Experiments show the projection features are better than the Hough represented features, and MLP is the best classifier with an overall accuracy of 98.6%. Among the unsupervised approaches, Georgopoulos et al. [1] identified the crack type by analyzing the geometrical properties of the cracks, which could also generate the severity information. Considering the spatial connectivity and directional consistency, Javidi et al. [20] approached the pavement crack detection and classification by a combination use of the wavelet multiscale edge detection and the Hough transformation. It was reported to achieve better results than the Wisecrax: a commercial product from Roadware company. Wang et al. [21] presented a chain coding algorithm to track the skeletonized cracks. Based on the statistic parameters of the tracking array, they classified the crack types. Beamlet transform-based approach was proposed by Ying and Salari [22], where the sub-images are represented with beamlets, based on which the segmentation, linking, and classification operations are implemented.
Most of those methods mentioned above can obtain certain accuracy under certain conditions such as noiseless source images, limited experimental results, and computer simulated experiment images. None of them have given complete consideration to the spatial distribution features of either direction or density.

Classification Framework
As has been discussed, the crack type is determined by the spatial distribution features of the crack points. Therefore, how to describe and generate the spatial distribution features: the direction features and the density features is the key point to the problem of pavement crack classification. The direction features mainly contain four types: the transverse direction (perpendicular to the road direction), longitudinal direction (parallel to the road direction), and two diagonal directions (with a 45 • or 135 • angle to the road direction). The density features are depending on the number and location of endpoints and junction points of the crack sections. Considering the characteristics of these different cracks, we propose the solution architecture shown in Figure 2.
In order to extract spatial distribution features of the cracks, we detect the singularity points on the binary skeletonized crack sections firstly, and then we introduce a direction coding (D-Coding) algorithm to compute the direction information for each crack subsection generated from the removal of singularity points. With singularity points, we can also create the Delaunay triangles to analyze the crack structure and extract the density features. Based on the D-Coding results and Delaunay analyzing results, we construct the feature vector with seven feature parameters. Finally, we employ a BP (Back Propagation) neural network classifier considering the feature vector and the classification output.

Methodology
We start this section with introducing the process of singularity points detection, followed by a presentation of the linear subsection extraction, and the Delaunay Triangulation technique for density feature expression. Then we describe the feature vector construction and the neural network design.

Singularity Points Detection.
A pavement crack is composed of linear subsections, and each subsection contains two general endpoints: the endpoints and the junction points. In this study, these two kinds of points are defined as singularity points. With singularity points, the whole pavement crack could be separated into several linear crack subsections.
Exactly, there are two types of junction structures owing to the centrosymmetric characteristic of the 8-connection binary skeletonized crack sections. One is the intersection with a transverse and a longitudinal line, and the other is the intersection with two lines in diagonal directions, which are illustrated in Figures 3(a) and 3(b), respectively. Also, two types of basic endpoint structures are illustrated in Figures 3(c) and 3(d). The first type includes points whose 8-connection neighborhood has only one crack point. The other type includes points whose 8-connection neighborhood contains one intersection point, while no other crack points exist in its 8-connection neighborhood except the ones belonging to the neighborhood of that intersection point.
The singularity points of the crack skeletons are detected based on the definitions and rules described above. Figure 4 illustrates the results of singularity point detection.

Linear Subsection Extraction Using D-Coding.
In order to express the spatial direction features of the crack, we divide crack skeletons into linear crack subsections by removal of the singularity points. As mentioned above, the direction feature of the crack subsections is very important to linear pavement distress classification.
On the basis of the classical 8-direction Freeman coding and the centrosymmetric characteristic of the 8-connection neighborhood, we propose a direction coding (D-Coding) to encode the crack subsections. On the one hand, since crack direction can be fully expressed by an angle between 0 to 180 degrees, we need only one direction code for one crack line. On the other hand, as a crack line is an 8-connection component, we cannot use a 4-direction encoding strategy, for example, the 4-direction Freeman coding. Therefore, we form our D-Coding structure by equalizing each two codes in centrosymmetric in an 8-direction Freeman coding structure. As illustrated in Figure 5, (a) is the conventional diagram of the classical 8-direction Freeman coding, (b) is for the 4-direction Freeman coding, and (c) is the diagram of the D-Coding. Considering the convenience of the subsequent analysis, the starting code begins with 1, and with an order of 1, 2, 3, and 4. As Figure 5(c) illustrated, codes 1 and 3 stand for the horizontal and vertical direction, while codes 2 and 4 stand for the two diagonal directions.
To activate the proposed D-Coding, two rules are formed: (1) to crack skeletons with junction points, the junction points are regarded as the start coding points and (2) to crack skeletons without junctions, the endpoints are regarded as the coding start.
Given the starting code be set as 0, the corresponding D-Coding results for cracks in Figures 4(e)-4(h) are shown in Figure 6.

Density Feature Expression with Delaunay Triangles.
The singularity points also provide important clues for crack density distribution features. The number and location of the singularity points show the complexity of the crack, and the structure of the crack region, which are highly related to the density property of the crack, and vital for identifying a crack's type. In order to describe these density properties, we apply the Delaunay Triangulation technique.
In mathematics and computational geometry, a Delaunay Triangulation for a set P of points in the plane is a triangulation DT(P) such that no point in P is inside the circumcircle of any triangle in DT(P). Delaunay Triangulations maximize the minimum angle of all the angles of the triangles in the triangulation; they tend to avoid skinny triangles [23].
Delaunay triangles can be created based on the Delaunay Graphs. Let P be a set of spatial points, DG(P) be the Delaunay Graph of P, then DG(P) can be obtained through the following two steps: Once DG(P) is obtained, the Delaunay Triangulation DT(P) could be gained simply by replacing the graph edges with lines.
With the singularity points detected in Section 4.1, we could implement the Delaunay Triangulation for the EURASIP Journal on Advances in Signal Processing The circumcircle excluding and minimum-angle maximizing properties of the Delaunay Triangulation make the Delaunay triangles compactly describe the crack region structures, which is helpful for representing the crack's spatial density features.

Feature Vector Construction.
According to the definition of different pavement cracks, the transverse and longitudinal cracks have less singularity points, while the block and alligator cracks have much more singularity points, especially junction points. In addition, a transverse crack is abundant in horizontal components, while a longitudinal crack is full of vertical components. Moreover, the average area value of the alligator crack regions is much smaller than that of the block crack regions. Based on such above analysis, we construct the feature vector as follows.
Let p 0 = n be the number of Delaunay triangles generated from singularity points, let S i be the area value of each triangle, and let a j ( j ∈ {1, 2, 3, 4}) be the sum Dcoding value of the direction j segments, then the maximum area value p 1 and the average area value p 2 of the triangles can be formulized as (1) and (2), respectively The ratio d j ( j ∈ {1, 2, 3, 4}) of each direction components which used to characterize the spatial distribution direction features can be defined by Then the feature vector P for pavement cracks classification can be constructed as

Neural Network Structure
Design. The back-propagation (BP) neural network is a typical multilayer feedforward neural network which adjusts the weights and bias through back propagation continuously until the Mean Square Error (MSE) of the network tends to the least. It has been widely applied to recognition, forecasting and procedure controlling because of its strong abilities on nonlinear mapping, flexible network structuring, and generalization. In this study, a single hidden layer BP neural network is employed, and the structure was designed as follows.
Let l be the input neurons number, let P (P ∈ R l ) be the input feature vector, let m be the number of hidden neurons, Z (Z ∈ R m ) be the middle hidden feature vector, let n be the output neurons number, and let T (T ∈ R n ) be the output   j ∈ {0, 1, . . . , m − 1}), W 2 j,k and b 2 k are the connect weights and bias value between hidden neuron z i and output neuron t k (k ∈ {0, 1, . . . , n − 1}), then the relationship of input layer, hidden layer and output layer can be formulate as follows: EURASIP Journal on Advances in Signal Processing  P [2] P [3] P [4] P [5] P [6] Z [1] . . .

Input layer
Hidden layer Output layer where f (•) is the stimulation function. Based on the fact that the input feature vector is a seven dimension space, while the output patterns are with the transverse, longitudinal, block, and alligator, the neuron numbers of the input layer and Table 1: Feature values corresponding to the images in Figure 11.
Column ( The suitable neurons number of the hidden layer m relies on the repeated training results and is initially set as 30.
According to above description, the neural network structure is illustrated by Figure 9.

Experimental Study
In this section, we first introduce the dataset used in this study, and then give a real example for feature vector construction. At last we examine and discuss the classification details.

Dataset.
A crack image database (SmartDB) containing 16000 image samples has been created based on 1200 pavement crack images captured by our SmartV system (see Figure 10). Each type of cracks has 4000 samples, where the crack type is labeled by two skilled workers and verified by an expert. In order to evaluate the performance of the proposed approach, the database images are divided into two batches, one contains 8000 as the training sample images, and the other contains 8000 as the testing sample images. Then, for each type of crack, we have one 2000 images for training and the other 2000 images for testing.

An Example for Feature Vector Construction with Real
Images. In order to illustrate the procedure of spatial distribution feature extraction on cracks, four typical images shown in Figure 11 (row 1) are selected to give a real example.
The results of each processing stage are arranged in rows. The feature vector P of each crack image is extracted according to the methods described in Section 4.4, and the feature values corresponding to the four images are listed in Table 1.
As mentioned in Section 4.4, p 0 is the number of triangles, p 1 is maximum area value of the triangles, while p 2 is the average area value, and d j ( j ∈ {1, 2, 3, 4}) is the direction components in four directions, respectively.

Evaluation. A series of training and testing experiments
were conducted under different number of hidden neurons and different training epochs. Firstly, two metrics are introduced for the result evaluation.
To a crack type i (i = 0, 1, 2, 3 denote the four types, respectively), given a true-positive TP i , the number of cracks which are correctly classified as type i, a false-positive FP i , the number of cracks which are with type i but not classified as type i, and a false-negative FN i , the number of cracks which are incorrectly classified as type i, then the two objective indices Precision i and Recall i are defined by [24] Precision i = TP i TP i + FP i , Leting TP, FP, and FN denote the total number of the corresponding cracks mentioned above, we have Then the total Precision and Recall can be defined as As each testing sample will be classified as one of the four crack types, a false-positive in one type will certainly cause a false-negative in another. Thus, we have FP = FN, and Precision = Recall. We simply adopt one of them, for example, Precision, to evaluate the overall performance.
Overall Performance. Taking the experimental strategies in [16] for reference, we conduct the training and testing of the constructed BP neural network under different hidden neuron numbers from 30 to 120 at an interval of 30, and under different epochs from 500 to 3000 at an interval of 500. The testing results are listed in Table 2. As can be seen from  [15,16], and the other two are feature vectors based on projection [9,18]. All results shown in Table 3 are gained under an optimal training epoch with 60 hidden neurons. The comparisons in terms of precision and recall are also illustrated by Figure 12, from which we can find that, the proposed feature vector outperforms the other three competing feature vectors in both precision and recall. Among the other three feature vectors, the projection-based feature vector used in [9] gains the most competing precision results with the proposed one, however, it achieves much lower recalls in handling block and alligator cracks.
Validity of the Selected Features. Also, we conduct a range of experiments to check the validity of the selected features.
To reach this, we construct the feature vector by excluding each of the seven features in turn, and their performances are illustrated in Figure 13. From Figure 13, we can find the feature vectors which have been element-excluded gains lower achievements than the full-element feature vector, which indicates the validity of each element of the proposed feature vector. Meanwhile, we can see that, the features p 0 , p 1 , and p 2 have high impact on results of the block and alligator cracks, which denote their density characteristics. And features d 1 and d 3 have high impact on results of the transverse and longitudinal cracks, respectively, which verifies their direction characteristics.

Conclusion
In this paper, we developed a new pavement crack classification approach by using spatial distribution features of the crack. In order to extract the direction distribution features, we presented the D-Coding algorithm. While to extract the density features, we adopt the Delaunay Triangulation EURASIP Journal on Advances in Signal Processing