Weakly supervised object extraction with iterative contour prior for remote sensing images

Abstract

This article presents a weakly supervised approach based on Markov random field model for the extraction of objects (e.g., aircrafts) in optical remote sensing images. This approach is capable of localizing and then segmenting objects in optical remote sensing images by relying only on several object samples without artificial labels. However, unlike direct combinations of object detection and segmentation, the proposed method develops a contour prior model based on detection results, thereby improving segmentation performance. Furthermore, we iteratively update the contour prior information based on the expectation-maximization algorithm. Numerical experiments illustrate that the proposed method can successfully be applied to the extraction of aircrafts in optical remote sensing images.

1 Introduction

Object detection and segmentation have received considerable attention as important procedures in automatic object identification in such fields as computer vision, remote sensing image processing, and so on. Based on the large number of works to which object detection and segmentation have been applied, a key distinction between these two methods can be found; object segmentation is usually interactive and incorporates guidance from the user throughout the analysis process, such as in GraphCut [1] and Snake [2], whereas object detection needs learning samples and/or supervising information from the user at the beginning of the analysis, such as in latent support vector machine LSVMs [3], Wu et al.’s Active Basis [4]. Nevertheless, object detection and object segmentation share numerous theoretical and methodological features, which if explored will be of benefit to each other. In this article, object detection results based on Active Basis [4] are developed to replace supervised learning samples in object segmentation. Object segmentation results can be obtained by providing several object samples. Furthermore, this combination employs a contour prior model based on the detection results, thereby improving the segmentation performance.

From a methodological perspective, the main idea of numerous methods that have recently been used for object detection and segmentation can be divided into shape-based methods and feature-based methods. Shape-based methods, such as Felzenszwalb et al.’s LSVMs [3], Wu et al.’s Active Basis [4], Laptev et al.’s Snake [2], and Ferrari’s kAS [5], exploit shapes similarities between objects by using different strategies and then obtain segmentation results through by connecting the segments. Shape-based methods are completed automatically without the need for human assistance. However, these methods hardly obtain segmentation results. Feature-based methods such as Cheng et al.’s [6] hierarchical lane detection system, Hassaballah et al.’s [7] independent components analysis, Borenstein and Ullman’s [8] top-down bottom-up segmentation, Weisenssel et al.’s [9] Markov random field (MRF) model-based method, and Jia and Hong-qi’s [10] interactive segmentation based on graph cuts, utilize different representations (color or texture features, distribution model) of image pixels or regions to distinguish objects from the background. However, these methods all require human assistance or strong supervised information.

Based on Active Basis [4], we developed detection results by using Morph-ActiveBasis [11] and thus proposed a contour prior model to improve segmentation performance, which will be detailed in Section 3. This article combines object detection (Morph-ActiveBasis [11]) and object segmentation (MRF) and proposes a contour prior model by using the above combination to improve segmentation performance.

2 Morph-ActiveBasis: from fragments to rough contours

Morph-ActiveBasis, which is presented in [11], is based on Active Basis. Morph-ActiveBasis can determine the basic edge contours of objects from a set of object samples without the need for artificial labels and can detect similar objects in given images. Unlike the scattered fragments obtained by using the original Active Basis, Morph-ActiveBasis employs fragment connection to link scattered fragments thus forming a sketch of the object contour (for details, see [11]).

2.1 Fragment detection

The Active Basis [4] model is utilized to detect the basic edge contours of objects. Active Basis represents contours through a set of Gabor Wavelet bases [12, 13]. Moreover, Active Basis does not require human guidance and can automatically detect objects in the image. However, the detected results are only scattered fragments that could not represent the integral contour of object. Therefore, we propose the use of fragment connection to link the scattered fragments, thus forming a sketch of the object contour.

2.2 Fragment connection

The principle of fragment connection in [11] is based on the structure information among fragments.

2.3 Rough contours extraction

The Morph-ActiveBasis detection algorithm estimates the Gaussian distribution models of objects and backgrounds by using contour sketches, and then segments the contours by utilizing GraphCut segmentation [14] algorithm. Given an image $I=\{s_1,s_2,\dots ,s_{N_I}\}$, s _i is the i th pixel in the image. The segmentation result is that each one of N _I pixels is assigned to a label y _i ∈{0,1}, where 0 and 1 represent objects and backgrounds, respectively, thus the segmentation yields result $Y=\{y_1,y_2,\dots ,y_{N_I}\}$.

$$\overline{Y}=arg{max}_Y\left\{P\right(Y|I,\Theta )\}$$

(1)

$$P\left(Y\right|I,\Theta )\propto \prod _{i=1}^{N_I}{p}_{L_i}(y_i|s_i,\Theta )p_{N_I}\left(y_i\right|y_{N_I})$$

(2)

Here, Θ is the parameter of the Gaussian distribution Θ={μ,σ}, and μ,σ, respectively, represent the mean and variance of Gaussian distribution. The likelihood probability $p_{L_i}$ can be obtained by using Gaussian distribution model of objects and background, as shown in Equation (6). $P_{N_I}$ denotes the Potts model.

3 Expectation-maximization (EM) contour MRF: from rough contours to further segmentation

In this section, we present a contour prior model of objects to improve segmentation performance under the MRF model framework. The idea of this contour prior model is to assign pixels (that are located inside an object contour) with a higher probability to become object. By contrast, pixels outside the contour are assigned a higher probability to be background. The probability of the contour prior is based on the distance between each pixel and its nearest contour point.

3.1 Contour prior information based on rough contours

As shown in Figure 1a, Ω is the initial contour of the object, s ₀₁,s ₀₂,s ₁₁,s ₁₂ are the four arbitrary pixels in the image (s ₀₁,s ₀₂ are located inside Ω, whereas s ₁₁,s ₁₂ are outside Ω) and d ₀₁,d ₀₂,d ₁₁,d ₁₂ denote the distances from s ₀₁,s ₀₂,s ₁₁,s ₁₂ to Ω, respectively. Inside the contour, a pixel that is located farther from the contour is more likely to become an object. However, outside the contour, a pixel that is farther from the contour has higher probability of becoming a background. For instance, in Figure 1a, s ₀₂ is more likely to become an object as compared with s ₀₁ because d ₀₁<d ₀₂, whereas s ₁₂ is more likely to become a background as compared with s ₁₁ because d ₁₁<d ₁₂.

Figure 1

Sketch map of contour prior. (a) Distance from point to contour; (b) probability map of shape prior.

Figure 1b shows the sketch map of the shape prior probability $p_{s_i}$. The data shown in Figure 1 can mathematically be expressed by Equations (3)–(5) such as in [15, 16]. Prior knowledge on the background and the object is shown in Equations (3) and (4), respectively. Equation (5) gives the distance of the pixel to the contour, where μ is a constant that indicates the distance coefficient s i g n(s _i ,Ω)=1 when s _i is inside Ω and s i g n(s _i ,Ω)=−1 when s _i is outside Ω; and l o c(s) is the coordinate position of pixel s in the image.

$$\begin{array}{rcl}{p}_{s_i}(y_i& =& 0\left|\Omega \right)\propto \frac{1}{1+exp(\mu \ast \text{dist}(s_i,\Omega \left)\right)}\end{array}$$

(3)

$$\begin{array}{rcl}{p}_{s_i}(y_i& =& 1\left|\Omega \right)\propto 1-\frac{1}{1+exp(\mu \ast \text{dist}(s_i,\Omega \left)\right)}\end{array}$$

(4)

$$\begin{array}{rcl}\text{dist}(s_i,\Omega )& =& \text{sign}(s_i,\Omega )\underset{s\in \Omega }{min}\parallel \text{loc}\left(s_i\right)-\text{loc}\left(s\right)\parallel \end{array}$$

(5)

3.2 Segmentation based on the MRF model and on EM iteration

To obtain better segmentation results from the contour prior information, we combine the EM algorithm [17] with GraphCut optimization. We apply the segmentation result to update the parameters of the Gaussian distribution model and then utilize the new distribution parameters to initialize GraphCut optimization. The E- and M-steps are as follows:

E-step: The likelihood probability of pixels to class y _i is computed based on the current distribution parameters of Gaussian distribution Θ ^(t)={μ ^(t),σ ^(t)}, where μ,σ, respectively, represent the mean and variance of Gaussian distribution and express as follows:

$$p_{L_i}\left(y_i\right|s_i,{\Theta }^{\left(t\right)})\propto \frac{p\left(s_i\right|y_i,{\Theta }^{\left(t\right)})}{\sum _{c=0}^{1}p\left(s_i\right|y_i=c,{\Theta }^{\left(t\right)})}$$

(6)

M-step: Under the framework of the MRF model, we denote each pixel as a node in the MRF model and describe its likelihood probability with $p_{L_i}$. The relationship between pixels (nodes) is represented by the combination of p _{s i} and p _{N I} , which is a pairwise probability in MRF. The probability $p_{L_i}$ which is obtained in the E-step is utilized to combined with the contour prior $p_{s_i}$ of the object and the prior probability $P_{N_I}$ of the Potts model to obtain the MRF model in Equations (7) and (8). The GraphCut optimization algorithm is then applied to solve Equations (7) and (8), thus obtaining the new labeling data Y. The new Gaussian distribution parameters Θ ^(t+1)={μ ^(t+1),σ ^(t+1)} of objects and background are computed, and the shape Ω ^(t+1) of the object is updated. The EM algorithm continuously repeats the E- and the M-steps until the convergence condition is satisfied.

$$\begin{array}{c}\overline{Y}={argmax}_Y\left\{P\right(Y\left|I\right)P(I/{\Omega }^{(t-1)})P\left(Y\right)\}\end{array}$$

(7)

$$\begin{array}{ll}\hfill P\left(Y^{\left(t\right)}\right|I,& {\Omega }^{t-1},{\Theta }^{t-1})\\ \hfill =& exp\left[\sum _{i\in I}\left[\mathrm{\alpha f}\left(s_i|y_i,{\Theta }^{t-1}\right)+\mathrm{\beta f}\left(y_i|d\left(s_i,{\Omega }^{t-1}\right)\right)\right.\right.\\ \left.\left.+\gamma \sum _{j\in C_i}\delta \left(y_i,y_j\right)\right]\right]\end{array}$$

(8)

The right-hand side of Equation (8) is consisted of three parts which, respectively, represent $p_{L_i}$, $p_{s_i}$, and $P_{N_I}$. The first two parts, respectively, represent Gaussian probability and contour, the last one is the Potts prior model. C _i is the cliques of pixel s _i . We use eight neighborhood cliques. Then we defined the potential or energy function for these cliques through Gibbs distribution which is computed by Potts prior model. d(s _i ,Ω ^t−1) is the distance from s _i to the contour Ω ^t−1, α,β,γ are the normalized parameters.

4 Flow of the proposed algorithm

The proposed weakly supervised object extraction algorithm aims at achieving improved object fragment detection and segmentation results with minimal supervision. As shown in Figure 2, the algorithm first utilizes fragment connection and GraphCut segmentation to obtain the initial contour Ω ⁰ of the object based on the Active Basis model. The algorithm then constructs the MRF on the initial contour of the object and adopts an iterative optimization approach combined with the EM algorithm to achieve improved object segmentation results. The algorithm process is shown as below:

Figure 2

Framework of Morph-ActiveBasis + EMContourMRF algorithm.

Initialization

Use Morph-ActiveBasis method to obtain the initial contour Ω ⁰ of objects

Step 1: E-Step: a: Use (3)–(5) to compute for the shape prior $p_{s_i}$ of each pixel in the testing image based on the current contour Ω ^t of the object; b: Compute the likelihood p(s _i |y _i ,Θ) and posterior probability $p_{L_i}$ of pixels according to the current distribution parameters of the objects and background, as shown in Equation (6);

Step 2: M-Step: a: Use of GraphCut optimization algorithm to solve (7)–(8) to obtain the new labeling data based on (3)–(6); b: Update object’s shape Ω ^(t+1) as well as Gaussian distribution parameters Θ ^(t+1);

Termination: Repeat 2 and 3 until the change in the segmented object regions is less than a certain threshold, that is, the iterative convergence.

end

5 Experimental results and analysis

5.1 Experimental data

During the experiment, two new and significantly extended databases have been formed through capturing screen shot images from Google earth. One database contains 100 small images with the size 128×128 obtained from the Beijing Capital International Airport; the other database consists of 10 big images with the size 5000×5000 obtained from Atlanta and other airports in the world. To obtain clearer images of the aircrafts, eye altitude is set to lower than 500 m when using Google Earth to obtain screenshots. Aircraft targets from the airport are selected as experimental targets from which the training and testing image samples are taken. At least one image is needed for training. We use ten images for training in the experiments.

5.2 Experiment setting

Comparisons are performed by using the conditional random field (CRF) [18] with RCC prior. CRF needs artificial labels to start segmentation. For the same experimental setting, we used the initial object pixels and background pixels (in Step 2, as shown in Figure 2) as the input of CRF.

To evaluate quantitatively the performance of proposed method in terms of object segmentation, measures of completeness and correctness are adopted which are similar to recall and precision in image retrieval [19], in which completeness denotes the ratio of correctly segmented object pixels to the sum of the true object pixels, whereas correctness denotes the ratio of correctly segmented object pixels to the sum of pixels in the object segmentation results. True object pixels are manually labeled in the original image. The detection rate and error detection rate are also employed to assess target detection performance, where the former indicates the ratio of the number of detected target pixels to the total number of targets whereas the latter stands for the number of wrongly detected target pixels to the total number of targets.

5.3 Results

5.3.1 Experiment 1

We first test the proposed method on 100 small images taken from Beijing International Airport (size: 128×128, each contains a single aircraft). Figure 3 shows ten samples of the experimental results. As shown in Figure 3, contours are relatively well segmented. The proposed Morph-ActiveBasis + EMContourMRF algorithm is compared with the CRF method. Table 1 shows the segmentation accuracy. The completeness of Morph-ActiveBasis is higher than that of CRF, but its correctness is significantly lower. Based on Morph-ActiveBasis, EM Contour MRF not only enhances the contour of objects, but also significantly improves the segmentation accuracy.

Figure 3

Experimental results of single-target detection. Top, the original; Middle, results of the CRF method; Bottom, results of proposed method.

Table 1 Completeness and correctness of object segmentation in the images taken from the Beijing Capital International Airport with a single aircraft

5.3.2 Experiment 2

The second experiment involved the extraction of multiple targets from images taken from Atlanta Airport (size: 5000×5000, each contains more than one aircraft) as shown in Figure 4. Figure 5 displays the results. More aircrafts have correctly been detected by using Morph-ActiveBasis method than CRF approach. Table 2 shows the numbered bounding box, detection rate, and the error detection rate. Figure 5 illustrates some of the corresponding segmented results. The completeness and correctness of segmentation of each image are shown in Table 3, and the results of all the ten images are shown in Table 4.

Figure 4

Experimental images for multi-target detection. Top, Atlanta Airport and Charles de Gaulle Airport; Bottom, Ground Truth.

Figure 5

Extraction results and some segmentation results of multi-aircrafts: (a) CRF approach and (b) Morph-ActiveBasis + EMContourMRF algorithm.

Table 2 Detection rate and error detection rate of aircraft detection in the Atlanta Airport and Charles de Gaulle Airport

Table 3 Completeness and correctness of aircraft detection in the Atlanta Airport and Charles de Gaulle Airport

Table 4 Completeness and correctness of aircraft detection in ten international airports

Although the completeness of the propose method is lower than that of CRF, the former significantly improves correctness. We can learn that the weakly supervised MRF model with iterative shape prior can also achieve satisfactory performance in multi-target extraction.

6 Conclusion

An automatic object extraction algorithm for optical remote sensing images is proposed in this article. First, the Active Basis algorithm was used to detect the fragment of object contour and to obtain the initial object contour by using fragment connection based on morphology. The Gaussian mixture models of objects and background were then built under the Bayesian framework and the prior information on object shapes was introduced. Finally, EM iteration and GraphCut optimization were combined for object segmentation. Our algorithm has the following advantages: (1) Morph-ActiveBasis needs only multiple images containing the objects or simple hand-drawn sketches of objects without other prior instructions. Less human intervention is required by the proposed method compared with other object segmentation methods that are based on feature classification; (2) Since EMContourMRF overcomes the problem of unorganized segmentation results, which occurs in general models that have only neighborhood prior, because EMContourMRF introduces prior information on object shapes and uses contours to constrain the segmentation process. Thus, EMContourMRF obtains better contours of objects; (3) EMContourMRF combines EM iteration and GraphCut optimization to estimate and to optimize the distribution models of objects and backgrounds repeatedly. EMContourMRF has relatively higher segmentation precision than object detection methods that are based on shape fragments.

The accuracy of object detection and segmentation is significantly affected by shadows. Thus, the removal of shadows is the focus of our follow-up work. Moreover, with the development of network information, automatic acquisition of images that contain more objects to construct a complete training database is a key step in our follow-up work to achieve better segmentation results independently.