Learning from label proportions for SAR image classification

3.1 Grid labeling concept

Grid labeling (GL) is introduced in this paper to acquire training data with high efficiency and at low cost. Instead of assigning one label to each pixel in the training set as in pixel labeling (PL), we assign a single label to each cell by grid labeling. All pixels in the cell will share the same label. Assume a proportion of the cells obtained by slicing the original SAR images are selected randomly and the training set is made up of pixels in these cells. The label for a cell C is determined by the major class with the maximum number of pixels in C. We call it “naive grid labeling” as it does not provide any other information than the label of the major class in a cell.

Comparison of grid labeling and pixel labeling is illustrated in Fig. 2. For pixel labeling in Fig. 2 c, the human expert carefully assigns a label for each pixel in the image, while for grid labeling, he or she just needs to estimate the major class in a cell and assigns the corresponding label to all the pixels in the cell, as in Fig. 2 b. Obviously, grid labeling will introduce label uncertainty into the training set, for the pixels with other classes than the major one are still assigned the major label forcefully, e.g., on the boundary of WL and WB in Fig. 2 b. But the potential of grid labeling in practical large-scale SAR image understanding makes it valuable to explore classification methods based on learning from grid labeling.

Although we usually accomplish pixel labeling by drawing a polygon on the image to annotate a region composed of pixels with the same class and fill it with a certain color, several factors will still make pixel labeling a hard and laborious task, such as various scales and shapes of the polygons, the unclear criterion how small a region can be ignored, and some regions with uncertain labels. Compared to traditional pixel labeling, the main advantage of grid labeling is its efficiency and operability, which includes the following: (1) Low cost and high efficiency. Grid labeling means greatly reduced cost and high efficiency, making large-scale SAR image classification more realizable; (2) Significant reduction of the difficulty in labeling HR SAR images. The backscattering effects of objects and scenes in HR SAR images are complex, making it difficult to precisely assign a label for each pixel. For grid labeling, the person in charge of labeling needs no special training and avoids expensive field surveys in the labeling work; (3) Grid labeling also emphasizes more context clues than pixel labeling, which means it is more suitable for scene understanding.

3.2 Proportional grid labeling

As mentioned above, naive grid labeling performs on the groups (grid cells) from directly slicing the images in spatial domain, which is an efficient and convenient way for sample labeling. But naive grid labeling provide too little information (only the label of the major class in a cell) on sample classes, which will inevitably introduce label uncertainty and eventually reduce the classification accuracy. According to the idea of learning with label proportions [9–11], we can learn a model to predict labels of the individual samples by grouping the training samples and providing proportions of the labels in each group. However, the current definition of label proportion [11] ignores spatial sources of samples in each group, which means samples in a group are not necessarily from the same local region, making it not convenient for sample labeling of remote sensing images. Thus, we introduce proportional grid labeling (proportional GL) by combining grid labeling with label proportions to implement rapid training set definition for SAR images. By estimating the proportions of each label in cell C, proportional GL not only can implement convenient and efficient sample labeling for SAR images as naive grid labeling but also can provide more information on sample classes.

where \({y_{j}^{*}}\) is the real pixel-level ground truth of x _j , |C _k | denotes the total number of samples in cell C _k . Then, proportional GL can be presented in three forms by giving label proportions for each class or the major class in a cell during the sample labeling process,

1) Multiclass GL: The task of multiclass GL is to estimate the proportion p _k (l _i ) for each \({{l}_{i}}\in \mathcal {L}\)(i=1,…,M) in a grid cell C _k by a human expert.

where \({{\mathcal {L}}^{\backslash {{l}^{(k)}}}}=\left \{{{l}_{j}}\in \mathcal {L}|{{l}_{j}}\ne {{l}^{(k)}}\right \}\) is the set of all labels except l ^(k). So the task of majorclass GL is just to give p _k for major class l ^(k) in a grid cell C _k by a human expert.

3) Naive GL: Based on the definition of majorclass GL, we can predefine p _k =1 for simplicity, which also means \({{p}_{k}}\left ({{\mathcal {L}}^{\backslash {{l}^{(k)}}}}\right)=0\). Then, the human expert only needs to indicate the majorclass label l ^(k) in a grid cell C _k , without estimating the proportion of any label. In this way, majorclass GL is indeed degraded to the naive grid labeling (naive GL) described in Section 3.1.

4.1 The learning model with label proportions

We build a learning model with label proportions based on the SVM classifier [16] with boosting strategy [17]. The SVM classifier takes the simple but efficient features (i.e., backscattering intensity, texture, and supertexture) proposed in [1] as its inputs and outputs the land cover types of the SAR images. Optimization of the training set from grid labeling is implemented by reweighting strategy of the boosting algorithm [17, 18], which has been improved by us through introducing a new penalty term for label proportions.

where ω _i is the weight for training sample x _i ; ω=[ ω ₁,…,ω _N ] ^T is the weight vector; y _i is the true label of x _i from grid labeling; L(·)≥0 is the loss function for the traditional SVM model; another loss function L _p (·)≥0 is designed to penalize the dissimilarity between the true label proportion p _k and the predicted label proportion \({{\tilde {p}}_{k}}\) based on ω; we define \({{\tilde {p}}_{k}}(\omega)=\frac {1}{|{{C}_{k}}|}\sum \limits _{i\in {{C}_{k}}}{[\!{{\omega }_{i}}>0]}\) because a nonzero value will be assigned to ω _i if the majorclass label l ^(k) is reliable to be the true label of x _i (i∈C _k ), i.e., y _i =l ^(k); φ(x _i ) maps x _i into a higher-dimensional space; λ and λ _p are the regularization parameters. Our task is to simultaneously optimize the weight ω and the SVM model parameters w and b to ensure \({{\tilde {p}}_{k}}\) estimated using ω is close enough to p _k .

Based on pixel-level training and label proportion constraints from grid labeling, this model not only can implement pixel-level classification for SAR images but also can improve the efficiency of sample labeling and reduce the corresponding cost. We build this model based on SVM because the training set can be optimized by reweighting the importance of each sample used for training in SVM model. This strategy has been adopted in transfer learning methods [7, 17]. And we can easily extend the number of classes for classification by defining a training set with rich classes for SVM classifier.

4.2 Model inference using LpcSVM algorithm

The unknown sample weight ω can be seen as a link between supervised learning loss L(·) and label proportion loss L _p (·) in Eq. (4). So optimization for Eq. (4) can be solved by alternating optimization strategy [11, 19], i.e., optimizing (w, b) and ω one at a time by fixing the others. For fixed sample weights ω=[ ω ₁,…,ω _N ] ^T , the optimization of Eq. (4) w.r.t w and b becomes a classic SVM problem. When w and b are fixed, ω can be updated according to label agreement and label proportion constraints. The baseline strategy of weight distribution update is to assign lower weights for misclassified or unreliable samples in the training set [20]. Assume the majority of samples in the training set are correctly labeled by grid labeling. So correctly classified (reliable) training samples are associated to strong weights and misclassified samples are assigned with weak weights in the reweighting process. An updated ω means an updated classifier, which is expected to be more trustworthy than previous ones, since it is trained with more reliable samples.

where the absolute loss \({{L}_{p}}\left ({{\tilde {p}}_{k}}(\omega),{{p}_{k}}\right)=|{{\tilde {p}}_{k}}(\omega)-{{p}_{k}}|\) is used in this paper. We adopt an approximate cell-wise optimization method for estimating weight ω _i in Eq. (7).

where E(l|x _i )=− log(P(l|x _i )). The class posterior distribution P(l|x _i ) is estimated by LIBSVM [16]. LIBSVM uses the method in [21] to estimate class probability for classification problem. The basic idea of the method is to estimate pairwise (i.e., one-against-one) class probability from decision values (or cost values) and then solve an optimization problem to get multi-class probability. Compared to the simple potential form E(l ^(k)|x _i ), the definition in Eq. (8) is expected to better depict the potential of x _i to be classified as l ^(k), by measuring the distance between the potential of label l ^(k) and the minimal value among all the other labels (i.e., the label most likely to replace l ^(k)). Smaller value of R(l ^(k)|x _i ) means l ^(k) is more reliable to reflect the true label of x _i .

Then, we sort x _i (i∈C _k ) in ascending order by R(l ^(k)|x _i ) and note the sorted sample sequence as \(\phantom {\dot {i}\!}{{x}_{{{\delta }_{i}}}}\), where δ _i is the index value (or position) of x _i in the sorted sequence.

where N _s =int(p _k |C _k |) is the sample number of major class l _k in C _k ; \({{N}_{m}}=\frac {\left | {{C}_{k}} \right |}{M}\) is the number of samples to be assigned with weight 1; and θ is a parameter to control the degree of weight weakening for N _m <δ _i ≤N _s . Typically, θ=0.5 will work well in most cases. According to the formula, we assign strong weight (ω _i >0) for the first N _s samples with smallest R(l ^(k)|x _i ) (where i satisfies δ _i ≤N _s ) and zero weight for the others. To further enhance the robustness of our method to the estimation error of label proportion p _k , the sample weight is designed to introduce a penalty term for N _m <δ _i ≤N _s , instead of simply assigning ω _i =1 for all δ _i ≤N _s , as illustrated in Fig. 3.

where we use L(x _i ) to denote L(y _i ,w ^T φ(x _i )+b) for short.

Let ω ^′ _i be the updated version of sample weights ω _i using the reweighting strategy in Eq. (9). Then, ω ^′ _i is decreasing (non-increasing) with δ _i (see Fig. 3). Recall that \(\phantom {\dot {i}\!}\{{{x}_{{{\delta }_{i}}}}\}\) is the sample sequence after sorting { x _i } in ascending order of the potential \(R\left ({{l}^{(k)}},{{x}_{{\delta }_{i}}}\right)\), which means \(R\left ({{l}^{(k)}},{{x}_{{\delta }_{i}}}\right)\) is increasing (non-decreasing) with δ _i . For a sample \(\phantom {\dot {i}\!}{{x}_{{{\delta }_{i}}}}\), a smaller value of the potential \(\phantom {\dot {i}\!}R\left ({{l}^{(k)}},{{x}_{{{\delta }_{i}}}}\right)\) indicates a greater probability of l ^(k) to be the true label of \(\phantom {\dot {i}\!}{{x}_{{\delta }_{i}}}\), which also means a smaller value of sample cost \(\phantom {\dot {i}\!}L({{x}_{{\delta }_{i}}})\) in Eq. (7). So \(\phantom {\dot {i}\!}L({{x}_{{\delta }_{i}}})\) is also increasing (non-decreasing) with δ _i .

which means the cost function in Eq. (7) is minimized by setting higher weight ω ^′ _i to lower sample cost \(\phantom {\dot {i}\!}L({{x}_{{\delta }_{i}}})\) using our LpcSVM algorithm.

Besides, we choose \({{N}_{m}}=\frac {\left | {{C}_{k}} \right |}{M}\) in Eq. (7) for the following reason. As there are M labels for classification, the infimum of p _k is \(\inf ({{p}_{k}})=\frac {1}{M}\). We get \(\inf ({{N}_{s}})=\inf ({{p}_{k}})\times \left | {{C}_{k}} \right |=\frac {\left | {{C}_{k}} \right |}{M}\). Despite possible noises in p _k , at least \({{N}_{m}}=\frac {\left | {{C}_{k}} \right |}{M}\) samples in each cell should be made full use of by assigning weight 1 to them. A larger value of δ _i means a less reliable sample. In our reweighting strategy, weakened weights are assigned to less reliable ones (i.e., N _m <δ _i ≤N _s ) among the N _s samples chosen for the next iteration (see Fig. 3). Then, small fluctuation of N _s will not cause considerable changes in the weight distribution, making the method of better robustness to possible estimation errors in label proportions. Experimental performance of the robustness of our method to the estimation error in label proportions will be described in Section 5.4.

By combining label agreement [17] and label proportion similarity between the predicted and true labels, our reweighting strategy will not only weaken the role of unreliable samples in the training set T but also keep the important role of a certain number of reliable samples within the constraints of label proportions for the major class in each cell.

The updated ω is used to retrain the SVM classifier. After a number of iterations, strong weights will be assigned to the reliable samples and weak weights to the others, producing an optimized pixel-level training set (in form of optimized sample weights). Finally, the classifier will predict the label for each sample in the test data using the optimized model parameters.

Although partially inspired by the idea of ∝SVM [11], our model is essentially different with ∝SVM in the following respects: (1) ∝SVM performs on multiclass label proportions, while our model performs on majorclass label proportion, which makes sample labeling more efficient; (2) ∝SVM only deals with binary classification, while our model deals directly with multi-class classification; (3) Besides, our model is optimized based on sample reweighting, which is totally different with ∝SVM and also makes multiclass classification realizable.

5.1 Dataset description and experiment setting

The SVM classifier is configured with radial basis function (RBF) kernel and probability estimation [16]. In our experiment, the parameter λ in Eq. (7) is tuned from {0.1, 1.0, 10} and we found λ=1.0 works better for our test. The parameter λ _p in Eq. (7) is tuned from { λ, 10 λ, 100 λ, 1000 λ}. A larger value of λ _p means introducing more penalty from label proportions. Theoretically, the parameter θ in Eq. (9) controls the penalty on the less reliable training samples with N _m <δ _i ≤N _s . A larger θ means assigning larger weights to the corresponding samples. In our test, we tune θ from {0.3, 0.4, 0.5, 0.6, 0.7} and found θ=0.5 works better for our experiment. We use efficient low-level features such as backscattering intensity, texture, and supertexture [1] for the SVM classifier to take as input. The features are defined as follows.

where \(\mu _{i}^{sup}\) is the average of \(f_{j}^{tex}\) and \(\sigma _{i}^{sup}\) is the standard deviation of \(f_{j}^{tex}\) in the neighborhood N(i) of patch i, n is the number of patches in N(i). A 5 ×5 rectangular neighborhood is exploited in our work for an appropriate measurement of scene homogeneity and texture context (such as a piece of residential area or green field). Then, the feature vector \({{f}_{i}}={{\left [f_{i}^{I},f_{i}^{tex},f_{i}^{sup}\right ]}^{T}}\) for each pixel x _i is taken as input for the SVM classifier.

We have designed four groups of experiments to test our method from different aspects. The first is to show overall experiment results by comparing our method to the baseline methods. The second is to analyze the classification performance of our method with cell size and iteration varying. The third is to explore the robustness of our method to p _k estimated in majorclass GL. The last is to further apply our method to classification of a simulated dataset. Besides, we also discuss the efficiency and cost advantages of our method over traditional methods at the end of this section.

5.2 Overall experimental results

As can be seen from Table 2, GL+SVM does not perform so well due to the uncertain labels introduced by grid labeling. But ∝SVM performs better than GL+SVM through learning from label proportions. And our method (GL+LpcSVM with and without noise n _k in p _k ) outperforms ∝SVM in classification accuracy with about 1.5%. Moreover, ∝SVM cannot deal with multiclass classification directly, which means extra computing will be needed for ∝SVM to deal with multiclass classification. And ∝SVM is based on detailed label proportions, which also means more cost than LpcSVM (based on majorclass GL) in sample labeling. Although it is unfair to compare our method with PL+SVM for the latter is based on the accuracy training set from pixel labeling, we still take PL+SVM as a baseline method. GL+LpcSVM (with and without noise n _k in p _k ) demonstrates comparable classification accuracy and Kappa coefficient with PL+SVM, while reducing labeling cost significantly by adopting grid labeling.

The classification results of Tianjin and Rosenheim areas using our method and the baseline methods are presented in Fig. 4. The ground truth maps are labeled manually by referring to the corresponding optical images, with undefined areas in black color. For all methods, no postprocessing steps are added after classification using SVM. In the experiment, we adopt a cell size of 200 ×200 and perform four iterations for LpcSVM. The criteria for selecting cell size and iteration number will be explained in Section 5.3.

5.3 Classification performance with cell size and iteration varying

The proportion of all mislabeled samples (i.e., \({{p}_{\text {mis}}}=\frac {1}{K}\sum \limits _{k=1}^{K}{(1-{{p}_{k}})}\)) in the training set as a function of the cell size of grid labeling is illustrated in Fig. 5. As can be seen from the figure, a larger cell size in grid labeling basically means a larger proportion of incorrectly labeled (mislabeled) samples in the training set. Thus, the cell size can be used as a direct measure of the total proportion of mislabeled samples, which will influence the classification accuracy of the proposed LpcSVM classifier. Larger cell size means lower cost for sample labeling as the number of cells to be labeled is smaller, but it also means lower classification accuracy as the training set contains more unreliable samples (i.e., higher mislabeled proportion). So a balance between labeling cost and classification accuracy should be found in the application of grid labeling. The balance can be reached by choosing the right cell size.

Classification accuracy using GL+LpcSVM for the two SAR images as a function of the iteration number of LpcSVM under different cell sizes is shown in Fig. 6. We can conclude that, for a certain cell size, the classification accuracy on the two SAR images increases with the iteration numbers of LpcSVM, remaining stable after four iterations. More than four iterations will not improve the classification accuracy obviously, even though not reducing the accuracy in our test. So about four iterations of LpcSVM is applied in our experiments. When the cell size changes from 1 ×1 to 200 ×200, the curve of classification accuracy falls gradually, but it drops dramatically from the cell size of 300 ×300. Consequently, we adopt a cell of about 200 ×200 pixels in the application of grid labeling to reach a balance between labeling cost and classification accuracy (as the results in Table 2 and Fig. 4).

5.4 Robustness to p _k estimated in majorclass GL

Since label proportions estimated by human in grid labeling may differ from the true label proportions, there possibly exist errors in label proportion p _k . As we have introduced a penalty of weights at the end of the sequence of N _s samples chosen for the next training in the reweighting formula of Eq. (9), small errors in p _k will not degrade the performance of our method. Influence of label proportion errors on classification accuracy for the two SAR images is presented in Fig. 7. The test is performed with a cell size of 200 ×200 and four iterations. We simulate the errors in label proportion by adding noise n _k of a normal distribution with mean μ=0 and standard deviation σ=0.05∼0.20 to the existing label proportion p _k . Then, the noisy label proportion is p _k =p _k +n _k , where n _k ∼ N(μ,σ ²). According to the three-sigma rule, the value of n _k will basically fall in [μ−3σ,μ+3σ], e.g., [-0.15, 0.15] for σ=0.05. As can be seen from the figure, our method still maintains a relative high level of classification accuracy under the influence of errors in label proportions.

In order to estimate the standard deviation value of actual label proportion errors, grid labeling is performed on nearly 4000 cell regions from the two full-scene TerraSAR-X images used in the experiment. Five persons participate in the grid labeling stage, and each person labels each image twice. The order of cell regions in each image is randomly generated for each labeling by each person. So we obtain ten groups of different labelings for the cell regions in each image. Based on our grid labeling activities, we found the standard deviation value of label proportion errors in grid labeling is basically around σ=0.05, which means the value of label proportion errors falls in the confidence interval [-0.15, 0.15] with the probability of 99.7%, based on the three-sigma rule mentioned above. However, our method can still provide desirable classification results (85.92% for Tianjin and 67.06% for Rosenheim) even under a noise with σ=0.10 (see Fig. 7).

Besides, if no label proportions are provided (i.e., p _k =1, k=1,…,K as defined in Naive GL), our method can still work relatively well, outperforming GL+SVM about 1% in classification accuracy (see the case of σ= max in Fig. 7). In this condition, all the samples in cell C _k are selected for the next iteration (i.e., N _s =|C _k |), but they are reweighted differently according to the reliability of their labels, without the support of label proportions.

5.5 Classification with the simulated dataset

The dataset of a simulated SAR image is also used to further test our method. In order to simulate a SAR image of a large scene for classification, we adopt a widely used speckle-statistics-based simulation method as in Section 4.1.2 of [24]. Four types of homogeneous areas are considered to represent four land cover types in the simulated SAR image, and the pixels in each type of homogeneous areas follow a Rayleigh distribution. We implement each type of homogeneous area by setting the real and imaginary components of each pixel in the area to follow an independent and identical Gaussian (normal) distribution with zero mean and a variance σ ². Then, the amplitude A of the pixels follows a Rayleigh probability density \(p(A)=\frac {A}{{\sigma }^{2}}\exp \left (-\frac {{A}^{2}}{2{{\sigma }^{2}}} \right),A\ge 0\) with its mean \(\sigma \sqrt {\frac {\pi }{2}}\) and the variance \(\frac {4-\pi }{2}{{\sigma }^{2}}\).

We s et σ=50, 110, 130, 150 respectively to generate the four types of homogeneous areas (from dark to bright). The truth map of the TerraSAR-X image of Rosenheim is used for spatial configuration (such as locations, sizes, and shapes) of different homogeneous areas in the simulated image. Finally, we obtain a simulated image with size of 9504 ×8330, which contains four types of homogeneous areas to represent four different land cover types.

5.6 Efficiency and cost advantages

As mentioned above, the classification accuracy of our method (GL+LpcSVM) is slightly lower than PL+SVM as the latter is based on the accuracy training set from pixel labeling. But we emphasize the efficiency and cost advantages of our method over PL+SVM. For efficiency measuring, we consider the total classification time t as the sum of training set definition time t ₁ and time consuming t ₂ of the SVM classifier. Even using polygon annotation, the efficiency of pixel labeling will still be very low due to various scales and shapes of the polygons, the unclear criterion how small a region can be ignored, and some regions with uncertain labels. It will be shown that the efficiency of classification can be improved significantly through the great reduction of training set definition time t ₁ based on our method.

We have developed a software tool for proportional grid labeling to define the training set of classification, as shown in Fig. 8. The average time consuming for classification on the TerraSAR-X images of Tianjin and Rosenheim areas using the baseline PL+SVM, and our GL+LpcSVM is provided in Table 4. The test was conducted on an Intel i5-2520M dual core processor clocked at 2.50 GHz. For PL+SVM, t ₁ is the time of pixel labeling, and for GL+LpcSVM, it is the time of grid labeling using the developed labeling tool. From the table, we can see that t ₁ is much larger than t ₂ for both methods. Our method makes the total classification time t reduced greatly by decreasing the training set definition time t ₁ greatly and keeping the LpcSVM inference algorithm reaching stable within just three or four iterations (see Fig. 6), which means a slightly increased t ₂.

Method	Training set	Classifier	Total time t
	definition time t 1	consuming time t 2
PL+SVM	514	22	536
GL+LpcSVM	128	37	165

In addition, the cost of land cover classification has been reduced significantly with the reduction of training set definition time. More importantly, expensive field surveys can be avoided for sample labeling, making our method suitable for practical large-scale SAR image understanding.