Impulse Noise Filtering Using Robust Pixel-Wise S-Estimate of Variance

Abstract

A novel method for impulse noise suppression in images, based on the pixel-wise S-estimator, is introduced. The S-estimator is an alternative for the well-known robust estimate of variance MAD, which does not require a location estimate and hence is more appropriate for asymmetric distributions, frequently encountered in transient regions of the image. The proposed computationally efficient modification of a robust S-estimator of variance is successfully utilized in iterative scheme for impulse noise filtering. Another novelty is that the proposed iterative algorithm has automatic stopping criteria, also based on the pixel-wise S-estimator. Performances of the proposed filter are independent of the image content or noise concentration. The proposed filter outperforms all state-of-the-art filters included in a large comparison, both objectively (in terms of PSNR and MSSIM) and subjectively.

1. Introduction

Corruption by the impulse noise is a frequent problem which appears in digital images. It occurs as a consequence of transmission errors, timing problems in analog-to-digital conversion, or damaged pixel elements in image sensors [1]. Regardless of its origin, the impulse noise has two important aspects: only certain parts of the image pixels are corrupted by the noise and the intensities of contaminated pixels are significantly different from the other noise-free pixels in their neighborhoods. These properties can easily make any kind of subsequent processing, such as segmentation, edge detection, or object recognition, difficult or even impossible. Therefore, the suppression of the impulse noise is usually a required preprocessing step.

The major issue in impulse noise suppression is to satisfy two opposing requests. The corrupted pixels should be filtered whereas the image details have to be preserved. This task is exceptionally difficult because even the smallest amount of noise impulses which are not detected and filtered causes significant deterioration of image quality due to the nature of impulse noise. There have been proposed a large number of filtering techniques for removal of impulse noise. The classical approach is based on using median or its modifications [2]. These space-invariant methods are applied uniformly throughout the whole image, that is, apart from the noisy pixels, they unnecessarily change the noise-free pixels and impair image details. Most of the modern impulse noise filters utilize the solution based on switching scheme [3]. The noisy pixels are detected first and filtered whereas the noise-free pixels are left intact. Thus, this approach is space variant, and it is proven to be effective in preserving image details.

The impulse noise detection is usually performed by comparison of some robust statistics calculated in a local neighborhood to the corresponding fixed or adaptively calculated thresholds. A plethora of the algorithms has been developed which uses this approach, for example, switching median (SM) filter [3], three-state median (TSM) filter [4], multistate median (MSM) [5], adaptive center weighted median (ACWM) filter [6], state-dependent rank-order mean (SDROM) filter [7], progressive switching median (PSM) filter [8], conditional signal-adaptive median (CSAM) filter [9], pixel-wise MAD (PWMAD) filter [10], threshold boolean filter (TBF) [11], and so forth. The detectors of the previous filters are constructed heuristically, but it is also possible to use previous knowledge and machine learning techniques in order to find an optimal decision rule. Genetic programming is utilized in GP [12–14] filters, and neural networks are employed in improved adaptive impulsive noise suppression (IAINS) filter [15].

The other popular approach relies on fuzzy logic. Impulse noise detection in fuzzy-based techniques models ambiguities between noisy impulses and image structures in order to preserve image details [16–18]. Further enhancement of the fuzzy techniques is achieved by combining them with neural networks into neurofuzzy systems [19].

Robust statistics play a central role in impulse detection, being capable of producing correct estimates in the presence of unreliable data. The most frequently used statistics are the median and its variants such as center-weighted median. Nevertheless, robust statistics based on absolute differences are proven to be successful. The trilateral filter [20] was the first one which employed rank-ordered absolute difference (ROAD) statistics. Effective modifications are given by the rank-ordered logarithmic difference (ROLD) detector [21] and rank-ordered-relative difference (RORD) detector [22]. A slightly different technique, which is also based on absolute differences but makes use of directional information, is employed in the directional weighted median (DWM) filter [23].

After the detection of noisy impulses, selective, space-variant estimation is applied. The classical approach is based on the utilization of robust estimates of location, but recently an edge-preserving regularization method emerged as an alternative. This method was applied for the first time in the detail-preserving variational method (DPVM) [24]. However, it was applied uniformly for all the pixels in the image, which resulted in relatively moderate results. This estimation method showed much better results when it was combined with the impulse detection scheme in the adaptive center-weighted median with edge-preserving regularization: (ACWM-EPR) filter [25] or ROLD-EPR filter [21].

In this paper we concentrate on denoising images corrupted by the mixture of salt-and-pepper and random-valued impulse noises. It has already been shown with the DUMMY filter [14] that detection of pure salt-and-pepper impulse noises is almost a trivial problem. Therefore, improvements in salt-and-pepper noise filtering are directed either toward efficient implementation, like in a decision-based algorithm (DBA) filter [26] or toward better estimation, implemented by a fuzzy impulse noise detection and reduction method (FIDRM) [27], switching-based adaptive weighted mean (SAWM) filter [28], and the edge-preserving (EP) filter [29]. The more challenging impulse noise model is the random-valued model, and the filters designed to remove it are, in general, capable of treating salt-and-pepper noise as well. However, there exist some filters for which that is not the case [13, 20]. Therefore, it is an important property of impulse noise filters that they are capable of suppressing both types of noise equally well. In order to evaluate the overall performance of impulse noise filters, the mixed impulse noise model was proposed in [14]. The same impulse noise model is used in this paper.

In Section 2 the assumed noise model is described. Section 3 introduces a pixel-wise S-estimate of variance, and in Section 4 the proposed method for impulse noise suppression based on usage of this robust estimate is presented. The results are given in Section 5.

2. Noise Model

In this paper we considered a mixed impulse noise model [14], which is basically the composition of two well-known impulse noise models: random-valued and salt-and-pepper impulse noises. Let be the pixel, at the location , of the image containing mixed impulse noise. It is given as follows:

(1)

where and denote noise impulses generated according to random-valued and salt-and-pepper impulse noises, respectively. A noise-free image pixel at location is symbolized by . If and are the minimum and maximum values from the dynamic range of pixel values in the image, then and . In this way, half of the noisy pixels are corrupted by the random-valued impulse noise and the other half by salt-and-pepper noise. The reason for using the mixed impulse noise model is twofold. Firstly, the mixed impulse noise model is more realistic than existing models. The impulse noise is a result of disturbances caused by noise signals with random amplitudes. The amplitude of the noise signal can fall either into the dynamic range or out of that range. If it is out of the range, the corrupted pixel in the resulting noisy image will be saturated to the maximal or minimal value of the dynamic range, and that situation corresponds to the salt-and-pepper model. Alternatively, if the impulse noise is within the dynamic range, it will appear as random-valued impulse noise in the noisy image. Secondly, it is expected from the high-quality impulse noise filter to perform well in the presence of both salt-and-pepper and random-valued models. Since the filters should handle both impulse noise types equally, it is reasonable to choose a percent ratio for testing. Accordingly, the mixed impulse noise model represents the model which is suitable for proper evaluation of the impulse noise filters.

3. Pixel-Wise S-estimate

The well-known robust estimate of variance is the median of absolute deviations from median—MAD. It was utilized for impulse noise suppression in the ACWM filter and its pixel-wise modification in PWMAD [10]. MAD is a good estimate of variance, but it requires the calculation of the location estimate, and therefore it is not suitable when the underlying distribution is not symmetric. The information of variance estimate is used to distinguish between the regions that contain image detail and flat, homogenous regions, that is, regions with high and low variance, respectively. In transitional regions where edges are present, the signal can be hardly modeled by a symmetric distribution. Accordingly, it would be advisable to use a robust estimator which works well even for asymmetric distributions. The robust estimate of variance with such a property was proposed in [30]. It was called the S-estimate and it is given as follows:

(2)

where , is the sample containing elements. Firstly, for each the inner median of is calculated. This yields a new sample of elements, and their median (the outer median) gives the final estimate S. A beneficial property of the S-estimate is that it does not rely on a calculation of the location estimate, and consequently it produces an accurate estimate of variance for nonsymmetric sample distributions, that is, the image regions containing edges. The major drawback of the S-estimate is its computational complexity. For a sample of elements, it is necessary to calculate the median exactly times. This could be a serious problem in image processing, where the size of the sample is usually the square of the filtering window dimension and it should be calculated for each pixel.

It has already been shown in [10] that pixel-wise modification of the robust operator MAD can lead to a significant reduction of complexity with negligible degradation of the estimator's performance. In a similar manner, we propose a new estimate of variance, pixel-wise S-estimate (PWS), which is the modification of the original S-estimate based on the fixed spatial structure of image pixels. Let denote the set of coordinates in a window centered at the position , where the size of the window is and . The set of coordinates is defined as

(3)

Also, let denote the same set without the central coordinate, that is, . The absolute difference between the central pixel and neighboring pixel , from the window , is defined as

(4)

First, we define the median of absolute differences from the central pixel as a median value of the set obtained by calculating for each coordinate in :

(5)

where is high median value, which is the order statistic of rank . Since the differences are calculated with respect to the central pixel, MAd is calculated on the set for the sake of more efficient implementation.

Finally, we define the pixel-wise S-estimate as the median value of medians of absolute differences in a window :

(6)

The example in Figure 1 shows the PWS estimate of variance for the image Lena corrupted with 15% of mixed impulse noise. It is clear that the overall number of required median calculations for PWS in the image is only two per pixel, which is a significant improvement over the S-estimator. The key difference between PWS and S-estimator is in the way of calculation of the outer median in (2). All median calculations for the S-estimator are performed within a fixed window, whereas the PWS in the outer median shares the data calculated for neighboring pixels, which facilitates practical implementation of the proposed algorithm.

Figure 1

Pixel-wise S-estimation of the variance for Lena image corrupted with 15% of mixed impulse noise.

4. Proposed Method

The proposed detection scheme is based on utilizing both PWS and MAd statistics. Test image Bridge corrupted with of artificially added mixed impulse noise is given in Figure 2. Since the position of noisy pixels is known, we can calculate PWS and MAd values for every pixel. Scatter plots of those values are given in Figure 3 for noisy and noise-free pixels separately. Each image pixel in a plot is represented as a dot with coordinates given by its PWS and MAd values. The overlapping area between two plots is relatively large, which can be expected since some noisy pixels are hidden in image details and simply cannot be distinguished from them. Nevertheless, Figure 3(a) contains in total fewer pixels than Figure 3(b), because of the noise distribution in Figure 2. Therefore, the overlapping area contains many more noise-free than noisy pixels. Still, due to this overlapping, we cannot build a detector which is able to separate noisy and noise-free pixels in just one step. It is necessary to apply the whole filtering scheme iteratively, with some bias towards detail preservation—it is more important to preserve noise-free pixels than to leave some impulses undetected. Undetected impulses are likely to be removed in future iterations, while filtered details cannot be recovered.

Figure 2

Test image Bridge corrupted with of mixed impulse noise.

Figure 3

PWS-MAD scatter plots of noisy and noise-free pixels for the image Bridge corrupted with of mixed impulse noise. NoisyNoise-free

4.1. Detection

Ideally, we would be capable of labeling all the pixels which do not belong to the area where noise-free pixels are concentrated in scatter plot in Figure 3(b). Still, this is not an easy task, and we need a solution which is robust. We introduce the usage of a simple affine classifier to separate the plane defined by PWS and MAd values into two regions: noisy and noise-free. The classifier is given as follows:

(7)

where and are parameters which determine the decision boundary of the classifier. Since we propose the usage of an iterative filtering procedure, in the th iteration step the classifier generates a noise map having "1" at positions where noise is detected and "0" at positions labeled as noise-free. Accordingly, the noise map in the th filtering iteration is given as

(8)

where and denote the statistics calculated in the th iteration for the pixel at position . The values of the constants and depend on the iteration and they are determined experimentally.

The detector uses a recursive approach for the calculation of the noise map . This means that the pixels detected as noisy are immediately replaced by the initial estimate , and in a subsequent processing the new estimates are used for processing of the neighboring pixels. As the initial estimate we use median value calculated in a filtering window :

(9)

where

(10)

Therefore, the initial estimate is calculated using estimate as an input instead of if the pixel at position is already processed. In the next filtering iteration the output of the previous iteration is used as the input:

(11)

The same recursive approach is used when and values are calculated. Instead of the input values , the initial estimates are always used if they are already calculated. The initial estimates are later replaced by final estimates obtained by edge-preserving regularization. The details about the final estimation are given in Section 4.2.

We denote the proposed detection scheme PWS detector since the noise classifier is based on PWS and MAd statistics, but MAd is actually calculated within the calculation of PWS. The detector parameters and are determined experimentally. Parameter defines the slope of the classification line in the PWS/MAd plane, defined by (7), whereas defines the offset from the origin. It has been found through experimentation with images having 8 bits per pixel that the optimal value of should be in the range . In our simulations we set it to be constant in every iteration and to be at the middle of this range, that is, . On the other hand, the slope parameter should change throughout the iterations. In the beginning it should be in the range and it should become smaller in subsequent iterations, in order to become less conservative and allow detection of more noise. We reduce this parameter for a constant factor in each iteration . These parameters are very robust and produce satisfactory results for different noise levels and different images.

Figure 4 shows the distribution of noisy (red) and noise-free (blue) pixels together with the decision boundary. In each iteration the pixels below the boundary are labeled as noisy, and the estimation procedure described in Section 4.2 is applied to them. Modification of the parameter moves the decision boundary slowly upwards, thus allowing more impulses to be detected. In the last iteration the noisy and noise-free regions are overlapping, so any further filtering will not give additional improvements to the filtered image. In Figure 5 the enlarged detail of the actual distribution of noise-free pixels from the scatter plots given in Figures 4(e) and 4(f) is presented. The decision boundaries of detectors are also given. The majority of the noise-free pixels are still labeled correctly in iteration 5 while in iteration 6 the decision boundary passed through the region where the distribution is more dense. Therefore, the filtering should be stopped in this case at iteration 5, before a significant number of noise-free pixels are wrongly marked as noisy. In each iteration a relatively small percent of noise-free pixels are unnecessarily filtered due to misclassification. This undesirable effect cannot be completely avoided but can be possibly reduced by finding the classifier which better fits the border between noisy and noise-free pixels. Accordingly, we should keep in mind that total separation between those pixels is often impossible because some percentage of noise is blended with the image details and cannot be distinguished.

Figure 4

Scatter plots of noisy and noise-free pixels for the image Bridge corrupted with mixed impulse noise. Each plot displays the distribution of noisy and noise-free pixels together with the classification line.Iter 1Iter 2Iter 3Iter 4Iter 5Iter 6

Figure 5

The distribution of noise-free pixels in the th and th iteration of filtering image Bridge corrupted with mixed impulse noise. Iter 5Iter 6

It has been experimentally verified that the proposed detection procedure is robust to the variation of parameters and . However, the quality of the output is mainly influenced by the optimal number of iterations , because iterative filtering has to be stopped before it starts to severely destroy image details. This issue is discussed and handled by the algorithm described in Section 4.3.

The last parameter, which is the only one that actually has to be set manually, is the window size , defined in (3). We follow the rules given in [20]: if the noise ratio is higher than , we use window; otherwise the window of the size is applied. This yields satisfactory performance in most cases.

4.2. Estimation

Many space-variant filtering methods designed for suppression of the impulse noise use estimators based on the median and its derivatives. We utilize this kind of estimate just as a first approximation during the iterative impulse detection. The final estimate is found by replacing the noisy pixels with values found through the procedure of edge-preserving regularization (EPR) similar to [24]. We combine the proposed PWS detector with edge-preserving regularization and denote it as PWS-EPR filter.

Firstly, the pixels marked as noise-free are estimated by the observed value, that is, . The noisy pixels are estimated by minimizing the functional defined as follows:

(12)

where is the set of all pixels marked as noisy in the noise map, is the set of the four closest neighbors of and is the edge-preserving potential function [31]. Particularly, we use the following function:

(13)

similarly as in [25], where Note that in contrast to [25] or [32] the functional in (12) contains only a regularization term and not the data term. This is because the data are fitted exactly for the uncorrupted pixels, while for the corrupted pixels it is expected that the difference between observed end estimated values will be large due to the nature of the impulse noise.

A global minimization of the functional is very difficult. Therefore, we apply the optimization procedure for each pixel separately and repeat the procedure iteratively across the whole image until the process converges to the stable solution. Since the edge-regularization potential function is strictly convex, we perform the local optimization by Brent optimization method [33]. The same convergence criteria as in [34] are applied.

4.3. Stopping Criteria

The important issue which is common for iterative filtering approaches is to determine the optimal number of iterations. Most of the state-of-the-art algorithms set that parameter to some fixed value which gives satisfactory results in most cases [14, 21, 25] or set it according to noise concentration [20, 23]. Still, this is a challenging problem because the optimal number of iterations usually depends on both image content and noise concentration. The proposed PWS-EPR filter in each iteration calculates the PWS estimate of the variance, and this is utilized to determine the stopping criteria for the iterative filtering.

The PWS can be comprehended as an approximation of the S-estimate of variance. Therefore, the result of the estimation produced by PWS is more susceptible, that is, less robust to the presence of impulse noise. As the filtering with PWS-EPR filter goes through the iterations, the PWS will give a better approximation, since the noise concentration is continuously reduced and PWS estimates between the consequent iteration are becoming more and more similar. At some point, as a result of detector design, further continuation of the filtering starts to significantly destroy image details. The point when the filtering has to be stopped is illustrated in Figure 5 where the decision boundary starts to misclassify a significant number of noise-free pixels and therefore degrade image details. Hence, the difference between two consecutive PWS estimates starts to become more significant. So, the criteria for stopping further filtering by the PWS-EPR filter is to find the point where the similarity between consecutive PWS estimates changes trends from descending to ascending. This simple approach has proven to be very effective. As a similarity measure between consecutive PWS estimates we use mean relative absolute difference calculated across the whole image:

(14)

where the summation goes over all pixel coordinates, is the number of pixels in the image, and is the iteration number.

5. Results

The performance of the proposed PWS-EPR filter was compared to a number of state-of-the-art impulse noise filters. The experiments were conducted on standard test images and for different concentrations of mixed impulse noise. The parameters of the PWS-EPR filter were set to the constant experimentally found values as explained in Section 4. The only parameter which needs to be tuned with respect to the noise level is the window size used by the detector. It is set to be when noise concentration % and otherwise. The other parameters were set as follows: , , and . In all experiments, the parameters were kept constant, that is, they were not tuned for a particular image or noise level. The optimal number of iterations is calculated dynamically, as explained in Section 4.3.

In the comparison we included the standard median filter, TSM [4], ACWM [6], SDROM [7], PSM [8], PWMAD [10], Trilateral [20], DWM [23], FRINR, GP [14], ACWM-EPR [25], and ROLD-EPR [21] filter, having parameters optimized according to the suggestions given in the original references. The quality measures which we used are the well-known peak signal-to-noise ratio and mean structure similarity index (MSSIM) [35], which quantifies the resemblance in image structures between the original and filtered image. Figure 6 shows the PSNR and MSSIM results of compared filters obtained for the test image Barbara over the range of impulse noise concentration from to . Table 1 shows the obtained PSNR values for images Lena, Goldhill, Boats, and Bridge corrupted by , and of mixed impulse noise, whereas in Table 2 MSSIM values are given for the same experiments. In most cases the PWS-EPR filter outperforms the other filtering methods. It should be noted that it produces better results than the ACWM-EPR and ROLD-EPR filters, whose estimation modules are similar to the edge-preserving method utilized in the PWS-EPR filter. The overall conclusion is that PSW-EPR is superior compared to other filters in both PSNR and MSSIM values.

Table 1 Filtering results in PSNR (dB) for images corrupted with mixed impulse noise.

Table 2 Filtering results in MSSIM for images corrupted with mixed impulse noise.

Figure 6

Peak signal-to-noise ratio (PSNR) and mean structure similarity index (MSSIM) obtained as a result of filtering the image Barbara corrupted by the mixed impulse noise whose ratio is spanning from to .

The subjective quality is assessed by visual inspection. An enlarged detail from image Lena corrupted by of mixed impulse noise and denoised using different filters is shown in Figure 7. Only the best performing methods are compared visually. The proposed filter gives output which is the closest to the original picture. Although, the test image is corrupted with a relatively high level of mixed impulse noise, the PWS-EPR filter does not produce noticeable artifacts like DWM, FRINR, and ACWM-EPR or slight blurring like ROLD-EPR.

Figure 7

Details from the Lena test image corrupted with of mixed impulse noise, and results obtained by applying different filters. NoisyFRINRDWMACWM-EPRROLD-EPRPWS-EPR

6. Conclusion

This paper has two major contributions. Firstly, we introduced a pixel-wise S-estimator as an effective and computationally efficient robust estimator of variance that can be successfully utilized in an iterative scheme for mixed impulse noise filtering. In addition, we developed a novel method for determining the optimal number of filtering iterations, also based on the pixel-wise S-estimator. The proposed PWS-EPR filter outperforms other filters included in the comparison, both objectively (in terms of PSNR and MSSIM) and subjectively. Further improvements of the proposed filter are possible since the linear classifier used for noise detection is a rather simple solution. The utilization of a quadratic classifier instead which could provide better separation of noisy and noise-free pixels should further reduce the number of iterations, increase the detector accuracy, and improve the overall filtering performance.