Local distortion resistant image watermarking relying on salient feature extraction

2.1 Image normalization

and $T=\left(\begin{array}{ccc}1& 0& -d1\\ 0& 1& -d2\\ 0& 0& 1\end{array}\right)$ is a translation matrix, $X=\left(\begin{array}{ccc}1& \beta & 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)$ is a x-shearing matrix, $Y=\left(\begin{array}{ccc}1& 0& 0\\ \gamma & 1& 0\\ 0& 0& 1\end{array}\right)$ is a y-shearing matrix, and $S=\left(\begin{array}{ccc}\alpha & 0& 0\\ 0& \delta & 0\\ 0& 0& 1\end{array}\right)$ is a scaling matrix.

where ${\mu }_{pq}^{\left(XT\right)}$ are the central moments of I _XT (x, y) which is the image I _T (x, y) after x-shearing normalization. Finally, the values of α and δ are derived given that I _{Y XT} (x, y) (the image I _XT (x, y) after y-shearing normalization) is resized to a specific size (e.g., 512 × 512 in our experiments) to provide the final normalized image I _{SY XT} (x, y). The signs of these parameter values are determined by the constraint that both ${\mu }_{50}^{\left(SYXT\right)}$ and ${\mu }_{05}^{\left(SYXT\right)}$ are positive. Examples of the original "Lena" and "Lake" images and respective normalized images using the above described method are shown in Figure 1.

This normalized representation of the original image is the input for the next step of preprocessing that is necessary for both watermark embedding and detection.

2.2 Feature extraction

The second step of the preprocessing stage is the feature extraction step. A great variety of feature extraction methods has been proposed in the literature. Lately, there is a tendency of using the so-called scale-space methods such as SIFT [20] for watermarking purposes [18, 21–23]. In our study, we employed this as well as other feature detectors proposed in the literature, but not in the context of image watermarking, during the past few years. These detectors are, more specifically, the radial symmetry transform (RST) introduced in [24], the speeded up robust features (SURF) [25, 26] and the features from accelerated segment test (FAST) [27, 28]. As we will show in the experimental results section, all of them perform adequately well for our application, although their relative performance varies.

2.2.1 Radial symmetry transform

To compute the RST first we have to construct two images, the magnitude projection image M _n and the orientation projection image O _n of the normalized image at every radius n that we have selected. These images are initialized to zero and are subsequently updated at each point depending on how the point is affected by the gradient vector at a point a distance n away. Let p = (x, y) be a point and g(p) the gradient vector at that point, determined by applying the 3 × 3 Sobel operator at the respective point of the normalized image. The coordinates of the so-called positively-affected pixel are

${\mathbf{p}}_{+ve}\left(\mathbf{p}\right)=\mathbf{p}+\mathsf{\text{round}}\left(\frac{g\left(\mathbf{p}\right)}{∥g\left(\mathbf{p}\right)∥}n\right),$

(7)

and those of the negatively-affected pixel are

${\mathbf{p}}_{-ve}\left(\mathbf{p}\right)=\mathbf{p}-\mathsf{\text{round }}\left(\frac{g\left(\mathbf{p}\right)}{∥g\left(\mathbf{p}\right)∥}n\right),$

(8)

The pixel values of the magnitude projection and orientation projection images are updated as follows

$M_n\left({\mathbf{p}}_{+ve}\left(\mathbf{p}\right)\right)=M_n\left({\mathbf{p}}_{+ve}\left(\mathbf{p}\right)\right)+∥g\left(\mathbf{p}\right)∥,$

(9)

$M_n\left({\mathbf{p}}_{-ve}\left(\mathbf{p}\right)\right)=M_n\left({\mathbf{p}}_{-ve}\left(\mathbf{p}\right)\right)-∥g\left(\mathbf{p}\right)∥,$

(10)

$O_n\left({\mathbf{p}}_{+ve}\left(\mathbf{p}\right)\right)=O_n\left({\mathbf{p}}_{+ve}\left(\mathbf{p}\right)\right)+1,$

(11)

$O_n\left({\mathbf{p}}_{-ve}\left(\mathbf{p}\right)\right)=O_n\left({\mathbf{p}}_{-ve}\left(\mathbf{p}\right)\right)-1.$

(12)

Next, we have to define

${\mathit{Õ}}_n\left(\mathbf{p}\right)=\left\{\begin{array}{cc}{O}_n\left(\mathbf{p}\right)\hfill & \mathsf{\text{if }}{O}_n\left(\mathbf{p}\right)<k_n\hfill \\ k_n\hfill & \mathsf{\text{otherwise}}.\hfill \end{array}\right.$

(13)

where k _n is a scaling factor to normalize M _n and O _n across different radii. Once Õ _n is defined, we compute

$F_n\left(\mathbf{p}\right)=\frac{M_n\left(\mathbf{p}\right)}{k_n}{\left(\frac{\left|{\mathit{Õ}}_n\left(\mathbf{p}\right)\right|}{k_n}\right)}^{\alpha },$

(14)

where α is the radial strictness parameter. The larger the value of α, the stricter the required radial symmetry. Finally, F _n is convolved with a 2D Gaussian filter A _n to produce the radial symmetry contribution at radius n

$S_n=F_n*A_n$

(15)

The overall RST (symmetry map) is calculated by simply averaging radial symmetry contributions for all of the radii considered

$S=\frac{1}{\left|N\right|}\sum _{n\in N}{S}_n$

(16)

where N is the set of radii. A non-maximum suppression and thresholding algorithm [29] is applied to the symmetry map S to localize the strongly symmetric points of the normalized image. An example for the images of Figure 1 is depicted in Figure 2 for N = {1, 3, 5} and α = 1. The value of the radius for non-maximum suppression was chosen to be 3 and that of the threshold to be 5.

2.2.2 Scale-invariant feature transform

The main idea of this detector is to search for candidate stable feature points across a series of image scales. First, the so-called scale space of the normalized image is constructed by convolving the image I(x, y) with a variable-scale Gaussian $G\left(x,y,\sigma \right)=\frac{1}{2\pi {\sigma }^2}{e}^{-\left(x^2+y^2\right)/2{\sigma }^2}$

$L\left(x,y,\sigma \right)=G\left(x,y,\sigma \right)*I\left(x,y\right)$

(17)

The potentially stable feature points are detected as local extrema of the function D(x, y, σ) constructed as follows

$D\left(x,y,\sigma \right)=\left(G\left(x,y,k\sigma \right)-G\left(x,y,\sigma \right)\right)*I\left(x,y\right)=L\left(x,y,k\sigma \right)-L\left(x,y,\sigma \right)$

(18)

that is, a convolution of the image with a difference of Gaussians. k is a factor that determines the difference between consecutive scales. An octave of scale space is a series of D(x, y, σ) functions spanning a doubling of σ. Each octave is divided in s intervals and, thus, k = 2^1/s. For each new octave, the Gaussian image produced with the doubled value of σ at the previous octave is first downsampled by a factor of 2 at each dimension. The local minima and maxima are found by 3D search in the 8 neighbors of the current scale and the respective 9 neighbors in each of the previous and the next scale.

To correctly localize feature points, candidate points are fitted to the nearby data by interpolation. The Taylor expansion of the function D(x, y, σ) is given by

$D\left(\mathbf{x}\right)=D+\frac{\partial D^T}{\partial \mathbf{x}}\mathbf{x}+\frac{1}{2}{\mathbf{x}}^T\frac{{\partial }^{2}D}{\partial {\mathbf{x}}^2}\mathbf{x}$

(19)

where D and its derivatives are calculated at the candidate feature point and x = (x, y, σ) ^T is the offset from this point. The location of the extremum $\widehat{\mathbf{x}}$ is found by taking the derivative of this expansion and setting it to zero, giving

$\widehat{\mathbf{x}}=-\frac{{\partial }^2{D}^{-1}}{\partial {\mathbf{x}}^2}\frac{\partial D}{\partial \mathbf{x}}$

(20)

If the offset $\widehat{\mathbf{x}}$ is larger than 0.5 in any dimension, then the extremum should be closer to another candidate feature point. If so, the interpolation is again performed around a different point. Otherwise the offset is added to the candidate point to produce the interpolated estimate of the extremum.

To discard feature points of low contrast, the value of the second-order Taylor expansion is computed at the offset $\widehat{\mathbf{x}}$. If this value is less than 0.03 then the candidate point is discarded. Otherwise it is kept, and its final location and scale are, respectively, y + $\widehat{\mathbf{x}}$ and σ, where y is the original location of the candidate point at scale σ.

Another action that should be taken is to eliminate feature points with strong edge response. To do so, we first have to compute the second-order Hessian matrix H

$H=\left[\begin{array}{cc}{D}_{xx}& D_{xy}\\ D_{xy}& D_{yy}\end{array}\right]$

(21)

whose eigenvalues are proportional to the principal curvatures of D. If we let α be the larger eigenvalue and β the smaller one, then it can be shown that

$R=\frac{Tr{\left(\mathbf{H}\right)}^2}{Det\left(\mathbf{H}\right)}=\frac{{\left(r+1\right)}^2}{r}$

(22)

where r = α/β, Tr(H) = D _xx + D _yy = α + β is the trace of H and Det(H) = D _xx D _yy -(D _xy )² = αβ is the determinant of H. If the ratio R for a certain candidate feature point is larger than (r _th + 1)²/r _th , then the feature point is rejected. The method sets the threshold eigenvalue ratio to r _th = 10.

In our experiments the values of the various parameters involved in this method were chosen in accordance with [20]. Only the strength threshold for local maxima of the scale space was chosen to be equal to 0.05 to reduce the number of produced feature points. Examples of feature points extracted from the normalized versions of "Lena" and "Lake" are shown in Figure 3.

2.2.3 Speeded up robust features

This method was introduced as an alternative to SIFT focusing on computational cost reduction. A fast way of computing the Hessian matrix using integral images is proposed. This approach approximates the second order Gaussian derivatives by box filters. These, in turn, are used to compute the approximate determinant of the Hessian matrix. Instead of subsampling the filtered image of a previous layer, the scale space is constructed by increasing the filter size. For each new octave, the filter size increase per layer is doubled, and so is the sampling interval for the extracted feature points.

In the experiments that we conducted, the number of octaves that were analyzed was 5, the initial sampling interval was 2 and the Hessian response threshold was chosen to be 0.004. The feature points extracted from the normalized versions of "Lena" and "Lake" are presented in Figure 4.

2.2.4 Features from accelerated segment test

This feature detector should be more precisely called a corner detector. To test if a certain pixel p is a corner, 16 pixels lying on a circle centered at this pixel (specifically, a Bresenham circle of radius 3) are tested for similarity of intensity to the center pixel. If N contiguous pixels lying on this circle are all brighter than the center pixel by a quantity T (that is I_p→x≥ I _p +T, x ∈ {1 . . . 16}) or darker than it by the same quantity (that is I_p→x≤ I _p - T, x ∈ {1 . . . 16}), then the center pixel is considered a corner. A non-maximum suppression step follows to reduce the number of corner points. Since there is no score function on which to apply the suppression, we define one as [28]

$V=max\left(\sum _{x\in S_{\mathsf{\text{bright}}}}|I_{p\to x}-I_p|-T,\sum _{x\in S_{\mathsf{\text{dark}}}}|I_p-I_{p\to x}|-T\right)$

(23)

where

$\begin{array}{c}{S}_{\mathsf{\text{bright}}}=\left\{x|I_{p\to x}\ge I_p+T\right\}\\ S_{\mathsf{\text{dark}}}=\left\{x|I_{p\to x}\le I_p-T\right\}\end{array}$

(24)

After suppression only the candidates having score value greater than all their 8 neighbors are preserved. The parameter values used in our experiments where N = 12 and T = 60. For the "Lena" and "Lake" images, the feature points extracted from their normalized versions are shown in Figure 5.

3.1 Watermark embedding

The original aim is to insert the watermark in the DCT transform domain - other domains such as the space/spatial-frequency domain [31] could alternatively be employed - of the normalized image or, equivalently, insert the inverse DCT of the watermark in the spatial domain of the normalized image. However, by doing so, we would afterwards have to inversely normalize the watermarked normalized image to obtain the watermarked original image so that the watermark embedding process would be complete. This, as pointed out in Section 1, would impose interpolation errors, resulting in a version of the image that would be visibly corrupt compared to the original, even in areas that would not be normally affected by watermark embedding. To avoid this image degradation we choose to embed the inversely normalized version of the inverse DCT of the original watermark in the original image. Additionally, the watermark is to be embedded in all areas corresponding to the extracted feature points of the normalized image in a similar fashion as in [32]. This is done to increase watermark robustness as it is possible that not all originally detected feature points will also be detected after some attack. The overall embedding procedure is depicted in Figure 7.

where w _o (x, y) = w(x _b , y _b ) according to Equation (28), f _i (x, y) with i = 1 . . . M are the areas of the original image where the watermark is to be embedded and $f_i^w\left(x,y\right)$ are the respective watermarked areas. An example of a watermarked version of the image "Lake" of PSNR = 24.69 dB using RST and its amplified difference from the original is given in Figure 8. We can notice that some embedding areas may overlap because of the proximity of the corresponding feature points. We prefer to use all feature points as embedding area centers instead of applying some kind of criterion to select some of them. That is because we cannot be certain about the repeatability of feature points (that is, the probability that a specific point will be extracted in any altered version of the image). Since the watermark is embedded around all extracted points, it is also going to be detected around all feature points extracted during the detection stage, as it will be described in the following section. Thus, to cover the case of overlapping areas, it would be more appropriate to describe embedding in an iterative manner

where i = 1, . . . , M, w _i (x, y) is the image with same size as f(x, y) and non-zero only in the i th embedding area (where w _o is located), and $f_0^w\left(x,y\right)=f\left(x,y\right)$.

An example of applying this rule is given in Figure 9. The watermarked image now has PSNR = 40 dB and, in contrast to Figure 8, the watermark is hardly visible.

3.2 Watermark detection

To perform watermark detection, the preprocessing step is needed as for watermark embedding. This means that the watermarked and, possibly, attacked image is first geometrically normalized and feature extraction is performed in the normalized image in the same manner as in the embedding stage (using one of the methods described in Section 2.2). Figure 10 shows the result for the watermarked image of Figure 9. As one can see, a great percentage of the originally extracted feature points used for watermark embedding (see Figure 2) are still present in the normalized watermarked image. Therefore, the watermark will be detected accurately in all respective areas. Since, as pointed out in Section 3.1, no algorithm for selection of certain feature points has been established, watermark detection is going to be performed in all corresponding areas. An outline of the detection procedure is shown in Figure 11.

4.1 Local geometric attacks

One classic non-geometric attack is column and line removal. In Figure 12 we can see results for this attack where the pair of values inside the parentheses denotes the number of columns and lines of the image that have been removed, and which are equidistant. We can notice that our technique performs better for all employed feature detectors. This was expected since the state-of-the-art techniques affect the image globally and cannot withstand attacks that modify image contents. The SIFT-based version of our technique demonstrated the best performance followed by RST-based and SURF-based which have similar performance and finally FAST-based which is still better than the older techniques.

The next local distortion considered was the Stirmark attack. The experiment involved varying the jitter strength parameter from 1 to 7. As one can see in Figure 13, the proposed technique is superior to the technique by Dong et al. for all versions and especially for the SIFT-based one, but the technique by Tian et al. provides better performance for all cases but one.

Another attack considered in this category was image band cropping. The idea is to crop a band of certain width around the boundaries of the image. The band width in our experiments varied from 3 to 11 pixels as one can see in Figure 14. We can notice that the state-of-the-art techniques are seriously affected even by a small amount of cropping, whereas the various versions of our technique are always more robust and slowly degrade as the band gets wider. Although all versions provide similar performance, the SIFT-based version appears to prevail. This is, again, an expected behavior since the state-of-the-art techniques are not designed to withstand attacks that severely modify the global spectral representation of the image.

4.2 Global geometric attacks

Another category of possible distortions is that of global geometric attacks. These include rotation, scaling, shearing and combinations of them (i.e., general affine transforms). The first of these attacks presented here is the shearing attack. In this experiment, the varying parameters were the shearing percentages in both x and y axes. The results shown in Figure 15 prove that the technique by Tian et al. is not resistant against such an attack, which is expected since the technique does not apply affine normalization on the original image prior to watermark embedding. Performance, however, is excellent for the rest of the methods, with the SIFT-based version providing slightly better robustness than the technique by Dong et al. which, in turn, is a little more robust than the rest of our versions.

In the case of scaling, we conducted experiments with the scaling factor taking values as shown in Figure 16. The various methods do not present great differences in performance. However, the technique by Dong et al. is the best in all cases but one. The SIFT-based version of our technique is the next in order of performance, followed by the SURF-based and the RST-based versions and the technique by Tian et al. which alternate in terms of performance for the various parameter values, and finally the FAST-based version which exhibits the lowest robustness.

Rotation followed by cropping out the central region that does not contain black border pixels and finally scaling to the original size has next been tested with the varying parameter being the rotation angle, as presented in Figure 17. The technique by Tian et al. cannot withstand this attack. On the contrary, the technique by Dong et al. is superior, although the SIFT-based version of our technique is very close to it in terms of robustness, followed by SURF-based, RST-based, and FAST-based.

Another example of an attack comprising of different stages is shown in Figure 18, where successive downsampling and upsampling has been performed in the watermarked images. The pairs of values in parentheses correspond to the downsampling and upsampling factor, respectively. We can notice that all methods display similar performance with the technique by Tian et al. presenting the least varying robustness. The SIFT-based version appears to be, again, the best among all versions of our technique.

Finally, an experiment involving general affine transform was conducted, which showed that the performance of the proposed technique is comparable to that provided by the technique by Dong et al., as one can see in Figure 19. All techniques but the one by Tian et al. survive this type of attack. The varying parameters, in this case, were the affine transform matrix coefficients, considering the form $\left(\begin{array}{cc}\hfill a_1\hfill & \hfill a_2\hfill \\ \hfill a_3\hfill & \hfill a_4\hfill \end{array}\right)$. As in the aforementioned experiments, the SIFT-based version demonstrated the best results among the four versions of our method, followed by SURF-based, RST-based, and FAST-based.

4.3 Signal processing attacks

The third and last attack category considered in our experiments was that of signal processing manipulations. A very usual attack is JPEG compression. Figure 20 presents results for quality factor ranging from 10% to 50%. We can notice that the state-of-the-art techniques are superior, with the technique by Tian et al. having the least variation in robustness. However, the performance of our method in all its versions is quite close to the one by Dong et al. especially for high compression ratios (low quality factor values). Of course, for higher quality factor values, the performance of all versions improves since the distortion is smaller. The SIFT-based version of our method is the best, followed by SURF-based, RST-based, and FAST-based.

A more modern compression technique, specifically H.264 intra-frame compression, has also been considered. As we can see in Figure 21, all methods have similar performance which improves with reduced quantization parameter value, as expected. The two state-of-the-art techniques perform slightly better, whereas the various versions of our method follow closely, with SIFT-based being the best, followed by SURF-based, RST-based, and FAST-based.

Another common distortion is noise addition. For the purpose of our experiments, we added Gaussian white noise of zero mean value and variance ranging from 0.001 to 0.006 to the watermarked images whose pixel values had previously been scaled to the range [0, 1]. As we can see in Figure 22, our technique is not as robust against Gaussian noise as the technique by Dong et al., but is better than the technique by Tian et al. in its SIFT-based and SURF-based versions. The RST-based version follows and, finally, the FAST-based version exhibits the lowest robustness.

Finally, we perform lowpass filtering using a rotationally symmetric Gaussian filter of size 3 × 3 with standard deviation varying from 0.1 to 0.6. As one can see in Figure 23, the technique by Dong et al. is flawless for all values of standard deviation, followed in order of performance by the SIFT-based and the SURF-based versions of our method, with RST-based and FAST-based following. The technique by Tian et al. is only better than the two latter versions for small values of standard deviation. However, the variation in performance is quite small for all methods.

In summary, the proposed technique, as expected due to its design, is more robust than the state-of-the-art techniques in terms of local geometric distortions. It is also better in terms of shearing attacks and downsampling followed by upsampling. It is only inferior compared to the method by Dong et al., yet with significant performance, under rotation, scaling, general affine transform and signal processing attacks, such as JPEG compression, H.264 intra-frame compression, lowpass filtering and noise addition. It is even better, in its SIFT-based and SURF-based versions, than the method by Tian et al. for all these attacks except compression attacks. The most competitive version of our method appears to be the SIFT-based one, followed by the SURF-based, the RST-based, and the FAST-based.