Similarity criterion for image resizing

Abstract

Based on bidirectional similarity measure between patches of image, in this study, we investigate the similarity criterion of image for resizing image. First, our scheme implements image resizing by Seam Carving step by step. For each step, we remove five seams, and then calculate the dissimilarity between the original image and its resized one as well as the relative difference of dissimilarity between neighboring steps. According to the relative differences of dissimilarity of all steps, we can assess the degree of distortion of the resized image and conclude the similarity criterion. On the basis of the similarity criterion, we present an effective image-resizing algorithm by combining Seam Carving, Scaling, and Bidirectional Similarity iteration. Before the salient feature gets damaged markedly, we change the resizing method from Seam Carving to Scaling, and resize the image up to the preferred size. Then, we can update the resized image to eliminate artifacts by iterative computations of Bidirectional Similarity measure. Experiments show that, even though the amount of adjustment is large, our algorithm can preserve the important information, local structures, and global visual effect adequately.

1. Introduction

With the rapid development of the multimedia technology, various display devices have been emerging endlessly, such as computer screen, digital TV, mobile media, MP4, digital camera, and so on. In order that digital image and video should be transmitted and displayed in different display devices, digital image and video need to be changed to different size or aspect ratio for displaying; hence, a variety of algorithms have been proposed to realize such a purpose. A sophisticated algorithm should be able to maintain the salient and interesting regions intact and authentic as much as possible.

Traditional image-resizing methods, such as Scaling and Cropping, have clear drawbacks because of the lack of attentions paid to the content and the feature distribution of images. Scaling will cause obvious distortion if the aspect ratio of the image is drastically changed. Cropping only removes pixels from the image periphery, and hence, it is likely to discard too much important information scattering over the image. Recently, there are growing interests with regard to image-resizing and retargeting algorithms that can protect both the global visual effect and some local structures of the original image [1]; image resizing can be realized by considering important content, unimportant region, image construction, or texture and so on. These methods can be used to resize image fairily, but there are still some problems remaining to be solved. For example, if the amount of adjustment exceeds some bound, then the original image will be warped significantly, and the resulting image will be dissimilar to the original image; even though a certain algorithm gets better resized result by iterative or traverse calculation, it is remarkably time-consuming.

In this article, we investigate the image resizing and the similarity value between the original image and the resized one, and summarize a similarity criterion for the image resizing through a number of experiments. Based on the similarity criterion, we propose an effective image-resizing algorithm which can be used to protect the salient and important information efficiently. First, our algorithm resizes the original image using Seam Carving step by step, and calculates the dissimilarity at each step, as well as the relative difference of dissimilarity between neighboring steps simultaneously. Second, we can estimate whether the deformation degree exceeds a threshold value calculated by a criterion formula; before reaching the critical value, our algorithm stops using Seam Carving and transfers the remnant task to Scaling and Bidirectional Similarity strategy. As a consequence, our algorithm can nicely preserve the authenticity and visibility of an image.

In summary, our main contributions in this article are as follows:

• Building a similarity criterion between an image and its changed version;

• Utilizing the similarity criterion to assess the degree of distortion of the resized images;

• Proposing a content-aware image-resizing algorithm which can preserve the salient information and the global visual effect, based on the similarity criterion.

The rest of this article is organized as follows: Section 2 introduces the background of image resizing and similarity measure. Section 3 shows the image-resizing and similarity measure algorithms used in this article. Section 4 describes similarity criterion between images. In Section 5, we present an effective image-resizing algorithm based on the similarity criterion. In Section 6, we compare the effects of our method with those of the other algorithms and present some discussion.

2. Related works

Until now, a number of algorithms have been proposed to resize image, such as the methods of preserving the visual consistency of the important regions [2–5], removing or duplicating unimportant content [6–11], bidirectional similarity of the patches [12, 13], and so on.

Suh et al. [14] proposed an automatic thumbnail creation based on either a saliency map or the output of a face detector. The large image is then cropped to capture the most salient region. However, if salient regions of the image are sparse, then the effect will not be very perfect. Other image-resizing algorithms [15, 16], based on saliency map, detected and transmitted the most important regions to the small display device, where users can browse the important regions through scrolling the pages, but the important regions could not be seen at the same time. Setlur et al. [17] proposed an automatic, non-photorealistic algorithm for retargeting images to small resolution displays. The retargeting algorithm segments an image into regions, identifies important regions, removes them, resizes the remaining image, and reinserts the important regions. Based on the conformal energy, Zhang et al. [18] employed tools to describe original image and minimized quadratic distortion energies to obtain a resized image.

With dynamic programming, Avidan and Shamir [6] presented a simple image-resizing method called Seam Carving, and then Rubinstein et al. [19] improved it by using graph cuts for image and video retargeting. Seam Carving pays more attention to the unimportant regions, and can retain important content through removing or duplicating the unimportant regions. However, if resizing image is done severely (e.g., the low gradient pixels have been removed), or the interesting objects span the entire image, then the interesting objects and the important regions would suffer from distortion, and, therefore, the local structures and global layout might be destroyed. By utilizing a stream, a path of several pixels width, instead of a seam, Domingues et al. [20] presented an improved algorithm called Stream Carving to induce an increase in the quality of the resized image. Mansfield et al. [21] proposed a scene-carving method, by decomposing the image-retargeting procedure into removing image content with minimal distortion and re-arrangement of known objects within the scene to maximize their visibility. Considering the distortion in both spatial and temporal dimensions, Grundmann et al. [22] presented a discontinuous Seam Carving for video retargeting to process the video frame sequentially and afford great flexibility.

Dong et al. [9] presented a resizing algorithm combining Seam Carving with Scaling. However, their algorithm needs to compute all the possible combinations of resizing amount by both Seam Carving and Scaling, and then chooses the best ratio for resizing image. Based on bidirectional similarity measure of the patches, Simakov et al. [13] proposed an image summary algorithm. Because bidirectional similarity algorithm takes into consideration the completeness and coherence between the resized image and the original image, it can get image summary effectively by iteration computation. Obviously, the above two algorithms are time-consuming. Rubinstein et al. [23] compared a number of state-of-the-art retargeting methods by creating a benchmark of images and conducting a user study.

In general, some artifacts and warping will be introduced to the resized image. If the magnitude of warping is within an acceptable range, then we think that the resized image is similar to the original image subjectively. If it exceeds certain bound, then one will feel that the original image is damaged and the resulting image is not similar to the original one any longer. In fact, a similarity (or dissimilarity) value can be taken into account for image resizing to judge the effect of the resulting image.

Similarity measure between images is an important part of image analysis, and it can be broadly used for image retrieval, visual tracking, and image quality assessment [24–26]. For image resizing and retargeting, Simakov et al. [13] proposed a similarity measure method which quantitatively captures the incompleteness and incoherence of the patches between the original image and the resized images. Rubinstein et al. [8] provided a similarity measure algorithm between images termed Bi-Directional Warping. It measures the similarity between every row (column), and then takes the maximum alignment error as the distance. This algorithm takes the positional concept of pixels into account, and hence, it can capture the overall-similarity between the images. Maalouf and Larabi [27] defined a multi-scale bandelet-based perceptual similarity measure for image retargeting, by measuring the geometric and perceptual similarities between two images to obtain the resulting image that contains as many as of the geometric and perceptual features of the original image. Dong et al. [9] presented a well-defined image distance function, which is formulated as a combination of patch-based bidirectional image Euclidean distance, image-dominant color similarity, and seam energy variation. In this article, we explore and summarize a similarity criterion on the basis of similarity measure and apply it to image resizing.

3. Image resizing and similarity measure

Seam Carving [6] is an efficient method for resizing images in a content-aware mode. As regards the image energy map, it involves dynamic programming to find optimal eight-connected paths of pixels, called seam, across the image from top to bottom or left to right. By greedily removing or duplicating seams passing through less important regions, it shrinks or expands an image in one direction or both to generate a retargeted one. Optimal seams can be found by either of the following methods.

• (1) Compute the energy of every pixel in the image using the gradient energy function, then calculate the accumulative energy of all the eight-connected paths, and find a connected seam which has minimum cumulative energy (see [6]).

• (2) Divide the image into 3 × 3 blocks, compute a uniformity measure of each block by working out the variances of R, G, and B intensity value in each block, then calculate the energy of all the eight-connected paths, and find one particular path having the minimum energy [28]. This way is a modified version of above method and can be implemented quickly.

Images can be resized or retargeted through removing/duplicating vertical and horizontal seams individually or both. As shown in Figure 1, we resize the original image by Seam Carving. If the amount of adjustment is within certain bound, then the resulting image is acceptable and enjoyable, such as shrinking image from the size 201 × 134 to 185 × 134 and 155 × 134. However, by proceeding to 126 × 134, the contents of the image would be destroyed markedly.

Figure 1

Image resizing by Seam Carving.

Such a state as the above motivates us to investigate the similarity between the original and resized image and conclude a similarity criterion for image resizing. Various similarity algorithms have been presented for image analysis and processing [8, 9, 12, 13, 24–26, 29]. In this article, considering the character of image resizing, we choose the Bidirectional Similarity measure to calculate dissimilarity (similarity) value between images.

The Bidirectional Similarity measure method is proposed by Simakov for summarizing image or video [13]. Its essential idea is that a good visual summary should satisfy two properties, namely, it should contain as much information as possible from the original image, and introduce as few artifacts as possible, which were not in the original. Hence, in Bidirectional Similarity measure, two aspects, completeness and coherence, are considered and described in the following.

For the source image S and the target image T, let P and Q denote patches in S and T, respectively, and N _S and N _T are the number of patches in S and T, respectively. For each patch P ⊂ S, we search for the nearest similar patch Q ⊂ T, and measure their distance D (,), and vice versa. The similarity measure can be formalized as follows [13]:

(1)

where the term d_complete(S,T) measures the deviation of T from "completeness" w.r.t.S, the term d_cohere(S,T) measures the deviation of T from "coherence" w.r.t. S. For details, please refer to the literature [13].

4. Similarity criterion

Experiments show that with the increase in the number of the removed seams, Seam Carving would cause the distortion of image contents. We adopt Equation 1 to compute the dissimilarity value between the original image and the resulting image accompanying each step, and further calculate the relative difference of dissimilarity between adjacent steps.

In this article, image resizing is performed under the following rules.

• (1) If the amount of adjustment in horizontal dimension is the same as that in vertical dimension, then at each step, we remove the seams of Δn _row rows and Δn _column columns with minimal energy in both horizontal and vertical dimensions, respectively, and then calculate the dissimilarity value d _i and the relative difference of dissimilarity. Δd _i . We repeat such process up to the preferred size.

• (2) If the amount of adjustment in vertical dimension is greater (less) than that in horizontal dimension, then we first perform the image resizing as mentioned above until completing the horizontal (vertical) adjustment, then resize the remaining adjustment in vertical (horizontal) dimension only. At each step, we remove the seams of Δn _row rows and/or Δn _column columns with minimum energy and compute d _i and Δd _i .

We calculate d _i by Equation 1 and Δd _i by

(2)

where d _i denotes the dissimilarity value between the original image and its resized image at the i^th step, d _i-1 denotes the dissimilarity value between the original image and its resized image at the (i-1) ^th step.

In our proposed algorithm, we empirically set Δn_row (Δn_column) equal to five, namely carving out 5 seams in horizontal or vertical direction at each step for facilitating the estimation of the distortion degree. If Δn_row (Δn_column) is smaller than 5, then the increment of distortion is not obvious in every step, and it is difficult to find the crucial step; conversely, if Δn_row (Δn_column) is much greater than 5, then it maybe omits some scenarios with sharp increment of relative difference.

Based on a large number of experiments, we observed that the change of Δd _i is mild at the beginning of resizing; at certain step, the value of Δd _i increases sharply, and the salient feature or object begins to be destroyed. As shown in Figure 2, we resize image in width direction from 201 × 134 to 126 × 134 according to the rule mentioned above. (a) is the original image, (b) is the graph of the dissimilarity value, (c) is the graph of the relative difference of dissimilarity. We can see that the relative difference of dissimilarity alters sharply when we remove 40 seams; the value changes from 46 to 192, and the salient object begins to get distorted. Similarly, in Figure 3, we resize image in high direction from 136 × 134 to 136 × 84. When we remove 25 seams, the relative difference of dissimilarity jumps from 8 to 152.

Figure 2

Dissimilarity and its relative difference of resizing image in width.

Figure 3

Dissimilarity and its relative difference of resizing image in height.

Based on the above observation and consideration, we think the change of the relative difference of dissimilarity can be adopted to judge the degree of image warping. If the relative difference of dissimilarity is within a certain threshold, then the important contents of image can be preserved; else if it exceeds the threshold, the important contents begin to be distorted. Considering the number of seams of each step, the change of dissimilarity, and various images synthetically, we build the equation of threshold as

(3)

where ΔE is the total dissimilarity between the original image and the resized image of ultimate size by Seam Carving, L indicates the total number of removed seams, α is the number of removed seams of every step, and β is a coefficient.

For different images, there exist different contents and distribution, and the magnitude of the relative difference of dissimilarity corresponding to the step with increment sharply is different. Hence, Equation 3 contains the average value of dissimilarity (ΔE/L) and an adjustive coefficient β In this article, we set β = 1.2.

On the basis of the Equation 3 and analysis above, we construct the similarity criterion for image resizing by comparing the relative difference of dissimilarity Δd _i with the threshold θ. If Δd _i is less than θ, we continue resizing image by Seam Carving; otherwise we should stop Seam Carving and switch to other resizing algorithms for remaining adjustment.

5. Our algorithm

Seam Carving resizes an image by carving out or duplicating unimportant regions gracefully, and hence, it has good performance for preserving important information of the original image as much as possible. While the unimportant pixels are almost removed, the global visual effect and some local structures of image will be damaged severely if we continue resizing the image this way.

Motivated by the similarity criterion mentioned in Section 4, we propose an effective image resizing algorithm, which combines Seam Carving with Bidirectional Similarity measure, to obtain better visual result. The details of the steps of our algorithm are as follows:

• (1) Resize the original image to a preferred size directly by Seam Carving and calculate the dissimilarity value between the resized image and the original image. Based on the dissimilarity value and the total number of seams removed, the threshold θ is computed.

• (2) Resize the original image by Seam Carving step by step.

• (3) Calculate dissimilarity value d _i between the original image and its resized one as well as relative difference of dissimilarity Δd _i associated with step i(i = 1,2,3,...), then judge whether Δd _i exceeds the threshold θ.

• (4) If Δd _i is less than θ, then go to step 2 and continue resizing; otherwise, go to step 5.

• (5) If Δd _i exceeds the threshold θ at the i^th step, then we will adopt following approach to complete the remnant tasks from the (i-1) ^th step:

We scale the (i-1) ^th step image to ultimate size directly, denoting the (i-1) ^th step image with S₁ and the final image with T₁, and then use the Bidirectional Similarity iteration to update the image T₁. We will obtain the target image until we get minimal dissimilarity value.

Iterative refinement is performed as follows:

Considering the coherence, traverse each pixel in image T_1. For a pixel q in T₁, let Q₁,Q₂,...,Q _m denote all the patches containing the pixel q. Let P₁,P₂,...,P _m denote the most similar patches in S₁ corresponding to Q₁,Q₂,...,Q _m , and p₁,p₂,...,p _m be the corresponding pixels in P₁,P₂,...,P _m to the pixel q within Q₁,Q₂,...,Q _m in geometric position. In this article, the size of patch is 7 × 7, and so m is 49.

Considering the completeness, traverse all patches in the image S₁. For each patch, search the most similar patch in T₁, and record all the corresponding pixels between the two pitches. Hence, for a pixel q in T₁, it can get the votes by corresponding pixels within the patches in S₁. The subscript symbol n is decided by the number of similar patches.

We update R, G, and B intensities of the pixel q with the following equation [13]:

(4)

where and denote the number of patches in S₁ and T₁, m and n denote the number of pixels p _i and in S₁ corresponding to the pixel q in T₁.

6. Experiment and discussion

From Figures 4, 5, 6 and 7, we compare resizing results by various methods. Moreover, we assess the similarity value between the resized image and the original image using Equation 1.

Figure 4

Result comparison by several methods.

Figure 5

Image resizing by various methods.

Figure 6

Resizing image in vertical direction.

Figure 7

Resizing image in both directions.

We resize the image from 201 × 134 to 126 × 134 as shown in Figure 4. The balloons are the important objects. Figure 4b is the resizing result only by Seam Carving. The balloons in the rectangles are damaged, and there are some aliases around the borders of the balloons. In Figure 4c, the image contents are shrunken uniformly. Using our method, the contents within image are preserved better and the verge of balloon is smooth (see Figure 4d). For dissimilarity value, relative to the original image, the dissimilarity value is 2,248 for Figure 4b; 3,188 for Figure 4c; and 1,292 for Figure 4d. Our result is the most similar to the original image.

Another example as shown in Figure 5 resizes the original image from 181 × 119 to 101 × 119 with several methods. We can observe that there are some artifacts appearing on the verge of petal within the image Figure 5b. Resizing the entire image in the same ratio using Scaling would change the shape of the flower, and the global visual effect is damaged (see Figure 5c). However, our method (see Figure 5d) can preserve the contents of image from distortion; yet, the edge of petal is smooth. While assessing the similarity, our result is the most similar to the original image due to the use of the iterative update, and the dissimilarity value is the smallest.

Analogously, we resize the image (see Figure 6) in vertical direction, from 136 × 134 to 136 × 84. The resulting image by Seam Carving is shown in Figure 6b, and the roof of the house is damaged. However, the result by our method, owing to adopting iterative computation, could mend a few interesting contents to some extent (see Figure 6d).

In Figure 7, we show an example of resizing image from 169 × 128 to 109 × 80. In this case, the size of image will be changed in both width and height. We can see that the fur of rabbits is damaged (see Figure 7b), and the entire image is shrunk uniformly (see Figure 7c). In our approach, the important information can be preserved and good visual effect obtained.

In our algorithm, the effect of resizing image is correlated with coefficients Δn_row, Δn_column and β. β is an adjustive coefficient, deciding the degree of image distortion by Seam Carving. If β is small, then distortion inspection will be rigorous, the number of carving steps may be small, and the great mass of resizing work will be realized by Scaling and Bidirectional Similarity iteration. Whereas the great mass of resizing work may be carried out by Seam Carving (see Figure 8).

Figure 8

Resizing results of our method with different coefficients β.

Because Bidirectional Similarity involves iterative computation which is time-consuming, our algorithm is slower than Seam Carving. However, pixels' updating computation is needed only in remnant summarizing task, and so it is faster than using Bidirectional Similarity alone. In order to improve the speed, multiple CPUs/GPU parallel calculation can be introduced for significant speedup.

In this article, we focused on the image reduction and image summary. For image enlarging, Seam Carving is a sophisticated method and can be implemented gracefully.

7. Conclusions

In this article, we investigate the image resizing and sum up the similarity criterion, which could be employed to judge the degree of deformation for the resized images relative to the original image. Based on the similarity criterion, we proposed an effective image resizing algorithm combining the Seam Carving with Bidirectional Similarity measure. Even though the amount of adjustment is large, the algorithm can still avoid the disorder and distortion of image contents and preserve both the important regions and the global visual effect of the original image.

For the future study, we will further research an adaptive resizing algorithm, to choose automatically the best resizing algorithm from several candidate methods in every step to obtain better results of resizing.