Open Access

Missing Texture Reconstruction Method Based on Perceptually Optimized Algorithm

EURASIP Journal on Advances in Signal Processing20102010:208976

https://doi.org/10.1155/2010/208976

Received: 23 August 2010

Accepted: 26 October 2010

Published: 26 October 2010

Abstract

This paper presents a simple and effective missing texture reconstruction method based on a perceptually optimized algorithm. The proposed method utilizes the structural similarity (SSIM) index as a new visual quality measure for reconstructing missing areas. Furthermore, in order to adaptively reconstruct target images containing several kinds of textures, the following two novel approaches are introduced into the SSIM-based reconstruction algorithm. First, the proposed method performs SSIM-based selection of the optimal known local textures to adaptively obtain subspaces for reconstructing missing textures. Secondly, missing texture reconstruction that maximizes the SSIM index in the known neighboring areas is performed. In this approach, the nonconvex maximization problem is reformulated as a quasi convex problem, and adaptive reconstruction of the missing textures based on the perceptually optimized algorithm becomes feasible. Experimental results show impressive improvements of the proposed method over previously reported reconstruction methods.

1. Introduction

Restoration of missing areas in digital images has been intensively studied since it can be applied to a number of fundamental applications such as restoration of corrupted old films, removal of unnecessary objects, and error concealment. Therefore, many methods have been proposed in order to realize these applications. Generally, they are broadly classified into two categories, structural and textural reconstruction approaches, and many papers on these approaches have been published. Attractive methods that perform simultaneous reconstruction of missing structures and textures in images have also been proposed [1, 2]. Most algorithms reported in the literature are based on structural inpainting techniques for accurate reconstruction of missing edges [35]. These techniques are effective for pure structure images. However, since general images also contain many textures, different methods work better in these areas. Thus, several methods have been proposed for accurate reconstruction of missing textures [612]. The remainder of this paper focuses on the texture reconstruction approach with discussion of its details.

Traditionally, missing texture reconstruction is realized as one of the applications of texture synthesis. Efros et al. firstly proposed a pioneered method for the texture synthesis [6, 7]. Their approach models textures by using the MRF (Markov random field) model and enables missing texture reconstruction by copying pixels of a target image itself, that is, nonparametric sampling in synthesis. Furthermore, Wei and Levoy proposed a fast algorithm for the searching step in the texture synthesis by utilizing multiresolution concepts [8]. Then many methods which perform the exemplar-based inpainting are mainly inspired by the nonparametric sampling in [6]. Drori et al. proposed a fragment-based algorithm for image completion which could preserve structures and textures [9]. Furthermore, the exemplar-based image inpainting method proposed by Criminisi et al. is a representative one based on the texture synthesis [10, 11]. This method adopts a patch-based greedy sampling scheme similar to the fragment-based completion, but it is simpler and faster. A good review of the exemplar-based inpainting methods based on [6] is shown in [12].

In the field of texture reconstruction, not only the methods based on the texture synthesis but also many methods, which estimate missing intensities by utilizing statistical features of known textures within a target image as training patterns, have been proposed. Generally, since the restoration of missing areas is an ill-posed problem, it is difficult to directly estimate the missing intensities. Thus, these methods perform approximation of textures within the target image in lower-dimensional subspaces and enable derivation of the inverse projection for the corruption. Amano and Sato proposed an effective PCA-based method for reconstructing missing textures using back projection for lost pixels and realized accurate reconstruction performance [13]. Furthermore, kernel methods have recently been developed and their achievements have been reported in a number of papers [1416]. Subspaces constructed on the basis of kernel methods are also suitable for approximating nonlinear texture features in target images. Several missing texture reconstruction methods that utilize projection schemes onto nonlinear subspaces obtained by kernel PCA and CCA have been proposed [17, 18]. Furthermore, image reconstruction based on sparse representation approaches [1921] have been intensively studied. By utilizing sparse representation, optimal signal atoms can be adaptively selected from a dictionary for representing target signals. This means that these methods can adaptively provide optimal subspaces for restoring missing areas. Several missing area reconstruction methods based on sparse representation have been proposed [2123].

It should be noted that in conventional methods, reconstruction is mostly performed by minimizing errors of intensities, that is, the mean squared error (MSE), which is the most popular metric. However, it has been reported that MSE optimal algorithms do not necessarily produce images of high visual quality [24]. Thus, it may not be appropriate to utilize the MSE as a quality measure for reconstruction. Recent advances in full-reference image quality assessment (IQA) have resulted in the emergence of several powerful perceptual distortion measures that outperform the MSE and its variants. Criteria such as PQS [25], NQM [26], IFC [27], and VIF [28] are well known as perceptual distortion measures, and their performances have been evaluated in detail [29]. The structural similarity (SSIM) index [30] is utilized as a representative quality measure in many fields of image processing. Since its formulation is simple and easy to be analyzed, the SSIM index can be applied to not only image quality assessment but also design of linear equalizers [31]. Therefore, by using this quality measure, accurate reconstruction of missing textures can be expected.

In this paper, we present a simple and effective missing texture reconstruction method based on a perceptually optimized algorithm. The proposed method utilizes the SSIM index as a criterion for reconstructing missing areas in the target image. Specifically, we introduce the following two novel approaches into the SSIM-based algorithm and realize adaptive reconstruction of missing textures.
  1. (1)

    SSIM-based selection of the optimal known local textures for reconstructing target textures including missing areas.

     
  2. (2)

    Reconstruction of the target textures maximizing the SSIM index in the known neighboring areas.

     

The first approach provides optimal subspaces for the following SSIM-based reconstruction approach by using an algorithm similar to several matching pursuit algorithms [32, 33]. Furthermore, in the second approach, we introduce the computation scheme in [31] into the SSIM-based reconstruction algorithm, and its nonconvex maximization problem is reformulated as a quasi convex problem. Then the optimal solution based on the SSIM index can be computed, and accurate reconstruction of the missing textures is expected.

This paper is organized as follows. First, in Section 2, we briefly explain the SSIM index used as the quality measure in the proposed method. Next, the missing texture reconstruction method based on the perceptually optimized algorithm is proposed in Section 3. Experimental results that verify the performance of the proposed method are shown in Section 4. Finally, conclusions are given in Section 5.

2. SSIM Index

The SSIM index represents the similarity between two signal vectors and ( ), and its specific definition is as follows:
(1)
where the terms and , respectively, compare the mean and variance of the two signal vectors. Furthermore, measures their structural correlation. These three terms, , , and , are obtained as
(2)
In the above equations, and are the means of and , and are the variances of and , and is the cross-covariance between and . The constants , and are necessary for avoiding instability when the denominators are very close to zero. The parameters , and determine the relative importance of the three components in (1). As shown in [30], those parameters are set as and , and formulation of the SSIM is simplified as follows:
(3)

As shown in (1)–(3), the SSIM index is consistent with luminance and contrast masking and the correlation.

In [30, 34], the effectiveness of the SSIM index as a quality measure and its superiority to the MSE and its variants are presented in detail. Generally, the MSE cannot reflect perceptual distortions, and its value becomes higher for images altered with some distortions such as mean luminance shift, contrast stretch, spatial shift, spatial scaling, and rotation, yet negligible loss of subjective image quality. Furthermore, blurring severely deteriorates the image quality, but its MSE becomes lower than those of the above alternation. On the other hand, the SSIM index is defined by separately calculating the three similarities in terms of the luminance, variance, and structure, which are derived based on the HVS (human visual system) not accounted for by the MSE. Therefore, it becomes a better quality measure providing a solution to the above problem, and this is also confirmed in [34]. Then we can expect that the use of this similarity for the reconstruction of missing areas will provide successful results. The specific effectiveness of the SSIM index for the reconstruction is discussed in Section 4.

3. Adaptive Missing Texture Reconstruction Based on SSIM Index

In this section, we present an adaptive SSIM-based missing texture reconstruction method. In the proposed method, a patch (  pixels) including missing areas is clipped from the target image, and its missing textures are estimated from the other known areas. An overview of the proposed method is shown in Figure 1. For the following explanations, we denote two areas whose intensities are unknown and known within the target patch as and , respectively. We also define vectors whose elements are intensities within and as and , respectively, where is the number of pixels within the area .
Figure 1

Outline of the proposed method including a perceptually optimized algorithm.

In the target image, there are several kinds of textures, that is, there are many known patches whose textures are quite different from that of the target patch . Such patches should not affect the reconstruction of the target patch . In order to reconstruct the missing textures within the target patch from only the same kinds of textures, we have to select those textures from the known areas. Therefore, the proposed method first performs selection of the optimal known patches utilized for reconstruction of the target patch based on the SSIM index. Furthermore, by using the selected patches, we derive the representation model optimized for the target patch in terms of the SSIM index to reconstruct the missing area . Then the proposed method can adaptively reconstruct the missing textures from only the same kinds of known textures based on the perceptually optimized scheme.

In this section, we first show the SSIM-based algorithm for selecting the optimal known patches in Section 3.1. The reconstruction algorithm of the missing textures based on the SSIM index is shown in Section 3.2.

3.1. SSIM-Based Optimal Texture Selection Algorithm

In this subsection, we present the SSIM-based optimal texture selection algorithm. First, we clip known patches ( ) whose size is pixels from the target image in the same interval. For the following explanation, two vectors that correspond to and of each patch are denoted as ( ) and ( ), respectively. From the clipped patch, we select patches that are optimal for reconstruction of the target patch . The order of the value is explained in Section 4. In the reconstruction algorithm shown in the following subsection, the target patch is represented by a linear combination of the selected known patches in such a way that the SSIM index in the known area becomes maximum. Therefore, we should select known patches that provide the optimal linear combination. Note that the selection of such optimal known patches is an NP-hard problem. Thus, we adopt the simplest algorithm that selects the optimal known patches one by one, and it is similar to several matching pursuit algorithms [32, 33]. In the rest of this subsection, the details of the th ( ) optimal patch selection are shown.

In the th iteration, we first define the following vector:
(4)
where is an matrix which contains vectors previously selected from ( ) in iterations. Furthermore,
(5)

is a coefficient vector for obtaining . The proposed method estimates the optimal vector of ( ) which provides the optimal representation performance based on the SSIM index. Then the best matched patch , whose vector approximating has a higher value of the SSIM index than those of other patches, is selected.

In order to calculate for each patch , we have to estimate the optimal coefficient vector of in (4) that satisfies
(6)
This means we have to solve the following equation:
(7)
where is defined as follows:
(8)
In the above equation, and are the means of and , and are the variances of and , and is the cross covariance between and . Furthermore, since is provided in (4), (8) is rewritten as follows:
(9)
where is an vector, and
(10)

The proposed method calculates the optimal vector in (7) by simply applying the steepest ascend algorithm to in (9). Note that we can calculate the optimal vector more accurately by using the algorithm shown in the following subsection. However, in order to reduce the computation time of the proposed method, we adopt the steepest ascend algorithm in this subsection. It is well known that the steepest ascend algorithm cannot necessarily provide the globally optimal solutions in (7), but this algorithm can save the computation time compared to the algorithm shown in the following subsection. The details are shown later. From the above reason, we utilize this scheme in the proposed method.

By iterating the above procedures times, we can select the optimal known patches based on the SSIM index and denote them as ( ). Algorithm 1 shows the specific procedures of this selection algorithm. Then by utilizing the obtained known patches, the proposed method can adaptively provide the optimal subspace for the target patch , and accurate reconstruction based on the SSIM index is also expected in the following subsection. For the following explanation, we denote two vectors obtained from in the same way as and as and , respectively.

Algorithm 1: Specific procedures to select optimal known patches ( ) for the target patch based on the SSIM index.
  1. (i)

    Initialization is performed as follows: ,

    , and . Furthermore, , and are, respectively, set to the empty matrix,and vector.

     
  2. (ii)

    For each patch included in the set , the optimal value of the SSIM index maximizing (8) is calculated.

     
  3. (iii)

    The best matched patch, whose maximized SSIM index is larger than those of the other patches in ,

     
is selected as . Furthermore, this patch is removed from and added to .
  1. (iv)

    , and the matrix is constructed from the vectors of the patches belonging to .

     
  2. (v)

    The procedures (ii)–(iv) are repeated until . If , outputs optimal known patches.

     

3.2. Texture Reconstruction Algorithm

In this subsection, we present the reconstruction algorithm of the missing area in the target patch based on the SSIM index. First, we approximate the known vector of the target patch by utilizing of the patches selected in the previous subsection as follows:
(11)
where is an matrix whose columns are ( ), and is a coefficient vector for representing . The proposed method estimates the optimal vector as follows:
(12)
In the above equation, is defined as
(13)
in the same way as (9), where
(14)

is an vector whose elements are the means of ( ) in .

It should be noted that the criterion defined in (13) is a nonconvex function of , and it is difficult to obtain its global optimal solution. Therefore, we introduce the calculation scheme utilized in [31] into the estimation of the optimal vector . Specifically, the above nonconvex problem is transformed into a quasiconvex formulation. First, we note that the first term in (13) is a function only of = . Thus, (13) can be rewritten as follows:
(15)
where
(16)
Then we can simplify the optimization problem by constraining . Specifically, the optimization problem can be simplified to find
(17)
Therefore, the overall problem is to find the highest SSIM index by searching over a range of . Furthermore, since the optimization problem in (17) is still nonconvex, it is converted into a quasiconvex optimization problem as follows:
(18)
The first equivalence relationship holds since minimizing is the same as finding the least upper bound of (17). The second equivalence relationship holds since the denominator in (17) is strictly positive, allowing us to multiply through and rearrange terms. Then becomes a true upper bound if the problem
(19)
has a non-negative optimal value, and the optimal vector in (17) can be obtained. Specifically, the proposed method utilizes the following Lagrange multiplier approach:
(20)
By solving for and , we can obtain
(21)

where we denote them as and since they depend on . Furthermore, in the above two equations, is a Moore-Penrose pseudoinverse matrix of . Then the proposed method estimates the optimal value of by using a standard bisection procedure, and the optimal vectors are calculated for several values of ( ) to select maximizing (13), where is a step size and determines the search range of . Algorithm 2 shows the details on the estimation of in the proposed method.

Algorithm 2: Specific procedures to search the optimal in the proposed method.
  1. (i)

    An initial value of (say ) is determined between. zero to one. Furthermore,

    and , where and , respectively, represent the upper

    limit and the lower limit of . In this paper, we set

     
  2. (ii)

    The optimization problem in (19) is solved by using .

     
  3. (iii)

    Two criteria and are calculated as

     

          ,

                  .
  1. (iv)

    According to the obtained criteria and , the following steps are operated:

     

  (a) If and , the final optimal solution of is output, where .

  (b) If but , and .

  (c) Otherwise, and .
  1. (v)

    The procedures (ii)–(iv) are iterated.

     

Note that the algorithm for calculating the optimal linear combination in this subsection provides better solutions than those in the previous subsection. However, this algorithm needs to perform iterations to determine the value of . Furthermore, it also needs the iteration to search the optimal value of as shown in Algorithm 2. Then since it is confirmed that the algorithm shown in this subsection takes more computation time compared to the algorithm shown in the previous subsection, we perform the selection of the optimal known patches ( ) as shown in the previous subsection.

By utilizing the coefficient vector , the estimation result of the unknown vector whose elements are the intensities within is calculated as follows:
(22)

where is a matrix whose columns are ( ). Finally, from the obtained result , the proposed method outputs the estimated intensities in the missing area .

As described above, we can reconstruct the missing texture in the target patch. The proposed method clips patches ( pixels) at the same interval from the upper-left of the target image in a rasterscanning order. If the clipped patches contain missing areas, we regard them as the target patches and reconstruct them by using the above approach. Note that each restored pixel has multiple estimation results if the clipping interval is smaller than the size of the patches. In this case, the proposed method regards the result maximizing (13) as the final one. The proposed method does not utilize the already obtained results for reconstructing other missing areas. Therefore, the performance of the proposed method does not depend on the order of the reconstruction, that is, the positions of the patches including missing areas do not influence the results.

4. Experimental Results

The performance of the proposed method is shown in this section. Figure 2(a) is a test texture image ( pixels, 24-bit color levels), and from the flag image shown in Figure 2(b), a corrupted image, which includes text regions "Grand Canyon" as missing areas, is obtained as shown in Figure 2(c)(Note that positions of the missing areas are previously provided in this experiment.) Figure 2(d) shows the results of reconstruction by the proposed method. In this experiment, we set the parameters of the proposed method as follows: , , , , , , where is the maximum value of intensities, and the clipping interval of patches is 10 and 8. The size of patches influences the reconstruction results. If the size of patches becomes smaller, the representation performance of their textures becomes better. However, these patches including missing areas must contain known intensities to select the optimal known patches in Section 3.1 and estimate the vector in Section 3.2. Thus, the size of patches should be determined in such a way that they necessarily contain several known intensities. In this experiment, we determine and to satisfy the above condition. Furthermore, the clipping interval is set to about quarter size of and , that is, the horizontal and vertical intervals are, respectively, set to 10 and 8. Next, and mean that the search range of in the proposed method is from to . From preliminary experiments, tends not to become smaller than or larger than . Thus, we set the range of as shown above. The parameters and are determined in the same way as [30].
Figure 2

(a) Original image (480 × 359 pixels, 24-bit color levels). (b) Flag image whose white regions correspond to missing areas. (c) Corrupted image including text regions (8.9% loss), (d) Reconstructed image by the proposed method. (e) Reconstructed image by the method based on the random selection. (f) Reconstructed image by the method which utilizes the MSE instead of the SSIM index. (g) Reconstructed image by the conventional method [11]. (h) Reconstructed image by the conventional method [13]. (i) Reconstructed image by the conventional method [21].

For comparison, Figures 2(e)2(i), respectively, show the results obtained by the method which selects known patches randomly but reconstructs missing areas in the same way as Section 3.2, the method which utilizes the MSE instead of the SSIM index, the exemplar-based texture reconstruction method [11], the PCA-based texture reconstruction method [13], and the method based on sparse representation [21]. In order to verify the effectiveness of the selection algorithm shown in Section 3.1, we show the results in Figure 2(e). The method in [11] is one of the most influential works in the field of the exemplar-based texture reconstruction, and we utilize this method for the comparison of the proposed method as shown in Figure 2(g). Furthermore, since subspaces optimized on the basis of the MSE criterion are utilized for the reconstruction of missing textures, the other conventional methods shown in Figures 2(f), 2(h), and 2(i) are suitable for verifying the performance of the proposed method. Particularly, the methods in [13, 21] are, respectively, representative works using PCA and sparse representation.

Note that the dimensions of the subspaces utilized in the proposed method and the conventional methods are set to the same value 40 ( ). In the proposed method, we have to set to a smaller value than the number of known pixels within the target patch . Furthermore, this should be satisfied for all target patches including missing areas within the target image. Otherwise, the calculation of the optimal vector in (11) and (12) generally becomes an underdetermined problem. This means we have to set to a sufficiently small value in order to avoid the problem in (12), being an underdetermined problem. Generally, if becomes larger than the number of the known pixels in , some constraints must be introduced as regularization terms for avoiding the system instability. Furthermore, if there is no limitation of the cost function utilized for the reconstruction, some constraints must be also adopted. Several existing studies for inpainting using a linear combination of patches adopt some restrictions such as the sum of the linear coefficients being one [35], and so forth. On the other hand, our method sets the value of to a much smaller value than the number of known pixels within the target patch , and the maximum range of the SSIM index is limited to one. Thus, since the system is not instable in (12), we think that our method does not have to utilize other restrictions. Furthermore, it seems that the value of should be set to about one-tenth of the dimension of . This means we assume the percentages of the known pixels within the target patches are larger than 10%. In the experiments, we set to 40, that is, a much smaller value than the above criterion to clearly show the difference of the reconstruction performance between the proposed method and the conventional methods.

For better subjective evaluation, the enlarged portions around the upper left of the images are shown in Figure 3. It can be seen that the use of the proposed method has achieved noticeable improvements compared to the conventional methods. From the results in Figures 3(d) and 3(e), the effectiveness of the algorithm for selecting the optimal known patches in Section 3.1 can be confirmed. Different experimental results are shown in Figures 4, 5, 6, and 7. Compared to the results obtained by the conventional methods, it can be seen that various kinds of textures are accurately restored by using the proposed method. Therefore, high performance of the proposed method was verified by the experiments.
Figure 3

(a) Zoomed portion of Figure 2(a). (b) Zoomed portion of Figure 2(b), (c) Zoomed portion of Figure 2(c), (d) Zoomed portion of Figure 2(d), (e) Zoomed portion of Figure 2(e), (f) Zoomed portion of Figure 2(f), (g) Zoomed portion of Figure 2(g), (h) Zoomed portion of Figure 2(h), (i) Zoomed portion of Figure 2(i).

Figure 4

(a) Original image (480 × 360 pixels, 24-bit color levels). (b) Flag image whose white regions correspond to missing areas. (c) Corrupted image including text regions (10.7% loss). (d) Reconstructed image by the proposed method. (e) Reconstructed image by the method based on the random selection. (f) Reconstructed image by the method which utilizes the MSE instead of the SSIM index. (g) Reconstructed image by the conventional method [11]. (h) Reconstructed image by the conventional method [13]. (i) Reconstructed image by the conventional method [21].

Figure 5

(a) Zoomed portion of Figure 4(a). (b) Zoomed portion of Figure 4(b). (c) Zoomed portion of Figure 4(c). (d) Zoomed portion of Figure 4(d). (e) Zoomed portion of Figure 4(e). (f) Zoomed portion of Figure 4(f). (g) Zoomed portion of Figure 4(g). (h) Zoomed portion of Figure 4(h). (i) Zoomed portion of Figure 4(i).

Figure 6

(a) Original image (480 × 360 pixels, 24-bit color levels). (b) Flag image whose white regions correspond to missing areas. (c) Corrupted image including text regions (11.9% loss). (d) Reconstructed image by the proposed method. (e) Reconstructed image by the method based on the random selection. (f) Reconstructed image by the method which utilizes the MSE instead of the SSIM index. (g) Reconstructed image by the conventional method [11]. (h) Reconstructed image by the conventional method [13]. (i) Reconstructed image by the conventional method [21].

Figure 7

(a) Zoomed portion of Figure 6(a). (b) Zoomed portion of Figure 6(b). (c) Zoomed portion of Figure 6(c). (d) Zoomed portion of Figure 6(d). (e) Zoomed portion of Figure 6(e). (f) Zoomed portion of Figure 6(f). (g) Zoomed portion of Figure 6(g). (h) Zoomed portion of Figure 6(h). (i) Zoomed portion of Figure 6(i).

In order to confirm the superiority of the SSIM index for evaluating visual qualities, we show the MSE and the SSIM index of the reconstruction results in Tables 1 and 2. It can be seen that our method has achieved improvement over the conventional methods in the SSIM index. Although the MSEs of the proposed method tend to become worse than those of the conventional methods, we can see that the MSE results cannot correctly reflect the visual quality in the subjective evaluation. On the other hand, the SSIM index can represent the visual quality more accurately. Therefore, we can conclude that the use of the SSIM index as a visual quality measure is appropriate for texture reconstruction.
Table 1

Performance comparison (MSE) of the proposed method and the conventional methods.

Test image

Random Selection

MSE-based method

Reference [11]

Reference [13]

Reference [21]

Proposed  method

Figure 2

155.27

95.97

208.94

174.94

138.11

153.70

Figure 4

170.91

116.11

224.23

167.96

169.26

173.19

Figure 6

123.44

105.96

170.73

110.65

121.73

127.28

Table 2

Performance comparison (SSIM) of the proposed method and the conventional methods.

Test image

Random Selection

MSE-based method

Reference [11]

Reference [13]

Reference [21]

Proposed  method

Figure 2

0.9250

0.9421

0.9273

0.9257

0.9324

0.9435

Figure 4

0.9069

0.9253

0.9263

0.9134

0.9202

0.9403

Figure 6

0.9046

0.9154

0.9202

0.9137

0.9119

0.9361

In the conventional methods, the subspace estimation and texture reconstruction schemes are based on the MSE criterion. However, the MSE optimal algorithms do not necessarily produce images of high visual quality, and the reconstruction results may be degraded. Specifically, it is well known that most images contain more low-frequency components than high-frequency components. Thus, models using the subspaces based on the MSE can only represent such low-frequency components, and it becomes difficult to reconstruct the missing high-frequency components of the missing areas. This means the reconstruction results tend to be blurred. Then since the representation performance, that is, the reconstruction performance of each patch, becomes worse, the color discontinuities at the border of the missing areas and that of patches also occurs. On the other hand, the proposed method adopts the SSIM index for obtaining subspaces and reconstructing missing textures. The basic formulation of the SSIM index is obtained from the three terms , , and as shown in (1). These terms respectively represent the mean similarity, the variance similarity, and the structural correlation. The first term and the third term compare the vector lengths and angles, and they separately provide those similarities. Note that the second term compares the contrast of the two vectors, that is, it enables the comparison of the texture roughness. Therefore, this can be regarded as the term comparing how much high-frequency components the target textures contain. This is also pointed out in [34], and they confirmed that the SSIM index of blurred images which were perceptually degraded severely became lower. Then it seems that the proposed method can avoid the oversmoothness of the reconstruction results by utilizing the SSIM index including the above useful term. Since the SSIM index outperforms the MSE as a perceptual distortion measure, our method can provide the solution to the conventional problems and realize more accurate reconstruction.

Finally, we show the computation time of the proposed method. The experiments shown above were performed on a personal computer using Intel(R) Core(TM) i7 950 CPU 3.06 GHz with 8.0 Gbytes RAM. The proposed method was implemented by using Matlab. The average computation times to perform the algorithms shown in Sections 3.1 and 3.2 for each target patch are, respectively, sec and  sec. Thus, from these results, we can see that the reduction of the computational cost in the optimal patch selection algorithm of the proposed method is necessary for practical use. This issue will be addressed in a future work.

5. Conclusions

In this paper, we have presented an adaptive method for reconstructing missing textures based on the SSIM index. The proposed method adaptively obtains subspaces utilized for the reconstruction of missing textures by selecting the optimal known local textures based on the SSIM index. Furthermore, missing texture reconstruction maximizing the SSIM index can be realized by reformulating the nonconvex problem as a quasi convex problem. Then the proposed method enables adaptive texture reconstruction based on the perceptually optimized algorithm. Consequently, impressive improvement of the proposed method over previously reported methods was confirmed.

In the experiments, we manually determine the parameters of the proposed method. It is desirable that these parameters be adaptively determined from the target image. Thus, we need to complement this determination algorithm. Extension of the framework to texture reconstruction of other types of missing imagery data is also needed for various applications. Finally, we would like to study these ideas for interpolation in video data. These topics will be the subjects of subsequent reports.

Declarations

Acknowledgment

This work was partly supported by Grant-in-Aid for Scientific Research (B) 21300030, Japan Society for the Promotion of Science (JSPS).

Authors’ Affiliations

(1)
Graduate School of Information Science and Technology, Hokkaido University

References

  1. Bertalmio M, Vese L, Sapiro G, Osher S: Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing 2003, 12(8):882-889. 10.1109/TIP.2003.815261View ArticleGoogle Scholar
  2. Rane SD, Sapiro G, Bertalmio M: Structure and texture filling-In of missing image blocks in wireless transmission and compression applications. IEEE Transactions on Image Processing 2003, 12(3):296-303. 10.1109/TIP.2002.804264MathSciNetView ArticleGoogle Scholar
  3. Ballester C, Bertalmio M, Caselles V, Sapiro G, Verdera J: Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing 2001, 10(8):1200-1211. 10.1109/83.935036MathSciNetView ArticleMATHGoogle Scholar
  4. Chan TF, Shen J: Nontexture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation 2001, 12(4):436-449. 10.1006/jvci.2001.0487View ArticleGoogle Scholar
  5. Rareş A, Reinders MJT, Biemond J: Edge-based image restoration. IEEE Transactions on Image Processing 2005, 14(10):1454-1468.View ArticleMATHGoogle Scholar
  6. Efros AA, Leung TK: Texture synthesis by non-parametric sampling. Proceedings of the 1999 7th IEEE International Conference on Computer Vision (ICCV'99), September 1999 1033-1038.Google Scholar
  7. Efros AA, Freeman WT: Image quilting for texture synthesis and transfer. Proceedings of the Computer Graphics Annual Conference (SIGGRAPH '01), August 2001 341-346.Google Scholar
  8. Wei L-Y, Levoy M: Fast texture synthesis using tree-structured vector quantization. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics (SIGGRAPH '00), 2000. Edited by: Akeley K. ACM Press /ACM SIGGRAPH /Addison Wesley Longman; 479-488.Google Scholar
  9. Drori I, Cohen-Or D, Yeshurun H: Fragment-based image completion. In Proceedings of the ACM SIGGRAPH 2003 Papers (SIGGRAPH '03), 2003, New York, NY, USA. ACM Press; 303-312.Google Scholar
  10. Criminisi A, Pérez P, Toyama K: Object removal by exemplar-based inpainting. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2003 721-728.Google Scholar
  11. Criminisi A, Pérez P, Toyama K: Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 2004, 13(9):1200-1212. 10.1109/TIP.2004.833105View ArticleGoogle Scholar
  12. Fidaner IB: A survey on variational image inpainting texture synthesis and image completion. http://www.scribd.com/doc/3012627/A-Survey-on-Variational-Image-Inpainting-Texture-Synthesis-and-Image-Completion
  13. Amano T, Sato Y: Image interpolation using BPLP method on the eigenspace. Systems and Computers in Japan 2007, 38(1):87-96. 10.1002/scj.10319View ArticleGoogle Scholar
  14. Schölkopf B, Mika S, Burges CJC, Knirsch P, Müller K-R, Rätsch G, Smola AJ: Input space versus feature space in kernel-based methods. IEEE Transactions on Neural Networks 1999, 10(5):1000-1017. 10.1109/72.788641View ArticleGoogle Scholar
  15. Mika S, Schölkoph B, Smola A, Müller K-R, Scholz M, Rätsch G: Kernel PCA and de-noising in feature spaces. Proceedings of the conference on Advances in Neural Information Processing Systems II, 1999 536-542.Google Scholar
  16. Kim KI, Franz MO, Schölkopf B: Iterative kernel principal component analysis for image modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 2005, 27(9):1351-1366.View ArticleGoogle Scholar
  17. Ogawa T, Haseyama M: POCS-based texture reconstruction method using clustering scheme by kernel PCA. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences 2007, 90(8):1519-1527.View ArticleGoogle Scholar
  18. Ogawa T, Haseyama M: Adaptive missing texture reconstruction method based on kernel canonical correlation analysis with a new clustering scheme. IEICE Transactions. Fundamentals 2009, 92(8):1950-1960.View ArticleGoogle Scholar
  19. Aharon M, Elad M, Bruckstein A: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing 2006, 54(11):4311-4322.View ArticleGoogle Scholar
  20. Elad M, Aharon M: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing 2006, 15(12):3736-3745.MathSciNetView ArticleGoogle Scholar
  21. Mairal J, Elad M, Sapiro G: Sparse representation for color image restoration. IEEE Transactions on Image Processing 2008, 17(1):53-69.MathSciNetView ArticleMATHGoogle Scholar
  22. Wohlberg B: Inpainting with sparse linear combinations of exemplars. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '09), April 2009 689-692.Google Scholar
  23. Shen B, Hu W, Zhang Y, Zhang Y-J: Image inpainting via sparse representation. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '09), April 2009 697-700.Google Scholar
  24. Girod B: What's wrong with mean-squared error? In Digital Images and Human Vision. Edited by: Watson AB. MIT Press, Cambridge, Mass, USA; 1993:207-220.Google Scholar
  25. Miyahara M, Kotani K, Algazi VR: Objective picture quality scale (PQS) for image coding. IEEE Transactions on Communications 1998, 46(9):1215-1226. 10.1109/26.718563View ArticleGoogle Scholar
  26. Damera-Venkata N, Kite TD, Geisler WS, Evans BL, Bovik AC: Image quality assessment based on a degradation model. IEEE Transactions on Image Processing 2000, 9(4):636-650. 10.1109/83.841940View ArticleGoogle Scholar
  27. Sheikh HR, Bovik AC, de Veciana G: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Transactions on Image Processing 2005, 14(12):2117-2128.View ArticleGoogle Scholar
  28. Sheikh HR, Bovik AC: Image information and visual quality. IEEE Transactions on Image Processing 2006, 15(2):430-444.View ArticleGoogle Scholar
  29. Sheikh HR, Sabir MF, Bovik AC: A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing 2006, 15(11):3440-3451.View ArticleGoogle Scholar
  30. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 2004, 13(4):600-612. 10.1109/TIP.2003.819861View ArticleGoogle Scholar
  31. Channappayya SS, Bovik AC, Caramanis C, Heath RW Jr.: Design of linear equalizers optimized for the structural similarity index. IEEE Transactions on Image Processing 2008, 17(6):857-872.MathSciNetView ArticleGoogle Scholar
  32. Mallat SG, Zhang Z: Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing 1993, 41(12):3397-3415. 10.1109/78.258082View ArticleMATHGoogle Scholar
  33. Pati YC, Rezaiifar R, Krishnaprasad PS: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Proceedings of the 27th Asilomar Conference on Signals, Systems & Computers, November 1993 1: 40-44.View ArticleGoogle Scholar
  34. Wang Z, Bovik AC: Modern Image Quality Assessment. Morgan & Claypool, San Rafael, Calif, USA; 2006.Google Scholar
  35. Arias P, Caselles V, Sapiro G: A variational framework for nonlocal image inpainting. Proceedings of the Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR '09), 2009 345-358.View ArticleGoogle Scholar

Copyright

© T. Ogawa and M. Haseyama. 2010

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.