Missing Texture Reconstruction Method Based on Perceptually Optimized Algorithm
© T. Ogawa and M. Haseyama. 2010
Received: 23 August 2010
Accepted: 26 October 2010
Published: 26 October 2010
This paper presents a simple and effective missing texture reconstruction method based on a perceptually optimized algorithm. The proposed method utilizes the structural similarity (SSIM) index as a new visual quality measure for reconstructing missing areas. Furthermore, in order to adaptively reconstruct target images containing several kinds of textures, the following two novel approaches are introduced into the SSIM-based reconstruction algorithm. First, the proposed method performs SSIM-based selection of the optimal known local textures to adaptively obtain subspaces for reconstructing missing textures. Secondly, missing texture reconstruction that maximizes the SSIM index in the known neighboring areas is performed. In this approach, the nonconvex maximization problem is reformulated as a quasi convex problem, and adaptive reconstruction of the missing textures based on the perceptually optimized algorithm becomes feasible. Experimental results show impressive improvements of the proposed method over previously reported reconstruction methods.
Restoration of missing areas in digital images has been intensively studied since it can be applied to a number of fundamental applications such as restoration of corrupted old films, removal of unnecessary objects, and error concealment. Therefore, many methods have been proposed in order to realize these applications. Generally, they are broadly classified into two categories, structural and textural reconstruction approaches, and many papers on these approaches have been published. Attractive methods that perform simultaneous reconstruction of missing structures and textures in images have also been proposed [1, 2]. Most algorithms reported in the literature are based on structural inpainting techniques for accurate reconstruction of missing edges [3–5]. These techniques are effective for pure structure images. However, since general images also contain many textures, different methods work better in these areas. Thus, several methods have been proposed for accurate reconstruction of missing textures [6–12]. The remainder of this paper focuses on the texture reconstruction approach with discussion of its details.
Traditionally, missing texture reconstruction is realized as one of the applications of texture synthesis. Efros et al. firstly proposed a pioneered method for the texture synthesis [6, 7]. Their approach models textures by using the MRF (Markov random field) model and enables missing texture reconstruction by copying pixels of a target image itself, that is, nonparametric sampling in synthesis. Furthermore, Wei and Levoy proposed a fast algorithm for the searching step in the texture synthesis by utilizing multiresolution concepts . Then many methods which perform the exemplar-based inpainting are mainly inspired by the nonparametric sampling in . Drori et al. proposed a fragment-based algorithm for image completion which could preserve structures and textures . Furthermore, the exemplar-based image inpainting method proposed by Criminisi et al. is a representative one based on the texture synthesis [10, 11]. This method adopts a patch-based greedy sampling scheme similar to the fragment-based completion, but it is simpler and faster. A good review of the exemplar-based inpainting methods based on  is shown in .
In the field of texture reconstruction, not only the methods based on the texture synthesis but also many methods, which estimate missing intensities by utilizing statistical features of known textures within a target image as training patterns, have been proposed. Generally, since the restoration of missing areas is an ill-posed problem, it is difficult to directly estimate the missing intensities. Thus, these methods perform approximation of textures within the target image in lower-dimensional subspaces and enable derivation of the inverse projection for the corruption. Amano and Sato proposed an effective PCA-based method for reconstructing missing textures using back projection for lost pixels and realized accurate reconstruction performance . Furthermore, kernel methods have recently been developed and their achievements have been reported in a number of papers [14–16]. Subspaces constructed on the basis of kernel methods are also suitable for approximating nonlinear texture features in target images. Several missing texture reconstruction methods that utilize projection schemes onto nonlinear subspaces obtained by kernel PCA and CCA have been proposed [17, 18]. Furthermore, image reconstruction based on sparse representation approaches [19–21] have been intensively studied. By utilizing sparse representation, optimal signal atoms can be adaptively selected from a dictionary for representing target signals. This means that these methods can adaptively provide optimal subspaces for restoring missing areas. Several missing area reconstruction methods based on sparse representation have been proposed [21–23].
It should be noted that in conventional methods, reconstruction is mostly performed by minimizing errors of intensities, that is, the mean squared error (MSE), which is the most popular metric. However, it has been reported that MSE optimal algorithms do not necessarily produce images of high visual quality . Thus, it may not be appropriate to utilize the MSE as a quality measure for reconstruction. Recent advances in full-reference image quality assessment (IQA) have resulted in the emergence of several powerful perceptual distortion measures that outperform the MSE and its variants. Criteria such as PQS , NQM , IFC , and VIF  are well known as perceptual distortion measures, and their performances have been evaluated in detail . The structural similarity (SSIM) index  is utilized as a representative quality measure in many fields of image processing. Since its formulation is simple and easy to be analyzed, the SSIM index can be applied to not only image quality assessment but also design of linear equalizers . Therefore, by using this quality measure, accurate reconstruction of missing textures can be expected.
SSIM-based selection of the optimal known local textures for reconstructing target textures including missing areas.
Reconstruction of the target textures maximizing the SSIM index in the known neighboring areas.
The first approach provides optimal subspaces for the following SSIM-based reconstruction approach by using an algorithm similar to several matching pursuit algorithms [32, 33]. Furthermore, in the second approach, we introduce the computation scheme in  into the SSIM-based reconstruction algorithm, and its nonconvex maximization problem is reformulated as a quasi convex problem. Then the optimal solution based on the SSIM index can be computed, and accurate reconstruction of the missing textures is expected.
This paper is organized as follows. First, in Section 2, we briefly explain the SSIM index used as the quality measure in the proposed method. Next, the missing texture reconstruction method based on the perceptually optimized algorithm is proposed in Section 3. Experimental results that verify the performance of the proposed method are shown in Section 4. Finally, conclusions are given in Section 5.
2. SSIM Index
As shown in (1)–(3), the SSIM index is consistent with luminance and contrast masking and the correlation.
In [30, 34], the effectiveness of the SSIM index as a quality measure and its superiority to the MSE and its variants are presented in detail. Generally, the MSE cannot reflect perceptual distortions, and its value becomes higher for images altered with some distortions such as mean luminance shift, contrast stretch, spatial shift, spatial scaling, and rotation, yet negligible loss of subjective image quality. Furthermore, blurring severely deteriorates the image quality, but its MSE becomes lower than those of the above alternation. On the other hand, the SSIM index is defined by separately calculating the three similarities in terms of the luminance, variance, and structure, which are derived based on the HVS (human visual system) not accounted for by the MSE. Therefore, it becomes a better quality measure providing a solution to the above problem, and this is also confirmed in . Then we can expect that the use of this similarity for the reconstruction of missing areas will provide successful results. The specific effectiveness of the SSIM index for the reconstruction is discussed in Section 4.
3. Adaptive Missing Texture Reconstruction Based on SSIM Index
In the target image, there are several kinds of textures, that is, there are many known patches whose textures are quite different from that of the target patch . Such patches should not affect the reconstruction of the target patch . In order to reconstruct the missing textures within the target patch from only the same kinds of textures, we have to select those textures from the known areas. Therefore, the proposed method first performs selection of the optimal known patches utilized for reconstruction of the target patch based on the SSIM index. Furthermore, by using the selected patches, we derive the representation model optimized for the target patch in terms of the SSIM index to reconstruct the missing area . Then the proposed method can adaptively reconstruct the missing textures from only the same kinds of known textures based on the perceptually optimized scheme.
In this section, we first show the SSIM-based algorithm for selecting the optimal known patches in Section 3.1. The reconstruction algorithm of the missing textures based on the SSIM index is shown in Section 3.2.
3.1. SSIM-Based Optimal Texture Selection Algorithm
In this subsection, we present the SSIM-based optimal texture selection algorithm. First, we clip known patches ( ) whose size is pixels from the target image in the same interval. For the following explanation, two vectors that correspond to and of each patch are denoted as ( ) and ( ), respectively. From the clipped patch, we select patches that are optimal for reconstruction of the target patch . The order of the value is explained in Section 4. In the reconstruction algorithm shown in the following subsection, the target patch is represented by a linear combination of the selected known patches in such a way that the SSIM index in the known area becomes maximum. Therefore, we should select known patches that provide the optimal linear combination. Note that the selection of such optimal known patches is an NP-hard problem. Thus, we adopt the simplest algorithm that selects the optimal known patches one by one, and it is similar to several matching pursuit algorithms [32, 33]. In the rest of this subsection, the details of the th ( ) optimal patch selection are shown.
is a coefficient vector for obtaining . The proposed method estimates the optimal vector of ( ) which provides the optimal representation performance based on the SSIM index. Then the best matched patch , whose vector approximating has a higher value of the SSIM index than those of other patches, is selected.
The proposed method calculates the optimal vector in (7) by simply applying the steepest ascend algorithm to in (9). Note that we can calculate the optimal vector more accurately by using the algorithm shown in the following subsection. However, in order to reduce the computation time of the proposed method, we adopt the steepest ascend algorithm in this subsection. It is well known that the steepest ascend algorithm cannot necessarily provide the globally optimal solutions in (7), but this algorithm can save the computation time compared to the algorithm shown in the following subsection. The details are shown later. From the above reason, we utilize this scheme in the proposed method.
By iterating the above procedures times, we can select the optimal known patches based on the SSIM index and denote them as ( ). Algorithm 1 shows the specific procedures of this selection algorithm. Then by utilizing the obtained known patches, the proposed method can adaptively provide the optimal subspace for the target patch , and accurate reconstruction based on the SSIM index is also expected in the following subsection. For the following explanation, we denote two vectors obtained from in the same way as and as and , respectively.
3.2. Texture Reconstruction Algorithm
where we denote them as and since they depend on . Furthermore, in the above two equations, is a Moore-Penrose pseudoinverse matrix of . Then the proposed method estimates the optimal value of by using a standard bisection procedure, and the optimal vectors are calculated for several values of ( ) to select maximizing (13), where is a step size and determines the search range of . Algorithm 2 shows the details on the estimation of in the proposed method.
Note that the algorithm for calculating the optimal linear combination in this subsection provides better solutions than those in the previous subsection. However, this algorithm needs to perform iterations to determine the value of . Furthermore, it also needs the iteration to search the optimal value of as shown in Algorithm 2. Then since it is confirmed that the algorithm shown in this subsection takes more computation time compared to the algorithm shown in the previous subsection, we perform the selection of the optimal known patches ( ) as shown in the previous subsection.
As described above, we can reconstruct the missing texture in the target patch. The proposed method clips patches ( pixels) at the same interval from the upper-left of the target image in a rasterscanning order. If the clipped patches contain missing areas, we regard them as the target patches and reconstruct them by using the above approach. Note that each restored pixel has multiple estimation results if the clipping interval is smaller than the size of the patches. In this case, the proposed method regards the result maximizing (13) as the final one. The proposed method does not utilize the already obtained results for reconstructing other missing areas. Therefore, the performance of the proposed method does not depend on the order of the reconstruction, that is, the positions of the patches including missing areas do not influence the results.
4. Experimental Results
For comparison, Figures 2(e)–2(i), respectively, show the results obtained by the method which selects known patches randomly but reconstructs missing areas in the same way as Section 3.2, the method which utilizes the MSE instead of the SSIM index, the exemplar-based texture reconstruction method , the PCA-based texture reconstruction method , and the method based on sparse representation . In order to verify the effectiveness of the selection algorithm shown in Section 3.1, we show the results in Figure 2(e). The method in  is one of the most influential works in the field of the exemplar-based texture reconstruction, and we utilize this method for the comparison of the proposed method as shown in Figure 2(g). Furthermore, since subspaces optimized on the basis of the MSE criterion are utilized for the reconstruction of missing textures, the other conventional methods shown in Figures 2(f), 2(h), and 2(i) are suitable for verifying the performance of the proposed method. Particularly, the methods in [13, 21] are, respectively, representative works using PCA and sparse representation.
Note that the dimensions of the subspaces utilized in the proposed method and the conventional methods are set to the same value 40 ( ). In the proposed method, we have to set to a smaller value than the number of known pixels within the target patch . Furthermore, this should be satisfied for all target patches including missing areas within the target image. Otherwise, the calculation of the optimal vector in (11) and (12) generally becomes an underdetermined problem. This means we have to set to a sufficiently small value in order to avoid the problem in (12), being an underdetermined problem. Generally, if becomes larger than the number of the known pixels in , some constraints must be introduced as regularization terms for avoiding the system instability. Furthermore, if there is no limitation of the cost function utilized for the reconstruction, some constraints must be also adopted. Several existing studies for inpainting using a linear combination of patches adopt some restrictions such as the sum of the linear coefficients being one , and so forth. On the other hand, our method sets the value of to a much smaller value than the number of known pixels within the target patch , and the maximum range of the SSIM index is limited to one. Thus, since the system is not instable in (12), we think that our method does not have to utilize other restrictions. Furthermore, it seems that the value of should be set to about one-tenth of the dimension of . This means we assume the percentages of the known pixels within the target patches are larger than 10%. In the experiments, we set to 40, that is, a much smaller value than the above criterion to clearly show the difference of the reconstruction performance between the proposed method and the conventional methods.
Performance comparison (MSE) of the proposed method and the conventional methods.
Performance comparison (SSIM) of the proposed method and the conventional methods.
In the conventional methods, the subspace estimation and texture reconstruction schemes are based on the MSE criterion. However, the MSE optimal algorithms do not necessarily produce images of high visual quality, and the reconstruction results may be degraded. Specifically, it is well known that most images contain more low-frequency components than high-frequency components. Thus, models using the subspaces based on the MSE can only represent such low-frequency components, and it becomes difficult to reconstruct the missing high-frequency components of the missing areas. This means the reconstruction results tend to be blurred. Then since the representation performance, that is, the reconstruction performance of each patch, becomes worse, the color discontinuities at the border of the missing areas and that of patches also occurs. On the other hand, the proposed method adopts the SSIM index for obtaining subspaces and reconstructing missing textures. The basic formulation of the SSIM index is obtained from the three terms , , and as shown in (1). These terms respectively represent the mean similarity, the variance similarity, and the structural correlation. The first term and the third term compare the vector lengths and angles, and they separately provide those similarities. Note that the second term compares the contrast of the two vectors, that is, it enables the comparison of the texture roughness. Therefore, this can be regarded as the term comparing how much high-frequency components the target textures contain. This is also pointed out in , and they confirmed that the SSIM index of blurred images which were perceptually degraded severely became lower. Then it seems that the proposed method can avoid the oversmoothness of the reconstruction results by utilizing the SSIM index including the above useful term. Since the SSIM index outperforms the MSE as a perceptual distortion measure, our method can provide the solution to the conventional problems and realize more accurate reconstruction.
Finally, we show the computation time of the proposed method. The experiments shown above were performed on a personal computer using Intel(R) Core(TM) i7 950 CPU 3.06 GHz with 8.0 Gbytes RAM. The proposed method was implemented by using Matlab. The average computation times to perform the algorithms shown in Sections 3.1 and 3.2 for each target patch are, respectively, sec and sec. Thus, from these results, we can see that the reduction of the computational cost in the optimal patch selection algorithm of the proposed method is necessary for practical use. This issue will be addressed in a future work.
In this paper, we have presented an adaptive method for reconstructing missing textures based on the SSIM index. The proposed method adaptively obtains subspaces utilized for the reconstruction of missing textures by selecting the optimal known local textures based on the SSIM index. Furthermore, missing texture reconstruction maximizing the SSIM index can be realized by reformulating the nonconvex problem as a quasi convex problem. Then the proposed method enables adaptive texture reconstruction based on the perceptually optimized algorithm. Consequently, impressive improvement of the proposed method over previously reported methods was confirmed.
In the experiments, we manually determine the parameters of the proposed method. It is desirable that these parameters be adaptively determined from the target image. Thus, we need to complement this determination algorithm. Extension of the framework to texture reconstruction of other types of missing imagery data is also needed for various applications. Finally, we would like to study these ideas for interpolation in video data. These topics will be the subjects of subsequent reports.
This work was partly supported by Grant-in-Aid for Scientific Research (B) 21300030, Japan Society for the Promotion of Science (JSPS).
- Bertalmio M, Vese L, Sapiro G, Osher S: Simultaneous structure and texture image inpainting. IEEE Transactions on Image Processing 2003, 12(8):882-889. 10.1109/TIP.2003.815261View ArticleGoogle Scholar
- Rane SD, Sapiro G, Bertalmio M: Structure and texture filling-In of missing image blocks in wireless transmission and compression applications. IEEE Transactions on Image Processing 2003, 12(3):296-303. 10.1109/TIP.2002.804264MathSciNetView ArticleGoogle Scholar
- Ballester C, Bertalmio M, Caselles V, Sapiro G, Verdera J: Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing 2001, 10(8):1200-1211. 10.1109/83.935036MathSciNetView ArticleMATHGoogle Scholar
- Chan TF, Shen J: Nontexture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation 2001, 12(4):436-449. 10.1006/jvci.2001.0487View ArticleGoogle Scholar
- Rareş A, Reinders MJT, Biemond J: Edge-based image restoration. IEEE Transactions on Image Processing 2005, 14(10):1454-1468.View ArticleMATHGoogle Scholar
- Efros AA, Leung TK: Texture synthesis by non-parametric sampling. Proceedings of the 1999 7th IEEE International Conference on Computer Vision (ICCV'99), September 1999 1033-1038.Google Scholar
- Efros AA, Freeman WT: Image quilting for texture synthesis and transfer. Proceedings of the Computer Graphics Annual Conference (SIGGRAPH '01), August 2001 341-346.Google Scholar
- Wei L-Y, Levoy M: Fast texture synthesis using tree-structured vector quantization. In Proceedings of the ACM SIGGRAPH Conference on Computer Graphics (SIGGRAPH '00), 2000. Edited by: Akeley K. ACM Press /ACM SIGGRAPH /Addison Wesley Longman; 479-488.Google Scholar
- Drori I, Cohen-Or D, Yeshurun H: Fragment-based image completion. In Proceedings of the ACM SIGGRAPH 2003 Papers (SIGGRAPH '03), 2003, New York, NY, USA. ACM Press; 303-312.Google Scholar
- Criminisi A, Pérez P, Toyama K: Object removal by exemplar-based inpainting. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2003 721-728.Google Scholar
- Criminisi A, Pérez P, Toyama K: Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 2004, 13(9):1200-1212. 10.1109/TIP.2004.833105View ArticleGoogle Scholar
- Fidaner IB: A survey on variational image inpainting texture synthesis and image completion. http://www.scribd.com/doc/3012627/A-Survey-on-Variational-Image-Inpainting-Texture-Synthesis-and-Image-Completion
- Amano T, Sato Y: Image interpolation using BPLP method on the eigenspace. Systems and Computers in Japan 2007, 38(1):87-96. 10.1002/scj.10319View ArticleGoogle Scholar
- Schölkopf B, Mika S, Burges CJC, Knirsch P, Müller K-R, Rätsch G, Smola AJ: Input space versus feature space in kernel-based methods. IEEE Transactions on Neural Networks 1999, 10(5):1000-1017. 10.1109/72.788641View ArticleGoogle Scholar
- Mika S, Schölkoph B, Smola A, Müller K-R, Scholz M, Rätsch G: Kernel PCA and de-noising in feature spaces. Proceedings of the conference on Advances in Neural Information Processing Systems II, 1999 536-542.Google Scholar
- Kim KI, Franz MO, Schölkopf B: Iterative kernel principal component analysis for image modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence 2005, 27(9):1351-1366.View ArticleGoogle Scholar
- Ogawa T, Haseyama M: POCS-based texture reconstruction method using clustering scheme by kernel PCA. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences 2007, 90(8):1519-1527.View ArticleGoogle Scholar
- Ogawa T, Haseyama M: Adaptive missing texture reconstruction method based on kernel canonical correlation analysis with a new clustering scheme. IEICE Transactions. Fundamentals 2009, 92(8):1950-1960.View ArticleGoogle Scholar
- Aharon M, Elad M, Bruckstein A: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing 2006, 54(11):4311-4322.View ArticleGoogle Scholar
- Elad M, Aharon M: Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing 2006, 15(12):3736-3745.MathSciNetView ArticleGoogle Scholar
- Mairal J, Elad M, Sapiro G: Sparse representation for color image restoration. IEEE Transactions on Image Processing 2008, 17(1):53-69.MathSciNetView ArticleMATHGoogle Scholar
- Wohlberg B: Inpainting with sparse linear combinations of exemplars. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '09), April 2009 689-692.Google Scholar
- Shen B, Hu W, Zhang Y, Zhang Y-J: Image inpainting via sparse representation. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '09), April 2009 697-700.Google Scholar
- Girod B: What's wrong with mean-squared error? In Digital Images and Human Vision. Edited by: Watson AB. MIT Press, Cambridge, Mass, USA; 1993:207-220.Google Scholar
- Miyahara M, Kotani K, Algazi VR: Objective picture quality scale (PQS) for image coding. IEEE Transactions on Communications 1998, 46(9):1215-1226. 10.1109/26.718563View ArticleGoogle Scholar
- Damera-Venkata N, Kite TD, Geisler WS, Evans BL, Bovik AC: Image quality assessment based on a degradation model. IEEE Transactions on Image Processing 2000, 9(4):636-650. 10.1109/83.841940View ArticleGoogle Scholar
- Sheikh HR, Bovik AC, de Veciana G: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Transactions on Image Processing 2005, 14(12):2117-2128.View ArticleGoogle Scholar
- Sheikh HR, Bovik AC: Image information and visual quality. IEEE Transactions on Image Processing 2006, 15(2):430-444.View ArticleGoogle Scholar
- Sheikh HR, Sabir MF, Bovik AC: A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing 2006, 15(11):3440-3451.View ArticleGoogle Scholar
- Wang Z, Bovik AC, Sheikh HR, Simoncelli EP: Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 2004, 13(4):600-612. 10.1109/TIP.2003.819861View ArticleGoogle Scholar
- Channappayya SS, Bovik AC, Caramanis C, Heath RW Jr.: Design of linear equalizers optimized for the structural similarity index. IEEE Transactions on Image Processing 2008, 17(6):857-872.MathSciNetView ArticleGoogle Scholar
- Mallat SG, Zhang Z: Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing 1993, 41(12):3397-3415. 10.1109/78.258082View ArticleMATHGoogle Scholar
- Pati YC, Rezaiifar R, Krishnaprasad PS: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Proceedings of the 27th Asilomar Conference on Signals, Systems & Computers, November 1993 1: 40-44.View ArticleGoogle Scholar
- Wang Z, Bovik AC: Modern Image Quality Assessment. Morgan & Claypool, San Rafael, Calif, USA; 2006.Google Scholar
- Arias P, Caselles V, Sapiro G: A variational framework for nonlocal image inpainting. Proceedings of the Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR '09), 2009 345-358.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.