 Research
 Open Access
 Published:
Automatic movie restoration based on wave atom transform and nonparametric model
EURASIP Journal on Advances in Signal Processing volume 2012, Article number: 132 (2012)
Abstract
This article presents a new scheme for movie descratching, denoising and deblotching based on wave atom transform and nonparametric model. According to the analysis of the noise, scratches and blotches in the film, we establish a hybrid model for movie signal. In the proposed model, we model scratch as directional additive noise, which can be represented effectively by a new multiscale analysis tool called wave atom transform. Based on wave atoms, denoising and descratching are combined in a nested thresholding framework which can suppress both homogeneous noise and structural artifacts. On the other hand, blotch is described as multiplicative noise in the model, which can be restored by a “detecting and inpainting” strategy. We use the inter and intraframe discontinuity of blotch to detect them and then restore them by kernel regression method. Combining the processing operations above together, we design an effective automatic movie restoration system. Experimental results show that the proposed method not only achieves excellent performances even for seriously damaged movies but also has low computational complexity compared with the existing methods.
Introduction
Movie restoration is a problem with both academic challenge and great practical value. Due to long display time and undesirable storage environment, movies are damaged by various kinds of artifacts such as noise, loss of contrast, scratch, blotch, etc. These artifacts have negative influences on the artistic and commercial values of movie. For recent years, many efforts have been devoted to movie restoration, especially to denoising, descratching and deblotching problems.
In the literature, wavelet domain softthresholding based algorithms [1–4] have shown good performances in image denoising. For those methods, the selection of threshold is of vital importance. Although Donoho [1] gives the universal upper bound of threshold and researchers try to achieve even better solutions in [2, 4], performances of these algorithms largely hinge on accurate noise variance estimation. An expectationmaximization (EM) algorithm for noise estimation is proposed in [3], but its computational complexity is very high. Besides transformbased methods mentioned above, nonparametric and parametric model based algorithms [5–7] also achieve good denoising results. Currently, the methods combining parametric and nonparametric model achieve stateoftheart results in image and video denoising, such as the BM3D in [8] algorithm and the trainingbased sparse denoising method in [9]. Both of them take advantages of selfsimilarity of the patches in the image and represent the image sparsely by a dictionary. However, although those algorithms are effective to remove Gaussian noise, they do not work well for structural artifacts.
Recently, some algorithms for structural artifacts removal are proposed. For example, in [10, 11], a trainingbased adaptive least square filter is designed to remove the coding artifacts in the image. Many descratching and deblotching algorithms have also been proposed in the literature [12–15]. A joint scratch and blotch restoration method using Markov Random Field (MRF) model was presented in [16]. However, this method is not robust enough for movies with large motion. The methods in [12, 17–20] exploit interframe information in the Bayesian framework for scratch detection and restoration. Because of using the directionality of scratch in the detection phase, these method cannot be extended to blotch restoration straightforwardly. In our opinion, because scratch has strong directionality whereas blotch is isotropic, scratch and blotch should be restored separately. How to restore them respectively to achieve better result is a problem we try to solve. On the other hand, a common point of all these methods is that all of them use sequential detection and restoration steps, thus the overall performance will be limited by the accuracy of the detection stage, which is often very ad hoc in nature. How to reduce the dependence of restoration result on the accuracy of detection is another interesting problem.
The contribution of this work includes three folds. Firstly, according to the analysis of the scratches in the film, we model scratch as directional additive noise which can be represented effectively by wave atoms. Secondly, inspired by softthresholding denoising method, we combine denoising and descratching in a nested thresholding framework based on wave atom transform, which is robust to the performance drift of detection step. Thirdly, after denoising and descratching, we detect blotches by the inter and intraframe discontinuity and restore them by kernel regression. Because scratches have been restored before, the computational complexity caused by applying kernel regression to deblotching.
The article is organized as follows. In Signal model of movie section, we establish a hybrid model of movie signal and analyze the properties of noise, scratch and blotch in the movie. In Joint descratching and denoising section, we propose a nested thresholding method in wave atom domain, which can be used for denoising and descratching. In Blotch detection and restoration section, we design a robust blotch detection and restoration algorithm based on motion estimation and nonparametric kernel regression. The integrated system and experimental results are given in Implementation details and experimental results section. Conclusions and future work section concludes this article.
Signal model of movie
Hybrid noise model
We model movie signal by a hybrid noise model, as shown in (2.1).
where X is the ideal pixel value of video signal and Y is the noisy observation. N is homogeneous additive noise and S is the scratch component. B is the blotch component. q is a random variable controlling the transparency of blotch, and q∈[0,1] generally. If B is regarded as additive noise, (2.1) becomes a pure additive noise model. However, because B has not homogeneous property, the additive noise model increases the difficulty of analysis and restoration. For simplification, we restrict q∈{0,1}. Based on the binarization of q, B is modeled as multiplicative noise and (2.1) becomes a hybrid noise model.
This hybrid model is different from the model proposed in [21–24], which regards scratch as multiplicative noise. This change is based on both physical and chemical causes of scratch. Scratch results from the shedding of coats on the film. As long as the contact surface between the projector and the film is not clean, scratches will appear in the film during displaying. We can model the phenomenon as an additive process, i.e. the film coat becomes thinner damaged by those foreign matters. On the other hand, blotch artifacts can be caused by mildews, dirty spots or the shedding of film coats. The spatial and temporal discontinuity of blotch makes it suitable to be modeled with the multiplicative noise model. The hybrid model guides us restore them by different methods.
The analysis of noise, scratch and blotch
Generally, distribution of N in our model (2.1) can be modeled as Gaussian noise. However, since the variance σ^{2} is not known beforehand, softthresholding based denoising algorithm cannot be used directly. For simplifying the problem, we make the following assumption: the noise in the movie is homogeneous, which obeys N(0,σ^{2}) in the spatiotemporal domain. In other words, the noise in the movie is assumed to be stationary in the spatiotemporal domain, which can be estimated by the residual signal of successive frames.
Considering its generative mechanism, scratch is a kind of anisotropic sudden interference, usually along an approximate vertical direction. It means that S is a directional noise. Further, scratch can be classified into two types according to its interframe continuity, as exemplified in Figure 1. In Figure 1a, we can find that the first type of scratch is relatively “weak”, which is short and narrow and does not have interframe continuity. This kind of scratch can be detected straightforwardly by motion estimation based method. Figure 1b shows another type of scratch whose location in adjacent frames is almost stable. Motion estimation based method cannot be useful for the second type of continuous scratch.
Because of the variety of scratch, the detection method based on either inter or intraframe discontinuity is sensitive to the choice of parameters. For detecting scratch more accurately, Bruni proposed a damped sinusoid model for scratch in [15], which is useful to detect isolated and serious scratch. Bruni’s model reveals the oscillatory property of scratch—to each individual scratch, it can be described as a damped sinusoid signal. In fact, this microscopic view can be extended to macroscopic situation. Because scratch can accumulate during multiple times of film projection, not only does the individual but also the whole scratches in the frame obey oscillatory pattern. As shown in Figure 2, we choose several frames from some old movies to investigate the pattern of scratch. It can be found that scratches in the film have local oscillatory patterns, and such pattern would change from a local phenomenon to a global one with the increase of the displaying time.
Bruni’s model aims to solve the detection problem only while the proposed model provides a guidance to design a restoration algorithm directly. Based on the analysis above, we can find that additivity, directionality and local (even global) oscillation are three features of scratch. According to the intrinsic properties, wave atombased method matches this problem perfectly, which will be addressed with details in Joint descratching and denoising section.
Different from scratch, blotch artifact shows the isotropic pattern in most situations. Generally, it does not have the spatiotemporal continuity between and among movie frames. The blotch artifact can be very bright, very dark or even semitransparent, as instanced in Figure 3. All of these properties increase the difficulty of removing blotch [16, 21–25].
Joint descratching and denoising
Overview of wave atom transform
Wave atom transform is developed firstly by Demanet and Ying [26, 27]. It is a variation of traditional wavelet aiming to overcome the weakness of wavelet in representing the directionality of high dimensional signals. The definition of wave atoms is closely related to the Fourier transform. Consider a 2D Fourier transform pair of f(x) and $\widehat{f}\left(\mathit{\omega}\right)$, where x=(x_{1}x_{2}), ω=(ω_{1}ω_{2})are the coordinates in spatial domain and frequency domain, then wave atoms can be defined as φ_{ μ }(x), where μ=(j m n)=(j m_{1}m_{2}n_{1}n_{2}) is a label representing the scale, direction and location of the corresponding wave atom in 2D situation. The indexed point (x_{ μ }ω_{ μ }) in phasespace is defined as
where C^{1}, C^{2} are positive constants.
It can be seen that the notations follow those of the wavelet and curvelet, where j controls the resolution scale, and m and n control the location in frequency and spatial domain, respectively. However, there are two differences between wavelet and wave atom. First, wavelets obey a linear scaling wavelength while wave atoms obey the parabolic scaling wavelength: at scale 2^{−2j} the essential frequency support is of size 2^{j}, and at frequency 2^{2j}, the essential time support is of size 2^{−j}. Second, the basis function of wave atom is different from that of conventional wavelet.
In the 1D situation, the frequency formula of basis function $\left(\right)close="">{\psi}_{m}^{0}\left(x\right)$ is
with ε_{ m }=(−1)^{m} and ${\alpha}_{m}=\frac{\Pi}{2}(m+\frac{1}{2})$. g is a realvalued, C^{∞} bump function on a compact support with length 2Π, which promises the energy of basis function is normalized ($\parallel {\psi}^{0}{\parallel}_{2}=1$). Similarly, the 2D basis function of wave atom can be written as
The transform process of wave atom transform is similar to that of curvelet, which is based on FFT and IFFT. For a 2D signal u∈R^{N×N}, we first perform a FFT of size N×N of u, and then get $\widehat{\mathbf{u}}$. For each pair (j m), we wrap the product ${\widehat{\psi}}_{\mathbf{m}}^{j}\widehat{\mathbf{u}}$ by periodicity insert the interval [−2^{j}Π,2^{j}Π×[−2^{j}Π,2^{j}Π. We then compute inverse FFT of size 2^{j} of the result to get c_{ μ }. We repeat the process above over each pair (j m)to get all the coefficients of wave atom transform. Finally, we permute these coefficients according to the label μ=(j m n) to get a matrix of coefficients. Please refer to [26, 27] for the details of wave atom transform.
Figure 4 gives some examples of wave atoms and their frequency representation. It can be found that wave atoms have outstanding performance in directional representation. This is one reason why wave atoms are used for scratch representation.
The features of wave atom transform to represent scratch
Recalling the analysis in “The analysis of noise, scratch and blotch” section, we know that scratch is a directional additive noise having an oscillatory pattern. Figure 5 illustrates the coefficient’s distribution of a straightline image in wave atom, curvelet, wavelet and dual tree complex wavelet [28] domains after 3level decomposition. All the four coefficient domains can be divided into several regions corresponding to different directions and scales. We can find that the coefficients corresponding to vertical, horizontal and diagonal lines are separated perfectly in wave atom domain. The coefficients in red boxes in Figure 5b,e,h,k correspond to vertical signal. An intuitive thresholding example is to remove the coefficients contained in red boxes in both domains. Figure 5c,f,i,l give the vertical line removal results. We find that under the same situation (removing the coefficients corresponding to the first two levels), the vertical line removal result using wave atom transform is clearly superior. It means that under the same decomposition level, the directionality of wave atom transform is better than three competitors.
Another feature of wave atom transform is its sparsity of representing oscillatory signals. In [27], wave atom transform has been proven to be able to achieve sparser representation of oscillatory signal than the existing transforms including wavelet, and curvelet. If O(N)wave atom coefficients suffice to represent an oscillatory signal to reasonable accuracy, O(N^{3/2}) curvelet coefficients or O(N^{2})wavelet coefficients will be needed to achieve the same accuracy.
The analysis above implies that if we decompose an image with scratches by wave atoms, the coefficients corresponding to scratches will be sparse and distribute in a narrow region in transform domain.
Nested thresholding in wave atom domain
We now consider how to threshold the coefficients of wave atoms so as to remove scratch and noise. The coefficients representing scratch will be limited in the narrow region corresponding to vertical component of signal, as shown in the red boxes in Figure 5b. A naive method that removes all the coefficients in this region will damage meaningful vertical image pixels rather than scratch. Another difficulty is to avoid ringing artifacts caused by the sidelobes of signal, which is common in wavelet shrinkage methods. To overcome these two problems, we propose a nested thresholding method.
We use a vertical 5×5Sobel detector to find where scratches are likely to appear and get a mask matrix M. To ensure an accurate scratch detection, morphological dilation operation [29] is applied to M. Besides Sobel detector, there exists many other effective edge and ridge detector, such as the method in [30]. However, considering the requirement for processing a large volume of data realtime, we apply the simple strategy.
The threshold of Sobel detector is critical to M. If the selected threshold is too low, it will not only generate false detection result of scratch; if the threshold is too high, then many of the scratches may not be detected or the detected scratches are too segmented. To make a trade off between computational complexity and robustness, we choose the threshold adaptively by Otsu threshold segmentation method [31]. According to the analysis below we will prove that the proposed method is robust to detection false.
Rather than classic shrinkage of the coefficients in transform domain, we propose to iteratively subtract a quantized residual from the reconstructed signal (or original signal in the first step) in the detected region. We define a quantization function as S_{ T }(∙)with quantization step T as follows.
Let F and $\widehat{F}$ be transform and inverse transform respectively. We perform wave atom transform to input frame Y and get coefficient matrix C=F(Y). C_{ s }is the set of coefficients in the region corresponding to vertical signal. The scale level of C_{ s }corresponding to the first and the second level. Figure 5b gives an example of 3level decomposition and the region of C_{ s } as the red boxes shows.
We compute the quantized residual ${S}_{T}\left(\widehat{F}\right({\mathbf{C}}_{s}\left)\right)$ and restore scratch in spatial domain by the following procedure
where ⊙ denotes elementwise multiplications. With the help of M and quantization, we can effectively avoid the ringing effect. The mask M ensure that the only those regions with scratch artifacts will be restored while other regions can be passed, thus protected well. Although M may carry some texture and edge regions besides scratches, the directionality selection of wave atom transform ensures only those pixels along vertical direction will be modified. The quantization function in (3.5) further reduces ringing effect by setting the small oscillatory tails to zero.
Meanwhile, we can incorporate softthresholding denoising process into our iterative framework. Denote the soft shrinkage function as Thres_{ σ }(∙), as following shows
Here we use the conclusion in [4] and define ${\delta}_{\sigma}\in [0,\sqrt{2{\sigma}^{2}logN}]$, N is the size of signal. Based on the assumption of noise mentioned in Signal model of movie section, the variance σ^{2} can be estimated according to the residual between adjacent frames after motion compensation. In the first iteration, $\left(\right)close="">{\delta}_{\sigma}^{1}$ is set as the upper bound of the interval. To the i th iteration, $\left(\right)close="">{\delta}_{\sigma}^{i}=0.8{\delta}_{\sigma}^{i1}$. This strategy ensures the convergence of residual signal X^{i}−X^{i−1}during iteration. When $max{\mathbf{X}}^{i}{\mathbf{X}}^{i1}<3$, the iteration is stopped. Under such stop criterion, we find that 48 iteration can achieve excellent performance for scratch restoration.
The algorithm of the nested thresholding scheme for joint descratching and deblotching is summarized in Algorithm 1. One of the contributions of our algorithm is reducing the dependence of the scratch restoration results on the accuracy of detection. A dilation operation will enlarge the region of scratch in the detection mask, which may introduce positive false into M. However, our method is robust to the enlarging of detection mask. The residual quantization step in our algorithm can reduce the influence of sidelobes. As a result, even if the scratched region is enlarged by dilation operation, the sidelobes can still be suppressed by iterative quantization of residual, which will not influence on visual effect seriously.
Algorithm 1. The nested thresholding scheme
Task:
Denoising and descratching restoration for Y.
Initialization:
Maximum iteration number I,
initial result X^{0}=Y,
quantization step T,
estimation of variance of noise σ^{2}.
Process:
Get mask matrix M=Dilation(Sobel(Y)).
In the i th iteration, i>0
C^{i}=F(X^{i−1});
${\mathbf{X}}^{i}={\mathbf{X}}^{i1}\mathbf{M}\odot {S}_{T}\left(\widehat{F}\right({\mathbf{C}}_{s}^{i}\left)\right)$;
$T=\frac{T}{i}$;
${\mathbf{X}}^{i}=\widehat{F}\left({\mathrm{Thres}}_{\sigma}\right(F\left({\mathbf{X}}^{i}\right)\left)\right)$.
If $max{\mathbf{X}}^{i}{\mathbf{X}}^{i1}<3$ or i=I
$\stackrel{~}{\mathbf{Y}}={\mathbf{X}}^{I}$.
Else i=i + 1.
Because our algorithm only processes the signal registered by detection mask, it is robust to the positive false of detection. In other words, even if an unscratched area is detected as scratch, the proposed method can still protect it. But if a scratch is not detected, the proposed method cannot restore it. For reducing the risk of negative false, we think the dilation operation is necessary, which ensures the robustness of our method.
To reduce the negative influence of negativefalse, we apply an adaptive dilation operation row by row. To each row of M, the detected scratched region is registered as continuous nonzero pixels. Generally, the longer the length of the continuous nonzero pixels is, the smaller size of dilation operator we should use. Based on this principle, we adaptively select the size of dilation operation according to Table 1.
Figure 6 gives an illustration of the restoration process for a row from an image with scratches, where it can found that with the increase of iteration, the scratch artifact can be restored without introducing serious ringing effect.
Blotch detection and restoration
Blotch detection
Recall that blotch is modeled as a multiplicative noise in our degradation model. Motion estimation can be used for blotch detection because the interframe residual of motion estimation in the location of blotch will be much higher than other regions. Currently, the general blotch detection method is using ranked order difference (ROD) detector. In [24], a twostage simple ROD detector (twostage SROD) is proposed, which measures the interframe discontinuity of blotch in the 3×3patches of three successive frames. However, abrupt motion of image objects can also cause large residual if the size of the search window in motion analysis is not large enough. On the other hand, toolarge search window size will increase the computational complexity significantly. So, merely taking advantage of the interframe residual is insufficient for blotch localization.
To detect blotch robustly, we will also exploit the intraframe information. Although both blotch and motion object have large interframe residual, we can discriminate them according to their intraframe features. Based on the analysis of Signal model of movie section, we know that blotch artifact is incoherent to its neighbor pixels. On the other hand, the pixels of the moving objects are more likely to have similar values in the neighborhood. We define the pixel in the i th row, j th column and t th frame as Y_{ ijt }. P_{ ijt }∈R^{L×L} is a patch with the center at Y_{ ijt }. L is the size of the patch. Calculating the residual between the patches in adjacent frames, we can get Res_{ ijt }=P_{ ijt }−P_{ij,t−1}. If there are pixels in P_{ ijt } belonging to blotch, they should satisfy Res_{ ijt }(a,b)>t_{1}, where t^{1} is a threshold and a, b is the coordinates of pixel in P_{ ijt }. We can get a mask M_{1}as following shows.
M_{1}chooses the candidates of blotch according to the interframe discontinuity of blotch. For getting a more accurate estimation, we should take intraframe discontinuity into consideration. So, we define E∈R^{L×L} as the residual among Y_{ ijt }and its neighborhood pixels as follows.
Similarly, if there are pixels in P_{ ijt } belonging to blotch, they should satisfy E(a,b)>t_{2}, where t^{2} is another threshold. We can get another mask M_{2}as following shows.
Only the pixels satisfying M_{2}(a,b)=M_{1}(a,b)=1are considered as blotch. After getting the primary detection result, a dilation operation is used to suppress the negative false of detection further.
It should be noted that if the blotch has almost uniform luminance value, M_{2} will contain the pixels of the boundary of the blotch rather than the whole region of the blotch. Generally, blotches having almost uniform luminance value are likely to have small areas. After finding their boundaries in M_{2}, we can detect their whole regions by dilation operation. On the other hand, if a blotch has large area, its luminance value is generally not uniform. In such situation, we can still detect blotch with the help of M_{1} and M_{2}.
Our method can be regarded as an extension of the work in [13, 24]. The improvement of our method is that we not only consider the interframe discontinuity of blotch but also consider the intraframe discontinuity. In [24], there is only one threshold in twostage SROD controlling the criterion of interframe discontinuity. The principle of the Bayesian method in [13] is also taking advantages of interframe discontinuity. In our method, we use two thresholds t^{1} and t^{2} to control the criterions of inter and intraframe discontinuity respectively. Because introducing intraframe information of blotch, our method is more robust to pathological motion [13] and occlusions in the movie.
Figure 7 shows an example of blotch detection, where it can be found that using intraframe discontinuity property, the detection is robust to large motion. In this algorithm, t^{1} and t^{2} are thresholds for interframe and intraframe discontinuity. Note that t^{2} is related to the search window size L. Generally, with the increase of L, intraframe coherence of arbitrary two pixels in the window becomes weaker, so the value of t^{2} should decrease. Too large t^{1} and t^{2} will increase the risk of falsenegative response while too small thresholds may cause a falsepositive response. In our experiments, we choose L=18, t_{2}=2, and t_{1}=15empirically based on the experimental results of over 4,000 frames. A dilation operation [29] is used to the detection result for improving the accuracy. The dilation operator is a square with size of 5×5 for the test frame with size of 1024×1024. From Figure 7, we can find that our detection result is good enough for a computationally efficient algorithm.
Blotch restoration
For blotch restoration of the old movie, we can take advantages of both inter and intra frame information with a nonparametric kernel regression based approach. For a missing pixel value y, its temporal and spatial context is denoted as a vector form x. We can approximate y using conditional expectation as
To get y=E(YX=x) we have to compute the joint probability P(y_{ i }x_{ i }) and the marginal probability P(x_{ i }), without presuming any parametric relationship between y_{ i }and x_{ i }. Kernel regression is a widely used nonparametric technique to estimate probability functions [32, 33]. With kernel regression, we have
where the kernels ${K}_{\Delta}(\circ )=\frac{1}{\Delta}K\left(\frac{\circ}{\Delta}\right)$ can be chosen from functions that are nonnegative, sum to 1 and symmetric around 0. h and g are called bandwidth of the kernels. Widely used kernel function includes the triangle kernel, Gaussian kernel, Epanechnikov kernel among others. Note that the product kernels rather than multivariate kernels are used in (4.6) to simplify the analysis. See [32] for more details.
Substituting $\widehat{P}\left({\mathbf{x}}_{i}\right)$ and $\widehat{P}({y}_{i},{\mathbf{x}}_{i})$ into (4.4), we have
Now we have to deal with $\int {y}_{i}{K}_{g}(\parallel {y}_{i}{y}_{j}\parallel )d{y}_{i}$. By defining a residual term as y^{′}=y_{ i }−y_{ j }, and omitting the bandwidth parameter g for simplicity, we have
where we used the properties of the kernel function that it sums to 1 and is symmetric around the origin. Letting (4.8) into (4.7), we have
The nonparametric conditional expectation estimator in (4.9) is known as the Nadaraya–Watson (NW) estimator, which is weighted average of the observations {y_{1},y_{2},…,y_{ n }} with the weight in inverse proportion to the distances between x_{ j },1≤j≤n and x_{ i }. For higher accuracy of the approximation, normalization is recommended and we can rewrite (4.9) as following shows
where $\widehat{\mathbf{x}}$ is the normalized vector of x and δ_{ j } is the ratio between the l^{2} norm of x_{ j } and that of x_{ i }. The kernel we choose is Gaussian kernel $\left(\frac{exp\left({(\bullet /h)}^{2}\right)}{h\sqrt{\Pi}}\right)$, where h is the bandwidth of the kernel. The adaptive bandwidth selection for the kernel is based on cross validation. By minimizing the following cost function, we get the optimized bandwidth.
where ${\u0177}_{j}$ is the estimated pixel value of y_{ j } without using the j th context x_{ j }. Because the cross validation method is computationally very demanding, we only take the neighbors of y_{ i } into consideration and set n≤9in (4.11).
Implementation details and experimental results
The diagram of the proposed movie restoration system is given in Figure 8. The system includes three modules: parameter estimation, joint scratch and noise removal, and blotch restoration. And the implementation details will be introduced in this section.
Noise estimation and denoising
We estimate the variance of noise using motion estimation. For each patch in the current frame, we search the patch with the shortest Euclidean distance in the preceding frame to get the motion vector and the residual patch. For simplicity, we employ the threestep motion estimation method [34] in this research. Define P_{ i } and P_{ j } as original patches, W_{ i } and W_{ j } the i.i.d. Gaussian random variables obeying N(0,σ^{2}). It is obvious that ${W}_{i}{W}_{j}\backsim N(0,2{\sigma}^{2})$. Moreover, according to the spatiotemporal stationary assumption of the noise in the movie, P_{ i }−P_{ j } has Laplacian distribution Laplace(0,b)[29]. So, we can estimate the variance of noise with the residual images from the sequential frames.
Compared with other professional noise estimator, such as the methods in [35, 36], the proposed estimator has lower computational complexity, which can be achieved realtime. Considering the large volume of data in the movie restoration system, the complexity of the algorithm should be limited. Moreover, the methods in [35, 36] focus on estimate the noise in the single image while the proposed method takes advantage of the interframe continuity of movie signal.
For testing the performance of the proposed method, we select six test sequences from the database http://sp.cs.tut.fi/mobile3dtv/stereovideo/ and add Gaussian noise to them. After applying different estimators to these sequence, we give comparison results of different methods on the accuracy of noise estimation in Table 2. We can find that the proposed method achieves a comparable even better performance on noise estimation.
Based on the noise estimation result, we can apply wave atom based denoising algorithm proposed in Nested thresholding in wave atom domain section. Figure 9 gives a comparison among original video frames, the degraded frames and denoising results. After performing the proposed denoising method, we can improve the visual effect of the frames greatly.
Scratch restoration
We compare our algorithm with some existing scratch restoration algorithms. The initial quantization step is 8. The four test movies are from China Film Archive, namely “TAO LI JIE”, “FANG ZHEN ZHU”, “DA LU” and “GUAN LIAN ZHANG”.
Figure 10 shows the experimental results of scratch removal (without blotch restoration) by different algorithms, including BM3D, nonlocal means restoration [6], MCA inpainting [37, 38], the method in [12, 15] (we used the method proposed in [15] to detect scratch, after that, we use the method in [12] to restore the degraded images), the simple wave atom coefficients pruning based method mentioned in Joint descratching and denoising section and our proposed algorithm.
It can be found that BM3D is only effective on noise removal. Both nonlocal means and MCA tend to produce oversmoothed results and remove some important details in the images (see the area of human eyes). The method in [15] gives falsenegative detection when on light vertical scratches, so that nonlocal means, MCA and the method in [12] have the risk of falsenegative judgment. On the other hand, our proposed scratch removal algorithm based on wave atom transform does not closely depend on accurate detection. The naive coefficient pruning method mentioned in Joint descratching and denoising section causes obvious ringing effect. The proposed algorithm overcomes the drawback and gets the best restoration results in our test.
Objective quality evaluation
We have also designed an objective evaluation test for the proposed algorithm. First, we select five test sequences from the database http://sp.cs.tut.fi/mobile3dtv/stereovideo/ and another four normally used test images as original data. Then we extract some frames with scratches from the beginning of old movies. After removing the mean of those scratched frames, we add the residual to the test sequences. As a result, we have both the original clean sequences and relevant degraded sequences, so we can calculate PSNR as an objective quality measure. We compared our algorithm with other restoration algorithms, including the nonlocal means, MCA and BM3D. Table 3 lists the PSNR comparison of these algorithms. Figure 11 gives some visual results as well. From the results, we can find that our algorithm outperforms other methods under most conditions. Note that for a faithful test, scratches are detected by the method in [15] and blotches are detected by the method in the tests before inpainted by the algorithms of nonlocal means or MCA.
Besides PSNR, another objective image quality assessment method called structural similarity indexing (SSIM) [39, 40] is used to evaluate the performance of the proposed method. Similarly, the quality index of our results is better in most situations, as Table 4 shows.
The reason for the outstanding performance of the proposed method can be summarized as follows. (1) As we have said that our method is robust to the positive false in the detection phase. The region not belonging to scratch and blotch can be protected better than other methods.
(2) The inpainting based restoration method abandons the information hidden by scratch and blotch. It estimates original signal merely according to the neighborhood of the signal. When the region of scratch or blotch is too large, this method can not get enough prior knowledge from the neighborhood of signal, which makes the result of restoration oversmooth. In the proposed method, instead of abandoning the information of original signal totally, we just suppress the interference added on the signal as much as possible. This strategy is better than inpainting based method when it comes to serious degraded image. Figure 12 gives an illustration of the advantage of our method. Because we only apply inpainting algorithm to blotches, to the same frame, the region waiting for inpainting in our method is much smaller than that of other method.
Overall performances of descratching, denoising and deblotching in old movie
Results of the proposed system on practical data are shown in Figure 13, 14, 15 and 16. In Figure 13, each subfigure reports the result from the step marked in Figure 8. The size of test frames is 1024×1024. We can find that the proposed system can produce outstanding results even to seriously damaged movies.
Our proposed system also has the advantage of high computational efficiency. On the platform of MATLAB with Inter Core2 CPU and with 3 Gb memory, the processing speed of our algorithm is on average 60.317 s per frame. In fact, compared with other existing methods, such as those based on pure nonparametric inpainting for both scratch and blotch, MCA inpainting and the method in [13], our method is much faster on the same hardware and software platform. Table 5 shows the average CPU time for processing a test frame of the testing algorithms. In Figure 16, we test our system on four different movie segments and give three consecutive frames and corresponding restoration results respectively.
The high efficiency of our proposed system can be explained by the following facts.

(1)
In scratch restoration stage, we do not have to perform complex detection, which is a large part of total time.

(2)
Wave atom transform is based on FFT and IFFT, whose complexity is O(N logN), so the complexity of our nestedthresholding method is only O(IN logN)with I the iteration number, which is much lower than wavelet based methods.

(3)
Our empirically test shows that the iterative thresholding algorithm converges fast and the iteration number we need is less than 10, whereas the iteration number of MCA can be hundreds times and even more.

(4)
It is noted that inpainting is the most computationally expensive step in our movie restoration system, whose complexity is about O(N L ^{2}), where N is the number of pixel to be inpainted and L ^{2}is the number of pixel in the neighborhood. Taking scratch out of nonparametric inpainting step can further reduce the computation time.
The average time cost of each phase and their percentage in the total processing time in our system is given in Table 6. The blotch inpainting phase costs most of the processing time while the denoising and descratching phases occupy much less processing time.
It should be mentioned that although our system in Figure 8 shows an obvious serial type of process manner for a single frame, it can be highly paralleled on GPU for the processing of movie sequence due to the following observations.

(1)
wave atom transform and inversetransform is blockbyblock, which can be done with a high degree of parallelism.

(2)
The kernel regression based inpainting process using only local data can also be conducted in parallel.
Therefore, our proposed system already has lower complexity as compared to the existing competitors, and it is expected by facilitating GPU and other parallel computing techniques in practical implementation.
Conclusions and future work
In this article, we propose an automatic system for movie restoration. Based on the analysis of features of noise, scratch and blotch artifacts in the damaged movie, we combine a nested thresholding method in wave atom domain with nonparametric model towards effective restoration. In scratch restoration stage, we use wave atom transform to achieve joint scratch detection, restoration and denoising. In blotch restoration stage, we take full advantages of inter and intraframe discontinuity and use nonparametric inpainting.
Although our work gets outstanding performances, there still exists some problems for further investigation in future work. In our model, blotch is modeled as multiplicative noise, so that the parameter q is binary. However, in practical situation, some blotches are semitransparent in the movie. How to design a more practical model for the degradation of movie is one of our future works. On the other hand, the values of some parameters in our system are empirical.

(1)
Based on the experimental results of four films from Chinese film archive (over 4,000 frames), the configuration of L, t ^{1} and t ^{2} in deblotching step achieves satisfying results for blotch restoration. However, we still need to investigate more evidence to support the configuration of them.

(2)
In descratching and denoising phase of our system, the quantization step T, the stop criterion and the multiplicative constant of the threshold value in the denoising step are fixed. We think that the selection of T may have a strong relationship with the intensity of scratch in the movie. For example, the value of T may not be global. To different regions, the quantization step can be adaptive according to the maximum value of the scratch in the region. According to noise level and image content, the stop criterion and the multiplicative constant of the threshold value in the denoising step should be adaptive as well. In this article, they are fixed to be 3 and 0.8, respectively. We think that the estimation of the variance of noise can provide the selection of them with a useful prior knowledge. It is reasonable to think that the values of them should be increase with the variance of noise. In summary, establishing an adaptive strategy for the choice of these parameters will be one of our future work.
References
 1.
Donoho DL: Denoising by softthresholding. IEEE Trans. Inf. Theory 1995, 41(3):613627. 10.1109/18.382009
 2.
Chang SG, Yu B, Vetterli M: Adaptive wavelet thresholding for image denoising and compression. IEEE Trans. Image Process 2000, 9(9):15321546. 10.1109/83.862633
 3.
Figueiredo MAT, Nowak RD: An EM algorithm for waveletbased image restoration. IEEE Trans. Image Process 2003, 12(8):906916. 10.1109/TIP.2003.814255
 4.
Luisier F, Blu T, Unser M: A new SURE approach to image denoising: interscale orthonormal wavelet thresholding. IEEE Trans. Image Process 2007, 16(3):593606.
 5.
Bertalmio M, Caselles V, Pardo A: Movie denoising by average of warped lines. IEEE Trans. Image Process 2007, 16(9):23332347.
 6.
Buades A, Coll B, Morel J: Nonlocal image and movie denoising. Int. J. Comput. Vis 2008, 76(2):123139. 10.1007/s1126300700521
 7.
Bose N, Ahuja N: Superresolution and noise filtering using moving least squares. IEEE Trans. Image Process 2006, 15(8):22392248.
 8.
Dabov K, Foi A, Katkovnik V, Egiazarian K: Image denoising by sparse 3D transformdomain collaborative filtering. IEEE Trans. Image Process 2007, 16(8):20802095.
 9.
Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A: Nonlocal sparse models for image restoration. IEEE 12th International Conference on Computer Vision, (Kyoto, Japan, 2009), pp. 2272–2279
 10.
Shao L, Zhang H, de Haan G: An overview and performance evaluation of classificationbased least squares trained filters. IEEE Trans. Image Process 2008, 17(10):17721782.
 11.
Shao L, Wang J, Kirenko I, de Haan G: Quality adaptive least squares filters for compression artifacts removal using a noreference bloc visibility metric. J. Vis. Commun. Image Represent 2011, 22: 2332. 10.1016/j.jvcir.2010.09.007
 12.
Kokaram AC: Removal of line artefacts for digital dissemination of archived film and video. IEEE International Conference on Multimedia Computing and Systems, (Florence, Italy, 1999), pp. 245–249
 13.
Kokaram AC: On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach. IEEE Trans. Image Process 2004, 13(3):397415. 10.1109/TIP.2004.823815
 14.
Bruni V, Vitulano D, Kokaram A: Line scratches detection and restoration via light diffraction. Proceedings of the 3rd International Symposium on Image and Signal Processing and Analysis, (2003), pp. 5–10
 15.
Bruni V, Vitulano D: A generalized model for scratch detection. IEEE Trans. Image Process 2004, 13: 4450. 10.1109/TIP.2003.817231
 16.
Nadenau MJ, Mitra SK: Blotch and scratch detection in image sequences based on rank ordered differences. TimeVarying Image Processing and Moving Object Recognition, (Florence, Italy, 1997), pp. 27–35
 17.
Haindl M, Filip J: Fast restoration of colour movie scratches. Proc. IEEE International Conference on Pattern Recognition, (Quebec, Canada, 2002), pp. 269–272
 18.
Joyeux L, Boukir S, Besserer B: Film line scratch removal using Kalman filtering and Bayesian restoration. Fifth IEEE Workshop on Applications of Computer Vision, (Palm Springs, America, 2000), p. 8
 19.
Maddalena L, Petrosino A: Restoration of blue scratches in digital image sequences. Image Vis. Comput 2008, 26(10):13141326. 10.1016/j.imavis.2006.04.013
 20.
Muller S, Buhler J, Thebault C, Doser I, Neisse O: Scratch detection supported by coherency analysis of motion vector fields. 16th IEEE International Conference on Image Processing (ICIP), (Cairo, Egypt, 2009), pp. 89–92
 21.
Stanco F, Ramponi G, Tenze L: Removal of semitransparent blotches in old photographic prints. In 5th COST 276 Workshop,. Prague, Czech; 2003.
 22.
Stanco F, Ramponi G, de Polo A: Towards the automated restoration of old photographic prints: a survey. EUROCON 2003. Computer as a Tool. The IEEE Region 8, vol. 2, (Ljubljana, Slovenia, 2003), pp. 370–374
 23.
Licsar A, Czuni L: Trainable postprocessing method to reduce false alarms in the detection of small blotches of archive films. IEEE International Conference on Image Processing, vol. 2, (Genoa, Italy, 2005), pp. 562–565
 24.
Gullu MK, Urhan O, Erturk S: Blotch detection and removal for archive film restoration. AEUInt. J. Electron. Commun 2008, 62(7):534543. 10.1016/j.aeue.2007.07.010
 25.
Kokaram AC: Advances in the detection and reconstruction of blotches in archived film and video. IEE Seminar on Digital Restoration of Film and Video Archives, (London, UK, 2001), pp. 1–7
 26.
Demanet L: Curvelets, wave atoms, and wave equations,Thesis of California Institute of Technology. 2006.
 27.
Demanet L, Ying L: Wave atoms and sparsity of oscillatory patterns. Appl. Comput. Harmon. Anal 2007, 23(3):368387. 10.1016/j.acha.2007.03.003
 28.
Selesnick I, Baraniuk R, Kingsbury N: The dualtree complex wavelet transform. IEEE Signal Process. Mag 2005, 22(6):123151.
 29.
Gonzalez RC, Woods RE: Digital Image Processing. (Publishing House of Electronics Industry, Prentice Hall, Beijing, 2002)
 30.
Nezhadarya E, Ward RK: A new scheme for robust gradient vector estimation in color images. IEEE Trans. Image Process 2011, 20(8):22112220.
 31.
Otsu NA: A threshold selection method from graylevel histograms. IEEE Trans. Syst. Man Cybern 1979, 9: 6266.
 32.
Hardle W: Nonparametric and Semiparametric Models. Springer Series in Statistics (Springer, New York, 2004)
 33.
Takeda H, Farsiu S, Milanfar P: Kernel regression for image processing and reconstruction. IEEE Trans. Image Process 2008, 16(2):349366.
 34.
Jing X, Chau LP: An efficient threestep search algorithm for block motion estimation. IEEE Trans. Multimed 2004, 6(3):435438. 10.1109/TMM.2004.827517
 35.
Zoran D, Weiss Y: Scale invariance and noise in natural images. IEEE 12th International Conference on Computer Vision, (Kyoto, Japan, 2010), pp. 2209–2216
 36.
Tang C, Yang X, Zhai G: Dualtransform based noise estimation. IEEE 12th International Conference on Multimedia Expo, (Melbourne, Australia, 2012), accepted
 37.
Elad M, Starck JL, Querreb P, Donoho D: Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal 2005, 19(3):340358. 10.1016/j.acha.2005.03.005
 38.
Fadili JM, Starck JL, Elad M, Donoho D: Mcalab: reproducible research in signal and image decomposition and inpainting. IEEE Des. Test 2010, 12: 4463.
 39.
Wang Z, Sheikh HR, Bovik AC: Objective Video Quality Assessment. The Handbook of Video Databases: Design and Applications. (CRC Press, Boca Raton, 2003), pp. 1041–1078
 40.
Wang Z, Bovik A, Sheikh H, Simoncelli E: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process 2004, 13(4):600612. 10.1109/TIP.2003.819861
Acknowledgements
This article was supported in part by the NSFC (60932006, 61025005, 61001145, 61102098), RFDP (20090073110022), postdoctoral foundation of China 20100480603, 201104276, postdoctoral foundation of Shanghai 11R21414200 and the 111 Project (B07022).
Author information
Affiliations
Corresponding author
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Xu, H., Zhai, G., Chen, L. et al. Automatic movie restoration based on wave atom transform and nonparametric model. EURASIP J. Adv. Signal Process. 2012, 132 (2012). https://doi.org/10.1186/168761802012132
Received:
Accepted:
Published:
Keywords
 Movie restoration
 Wave atom transform
 Nested thresholding
 Nonparametric inpainting