This article presents a new scheme for movie de-scratching, de-noising and de-blotching based on wave atom transform and nonparametric model. According to the analysis of the noise, scratches and blotches in the film, we establish a hybrid model for movie signal. In the proposed model, we model scratch as directional additive noise, which can be represented effectively by a new multi-scale analysis tool called wave atom transform. Based on wave atoms, de-noising and de-scratching are combined in a nested thresholding framework which can suppress both homogeneous noise and structural artifacts. On the other hand, blotch is described as multiplicative noise in the model, which can be restored by a “detecting and inpainting” strategy. We use the inter- and intra-frame discontinuity of blotch to detect them and then restore them by kernel regression method. Combining the processing operations above together, we design an effective automatic movie restoration system. Experimental results show that the proposed method not only achieves excellent performances even for seriously damaged movies but also has low computational complexity compared with the existing methods.

Introduction

Movie restoration is a problem with both academic challenge and great practical value. Due to long display time and undesirable storage environment, movies are damaged by various kinds of artifacts such as noise, loss of contrast, scratch, blotch, etc. These artifacts have negative influences on the artistic and commercial values of movie. For recent years, many efforts have been devoted to movie restoration, especially to de-noising, de-scratching and de-blotching problems.

In the literature, wavelet domain soft-thresholding based algorithms [1–4] have shown good performances in image de-noising. For those methods, the selection of threshold is of vital importance. Although Donoho [1] gives the universal upper bound of threshold and researchers try to achieve even better solutions in [2, 4], performances of these algorithms largely hinge on accurate noise variance estimation. An expectation-maximization (EM) algorithm for noise estimation is proposed in [3], but its computational complexity is very high. Besides transform-based methods mentioned above, non-parametric and parametric model based algorithms [5–7] also achieve good de-noising results. Currently, the methods combining parametric and nonparametric model achieve state-of-the-art results in image and video de-noising, such as the BM3D in [8] algorithm and the training-based sparse de-noising method in [9]. Both of them take advantages of self-similarity of the patches in the image and represent the image sparsely by a dictionary. However, although those algorithms are effective to remove Gaussian noise, they do not work well for structural artifacts.

Recently, some algorithms for structural artifacts removal are proposed. For example, in [10, 11], a training-based adaptive least square filter is designed to remove the coding artifacts in the image. Many de-scratching and de-blotching algorithms have also been proposed in the literature [12–15]. A joint scratch and blotch restoration method using Markov Random Field (MRF) model was presented in [16]. However, this method is not robust enough for movies with large motion. The methods in [12, 17–20] exploit inter-frame information in the Bayesian framework for scratch detection and restoration. Because of using the directionality of scratch in the detection phase, these method cannot be extended to blotch restoration straightforwardly. In our opinion, because scratch has strong directionality whereas blotch is isotropic, scratch and blotch should be restored separately. How to restore them respectively to achieve better result is a problem we try to solve. On the other hand, a common point of all these methods is that all of them use sequential detection and restoration steps, thus the overall performance will be limited by the accuracy of the detection stage, which is often very ad hoc in nature. How to reduce the dependence of restoration result on the accuracy of detection is another interesting problem.

The contribution of this work includes three folds. Firstly, according to the analysis of the scratches in the film, we model scratch as directional additive noise which can be represented effectively by wave atoms. Secondly, inspired by soft-thresholding de-noising method, we combine de-noising and de-scratching in a nested thresholding framework based on wave atom transform, which is robust to the performance drift of detection step. Thirdly, after de-noising and de-scratching, we detect blotches by the inter- and intra-frame discontinuity and restore them by kernel regression. Because scratches have been restored before, the computational complexity caused by applying kernel regression to de-blotching.

The article is organized as follows. In Signal model of movie section, we establish a hybrid model of movie signal and analyze the properties of noise, scratch and blotch in the movie. In Joint de-scratching and de-noising section, we propose a nested thresholding method in wave atom domain, which can be used for de-noising and de-scratching. In Blotch detection and restoration section, we design a robust blotch detection and restoration algorithm based on motion estimation and nonparametric kernel regression. The integrated system and experimental results are given in Implementation details and experimental results section. Conclusions and future work section concludes this article.

Signal model of movie

Hybrid noise model

We model movie signal by a hybrid noise model, as shown in (2.1).

where X is the ideal pixel value of video signal and Y is the noisy observation. N is homogeneous additive noise and S is the scratch component. B is the blotch component. q is a random variable controlling the transparency of blotch, and q∈[0,1] generally. If B is regarded as additive noise, (2.1) becomes a pure additive noise model. However, because B has not homogeneous property, the additive noise model increases the difficulty of analysis and restoration. For simplification, we restrict q∈{0,1}. Based on the binarization of q, B is modeled as multiplicative noise and (2.1) becomes a hybrid noise model.

This hybrid model is different from the model proposed in [21–24], which regards scratch as multiplicative noise. This change is based on both physical and chemical causes of scratch. Scratch results from the shedding of coats on the film. As long as the contact surface between the projector and the film is not clean, scratches will appear in the film during displaying. We can model the phenomenon as an additive process, i.e. the film coat becomes thinner damaged by those foreign matters. On the other hand, blotch artifacts can be caused by mildews, dirty spots or the shedding of film coats. The spatial and temporal discontinuity of blotch makes it suitable to be modeled with the multiplicative noise model. The hybrid model guides us restore them by different methods.

The analysis of noise, scratch and blotch

Generally, distribution of N in our model (2.1) can be modeled as Gaussian noise. However, since the variance σ^{2} is not known beforehand, soft-thresholding based de-noising algorithm cannot be used directly. For simplifying the problem, we make the following assumption: the noise in the movie is homogeneous, which obeys N(0,σ^{2}) in the spatiotemporal domain. In other words, the noise in the movie is assumed to be stationary in the spatiotemporal domain, which can be estimated by the residual signal of successive frames.

Considering its generative mechanism, scratch is a kind of anisotropic sudden interference, usually along an approximate vertical direction. It means that S is a directional noise. Further, scratch can be classified into two types according to its inter-frame continuity, as exemplified in Figure 1. In Figure 1a, we can find that the first type of scratch is relatively “weak”, which is short and narrow and does not have inter-frame continuity. This kind of scratch can be detected straightforwardly by motion estimation based method. Figure 1b shows another type of scratch whose location in adjacent frames is almost stable. Motion estimation based method cannot be useful for the second type of continuous scratch.

Because of the variety of scratch, the detection method based on either inter- or intra-frame discontinuity is sensitive to the choice of parameters. For detecting scratch more accurately, Bruni proposed a damped sinusoid model for scratch in [15], which is useful to detect isolated and serious scratch. Bruni’s model reveals the oscillatory property of scratch—to each individual scratch, it can be described as a damped sinusoid signal. In fact, this microscopic view can be extended to macroscopic situation. Because scratch can accumulate during multiple times of film projection, not only does the individual but also the whole scratches in the frame obey oscillatory pattern. As shown in Figure 2, we choose several frames from some old movies to investigate the pattern of scratch. It can be found that scratches in the film have local oscillatory patterns, and such pattern would change from a local phenomenon to a global one with the increase of the displaying time.

Bruni’s model aims to solve the detection problem only while the proposed model provides a guidance to design a restoration algorithm directly. Based on the analysis above, we can find that additivity, directionality and local (even global) oscillation are three features of scratch. According to the intrinsic properties, wave atom-based method matches this problem perfectly, which will be addressed with details in Joint de-scratching and de-noising section.

Different from scratch, blotch artifact shows the isotropic pattern in most situations. Generally, it does not have the spatiotemporal continuity between and among movie frames. The blotch artifact can be very bright, very dark or even semi-transparent, as instanced in Figure 3. All of these properties increase the difficulty of removing blotch [16, 21–25].

Joint de-scratching and de-noising

Overview of wave atom transform

Wave atom transform is developed firstly by Demanet and Ying [26, 27]. It is a variation of traditional wavelet aiming to overcome the weakness of wavelet in representing the directionality of high dimensional signals. The definition of wave atoms is closely related to the Fourier transform. Consider a 2D Fourier transform pair of f(x) and \widehat{f}\left(\mathit{\omega}\right), where x=(x_{1}x_{2}), ω=(ω_{1}ω_{2})are the coordinates in spatial domain and frequency domain, then wave atoms can be defined as φ_{
μ
}(x), where μ=(jmn)=(jm_{1}m_{2}n_{1}n_{2}) is a label representing the scale, direction and location of the corresponding wave atom in 2-D situation. The indexed point (x_{
μ
}ω_{
μ
}) in phase-space is defined as

It can be seen that the notations follow those of the wavelet and curvelet, where j controls the resolution scale, and m and n control the location in frequency and spatial domain, respectively. However, there are two differences between wavelet and wave atom. First, wavelets obey a linear scaling wavelength while wave atoms obey the parabolic scaling wavelength: at scale 2^{−2j} the essential frequency support is of size 2^{j}, and at frequency 2^{2j}, the essential time support is of size 2^{−j}. Second, the basis function of wave atom is different from that of conventional wavelet.

In the 1D situation, the frequency formula of basis function \left(\right)close="">\n \n \n \n \psi \n \n \n m\n \n \n 0\n \n \n (\n x\n )\n \n is

with ε_{
m
}=(−1)^{m} and {\alpha}_{m}=\frac{\Pi}{2}(m+\frac{1}{2}). g is a real-valued, C^{∞} bump function on a compact support with length 2Π, which promises the energy of basis function is normalized (\parallel {\psi}^{0}{\parallel}_{2}=1). Similarly, the 2D basis function of wave atom can be written as

The transform process of wave atom transform is similar to that of curvelet, which is based on FFT and IFFT. For a 2-D signal u∈R^{N×N}, we first perform a FFT of size N×N of u, and then get \widehat{\mathbf{u}}. For each pair (jm), we wrap the product {\widehat{\psi}}_{\mathbf{m}}^{j}\widehat{\mathbf{u}} by periodicity insert the interval [−2^{j}Π,2^{j}Π×[−2^{j}Π,2^{j}Π. We then compute inverse FFT of size 2^{j} of the result to get c_{
μ
}. We repeat the process above over each pair (jm)to get all the coefficients of wave atom transform. Finally, we permute these coefficients according to the label μ=(jmn) to get a matrix of coefficients. Please refer to [26, 27] for the details of wave atom transform.

Figure 4 gives some examples of wave atoms and their frequency representation. It can be found that wave atoms have outstanding performance in directional representation. This is one reason why wave atoms are used for scratch representation.

The features of wave atom transform to represent scratch

Recalling the analysis in “The analysis of noise, scratch and blotch” section, we know that scratch is a directional additive noise having an oscillatory pattern. Figure 5 illustrates the coefficient’s distribution of a straight-line image in wave atom, curvelet, wavelet and dual tree complex wavelet [28] domains after 3-level decomposition. All the four coefficient domains can be divided into several regions corresponding to different directions and scales. We can find that the coefficients corresponding to vertical, horizontal and diagonal lines are separated perfectly in wave atom domain. The coefficients in red boxes in Figure 5b,e,h,k correspond to vertical signal. An intuitive thresholding example is to remove the coefficients contained in red boxes in both domains. Figure 5c,f,i,l give the vertical line removal results. We find that under the same situation (removing the coefficients corresponding to the first two levels), the vertical line removal result using wave atom transform is clearly superior. It means that under the same decomposition level, the directionality of wave atom transform is better than three competitors.

Another feature of wave atom transform is its sparsity of representing oscillatory signals. In [27], wave atom transform has been proven to be able to achieve sparser representation of oscillatory signal than the existing transforms including wavelet, and curvelet. If O(N)wave atom coefficients suffice to represent an oscillatory signal to reasonable accuracy, O(N^{3/2}) curvelet coefficients or O(N^{2})wavelet coefficients will be needed to achieve the same accuracy.

The analysis above implies that if we decompose an image with scratches by wave atoms, the coefficients corresponding to scratches will be sparse and distribute in a narrow region in transform domain.

Nested thresholding in wave atom domain

We now consider how to threshold the coefficients of wave atoms so as to remove scratch and noise. The coefficients representing scratch will be limited in the narrow region corresponding to vertical component of signal, as shown in the red boxes in Figure 5b. A naive method that removes all the coefficients in this region will damage meaningful vertical image pixels rather than scratch. Another difficulty is to avoid ringing artifacts caused by the sidelobes of signal, which is common in wavelet shrinkage methods. To overcome these two problems, we propose a nested thresholding method.

We use a vertical 5×5Sobel detector to find where scratches are likely to appear and get a mask matrix M. To ensure an accurate scratch detection, morphological dilation operation [29] is applied to M. Besides Sobel detector, there exists many other effective edge and ridge detector, such as the method in [30]. However, considering the requirement for processing a large volume of data real-time, we apply the simple strategy.

The threshold of Sobel detector is critical to M. If the selected threshold is too low, it will not only generate false detection result of scratch; if the threshold is too high, then many of the scratches may not be detected or the detected scratches are too segmented. To make a trade off between computational complexity and robustness, we choose the threshold adaptively by Otsu threshold segmentation method [31]. According to the analysis below we will prove that the proposed method is robust to detection false.

Rather than classic shrinkage of the coefficients in transform domain, we propose to iteratively subtract a quantized residual from the reconstructed signal (or original signal in the first step) in the detected region. We define a quantization function as S_{
T
}(∙)with quantization step T as follows.

Let F and \widehat{F} be transform and inverse transform respectively. We perform wave atom transform to input frame Y and get coefficient matrix C=F(Y). C_{
s
}is the set of coefficients in the region corresponding to vertical signal. The scale level of C_{
s
}corresponding to the first and the second level. Figure 5b gives an example of 3-level decomposition and the region of C_{
s
} as the red boxes shows.

We compute the quantized residual {S}_{T}\left(\widehat{F}\right({\mathbf{C}}_{s}\left)\right) and restore scratch in spatial domain by the following procedure

where ⊙ denotes element-wise multiplications. With the help of M and quantization, we can effectively avoid the ringing effect. The mask M ensure that the only those regions with scratch artifacts will be restored while other regions can be passed, thus protected well. Although M may carry some texture and edge regions besides scratches, the directionality selection of wave atom transform ensures only those pixels along vertical direction will be modified. The quantization function in (3.5) further reduces ringing effect by setting the small oscillatory tails to zero.

Meanwhile, we can incorporate soft-thresholding de-noising process into our iterative framework. Denote the soft shrinkage function as Thres_{
σ
}(∙), as following shows

Here we use the conclusion in [4] and define {\delta}_{\sigma}\in [0,\sqrt{2{\sigma}^{2}logN}], N is the size of signal. Based on the assumption of noise mentioned in Signal model of movie section, the variance σ^{2} can be estimated according to the residual between adjacent frames after motion compensation. In the first iteration, \left(\right)close="">\n \n \n \n \delta \n \n \n \sigma \n \n \n 1\n \n \n \n is set as the upper bound of the interval. To the i th iteration, \left(\right)close="">\n \n \n \n \delta \n \n \n \sigma \n \n \n i\n \n \n =\n 0\n .\n 8\n \n \n \delta \n \n \n \sigma \n \n \n i\n \u2212\n 1\n \n \n \n. This strategy ensures the convergence of residual signal X^{i}−X^{i−1}during iteration. When max|{\mathbf{X}}^{i}-{\mathbf{X}}^{i-1}|<3, the iteration is stopped. Under such stop criterion, we find that 4-8 iteration can achieve excellent performance for scratch restoration.

The algorithm of the nested thresholding scheme for joint de-scratching and de-blotching is summarized in Algorithm 1. One of the contributions of our algorithm is reducing the dependence of the scratch restoration results on the accuracy of detection. A dilation operation will enlarge the region of scratch in the detection mask, which may introduce positive false into M. However, our method is robust to the enlarging of detection mask. The residual quantization step in our algorithm can reduce the influence of sidelobes. As a result, even if the scratched region is enlarged by dilation operation, the sidelobes can still be suppressed by iterative quantization of residual, which will not influence on visual effect seriously.

If max|{\mathbf{X}}^{i}-{\mathbf{X}}^{i-1}|<3 or i=I

\stackrel{~}{\mathbf{Y}}={\mathbf{X}}^{I}.

Else i=i + 1.

Because our algorithm only processes the signal registered by detection mask, it is robust to the positive false of detection. In other words, even if an unscratched area is detected as scratch, the proposed method can still protect it. But if a scratch is not detected, the proposed method cannot restore it. For reducing the risk of negative false, we think the dilation operation is necessary, which ensures the robustness of our method.

To reduce the negative influence of negative-false, we apply an adaptive dilation operation row by row. To each row of M, the detected scratched region is registered as continuous non-zero pixels. Generally, the longer the length of the continuous non-zero pixels is, the smaller size of dilation operator we should use. Based on this principle, we adaptively select the size of dilation operation according to Table 1.

Figure 6 gives an illustration of the restoration process for a row from an image with scratches, where it can found that with the increase of iteration, the scratch artifact can be restored without introducing serious ringing effect.

Blotch detection and restoration

Blotch detection

Recall that blotch is modeled as a multiplicative noise in our degradation model. Motion estimation can be used for blotch detection because the inter-frame residual of motion estimation in the location of blotch will be much higher than other regions. Currently, the general blotch detection method is using ranked order difference (ROD) detector. In [24], a two-stage simple ROD detector (two-stage SROD) is proposed, which measures the inter-frame discontinuity of blotch in the 3×3patches of three successive frames. However, abrupt motion of image objects can also cause large residual if the size of the search window in motion analysis is not large enough. On the other hand, too-large search window size will increase the computational complexity significantly. So, merely taking advantage of the inter-frame residual is insufficient for blotch localization.

To detect blotch robustly, we will also exploit the intra-frame information. Although both blotch and motion object have large inter-frame residual, we can discriminate them according to their intra-frame features. Based on the analysis of Signal model of movie section, we know that blotch artifact is incoherent to its neighbor pixels. On the other hand, the pixels of the moving objects are more likely to have similar values in the neighborhood. We define the pixel in the i th row, j th column and t th frame as Y_{
ijt
}. P_{
ijt
}∈R^{L×L} is a patch with the center at Y_{
ijt
}. L is the size of the patch. Calculating the residual between the patches in adjacent frames, we can get Res_{
ijt
}=|P_{
ijt
}−P_{ij,t−1}|. If there are pixels in P_{
ijt
} belonging to blotch, they should satisfy Res_{
ijt
}(a,b)>t_{1}, where t^{1} is a threshold and a, b is the coordinates of pixel in P_{
ijt
}. We can get a mask M_{1}as following shows.

M_{1}chooses the candidates of blotch according to the inter-frame discontinuity of blotch. For getting a more accurate estimation, we should take intra-frame discontinuity into consideration. So, we define E∈R^{L×L} as the residual among Y_{
ijt
}and its neighborhood pixels as follows.

Similarly, if there are pixels in P_{
ijt
} belonging to blotch, they should satisfy E(a,b)>t_{2}, where t^{2} is another threshold. We can get another mask M_{2}as following shows.

Only the pixels satisfying M_{2}(a,b)=M_{1}(a,b)=1are considered as blotch. After getting the primary detection result, a dilation operation is used to suppress the negative false of detection further.

It should be noted that if the blotch has almost uniform luminance value, M_{2} will contain the pixels of the boundary of the blotch rather than the whole region of the blotch. Generally, blotches having almost uniform luminance value are likely to have small areas. After finding their boundaries in M_{2}, we can detect their whole regions by dilation operation. On the other hand, if a blotch has large area, its luminance value is generally not uniform. In such situation, we can still detect blotch with the help of M_{1} and M_{2}.

Our method can be regarded as an extension of the work in [13, 24]. The improvement of our method is that we not only consider the inter-frame discontinuity of blotch but also consider the intra-frame discontinuity. In [24], there is only one threshold in two-stage SROD controlling the criterion of inter-frame discontinuity. The principle of the Bayesian method in [13] is also taking advantages of inter-frame discontinuity. In our method, we use two thresholds t^{1} and t^{2} to control the criterions of inter- and intra-frame discontinuity respectively. Because introducing intra-frame information of blotch, our method is more robust to pathological motion [13] and occlusions in the movie.

Figure 7 shows an example of blotch detection, where it can be found that using intra-frame discontinuity property, the detection is robust to large motion. In this algorithm, t^{1} and t^{2} are thresholds for inter-frame and intra-frame discontinuity. Note that t^{2} is related to the search window size L. Generally, with the increase of L, intra-frame coherence of arbitrary two pixels in the window becomes weaker, so the value of t^{2} should decrease. Too large t^{1} and t^{2} will increase the risk of false-negative response while too small thresholds may cause a false-positive response. In our experiments, we choose L=18, t_{2}=2, and t_{1}=15empirically based on the experimental results of over 4,000 frames. A dilation operation [29] is used to the detection result for improving the accuracy. The dilation operator is a square with size of 5×5 for the test frame with size of 1024×1024. From Figure 7, we can find that our detection result is good enough for a computationally efficient algorithm.

Blotch restoration

For blotch restoration of the old movie, we can take advantages of both inter- and intra- frame information with a nonparametric kernel regression based approach. For a missing pixel value y, its temporal and spatial context is denoted as a vector form x. We can approximate y using conditional expectation as

To get y=E(Y|X=x) we have to compute the joint probability P(y_{
i
}x_{
i
}) and the marginal probability P(x_{
i
}), without presuming any parametric relationship between y_{
i
}and x_{
i
}. Kernel regression is a widely used nonparametric technique to estimate probability functions [32, 33]. With kernel regression, we have

where the kernels {K}_{\Delta}(\circ )=\frac{1}{\Delta}K\left(\frac{\circ}{\Delta}\right) can be chosen from functions that are nonnegative, sum to 1 and symmetric around 0. h and g are called bandwidth of the kernels. Widely used kernel function includes the triangle kernel, Gaussian kernel, Epanechnikov kernel among others. Note that the product kernels rather than multivariate kernels are used in (4.6) to simplify the analysis. See [32] for more details.

Substituting \widehat{P}\left({\mathbf{x}}_{i}\right) and \widehat{P}({y}_{i},{\mathbf{x}}_{i}) into (4.4), we have

Now we have to deal with \int {y}_{i}{K}_{g}(\parallel {y}_{i}-{y}_{j}\parallel )d{y}_{i}. By defining a residual term as y^{′}=y_{
i
}−y_{
j
}, and omitting the bandwidth parameter g for simplicity, we have

The nonparametric conditional expectation estimator in (4.9) is known as the Nadaraya–Watson (NW) estimator, which is weighted average of the observations {y_{1},y_{2},…,y_{
n
}} with the weight in inverse proportion to the distances between x_{
j
},1≤j≤n and x_{
i
}. For higher accuracy of the approximation, normalization is recommended and we can rewrite (4.9) as following shows

where \widehat{\mathbf{x}} is the normalized vector of x and δ_{
j
} is the ratio between the l^{2} norm of x_{
j
} and that of x_{
i
}. The kernel we choose is Gaussian kernel \left(\frac{exp\left({(\bullet /h)}^{2}\right)}{h\sqrt{\Pi}}\right), where h is the bandwidth of the kernel. The adaptive bandwidth selection for the kernel is based on cross validation. By minimizing the following cost function, we get the optimized bandwidth.

where {\u0177}_{j} is the estimated pixel value of y_{
j
} without using the j th context x_{
j
}. Because the cross validation method is computationally very demanding, we only take the neighbors of y_{
i
} into consideration and set n≤9in (4.11).

Implementation details and experimental results

The diagram of the proposed movie restoration system is given in Figure 8. The system includes three modules: parameter estimation, joint scratch and noise removal, and blotch restoration. And the implementation details will be introduced in this section.

Noise estimation and de-noising

We estimate the variance of noise using motion estimation. For each patch in the current frame, we search the patch with the shortest Euclidean distance in the preceding frame to get the motion vector and the residual patch. For simplicity, we employ the three-step motion estimation method [34] in this research. Define P_{
i
} and P_{
j
} as original patches, W_{
i
} and W_{
j
} the i.i.d. Gaussian random variables obeying N(0,σ^{2}). It is obvious that {W}_{i}-{W}_{j}\backsim N(0,2{\sigma}^{2}). Moreover, according to the spatiotemporal stationary assumption of the noise in the movie, P_{
i
}−P_{
j
} has Laplacian distribution Laplace(0,b)[29]. So, we can estimate the variance of noise with the residual images from the sequential frames.

Compared with other professional noise estimator, such as the methods in [35, 36], the proposed estimator has lower computational complexity, which can be achieved real-time. Considering the large volume of data in the movie restoration system, the complexity of the algorithm should be limited. Moreover, the methods in [35, 36] focus on estimate the noise in the single image while the proposed method takes advantage of the inter-frame continuity of movie signal.

For testing the performance of the proposed method, we select six test sequences from the database http://sp.cs.tut.fi/mobile3dtv/stereo-video/ and add Gaussian noise to them. After applying different estimators to these sequence, we give comparison results of different methods on the accuracy of noise estimation in Table 2. We can find that the proposed method achieves a comparable even better performance on noise estimation.

Based on the noise estimation result, we can apply wave atom based de-noising algorithm proposed in Nested thresholding in wave atom domain section. Figure 9 gives a comparison among original video frames, the degraded frames and de-noising results. After performing the proposed de-noising method, we can improve the visual effect of the frames greatly.

Scratch restoration

We compare our algorithm with some existing scratch restoration algorithms. The initial quantization step is 8. The four test movies are from China Film Archive, namely “TAO LI JIE”, “FANG ZHEN ZHU”, “DA LU” and “GUAN LIAN ZHANG”.

Figure 10 shows the experimental results of scratch removal (without blotch restoration) by different algorithms, including BM3D, non-local means restoration [6], MCA inpainting [37, 38], the method in [12, 15] (we used the method proposed in [15] to detect scratch, after that, we use the method in [12] to restore the degraded images), the simple wave atom coefficients pruning based method mentioned in Joint de-scratching and de-noising section and our proposed algorithm.

It can be found that BM3D is only effective on noise removal. Both non-local means and MCA tend to produce over-smoothed results and remove some important details in the images (see the area of human eyes). The method in [15] gives false-negative detection when on light vertical scratches, so that non-local means, MCA and the method in [12] have the risk of false-negative judgment. On the other hand, our proposed scratch removal algorithm based on wave atom transform does not closely depend on accurate detection. The naive coefficient pruning method mentioned in Joint de-scratching and de-noising section causes obvious ringing effect. The proposed algorithm overcomes the drawback and gets the best restoration results in our test.

Objective quality evaluation

We have also designed an objective evaluation test for the proposed algorithm. First, we select five test sequences from the database http://sp.cs.tut.fi/mobile3dtv/stereo-video/ and another four normally used test images as original data. Then we extract some frames with scratches from the beginning of old movies. After removing the mean of those scratched frames, we add the residual to the test sequences. As a result, we have both the original clean sequences and relevant degraded sequences, so we can calculate PSNR as an objective quality measure. We compared our algorithm with other restoration algorithms, including the non-local means, MCA and BM3D. Table 3 lists the PSNR comparison of these algorithms. Figure 11 gives some visual results as well. From the results, we can find that our algorithm outperforms other methods under most conditions. Note that for a faithful test, scratches are detected by the method in [15] and blotches are detected by the method in the tests before inpainted by the algorithms of non-local means or MCA.

Besides PSNR, another objective image quality assessment method called structural similarity indexing (SSIM) [39, 40] is used to evaluate the performance of the proposed method. Similarly, the quality index of our results is better in most situations, as Table 4 shows.

The reason for the outstanding performance of the proposed method can be summarized as follows. (1) As we have said that our method is robust to the positive false in the detection phase. The region not belonging to scratch and blotch can be protected better than other methods.

(2) The inpainting based restoration method abandons the information hidden by scratch and blotch. It estimates original signal merely according to the neighborhood of the signal. When the region of scratch or blotch is too large, this method can not get enough prior knowledge from the neighborhood of signal, which makes the result of restoration over-smooth. In the proposed method, instead of abandoning the information of original signal totally, we just suppress the interference added on the signal as much as possible. This strategy is better than inpainting based method when it comes to serious degraded image. Figure 12 gives an illustration of the advantage of our method. Because we only apply inpainting algorithm to blotches, to the same frame, the region waiting for inpainting in our method is much smaller than that of other method.

Overall performances of de-scratching, de-noising and de-blotching in old movie

Results of the proposed system on practical data are shown in Figure 13, 14, 15 and 16. In Figure 13, each sub-figure reports the result from the step marked in Figure 8. The size of test frames is 1024×1024. We can find that the proposed system can produce outstanding results even to seriously damaged movies.

Our proposed system also has the advantage of high computational efficiency. On the platform of MATLAB with Inter Core2 CPU and with 3 Gb memory, the processing speed of our algorithm is on average 60.317 s per frame. In fact, compared with other existing methods, such as those based on pure nonparametric inpainting for both scratch and blotch, MCA inpainting and the method in [13], our method is much faster on the same hardware and software platform. Table 5 shows the average CPU time for processing a test frame of the testing algorithms. In Figure 16, we test our system on four different movie segments and give three consecutive frames and corresponding restoration results respectively.

The high efficiency of our proposed system can be explained by the following facts.

(1)

In scratch restoration stage, we do not have to perform complex detection, which is a large part of total time.

(2)

Wave atom transform is based on FFT and IFFT, whose complexity is O(N logN), so the complexity of our nested-thresholding method is only O(IN logN)with I the iteration number, which is much lower than wavelet based methods.

(3)

Our empirically test shows that the iterative thresholding algorithm converges fast and the iteration number we need is less than 10, whereas the iteration number of MCA can be hundreds times and even more.

(4)

It is noted that inpainting is the most computationally expensive step in our movie restoration system, whose complexity is about O(NL^{2}), where N is the number of pixel to be inpainted and L^{2}is the number of pixel in the neighborhood. Taking scratch out of non-parametric inpainting step can further reduce the computation time.

The average time cost of each phase and their percentage in the total processing time in our system is given in Table 6. The blotch inpainting phase costs most of the processing time while the de-noising and de-scratching phases occupy much less processing time.

It should be mentioned that although our system in Figure 8 shows an obvious serial type of process manner for a single frame, it can be highly paralleled on GPU for the processing of movie sequence due to the following observations.

(1)

wave atom transform and inverse-transform is block-by-block, which can be done with a high degree of parallelism.

(2)

The kernel regression based inpainting process using only local data can also be conducted in parallel.

Therefore, our proposed system already has lower complexity as compared to the existing competitors, and it is expected by facilitating GPU and other parallel computing techniques in practical implementation.

Conclusions and future work

In this article, we propose an automatic system for movie restoration. Based on the analysis of features of noise, scratch and blotch artifacts in the damaged movie, we combine a nested thresholding method in wave atom domain with nonparametric model towards effective restoration. In scratch restoration stage, we use wave atom transform to achieve joint scratch detection, restoration and de-noising. In blotch restoration stage, we take full advantages of inter- and intra-frame discontinuity and use nonparametric inpainting.

Although our work gets outstanding performances, there still exists some problems for further investigation in future work. In our model, blotch is modeled as multiplicative noise, so that the parameter q is binary. However, in practical situation, some blotches are semi-transparent in the movie. How to design a more practical model for the degradation of movie is one of our future works. On the other hand, the values of some parameters in our system are empirical.

(1)

Based on the experimental results of four films from Chinese film archive (over 4,000 frames), the configuration of L, t^{1} and t^{2} in de-blotching step achieves satisfying results for blotch restoration. However, we still need to investigate more evidence to support the configuration of them.

(2)

In de-scratching and de-noising phase of our system, the quantization step T, the stop criterion and the multiplicative constant of the threshold value in the de-noising step are fixed. We think that the selection of T may have a strong relationship with the intensity of scratch in the movie. For example, the value of T may not be global. To different regions, the quantization step can be adaptive according to the maximum value of the scratch in the region. According to noise level and image content, the stop criterion and the multiplicative constant of the threshold value in the de-noising step should be adaptive as well. In this article, they are fixed to be 3 and 0.8, respectively. We think that the estimation of the variance of noise can provide the selection of them with a useful prior knowledge. It is reasonable to think that the values of them should be increase with the variance of noise. In summary, establishing an adaptive strategy for the choice of these parameters will be one of our future work.

References

Donoho DL: Denoising by soft-thresholding. IEEE Trans. Inf. Theory 1995, 41(3):613-627. 10.1109/18.382009

Dabov K, Foi A, Katkovnik V, Egiazarian K: Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Trans. Image Process 2007, 16(8):2080-2095.

Mairal J, Bach F, Ponce J, Sapiro G, Zisserman A: Non-local sparse models for image restoration. IEEE 12th International Conference on Computer Vision, (Kyoto, Japan, 2009), pp. 2272–2279

Shao L, Zhang H, de Haan G: An overview and performance evaluation of classification-based least squares trained filters. IEEE Trans. Image Process 2008, 17(10):1772-1782.

Shao L, Wang J, Kirenko I, de Haan G: Quality adaptive least squares filters for compression artifacts removal using a no-reference bloc visibility metric. J. Vis. Commun. Image Represent 2011, 22: 23-32. 10.1016/j.jvcir.2010.09.007

Kokaram AC: Removal of line artefacts for digital dissemination of archived film and video. IEEE International Conference on Multimedia Computing and Systems, (Florence, Italy, 1999), pp. 245–249

Kokaram AC: On missing data treatment for degraded video and film archives: a survey and a new Bayesian approach. IEEE Trans. Image Process 2004, 13(3):397-415. 10.1109/TIP.2004.823815

Bruni V, Vitulano D, Kokaram A: Line scratches detection and restoration via light diffraction. Proceedings of the 3rd International Symposium on Image and Signal Processing and Analysis, (2003), pp. 5–10

Bruni V, Vitulano D: A generalized model for scratch detection. IEEE Trans. Image Process 2004, 13: 44-50. 10.1109/TIP.2003.817231

Nadenau MJ, Mitra SK: Blotch and scratch detection in image sequences based on rank ordered differences. Time-Varying Image Processing and Moving Object Recognition, (Florence, Italy, 1997), pp. 27–35

Haindl M, Filip J: Fast restoration of colour movie scratches. Proc. IEEE International Conference on Pattern Recognition, (Quebec, Canada, 2002), pp. 269–272

Joyeux L, Boukir S, Besserer B: Film line scratch removal using Kalman filtering and Bayesian restoration. Fifth IEEE Workshop on Applications of Computer Vision, (Palm Springs, America, 2000), p. 8

Maddalena L, Petrosino A: Restoration of blue scratches in digital image sequences. Image Vis. Comput 2008, 26(10):1314-1326. 10.1016/j.imavis.2006.04.013

Stanco F, Ramponi G, de Polo A: Towards the automated restoration of old photographic prints: a survey. EUROCON 2003. Computer as a Tool. The IEEE Region 8, vol. 2, (Ljubljana, Slovenia, 2003), pp. 370–374

Licsar A, Czuni L: Trainable post-processing method to reduce false alarms in the detection of small blotches of archive films. IEEE International Conference on Image Processing, vol. 2, (Genoa, Italy, 2005), pp. 562–565

Gullu MK, Urhan O, Erturk S: Blotch detection and removal for archive film restoration. AEU-Int. J. Electron. Commun 2008, 62(7):534-543. 10.1016/j.aeue.2007.07.010

Kokaram AC: Advances in the detection and reconstruction of blotches in archived film and video. IEE Seminar on Digital Restoration of Film and Video Archives, (London, UK, 2001), pp. 1–7

Demanet L: Curvelets, wave atoms, and wave equations,Thesis of California Institute of Technology. 2006.

Zoran D, Weiss Y: Scale invariance and noise in natural images. IEEE 12th International Conference on Computer Vision, (Kyoto, Japan, 2010), pp. 2209–2216

Tang C, Yang X, Zhai G: Dual-transform based noise estimation. IEEE 12th International Conference on Multimedia Expo, (Melbourne, Australia, 2012), accepted

Elad M, Starck JL, Querreb P, Donoho D: Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA). Appl. Comput. Harmon. Anal 2005, 19(3):340-358. 10.1016/j.acha.2005.03.005

Fadili JM, Starck JL, Elad M, Donoho D: Mcalab: reproducible research in signal and image decomposition and inpainting. IEEE Des. Test 2010, 12: 44-63.

Wang Z, Sheikh HR, Bovik AC: Objective Video Quality Assessment. The Handbook of Video Databases: Design and Applications. (CRC Press, Boca Raton, 2003), pp. 1041–1078

Wang Z, Bovik A, Sheikh H, Simoncelli E: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process 2004, 13(4):600-612. 10.1109/TIP.2003.819861

This article was supported in part by the NSFC (60932006, 61025005, 61001145, 61102098), RFDP (20090073110022), postdoctoral foundation of China 20100480603, 201104276, postdoctoral foundation of Shanghai 11R21414200 and the 111 Project (B07022).

Author information

Authors and Affiliations

Institute of Image Communication and Information Processing, Shanghai Jiao Tong University, Dongchuan Road 800, Shanghai, China

Hongteng Xu, Guangtao Zhai, Li Chen & Xiaokang Yang

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Xu, H., Zhai, G., Chen, L. et al. Automatic movie restoration based on wave atom transform and nonparametric model.
EURASIP J. Adv. Signal Process.2012, 132 (2012). https://doi.org/10.1186/1687-6180-2012-132