Automatic movie restoration based on wave atom transform and nonparametric model

Abstract

This article presents a new scheme for movie de-scratching, de-noising and de-blotching based on wave atom transform and nonparametric model. According to the analysis of the noise, scratches and blotches in the film, we establish a hybrid model for movie signal. In the proposed model, we model scratch as directional additive noise, which can be represented effectively by a new multi-scale analysis tool called wave atom transform. Based on wave atoms, de-noising and de-scratching are combined in a nested thresholding framework which can suppress both homogeneous noise and structural artifacts. On the other hand, blotch is described as multiplicative noise in the model, which can be restored by a “detecting and inpainting” strategy. We use the inter- and intra-frame discontinuity of blotch to detect them and then restore them by kernel regression method. Combining the processing operations above together, we design an effective automatic movie restoration system. Experimental results show that the proposed method not only achieves excellent performances even for seriously damaged movies but also has low computational complexity compared with the existing methods.

Introduction

Movie restoration is a problem with both academic challenge and great practical value. Due to long display time and undesirable storage environment, movies are damaged by various kinds of artifacts such as noise, loss of contrast, scratch, blotch, etc. These artifacts have negative influences on the artistic and commercial values of movie. For recent years, many efforts have been devoted to movie restoration, especially to de-noising, de-scratching and de-blotching problems.

In the literature, wavelet domain soft-thresholding based algorithms [1–4] have shown good performances in image de-noising. For those methods, the selection of threshold is of vital importance. Although Donoho [1] gives the universal upper bound of threshold and researchers try to achieve even better solutions in [2, 4], performances of these algorithms largely hinge on accurate noise variance estimation. An expectation-maximization (EM) algorithm for noise estimation is proposed in [3], but its computational complexity is very high. Besides transform-based methods mentioned above, non-parametric and parametric model based algorithms [5–7] also achieve good de-noising results. Currently, the methods combining parametric and nonparametric model achieve state-of-the-art results in image and video de-noising, such as the BM3D in [8] algorithm and the training-based sparse de-noising method in [9]. Both of them take advantages of self-similarity of the patches in the image and represent the image sparsely by a dictionary. However, although those algorithms are effective to remove Gaussian noise, they do not work well for structural artifacts.

Recently, some algorithms for structural artifacts removal are proposed. For example, in [10, 11], a training-based adaptive least square filter is designed to remove the coding artifacts in the image. Many de-scratching and de-blotching algorithms have also been proposed in the literature [12–15]. A joint scratch and blotch restoration method using Markov Random Field (MRF) model was presented in [16]. However, this method is not robust enough for movies with large motion. The methods in [12, 17–20] exploit inter-frame information in the Bayesian framework for scratch detection and restoration. Because of using the directionality of scratch in the detection phase, these method cannot be extended to blotch restoration straightforwardly. In our opinion, because scratch has strong directionality whereas blotch is isotropic, scratch and blotch should be restored separately. How to restore them respectively to achieve better result is a problem we try to solve. On the other hand, a common point of all these methods is that all of them use sequential detection and restoration steps, thus the overall performance will be limited by the accuracy of the detection stage, which is often very ad hoc in nature. How to reduce the dependence of restoration result on the accuracy of detection is another interesting problem.

The contribution of this work includes three folds. Firstly, according to the analysis of the scratches in the film, we model scratch as directional additive noise which can be represented effectively by wave atoms. Secondly, inspired by soft-thresholding de-noising method, we combine de-noising and de-scratching in a nested thresholding framework based on wave atom transform, which is robust to the performance drift of detection step. Thirdly, after de-noising and de-scratching, we detect blotches by the inter- and intra-frame discontinuity and restore them by kernel regression. Because scratches have been restored before, the computational complexity caused by applying kernel regression to de-blotching.

The article is organized as follows. In Signal model of movie section, we establish a hybrid model of movie signal and analyze the properties of noise, scratch and blotch in the movie. In Joint de-scratching and de-noising section, we propose a nested thresholding method in wave atom domain, which can be used for de-noising and de-scratching. In Blotch detection and restoration section, we design a robust blotch detection and restoration algorithm based on motion estimation and nonparametric kernel regression. The integrated system and experimental results are given in Implementation details and experimental results section. Conclusions and future work section concludes this article.

Signal model of movie

Hybrid noise model

We model movie signal by a hybrid noise model, as shown in (2.1).

$$\mathbf{Y}=q\mathbf{X}+(1-q)\mathbf{B}+\mathbf{S}+\mathbf{N},$$

(2.1)

where X is the ideal pixel value of video signal and Y is the noisy observation. N is homogeneous additive noise and S is the scratch component. B is the blotch component. q is a random variable controlling the transparency of blotch, and q∈[0,1] generally. If B is regarded as additive noise, (2.1) becomes a pure additive noise model. However, because B has not homogeneous property, the additive noise model increases the difficulty of analysis and restoration. For simplification, we restrict q∈{0,1}. Based on the binarization of q, B is modeled as multiplicative noise and (2.1) becomes a hybrid noise model.

This hybrid model is different from the model proposed in [21–24], which regards scratch as multiplicative noise. This change is based on both physical and chemical causes of scratch. Scratch results from the shedding of coats on the film. As long as the contact surface between the projector and the film is not clean, scratches will appear in the film during displaying. We can model the phenomenon as an additive process, i.e. the film coat becomes thinner damaged by those foreign matters. On the other hand, blotch artifacts can be caused by mildews, dirty spots or the shedding of film coats. The spatial and temporal discontinuity of blotch makes it suitable to be modeled with the multiplicative noise model. The hybrid model guides us restore them by different methods.

The analysis of noise, scratch and blotch

Generally, distribution of N in our model (2.1) can be modeled as Gaussian noise. However, since the variance σ² is not known beforehand, soft-thresholding based de-noising algorithm cannot be used directly. For simplifying the problem, we make the following assumption: the noise in the movie is homogeneous, which obeys N(0,σ²) in the spatiotemporal domain. In other words, the noise in the movie is assumed to be stationary in the spatiotemporal domain, which can be estimated by the residual signal of successive frames.

Considering its generative mechanism, scratch is a kind of anisotropic sudden interference, usually along an approximate vertical direction. It means that S is a directional noise. Further, scratch can be classified into two types according to its inter-frame continuity, as exemplified in Figure 1. In Figure 1a, we can find that the first type of scratch is relatively “weak”, which is short and narrow and does not have inter-frame continuity. This kind of scratch can be detected straightforwardly by motion estimation based method. Figure 1b shows another type of scratch whose location in adjacent frames is almost stable. Motion estimation based method cannot be useful for the second type of continuous scratch.

Figure 1

The comparison of two kinds of scratch. (a) Two adjacent frames with the type of discontinuous scratch. (b) Two adjacent frames with the type of continuous scratch.

Because of the variety of scratch, the detection method based on either inter- or intra-frame discontinuity is sensitive to the choice of parameters. For detecting scratch more accurately, Bruni proposed a damped sinusoid model for scratch in [15], which is useful to detect isolated and serious scratch. Bruni’s model reveals the oscillatory property of scratch—to each individual scratch, it can be described as a damped sinusoid signal. In fact, this microscopic view can be extended to macroscopic situation. Because scratch can accumulate during multiple times of film projection, not only does the individual but also the whole scratches in the frame obey oscillatory pattern. As shown in Figure 2, we choose several frames from some old movies to investigate the pattern of scratch. It can be found that scratches in the film have local oscillatory patterns, and such pattern would change from a local phenomenon to a global one with the increase of the displaying time.

Figure 2

The illustration of the oscillatory pattern of scratch. (a) Scratched movie frame. (b) Red circles highlight the region with oscillating pattern. (c) 3D plots of regions marked by red circles in (b).

Bruni’s model aims to solve the detection problem only while the proposed model provides a guidance to design a restoration algorithm directly. Based on the analysis above, we can find that additivity, directionality and local (even global) oscillation are three features of scratch. According to the intrinsic properties, wave atom-based method matches this problem perfectly, which will be addressed with details in Joint de-scratching and de-noising section.

Different from scratch, blotch artifact shows the isotropic pattern in most situations. Generally, it does not have the spatiotemporal continuity between and among movie frames. The blotch artifact can be very bright, very dark or even semi-transparent, as instanced in Figure 3. All of these properties increase the difficulty of removing blotch [16, 21–25].

Figure 3

Examples of blotch artifacts.

Joint de-scratching and de-noising

Overview of wave atom transform

Wave atom transform is developed firstly by Demanet and Ying [26, 27]. It is a variation of traditional wavelet aiming to overcome the weakness of wavelet in representing the directionality of high dimensional signals. The definition of wave atoms is closely related to the Fourier transform. Consider a 2D Fourier transform pair of f(x) and $\widehat{f}\left(\mathit{\omega }\right)$, where x=(x₁x₂), ω=(ω₁ω₂)are the coordinates in spatial domain and frequency domain, then wave atoms can be defined as φ _μ (x), where μ=(j m n)=(j m₁m₂n₁n₂) is a label representing the scale, direction and location of the corresponding wave atom in 2-D situation. The indexed point (x _μ ω _μ ) in phase-space is defined as

$$\begin{array}{l}{\mathbf{x}}_{\mathit{\mu }}=2^{-j}\mathbf{n},{\mathit{\omega }}_{\mathit{\mu }}=\Pi 2^j\mathbf{m},\\ C_1{2}^j\le max\left|m_i\right|\le C_2{2}^j,\end{array}$$

(3.1)

where C¹, C² are positive constants.

It can be seen that the notations follow those of the wavelet and curvelet, where j controls the resolution scale, and m and n control the location in frequency and spatial domain, respectively. However, there are two differences between wavelet and wave atom. First, wavelets obey a linear scaling wavelength while wave atoms obey the parabolic scaling wavelength: at scale 2^−2j the essential frequency support is of size 2^j, and at frequency 2^2j, the essential time support is of size 2^−j. Second, the basis function of wave atom is different from that of conventional wavelet.

In the 1D situation, the frequency formula of basis function ${\psi }_m^0\left(x\right)$ is

$$\begin{array}{l}{\widehat{\psi }}_m^0\left(\omega \right)=e^{-\mathrm{i\omega }/2}\left[e^{i{\alpha }_m}g\left({\epsilon }_m\left(\omega -\Pi \left(m+\frac{1}{2}\right)\right)\right)+\right.\\ \left.e^{-i{\alpha }_m}g\left({\epsilon }_{m+1}\left(\omega +\Pi \left(m+\frac{1}{2}\right)\right)\right)\right],\\ \sum _m\left|{\widehat{\psi }}_m^0\right(\omega ){|}^2=1,\end{array}$$

(3.2)

with ε _m =(−1)^m and ${\alpha }_m=\frac{\Pi }{2}(m+\frac{1}{2})$. g is a real-valued, C^∞ bump function on a compact support with length 2Π, which promises the energy of basis function is normalized ($\parallel {\psi }^0{\parallel }_2=1$). Similarly, the 2D basis function of wave atom can be written as

$${\phi }_{\mathit{\mu }}(x_1,x_2)=2^j{\psi }_{m_1}^0(2^j{x}_1-n_1){\psi }_{m_2}^0(2^j{x}_2-n_2).$$

(3.3)

The transform process of wave atom transform is similar to that of curvelet, which is based on FFT and IFFT. For a 2-D signal u∈R^N×N, we first perform a FFT of size N×N of u, and then get $\widehat{\mathbf{u}}$. For each pair (j m), we wrap the product ${\widehat{\psi }}_{\mathbf{m}}^j\widehat{\mathbf{u}}$ by periodicity insert the interval [−2^jΠ,2^jΠ×[−2^jΠ,2^jΠ. We then compute inverse FFT of size 2^j of the result to get c _μ . We repeat the process above over each pair (j m)to get all the coefficients of wave atom transform. Finally, we permute these coefficients according to the label μ=(j m n) to get a matrix of coefficients. Please refer to [26, 27] for the details of wave atom transform.

Figure 4 gives some examples of wave atoms and their frequency representation. It can be found that wave atoms have outstanding performance in directional representation. This is one reason why wave atoms are used for scratch representation.

Figure 4

The examples of wave atoms. The first row shows two wave atoms with different directions and scales. The second row shows their frequency responses. The corresponding labels are μ _a =(3,6,3,0,0), μ _b =(4,10,5,0,0), μ _c =(3,1,6,0,0), μ _d =(4,1,10,0,0).

The features of wave atom transform to represent scratch

Recalling the analysis in “The analysis of noise, scratch and blotch” section, we know that scratch is a directional additive noise having an oscillatory pattern. Figure 5 illustrates the coefficient’s distribution of a straight-line image in wave atom, curvelet, wavelet and dual tree complex wavelet [28] domains after 3-level decomposition. All the four coefficient domains can be divided into several regions corresponding to different directions and scales. We can find that the coefficients corresponding to vertical, horizontal and diagonal lines are separated perfectly in wave atom domain. The coefficients in red boxes in Figure 5b,e,h,k correspond to vertical signal. An intuitive thresholding example is to remove the coefficients contained in red boxes in both domains. Figure 5c,f,i,l give the vertical line removal results. We find that under the same situation (removing the coefficients corresponding to the first two levels), the vertical line removal result using wave atom transform is clearly superior. It means that under the same decomposition level, the directionality of wave atom transform is better than three competitors.

Figure 5

The comparison of different multi-scale transforms. (a), (d), (g) and (j) are the same test image with size of 128×128. (b) The green boxes illustrate the scale segments of wave atom domain and the red boxes contain the coefficients corresponding to the vertical signals. (e) The red boxes contain the curvelet coefficients corresponding to the vertical signals. (h) The red boxes contain the wavelet (db1) coefficients corresponding to the vertical signals. (k) The red boxes contain the dual tree complex wavelet coefficients corresponding to the vertical signals. (c), (f), (i) and (l) are the images by the inverse transform after removing the coefficients inside the red boxes.

Another feature of wave atom transform is its sparsity of representing oscillatory signals. In [27], wave atom transform has been proven to be able to achieve sparser representation of oscillatory signal than the existing transforms including wavelet, and curvelet. If O(N)wave atom coefficients suffice to represent an oscillatory signal to reasonable accuracy, O(N^3/2) curvelet coefficients or O(N²)wavelet coefficients will be needed to achieve the same accuracy.

The analysis above implies that if we decompose an image with scratches by wave atoms, the coefficients corresponding to scratches will be sparse and distribute in a narrow region in transform domain.

Nested thresholding in wave atom domain

We now consider how to threshold the coefficients of wave atoms so as to remove scratch and noise. The coefficients representing scratch will be limited in the narrow region corresponding to vertical component of signal, as shown in the red boxes in Figure 5b. A naive method that removes all the coefficients in this region will damage meaningful vertical image pixels rather than scratch. Another difficulty is to avoid ringing artifacts caused by the sidelobes of signal, which is common in wavelet shrinkage methods. To overcome these two problems, we propose a nested thresholding method.

We use a vertical 5×5Sobel detector to find where scratches are likely to appear and get a mask matrix M. To ensure an accurate scratch detection, morphological dilation operation [29] is applied to M. Besides Sobel detector, there exists many other effective edge and ridge detector, such as the method in [30]. However, considering the requirement for processing a large volume of data real-time, we apply the simple strategy.

The threshold of Sobel detector is critical to M. If the selected threshold is too low, it will not only generate false detection result of scratch; if the threshold is too high, then many of the scratches may not be detected or the detected scratches are too segmented. To make a trade off between computational complexity and robustness, we choose the threshold adaptively by Otsu threshold segmentation method [31]. According to the analysis below we will prove that the proposed method is robust to detection false.

Rather than classic shrinkage of the coefficients in transform domain, we propose to iteratively subtract a quantized residual from the reconstructed signal (or original signal in the first step) in the detected region. We define a quantization function as S _T (∙)with quantization step T as follows.

$$S_T(\bullet )=T\times \text{round}\left(\frac{\bullet }{T}\right).$$

(3.4)

Let F and $\widehat{F}$ be transform and inverse transform respectively. We perform wave atom transform to input frame Y and get coefficient matrix C=F(Y). C _s is the set of coefficients in the region corresponding to vertical signal. The scale level of C _s corresponding to the first and the second level. Figure 5b gives an example of 3-level decomposition and the region of C _s as the red boxes shows.

We compute the quantized residual $S_T\left(\widehat{F}\right({\mathbf{C}}_s\left)\right)$ and restore scratch in spatial domain by the following procedure

$$\tilde{\mathbf{Y}}=\mathbf{Y}-\mathbf{M}\odot S_T\left(\widehat{F}\left({\mathbf{C}}_s\right)\right),$$

(3.5)

where ⊙ denotes element-wise multiplications. With the help of M and quantization, we can effectively avoid the ringing effect. The mask M ensure that the only those regions with scratch artifacts will be restored while other regions can be passed, thus protected well. Although M may carry some texture and edge regions besides scratches, the directionality selection of wave atom transform ensures only those pixels along vertical direction will be modified. The quantization function in (3.5) further reduces ringing effect by setting the small oscillatory tails to zero.

Meanwhile, we can incorporate soft-thresholding de-noising process into our iterative framework. Denote the soft shrinkage function as Thres _σ (∙), as following shows

$${\text{Thres}}_{\sigma }\left(x\right)=\left\{\begin{array}{ll}\text{sign}\left(x\right)\left(\right|x|-{\delta }_{\sigma }),& \left|x\right|>{\delta }_{\sigma }\\ 0,& \text{otherwise.}\end{array}\right.$$

(3.6)

Here we use the conclusion in [4] and define ${\delta }_{\sigma }\in [0,\sqrt{2{\sigma }^{2}logN}]$, N is the size of signal. Based on the assumption of noise mentioned in Signal model of movie section, the variance σ² can be estimated according to the residual between adjacent frames after motion compensation. In the first iteration, ${\delta }_{\sigma }^1$ is set as the upper bound of the interval. To the i th iteration, ${\delta }_{\sigma }^i=0.8{\delta }_{\sigma }^{i-1}$. This strategy ensures the convergence of residual signal Xⁱ−Xⁱ⁻¹during iteration. When $max|{\mathbf{X}}^i-{\mathbf{X}}^{i-1}|<3$, the iteration is stopped. Under such stop criterion, we find that 4-8 iteration can achieve excellent performance for scratch restoration.

The algorithm of the nested thresholding scheme for joint de-scratching and de-blotching is summarized in Algorithm 1. One of the contributions of our algorithm is reducing the dependence of the scratch restoration results on the accuracy of detection. A dilation operation will enlarge the region of scratch in the detection mask, which may introduce positive false into M. However, our method is robust to the enlarging of detection mask. The residual quantization step in our algorithm can reduce the influence of sidelobes. As a result, even if the scratched region is enlarged by dilation operation, the sidelobes can still be suppressed by iterative quantization of residual, which will not influence on visual effect seriously.

Algorithm 1. The nested thresholding scheme

Task:

De-noising and de-scratching restoration for Y.

Initialization:

Maximum iteration number I,

initial result X⁰=Y,

quantization step T,

estimation of variance of noise σ².

Process:

Get mask matrix M=Dilation(Sobel(Y)).

In the i th iteration, i>0

Cⁱ=F(Xⁱ⁻¹);

${\mathbf{X}}^i={\mathbf{X}}^{i-1}-\mathbf{M}\odot S_T\left(\widehat{F}\right({\mathbf{C}}_s^i\left)\right)$;

$T=\frac{T}{i}$;

${\mathbf{X}}^i=\widehat{F}\left({\mathrm{Thres}}_{\sigma }\right(F\left({\mathbf{X}}^i\right)\left)\right)$.

If $max|{\mathbf{X}}^i-{\mathbf{X}}^{i-1}|<3$ or i=I

$\tilde{\mathbf{Y}}={\mathbf{X}}^I$.

Else i=i + 1.

Because our algorithm only processes the signal registered by detection mask, it is robust to the positive false of detection. In other words, even if an unscratched area is detected as scratch, the proposed method can still protect it. But if a scratch is not detected, the proposed method cannot restore it. For reducing the risk of negative false, we think the dilation operation is necessary, which ensures the robustness of our method.

To reduce the negative influence of negative-false, we apply an adaptive dilation operation row by row. To each row of M, the detected scratched region is registered as continuous non-zero pixels. Generally, the longer the length of the continuous non-zero pixels is, the smaller size of dilation operator we should use. Based on this principle, we adaptively select the size of dilation operation according to Table 1.

Table 1 Choosing dilation operator based on the length of continuous non-zero pixels in the row of M

Figure 6 gives an illustration of the restoration process for a row from an image with scratches, where it can found that with the increase of iteration, the scratch artifact can be restored without introducing serious ringing effect.

Figure 6

An illustration of de-scratching. (a) The original signal with obvious catastrophe points. (b, e, h) The residual signal $\mathbf{M}\odot S_T\left(\widehat{f}\right({\mathbf{C}}_s^i\left)\right)$ in each iteration. (c, f, i) After subtracting the residual signals in (b, e, h) during each iteration, we get restored signal marked by blue dots. (d, g) The initial signal of each iteration is the restoration result from the former iteration. (j) Compared with original signal represented by red points, after 3 times iteration, our algorithm produces a smooth result without ringing effect. (k) The final results of joint de-noising with scratch restoration (blue dots) and the original signal (red points).

Blotch detection and restoration

Blotch detection

Recall that blotch is modeled as a multiplicative noise in our degradation model. Motion estimation can be used for blotch detection because the inter-frame residual of motion estimation in the location of blotch will be much higher than other regions. Currently, the general blotch detection method is using ranked order difference (ROD) detector. In [24], a two-stage simple ROD detector (two-stage SROD) is proposed, which measures the inter-frame discontinuity of blotch in the 3×3patches of three successive frames. However, abrupt motion of image objects can also cause large residual if the size of the search window in motion analysis is not large enough. On the other hand, too-large search window size will increase the computational complexity significantly. So, merely taking advantage of the inter-frame residual is insufficient for blotch localization.

To detect blotch robustly, we will also exploit the intra-frame information. Although both blotch and motion object have large inter-frame residual, we can discriminate them according to their intra-frame features. Based on the analysis of Signal model of movie section, we know that blotch artifact is incoherent to its neighbor pixels. On the other hand, the pixels of the moving objects are more likely to have similar values in the neighborhood. We define the pixel in the i th row, j th column and t th frame as Y _ijt . P _ijt ∈R^L×L is a patch with the center at Y _ijt . L is the size of the patch. Calculating the residual between the patches in adjacent frames, we can get Res _ijt =|P _ijt −P_ij,t−1|. If there are pixels in P _ijt belonging to blotch, they should satisfy Res _ijt (a,b)>t₁, where t¹ is a threshold and a, b is the coordinates of pixel in P _ijt . We can get a mask M₁as following shows.

$${\mathbf{M}}_1(a,b)=\left\{\begin{array}{ll}1,& {\mathbf{\text{Res}}}_{\mathrm{ijt}}(a,b)>t_1\\ 0,& \text{otherwise}\end{array}\right.a,b=1,\dots ,\mathrm{L.}$$

(4.1)

M₁chooses the candidates of blotch according to the inter-frame discontinuity of blotch. For getting a more accurate estimation, we should take intra-frame discontinuity into consideration. So, we define E∈R^L×L as the residual among Y _ijt and its neighborhood pixels as follows.

$$\begin{array}{l}\mathbf{E}(a,b)=|Y_{\mathrm{ijt}}-{\mathbf{P}}_{\mathrm{ijt}}(a,b\left)\right|,\\ a,b=1,\dots ,L/2-1,L/2+1,\dots ,\mathrm{L.}\end{array}$$

(4.2)

Similarly, if there are pixels in P _ijt belonging to blotch, they should satisfy E(a,b)>t₂, where t² is another threshold. We can get another mask M₂as following shows.

$${\mathbf{M}}_2(a,b)=\left\{\begin{array}{l}1,\mathbf{E}(a,b)>t_2\\ a,b=1,\dots ,L/2-1,\\ L/2+1,\dots ,\mathrm{L.}\\ 0,\text{otherwise}\end{array}\right.$$

(4.3)

Only the pixels satisfying M₂(a,b)=M₁(a,b)=1are considered as blotch. After getting the primary detection result, a dilation operation is used to suppress the negative false of detection further.

It should be noted that if the blotch has almost uniform luminance value, M₂ will contain the pixels of the boundary of the blotch rather than the whole region of the blotch. Generally, blotches having almost uniform luminance value are likely to have small areas. After finding their boundaries in M₂, we can detect their whole regions by dilation operation. On the other hand, if a blotch has large area, its luminance value is generally not uniform. In such situation, we can still detect blotch with the help of M₁ and M₂.

Our method can be regarded as an extension of the work in [13, 24]. The improvement of our method is that we not only consider the inter-frame discontinuity of blotch but also consider the intra-frame discontinuity. In [24], there is only one threshold in two-stage SROD controlling the criterion of inter-frame discontinuity. The principle of the Bayesian method in [13] is also taking advantages of inter-frame discontinuity. In our method, we use two thresholds t¹ and t² to control the criterions of inter- and intra-frame discontinuity respectively. Because introducing intra-frame information of blotch, our method is more robust to pathological motion [13] and occlusions in the movie.

Figure 7 shows an example of blotch detection, where it can be found that using intra-frame discontinuity property, the detection is robust to large motion. In this algorithm, t¹ and t² are thresholds for inter-frame and intra-frame discontinuity. Note that t² is related to the search window size L. Generally, with the increase of L, intra-frame coherence of arbitrary two pixels in the window becomes weaker, so the value of t² should decrease. Too large t¹ and t² will increase the risk of false-negative response while too small thresholds may cause a false-positive response. In our experiments, we choose L=18, t₂=2, and t₁=15empirically based on the experimental results of over 4,000 frames. A dilation operation [29] is used to the detection result for improving the accuracy. The dilation operator is a square with size of 5×5 for the test frame with size of 1024×1024. From Figure 7, we can find that our detection result is good enough for a computationally efficient algorithm.

Figure 7

The comparison of different blotch detectors. (a,b) Test frame 1,2 of size 1024×1024for blotch detection. (c) The inter-frame blotch detection result of frame 1. (d) The inter- and intra-frame detection result of frame 1.

Blotch restoration

For blotch restoration of the old movie, we can take advantages of both inter- and intra- frame information with a nonparametric kernel regression based approach. For a missing pixel value y, its temporal and spatial context is denoted as a vector form x. We can approximate y using conditional expectation as

$$y=E\left(Y\right|X={\mathbf{x}}_i)=\int y_{i}P(y_i\left|{\mathbf{x}}_i\right)d{y}_i=\int y_i\frac{P(y_i,{\mathbf{x}}_i)}{P\left({\mathbf{x}}_i\right)}d{y}_i.$$

(4.4)

To get y=E(Y|X=x) we have to compute the joint probability P(y _i x _i ) and the marginal probability P(x _i ), without presuming any parametric relationship between y _i and x _i . Kernel regression is a widely used nonparametric technique to estimate probability functions [32, 33]. With kernel regression, we have

$$\widehat{P}\left({\mathbf{x}}_i\right)=\frac{1}{n}\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel ),$$

(4.5)

$$\widehat{P}(y_i,{\mathbf{x}}_i)=\frac{1}{n}\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )K_g\left(\parallel y_i-y_j\parallel \right),$$

(4.6)

where the kernels $K_{\Delta }(\circ )=\frac{1}{\Delta }K\left(\frac{\circ }{\Delta }\right)$ can be chosen from functions that are nonnegative, sum to 1 and symmetric around 0. h and g are called bandwidth of the kernels. Widely used kernel function includes the triangle kernel, Gaussian kernel, Epanechnikov kernel among others. Note that the product kernels rather than multivariate kernels are used in (4.6) to simplify the analysis. See [32] for more details.

Substituting $\widehat{P}\left({\mathbf{x}}_i\right)$ and $\widehat{P}(y_i,{\mathbf{x}}_i)$ into (4.4), we have

$$\begin{array}{l}y=E\left(Y\right|X={\mathbf{x}}_i)\\ =\frac{\int y_i\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )K_g(\parallel y_i-y_j\parallel )d{y}_i}{\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )}\\ =\frac{\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )\left(\right.\int y_i{K}_g(\parallel y_i-y_j\parallel )d{y}_i\left)\right.}{\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )}.\end{array}$$

(4.7)

Now we have to deal with $\int y_i{K}_g(\parallel y_i-y_j\parallel )d{y}_i$. By defining a residual term as y^′=y _i −y _j , and omitting the bandwidth parameter g for simplicity, we have

$$\int y_{i}K(\parallel y_i-y_j\parallel )d{y}_i=\int (y_j+y^{\prime })K(\parallel y^{\prime }\parallel )d{y}_i\approx y_j,$$

(4.8)

where we used the properties of the kernel function that it sums to 1 and is symmetric around the origin. Letting (4.8) into (4.7), we have

$$y=E\left(Y\right|X={\mathbf{x}}_i)\approx \frac{\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )y_j}{\sum _{j=1}^n{K}_h(\parallel {\mathbf{x}}_i-{\mathbf{x}}_j\parallel )}.$$

(4.9)

The nonparametric conditional expectation estimator in (4.9) is known as the Nadaraya–Watson (NW) estimator, which is weighted average of the observations {y₁,y₂,…,y _n } with the weight in inverse proportion to the distances between x _j ,1≤j≤n and x _i . For higher accuracy of the approximation, normalization is recommended and we can rewrite (4.9) as following shows

$$y=E\left(Y\right|X={\mathbf{x}}_i)\approx \frac{\sum _{j=1}^n{K}_h(\parallel {\widehat{\mathbf{x}}}_i-{\widehat{\mathbf{x}}}_j\parallel ){\delta }_j{y}_j}{\sum _{j=1}^n{K}_h(\parallel {\widehat{\mathbf{x}}}_i-{\widehat{\mathbf{x}}}_j\parallel )},$$

(4.10)

where $\widehat{\mathbf{x}}$ is the normalized vector of x and δ _j is the ratio between the l² norm of x _j and that of x _i . The kernel we choose is Gaussian kernel $\left(\frac{exp\left({(\bullet /h)}^2\right)}{h\sqrt{\Pi }}\right)$, where h is the bandwidth of the kernel. The adaptive bandwidth selection for the kernel is based on cross validation. By minimizing the following cost function, we get the optimized bandwidth.

$$h=argmin\frac{1}{n}\sum _{j=1}^n{({ŷ}_j-y_j)}^2,$$

(4.11)

where ${ŷ}_j$ is the estimated pixel value of y _j without using the j th context x _j . Because the cross validation method is computationally very demanding, we only take the neighbors of y _i into consideration and set n≤9in (4.11).

Implementation details and experimental results

The diagram of the proposed movie restoration system is given in Figure 8. The system includes three modules: parameter estimation, joint scratch and noise removal, and blotch restoration. And the implementation details will be introduced in this section.

Figure 8

The architecture of final system.

Noise estimation and de-noising

We estimate the variance of noise using motion estimation. For each patch in the current frame, we search the patch with the shortest Euclidean distance in the preceding frame to get the motion vector and the residual patch. For simplicity, we employ the three-step motion estimation method [34] in this research. Define P _i and P _j as original patches, W _i and W _j the i.i.d. Gaussian random variables obeying N(0,σ²). It is obvious that $W_i-W_j\backsim N(0,2{\sigma }^2)$. Moreover, according to the spatiotemporal stationary assumption of the noise in the movie, P _i −P _j has Laplacian distribution Laplace(0,b)[29]. So, we can estimate the variance of noise with the residual images from the sequential frames.

Compared with other professional noise estimator, such as the methods in [35, 36], the proposed estimator has lower computational complexity, which can be achieved real-time. Considering the large volume of data in the movie restoration system, the complexity of the algorithm should be limited. Moreover, the methods in [35, 36] focus on estimate the noise in the single image while the proposed method takes advantage of the inter-frame continuity of movie signal.

For testing the performance of the proposed method, we select six test sequences from the database http://sp.cs.tut.fi/mobile3dtv/stereo-video/ and add Gaussian noise to them. After applying different estimators to these sequence, we give comparison results of different methods on the accuracy of noise estimation in Table 2. We can find that the proposed method achieves a comparable even better performance on noise estimation.

Table 2 Comparison among different noise estimator

Based on the noise estimation result, we can apply wave atom based de-noising algorithm proposed in Nested thresholding in wave atom domain section. Figure 9 gives a comparison among original video frames, the degraded frames and de-noising results. After performing the proposed de-noising method, we can improve the visual effect of the frames greatly.

Figure 9

To each sub-figure, the images from light to right are original image, the image having Gaussian noise with σ =25 and the de-noising result gotten by the proposed method.

Scratch restoration

We compare our algorithm with some existing scratch restoration algorithms. The initial quantization step is 8. The four test movies are from China Film Archive, namely “TAO LI JIE”, “FANG ZHEN ZHU”, “DA LU” and “GUAN LIAN ZHANG”.

Figure 10 shows the experimental results of scratch removal (without blotch restoration) by different algorithms, including BM3D, non-local means restoration [6], MCA inpainting [37, 38], the method in [12, 15] (we used the method proposed in [15] to detect scratch, after that, we use the method in [12] to restore the degraded images), the simple wave atom coefficients pruning based method mentioned in Joint de-scratching and de-noising section and our proposed algorithm.

Figure 10

Scratch removal results. In each sub-figure, images from top to bottom and from left to right are: original image; the de-scratching results gotten by BM3D restoration, non-local means inpainting, MCA inpainting, the method in [12], coefficients removing in wave atom domain and the proposed method. The test frames are: (a) “TAO LI JIE”; (b) “FANG ZHEN ZHU”; (c) the test image in [12, 15].

It can be found that BM3D is only effective on noise removal. Both non-local means and MCA tend to produce over-smoothed results and remove some important details in the images (see the area of human eyes). The method in [15] gives false-negative detection when on light vertical scratches, so that non-local means, MCA and the method in [12] have the risk of false-negative judgment. On the other hand, our proposed scratch removal algorithm based on wave atom transform does not closely depend on accurate detection. The naive coefficient pruning method mentioned in Joint de-scratching and de-noising section causes obvious ringing effect. The proposed algorithm overcomes the drawback and gets the best restoration results in our test.

Objective quality evaluation

We have also designed an objective evaluation test for the proposed algorithm. First, we select five test sequences from the database http://sp.cs.tut.fi/mobile3dtv/stereo-video/ and another four normally used test images as original data. Then we extract some frames with scratches from the beginning of old movies. After removing the mean of those scratched frames, we add the residual to the test sequences. As a result, we have both the original clean sequences and relevant degraded sequences, so we can calculate PSNR as an objective quality measure. We compared our algorithm with other restoration algorithms, including the non-local means, MCA and BM3D. Table 3 lists the PSNR comparison of these algorithms. Figure 11 gives some visual results as well. From the results, we can find that our algorithm outperforms other methods under most conditions. Note that for a faithful test, scratches are detected by the method in [15] and blotches are detected by the method in the tests before inpainted by the algorithms of non-local means or MCA.

Figure 11

Each sub-figure gives the processing result of the proposed old movie restoration algorithm, from left to right: original images, degraded images and restored images by proposed method.

Table 3 Comparison among different algorithms on PSNR (dB)

Besides PSNR, another objective image quality assessment method called structural similarity indexing (SSIM) [39, 40] is used to evaluate the performance of the proposed method. Similarly, the quality index of our results is better in most situations, as Table 4 shows.

Table 4 Comparison among different algorithms on SSIM

The reason for the outstanding performance of the proposed method can be summarized as follows. (1) As we have said that our method is robust to the positive false in the detection phase. The region not belonging to scratch and blotch can be protected better than other methods.

(2) The inpainting based restoration method abandons the information hidden by scratch and blotch. It estimates original signal merely according to the neighborhood of the signal. When the region of scratch or blotch is too large, this method can not get enough prior knowledge from the neighborhood of signal, which makes the result of restoration over-smooth. In the proposed method, instead of abandoning the information of original signal totally, we just suppress the interference added on the signal as much as possible. This strategy is better than inpainting based method when it comes to serious degraded image. Figure 12 gives an illustration of the advantage of our method. Because we only apply inpainting algorithm to blotches, to the same frame, the region waiting for inpainting in our method is much smaller than that of other method.

Figure 12

The comparison of different resotration methods in blotch detection. (a) The region waiting for inpainting in our method, which only contains blotches. (b) The region waiting for inpainting in other methods, which contains both blotches and scratches.

Overall performances of de-scratching, de-noising and de-blotching in old movie

Results of the proposed system on practical data are shown in Figure 13, 14, 15 and 16. In Figure 13, each sub-figure reports the result from the step marked in Figure 8. The size of test frames is 1024×1024. We can find that the proposed system can produce outstanding results even to seriously damaged movies.

Figure 13

Restoration process of our system. Each sub-figure corresponds to a processing step numbered in Figure 8. The test frames are from “TAO LI JIE”.

Figure 14

The comparison of different methods. The first row is detection results of frames before inpainting. The second row is results of non-local means inpainting based on the detection results from the first row. The third row is results of MCA inpainting [37] based on the detection results from the first row. The fourth row is the results gotten by the method in [13]. The final row are results by the proposed method, which corresponds to the output marked by (4) in Figure 11. The test frames are from “TAO LI JIE”.

Figure 15

The enlarged images of results. In each sub-figure, the images from left to right are: the original image; non-local mean inpainting; MCA inpainting; the method in [12]; the proposed method. The test frames are: (a) “TAO LI JIE”; (b) FANG ZHEN ZHU”.

Figure 16

More test results. The first row of each sub-figure: to original frames. The second row of each sub-figure: automatic restoration result of our proposed system. The test frames are: (a) “DA LU”; (b) “TAO LI JIE”; (c) “GUAN LIAN ZHANG” and (d) “FANG ZHEN ZHU”.

Our proposed system also has the advantage of high computational efficiency. On the platform of MATLAB with Inter Core2 CPU and with 3 Gb memory, the processing speed of our algorithm is on average 60.317 s per frame. In fact, compared with other existing methods, such as those based on pure nonparametric inpainting for both scratch and blotch, MCA inpainting and the method in [13], our method is much faster on the same hardware and software platform. Table 5 shows the average CPU time for processing a test frame of the testing algorithms. In Figure 16, we test our system on four different movie segments and give three consecutive frames and corresponding restoration results respectively.

Table 5 Run time per frame

The high efficiency of our proposed system can be explained by the following facts.

• (1) In scratch restoration stage, we do not have to perform complex detection, which is a large part of total time.

• (2) Wave atom transform is based on FFT and IFFT, whose complexity is O(N logN), so the complexity of our nested-thresholding method is only O(IN logN)with I the iteration number, which is much lower than wavelet based methods.

• (3) Our empirically test shows that the iterative thresholding algorithm converges fast and the iteration number we need is less than 10, whereas the iteration number of MCA can be hundreds times and even more.

• (4) It is noted that inpainting is the most computationally expensive step in our movie restoration system, whose complexity is about O(N L ²), where N is the number of pixel to be inpainted and L ²is the number of pixel in the neighborhood. Taking scratch out of non-parametric inpainting step can further reduce the computation time.

The average time cost of each phase and their percentage in the total processing time in our system is given in Table 6. The blotch inpainting phase costs most of the processing time while the de-noising and de-scratching phases occupy much less processing time.

Table 6 Average run time per frame and percentage in total time of each phase

It should be mentioned that although our system in Figure 8 shows an obvious serial type of process manner for a single frame, it can be highly paralleled on GPU for the processing of movie sequence due to the following observations.

• (1) wave atom transform and inverse-transform is block-by-block, which can be done with a high degree of parallelism.

• (2) The kernel regression based inpainting process using only local data can also be conducted in parallel.

Therefore, our proposed system already has lower complexity as compared to the existing competitors, and it is expected by facilitating GPU and other parallel computing techniques in practical implementation.

Conclusions and future work

In this article, we propose an automatic system for movie restoration. Based on the analysis of features of noise, scratch and blotch artifacts in the damaged movie, we combine a nested thresholding method in wave atom domain with nonparametric model towards effective restoration. In scratch restoration stage, we use wave atom transform to achieve joint scratch detection, restoration and de-noising. In blotch restoration stage, we take full advantages of inter- and intra-frame discontinuity and use nonparametric inpainting.

Although our work gets outstanding performances, there still exists some problems for further investigation in future work. In our model, blotch is modeled as multiplicative noise, so that the parameter q is binary. However, in practical situation, some blotches are semi-transparent in the movie. How to design a more practical model for the degradation of movie is one of our future works. On the other hand, the values of some parameters in our system are empirical.

• (1) Based on the experimental results of four films from Chinese film archive (over 4,000 frames), the configuration of L, t ¹ and t ² in de-blotching step achieves satisfying results for blotch restoration. However, we still need to investigate more evidence to support the configuration of them.

• (2) In de-scratching and de-noising phase of our system, the quantization step T, the stop criterion and the multiplicative constant of the threshold value in the de-noising step are fixed. We think that the selection of T may have a strong relationship with the intensity of scratch in the movie. For example, the value of T may not be global. To different regions, the quantization step can be adaptive according to the maximum value of the scratch in the region. According to noise level and image content, the stop criterion and the multiplicative constant of the threshold value in the de-noising step should be adaptive as well. In this article, they are fixed to be 3 and 0.8, respectively. We think that the estimation of the variance of noise can provide the selection of them with a useful prior knowledge. It is reasonable to think that the values of them should be increase with the variance of noise. In summary, establishing an adaptive strategy for the choice of these parameters will be one of our future work.