Efficient nonuniform deblurring based on generalized additive convolution model
 Hong Deng^{1},
 Dongwei Ren^{1},
 David Zhang^{1},
 Wangmeng Zuo^{1}Email author,
 Hongzhi Zhang^{1} and
 Kuanquan Wang^{1}
https://doi.org/10.1186/s1363401603182
© Deng et al. 2016
Received: 4 August 2015
Accepted: 3 February 2016
Published: 19 February 2016
Abstract
Image with nonuniform blurring caused by camera shake can be modeled as a linear combination of the homographically transformed versions of the latent sharp image during exposure. Although such a geometrically motivated model can well approximate camera motion poses, deblurring methods in this line usually suffer from the problems of heavy computational demanding or extensive memory cost. In this paper, we develop generalized additive convolution (GAC) model to address these issues. In GAC model, a camera motion trajectory can be decomposed into a set of camera poses, i.e., inplane translations (slice) or roll rotations (fiber), which can both be formulated as convolution operation. Moreover, we suggest a greedy algorithm to decompose a camera motion trajectory into a more compact set of slices and fibers, and together with the efficient convolution computation via fast Fourier transform (FFT), the proposed GAC models concurrently overcome the difficulties of computational cost and memory burden, leading to efficient GACbased deblurring methods. Besides, by incorporating group sparsity of the pose weight matrix into slice GAC, the nonuniform deblurring method naturally approaches toward the uniform blind deconvolution. Experimental results show that GACbased deblurring methods can obtain satisfactory deblurring results compared to both stateoftheart uniform and nonuniform deblurring methods and are much more efficient than nonuniform deblurring methods.
Keywords
1 Introduction
Image blur is generally inevitable due to various factors such as defocus and camera shake. Blind deblurring from a realworld blurry image that needs to estimate the blur procedure and latent sharp image is a very illposed problem. Recently, several hardwareassisted approaches had been developed [1–6], by which additional information can be acquired to reduce the illposedness of the blind deblurring. These hardwareassisted approaches can provide much easier deviation of the blur or sharp image, but require complex camera configurations [1, 2] or dedicatedly designed hardware support [3–6], far from taking the place of traditional imaging devices. At the same time, in the era of ubiquitous acquisition of digital images using portable imaging devices, e.g., digital camera, mobile phone, camera shake is often unavoidable during exposure procedure, a major cause that ruins a photograph. For effective and efficient deblurring, it is crucial to develop appropriate forward blur model that can well explain the image degradation process of camera shake and then to design proper regularizers and efficient optimization algorithms.
Earlier approaches on camera shake deblurring usually assume that the blur is spatially invariant, due to which the blurry observation can be simply modeled as convolution of sharp image and blur kernel, and deblurring can thus be modeled as a blind deconvolution problem [7–12], where much attention had been devoted to design effective optimization algorithms. Since the convolution operation can be computed efficiently via fast Fourier transform (FFT), these blind deconvolution algorithms are commonly efficient. On one hand, Fergus et al. [7] adopted a mixture of Gaussians for representing the distribution of gradients and introduced a variational Bayesian (VB) approach [13] to estimate the blur kernel. The theoretical and experimental analysis by Levin et al. [8] demonstrated the advantages of VB over maximum a posterior (MAP) and inspired a number of VBbased blind deconvolution algorithms [7, 8, 14, 15]. On the other hand, by enforcing sparser priors [10, 16, 17] or exploiting edge prediction step [9, 18–21] to select salient edges in latent image, several carefully designed MAP algorithms can also exhibit promising performance. Most recently, discriminative learning approaches had been developed to learn proper priors for better estimating blur kernel [22–25].
However, camera shake blurring is often spatially varying and consequently cannot be simply modeled as a single blur kernel. Thus, how to reasonably model camera shake blurring plays a central role in nonuniform deblurring problem that can be categorized from two aspects. One direct strategy is to approximate the nonuniform blur as multiple blur kernels, where the blurry image is divided into several regions, and each region takes uniform blur. The multiple blur kernels can then be estimated from each region by performing uniform blind deconvolution method locally [26, 27]. The method proposed by Cao et al. [27] incorporated natural and textspecific dictionaries for the blind deblurring of natural scene text image. For nonuniform deblurring, they simply adopted the strategy of estimating the dedicated blur kernel for each text field. However, this kind of approach does not consider the global constraints on local blur kernels based on camera motion trajectory. The other geometrical strategy is based on the projective motion path blur model (PMPB) [28], where a camera shake trajectory can be decomposed into a sequence of camera poses lying in the 6D camera pose space (x, y, zaxes rotation and x, y, zaxes translation), and consequently, the sharp image projected by each pose is a homography, which is then weighted according to its exposure time, resulting in the blurry image. By far, there are mainly two simplified 3D geometrical models to approximate the 6D subspace, i.e., Whyte et al. [29] suggested to employ x(pitch), y(yaw), and z(roll) axes rotation, while Gupta et al. [30] proposed to adopt x and yaxes (inplane) translation and zaxis rotation. By combining the global camera motion constraint and the efficient filter flow (EFF) framework [31], Hirsch et al. suggested to construct local uniform blur models guided by the geometrical constraint [32].
Even in the simplified 3D camera pose subspace [29, 30], nonuniform deblurring methods still suffer the problem of high computational cost or extensive memory burden, placing prominent restriction on its wide applications. To speed up the blur estimation step, Gupta et al. [30] precomputed a sparse matrix for each homography transform, and thus, the forward nonuniform blur operator can be equivalently defined as the weighted sum of the homography transform matrices, and Hu and Yang [33] further restricted the possible camera pose in a low dimensional subspace. Although the precomputation techniques can relatively relax the computational inefficiency, these accelerated methods [30, 33] take increasing memory burden to store the huge homography transform matices. Even so, subsequent computation of homography transform matrices is still computationally costly, and thus, it is interesting to ask that is it possible to design forward blur model to benefit from FFT for efficient nonuniform deblurring?
In this paper, we propose generalized additive convolution (GAC) model, by which a camera motion trajectory can be decomposed into inplane translations (slice) or roll rotations (fiber), resulting in slice GAC model or fiber GAC model. In slice GAC model, a homography is formulated as the rotation of the convolution image of slice and sharp image, and in fiber GAC model, a homography is the inverse polar transform of the convolution of a kernel with the transformed sharp image (please refer to Section 3.2 for detailed proof), in which the convolution operation can be efficiently computed with the help of FFT. By this way, the GACbased forward blur models only require several FFTs and a number of pixelwise operations, significantly reducing computational complexity, and concurrently only the basis kernels and a number of rotation matrices are required to be stored so that the problem of memory burden is also relaxed. Furthermore, a greedy algorithm is proposed to generate a more compact set of slices and fibers from any camera motion trajectory, resulting in hybrid GAC model, a more promising way to concurrently solve the problems of computational inefficiency and memory burden. As for the optimization algorithm, we adopt the MAP framework to alternatively estimate nonuniform blur and latent image, in which generalized accelerated proximal gradient (GAPG) algorithm [34], a much efficient optimization algorithm for nonblind deconvolution, is employed for GACbased nonuniform deblurring.
Moreover, in slice GAC, we introduce group sparsity, i.e., l _{2,1}norm, on the pose weight matrix along the rotation angle dimension, interestingly providing a way to make nonuniform deblurring approach toward uniform blind deconvolution. Under alternative minimization framework, we also propose an effective solution to solve this problem. With the group sparsity constraint, more slices with rotation angles around 0 can be activated, and extremely when all the slices except angle 0 are inactivated, the nonuniform delburring method can degrade to uniform blind deconvolution.
Experimental results show that the proposed GAC method can obtain comparable or better deblurring results than the competing uniform and nonuniform deblurring methods. Compared to nonuniform deblurring method, GAC has a much lower peak memory usage than [16, 30, 33] and is much more efficient than the stateoftheart camera shake removal methods [16, 29, 30, 32, 33].

We develop GAC framework together with two GAC models, i.e., slice GAC and fiber GAC, for forward modeling of camera shake blurring, by which FFT can be employed for efficient computation.

To further reduce the computational complexity, a greedy algorithm is proposed to generate hybrid GAC from any camera motion trajectory, resulting in a more compact GAC model.

As for the optimization algorithm, we adopt a fast gradient method, i.e., GAPG [34], to estimate latent image, contributing to the faster convergence of the deblurring algorithm.

By imposing group sparsity of pose weight matrix in slice GAC, we interestingly provide a way to connect uniform and nonuniform deblurring.
The rest of the paper is organized as follows. Section 2 provides a brief review on camera shake models and optimization algorithms. Section 3 presents the proposed GAC models, which in Section 4 are embedded in nonuniform deblurring, and can be efficiently solved via GAPG. Section 5 presents experimental results, and finally, Section 6 concludes this paper.
2 Prerequisites and related work
For efficient nonuniform deblurring, it is crucial to design forward blur models and efficient optimization algorithms. In this section, we briefly review the original projective motion path blur (PMPB) model and its simplified 3D approximation, and the optimization algorithms in MAPbased deblurring.
2.1 Homographybased model
In the PMPB model, a camera shaken image is geometrically the integration of what the camera “sees” along the motion trajectory that can be modeled in a 6D camera pose space including x, y, and zaxes rotation and x, y, and zaxes translation. With the discretion of the time t and camera pose space, the integration can be rewritten as the weighted summation of all homographies in the discrete camera pose space, where a homography is the transformed sharp image projected by corresponding camera pose.
where C is the matrix of camera intrinsic parameters, d is the depth of the scene, and R _{ j } and t _{ j } are the rotation matrix and translation vector, respectively.
where w _{ j } is the fraction of time the camera spent at pose j, \(\boldsymbol {b}\in {{\mathbb {R}}^{n\times n}}\) is the blurry image, \(\boldsymbol {x}\in {{\mathbb {R}}^{n\times n}}\) is the latent sharp image, K is the sparse warping matrix with size n ^{2}×n ^{2}, and v denotes the additive Gaussian white noise.
PMPB is faithful to realworld camera motion, but has too many unknowns to be estimated. Fortunately, recent studies have shown that the full 6D camera pose space can be well approximated by the discrete 3D subspace of pitch, yaw, and roll rotations [29], or the 3D subspace of inplane translations and roll rotation in wide focal lengths range [30], which are both effective [35]. In [33], constrained camera pose subspace was further introduced to refine the set of camera poses. Interestingly, the global camera motion constraint can be adopted to guide the construction of local blur kernels [32]. However, the method in [29] suffers from the problem of heavy computational load to compute huge warping matrices, while the methods in [30, 33] suffer from the extensive memory burden problem to store huge warping matrices.
2.2 Additive convolution model
where k _{ i } is the ith basis convolution kernel, α _{ i } denotes the ith weight matrix, ⊗ denotes the convolution operation, ∘ denotes the pixelwise multiplication operator, and C is the number of basis convolution kernels. With FFT, the computational complexity of AC model is only \(O(Cn^{2}\mathop {\log }n)\). Furthermore, principal component analysis (PCA) is adopted to learn the basis convolution kernels in advance. However, when applied to blind nonuniform deblurring, both the basis kernels and the weighted maps should be updated in each iteration, making it unpractical in blind deblurring of camera shaken images.
In this paper, we propose a fast forward blur model, i.e., GAC, to represent nonuniform blur caused by camera shake, which is distinctly different with the existing models. Instead of utilizing sparse warping matrix [29, 30, 33], GAC models reformulate homography as convolution related to slice and fiber and can achieve better tradeoff between computational cost and memory complexity. Different from [32], GAC is an efficient implementation of the global geometric model rather than the locally uniform approximation. Compared with [36], the basis convolution kernels and the weighted maps can be efficiently constructed based on the camera motion trajectory, and thus is easy to update.
2.3 MAPbased blind deblurring
where Φ _{1}(x) denotes the regularizer on the latent sharp image and Φ _{2}(K) denotes the regularizer on the blur operator. For nonuniform blur, \(\boldsymbol {K} \boldsymbol {x} = {\sum \nolimits }_{j} {{w_{j}}} {{\boldsymbol {K}}_{j}}{\boldsymbol {x}}\phantom {\dot {i}\!}\), where the blur operator is parameterized by the weights w _{ j }. Like the alternative updating strategy commonly adopted in uniform blind deconvolution [16, 18], the estimation of weight matrix W and latent image x should be alternatively performed.
To constrain the illposed problem for better deblurring quality, proper regularizers should be imposed. In 3D camera pose subspace, only a few poses are active given a camera motion trajectory so that it is reasonable to introduce sparsity on W. In [29, 33], l _{2}norm of weight matrix or its gradient is imposed, where the estimation of W is a linear least square problem, which can be solved by gradientbased optimization method, while in [30], nonconvex regularizer is introduced, which is optimized by an iterative reweighted least squares (IRLS).
As for updating the latent clear image, the edge prediction step is usually necessary to guarantee the algorithm converge to the desired solution, so that the sparsity, e.g., total variation [29, 33], is also imposed. In earlier researches, simple but efficient optimization method, e.g., RachardsonLucy (RL) algorithm, was adopted to solve the problem [28]; however, the deblurring results often suffer from ringing artifacts. As a nonblind deconvolution problem, fast gradientbased optimization method has been intensively studied [37, 38]. The iterative shrinkage thresholding algorithm (IST) [37] was first proposed, and due to its simplicity and efficiency, two accelerated ISTbased algorithms had been developed, i.e., FISTA [39] and TwIST [38], which both possess higher convergence rate. Furthermore, Zuo and Lin proposed the GAPG algorithm [34], which further accelerated the convergence rate of APG algorithm.
In this paper, the GACbased deblurring problem is solved in the alternative minimization framework, where the updating of pose weight matrix with both sparsity and group sparsity regularizers can be solved, and the updating of latent clear image can be efficiently solved by GAPG algorithm.
3 Generalized additive convolution model for camera shake
In this section, we first propose the general form of the GAC model and then decompose camera motion trajectory into slices and fibers, which provides a solution to specify GAC model for efficient modeling of camera shake blurring. Finally, we propose a greedy algorithm to generate hybrid GAC model and discuss the memory and computational complexity of the GAC model.
3.1 Generalized additive convolution model
where f _{ i } and g _{ i } are two pixelwise operators. GAC is the generalization of the AC model [36], by defining f _{ i }(x)=α _{ i }∘x and g _{ i }(x)=x. Moreover, since both f _{ i } and g _{ i } are pixelwise operators, the computational complexity of GAC is the same as that of the AC model, i.e., O(C n ^{2}log_{2} n), where n ^{2} is the number of pixels of the image x. The smaller the C value indicates that GAC model is more efficient. Thus, the key issue of GAC model is to better specify k _{ i }, f _{ i }, and g _{ i } to reduce the C value.
3.2 Decomposition of slice and fiber
In this subsection, we show that camera poses in the 3D camera pose subspace [30] can be decomposed into slices and fibers, which then provides proper choices to design k _{ i }, f _{ i }, and g _{ i } in GAC model to reduce C value.
where t _{ j }=[t _{ x,j },t _{ y,j },1]^{ T } is the translation vector, d is the depth of the scene.
where \(\alpha _{x}=f*\frac {{{m}_{x\_{\text {ccd}}}}}{{{m}_{\textit {x\_im}}}}\phantom {\dot {i}\!}\) is the scale factors relating pixels to distance, \({{m}_{x\_{\text {ccd}}}}\) is the maximum width of the CCD, \({{m}_{x\_{\text {im}}}}\) is the maximum width of the related image, \(\alpha _{y}=f*\frac {{{m}_{y\_{\text {ccd}}}}}{{{m}_{y\_{\text {im}}}}}\phantom {\dot {i}\!}\) is quantified in the same manner, f is the focal length, and (x _{0},y _{0}) is the center coordinate.
Given \({{\boldsymbol {H}}_{\boldsymbol {\theta }}}_{_{j}}\phantom {\dot {i}\!}\), we can construct the corresponding warping matrix \({{\boldsymbol {K}}_{\boldsymbol {\theta }}}_{_{j}}\phantom {\dot {i}\!}\) (more detailed explanations can be found in [30, 33, 36]). We then have that the homography transform can be decomposed into inplane translation followed by roll rotation, yielding the following proposition.
Proposition 1.
Proof.
Based on the definition of \({{\boldsymbol {K}}_{\boldsymbol {\theta }}}_{_{j}}{\boldsymbol {x}}\phantom {\dot {i}\!}\), for the pixel at the location [l _{ x1},l _{ x2},1]^{ T }, \({{\boldsymbol {K}}_{\boldsymbol {\theta }}}_{_{j}}{\boldsymbol {x}}\phantom {\dot {i}\!}\) assigns the value of the pixel located at \({\boldsymbol {H}}_{{{\boldsymbol {\theta }}_{j}}}^{ 1}\left ({\left [l_{x1}, l_{x2}, 1\right ]^{T}}\right)\) of x to the pixel located at [l _{ x1},l _{ x2},1] of \({{\boldsymbol {K}}_{\boldsymbol {\theta }}}_{_{j}}{\boldsymbol {x}}\phantom {\dot {i}\!}\). Based on Eq. (10), \({\boldsymbol {H}}_{{{\boldsymbol {\theta }}_{j}}}^{ 1}\left ({\left [l_{x1}, l_{x2}, 1\right ]^{T}}\right)\) can also be explained as \({\boldsymbol {H}}_{{{\boldsymbol {\theta }}_{j,r}}}^{ 1}{\boldsymbol {H}}_{{{\boldsymbol {\theta }}_{j,t}}}^{ 1}\left ({\left [l_{x1}, l_{x2}, 1\right ]^{T}}\right)\), which means that \({{\boldsymbol {K}}_{{{\boldsymbol {\theta }}_{j}}}}{\boldsymbol {x}} = {{\boldsymbol {K}}_{{{\boldsymbol {\theta }}_{j,r}}}}{{\boldsymbol {K}}_{{{\boldsymbol {\theta }}_{j,t}}}}{\boldsymbol {x}}\phantom {\dot {i}\!}\).
where \({w_{{{\boldsymbol {\theta }}_{j}}}}\phantom {\dot {i}\!}\) denotes the contribution (weight) of pose θ _{ j }. It should be noted that the poses along a real camera motion trajectory form a connected 1D path, and thus, the weights of most poses are zeros. So in Eq. (14), we can only consider the subset of poses with positive weights, \({\mathcal {P}} = \left \{ {{{\boldsymbol {\theta }}_{j}}:{w_{{{\boldsymbol {\theta }}_{j}}}} > 0} \right \}\phantom {\dot {i}\!}\).
Finally, we define two special classes of subsets of \({\mathcal {P}}\): slice and fiber. A slice \({{\mathcal {S}}_{\theta } }\) is defined as \({{\mathcal {S}}_{\theta }} = \{ {{\boldsymbol {\theta }}_{j}} = \{ {\theta _{z,j}},{t_{x,j}},{t_{y,j}}\} : {{\boldsymbol {\theta }}_{j}} \in {\mathcal {P}} {\; \text {and} \;} {\theta _{z,j}} = \theta \}\phantom {\dot {i}\!}\), while a fiber \({{\mathcal {F}}_{\boldsymbol {t}}}\) is defined as \({{\mathcal {F}}_{\boldsymbol {t}}} = \{ {{\boldsymbol {\theta }}_{j}} = \{ {\theta _{z,j}}, {t_{x,j}},{t_{y,j}}\} :{{\boldsymbol {\theta }}_{j}} \in {\mathcal {P}} {\; \text {and} \;}({t_{x,j}},{t_{y,j}}) = {\boldsymbol {t}}\}\phantom {\dot {i}\!}\). Actually, \({\mathcal {P}}\) can be decomposed into a number of nonintersected slices and fibers. In the following, we will introduce how to construct slices and fibers from \({\mathcal {P}}\) and how to reformulate camera shake as the GAC model.
3.3 Slicebased GAC
where \({{\boldsymbol {k}}_{{{\mathcal {S}}_{\theta } }}}\) denotes the slice kernel with respect to the roll angle θ.
It is obvious that this is a GAC model with f _{ θ }(x)=R _{ θ }(x) and g _{ θ }(x)=x. If the range of the roll angles is discretized into n _{ z } intervals, we can see that the number C in slice GAC should be not higher than n _{ z }.
where \({{\boldsymbol {\tilde k}}_{{{\mathcal {S}}_{\theta } }}}\) is the adjoint operator of \({{\boldsymbol {k}}_{{{\mathcal {S}}_{\theta } }}}\) constructed by flipping the \({{\boldsymbol {k}}_{{{\mathcal {S}}_{\theta } }}}\) upsidedown and lefttoright, and \({\boldsymbol {R}}_{\theta }^{T}\) is the strict adjoint operator of the discrete version of R _{ θ }. It is obvious that the adjoint operator in Eq. (17) is also a GAC model and can be efficiently computed.
Finally, we discuss some implementation issues of slice GAC. To implement R _{ θ }, one can simply adopt the Matlab command imrotate. To enhance the efficiency, we maintain a lookup table (LUT) for each discrete roll angle to record the correspondence of the coordinates before and after rotation. By discretizing the range of roll angles into n _{ z } intervals, we precompute and store n _{ z } LUTs. In the continuous case, R _{−θ } is the adjoint operator of R _{ θ }, but in the discrete case, the error caused by discretization and interpolation cannot be overlooked. Thus, instead of using R _{−θ }, we adopt the strictly adjoint operator \({\boldsymbol {R}}_{\theta }^{T}\).
3.4 Fiberbased GAC
where θ _{1} is the minimal roll angle and θ _{ nz } is the maximal roll angle.
We compare the model in Eq. (18) with the 3D camera pose subspace model in [30] for the simulation of blur caused by camera shake along a fiber. Figure 3 shows the blurry images obtained by using these two methods. One can easily see that the difference between Fig. 3 c, d is insignificant, which demonstrates that the model in Eq. (18) can be used for modeling camera shaken image with a series of roll rotations.
It is obvious that this is a GAC model with f _{ t }(x)=IPT(x) and \({{\boldsymbol {g}}_{\boldsymbol {t}}}({\boldsymbol {x}}) = \text {PT}({{\boldsymbol {K}}_{\boldsymbol {\theta }_{\boldsymbol {t}}}}{\boldsymbol {x}})\phantom {\dot {i}\!}\). If the range of the inplane translations is discretized into n _{ x }×n _{ y } intervals, then the number C in fiber GAC should be not higher than n _{ x } n _{ y }.
where \({{\boldsymbol {\tilde w}}_{\boldsymbol {t}}}\) is the adjoint operator of w _{ t }, and PT^{ T } and IPT^{ T } are the adjoint operators of PT and IPT, respectively. To enhance the computational efficiency, two extra LUTs are precomputed to record the correspondence of polar and inverse polar transform, respectively. Moreover, we use the strict adjoint operators of PT and IPT, i.e., PT^{ T } and IPT^{ T }, to avoid the inconsistence caused by discretization and interpolation.
3.5 Hybrid GAC for modeling
Given the pose subset \({\mathcal {P}}\) and the 3D weight matrix W with W(θ _{ z },t _{ x },t _{ y }), the weight of the pose θ=(θ _{ z },t _{ x },t _{ y }), in each iteration, the proposed method first finds a candidate slice \({{\mathcal {S}}_{{{\hat \theta }_{z}}}}\), and a candidate fiber \(\phantom {\dot {i}\!}{\mathcal {F}}_{{{\hat t}_{x}},{{\hat t}_{y}}}\), compare their relative contributions, and then choose the slice or fiber with higher weights. By this way, we can obtain a slice set \(\phantom {\dot {i}\!}\left \{ {\left ({{\theta _{j}},{{\mathcal {S}}_{j}}, {{\boldsymbol {k}}_{j}}} \right):j = 1,...,{n_{s}}} \right \}\) and a fiber set \(\phantom {\dot {i}\!}\left \{ {\left ({{\boldsymbol {t}_{i}},{{\mathcal {F}}_{i}},{{\boldsymbol {w}}_{j}}} \right):i = 1,...,{n_{f}}} \right \}\). As shown in Fig. 4, the proposed greedy algorithm can successfully decompose the camera motion trajectory into two slices and two fibers. The detailed algorithm is summarized in Algorithm 1.
3.6 Discussions
With FFT, the computational complexity of the model in Eq. (22) is O((n _{ s }+n _{ f })n ^{2} log2n). If n _{ s } and n _{ f } are small, GAC would be more efficient than the other methods. Let n _{ z } be the number of intervals for the roll angle. It is reasonable to assume that (n _{ s }+n _{ f })<n _{ z }; otherwise, we can use the pure slicebased GAC model.
To further improve the computational efficiency, the LUT method can be adopted for fast image rotation, polar and inverse polar transform, and n _{ z }+2 LUTs should be precomputed and stored in memory. In [30, 33], a sparse n ^{2}×n ^{2} matrix was constructed for each pose in the 3D camera pose subspace. Compared with the model in [30, 33], GAC can achieve much better tradeoff between memory and computational complexity.
In [32], camera shake blurring is approximated as the sum of R ^{2} uniformly blurry patches, and the computational complexity of the model in [32] is O(R ^{2}(n/R+w)^{2} log2(n/R+w)), where w×w is the size of the blur kernel. Compared with [32], when R and w are higher, GAC would be computationally more efficient, and our experimental results also validates the efficiency of GAC against the model by Hirsch et al. [32]. Moreover, the model in [32] is a locally uniform approximation of the camera shake blurring model, while GAC can be strictly equivalent with the geometric model in [30].
4 GACbased nonuniform deblurring
In this section, by incorporating the proposed GAC forward blur model into existing deblurring model, the GACbased nonuniform deblurring can be efficiently solved. Then, by imposing l _{2,1}norm regularizer on pose weight matrix W, the slice GAC deblurring can approach toward uniform blind deconvolution.
4.1 GACbased deblurring via GAPG
where λ and τ are tradeoff parameters. By substituting the forward blur model with any GAC model, we come to the GACbased nonuniform deblurring method, which can be solved by alternatively updating pose weight matrix W and latent clear image x. By fixing x, we use the method in [33] to update W, while by fixing W, we develop an efficient solution to x based on GAPG algorithm.
By using smaller λ _{max} and η, GAPG algorithm will converge faster. For nonuniform blur, we choose ∥K∥_{2}=1 based on the following proposition.
Proposition 2.
Proof.
where ∥·∥_{1} and ∥·∥_{ ∞ } denote the largest l _{1}norm of the columns and rows, respectively. For the uniform blur procedure b=Kx, where each row of K is a shifted version of blur kernel K, Zuo and Lin had proved that the l _{1}norm of the row of matrix K corresponding to the (i,j)th entry of b is k_{1} [34]. For the nonuniform blur matrix K, (i,j)th entry of b has its own blur kernel k ^{(i,j)}, and consequently, K_{ ∞ }=max_{(i,j)}k ^{(i,j)}_{1}. By the nonnegative constraint k _{ i }≥0,∀i and the normalization constraint \({\sum \nolimits }_{i}{{{k}_{i}}=1}\), l _{1}norm of any blur kernel is 1, i.e., k ^{(i,j)}_{1}=1. Thus, we have K_{ ∞ }=1. Similarly, K_{1}=1 can be derivated from [34]. Thus, the inequality is proved.
The updating of subproblems should be performed several iterations. With the properly chosen Lipschitz matrix, each variable has it own Lipschitz constant, rather than the largest one adopted in APG algorithm, so that the GAPGbased deblurring method has a faster convergence rate.
4.2 Connection with uniform blind deconvolution
where \(\tau _{\theta _{j}}\) is the tradeoff parameter controlling the weight of all slices with angle θ _{ j }, and by setting \(\tau _{\theta _{j}=0}<\tau _{\theta _{i}},\forall i> j\), the slices with larger rotation angle will be gradually inactivated, and naturally, the nonuniform deblurring will approach toward uniform blind deconvolution.
where δ is the positive penalty parameter. For the Wsubproblem, it is also a linear least square problem, which can be efficiently solved [33], and for the W ^{′}subproblem, it is a standard l _{2,1}norm optimization problem, which can be solved by group shrinkage operator [42].
The slice GAC with group sparsity provides a natural connection of nonuniform deblurring and uniform deconvolution. When there are only camera inplane translations, i.e., no roll rotation, the camera shake blurring will be uniform, and the slice GAC can play like the traditional uniform blind deconvolution method. Otherwise, roll rotations will cause the nonuniform blur, where slice GAC can play like nonuniform deblurring method.
5 Experimental results
In this section, we evaluate the performance of proposed GAC deblurring methods. First, three GAC variants, i.e., slicebased GAC (SGAC), fiberbased GAC (FGAC), and hybrid GAC (HGAC), are compared in terms of computational time, memory usage, and visual quality. Due to its superiority over SGAC and FGAC, HGAC is then compared with the stateoftheart geometrical methods, i.e., Whyte et al. [29], Gupta et al. [30], and Hu and Zhang [33], and the stateoftheart EFFbased methods, i.e., Hirsch et al. [32], Xu et al. [16], and Cao et al. [27]. Finally, in the comparison with the stateoftheart uniform deblurring methods [9, 21, 24, 43], slice GAC with GSGAC is adopted. All the experiments are conducted on a PC with Intel Core i52400 3.10 GHz CPU and 16 G RAM memory, and the proposed method is implemented in Matlab.
5.1 Comparison of three GAC variants
Running time (s) and peak memory usage (GB) of different GAC models
Image ID  Image size  Time  Memory  

FGAC  SGAC  HGAC  FGAC  SGAC  HGAC  
Books  512×768  947  698  543  1.71  1.75  1.60 
Butcher Shop  401×601  486  403  351  0.95  0.97  0.97 
Statue  710×523  979  694  664  1.39  1.49  1.51 
From the restoration quality shown as Fig. 6, HGAC and SGAC perform better than FGAC in recovering more details and achieving more visual plausible deblurring quality. Due to its superiority over both SGAC and FGAC, HGAC is adopted in the following comparison with nonuniform deblurring methods.
5.2 Comparison with geometrical methods
Running time (s) of the four geometrical methods
Image ID  Image size  Whyte  Gupta  Hu and  GAC 

et al.  et al.  Yang  
Cabin  512×768  98,872  7789  2113  514 
Carinyard  512×768  87,527  8829  7495  531 
DimPetrolStation  512×768  88,767  8512  2512  596 
PetrolStation  406×679  89,673  7982  1402  308 
SixBooks  512×768  96,732  7356  2601  572 
Peak memory usage (GB) of the four geometrical methods
Image ID  Image size  Whyte  Gupta  Hu and  GAC 

et al.  et al.  Yang  
Cabin  512×768  1.00  14.00  10.06  2.06 
Carinyard  512×768  0.98  14.04  10.20  1.93 
DimPetrolStation  512×768  0.91  13.98  10.20  1.03 
PetrolStation  406×679  0.92  14.01  8.25  1.44 
SixBooks  512×768  0.91  14.05  11.4  1.95 
5.3 Comparison with nongeometrical methods
Running time (s) and peak memory usage (GB) comparison of Cao et al. [27] and hybrid GAC
Image ID  Image size  Time  Memory  

Xu et al.  Cao et al.  GAC  Xu et al.  Cao et al.  GAC  
Butcher shop  401 × 601  922  1520  351  2.24  0.43  1.50 
Coke  680 × 731  1434  800  952  4.54  0.66  3.61 
Statue  710 × 523  1334  4236  716  2.90  0.48  1.67 
Books  512 × 768  4698  2053  596  2.94  0.54  2.03 
Sixbooks  512 × 768  2270  3786  572  2.91  0.55  1.95 
Dimpetrolstation  512 × 768  1842  4582  596  3.57  0.53  1.03 
5.4 Comparison with uniform deblurring methods
6 Conclusions
In this paper, by designing GAC model to geometrically represent camera shake blurring, nonuniform deblurring is efficiently addressed. Since the slices and fibers decomposed from the camera motion trajectory can be formulated as convolution, all the GAC methods can exploit FFT for efficient optimization. Compared with the methods in [30, 33], the proposed method only needs several FFTs and to store several basis convolution kernels and lookuptables in memory. By incorporating group sparsity into pose weight matrix, the GACbased deblurring methods can also work like uniform blind deconvolution, better handling uniform blurry images. Compared with nonuniform deblurring methods, GAC method has a much lower peak memory usage and is much more efficient. Compared with uniform deblurring methods, GAC method also can achieve satisfactory deblurring results.
Declarations
Acknowledgements
This work is supported in part by NSFC grants (61271093, 61471146, 61173086) and the program of ministry of education for new century excellent talents (NCET120150).
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
 S McCloskey, Y Ding, J Yu, Design and estimation of coded exposure point spread functions. IEEE Trans. Pattern Anal. Mach. Intell.34(10), 2071–2077 (2012).View ArticleGoogle Scholar
 S McCloskey, in ECCV. Velocitydependent shutter sequences for motion deblurring (SpringerCrete, Greece, 2010).Google Scholar
 A Levin, R Fergus, F Durand, WT Freeman, Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph.26(3), 70 (2007).View ArticleGoogle Scholar
 N Joshi, SB Kang, CL Zitnick, R Szeliski, Image deblurring using inertial measurement sensors. ACM Trans. Graph.29(4), 30 (2010).View ArticleGoogle Scholar
 YW Tai, H Du, MS Brown, S Lin, Correction of spatially varying image and video motion blur using a hybrid camera. IEEE Trans. Pattern Anal. Mach. Intell.32(6), 1012–1028 (2010).View ArticleGoogle Scholar
 YW Tai, N Kong, S Lin, SY Shin, in CVPR. Coded exposure imaging for projective motion deblurring (IEEESan Francisco, USA, 2010), pp. 2408–2415.Google Scholar
 R Fergus, B Singh, A Hertzmann, ST Roweis, WT Freeman, Removing camera shake from a single photograph. ACM Trans. Graph.25(3), 787–794 (2006).View ArticleGoogle Scholar
 A Levin, Y Weiss, F Durand, WT Freeman, in CVPR. Understanding and evaluating blind deconvolution algorithms (IEEEMiami, USA, 2009), pp. 1964–1971.Google Scholar
 Q Shan, J Jia, A Agarwala, Highquality motion deblurring from a single image. ACM Trans. Graph.27(3), 73 (2008).View ArticleGoogle Scholar
 D Krishnan, T Tay, R Fergus, in CVPR. Blind deconvolution using a normalized sparsity measure (IEEEColorado Springs, USA, 2011), pp. 233–240.Google Scholar
 TS Cho, S Paris, BKP Horn, WT Freeman, in CVPR. Blur kernel estimation using the radon transform (IEEEColorado Springs, USA, 2011), pp. 241–248.Google Scholar
 Z Hu, MH Yang, in ECCV, 7576. Good regions to deblur (SpringerFirenze, Italy, 2012), pp. 59–72.Google Scholar
 J Miskin, DJ MacKay, in Adv. Ind. Compon. Anal. Ensemble learning for blind image separation and deconvolution (SpringerEdinburgh, UK, 2000), pp. 123–141.View ArticleGoogle Scholar
 A Levin, Y Weiss, F Durand, WT Freeman, in CVPR. Efficient marginal likelihood optimization in blind deconvolution (IEEEColorado Springs, USA, 2011), pp. 2657–2664.Google Scholar
 SD Babacan, R Molina, MN Do, AK Katsaggelos, in ECCV, 7577. Bayesian blind deconvolution with general sparse image priors (SpringerFirenze, Italy, 2012), pp. 341–355.Google Scholar
 L Xu, S Zheng, J Jia, in CVPR. Unnatural l0 sparse representation for natural image deblurring (IEEEColumbus, USA, 2013), pp. 1107–1114.Google Scholar
 J Kotera, F Šroubek, P Milanfar, in Compt. Anal. Images Patterns, 8048. Blind deconvolution using alternating maximum a posteriori estimation with heavytailed priors (SpringerYork, UK, 2013), pp. 59–66.View ArticleGoogle Scholar
 S Cho, S Lee, Fast motion deblurring. ACM Trans. Graph.28(5), 145 (2009).View ArticleGoogle Scholar
 N Joshi, R Szeliski, D Kriegman, in CVPR. PSF estimation using sharp edge prediction (IEEEAnchorage, USA, 2008), pp. 1–8.Google Scholar
 L Sun, S Cho, J Wang, J Hays, in ICCP. Edgebased blur kernel estimation using patch priors (IEEEClujNapoca, Romania, 2013), pp. 1–8.Google Scholar
 L Xu, J Jia, in ECCV. Twophase kernel estimation for robust motion deblurring (SpringerCrete, Greece, 2010), pp. 157–170.Google Scholar
 K Schelten, S Nowozin, J Jancsary, C Rother, S Roth, in WACV. Interleaved regression tree field cascades for blind image deconvolution (IEEEHawaii, USA, 2015).Google Scholar
 CJ Schuler, M Hirsch, S Harmeling, B Schölkopf, in NIPS. Learning to deblur (NIPSMontréal, Canada, 2014).Google Scholar
 W Zuo, D Ren, S Gu, L Lin, L Zhang, in CVPR. Discriminative learning of iterationwise priors for blind deconvolution (IEEEBoston, USA, 2015), pp. 3232–3240.Google Scholar
 Z Hu, MH Yang, Learning good regions to deblur images. Int. J. Comp. Vis.115(3), 345–362 (2015).View ArticleMathSciNetGoogle Scholar
 H Ji, K Wang, in CVPR. A twostage approach to blind spatiallyvarying motion deblurring (IEEERhode Island, USA, 2012), pp. 73–80.Google Scholar
 X Cao, W Ren, W Zuo, X Guo, H Foroosh, Scene text deblurring using textspecific multiscale dictionaries. IEEE Trans. Image Process.24(4), 1302–1314 (2015).View ArticleMathSciNetGoogle Scholar
 YW Tai, P Tan, MS Brown, RichardsonLucy deblurring for scenes under a projective motion path. IEEE Trans. Pattern Anal. Mach. Intell.33(8), 1603–1618 (2011).View ArticleGoogle Scholar
 O Whyte, J Sivic, A Zisserman, J Ponce, Nonuniform deblurring for shaken images. Int. J. Comp. Vis.98(2), 168–186 (2012).View ArticleMathSciNetMATHGoogle Scholar
 A Gupta, N Joshi, CL Zitnick, M Cohen, B Curless, in ECCV, 6311. Single image deblurring using motion density functions (SpringerCrete, Greece, 2010), pp. 171–184.Google Scholar
 M Hirsch, S Sra, B Schölkopf, S Harmeling, in CVPR. Efficient filter flow for spacevariant multiframe blind deconvolution (IEEESan Francisco, USA, 2010), pp. 607–614.Google Scholar
 M Hirsch, CJ Schuler, S Harmeling, B Schölkopf, in ICCV. Fast removal of nonuniform camera shake (IEEEBarcelona, Spain, 2011), pp. 463–470.Google Scholar
 Z Hu, MH Yang, in BMVC. Fast nonuniform deblurring using constrained camera pose subspace (BMVAGuildford, UK, 2012).Google Scholar
 W Zuo, Z Lin, A generalized accelerated proximal gradient approach for totalvariationbased image restoration. IEEE Trans. Image Process.20(10), 2748–2759 (2011).View ArticleMathSciNetGoogle Scholar
 R Köhler, M Hirsch, B Mohler, B Schölkopf, S Harmeling, in ECCV. Recording and playback of camera shake: benchmarking blind deconvolution with a realworld database (SpringerFirenze, Italy, 2012), pp. 27–40.Google Scholar
 H Deng, W Zuo, H Zhang, D Zhang, An additive convolution model for fast restoration of nonuniform blurred images. Int. J. Comp. Math.91(11), 2446–2466 (2014).View ArticleMathSciNetMATHGoogle Scholar
 I Daubechies, M Defrise, C De Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Comm. Pure. Appl. Math.57(11), 1413–1457 (2004).View ArticleMathSciNetMATHGoogle Scholar
 JM BioucasDias, MA Figueiredo, A new twist: twostep iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process.16(12), 2992–3004 (2007).View ArticleMathSciNetGoogle Scholar
 A Beck, M Teboulle, Fast gradientbased algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process.18(11), 2419–2434 (2009).View ArticleMathSciNetGoogle Scholar
 S Stančin, S Tomažič, Angle estimation of simultaneous orthogonal rotations from 3d gyroscope measurements. Sensors. 11(9), 8536–8549 (2011).View ArticleGoogle Scholar
 R Matungka, YF Zheng, RL Ewing, Image registration using adaptive polar transform. IEEE Trans. Image Process.18(10), 2340–2354 (2009).View ArticleMathSciNetGoogle Scholar
 J Yang, W Yin, Y Zhang, Y Wang, A fast algorithm for edgepreserving variational multichannel image restoration. SIAM J. Imag. Sci.2(2), 569–592 (2009).View ArticleMathSciNetMATHGoogle Scholar
 J Pan, Z Hu, Z Su, MH Yang, in CVPR. Deblurring text images via l0regularized intensity and gradient prior (IEEEColumbus, USA, 2014), pp. 2901–2908.Google Scholar
 S Harmeling, M Hirsch, B Schölkopf, in NIPS. Spacevariant singleimage blind deconvolution for removing camera shake (NIPSVancouver, Canada, 2010), pp. 829–837.Google Scholar