3.1 Algorithm description
To address the above issue, we propose a novel greedy algorithm for blind cosparse reconstruction, where the cosupports are refined iteratively and the information on cosparsity is extracted automatically. The proposed CSMP algorithm, as shown in Algorithm 1, is composed of two processes, namely, the forward and backward processes. The forward process estimates the cosparsity, constructs the cosupport starting from a cosupport with all rows in Ω, and updates the measurement residual simultaneously. The procedure ends with a backward process that tries to add the rows in Ω with a smaller correlation until the terminating condition is reached. The terminating condition for CSMP is controlled by a threshold, which ensures that the estimated cosparsity is fairly close to the actual one and the target signal has been well reconstructed. The main steps of CSMP are summarized in Algorithm 1.
In Algorithm 1, cosupp(x) = {i: Ω
i
x = 0}, cosupp(x, l
est) returns the index set of l
est smallest (in absolute value) elements in Ω
x, 2l
est − p is the size of |Γ| and for a vector α, function Min(α, 2l
est − p) returns the 2l
est − p indices corresponding to the 2l
est − p smallest values of α, index(Ω
x, q) returns q elements from the (l
est + 1)-th to (l
est + q)-th smallest (in absolute value) rows in Ω
x, and Γ, Λ
Δ, \( {\tilde{\varLambda}}^k \) and Λ
k are subsets of {1, 2, ⋯, p}, and ⌈s/q⌉ returns the smallest integer which is larger than s / q.
The CSMP algorithm adopts a stagewise approach [6] to estimate the real cosparsity in each stage in the forward process, which only requires the step size s to be set in initialization. Here, l is defined as the real cosparsity of the original signal, and s should not be larger than d − l normally [11]. The initial cosupport Λ
0 has the maximum number of rows in Ω that enables CSMP to recover the signal. To avoid under-estimation of the cosparsity, a safe choice is s = 1 if l is unknown. Nevertheless, there is a tradeoff between s and the recovery speed as a smaller s requires more stages. We can see that GAP and ASP/ACoSaMP are the special cases of the proposed algorithm when CSMP has a step size of s = 1 and s = d − l, respectively.
Suppose CSMP has a cosparsity of l
est in the forward process. With this cosparsity, the candidate set is constructed by selecting some rows in Ω with smaller correlations. Here, we should explain the relation between greedy synthesis algorithms and their analysis counterparts. Given two vectors v
1, v
2 ∈ ℜ
n such that Λ
1 = cosupp(Ω
v
1) and Λ
2 = cosupp(Ω
v
2). Assuming that ‖Λ
1‖0 ≤ (l
est)1 and ‖Λ
2‖0 ≤ (l
est)2, it holds that ‖Λ
1 ∩ Λ
2‖0 ≥ (l
est)1 + (l
est)2 − p. For the case ‖Λ
1‖0 = ‖Λ
2‖0 = l
est, we have 2l
est − p ≤ ‖Λ
1 ∩ Λ
2‖0 ≤ l
est. So 2l
est − p is a reasonable size of the candidate set for CSMP, which corresponds to the candidate set size of 2k of CoSaMP in the synthesis model. Denoting T
1 = supp(Ω
v
1) and T
2 = supp(Ω
v
2), it is clear that supp(Ω(v
1 + v
2)) ⊆ T
1 ∪ T
2. Noticing that supp(⋅) = cosupp(⋅)C, we get cosupp(Ω(v
1 + v
2)) ⊇ (T
1 ∪ T
2)C = T
1
C ∩ T
2
C = Λ
1 ∩ Λ
2, where the superscript C denotes the complementary set. This implies that the union of the supports in the synthesis case is parallel to the intersection of the cosupports in the analysis case.
Now, with the candidate set, we begin to construct the cosupport and update the measurement residual. The rows in the analysis operator, which correspond to the smallest l
est components in the temporarily estimated signal (calculated in Step 4 in Algorithm 1), are used to form the cosupport. Then, we approximate the signal and update the measurement residual of the current iteration with this cosupport. The backtracking strategy [12, 19], which can be described as not only selecting rows that match the current residual signal better from the candidate set in each iteration but also excluding the other rows from the cosupport, provides the basis for constructing a more accurate cosupport and obtaining a smaller measurement residual. Here, \( \mathrm{index}\left(\boldsymbol{\Omega} {\widehat{\boldsymbol{x}}}_{\mathrm{temp}},q\right) \) is reserved for sharing in the backward process. An efficient mechanism is required for stage switching when each stage finishes till l
est < l. This can be performed if the current measurement residual energy no longer decreases when compared with that in the last iteration. From Step 9 in Algorithm 1, we can see that the algorithm will run for some iterations with the same cosparsity l
est until it reaches the stage switching condition. The proposed algorithm will perform with a cosparsity of l
est − s in the next stage. The forward process will not stop until the measurement residual reaches a pre-set threshold such as ‖y
r
‖2/‖y‖2 ≤ 10‐ 6 for the noiseless case or ‖y
r
‖2 ≤ ε for the noisy case.
In the backward process, the algorithm tries to further increase the cosparsity by adding the less used rows into the cosupport. These q rows have been chosen in the last iteration of the forward process. The choice of the value of q can be made in terms of the value of s. As a rule of thumb, a small q is chosen when s is relatively small, and likewise, a greater q should be chosen if s is large. Typically, in our experiments, we choose q = 1 when s = 10, but we select q = 50 when s is in the order of thousands. With this strategy, we can get a more accurate cosupport for signal approximation. The iterations stop as soon as the measurement residual reaches the terminating threshold used in the forward process. The backward process needs to repeat less than ⌈s/q⌉ times since it is enough for the cosparsity to change from l
est to l
est + s in this process.
3.2 Relaxed versions for high-dimensional problems
In CSMP, the constrained minimization problem
$$ \underset{\boldsymbol{x}}{min}\kern0.28em \parallel \boldsymbol{y}-\boldsymbol{Mx}{\parallel}_2^2\kern1.2em \mathrm{s}.\mathrm{t}.\kern1em {\boldsymbol{\Omega}}_{\varLambda}\boldsymbol{x}=\mathbf{0} $$
(7)
is hard to solve for high-dimensional signals, and we propose to replace it with the minimization of the following cost function in Steps 4, 6, 12, and 14 in Algorithm 1:
$$ \parallel \boldsymbol{y}-\boldsymbol{Mx}{\parallel}_2^2+\lambda \kern0.5em \parallel {\boldsymbol{\Omega}}_{\varLambda}\boldsymbol{x}{\parallel}_2^2 $$
(8)
where λ is a relaxation constraint and we choose λ = 0.001 as in [12] in our experiments. This results in a relaxed version of the algorithm. We refer hereafter to this version as relaxed CSMP (RCSMP).
3.3 Theoretical performance analysis
This section describes our theoretical analysis of the behavior of CSMP for the cosparse model in both noiseless and noisy cases. Because the proposed algorithm has a similar strategy of backtracking which is used in ASP, the proofs are mainly based on the proof framework of ASP/ACoSaMP. The following theorems are formed in parallel with those in [12], except for the unknown cosparsity and the initial cosupport and measurement vectors.
To show the ability of exact and stable recovery of cosparse signals by the CSMP algorithm, we define all variables involved in the process of signal reconstruction as follows:
Definition 3.1 [12] Let \( {\boldsymbol{Q}}_{\varLambda }=\boldsymbol{I}-{\boldsymbol{\Omega}}_{\varLambda}^{+}{\boldsymbol{\Omega}}_{\varLambda } \) be the orthogonal projection onto Ω
Λ, where Ω
Λ is a sub-matrix of the analysis operator Ω ∈ ℜp × d and Λ is a subset of {1, 2, ⋯, p}. A cosupport Ŝ
l
which implies a near-optimal projection is defined as
$$ \hat{S}_{l}(\text{\textbf{\textit{v}}})={\underset{\varLambda\in \mathrm{L}_{l}}{\text{argmin}}}\|v-\text{\textbf{\textit{Q}}}_{\varLambda}\text{\textbf{\textit{v}}}\|^{2}_{2} $$
(9)
where v ∈ ℜ
d and \( \mathrm{L}_{l}=\left\{\varLambda\in\left[1,2,\cdots,p\right],\mid\varLambda\mid\geq l \right\} \)
l-cosparse cosupports.
Definition 3.2 (Problem p) [12] Consider a measurement vector y ∈ ℜm such that y = Mx + e, where x ∈ ℜd is l-cosparse, M ∈ ℜm × d is a measurement matrix, and e ∈ ℜm is a bounded additive noise. The largest singular value of M is σ
M
and its Ω
-RIP constant is δ
l
. The analysis operator Ω ∈ ℜp × d is given and fixed. Define C
l
to be the fraction of the largest and the smallest eigenvalues (which are not zero) of the sub-matrix composed of l rows from Ω. Assume that Ŝ
l
= cosupp(Ω
x,l). According to Definition 3.1, the cosupport of x is a near-optimal projection. Our task is to recover x from y. The recovery result is denoted by \( \widehat{\boldsymbol{x}} \).
Now, we give the guarantees on exact recovery and stability of CSMP for recovering the cosparse signals.
Theorem 3.3 (Convergence for cosparse recovery) Consider the problem p, and suppose that there exists γ > 0 such that
$$ \left(1+{C}_{2{l}_q-p}\right)\left(1-\left({\scriptscriptstyle \frac{C_{2{l}_q-p}}{\left(1+{\gamma}^2\right)}}-\left({C}_{2{l}_q-p}-1\right){\sigma}_{\boldsymbol{M}}^2\right)\right)<1, $$
(10)
then there exists
\( \delta \left({C}_{2{l}_q-p},{\sigma}_{\boldsymbol{M}}^2,\gamma \right)>0, \)
whenever
\( {\delta}_{4{l}_q-3p}\le \delta \left({C}_{2{l}_q-p},{\sigma}_{\boldsymbol{M}}^2,\gamma \right), \)
such that the k-th iteration of the algorithm satisfies
$$ \parallel \boldsymbol{x}-{\widehat{\boldsymbol{x}}}^k{\parallel}_2\le \frac{1+{\delta}_{2{l}_q-p}}{1-{\delta}_{2{l}_q-p}}{\rho}_1{\rho}_2\parallel \boldsymbol{x}-{\widehat{\boldsymbol{x}}}^{k-1}{\parallel}_2+\left(\frac{1+{\delta}_{2{l}_q-p}}{1-{\delta}_{2{l}_q-p}}\left({\eta}_1+{\rho}_1{\eta}_2\right)+\frac{2}{1-{\delta}_{2{l}_q-p}}\right)\parallel \boldsymbol{e}{\parallel}_2, $$
(11)
where
$$ \begin{array}{ll}{\eta}_1\triangleq \kern1em & \frac{\sqrt{{\scriptscriptstyle \frac{2+{C}_{l_q}}{1+{C}_{l_q}}}+2\sqrt{C_{l_q}}+{C}_{l_q}}\sqrt{1+{\delta_3}_{l_q-2p}}}{1-{\delta_4}_{l_q-3p}},\kern1em \\ {}{\eta}_2^2\triangleq \kern1em & \left(\frac{1+{\delta_3}_{l_q-2p}}{\gamma \left(1+\alpha \right)}+\frac{\left(1+{\delta_2}_{l_q-p}\right){C_2}_{l_q-p}}{\gamma \left(1+\alpha \right)\left(1+\gamma \right)}+\frac{\left({C_2}_{l_q-p}-1\right)\left(1+\gamma \right){\sigma}_{\boldsymbol{M}}^2}{\left(1+\alpha \right)\left(1+\gamma \right)\gamma}\right),\kern1em \\ {}{\rho}_1^2\triangleq \kern1em & \frac{1+2{\delta_4}_{l_q-3p}\sqrt{C_{l_q}}+{C}_{l_q}}{1-{\delta}_{4{l}_q-3p}^2}\kern1em \\ {}{\rho}_2^2\triangleq \kern1em & 1-{\left(\sqrt{{\delta_4}_{l_q-3p}}-\sqrt{\frac{{C_2}_{l_q-p}}{{\left(1+\gamma \right)}^2}{\left(1-\sqrt{{\delta_2}_{l_q-p}}\right)}^2-\left({C_2}_{l_q-p}-1\right)\left(1+{\delta_2}_{l_q-p}\right){\sigma}_{\boldsymbol{M}}^2}\right)}^2\kern1em \end{array} $$
and
$$ \alpha =\frac{\sqrt{\delta_{4{l}_q-3p}}}{\sqrt{{\scriptscriptstyle \frac{C_{2{l}_q-p}}{{\left(1+\gamma \right)}^2}}{\left(1-{\delta}_{2{l}_q-p}\right)}^2-\left({C}_{2{l}_q-p}-1\right)\left(1+{\delta}_{2{l}_q-p}\right){\sigma}_{\boldsymbol{M}}^2}-\sqrt{\delta_{4{l}_q-3p}}} $$
(12)
Moreover, when
\( {\rho}_1^2{\rho}_2^2<1, \)
the iteration converges. The constant γ gives a tradeoff between satisfying the theorem conditions and the noise level, and the conditions for the noiseless case are achieved when γ tends to zero.
Theorem 3.4 (Exact recovery for cosparse signals) Consider the problem p when ‖e‖2 = 0. Let l
s
= d − s⌈(d − l)/s⌉ and l
q
= l
s
+ q⌊(l − l
s
)/q⌋. If the measurement matrix
M
satisfies the
Ω-RIP
with parameter
\( {\delta}_{4{l}_q-3p}\le \delta \left({C}_{2{l}_q-p},{\sigma}_{\boldsymbol{M}}^2,\gamma \right), \)
where
\( {C}_{2{l}_q-p} \)
and γ are as in Theorem 3.3 and
\( \delta \left({C}_{2{l}_q-p},{\sigma}_{\boldsymbol{M}}^2,\gamma \right) \)
is a constant guaranteed to be greater than zero whenever (
10) is satisfied, the CSMP algorithm guarantees an exact recovery of
x
from
y
via a finite number of iterations.
The proof is mainly based on the following lemma:
Lemma 3.5
If the measurement matrix
M
satisfies the
Ω-RIP
with the same conditions as in Theorem 3.4, then
-
The ⌈(d − l)/s⌉ + ⌊(l − l
s
)/q⌋ − th stage of the algorithm is equivalent to the ASP algorithm with estimated cosparsity l
q
, except that they have different initial cosupports and initial measurement vectors.
-
CSMP recovers the target signal exactly after completing the ⌈(d − l)/s⌉ + ⌊(l − l
s
)/q⌋ − th
stage.
In the Appendix, Lemma 3.5 is proved in detail.
Lemma 3.5 describes that CSMP has a process of signal reconstruction that is equivalent to ASP, and could complete the exact recovery of the cosparse signals in a finite number of stages. To complete the proof, it is sufficient to show that the CSMP algorithm never gets stuck at any iteration of either stage, i.e., it takes a finite number of iterations up to ⌈(d − l)/s⌉ + ⌊(l − l
s
)/q⌋ stages. At each stage, the cosupport (whose size is assumed to be l
est) adds and discards some rows of Ω, and the number of rows is fixed and finite. Hence, there are a finite number of combinations, at most, \( \left(\begin{array}{c}\hfill d\hfill \\ {}\hfill {l}_{\mathrm{est}}\hfill \end{array}\right), \) where d is the length of the signal. Thus, if CSMP takes an infinite number of iterations in this stage, the construction of cosupport would be repeated after at most \( \left(\begin{array}{c}\hfill d\hfill \\ {}\hfill {l}_{\mathrm{est}}\hfill \end{array}\right) \) iterations. Hence, Theorem 3.6 follows.
Lemma 3.6 (Stability for cosparse recovery) Consider the problem p. If (10) holds and
\( {\delta}_{4{l}_q-3p}\le \delta \left({C}_{2{l}_q-p},{\sigma}_{\boldsymbol{M}}^2,\gamma \right) \)
, where γ is as in Theorem 3.3 and
\( \delta \left({C}_{2{l}_q-p},{\sigma}_{\boldsymbol{M}}^2,\gamma \right) \)
is a constant guaranteed to be greater than zero whenever
\( \frac{\left({C}_{2{l}_q-p}^2-1\right){\sigma}_{\boldsymbol{M}}^2}{C_{2{l}_q-p}^2}<1 \)
is satisfied, then for any
$$ k\ge \left|\frac{ \log \left(\parallel \boldsymbol{x}{\parallel}_2/\parallel \boldsymbol{e}{\parallel}_2\right)}{ \log \left(1/{\scriptscriptstyle \frac{1+{\delta}_{2{l}_q-p}}{1-{\delta}_{2{l}_q-p}}}{\rho}_1{\rho}_2\right)}\right|, $$
we have
$$ \parallel \widehat{\boldsymbol{x}}-\boldsymbol{x}{\parallel}_2\le \left(1+\frac{1-{\left({\scriptscriptstyle \frac{1+{\delta}_{2{l}_q-p}}{1-{\delta}_{2{l}_q-p}}}{\rho}_1{\rho}_2\right)}^k}{1-{\scriptscriptstyle \frac{1+{\delta}_{2{l}_q-p}}{1-{\delta}_{2{l}_q-p}}}{\rho}_1{\rho}_2}\cdot \left({\scriptscriptstyle \frac{1+{\delta}_{2{l}_q-p}}{1-{\delta}_{2{l}_q-p}}}\left({\eta}_1+{\rho}_1{\eta}_2\right)+\frac{2}{1-{\delta}_{2{l}_q-p}}\right)\right)\parallel \boldsymbol{e}{\parallel}_2 $$
(13)
implying that CSMP leads to a stable recovery. The constants η
1
, η
2
, ρ
1
, and ρ
2
are the same as in Theorem 3.3.
Similarly, the proof of Lemma 3.6 is based on Lemma 3.5 and the corresponding theorems of ASP algorithm in [12], and we omit the detailed proof here.
The above theorems are sufficient conditions of CSMP for exact recovery and stability. They are slightly more restrictive than the corresponding results of ASP algorithms because the true cosparsity level l is always larger than or equal to the estimated one l
q
. This may be regarded as an additional cost for not having precise information of cosparsity. On the other hand, the proofs also show that these sufficient conditions may not be optimal or tight enough because they only consider the final stage and ignore the influence of previous stages on the performance of the algorithm.