This section presents the convergence property of Algorithm 1.
First, let us define the following schemes with regard to the tth iteration of the second iteration statements in Algorithm 1:
$$\begin{array}{*{20}l} \left\{ \begin{array}{lllll} d_{i,j}^{(t)}&\!\,=\,\!& h_{\beta,\epsilon_{i}}(\boldsymbol{x}_{j},\boldsymbol{x}_{i},\boldsymbol{z}_{i,j}^{(t)}) & \text{for} \ (i,j)\in\mathcal{I}^{2},\\ \boldsymbol{Z}_{i}^{(t+1)}&\!\,=\,\!&\boldsymbol{\mathcal{T}}_{r,\gamma}\left\{(\boldsymbol{X}-\boldsymbol{x}_{i}\boldsymbol{1}_{N}^{T})\boldsymbol{D}_{i}^{(t)}\right\} &\text{for} \ i\in\mathcal{I}, \end{array} \right. \end{array} $$
(18)
u(t1,t2) behaves as:
$$u(t_{1},t_{2})=g_{\beta,\gamma,r}\left(\boldsymbol{X},\left\{\boldsymbol{Z}_{i}^{(t_{1})}\right\}_{i\in \mathcal{I}}, \left\{d_{i,j}^{(t_{2})}\right\}_{(i,j)\in \mathcal{I}^{2}}\right),$$
for t,t1,t2≥0 and a given \( \boldsymbol {X} \in {\mathbb {R}^{{M}\times {N}}}\).
Lemma 1
For t≥0 and a given \(\boldsymbol {X}\in {\mathbb {R}^{{M}\times {N}}}\), the \( d_{i,j}^{(t)} \) generated by the update schemes (18) satisfies:
$$\begin{array}{*{20}l} u(t,t)- u(t+1,t+1) \geq \ \ \ \\ \ \ \ \frac{1}{2}\sum_{(i,j)\in\mathcal{I}^{2}} \|\boldsymbol{x}_{j}-\boldsymbol{x}_{i}\|_{2}^{2}\left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right)^{2}. \end{array} $$
Proof
u(t1,t2) satisfies u(t,t)≥u(t+1,t)≥u(t+1,t+1)≥⋯≥−βN2 for t≥0, since \(d_{i,j}^{(t)}= h_{\beta,\epsilon _{i}}\left (\boldsymbol {x}_{j},\boldsymbol {x}_{i},\boldsymbol {z}_{i,j}^{(t)}\right)\) is the closed-form optimal solution of the convex quadratic-minimization problem with linear constraints for fixed \(\boldsymbol {Z}_{i}^{(t)}\), and \(\boldsymbol {Z}_{i}^{(t+1)}\) represents the optimal solution for fixed \(d_{i,j}^{(t)}\) (from Theorem 1 of [5]). Each \( d_{i,j}^{(t+1)} \) satisfies the following KKT condition of problem (16) with \(\left (\|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2}-\epsilon _{i}\right){d_{i,j}^{(t+1)}}\leq 0,{d_{i,j}^{(t)}}-1\leq 0,-{d_{i,j}^{(t+1)}}\leq 0\) for \( (i,j)\in \mathcal {I}^{2}\):
$$\begin{array}{*{20}l} \left\{ \begin{array}{l} \|\boldsymbol{x}_{j}-\boldsymbol{x}_{i}\|_{2}^{2}d_{i,j}^{(t+1)}-\langle \boldsymbol{x}_{j}-\boldsymbol{x}_{i}, \boldsymbol{z}_{i,j}^{(t+1)}\rangle -\beta\\ =\mu_{1,i,j}^{(t+1)}\left(\|\boldsymbol{x}_{j}-\boldsymbol{x}_{i}\|_{2}^{2}-\epsilon_{i}\right)+\mu_{2,i,j}^{(t+1)}-\mu_{3,i,j}^{(t+1)},\\ \mu_{1,i,j}^{(t+1)}\left(\|\boldsymbol{x}_{j}-\boldsymbol{x}_{i}\|_{2}^{2}-\epsilon_{i}\right)d_{i,j}^{(t+1)}=0,\\ \mu_{2,i,j}^{(t+1)}(d_{i,j}^{(t+1)}-1) =0,\\ \mu_{3,i,j}^{(t+1)}(-d_{i,j}^{(t+1)}) =0,\\ \mu_{1,i,j}^{(t+1)},\mu_{2,i,j}^{(t+1)},\mu_{3,i,j}^{(t+1)},\geq 0 \end{array} \right. \end{array} $$
where \(\mu _{1,i,j}^{(t+1)},\mu _{2,i,j}^{(t+1)}\) and \(\mu _{3,i,j}^{(t+1)}\) denote KKT multipliers for \( d_{i,j}^{(t+1)} \). Therefore, u(t1,t2) satisfies:
$$\begin{array}{*{20}l} &u(t+1,t)- u(t+1,t+1)\\ &=\!\sum_{(i,j)\in\mathcal{I}^{2}} \left\{ \begin{array}{l} \frac{1}{2}\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^{2}\left({d_{i,j}^{(t)}}^2-{d_{i,j}^{(t+1)}}^{2}\right)\\ -\langle\boldsymbol{x}_j-\boldsymbol{x}_i,\boldsymbol{z}_{i,j}^{(t+1)}\rangle\left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right)\\ -\beta \left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right) \end{array} \right\}\\ &=\!\sum_{(i,j)\in\mathcal{I}^{2}} \frac{1}{2}\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^{2}\left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right)^{2}\\ &+\!\sum_{(i,j)\in\mathcal{I}^{2}} \begin{array}{l} \left(d_{i,j}^{(t)}-d_{i,j}^{(t+1)}\right)\left(\begin{array}{l} \|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^2d_{i,j}^{(t+1)}\\-\langle\boldsymbol{x}_j-\boldsymbol{x}_i,\boldsymbol{z}_{i,j}^{(t+1)}\rangle\\ -\beta \end{array} \right)\\ \end{array} \\ &=\!\sum_{(i,j)\in\mathcal{I}^{2}} \frac{1}{2}\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^{2}\left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right)^{2}\\ &-\!\sum_{(i,j)\in\mathcal{I}^{2}} \begin{array}{l} \left(d_{i,j}^{(t)}-d_{i,j}^{(t+1)}\right)\left\{ \begin{array}{l} \mu_{1,i,j}^{(t+1)}\left(\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^2-\epsilon_{i}\right)\\ +\mu_{2,i,j}^{(t+1)}-\mu_{3,i,j}^{(t+1)} \end{array} \right\}\\ \end{array} \\ &=\!\sum_{(i,j)\in\mathcal{I}^{2}} \left\{ \begin{array}{l} \frac{1}{2}\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^{2}\left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right)^{2}\\ +\mu_{1,i,j}^{(t+1)}\left(\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^2-\epsilon_{i}\right)\left({d_{i,j}^{(t+1)}}-{d_{i,j}^{(t)}}\right)\\ +\mu_{2,i,j}^{(t+1)}\left({d_{i,j}^{(t+1)}}-1-{d_{i,j}^{(t)}}+1\right)\\ +\mu_{3,i,j}^{(t+1)}\left(-{d_{i,j}^{(t+1)}}+{d_{i,j}^{(t)}}\right)\\ \end{array} \right\}, \end{array} $$
Since \(\left (\|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2}-\epsilon _{i}\right){d_{i,j}^{(t+1)}}=0\) if \(\mu _{1,i,j}^{(t+1)}>0\), \( {d_{i,j}^{(t+1)}}-1 =0 \) if \( \mu _{2,i,j}^{(t+1)}>0 \) and \( -{d_{i,j}^{(t+1)}} = 0 \) if \( \mu _{3,i,j}^{(t+1)}>0 \), and each \( d_{i,j}^{(t)} \) satisfies the constraint condition:
$$\begin{array}{*{20}l} u(t,t)- u(t+1,t+1) \geq\\ u(t+1,t)- u(t+1,t+1) \geq\\ \frac{1}{2}\sum_{(i,j)\in\mathcal{I}^{2}} \|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^{2}\left({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}}\right)^2. \end{array} $$
Therefore, each sequence \( \{d_{i,j}^{(t)}\} \) converges to a limit point \(\bar {d}_{i,j}\) if u(0,0)<∞, because \({d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}} \rightarrow 0\) when t→∞ if \( \|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2}>0 \) and \( {d_{i,j}^{(t)}}-{d_{i,j}^{(t+1)}} = 1-1 = 0 \), even if \(\|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2}=0\) for \(t\geq 0, (i,j)\in \mathcal {I}^{2}\). Then, each sequence \( \{\boldsymbol {Z}_{i}^{(t)}\} \) converges to a limit point \(\bar {\boldsymbol {Z}}_{i}\) because each \( \boldsymbol {Z}_{i}^{(t+1)} \) can be obtained by the soft-thresholding operator using fixed \( d_{i,j}^{(t)} \) for \(t\geq 0, i\in \mathcal {I}\). □
Lemma 2
If β≥εi, the optimal solution of (17) under the constraint conditions for di,j and Zi can be obtained by initializing \(\boldsymbol {Z}_{i}^{(0)}\) as \(\boldsymbol {Z}_{i}^{(0)}=\boldsymbol {0}_{M,N}\) and updating \(d_{i,j}^{(0)}\) and \(\boldsymbol {Z}_{i}^{(1)}\) using the update schemes (18) for a given \(\boldsymbol {X}\in {\mathbb {R}^{{M}\times {N}}}\).
Proof
From Theorem 1 of [5], any \( \boldsymbol {X}\in {\mathbb {R}^{{M}\times {N}}} \) and each optimal solution \(\bar {\boldsymbol {Z}}_{i} \) and \( \bar {\boldsymbol {D}}_{i} \) satisfies \(\bar {\boldsymbol {Z}}_{i}=\boldsymbol {\mathcal {T}}_{r,\gamma }\left \{\left (\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}\right)\bar {\boldsymbol {D}}_{i}\right \} \). For a given di,j≥0, a matrix \(\boldsymbol {Z}_{i}=\mathcal {T}_{r,\gamma }\left \{\left (\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}\right){\boldsymbol {D}}_{i}\right \}\) satisfies 0≤〈xj−xi,zi,j〉 because, when di,j>0,
$$\begin{array}{*{20}l} \langle \boldsymbol{x}_j-\boldsymbol{x}_i, \boldsymbol{z}_{i,j} \rangle d_{i,j} &= \langle \boldsymbol{y}_{i,j}, \boldsymbol{z}_{i,j} \rangle\\ &=\sum_{l=1}^{r}\sigma_{l}^{2}(V)_{j,l}^{2}\\ &\ \ +\sum_{l=r+1}^M \sigma_{l}(\sigma_l-\gamma)_+ (\boldsymbol{V})_{j,l}^{2}\\ &\geq 0. \end{array} $$
Here, yi,j denotes the jth column of \(\boldsymbol {Y}_{i}=\left (\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}\right)\boldsymbol {D}_{i} =\boldsymbol {U}\text {diag}(\boldsymbol {\sigma })\boldsymbol {V}^{T}\) and σ,U,V denotes the singular values and vectors of \(\left (\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}\right)\boldsymbol {D}_{i}\); when di,j=0,〈yi,j,zi,j〉=0 because of yi,j=0M, where \( \boldsymbol {0}_{M}\in {\mathbb {R}^{M}} \) denotes the zero vector. Then, \(\bar {d}_{i,j}\) satisfies:
$$\begin{array}{*{20}l} \bar{d}_{i,j}=h_{\beta,\epsilon_{i}}(\boldsymbol{x}_j,\boldsymbol{x}_i,\bar{\boldsymbol{z}}_{i,j})=\left\{\begin{array}{cl}0 &:\|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^2>\epsilon_{i}\\1&: \|\boldsymbol{x}_j-\boldsymbol{x}_{i}\|_{2}^{2}\leq \epsilon_{i}\end{array}\right., \end{array} $$
because \(\beta \geq \epsilon _{i}\geq \|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2}\), which does not depend on \(\bar {\boldsymbol {Z}}_{i}\). Therefore, \(d_{i,j}^{(0)}=h_{\beta,\epsilon _{i}}(\boldsymbol {x}_{j},\boldsymbol {x}_{i},\boldsymbol {0}_{M})\in \{0,1\}\) and \(\boldsymbol {Z}_{i}^{(1)}=\boldsymbol {\mathcal {T}}_{r,\gamma }\left \{\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}){D}_{i}^{(0)}\right \}\) is the optimal solution for (16). □
Next, let us define the following schemes with regard to the kth iteration of the first-iteration statements in Algorithm 1 with β(k),γ(k),r(k) for k≥0,
$$\begin{array}{*{20}l} \left\{ \begin{array}{llll} d_{i,j}^{(k)}&=& \bar{d}_{i,j} \ \text{for} \ (i,j)\in\mathcal{I}^{2},\\ \boldsymbol{Z}_{i}^{(k)}&=&\bar{\boldsymbol{Z}}_{i} \ \text{for} \ i\in\mathcal{I},\\ \boldsymbol{x}^{(k+1)}&=&\underset{\boldsymbol{x}}{\text{argmin}} \boldsymbol{x}^{T} (\boldsymbol{L}^{(k)}\otimes \boldsymbol{I}_{M,M}) \boldsymbol{x}-2\boldsymbol{x}^{T} \boldsymbol{c}^{(k)}\\ &&\text{s.t.} \ (\boldsymbol{X})_{m,n}=(\boldsymbol{X}^{(0)})_{m,n} \ \text{for}\ (m,n)\in\Omega\\ && \ \ \ \ \ \ \|\boldsymbol{x}_{j}-\boldsymbol{x}_{i}\|_{2}^{2} \leq \epsilon_{i} \ \text{if}\ d_{i,j}^{(k)}>0, \end{array} \right. \end{array} $$
(19)
where \( \bar {d}_{i,j} \) and \( \bar {\boldsymbol {Z}}_{i} \) are the tth elements of the sequences obtained by the schemes (18) with β(k),γ(k),r(k),X(k), and vector c(k) as:
$$\begin{array}{*{20}l} \boldsymbol{c}^{(k)}&=\left[{\boldsymbol{c}_{1}^{(k)}}^T \ {\boldsymbol{c}_{2}^{(k)}}^{T}\ \cdots\ {\boldsymbol{c}_{M}^{(k)}}^{T}\right]^{T}\in {\mathbb{R}^{MN}}\\ \boldsymbol{c}_{l}^{(k)}&=\left(\tilde{\boldsymbol{D}}^{(k)}\odot \tilde{\boldsymbol{Z}}_{l}^{(k)}-\left(\tilde{\boldsymbol{D}}^{(k)}\odot \tilde{\boldsymbol{Z}}_{l}^{(k)}\right)^{T}\right)\boldsymbol{1}_N \\ &\in {\mathbb{R}^{N}} \ \text{for} \ l=1,2,\cdots, M. \end{array} $$
Here, \( \tilde {\boldsymbol {D}}^{(k)}\in {\mathbb {R}^{{N}\times {N}}}\) and \(\tilde {\boldsymbol {Z}}_{l}^{(k)} \in {\mathbb {R}^{{N}\times {N}}}\) denote matrices defined as \((\tilde {\boldsymbol {D}})_{i,j}^{(k)}= d_{i,j}^{(k)}\) and \((\tilde {\boldsymbol {Z}}_{l}^{(k)})_{i,j}= (\boldsymbol {Z}_{i}^{(k)})_{l,j}\) for \( (i,j)\in \mathcal {I}^{2}\), and the graph Laplacian L(k) is:
$$\begin{array}{*{20}l} \boldsymbol{L}^{(k)}=\text{diag}\left(\hat{\boldsymbol{D}}^{(k)}\boldsymbol{1}_{N}\right)-\hat{\boldsymbol{D}}^{(k)} \end{array} $$
where \(\hat {\boldsymbol {D}}^{(k)}\in {\mathbb {R}^{{N}\times {N}}}\) denotes a matrix whose every element is given by \(\left (\hat {\boldsymbol {D}}^{(k)}\right)_{i,j}={d_{i,j}^{(k)}}^{2}+{d_{j,i}^{(k)}}^{2}\).
Lemma 3
For k≥0,L(k) satisfies kernel(L(k))⊇kernel(L(k+1)).
Proof
Since a vector a∈kernel(L(k)) satisfies:
$$ \boldsymbol{a}^{T}\boldsymbol{L}^{(k)}\boldsymbol{a}= \sum_{(i,j)\in\mathcal{I}^{2}} \left({d_{i,j}^{(k)}}^{2}+{d_{j,i}^{(k)}}^{2} \right) (a_{i}-a_{j})^{2}=0, $$
kernel(L(k)) is written as:
$$\begin{array}{*{20}l} &\text{kernel}\left(\boldsymbol{L}^{(k)}\right)\\ &=\left\{\boldsymbol{a}\in{\mathbb{R}^{N}} \ \mid\ a_i=a_j \ \text{for} \ (i,j) \ \text{s.t.} \ d_{i,j}^{(k)}+d_{j,i}^{(k)}>0\right\}. \end{array} $$
Since \(d_{i,j}^{(k+1)}\) and \(d_{i,j}^{(k)}\) generated by the schemes (18) and (19) satisfy \( d_{i,j}^{(k+1)}>0 \) when \( d_{i,j}^{(k)}>0 \), L(k) satisfies kernel(L(k))⊇kernel(L(k+1)). □
Now, let us describe the properties of the sequences generated by Algorithm 1 \(\left \{\boldsymbol {X}^{(k)}\right \}, \left \{\boldsymbol {Z}_{i}^{(k)}\right \}, \left \{d_{i,j}^{(k)}\right \}\). We define the evaluation function:
$$\begin{array}{*{20}l} &v(k_1,k_2,k_3,k_4)=\\&g_{\beta^{(k_1)},\gamma^{(k_1)},r^{(k_1)}}\left(\boldsymbol{X}^{(k_2)},\left\{\boldsymbol{Z}_{i}^{(k_3)}\right\}_{i\in \mathcal{I}}, \left\{d_{i,j}^{(k_4)}\right\}_{(i,j)\in \mathcal{I}^{2}}\right) \end{array} $$
and replace the linear-constraint condition (X(k))m,n=(X(0))m,n for (m,n)∈Ω with Ax(k)=b, where \( \boldsymbol {b}\in {\mathbb {R}^{|\Omega |}} \) denotes a vector whose elements are observed values {(X(0))m,n}(m,n)∈Ω and A∈{0,1}|Ω|×MN denotes a selector matrix.
Theorem 1
The sequences \(\left \{\boldsymbol {X}^{(k)}\right \}, \left \{\boldsymbol {Z}_{i}^{(k)}\right \}\) and \(\left \{d_{i,j}^{(k)}\right \}\) converge to the limit points \( \bar {\boldsymbol {X}}, \bar {\boldsymbol {Z}}_{i}\), and \( \bar {d}_{i,j} \) under repetition of the iteration schemes of (19) when \(\text {kernel}\left (\tilde {\boldsymbol {L}}^{(0)}\right) \cap v(\boldsymbol {A}) = \{\boldsymbol {0}_{MN}\} \), where \(\tilde {\boldsymbol {L}}^{(k)}=\boldsymbol {L}^{(k)}\otimes \boldsymbol {I}_{M,M}\).
Proof
The scheme (15) can be written as:
$$\begin{array}{*{20}l} \begin{array}{cc} \underset{\boldsymbol{x}}{\text{argmin}}& \boldsymbol{x}^{T} \left(\boldsymbol{L}\otimes \boldsymbol{I}_{M,M}\right) \boldsymbol{x}-2\boldsymbol{x}^{T} \boldsymbol{c}\\ {{\text{subject to}}} & \boldsymbol{A}\boldsymbol{x}=\boldsymbol{b}\\ &\|\boldsymbol{Q}_{i,j}\boldsymbol{x}\|_{2}^{2}-\epsilon_{i}\leq0 \ \text{if}\ d_{i,j}>0, \end{array} \end{array} $$
where \(\boldsymbol {Q}_{i,j}\in {\mathbb {R}^{{M}\times {MN}}}\) denotes a matrix defined as \(\boldsymbol {Q}_{i,j}=\boldsymbol {q}_{i,j}^{T}\otimes \boldsymbol {I}_{M,M}\) and \(\boldsymbol {q}_{i,j}\in {\mathbb {R}^{N}}\) is defined such that the ith element is 1, the jth element is −1, and the others are 0 (Qi,j satisfies \( \|\boldsymbol {Q}_{i,j}\boldsymbol {x}\|_{2}^{2}=\|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2} \) for \( \boldsymbol {x}\in {\mathbb {R}^{MN}} \)). Since x(k+1) satisfies the following KKT condition for v(k,k+1,k,k):
$$\begin{array}{*{20}l} \left\{ \begin{array}{l} \tilde{\boldsymbol{L}}^{(k)}\boldsymbol{x}^{(k+1)}-\boldsymbol{c}^{(k)}+\lambda^{(k+1)} \boldsymbol{A}^{T}\boldsymbol{A}\boldsymbol{x}^{(k+1)}\\ +\sum_{d_{i,j}^{(k)}>0}\mu_{i,j}^{(k+1)}\boldsymbol{Q}_{i,j}^{T}\boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k+1)}}=\boldsymbol{0}_{MN},\\ \boldsymbol{A}\boldsymbol{x}^{(k+1)}=\boldsymbol{b},\\ \mu_{i,j}^{(k+1)}(\|\boldsymbol{Q}_{i,j}\boldsymbol{x}^{(k+1)}\|_{2}^{2}-\epsilon_{i})=0\ \text{for} \ d_{i,j}^{(k)} >0,\\ \mu_{i,j}^{(k+1)}\geq 0, \end{array} \right. \end{array} $$
where λ(k+1) and \(\mu _{i,j}^{(k+1)}\) denote the KKT multipliers, v(k1,k2,k3,k4) satisfies:
$$\begin{array}{*{20}l} &2v(k,k,k,k)-2v(k,k+1,k,k)\\ &\begin{array}{l} ={\boldsymbol{x}^{(k)}}^{T}\tilde{\boldsymbol{L}}^{(k)}{\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}^{T}\tilde{\boldsymbol{L}}^{(k)}{\boldsymbol{x}^{(k+1)}}\\ \ \ -2{\boldsymbol{x}^{(k)}}^{T}\boldsymbol{c}^{(k)}+2{\boldsymbol{x}^{(k+1)}}^{T}\boldsymbol{c}^{(k)} \end{array}\\ &\begin{array}{l} =\left({\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}\right)^{T}\tilde{\boldsymbol{L}}^{(k)}\left({\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}\right)\\ \ \ -2{\boldsymbol{x}^{(k)}}^{T}\sum_{d_{i,j}^{(k)}>0}\mu_{i,j}^{(k+1)}\boldsymbol{Q}_{i,j}^{T}\boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k+1)}}\\ \ \ +2{\boldsymbol{x}^{(k+1)}}^{T}\sum_{d_{i,j}^{(k)}>0}\mu_{i,j}^{(k+1)}\boldsymbol{Q}_{i,j}^{T}\boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k+1)}}, \end{array}\\ \end{array} $$
where the second equality uses the fact that Ax(k)=b. Since \(\|\boldsymbol {Q}_{i,j}\boldsymbol {x}^{(k+1)}\|_{2}^{2}=\epsilon _{i}\) when \(\mu _{i,j}^{(k+1)}>0\),
$$\begin{array}{*{20}l} &v(k,k,k,k)-v(k,k+1,k,k)\\ &\begin{array}{l} =\frac{1}{2}\left({\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}\right)^{T}\tilde{\boldsymbol{L}}^{(k)}\left({\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}\right)\\ \ \ +\sum_{d_{i,j}^{(k)}>0}\mu_{i,j}^{(k+1)}\left\{\epsilon_i-{{\boldsymbol{x}^{(k)}}^{T}\boldsymbol{Q}_{i,j}^{T}\boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k+1)}}}^{T}\right\}\\ \end{array}\\ &\begin{array}{l} \geq\frac{1}{2}\left({\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}\right)^{T}\tilde{\boldsymbol{L}}^{(k)}\left({\boldsymbol{x}^{(k)}}-{\boldsymbol{x}^{(k+1)}}\right). \end{array} \end{array} $$
The second inequality uses:
$$\left|\langle \boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k)}},\boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k+1)}}\rangle\right| \!\leq\! \| \boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k)}}\|_{2}\| \boldsymbol{Q}_{i,j}{\boldsymbol{x}^{(k+1)}}\|_{2}\!\leq\! \epsilon_{i}.$$
Obviously, v(k,k+1,k,k)≥v(k+1,k+1,k,k) because the parameters {β(k),γ(k),r(k)} decrease the objective function (17), and v(k+1,k+1,k,k)≥v(k+1,k+1,k+1,k+1) from Lemma 1. Since the sequence {v(k,k,k,k)} generated by (19) converges to a limit point because of:
$$\begin{array}{*{20}l} v(k,k,k,k)&\geq v(k,k+1,k,k)\\ &\geq v(k+1,k+1,k,k)\\ &\geq v(k+1,k+1,k+1,k+1)\\ &\ \vdots\\ &\geq -\beta_{\text{max}}N^{2}, \end{array} $$
x(k)−x(k+1)→0MN when k→∞ and v(0,0,0,0)<∞ because each L(k) satisfies \(\text {kernel}\left (\tilde {\boldsymbol {L}}^{(k)}\right) \cap \text {kernel}(\boldsymbol {A}) = \{\boldsymbol {0}_{mn}\} \) for k≥0 if \(\text {kernel}\left (\tilde {\boldsymbol {L}}^{(0)}\right) \cap \text {kernel}(\boldsymbol {A}) = \{\boldsymbol {0}_{MN}\} \) from Lemma 3. X(k) reaches a limit point \(\bar {\boldsymbol {X}}\); then, the sequence \(\left \{\boldsymbol {Z}_{i}^{(k)}\right \}\) and \(\left \{d_{i,j}^{(k)}\right \}\) converges to limit points \(\bar {\boldsymbol {Z}}_{i}\) and \(\bar {d}_{i,j}\) with a fixed \(\bar {\boldsymbol {X}}\) from Lemma 1. □
Finally, some improvements to Algorithm (1) are offered in this section. First, the dimension of the LDDM is unknown in actual applications, although Algorithm 1 requires a suitable r. In order to solve this issue, we adopt a method that estimates the dimension r based on the ratio of the singular value σr/σ1, just as [5] did for each column \( i\in \mathcal {I} \). Second, we consider ways to reduce the computational complexity. Two key possibilities are considered: one is to ignore the quadratic-constraint condition \( \left (\|\boldsymbol {x}_{j}-\boldsymbol {x}_{i}\|_{2}^{2}-\epsilon _{i}\right)d_{i,j}\leq 0 \) when we update X and the other is to update X for only the columns in the ith neighborhoods, for example, by minimizing the only ith Frobenius norm term of (17) \( \left \|(\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T})\boldsymbol {D}_{i}-\boldsymbol {Z}_{i}\right \|_{F}^{2} \) with regard to the column xi, which is expected to work like a stochastic gradient-descent algorithm. Furthermore, this paper utilizes the parameter β= maxiε(i) because the update schemes (18) yield limit points for Zi and di,j only once for each \( i\in \mathcal {I} \) from Lemma 2. Thus, this paper proposes a heuristic algorithm for reducing the calculation time, as shown in Algorithm 2. There, the parameters satisfy 1>α(0)≥α(1)≥⋯≥αmin>0 for k=0,1,⋯,kmax and δ>0, just as in [5].
We consider here the time and space complexities of Algorithm 2. The major computational cost of Algorithm 2 is derived from computing the singular value decomposition of \(\left (\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}\right)\boldsymbol {D}_{i} \) for all i=1,2,⋯,N at each iteration. For simplicity, this paper assumes that the number of non-zero vectors of \(\left (\boldsymbol {X}-\boldsymbol {x}_{i}\boldsymbol {1}_{N}^{T}\right)\boldsymbol {D}_{i} \) is M for each iteration and each i. Then, since the algorithm requires the singular value decomposition of the M×M matrix, the time and space complexities of Algorithm 2 are O(M3N) and O(M2) for each iteration. As written in [20], since the method VMC [15] requires the time complexity O(N3+MN2) and the space complexity O(N2), the time and space complexities of Algorithm 2 are lower than those of VMC when the numbers of rows M and columns N satisfy M3<N2. Hence, Algorithm 2 is effective for datasets such as those used in Section 5.2.