Optimized Projection Matrix for Compressive Sensing
© Jianping Xu et al. 2010
Received: 27 September 2009
Accepted: 22 June 2010
Published: 11 July 2010
Compressive sensing (CS) is mainly concerned with low-coherence pairs, since the number of samples needed to recover the signal is proportional to the mutual coherence between projection matrix and sparsifying matrix. Until now, papers on CS always assume the projection matrix to be a random matrix. In this paper, aiming at minimizing the mutual coherence, a method is proposed to optimize the projection matrix. This method is based on equiangular tight frame (ETF) design because an ETF has minimum coherence. It is impossible to solve the problem exactly because of the complexity. Therefore, an alternating minimization type method is used to find a feasible solution. The optimally designed projection matrix can further reduce the necessary number of samples for recovery or improve the recovery accuracy. The proposed method demonstrates better performance than conventional optimization methods, which brings benefits to both basis pursuit and orthogonal matching pursuit.
Compressive sensing (CS) [1–3] has received much attention as it has shown promising results in many applications. CS is an emerging framework, stating that signals which have a sparse representation on an appropriate basis can be recovered from a number of random linear projections of dimension considerably lower than that required by the Shannon-Nyquist theorem. Moreover, compressible signals, that is, the signals' transform coefficients on appropriate basis decay rapidly, can also be sampled at a much lower rate than that required by the Shannon-Nyquist theorem and then reconstructed with little loss of information.
Until now, almost all works on CS made the assumption that is drawn at random except the one by Elad  and the one by Duarte-Carvajalino and Sapiro . In , Elad proposed an iterative algorithm. The algorithm tries to minimize the -averaged mutual coherence between and , where is fixed. Although the reconstruction performance can be obviously improved, the method is time-consuming because it needs many iterative steps to achieve good performance. Some large mutual coherence values that are not present in the original Gram matrix are also created, which ruin completely the worst case guarantees of the reconstruction algorithms. Instead of targeting on the -averaged mutual coherence between and , in , Duarte-Carvajalino and Sapiro addressed the problem by making any subset columns of as orthogonal as possible, or equivalently, making the Gram matrix as closely as possible to identity matrix. The method is much faster than Elad's but the reconstruction performance is not very good because is overcomplete and it could not be an orthogonal basis.
In this paper, a method to optimize the projection matrix is proposed from the perspective of ETF design . The object is to find an equivalent dictionary whose Gram matrix is as close as possible to an ETF's because an ETF has minimum coherence . It is impossible to find an exact solution so an alternating minimization type method is used to find a feasible solution. The proposed method needs few iterative steps to achieve good performance and the reconstruction performance is much better than the existed methods, with both BP and OMP.
The remainder of the paper is organized as follows. In Section 2, the basics of CS are provided along with a statement of the main results in literature relating to this paper. In Section 3, after briefly describing the methods suggested by Elad and Duarte-Carvajalino and Sapiro, a method to optimize the projection matrix is proposed from the aspect of ETF design and an alternative minimization method is proposed to solve the problem. In Section 4, experimental results are presented and the performance obtained with all the optimization methods is compared. Finally, concluding remarks and directions for future research are presented in Section 5.
2. Compressive Sensing: The Basics
A different way to measure mutual coherence is considering the Gram matrix of equivalent dictionary which is computed after normalizing each column of . The off-diagonal entries of are the inner products that appear in (4). Mutual coherence is the off-diagonal entry with largest magnitude.
Mutual coherence measures the maximal correlation between both matrix elements and plays an important role in the success of reconstruction algorithm. It has been demonstrated that mutual coherence should be as small as possible in CS .
where is a positive constant, is the sparsity level of signal, and is the dimension. It is obvious to see that the smaller the coherence, the fewer samples are needed .
From the former discussion, it is easy to see that CS deals with the case of low coherence between and . With the properties of , there is a sensible reason to design the projection matrix in a way that minimizes the mutual coherence which may lead to better performance of reconstruction algorithms.
3. Optimizing Projection Matrix for Compressive Sensing
In this paper, only the case that is fixed while can be arbitrary is considered. Hence, the object is to optimize that will minimize the mutual coherence . After reviewing the former related work, the proposed algorithm is introduced.
3.1. Elad's Method 
The former shrinking operation causes the resulting Gram matrix to become full rank in general case. Thus, the next steps should mend this by forcing the rank to be and find the matrix that could best describe the squared root of the obtained Gram matrix. The process could be realized using SVD. The details can be found in .
3.2. Duarte-Carvajalino and Sapiro's Method
Unlike the previous one, Duarte-Carvajalino and Sapiro's method is noniterative. Instead of targeting on -averaged mutual coherence between and , this method addressed the problem by making any subset of columns in as orthogonal as possible, or equivalently, making the Gram matrix as close as possible to an identity matrix. Their approach was carried out as follows.
By solving the problem of (15), they achieved to optimize the projection matrix. The details to solve the problem can be found in .
3.3. The Proposed Method
Elad's method is time-consuming and the shrinkage function creates some large values that are not present in the original Gram matrix. Large off-diagonal values in the Gram matrix ruin completely the worst case guarantees of the reconstruction algorithms. Duarte-Carvajalino and Sapiro's method is noniterative and the reconstruction relative error rate is high. To overcome these drawbacks, a method based on ETF design is proposed in this paper. The object is to find an equivalent dictionary which is as close as possible to an ETF because of the minimum coherence property of ETF, and then from the equivalent dictionary, the optimized projection matrix can be constructed. It is impossible to solve the problem exactly because of the complexity, so an alternative minimization type method is used to find a feasible solution.
Firstly, model the problem as an optimization problem. Let be the Gram matrix of . The mutual coherence of is the maximum absolute value of the off-diagonal entry of , supposing the columns of are normalized. For such , if the magnitudes of all off-diagonal entries of are equal, has minimum coherence . This normalized dictionary is called an ETF. Although this type of frame has many nice properties, ETF does not exist for any arbitrary selection of dimension. Therefore the optimization process aims at finding the nearest admissible solution which is as close as possible to an ETF.
A standard method to solve (23) is alternating projection . In this work, a different method which has similarities with alternating projection is used. Although the proposed solution has similarities with alternating projection, it does not follow its steps exactly. The difference lies in the stage of updating the current solution with respect to . A point between the current solution and the projection on is chosen, that is because after being projected onto , the structure of the Gram matrix changes significantly and the selection of a new point in the following step is very difficult. After performing alternating minimization, the optimized projection matrix can be constructed from the output Gram matrix with a rank revealing QR factorization with eigenvalue decomposition. The details can be found in .
The conditions under which the algorithm converges can be found in .
The following are the steps of the proposed algorithm for optimizing the projection matrix, supposing the sparsifying matrix is known.
4. Experiment Results
Then, the performance of recovery algorithm in CS is evaluated before and after projection matrix optimization. In order to compare the performance, the test proposed by Elad in  is used. The test includes the following steps.
Performance test: apply BP and OMP to reconstruct the signals by approximating the solution of subject to . Test the error . Measure the average error rate—a reconstructionwith a mean squared error above some threshold is considered as a reconstruction failure.
Optimize the projection matrix: use the three methods mentioned above to optimize the projection matrix.
Reapply Steps 2 and 3 using the optimized projection matrix.
In the first experiment (Figures 2 and 3), a random dictionary of size was used. This size was chosen since it enabled the CS performance evaluation in reasonable time. sparse vectors were generated of length with cardinality . The nonzero's locations were chosen at random. These sparse vectors were used to create the test signals with which the CS performance was evaluated. CS performance was tested with varying values of . The relative error rate was evaluated as a function of for both OMP and BP before and after the optimization. Each point in the graph represents an average performance, accumulated over a possible varying number of experiments. While every point was supposed to present an average performance over signals, in cases where more than 300 errors was accumulated, the test stopped and the average so far was used instead. This was done to reduce the overall runtime.
From Figures 2 and 3, the CS performance improves for both BP and OMP as increases and in this experiment, BP outperforms OMP. Also, as expected, all the optimization methods lead to improved performance of CS. For some values of , the performance is improved by 10 : 1 for BP and 100 : 1 for OMP. The optimization method proposed by us outperforms both Elad's method and Duarte-Carvajalino and Sapiro's method. For some values of , our method improves the CS performance by 5 : 1 than Elad's method and even 10 : 1 than Duarte-Carvajalino and Sapiro's method for OMP. The improvement for BP is smaller than OMP, about 2 : 1 for some values of .
In the following experiment, the effect of signal dimension on CS performance before and after optimization of the projection matrix is considered. In this experiment, the signals' dimension was varied while proportionally updating measurements , dictionary dimension and signal cardinality with it. The object was to get a better indication to the asymptotic performance as studied in [1–3].
Afterwards, a test image was used to assess the recovery algorithm performance in CS before and after optimizing the projection matrix. The testing image consists of non-overlapping patches reconstructed from their noisy projections (5% level of noise).
In this experiment, is set to be 15 and the sparsifying matrix to be DCT. The reconstruction algorithm used is OMP. From the pictures above, it is obvious to see that the recovery performance is improved for both the optimization methods. The Elad's method can improve PSNR about 1 db than using random , and the proposed method in this paper can improve about 0.5 db than Elad's method. If larger number of measurements is used for recovery, the reconstruction performance could be better, but it is time-consuming.
A crucial ingredient in the deployment of the CS idea is the process of random linear projections that mix the signal. This operation has been traditionally chosen as a random matrix. This paper aimed to show that the optimally designed projection matrix can further improve the CS performance. The method was that constructing the projection matrix so as to minimize the coherence between and based on ETF design. The experimental results demonstrated that the optimally designed projection matrix indeed lead to a better CS performance, not only improvement of the reconstruction accuracy but also reduction in the necessary number of samples for recovery. It also demonstrated that after optimizing the projection matrix using the proposed method, the CS performance was greatly improved than using random projection matrix and the already existed optimization methods.
As this is only one of the very few works to address the problem of optimization the projection matrix, there is still great work to do. Here are some advices for future research. ( ) How to perform the proposed method when the signals are of high dimension. ( ) Whether there is a more direct method to address the problem which may be easier.
Supported by the Fundamental Research Funds for the Central Universities China.
- Candes EJ, Romberg J, Tao T: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory 2006, 52(2):489-509.MathSciNetView ArticleMATHGoogle Scholar
- Candes EJ, Wakin MB: An introduction to compressive sampling. IEEE Signal Processing Magazine 2008, 25(2):21-30.View ArticleGoogle Scholar
- Donoho DL: Compressed sensing. IEEE Transactions on Information Theory 2006, 52(4):1289-1306.MathSciNetView ArticleMATHGoogle Scholar
- Natarajan BK: Sparse approximate solutions to linear systems. SIAM Journal on Computing 1995, 24(2):227-234. 10.1137/S0097539792240406MathSciNetView ArticleMATHGoogle Scholar
- Pati YC, Rezaiifar R, Krishnaprasad PS: Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. Proceedings of the 27th Asilomar Conference on Signals, Systems & Computers, November 1993 40-44.View ArticleGoogle Scholar
- Tropp JA, Gilbert AC: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on Information Theory 2007, 53(12):4655-4666.MathSciNetView ArticleMATHGoogle Scholar
- Elad M: Optimized projections for compressed sensing. IEEE Transactions on Signal Processing 2007, 55(12):5695-5702.MathSciNetView ArticleGoogle Scholar
- Duarte-Carvajalino JM, Sapiro G: Learning to sense sparse signals: simultaneous sensing matrix and sparsifying dictionary optimization. IEEE Transactions on Image Processing 2009, 18(7):1395-1408.MathSciNetView ArticleGoogle Scholar
- Sustik MA, Tropp JA, Dhillon IS, Heath RW Jr.: On the existence of equiangular tight frames. Linear Algebra and Its Applications 2007, 426(2-3):619-635. 10.1016/j.laa.2007.05.043MathSciNetView ArticleMATHGoogle Scholar
- Sardy S, Bruce AG, Tseng P: Block coordinate relaxation methods for nonparametric wavelet denoising. Journal of Computational and Graphical Statistics 2000, 9(2):361-379. 10.2307/1390659MathSciNetGoogle Scholar
- Gribonval R, Nielsen M: Sparse representations in unions of bases. IEEE Transactions on Information Theory 2003, 49(12):3320-3325. 10.1109/TIT.2003.820031MathSciNetView ArticleMATHGoogle Scholar
- Mallat SG, Zhang Z: Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing 1993, 41(12):3397-3415. 10.1109/78.258082View ArticleMATHGoogle Scholar
- Tropp JA: Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory 2004, 50(10):2231-2242. 10.1109/TIT.2004.834793MathSciNetView ArticleMATHGoogle Scholar
- Donoho DL, Elad M:Optimally sparse representation in general (nonorthogonal) dictionaries via minimization. Proceedings of the National Academy of Sciences of the United States of America 2003, 100(5):2197-2202. 10.1073/pnas.0437847100MathSciNetView ArticleMATHGoogle Scholar
- Chen SS, Donoho DL, Saunders MA: Atomic decomposition by basis pursuit. SIAM Journal of Scientific Computing 1998, 20(1):33-61. 10.1137/S1064827596304010MathSciNetView ArticleMATHGoogle Scholar
- Strohmer T, Heath RW Jr.: Grassmannian frames with applications to coding and communication. Applied and Computational Harmonic Analysis 2003, 14(3):257-275. 10.1016/S1063-5203(03)00023-XMathSciNetView ArticleMATHGoogle Scholar
- Golub GH, Van Loan CF: Matrix Computation. Johns Hopkins University Press, Baltimore, Md, USA; 1996.MATHGoogle Scholar
- Tropp JA, Dhillon IS, Heath RW Jr., Strohmer T: Designing structured tight frames via an alternating projection method. IEEE Transactions on Information Theory 2005, 51(1):188-209.MathSciNetView ArticleMATHGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.