Removing non-stationary noise in spectrum sensing using matrix factorization

Abstract

Spectrum sensing is key to many applications like dynamic spectrum access (DSA) systems or telecom regulators who need to measure utilization of frequency bands. The International Telecommunication Union (ITU) recommends a 10 dB threshold above the noise to decide whether a channel is occupied or not. However, radio frequency (RF) receiver front-ends are non-ideal. This means that the obtained data is distorted with noise and imperfections from the analog front-end. As part of the front-end the automatic gain control (AGC) circuitry mainly affects the sensing performance as strong adjacent signals lift the noise level. To enhance the performance of spectrum sensing significantly we focus in this article on techniques to remove the noise caused by the AGC from the sensing data. In order to do this we have applied matrix factorization techniques, i.e., SVD (singular value decomposition) and NMF (non-negative matrix factorization), which enables signal space analysis. In addition, we use live measurement results to verify the performance and to remove the effects of the AGC from the sensing data using above mentioned techniques, i.e., applied on block-wise available spectrum data. In this article it is shown that the occupancy in the industrial, scientific and medical (ISM) band, obtained by using energy detection (ITU recommended threshold), can be an overestimation of spectrum usage by 60%.

1 Introduction

Nowadays the application of short range wireless technologies continues to grow, which requires the radio spectrum, a scarce resource, to be well-managed. Most of these modern wireless technologies take place in a frequency range from 200 MHz to 3 GHz where spectrum sensing is key to many applications. For instance, several applications in this frequency range need spectrum sensing to identify white spaces (dynamic spectrum access (DSA) applications) but also telecom operators use spectrum sensing to measure utilization and interference. An important part of these short range radio communication systems is currently active in the license-free 2.4 GHz industrial, scientific and medical (ISM) band, i.e., systems like Wi-Fi (IEEE 802.11 b/g/n), Bluetooth, wireless sensor networks, wireless A/V links (e.g., wireless cameras). In this respect spectrum management becomes more and more of importance in addressing coexistence issues between such systems in the 2.4 GHz ISM band.

In this article, we have used this band as a test case, however, the applied techniques can be used for other bands as well. To monitor the radio spectrum usage in this band we issue the RFeye mobile spectrum monitoring equipment of CRFS [1]. The mobile monitoring system can be built into vehicles to continuously measure spectrum usage on a country-wide scale while driving [2]. However, in a number of areas (very close to high power transmitters, e.g., UMTS stations) the detected input signals were significantly stronger than expected and measurements were affected as the automatic gain control (AGC) system responded causing the noise floor in that particular band to be lifted (see Figure 1). Note that these AGC systems can be found in virtually all radio receivers, which means that the lifting effect is not limited to this specific equipment. In several cases the AGC was fully engaged and the front-end of the receiver saturated thereby corrupting the data for that particular sweep [3]. The non-stationary white Gaussian noise caused by the AGC in the measurement data is the focus of this work. In the light of spectrum monitoring it is paramount to have the AGC noise removed from the measurement data to ensure that the in-band signals, and inherently the spectrum occupancy, can be assessed more accurately. Several methods are described in literature [4, 5] to separate the noise and signal space. In this article, we consider matrix factorization techniques in order to detect and remove the AGC noise from a block of data. This relates to the mobile monitoring equipment where the measurement data is provided in a spectrogram representation corrupted by noise from the AGC. Along this line, the concept of AGC noise and how it affects sensing performance is illustrated in Figure 2. The aforementioned matrix factorization techniques, known from the field of principal component analysis [4, 6] and image processing, lend themselves for reducing spectral distortion and thus are considered for removal of AGC noise from the spectrogram. In what follows, the performance of the proposed scheme is provided in terms of probability of detection, probability of false alarm and spectrum occupancy.

Figure 1

UMTS base stations account for AGC noise in the ISM band. The AGC lifts the noise floor due to strong signals in the UMTS downlink band (2110–2170 MHz) and is visible as wide-band artifacts in the spectrogram. The noise floor is lifted for tens of MHz, thereby also causing distortion in the 2.4 GHz ISM band. Note that the mobile monitoring equipment picks up the high signal power when an UMTS base station is in the vicinity.

Figure 2

Noise distributions. Both the thermal noise power ${\sigma }_{\text{thermal}}^2$ and AGC noise power ${\sigma }_{\text{AGC}}^2$ contribute to the total noise power ${\sigma }_{\text{noise}}^2$ as indicated by the dashed curve. Here the thermal noise is accentuated in Figure 1 a and the AGC noise in Figure 1 b. Additionally, both are modeled as white Gaussian processes whereas the sum of distributions provides a nearly perfect match with the measured noise from the mobile spectrum monitoring equipment. The latter is depicted by the bar plot as depicted in Figure 1 c. The objective is to remove the dark-shaded Gaussian distributed AGC noise contribution which is shown in Figure 1 b. This enhances the detection performance as illustrated in Figure 1 d.

The rest of the article is organized as follows. We first provide the background of AGC systems in Section 2. Next, in Section 3 an overview of related work in literature on spectrum sensing is presented supplemented with the research objectives. In Section 4, we introduce the mobile monitoring network and measurements using the monitoring equipment to assess the influence of AGC on spectrum sensing. Then the system model used for analysis, including the matrix factorization techniques, is provided in Section 5. The proposed method for removal of non-stationary AGC noise with the aid of matrix factorization is presented in Section 6. Along this line, in Section 7 we present the performance evaluation of the proposed method and a comparison is made with energy detection based spectrum sensing. In addition, a validation of the proposed method is provided to verify that the proposed method eliminates exclusively the AGC noise. Finally, the conclusions are drawn in Section 8.

2 Background

Nowadays, lots of radio systems and most of the spectrum analyzers are based on the super-heterodyne principle, i.e., the received signal is down-converted to a fixed intermediate frequency by the application of frequency mixing (or heterodyning). The conversion to an intermediate frequency is useful since filters can be set to a fixed frequency for which very sharp selective filters can be built. Furthermore, this makes them easier to tune and fewer stages for filtering are required when compared with the low-cost direct conversion (zero-IF) receivers which operate directly on the original radio carrier frequency. Due to the heterodyne principle the super-heterodyne radio comprises two stages: radio frequency (RF) and intermediate frequency (IF). Note that a super-heterodyne receiver can also be built of several IF stages to improve the radio reception; two or three IF stages are called double or triple conversion. However, these types of receivers are more costly and come at the expense of additional circuitry.

Each stage of the super-heterodyne receiver requires an AGC control loop in order to maintain the signal levels within an acceptable range. As a consequence, due to the AGC operation the weaker signals receive more gain and the stronger signals receive attenuation, i.e., an AGC system uses a high and low gain setting which are defined by respectively a lower and upper threshold.

An outline of the heterodyne receiver and the AGC loops is shown in Figure 3 for a system with one IF-stage. Regarding the RF stage, an AGC loop is necessary to prevent strong RF signals from overdriving the RF amplifier and mixer. This is considered paramount while overdriving can cause undesirable side-effects such as inter-modulation products and higher harmonics which can degrade the sensitivity of the receiver and may mask the wanted signal. In line with this, AGC control entails adjustments in RF-gain in order to ensure that the RF-amplifier operates in its linear region. This to prevent the RF-amplifier from running into saturation (non-linear region) which in turn would cause the above-mentioned inter-modulation problems. On the other hand, as a drawback note that a reduction in RF gain will lead to a higher noise figure (NF) of the system.

Figure 3

The super-heterodyne radio receiver. The super-heterodyne radio receiver with RF stage and one IF-stage. The first AGC loop operates in the RF stage in order to prevent the RF-amplifier (i.e., RF VGA circuit) from overloading the RF-mixer (down-converter). The second AGC loop operates in the baseband domain and sets the IF-gain of the IF-amplifier (i.e., IF VGA) to scale the signal input for the A/D conversion. Each AGC loop consists of the following building blocks: VGA, detector and comparator. In case of a super-heterodyne spectrum analyzer the principle is the same as for the radio receiver case with the addition that the oscillators are driven by a ramp generator to sweep through the frequency range.

In the IF-stage a second AGC system is active to maintain a constant signal level at the input of the analogue-to-digital converter (ADC), this because of the limited dynamic range of the ADC. This requires the signal input to be scaled properly for A/D conversion so that—despite large signal variations—the average ADC output falls within its pre-set range. For this purpose the upper-threshold can be set to the highest detected signal strength which also determines the lower-threshold level, i.e., is defined to be positioned at a fixed distance from the upper-threshold (see Figure 4). As a consequence, this means that in response to strong incoming signals the second AGC will raise the lower-threshold and herewith up scaling the IF-gain. For all signals below the lower-threshold the IF-gain applies which in turn causes the noise floor to be lifted. Moreover, the AGC operation also improves the quantization of weak signals because smaller signal variations will toggle now more bits at the A/D conversion. Paradoxically, note that an increasing IF-gain (AGC loop 2) can be assigned to scale up the signals for A/D conversion, whilst the preceding RF-amplifier adversely applied attenuation (AGC loop 1) to prevent overdriving [7].

Figure 4

AGC thresholds: IF stage. The second AGC loop entails respectively an upper and lower threshold. The upper-threshold can be set to the highest detected signal within a specific frequency range (typically tens of MHz) which depends on the AGC settling time. The lower threshold may be defined on a fixed distance from the upper-threshold, i.e., in this case 40 dB. As a result the signals are pulled up by the IF-gain.

To elaborate on this matter, the AGC loop is described in more detail below. First, note that for most applications an AGC system with closed-loop feedback (see in Figure 3) is used to settle the output signal amplitude to the desired level; feed-forward loops also exist but are less common. In general the AGC loop consists of a VGA (variable gain amplifier), a peak detector, and a comparator (using the defined thresholds) where both peak detector and VGA are non-linear components. Regarding the design of the AGC loop, the important parameters are the settling time, i.e., the loop response time, and the loop stability. Note that many AGC systems are required to have a constant settling time for the full dynamic range of input signal levels, which can be up to 80 dB. To achieve this the VGA operates with an exponential gain to ensure that the AGC loop behaves as a first-order linear system in decibels (dB). Moreover, to accomplish loop stability the settling time is much slower than the input signal change, i.e., the AGC loop bandwidth must be significantly smaller than the bandwidth of the VGA.

For many heterodyne radio receivers the settling time of the AGC loop takes at least 0.5 ms for a fast loop implementation in mobile applications [8]. For super-heterodyne spectrum analyzers the architecture is very similar to the above-described super-heterodyne radio receivers with a comparable settling time. However, the settling time is relatively slow and may corrupt the performance of the spectrum analyzer which carries out a full frequency sweep over many GHz typically within 100 ms. During the settling time period a wide range of frequencies is swept through (usually tens of MHz) wherefore the AGC gain is fixed. This means that for the duration of AGC settling time the noise floor may lift up for a broad frequency range which scales non-linearly with the detected strongest signals in this range and thus introduces noise uncertainty into the system. This has been referred to as non-stationary noise and is of particular concern in receiver design because it decreases the sensitivity of the receiver’s signal detection due to the higher noise floor.

3 Related study

In literature the AGC is described mainly from a RF point of view with focus on front-end design among which [9] is a classical article on AGC design. The two basic topics of AGC in literature are the loop stability and the settling time and the basic theory can be found in [10, 11]. In addition, the classical exponential constraint on the gain characteristic of the VGA to obtain constant settling time is discussed in [12–14]. However, achieving an exponential relationship in CMOS and other technologies is less obvious as pointed out in [15, 16]. In recent years, many articles, among which [7, 8], have issued digital AGC circuits in the design of wireless radio communication systems primarily intended for mobile applications. Such AGC circuits are required to be fast in order to compensate for the strong signal variations introduced by Rayleigh fading. To cope with this type of small scale fading the design of AGC systems for super-heterodyne receivers is discussed in [8]. Furthermore, it turns out that the optimal AGC settling time depends on the mobile speed [17] and has the aforementioned typical value of at least 0.5 ms.

The influence of AGC on monitoring and spectrum received less attention in literature. For instance, the distortion due to imperfections in the RF-frontend were addressed in [18] and is herein referred to as ‘dirty RF’. To improve on this in [19] algorithms were proposed to compensate for particular RF imperfections in the digital domain. Similar approaches may be needed to compensate for the imperfections of AGC, particularly for the case of spectrum sensing applications. To elaborate on this matter, spectrum sensing is a signal-processing technique to determine the activities of users within a particular spectrum band. Important wireless communication systems, for instance cognitive radio [5, 20], rely on such spectrum sensing techniques. The main task of spectrum sensing technology is to detect the available portions of radio spectrum for further efficient spectrum resource allocation. The Federal Communications Commission (FCC) has already adopted the concept cognitive radio for better use of the spectrum [21] and issued the requirements of reliable spectrum sensing in [22]. Along this line, the Electronics Communication Committee (ECC) provided technical and operational requirements for cognitive radio systems in the UHF/VHF bands [23].

The state-of-the-art spectrum sensing techniques can meet the above-mentioned regulatory requirements under special conditions [24, 25]. Among these signal sensing techniques cyclostationary detection is recognized as one of the most robust techniques which can provide an excellent sensing performance [5, 26, 27], however, with the disadvantage of introducing a long latency into the system [28].

The match filtering based approach can deliver the optimal detection performance in the sense that it maximizes the received SNR. On the other hand, match filtering detection comes at the cost of demanding synchronization and high computational complexity. Additionally, note that both cyclostationary detection and matched filtering require prior knowledge of the transmitted signal at the receiver. Thus, in scenarios with lack of signal information both methods are not applicable anymore. In the case of unknown signal information so-called blind spectrum sensing techniques are needed to provide a spectral estimate. However, blind spectrum sensing techniques require proper noise power estimation in order to determine the spectrum occupancy [29]. Blind sensing techniques thus operate ideally in scenarios of stationary noise, whereas in practice [30] most of the noise turns out to be non-stationary (i.e., noise uncertainty). Furthermore, the non-stationary noise may lead to a SNR wall and high probability of false alarm at the receiver site [31, 32].

In case of blind spectrum sensing and stationary noise the optimal detection technique is energy detection which provides low computational complexity and is recognized for its simple architecture [28, 33, 34]. However, energy detection has the shortcoming that it requires perfect information of the noise variance to perform satisfactorily and thus is susceptible to noise uncertainty at the receiver. Energy detection works as follows: in a certain frequency band of interest, the received signal power is compared with a predefined threshold based on which a decision is made whether the frequency spectrum is occupied or vacant. The placement of the threshold is thus crucial and is subject to fluctuations in noise power level, i.e., is limited in case of non-stationary noise. Furthermore, the International Telecommunication Union (ITU) recommends a threshold to be placed 10 dB above the ambient noise floor for energy detection as stated in ([35], p. 168) and [36].

As a solution for this problem alternative blind spectrum sensing techniques were proposed to enhance noise power estimation with the aid of eigenvalues derived from the sample covariance matrix of the received signal for instance. Along this line, the provided set of eigenvalues are used to compute the detection probabilities wherein the ratio maximum to minimum eigenvalue λ _max/λ _min determines the threshold position. In [37] the use of random matrix theory (RMT) is suggested to estimate the eigenvalues, whereas in [38] the multiple signal classification (MUSIC) algorithm is used to distinguish between noise and signal space for adaptive threshold placement. However, it is very complex to determine the threshold due to the biased lower eigenvalues [39]. In addition, another blind spectrum sensing technique has been proposed [40] known as “multitaper method (MTM)” to provide spectral estimation by application of orthogonal tapers/windows to determine the average power in each sub-band of the spectrum [41]. To achieve this MTM operates with a bank of optimal FIR bandpass filters in order to compute the spectral estimate where the filter coefficients are based on the discrete set of eigenfunctions (tapers) or Slepian sequences. Further, the use of MTM as an efficient method for spectrum sensing in cognitive radio systems has been suggested in [42]. A drawback of MTM is that it requires a lot of computation as compared with energy detection.

In addition, recently a blind sensing spectrum technique has been introduced based on the wavelet transform which is able to deliver spectrum estimation over a wide-band of frequencies [43, 44]. In this method the wavelets are used for detecting irregularities in the spectrum to determine the boundaries of important image structures. In this light the boundaries indicate the transitions from an occupied band to an empty band and the spectrum occupancy can be assessed based on the estimated powers within these bands. However, this method is currently not feasible for practical use while it requires high costs in terms of computational complexity and a very high spectral sampling rate.

All of the above spectrum sensing techniques operate with the objective of assessing spectrum utilization where the focus is on estimating the noise variance in order to determine the optimal detection threshold. However, these spectrum sensing algorithms do not incorporate denoising techniques to improve on detection as such and to reduce the impact of non-stationary noise. This is needed as suggested in [19] to compensate for the effects of ‘dirty RF’. To elaborate on this matter, in [45–47] the SVD as denoising technique is considered which can operate in combination with MTM and this has been referred to as MTM-SVD. It has been demonstrated [45, 47] that MTM-SVD improves the decision process for spectrum sensing due to its noise reduction scheme, however, the improvement comes at the cost of unwanted bias problems where the number of tapers being used is critical in this respect. Summarizing, in literature the use of spectrum sensing in combination with denoising techniques to compensate for ‘dirty RF’ influences is not well-exposed, particularly its application to AGC noise reduction for spectrum monitoring is not addressed in that aspect.

3.1 Research objectives

To resolve the above-described spectrum monitoring in the context of ‘dirty RF’, we are interested in matrix factorization techniques for a different purpose: aim to remove the dominant noise (the non-stationary AGC noise) in order to project out the signal space using the multidimensional spectrum measurement data (i.e., the frequency, time/space and field strength). In this way the spectrum occupancy may be assessed more accurately in the case of mobile spectrum monitoring. To achieve this two state-of-the-art matrix factorization techniques are issued, i.e., singular value decomposition (SVD) [45, 48] and non-negative matrix factorization (NMF) [49–51]. In line with this, recently in [52] the initial results on spectrum occupancy for different types of areas were presented with the aid of SVD processing (applied to the UMTS band). Building upon this idea, we aim to enhance blind spectrum sensing for mobile spectrum monitoring by application of matrix factorization techniques for AGC noise removal. In doing so, the above-mentioned matrix factorization techniques are analyzed for blind spectrum sensing in the 2.4 GHz ISM band and a performance evaluation is presented together with an experimental verification. To our knowledge the proposed method to remove AGC noise for spectrum sensing applications is novel and not found in literature.

4 Mobile spectrum sensing

4.1 Mobile monitoring system

To analyze the effect of non-stationary AGC noise, we use the measurement data obtained from mobile monitoring vehicles used by the Dutch regulator. These vehicles continuously perform full spectrum sweeps while driving in a particular area. As a test case we used the monitoring recordings from the 2.4 GHz ISM band, one of the most dense frequency bands in the range 2400–2483.5 MHz. These measurements include many days of spectrum monitoring in the Netherlands, thereby employing the monitoring system [1]. In our configuration the measurements take place at a 2 s time interval in which a frequency sweep is carried out (takes in total 100 ms), using a step-size of 156.25 kHz, to sample the spectrum in the 2.4 GHz ISM band. The monitoring data, which results as output of the measurement equipment, is referred to as raw/unprocessed data. This means that for each frequency bin and time instant the received average power P _{r x} is stored per hertz of bandwidth. The spectrogram of the raw data for a sample area is provided in Figure 5 and the corresponding statistics and data distribution are shown in Figures 6 and 7. Here, the Wi-Fi channels can be identified from the monitoring results in the spectrogram because it entails most of the traffic in this band. Note that there are in total 13 overlapping channels which are actually in use, only in Japan there is a 14th channel. In practice it turns out that Wi-Fi systems mainly use the non-overlapping channels 1, 6, and 11 (due to the manufacturer default settings).

Figure 5

Monitoring the 2.4 GHz ISM band: spectrogram. The spectrogram of the raw monitoring data (field strengths in dBm/Hz) w.r.t. the 2.4 GHz ISM band in the frequency range 2400–2483.5 MHz; the time instants are denoted as traces. Here the main Wi-Fi channels are 1, 6, and 11 which correspond to frequency bandwidths of respectively, 2402–2422 MHz, 2427–2447 MHz, and 2452–2472 MHz.

Figure 6

Monitoring in the 2.4 GHz ISM band: the statistics. The statistics throughout the 2.4 GHz ISM band, i.e., the mean, median, maximum and minimum signal levels. The latter corresponds with the thermal noise floor.

Figure 7

Monitoring the 2.4 GHz ISM band: the probability density function. The PDF of the signal levels of the raw monitoring data which includes the AGC noise components.

4.2 AGC calibration

The influence of the AGC on the measurement data is investigated in this section by setting up an experiment. Note that the monitoring equipment operates with the aid of a super-heterodyne based spectrum analyzer including AGC. To elaborate on this matter, the equipment is capable to log digitally both the IF-gain and the IF lower-threshold (scaled w.r.t. the thermal noise floor) while the second AGC loop operates in the baseband domain(see Figure 3). In addition, recall that the IF-gain is set to properly scale the input signals for A/D conversion. Furthermore, in what follows the IF-gain and IF lower-threshold will be meant when reference is made to respectively the AGC gain and AGC threshold.

To investigate the influence of the AGC on the measurement equipment an experiment is set up in a controlled environment. Here the same configuration settings are used in a similar fashion as for the nationwide monitoring readings, i.e., using the same frequency sampling resolution, time observation interval, etc. In doing so, the transmission power of an unmodulated test signal is increased in a controllable way, in steps of several dBs, at the input of the monitoring equipment. Meanwhile spectrum monitoring is performed and the AGC threshold is stored for each time observation. Next, the frequency sweep results are shown in the spectrogram of Figure 8; the corresponding AGC threshold, the average AGC gain (G _AGC), signal field strength, and trace interval are listed in Table 1. Note that here the AGC gain is constant throughout the ISM band and scales with the highest detected signal which in turn causes the noise level to be lifted significantly. This effect is visible in the spectrogram of Figure 8, indicated by the horizontal strips which we refer to as wide-band artifacts. Furthermore, the probability density function (PDF) of the measurement data is plotted in Figure 9. The PDF shows three distinguishable peaks (local maxima). The leftmost distribution is a mixture of on one-hand thermal noise and on the other hand AGC noise contributions (due to the lower AGC gain of G _AGC=3.5 dB); here the dominant component is thermal noise (i.e., G _AGC=0 dB) which accounts for the Gaussian shaped distribution. In addition, the second and third peak are AGC noise contributions which correspond to the following AGC gain values: G _AGC=10.5 dB and G _AGC=19.5 dB. Finally, the signal contributions are present from -135 dBm/Hz onwards and are clipped at -110 dBm/Hz for the sake of clarity/visibility.

Figure 8

AGC Experiment: spectrogram. To show the influence of the AGC, a power varying test signal is fed to the receiver input; the spectrogram shows the AGC influence (field strengths in dBm/Hz). The input signal increases in power over time (traces) and is visible by the stepwise square-shaped signals in the spectrogram. The noise level lifts up which results in wide-band artifacts in the spectrogram. In addition, the specification of the input signal power levels are listed in Table 1.

Table 1 AGC levels

Figure 9

AGC calibration experiment: the probability density function. The distribution of the received field strengths as based on the raw spectrum monitoring data in the spectrogram of Figure 8.

Relating the test measurement back to the mobile monitoring readings sheds light on the AGC distortion in the spectrogram of Figure 5. Here the wide-band artifacts stretch several hundreds of MHz primarily due to strong signals in the nearby UMTS downlink band. In the following section, we introduce a system model to remove the AGC noise components.

5 Spectrum sensing: matrix factorization techniques

5.1 Notation

Let the unprocessed data be represented as X=[X(t,f)] in which t∈{1…N _t } denotes the discrete time instants and where f∈{1…N _f } denotes the frequency bins. Regarding X, the number of rows N _t relates to the time-frame of monitoring, which in turn consists of a fixed number of t independent time instants (a.k.a. ‘traces’). In addition, each recording at a given time instant is linked to a specific spatial coordinate. Similarly, N _f defines the total number of frequency bins, also referred to as frequency observations. Note that each element in matrix X can either be classified as a signal (denoted by ‘1’) or as noise (represented as ‘0’), based on a predefined threshold η. In line with this the spectrum availability function X _c =[X _c (t _i ,f _j )] is defined through,

$$X_c(t_i,f_j)=\left\{\begin{array}{ll}1,& X(t_i,f_j)\ge \eta \\ 0,& X(t_i,f_j)<\eta \end{array}\right\}$$

(1)

which represents the matrix of classifications where the time traces are associated with the matrix rows and the frequencies with the matrix columns. Now, a signal is detected if the measured field strength exceeds the threshold. Otherwise the data is classified as noise. As a result, the choice of threshold value is critical for correct signal detection. To analyze the distribution of the data, it is common use to look at the second order statistics and at the covariance matrix which is defined by

$$\mathbf{C}=E\left[\left(\mathbf{\text{X}}-E\left[\mathbf{\text{X}}\right]\right){\left(\mathbf{\text{X}}-E\left[\mathbf{\text{X}}\right]\right)}^H\right]$$

(2)

where E denotes the expectation operator. Furthermore, (.)^H denotes the Hermitian transpose and x _k represents the k th column of X. However, C is not known beforehand and has to be estimated. For this purpose an unbiased estimate, i.e., sample covariance matrix $\widehat{\mathbf{C}}$, is computed; the latter is derived based on samples of collected measurement data and is an approximation of the actual covariance matrix C. Now, based on a given sample set consisting of N _f frequency domain observations of the N _t dimensional random variable $X\in {ℝ}^{N_t\times 1}$ (a N _t ×1 column vector) an estimation of C is defined through,

$$\widehat{\mathbf{C}}=\frac{1}{N_f-1}\sum _{k=1}^{N_f}\left({\mathbf{\text{x}}}_k-\mathit{\mu }\right){\left({\mathbf{\text{x}}}_k-\mathit{\mu }\right)}^H$$

(3)

as the sample covariance matrix, also referred to as the estimated covariance matrix. Notice that $\widehat{\mathbf{C}}$ has N _f -1 in the denominator rather than N _f in contrast to C (see [39]). For a certain time instant t the mean received field strength is computed through integration of the corresponding N _f samples in the ISM band as follows:

$$\mathit{\mu }=\left[\begin{array}{l}{\mu }_1\\ ⋮\\ {\mu }_{N_t}\end{array}\right]=\frac{1}{N_f}\sum _{k=1}^{N_f}{\mathbf{\text{x}}}_k$$

(4)

where the column vector μ contains the mean received field strength for each time instant.

5.2 Singular value decomposition

Based on the covariance matrix, we apply a technique known as SVD, in order to project out the signal space. Note that the SVD technique is preferred over an alternative method, named eigendecomposition. This is because SVD is numerically stable, yielding always non-negative eigenvalues, unlike eigendecomposition.

As a first step the unbiased data matrix, denoted by M, is defined in order to show how to decompose the sample covariance matrix into its sample eigenvectors and eigenvalues. Here $\widehat{\mathbf{C}}=\mathbf{M}{\mathbf{M}}^H$ holds, where matrix M is derived as

$$\mathbf{M}=\frac{1}{\sqrt{N_f-1}}\left(\mathbf{\text{X}}-E\left[\mathbf{\text{X}}\right]\right)$$

(5)

Now, the singular value decomposition of matrix M can take place, which leads to the following factorization:

$$\mathbf{M}={\mathbf{W\Sigma U}}^H$$

(6)

where the diagonal entries Σ _{i i} of Σ are known as the singular values of M. It is known, from the field of principal component analysis (PCA) [4, 6, 53], that the columns of W (left singular vectors) are eigenvectors of M M ^H and the columns of U (right singular vectors) are eigenvectors of M ^H M. Hence, it can be shown that the sample covariance matrix can be decomposed into its sample eigenvectors and eigenvalues:

$$\begin{array}{l}\widehat{\mathbf{C}}=\mathbf{M}{\mathbf{M}}^H\end{array}$$

(7)

$$\begin{array}{c}=\left(\mathbf{W}\mathbf{\Sigma }{\mathbf{U}}^H\right)\left(\mathbf{U}{\mathbf{\Sigma }}^H{\mathbf{W}}^H\right)\hfill \end{array}$$

(8)

$$\begin{array}{c}=\mathbf{W}\left(\mathbf{\Sigma }{\mathbf{\Sigma }}^H\right){\mathbf{W}}^H\hfill \end{array}$$

(9)

where U ^H U=I, due to the orthogonal column structure of matrix U. Furthermore, W denotes the sample eigenvector matrix of $\widehat{\mathbf{C}}$. Thus, the sample eigenvalues and eigenvectors of $\widehat{\mathbf{C}}$ are derived by computing SVD(M)=W Σ U ^H. The sample eigenvalues can be computed, since the singular values of diagonal matrix Σ correspond to the square roots of the sample eigenvalues of matrix $\widehat{\mathbf{C}}$. The obtained sample eigenvectors, contained in matrix W, can be used to create a projection $\widetilde{\mathbf{\text{X}}}$ of the measured data X, containing solely the strongest signal space components:

$$\widetilde{\mathbf{\text{X}}}=\sum _{i=1}^L{\mathbf{w}}_i{\mathbf{w}}_i^H\left(\mathbf{\text{X}}-E\left[\mathbf{\text{X}}\right]\right)+E\left[\mathbf{\text{X}}\right]$$

(10)

where w _i are the column vectors of W., i.e., the columns w ₁…w _L correspond to the first L principal components of X. As a consequence, the principal subspace, $F={\left\{{\mathbf{W}}_i\right\}}_1^L$, is spanned by the first L principal components of X. Since mean subtraction, i.e., mean centering, is necessary for performing PCA, we get a corrected version of matrix X, by adding E[X]. As a drawback, the SVD method is not constrained to output non-negative data, which can result in a small percentage of outliers—negative valued field strength data—which causes numerical problems when translated back to logarithmic values. To overcome this problem, we manually map the negative data values to a fixed arbitrary small positive value.

5.3 Non-negative matrix factorization

Since SVD is not constrained to output positive valued data we investigate a second technique known as NMF [49] which is referred to as a dimension-reduction technique. The basic idea is the following: given the non-negative data matrix $\mathbf{X}\in {ℝ}_{+}^{N_t\times N_f}$ (i.e., the non-logarithmic representation is used) the NMF algorithm aims to construct a low-rank approximation of X of rank L. Doing so, NMF opts to find two nonnegative matrices U and W such that X≈U W. Note that the latter expression holds with equality once the residual matrix R is taken into account: X=U W+R. As such, the elements of the residual matrix R can either be negative or positive valued. Recall that the NMF algorithm provides as factorization output the two matrices U and W which are nonnegative factors of X and are defined as U=[u ₁,…,u _L ] and $\mathbf{W}={\left[{\mathbf{w}}_1^H,\dots ,{\mathbf{w}}_L^H\right]}^H$; here U is a N _t ×L matrix containing the basis vectors as its columns whereas W denotes a L×N _f matrix containing the coefficients vectors. Now, by applying NMF a low-rank approximation of the feature space of X can be achieved, i.e., the data matrix X can be estimated by using a linear combination of the of basis vectors with the coefficients vectors through,

$$\mathbf{X}=\sum _{j=1}^L{\mathbf{u}}_j{\mathbf{w}}_j^H+\mathbf{R}$$

(11)

where rank L are the number of independent principal components present in the constructed lower-ranked approximation of X. Thus, L≤ min(N _t ,N _f ) is required to establish the desired rank-reduction. Next, in order to carry out the factorization of matrix X a cost function D(X,U W) needs to be defined to quantify the quality of the approximation. Such a cost function can be constructed using some measure of distance between X and the product U W. Examples of such measures include Euclidean distance and Kullback-Leibler (KL) divergence [50, 54–57]. Henceforth the KL measure is used as cost function since the advantage of KL compared to other cost functions lies in the guaranteed convergence to a fixed point. This occurs at the expense of lower processing speed, i.e., many iterations are needed. However, this is not the case for AGC noise removal due to the off-line processing possibilities. The optimization problem is now defined as follows:

$$\underset{\mathbf{U}\in {ℝ}_{+}^{N_t\times L},\mathbf{W}\in {ℝ}_{+}^{L\times N_f}}{\text{min}}\left[D\left(\mathbf{X},\mathbf{U}\mathbf{W}\right)\right]$$

(12)

where the cost function D(X,U W) equals

$$\sum _{i=1}^{N_t}\sum _{j=1}^{N_f}\left({\mathbf{X}}_{\mathit{\text{ij}}}log\frac{{\mathbf{X}}_{\mathit{\text{ij}}}}{{\left(\mathbf{U}\mathbf{W}\right)}_{\mathit{\text{ij}}}}-{\mathbf{X}}_{\mathit{\text{ij}}}+{\left(\mathbf{U}\mathbf{W}\right)}_{\mathit{\text{ij}}}\right)$$

(13)

The objective function is convex in U, and in W, but not convex in both simultaneously. Therefore an algorithm cannot guarantee to converge to a global optimum. Using an iterative update algorithm, such as gradient search, will therefore find a local optimum of the objective function. In fact, the NMF problem does not have a unique global minimum. Fortunately, it turns out that the achieved feature extraction (i.e., rank reduction) with only local minima has been shown to be of significant quality for many applications [55].

Finally, note that the most important difference between NMF and other matrix factorization methods, such as SVD, is that the data is described by using additive components only. The reason therefore are the non-negative constraints on U and W.

6 Methodology: AGC noise removal

This section illustrates the application of the matrix factorization techniques, i.e., SVD and NMF, to remove the AGC distortion and to assess spectrum occupancy more accurately for monitoring applications in the 2.4 GHz ISM band. To start with, a performance evaluation of both techniques is carried out in order to find out which technique suits best for further analysis. Basically both techniques can be employed for the following: a decomposition of the unprocessed monitoring data into independent principal components to project out the signal space from the noise space. Let L=300 and N _t =2400; for both NMF and SVD the corrected spectrograms are derived and the associated PDFs are depicted in Figure 10. As a result it turns out that the SVD produces negative valued data (outliers) that in turn distorts the spectrogram and causes problems whilst applying the required logarithmic conversion. Here, the peak (local maximum) in the PDF of Figure 10 corresponds with the probability mass of the negative valued outliers which were mapped numerically to an arbitrary small positive valued number, as referred to in Section 5.2. Furthermore, Figure 10 shows that NMF provides similar results as SVD but without the outliers, i.e., no instant peak. Thus, in fact NMF outperforms SVD and in what follows NMF is set as the preferred matrix factorization technique for spectrum occupancy assessment in the 2.4 GHz ISM band.

Figure 10

NMF versus SVD. A performance comparison: NMF versus SVD. Here the SVD curve displays an instant peak at a received power value of P _{r x} =-168 dBm/Hz which is due to the outliers, i.e., non-negative values. This is not the case for NMF which shows a curve without the instant outlier peak. Both algorithms ran with the same number of independent principal components: L=300.

For projecting out the signal space, it is essential to select the proper number of principal components L so that the non-stationary (AGC) noise components can be removed from the spectrogram without removing signal components. This is possible since it is known from literature that the lower ranked principal components contribute to the noise space [37]. To achieve this NMF is applied to the monitoring data for several values of L in the range of {1,2400}, the corresponding PDF curves are depicted in Figure 11. The interpretation of the PDFs for different values of L is the following: a value of L>500 for instance contains many noise components, indicated by its non-smooth character; on the other hand, the PDF curves for L<50 are smoothed out and no threshold can be defined based on its shape. To derive the optimum number of principal components a performance metric is required; the performance metric in this case is the thermal noise which is distributed according to a Gaussian distribution and is provided by the calibration test measurements. By fitting the Gaussian curve with the NMF curves for different L values the corresponding fitting errors can be derived; the results show that the optimal solution, i.e., minimal fitting error, lies in the range L∈{50,500}. Note that for instance the L=10 curve shows no resemblance with the Gaussian metric curve and thus yields a high fitting error. In addition, it turns out that the optimal fitting result is obtained for L=100, which is depicted in Figure 12. Along this line, the application of NMF with L=100 to the spectrogram data in Figure 13a provides a corrected version of the spectral data with less noise as depicted in Figure 13b. The latter spectrogram is smoother since only the main principal components are left over. Furthermore, by applying NMF the AGC noise is removed in the range of P _{r x} ={-160,-157} dBm/Hz and this shows the importance of choosing a good threshold value. Note that this part of the AGC noise, present in the spectrogram as small random variations, is not covered by the first L=100 components that mainly span the signal space. In line with this, recall that the PDF of the corrected spectrogram, as depicted in Figure 12, shows more contrast and follows a bimodal distribution which arises as a mixture of two unimodal Gaussian distributions. Here, the threshold is required to be chosen at the lowest field-strength in between the two local maxima (see Figure 12). Moreover, it turns out that the derived threshold is around the same value for each selected geographical area. This is the optimal threshold value while according to the conducted calibration test experiment the first distribution is classified as thermal noise whereas the second distribution contains both (Wi-Fi) signal and AGC noise contributions. Note that the latter distribution is caused by very strong signals in nearby bands (e.g., UMTS signals) that in turn trigger the AGC to lift up the thermal noise to values in the range of P _{r x} ={-157,-152} dBm/Hz; this is indicated by the wide-band artifacts in the binary frequency occupancy spectrogram of Figure 14a.

Figure 11

NMF curves. PDF curves of the monitoring data obtained from several NMF sessions which ran with different values of L, i.e., the number of independent signal components (field strengths in dBm/Hz).

Figure 12

NMF curve fitting. The best fit, taking the Gaussian distribution as the reference, turns out to be for L = 100 independent principal components.

Figure 13

Application of NMF. Applying NMF (L=100) on the monitoring data yields a corrected version of the spectrogram. The spectrogram before applying NMF is depicted in Figure 13 a; the corrected version after applying NMF is depicted in Figure 13 b.

Figure 14

Binary occupancy plots. Occupancy plots after thresholding: black denotes used spectrum and white denotes unused spectrum. Here, the occupancy before the final AGC removal step is depicted in Figure 14 a, whereas the occupancy after the final AGC removal is depicted in Figure 14 b.

To reduce the influence of AGC the wide-band artifacts are removed from the spectrogram, i.e., the traces with more then 80 percent in-band frequency occupancy are selected and deleted. In the sequel, this step is referred to as the second AGC removal step. Notice that NMF has paved the way for this second AGC step by smoothing the wide-band artifacts which enhances their identification for removal (thus referred to as AGC removal step 1). In Figure 14b the resulting binary frequency occupancy map is depicted after the wide-band artifacts removal. In addition, the corresponding PDFs are derived and depicted in Figure 15 where the PDF of the spectrogram after application of NMF (with L=100) is shown in Figure 15a; the PDF of the spectrogram after removal of the wide-band artifacts is depicted in Figure 15b. A significant decrease in probability mass is shown with respect to the second distribution where on the other hand the thermal distribution retains its shape; based on this it turns out that the second distribution mainly contained AGC noise.

Figure 15

Final AGC removal step. A comparison of the field strength distributions: before and after applying the final AGC removal step respectively depicted in Figure 15 a and Figure 15 b.

7 Performance evaluation

7.1 Results: live spectrum monitoring

In this section, the performance of the proposed AGC removal method is evaluated in terms of spectrum occupancy, detection probability and false alarm rate. The application and the results of the proposed method to mobile monitoring data are issued first, followed up by a validation of its performance.

The proposed AGC removal method is applied to the live mobile spectrum monitoring data which in turn provides a denoised spectrogram of the monitoring data. Based on this the spectrum occupancy in the 2.4 GHz ISM band is assessed for the associated geographical area of interest. The performance of the method is compared with energy detection where a fixed threshold of η=-158 dBm/Hz is used, i.e., based on the ITU recommendation [35] wherein the threshold is placed 10 dB above the noise floor. The obtained occupancy results are listed in Table 2 for the whole Wi-Fi band, i.e., {2.4-2.4835} GHz, and for the non-overlapping default channels 1, 6 and 11. These results show that after each processing step the estimation of spectrum occupancy decreases. Remarkably, it shows that the Wi-Fi band is around 40% less occupied than one would expect based on the energy detection (ITU recommended threshold) method. The other way around, the derived spectrum occupancy based on the energy detection method overestimates the spectrum occupancy by 60%.

Table 2 Spectrum occupancy: the ISM band

In addition, the results in Table 2 show that the occupancy of the default channels is always higher than the overall Wi-Fi band occupancy, indicating in line with the expectations that the default channels convey more Wi-Fi traffic than the other channels. Furthermore, Figure 16 depicts the spectrum occupancy of the ISM band (from 2400 MHz to 2483.5 MHz) to illustrate graphically the difference in spectrum usage estimation as computed by respectively the energy detection method and the AGC removal algorithm.

Figure 16

Spectrum occupancy: a comparison. The occupancy of the 2.4 GHz ISM band for the area of interest (see Figure 5). The blue curve represents the occupancy according to energy detection using the ITU recommended threshold whereas the green curve denotes the occupancy as computed by the AGC removal algorithm (AGC removal step 2 in Table 2).

7.2 Detection performance

For the purpose of spectrum monitoring the proposed sensing method is required to operate satisfactory in the lower SNR regime as well. According to literature [24, 25] the receiver should operate with a target probability of detection of P _d =0.9 and a target probability of false alarm of P _f =0.1 at a SNR of -18 dB in the UHF/VHF bands. However, in the case of mobile spectrum monitoring the detection performance is expected to be less due to the introduced AGC noise. In this scenario spectrum sensing typically leads to a higher probability of false alarm (P _f ) while AGC noise is likely to be detected as signal power. The total noise power ${\sigma }_{\text{noise}}^2$ is thus higher and includes both the contribution due to thermal noise ${\sigma }_{\text{thermal}}^2$ and the AGC noise ${\sigma }_{\text{AGC}}^2$:

$${\sigma }_{\text{noise}}^2={\sigma }_{\text{thermal}}^2+{\sigma }_{\text{AGC}}^2$$

(14)

Here the thermal noise power ${\sigma }_{\text{thermal}}^2$ depends on the temperature T, the Boltzmann’s constant K, the system’s noise figure NF, and the frequency bandwidth B through ${\sigma }_{\text{thermal}}^2=K·T·\left(\mathit{\text{NF}}\right)·B$. The thermal noise is assumed to be Gaussian, i.e., $N(0,{\sigma }_{\text{noise}}^2)$ with a variance of ${\sigma }_{\text{noise}}^2$ in the linear domain. Most of the receivers work under the Gaussian assumption of noise and attempt to measure the noise variance. This is valid in the case of a large number of sensing samples as according to the central limit theory. Additionally, also the AGC noise is approximately Gaussian as demonstrated in Figure 2.

At the receiver detection is performed with the aid of test statistic T(X) which is based on the received power X in the frequency band of interest for N _t time observations. Here, the number of frequency band samples equals N _f which leads to the following statistic for the log-likelihood ratio test at the detector:

$$T\left(X\right)=\frac{1}{N_t{N}_f}\sum _i^{N_t}\sum _j^{N_f}X(t,f)$$

(15)

The outcome of the test can be twofold:

$$\begin{array}{ll}{H}_0:T\left(X\right)& ={\sigma }_{\text{noise}}^2\end{array}$$

(16)

$$\begin{array}{ll}{H}_1:T\left(X\right)& ={\sigma }_{\text{noise}}^2+P_{\text{sig}}\end{array}$$

(17)

Here the hypothesis H ₁ means that the frequency band is occupied; T(X) converges to ${\sigma }_{\text{noise}}^2+P_{\text{sig}}$ under H ₁, where P _sig denotes the signal power. The other way around, hypothesis H ₀ refers to a vacant band where the received power corresponds with the total noise power ${\sigma }_{\text{noise}}^2$. Next, the probabilities of detection P _d and false alarm P _f can be derived at the receiver given a chosen threshold λ:

$$\begin{array}{l}{P}_d=\text{Pr}\left\{T\left(X\right)>\lambda |H_1\right\}\end{array}$$

(18)

$$\begin{array}{l}{P}_f=\text{Pr}\left\{T\left(X\right)>\lambda |H_0\right\}\end{array}$$

(19)

In doing so the probabilities of detection (P _d ) and false alarm (P _f ) are derived for the scenario of mobile spectrum monitoring with AGC noise by using the measured sensing data. A vacant frequency channel in the 2.4 GHz range is selected for monitoring where the time bandwidth product is defined by N _f =70 and N _t =1300. A controlled signal is added to the spectral data for which the signal power is exactly known so that as a function of SNR the detection performance can be determined for the proposed scheme. A comparison is made with the energy detection method and the results are presented in Figure 17 for different fixed probabilities of false alarm. The results show the benefit of using the proposed method when compared with the energy detection method. In all three the scenarios the performance gain at a probability of detection of 90% is at least 3 dB.

Figure 17

Detection performance. The detection performance of the proposed scheme in the low SNR regime compared with the energy detection method. For different fixed probabilities of false alarm the detection results are provided. In all cases the proposed AGC removal method outperforms energy detection.

The results show that the aforementioned target probabilities as defined for the UHF/VHF bands can not be achieved, however, in this scenario the noise conditions are different and the performance gain over classic spectrum sensing methods (i.e., without mechanisms to compensate for AGC imperfections) is significantly large in this case. Nevertheless, note that an increase in sensing time can compensate for the limited performance in the lower SNR regime.

7.3 Validation of the results

In what follows a verification is carried out of the above-described country-wide monitoring results to verify the removal of exclusively AGC noise from the raw monitoring data. The validation is carried out with the aid of the monitoring data obtained from the test experiment (Section 4.2) in which the AGC was turned on. Importantly, to examine the operation of AGC removal the spectrum monitoring data is provided for the same experiment but strictly without the presence of any AGC noise. Furthermore, the verification needs to cover both stages of the AGC removal algorithm. In doing so the AGC removal step 1 (i.e., NMF) is applied to the experimental monitoring data (AGC turned on) to start with. The corresponding PDF is shown in Figure 18 together with respectively the PDF of the raw monitoring data (AGC turned on) and the PDF of the actual field-strengths (i.e., AGC turned off).

Figure 18

Validation: first AGC removal step. The PDF after AGC removal step 1 versus the PDF of the field-strengths of the raw monitoring data (with AGC) and the PDF of the actual monitoring data (i.e., without AGC). The focus is on the leftmost distribution peak which is in the field-strength range of P _{r x} ={170,-155} dBm/Hz. For the actual field-strength data this corresponds to thermal noise only whereas for the raw monitoring data also AGC noise is included. As a consequence the difference in distribution width between these two leftmost peaks equals the AGC noise contribution. Due to NMF the thermal noise and the lower AGC contribution (G _AGC=3.5 dB) become separated and turn visible as distinguishable distributions.

Figure 18 shows the AGC noise distribution (i.e., G _AGC=3.5 dB) which becomes visible after application of NMF and can be clearly distinguished from the thermal noise distribution. This paves the way for the second AGC removal step which is now capable to properly remove the AGC noise by deleting the smoothened wide-band artifacts. This is illustrated in Figure 19 where the distribution of the field-strengths is shown after application of AGC removal step 2. The latter distribution now resembles the PDF of the actual monitoring data (i.e., AGC turned off) and thus represents the thermal noise contributions only.

Figure 19

Validation: second AGC removal step. The PDF after AGC removal step 2 versus respectively the PDF of the field-strengths of the raw monitoring data (with AGC) and the PDF of the actual monitoring data (i.e., without AGC). In this step the wide-band artifacts can be removed due to the pre-work of the NMF operation. After the application of the second AGC removal step the PDF contains primarily thermal noise. Due to NMF the thermal noise distribution shows lower variance, however the probability mass herein remained the same (see Table 3).

Table 3 Validation: spectrum occupancy

Finally, the spectrum sensing performance of the proposed AGC removal method is compared with the application of energy detection (ITU-based) and with the statistics of the actual monitoring data (i.e., without AGC noise). Table 3 shows that the AGC removal algorithm outperforms energy detection and that the provided spectral estimate matches with the actual monitoring data (i.e., without AGC noise). This confirms that the proposed algorithm removes exclusively AGC noise and thus is able to provide a good estimate of the spectrum occupancy.

8 Conclusion

In this study, we have presented a novel approach based on matrix factorization techniques to improve the reliability of spectrum sensing significantly by eliminating AGC noise. To verify our method we used the measured spectrum monitoring data in the 2.4 GHz ISM band. As we know from AGC test experiments, the AGC lifts the thermal noise by a few dBs. We showed that by applying matrix factorization techniques, i.e., SVD and NMF, this part of the AGC noise can be removed; moreover, it turns out that NMF outperforms SVD. In addition, we employed a curve fitting approach to find the optimal number of components to be included for application of NMF; the resulting PDF follows a bimodal distribution and shows more contrast compared to PDF of the raw monitoring data. This is due to the fact that a large part of AGC noise is centered around the threshold. Furthermore, NMF sets the stage for the final AGC removal step in order to delete the remaining AGC noise contributions (visible as wide-band artifacts in the spectrogram). Regarding the ISM band, by removing the AGC noise from the spectrogram we have demonstrated a reduction in spectrum usage of around 40% as when compared with energy detection using the ITU recommended threshold. Additionally, the detection performance of the proposed method shows a performance gain of at least 3 dB at a probability of detection of 90% in the lower SNR regime. Further research could focus on the real-time implementation of AGC removal techniques and on its application for different frequency bands.