 Research
 Open Access
 Published:
Dynamic rainfall monitoring using microwave links
EURASIP Journal on Advances in Signal Processing volume 2016, Article number: 77 (2016)
Abstract
In this work, we propose a sparsityexploiting dynamic rainfall monitoring methodology using raininduced attenuation measurements from microwave links. To estimate rainfall field intensity dynamically from a limited number of nonlinear measurements, we exploit physical properties of the rainfall such as spatial sparsity and nonnegativity along with the dynamics of rainfall intensity. We develop a dynamic state estimation algorithm, where the aforementioned spatial properties are utilized as prior information. To exploit spatial sparsity, we use a basis function to tailor the sparse representation of the rainfall intensity. The basis is selected based on some criteria for sparse reconstruction such as orthonormality and mutual coherence. The tuning parameter that controls the sparsity in the spatial rainfall distribution is dynamically updated at every correction step. The developed methodology is applied to dynamically monitor the rainfall field intensity in an area with a specified spatial resolution using less number of simulated nonlinear measurements than pixels. The proposed methodology can be generalized for any dynamic field reconstruction, where the limited number of nonlinear measurements are field intensities integrated over a linear path.
Introduction
Spatial rainfall mapping from the measurements of raininduced attenuations collected from microwave links (used by cellular telecommunication networks) is an emerging technology which can serve as an alternative to traditional approaches like rain gauges and weather radar [1]. The practicability of the method is illustrated in [2] by comparing its performance with rain gauges and radar. The motivation behind this methodology is to utilize existing systems such as cellular networks to improve the quality of rainfall estimates using rain gauges and radar, as well as to use it as an independent rainfall measuring unit in areas, where traditional measuring modalities are scarce. The attenuation measurements from microwave links can also be used for monitoring snowfall, fog, and humidity [3]. Seminal works in this domain include tomographic rainfall mapping [4] and a stochastic implementation of the microwave tomographic inversion technique (MTIT) [5]. Recently, it has been observed that signal processing algorithms like a modified weighted least squares method can be implemented to spatially map the rainfall intensity on a regular grid, using microwave link attenuation measurements [6]. Also, a direct spatial reconstruction from nonlinear measurements using a variable grid size is exhibited in [7]. The robustness of a practical application of such techniques is illustrated in [8], where a countrywide (the Netherlands) rainfall mapping is shown to be possible using link attenuation measurements using a data set of 12 days (with a temporal resolution of 15 min). However, in order to achieve some desired spatial resolution of the rainfall field estimate (in terms of number of pixels), the number of microwave links, i.e., the number of attenuation measurements, is always much smaller than the number of pixels in a given service area. In this case, to dynamically monitor the rainfall intensity, physical properties of rainfall like spatial sparsity and nonnegativity can be exploited as extra information. In [9], a sparse reconstruction of the rainfall field from a limited number of nonlinear measurements is presented. In [10], a sparsity as well as a ridgepenalized, nonnegativity constrained, ordinary least squares method is used to estimate the spatial rainfall map from linear pathaveraged rainfall intensities, albeit for a single snapshot. Furthermore, incorporating the nonlinearity of the measurements as well as a statespace model, a spatiotemporal rainfall monitoring method using an extended Kalman filter (EKF) is described in [11]. Recently, a linear Kalman filter is used for the reconstruction of rainfall maps inspired by object tracking algorithms [12]. However, none of the above dynamic rainfall monitoring methods exploits structural properties of the rainfall field like sparsity or nonnegativity.
Commingling the concepts of the aforementioned literature, estimating a spatiotemporally evolving rainfall field can be viewed as a dynamic sparse field estimation problem, where the spatial sparsity of the rainfall field can be tailored by representing it as a sparse signal in a suitable “sparsifying” basis [13]. Such a dynamic estimation of sparse signals, also known as sparsityaware Kalman filtering, is a wellstudied problem in the field of signal processing with quite a number of applications like target tracking and video coding. Next to the spatial sparsity, also, the temporal sparsity can be exploited in the state estimation [14]. Sparsity penalties lead to a faster convergence than a clairvoyant Kalman filter, as illustrated in [14]. Also, a nonnegativity constrained sparsityaware Kalman filter is applied to the target tracking problem in [15]. In [16], the “dynamic filtering” is implemented by introducing an iterative reweighted ℓ _{1}norm penalty. In that work, a Bayesian hierarchical model is used for the dynamically varying sparse coefficients of the signal. Also, in [17], the convergence of the aforementioned approach has been illustrated, formulating it as a basis pursuit denoising (BPDN) problem. Another notable approach of tracking a sparse signal in an underdetermined measurement scenario is viewing sparsity as a pseudomeasurement and implementing a parallel state and covariance update scheme for this extra measurement [18]. In the Bayesian paradigm, a sparsityaware state estimation can be formulated as a constrained maximum a posteriori estimator (MAP)[19].
In this work, we assume that the spatial rainfall intensity can be represented as a sparse environmental signal. We assume two scenarios for the spatiotemporal evolution of the rainfall field. In the first case, we assume that the dynamics of the rainfall field are perfectly known. In this case, we use a linear but nonstationary dynamical model for the spacetime evolution of the rainfall event, which incorporates physical phenomena like advection, diffusion, and convection [20, 21]. In the second case, we assume that the information regarding the dynamics is not perfectly known. In this case, we approximate the spatiotemporal evolution by a simple Gaussian random walk model.
We develop a complete structured framework to dynamically monitor the rainfall intensity exploiting a priori knowledge, i.e., the spatial sparsity and the nonnegativity of the rainfall field. The overall dynamic rainfall monitoring setup is pictorially represented in Fig. 1. The proposed setup accepts attenuation measurements, in a given service area at any given snapshot from the operating links, whose geometry and operating frequencies are known. Accumulating these nonlinear measurements, the spatial rainfall intensity in the given service area is computed in a centralized approach with a specified resolution. The developed dynamic sparsityaware rainfall monitoring algorithm has the following salient features:

The used measurement model is nonlinear, underdetermined, and also timevarying. We perform a dynamic linearization, followed by a state estimation where sparsity and nonnegativity is utilized, in order to achieve a stable solution from the underdetermined measurement setup.

The selection of the tuning parameter controlling the sparsity in the solution can be dynamically updated in every correction step.

The algorithm is also generalized to dynamically select the representation basis that minimizes the mutual coherence between the basis matrix and the measurement matrix at a particular time instance, which represents the geometry of the available link measurements at that time instance.
The rest of the paper is organized as follows. In Section 2, the measurement model, state model, and the spatial covariance structure of the rainfall field are presented. In Section 3, the dynamic rainfall monitoring algorithm is discussed. Issues such as the selection of an appropriate sparsifying basis and the selection of the tuning parameter controlling the sparsity is discussed in Section 4. In Section 5, simulation results are presented. Here, we mention that the true value of the rainfall, i.e., the gauge adjusted radar images and the location of the microwave links are known to us. But we do not have real attenuation measurements from the microwave links. We simulate the measurements using the ground truth, available link locations using the nonlinear measurement model mentioned in Section 2 and add additive white Gaussian noise (AWGN) of known variance. Section 5 summarizes this paper and looks at future directions.
Notations: Matrices are in upper case bold while column vectors are in lower case bold. The symbol [X]_{ ij } is the (i,j)th entry of the matrix X and [x]_{ i } is the ith entry of the vector x. The identity matrix of size N×N is denoted by I _{ N }. The transpose operator is denoted by (·)^{T}, \(\hat {{\mathbf {x}}}\) is the estimate of x, \(\triangleq \) defines an entity, and \({\{\mathbf {x}}\}_{p} = {\left (\sum _{i=0}^{N1}\left [ {x}]_{i}\right ^{p}\right)}^{1/p}\) is the ℓ _{ p } norm of x. The symbols 0_{ N } and 1_{ N } are the vectors of all zeros and ones of length N, respectively. The set of N×N symmmetric positive semidefinite and positive definite matrices are denoted by \({\mathbb {S}}^{N}_{+}\) and \({\mathbb {S}}^{N}_{++}\), respectively.
Signal model
In this section, we first describe the nonlinear measurement model which can be used to dynamically estimate the rainfall intensity with a prescribed spatial resolution by the timevarying attenuation measurements. Next, we describe the structure of the spatial covariance matrix of the rainfall field. After that, we describe the dynamic state model, based on the physics behind the movement of a rainstorm both spatially and temporally.
Nonlinear measurement model
The geometry of the microwave links deployed by any telecommunication service provider in any area is fixed. These links can be viewed as a fixed network of sensors to monitor rainfall since the received signal level (RSL) measurements related to these links depend on the rainfall. Note that the signal attenuation on a microwave link is not only due to rainfall but also depends on other atmospheric effects like humidity, wet antenna attenuation, and propagation loss [6]. For simplicity, we assume that the attenuation caused by these other effects (except precipitation) can be precomputed, e.g., during “dry periods”, and subtracted from the recorded RSL measurements. So, the effective measurements only include the raininduced attenuation. The conventional empirical relationship between the raininduced specific attenuation and the pathaveraged rainfall rate is given by y _{ s }=a r ^{b}, where y _{ s } is the specific attenuation of the link (dB/km), and r is the pathaveraged rainfall rate over the link (mm/h) [22]. If L is the length (km) of the microwave link, then the total raininduced attenuation over the link is y=a r ^{b} L dB. The parameters a and b are related to the drop size distribution (DSD) of rain, the polarization and frequency of the transmitted electromagnetic wave, the length of the link, the ambient temperature, etc. It has been extensively studied and shown in several works that variations of the aforementioned environmental and nonenvironmental parameters affect the estimate of the pathaveraged rainfall rate. A quantitative analysis of DSD related errors in estimating the pathaveraged rainfall from direct raininduced attenuation measurements is illustrated in [23, 24]. It can be observed that the attenuation for links operating in frequencies around 35 GHz can be treated as a linear measurement of the pathaveraged rainfall rate [23]. A detailed analysis of the effects of the frequency, DSD, link length, and temporal sampling in estimating the pathaveraged rainfall rate has been presented in [25, 26]. Also, in a wide coverage area, the link (measurement) availability in different hours of the day may significantly vary. All of these aforementioned studies advocate a dynamic tuning of the a and b coefficients in order to better monitor the rainfall from link attenuations.
The nonlinear attenuation measurements from the microwave links in any given service area for a fixed time can be used to estimate the spatial rainfall intensity over the same area. Let us consider a uniform discretization of the specified service area \({\mathcal {A}}\) (square) into N pixels where we would like to estimate the rainfall intensity. Here, we make the assumption that the rainfall intensity is constant within any pixel. This assumption is flexible as any resolution can be attained by tailoring the size of the square pixels. Let us assume that there are M links in the given service area. The length of the ith link can be written as \(L_{i}=\sum _{j=1}^{N}\;l_{ij}\), where l _{ ij } is the length of the ith link passing through the jth pixel, where i=1,…,M. If the ithe link does not pass through the jth pixel then l _{ ij }=0; otherwise, it is computed by the link and the pixel coordinates. The total attenuation over a link can be modelled as the sum of the attenuations over the link segments [6]. Using this, the attenuation over the ith link at time t can be expressed as \(y_{i,t} \approx \sum _{j=1}^{N} y_{ij,t}\), where y _{ i j,t } is the attenuation over the linksegment of length l _{ ij }. Using the powerlaw relationship for the attenuations over the link segments, the measurement model can be constructed in the following way,
where y _{ i,t } is the attenuation measurement of the ith link and u _{ j,t } is the intensity of the rainfall field in the jth pixel at time t. The powerlaw coefficients of the ith link at time t are given by a _{ i,t } and b _{ i,t }. The measurement model in (1) is a generalized timevarying nonlinear tomographic measurement model. In this work, we consider the fact that all the M links are operated in the same frequency and that the other environmental conditions (e.g., DSD, temperature) are fixed for all t. Based on these assumptions, the aforementioned measurement model can be simplified as
The measurement noise incurred at the ith link measurement at time t is given by e _{ i,t }. The measurements are corrupted by errors which are mainly due to quantization but also other sources of noise exist. A more detailed description of the statistical nature of the measurement noise can be found in [6]. For the sake of simplicity, let us assume that e _{ i,t } is zeromean spatiotemporally white Gaussian noise with variance \({\sigma _{e}^{2}}\). Further, we assume that e _{ i,t } is uncorrelated with u _{ j,t }.
Combining all the measurements from the M links at time t, we can construct the following nonlinear measurement model at time t:
where \(\mathbf {y}_{t}\in {\mathbb {R}}^{M}\) stacks the measurements from the M links at time t, whereas \({\bf {e}}_{t}\in {\mathbb {R}}^{M}\) does the same for the noise. The vector \(\mathbf {u}_{t}\in {\mathbb {R}}^{N}\) gathers the rainfall intensities for all of the N pixels at time t, i.e., it is the parameter to be estimated dynamically. The nonlinear mapping between the rainfall intensities and the attenuation measurements is given by \({\boldsymbol {\Phi }}: {\mathbb {R}}^{N}\rightarrow {\mathbb {R}}^{M}\) and it is assumed to be perfectly known. The elements of u _{ t } are given by [u _{ t }]_{ j }=u _{ j,t }=u _{ t }(x _{ j }), where u _{ t }(x) represents the continuous rainfall field at any arbitrary location \({\mathbf {x}}\in {\mathbb {R}}^{2}\) and x _{ j }=[x _{ j },y _{ j }]^{T} is the centroid of the jth pixel of the service area. The measurement noise components associated with the M link measurements are characterized by \({\mathbf {e}}_{t}\sim {\mathcal {N}}({\mathbf {0}},{\mathbf {R}}_{t})\), where \({\mathbf {R}}_{t} = {\mathbf {R}} = {\sigma _{e}^{2}}{\mathbf {I}}_{M}\).
Spatial variability of u _{ t }
At any snapshot t, the spatial rainfall intensity u _{ t }(x _{ j }), for j=1,…,N can be viewed as a widesense stationary (WSS) random process. In spatial statistics, u _{ t }(x _{ j }) is WSS (or secondorder stationary) if it satisfies \(\mathbb {E}[u_{t}({\mathbf {x}}_{j})]=\mu _{t}\) (for all j=1,…,N in the service area) and if the spatial covariance between any two points is dependent only on the distance between them (i.e., isotropic) [27]. The parameter μ _{ t } is the mean/trend of the rainfall field. A variogram model can be used to represent the spatial variations. The variogram (i.e., 2ξ(h)) or semivariogram (i.e., ξ(h)), as a function of \(h\triangleq \\mathbf {x}_{i}\mathbf {x}_{j}\_{2}\), can be written as \(\xi (h)=\frac {1}{2}{\mathbb {E}}{[u_{t}(\mathbf {x}_{i})u_{t}(\mathbf {x}_{j})]^{2}}\) for any two spatial points, x _{ i }=[x _{ i },y _{ i }]^{T}, x _{ j }=[x _{ j },y _{ j }]^{T}, ∀i,j∈{1,…,N} [27]. Generally, several variogram models are used as it is computationally hard to calculate the spatial dependency for every lag distance h. Some statistical functions like a Gaussian, exponential, or empirically fitted models like spherical functions are often used as variogram models [28]. From the analysis of [29], the spherical variogram model is seen to be an appropriate model to describe the spatial variability of rainfall. This is given as,
The parameters that characterize a variogram model are the sill N _{0}+S _{0} of the variogram (ξ(h) for h→∞) with S _{0} as the partial sill, the nugget N _{0} (nonzero value of ξ(h) for h→0), and the range d (value of h for which the variogram reaches the sill). The advantage of the spherical variogram model is that the parameters S _{0}, N _{0}, and d can be well estimated in hourly scales for a specific day of the year [29]. Now, the spatial covariance function C _{ v }(h) can be defined as \(C_{v}(h)=\mathbb {E} [({{u}}_{t}({\mathbf {x}}_{i})\mu _{t})({{u}}_{t}({\mathbf {x}}_{j})\mu _{t})]\). Using the secondorder stationarity of the random process u _{ t }(x _{ j }), the semivariogram can be related to the spatial covariance function C _{ v }(h) by the relation ξ(h)=(N _{0}+S _{0})−C _{ v }(h) [27]. Now, the elements of the spatial covariance matrix Σ _{ u } can be computed as [Σ _{ u }]_{ ij }=C _{ v }(∥x _{ i }−x _{ j }∥_{2}), ∀i,j∈{1,…,N}. The covariance matrix, constructed in this way, is symmetric and positive definite.
State model
A Kernelbased state model
One standard approach of modelling the spatiotemporal evolution of any environmental field is based on the integrodifference equation (IDE) [28]. Following this approach, the dynamics of the rainfall field for any specific temporal sampling interval δ _{ t } can be modelled as the following discrete time IDE
Here, g(x,x ^{′};θ) is the spacetime interaction function parameterized by θ, which can be deterministic or random and dependent on the temporal sampling interval δ _{ t }. The quantity q _{ t }(x) is the process noise which is generally modelled as independent in time but correlated in space.
The spacetime interaction function g(·) can be modelled as a parameterized Gaussian dispersal kernel which captures the underlying physical processes behind the spatiotemporal evolution of rainfall, i.e., diffusion, advection, and convection [20, 21]. In this case, the spacetime interaction function is given as g(x,x ^{′};w _{ t },D,α)=α exp[−(x−x ^{′}−w _{ t })^{T} D ^{−1}(x−x ^{′}−w _{ t })], i.e., a Gaussian kernel. The translation parameter of the kernel, i.e., \({\mathbf {w}}_{t}\in {\mathbb {R}}^{2}\) models the timevarying advective displacement, i.e., the spatial drift of the rain storm, and the dilation parameter of the kernel, i.e., \({\mathbf {D}}\in {\mathbb {S}}^{2}_{++}\) models the diffusion. Note that w _{ t } can also vary with space but we assume that it is averaged over the entire area and fixed. The diffusion coefficient D can be used to model isotropic as well as anisotropic diffusion. The amount and the directions of the spatial anisotropy can be introduced by D. The parameter D can also vary with time but this is not considered here. The scalar scaling parameter \(\alpha \in {\mathbb {R}}_{++}\) is used to control the stability (i.e., to avoid the explosive growth) of the dynamic process.
Here, the entire service area is uniformly discritized into N pixels. We assume a state transition matrix \({\mathbf {H}}_{t}\in {\mathbb {R}}^{N\times N}\) whose elements are modelled by the aforementioned simple 2D Gaussian kernel. After proper vectorization of the field intensities and state noise for N pixels, we obtain
where the elements of the state transition matrix H _{ t } are given by [H _{ t }]_{ ij }=α exp[−(x _{ i }−x _{ j }−w _{ t })^{T} D ^{−1}(x _{ i }−x _{ j }−w _{ t })], and q _{ t } is the spatially colored yet temporally white Gaussian state noise vector. The quantity w _{ t } is the advective displacement during the temporal sampling interval δ _{ t }, which can be represented more precisely as w _{ t }=v _{ t } δ _{ t }, where v _{ t } is the advection velocity. Note that the aforementioned model is nonstationary when the advection vector w _{ t } changes with time, which happens in many real scenarios [20]. If there is no advection, i.e., w _{ t }=0 and D=I, the model is stationary and isotropic. We assume that the dynamic model, i.e., the state transition matrix H _{ t } is perfectly known through the parameters w _{ t }, D, and α which are considered to be deterministic and known. Without loss of generality, we follow the assumptions of [20] and [30] that the distribution of q _{ t } is given by \({\mathbf {q}}_{t}\sim {\mathcal {N}}({\mathbf {0}}_{N},{\mathbf {Q}}_{t})\). But this assumption is not true in practical scenarios because the rainfall process cannot be negative. In the simulation section, after generating u _{ t } using the sate model of (6), we set the negative elements of u _{ t } to 0. This is a modelling approximation.
One notable advantage of the model in [20] is the linear relation of the rainfall intensities in one snapshot with the ones in the previous snapshot.
Gaussian random walk model
In the last section, we assume that the parameters of the state model are perfectly known. But in many practical scenarios for a large N, it can be computationally intractable to estimate the N ^{2} elements of the state transition matrix H _{ t } using the available data. In this case, without any prior knowledge regarding the parameterization of H _{ t }, one way to approximate the dynamics is by assuming that the process follows a Gaussian random walk model [31]. In this case, we assume that H _{ t }=H=I and the process model is given by
The benefit of a Gaussian random walk model is that it has very few model parameters rather than a parameterized process model as mentioned in Section 2.3.1.
Note that the parameterized state model of (6) can be viewed as a random walk model by incorporating negligible diffusion, i.e., D=ε I, where ε≪1 and no advection, i.e., w _{ t }=0. In this case, we have H _{ t }≈I assuming α=1.
Structure of the state error covariance matrix
It is assumed that the state error, i.e., q _{ t }, is a spatially colored but temporally white Gaussian process. Assuming spatial isotropy and stationarity of the state error q _{ t }, the elements of the covariance matrix Q _{ t }=Q can be represented using the Matern covariance function as
where Γ(·) is the Gamma function, K _{ p }(·) is the modified Bessel function of the second kind, and γ is a positive shaping parameter [28]. With p→∞ and p=1/2, (8) becomes the squared exponential and the exponential covariance functions, i.e., \([\!{\mathbf {Q}}]_{ij}=\sigma _{s}^{2} \exp \left (\frac {\{\mathbf {x}}_{i}}{\mathbf {x}}_{j}\_{2}^{2}{2\gamma ^{2}}\right)\), and \([\!{\mathbf {Q}}]_{ij}=\sigma _{s}^{2} \exp \left (\frac {\{\mathbf {x}}_{i}}{\mathbf {x}}_{j}\_{2}{\gamma }\right)\), respectively.
Dynamic rainfall mapping
We dynamically estimate the rainfall intensities at the N pixels, i.e., u _{ t } at t=1,…,T snapshots from the attenuation measurements y _{ t } at t=1,…,T. The measurement and state models can be represented in the following forms
A standard practice to estimate the rainfall intensity u _{ t } at every time t=1,…,T from the measurement and state equations of (9) and (10) is the nonlinear semblance of the standard Kalman filter, i.e., the extended EKF [32]. Note that we have nonlinearity only in the measurements.
As one of the criteria for the optimal behavior of the Kalman fiter, we assume that the measurement and the state noise statistics are completely known. The measurement and the state noises are characterized by \({\mathbf {e}}_{t}\sim {\mathcal {N}}({\mathbf {0}}_{M},{\mathbf {R}})\), and \({\mathbf {q}}_{t}\sim {\mathcal {N}}({\mathbf {0}}_{N},{\mathbf {Q}})\), respectively. The dimension of the measurement noise covariance matrix depends on the number of the available measurements at time t. As the state model is a linear function of u _{ t }, the standard Kalman fiter prediction steps are given by
where the prediction of u _{ t } from the last t−1 observations is given by \({\hat {\mathbf {u}}}_{t\vert t1}\) with the error covariance matrix \({{\mathbf {M}}}_{t\vert t1}={\mathbb {E}}[({\mathbf {u}}_{t}{\hat {\mathbf {u}}}_{tt1})({\mathbf {u}}_{t}{\hat {\mathbf {u}}}_{tt1})^{T}]\) [32]. The terms \({\hat {\mathbf {u}}}_{t1\vert t1}\) and M _{ t−1t−1} are calculated in the previous time step.
The prediction based on the state model is corrected by the measurements. But here, we have a nonlinear measurement model. To linearize that model, let us introduce the M×N Jacobian matrix computed at \({\mathbf {u}}_{t}={\hat {\mathbf {u}}}_{tt1}\) as \({\mathbf {J}}_{t}=\left.\frac {\partial {\boldsymbol {\Phi }}}{\partial {\mathbf {u}}_{t}^{T}}\right _{{\mathbf {u}}_{t}={\hat {\mathbf {u}}}_{tt1}}\). The elements of the Jacobian matrix are given by \([{\mathbf {J}}_{t}]_{ij}=a b l_{ij} [{\hat {\mathbf {u}}}_{tt1}]_{j}^{b1}\), with i=1,…,M, and j=1,…,N. A first order Taylor series expansion of the nonlinear measurement function around \({\hat {\mathbf {u}}}_{tt1}\) is then given as \({\boldsymbol {\Phi }}({\mathbf {u}}_{t})\approx {\boldsymbol {\Phi }}({\hat {\mathbf {u}}}_{tt1}) + {\mathbf {J}}_{t}[{\mathbf {u}}_{t}{\hat {\mathbf {u}}}_{tt1}]\). Substituting this in (9), we obtain the following linearized measurement equation:
where \({\tilde {\mathbf {y}}}_{t}={\mathbf {y}}_{t}{\boldsymbol {\Phi }}({\hat {\mathbf {u}}}_{tt1})+{\mathbf {J}}_{t}{\hat {\mathbf {u}}}_{tt1}\). Note that here, we have less observations than unknowns, i.e., the number of links (M) is much smaller than the number of pixels (N), i.e., the dimension of u _{ t }. Hence, in the correction step, to utilize the measurements along with the state model, we need to solve the underdetermined system (13) in order to update \({\hat {\mathbf {u}}}_{tt1}\) leading to \({\hat {\mathbf {u}}}_{tt}\). After the dynamic linearization, the state estimates can be obtained using a standard Kalman filter. In this case, both the expressions for the state estimate \({\hat {\mathbf {u}}}_{tt}\) and its state error covariance M _{ tt } can be obtained in closed form [32].
Limitations of a standard EKF
The estimation of u _{ t } from only M measurements using an ordinary EKF has the following uncertainties.

First of all, the quality of the estimate strongly depends on the degree of nonlinearity and the accuracy of the linearization [32]. Also, for a highly underdetermined (M≪N) and unpredictable measurement matrix (many rows of J _{ t } can be zero for any \({\hat {\mathbf {u}}}_{tt1}\)), the solution can be highly inaccurate and dependent mainly on the predictions using the state model and the initialization.

In the above case, if the available information regarding the dynamics are incomplete or imperfectly known, then the prediction using the state model will be inaccurate. In this case, an ordinary EKF may produce unrealistic estimates in the presence of high measurement noise.

Also, there is no guarantee that an ordinary EKF will always produce nonnegative estimate of \({\hat {\mathbf {u}}}_{tt}\). For instance, let us assume that an element of the predicted value, i.e., \([{\hat {\mathbf {u}}}_{tt1}]_{j}\) (predicted using (11)) is less than 0 at any t. In that case, if l _{ ij }≠0, we may have an imaginary \([{\mathbf {J}}_{t}]_{ij}=a b l_{ij} [{\hat {\mathbf {u}}}_{tt1}]_{j}^{b1}\), if b−1 is a fractional quantity. As mentioned in [22], the standard values for b mainly lie in the interval of 0<b<2.
In these circumstances, any further prior information about u _{ t } (beyond the dynamics) is desirable to achieve a stable and more accurate solution.
Available prior knowledge regarding rainfall field
Prior information about u _{ t } can be acquired from the physical properties of rainfall such as sparsity and nonnegativity. In a given area, the rainfall intensity itself can be assumed to be a sparsely distributed environmental field over the entire service area [9, 33]. But sparsity can also be introduced by representing u _{ t } in an orthonormal basis Ψ _{ t }, which can in principle be timevarying. When rainfall itself is sparse, we simply have Ψ _{ t }=I. Denoting u _{ t }=Ψ _{ t } z _{ t }, sparsity is measured by the number of nonzero entries in z _{ t }, i.e., ∥z _{ t }∥_{0}.
As the rainfall intensity cannot be negative, another prior knowledge about u _{ t } is the nonnegativity of the rainfall field. For N pixels, this is represented as u _{ t }≥0 _{ N }.
Comment: Here, we mention that the prior information regarding sparsity and nonnegativity along with the measurements can be efficiently utilized to monitor the rainfall over multiple snapshots. For this, we do not need any information regarding the dynamics. This can be implemented for both linear [10] as well as nonlinear [9] measurement models. However, one limitation of this dynamicsagnostic method is that the rainfall events should occur in areas where microwave links are present for accurate estimation. Otherwise, the effect of the measurement noise can be dominant. In this case, we need other spatial/temporal information (e.g., covariance structure, dynamics) to interpolate the rainfall field over the entire service area.
In the next section, we illustrate iterative approaches to dynamically estimate the state of u _{ t } for t=1,…,T, exploiting both sparsity and nonnegativity.
Estimation of u _{ t }
A simple Kalman estimation step without the sparsity and the nonnegativity constraint can be formulated as the following weighted least squares optimization problem [34]:
This estimation step is not aware of sparsity or nonnegativity. The sparsity information can be incorporated in the optimization problem of (14), by adding an ℓ _{1}penalty that enforces sparsity. Note that here, we use the ℓ _{1} norm as a convex relaxation of the nonconvex ℓ _{0} norm. Using the sparse representation of u _{ t }, i.e., z _{ t }, the optimization problem of (14) can be formulated as a sparsity and nonnegativity constrained optimization problem. This can be given as
where λ _{ t } is the tuning parameter that controls sparsity. The standard error covariance update of \({\hat {\mathbf {u}}}_{tt}\) for the estimation step (14) is given by \({\mathbf {M}}_{tt}={\mathbf {M}}_{tt1}{\mathbf {M}}_{tt1}{\mathbf {J}}_{t}^{T}({\mathbf {R}}_{t}+ {\mathbf {J}}_{t}{\mathbf {M}}_{tt1}{\mathbf {J}}_{t}^{T})^{1}{\mathbf {J}}_{t}{\mathbf {M}}_{tt1}=({\mathbf {M}}_{tt1}^{1}+{\mathbf {J}}_{t}^{T}{\mathbf {R}}_{t}^{1}{\mathbf {J}}_{t})^{1}\) [32]. This expression of M _{ tt } can be used to update the covariance of the estimate of (15) but is an approximation as it is not aware of the sparsity and the nonnegativity constraint. If we do not consider to propagate the secondorder statistics of the estimate, like in the traditional Kalman filter, the state noise minimization term in (15) can also be regularized by Q instead of M _{ tt−1}. This can be viewed as a weighted least squares problem to estimate u _{ t } using the measurement (13) and the state Eq. (10) constrained by sparsity and nonnegativity. In this case, the simple iterative state estimates are given by
Note that this is a suboptimal approach to dynamically estimate the states u _{ t } avoiding the computation of M _{ tt }. As mentioned in [14], different penalties (like ℓ _{2} or ℓ _{1}) can be applied to the state error minimization term in (18) depending on the nature of the sparse state (u _{ t }) and/or the state noise (q _{ t }).
Formulation of a constrained MAP estimator for u _{ t }
Using the representation of u _{ t } in the Ψ _{ t }=Ψ domain, the measurement and state equation of (13) and (10) can be written as
where \({\tilde {\mathbf {H}}}_{t}={\boldsymbol {\Psi }}^{T}{\mathbf {H}}_{t}{\boldsymbol {\Psi }}\), \({\tilde {\mathbf {q}}}_{t}={\boldsymbol {\Psi }}^{T}{\mathbf {q}}_{t}\), and \({\tilde {\mathbf {q}}}_{t}\sim {\mathcal {N}}({\mathbf {0}},{\boldsymbol {\Psi }}^{T}{\mathbf {Q}}{\boldsymbol {\Psi }})\), where \({\mathbf {q}}_{t}\sim {\mathcal {N}}({\mathbf {0}},{\mathbf {Q}})\). From the above measurement and state equations, we can derive the conditional probability density functions \(p({\tilde {\mathbf {y}}}_{t}{\mathbf {z}}_{t})\sim {\mathcal {N}}({\mathbf {J}}_{t}{\boldsymbol {\Psi }}{\mathbf {z}}_{t},{\mathbf {R}})\) and \(p({\mathbf {z}}_{t}{\mathbf {z}}_{t1})\sim {\mathcal {N}}({\tilde {\mathbf {H}}}_{t}{\mathbf {z}}_{t1},{\boldsymbol {\Psi }}^{T}{\mathbf {Q}}{\boldsymbol {\Psi }})\). Using Bayes’ rule, the posterior pdf \(p({\mathbf {z}}_{t}{\tilde {\mathbf {y}}}_{t})\) can be given as \(p({\mathbf {z}}_{t}{\tilde {\mathbf {y}}}_{t})\propto p({\tilde {\mathbf {y}}}_{t}{\mathbf {z}}_{t})p({\mathbf {z}}_{t}{\mathbf {z}}_{t1})\). So, a MAP estimator for z _{ t } can be formulated as,
where z _{ t−1} is computed from the previous time step. However, there is no guarantee that the estimator in (22) will produce a sparse estimate of z _{ t }. On the other hand, the representation of u _{ t } in the Ψ domain is targeted to exploit sparsity. So, the estimator of (22) can be formulated as a constrained MAP estimator by adding the sparsity and nonnegativity constraint in the optimization problem of (22). After substituting the pdfs, following the same approach as used in (15), the sparsity and nonnegativity constrained MAP estimator can be given as
where \({\tilde {\lambda _{t}}}\) controls the sparsity in the estimate \({\hat {\mathbf {z}}}_{t}\). From the solution of (23), the state estimate is given as \({\hat {\mathbf {u}}}_{tt}={\boldsymbol {\Psi }}{\hat {\mathbf {z}}}_{t}\). It is seen that resorting to a Bayesian paradigm, the developed constrained MAP estimator of (23) has a structure similar to the optimization problem of (18).
Selection of the basis (Ψ _{ t }) and the tuning parameter (λ _{ t })
Selection of the basis (Ψ _{ t })
If the spatial rainfall distribution is physically sparse, then we simply solve the optimization problem of (18) for Ψ _{ t }=I. But in the absence of physical sparsity, which is a more general case in many practical scenarios, we use a basis Ψ _{ t }. Revisiting the celebrated theory of compressive sampling, we know that some of the properties of both Ψ _{ t } and J _{ t }, (or Θ _{ t }=J _{ t } Ψ _{ t }) like mutual coherence and restricted isometry property (RIP) are important in the framework of sparse reconstruction [13]. Let us denote the quantity μ(Θ _{ t }) as the mutual coherence of the matrix Θ _{ t }, which is the maximum absolute inner product of different columns of Θ _{ t } [35]. Without the state error minimization term and the nonnegativity constraint in (18), the problem is a simple BPDN problem. As derived in [36], if a suitable sparse representation of z _{ t } is possible, which is given by \(\{\mathbf {z}}_{t}\_{0}< {\frac {1}{2}}(1+\frac {1}{{\mu }({\boldsymbol {\Theta }}_{t})})\), then a “stable” solution with the standard BPDN algorithm can be obtained with a bound on the estimation error. Along the same lines, the cost function of (18) can be viewed as a BPDN problem by augmenting the measurement and the state noise minimization terms into a single least squares term [17]. In [17], the convergence guarantees of the aforementioned BPDN problem are also derived based on some assumptions on the dynamics and the measurement matrix (here J _{ t }).
In our application, the design of the measurement matrix J _{ t }, in every snapshot, is dictated by the link locations and the predicted state estimates (\({\hat {\mathbf {u}}}_{tt1}\)). So, to maximally exploit the sparsity information of the rainfall field, we focus mainly on a suitable sparse representation of the state u _{ t }. Standard orthonormal bases such as a discrete cosine transform (DCT) basis C or wavelet basis W are quite popular in sparse signal representation for communications as well as image processing. Also, a Gaussian basis function can be used to sparsely represent environmental signals [37]. However, an orthonormal basis can also be constructed using the spatial covariance matrix of the rainfall field. An orthonormal basis can be constructed by the spatial covariance matrix Σ _{ u } described in Section 2.2, by simply choosing Ψ _{ t }=U, where Σ _{ u }=U Λ U ^{T} is the eigenvalue decomposition of Σ _{ u } with U ^{T} U=I and Λ a diagonal matrix. In this case, z _{ t }=U ^{T} u _{ t } is similar to applying a KarhunenLoeve transform (KLT), which is also advocated as a sparse representation technique [38].
We choose a basis Ψ _{ t } that has a minimum mutual coherence with the measurement matrix J _{ t }. Mutual coherence can be measured for the overall dictionary Θ _{ t }=J _{ t } Ψ _{ t }. In this case, μ(Θ _{ t }) can be quantified as the maximum magnitude offdiagonal element of \({\mathbf {D}}_{t}={\breve {\boldsymbol {\Theta }}}_{t}^{T}{\breve {{\boldsymbol {\Theta }}}_{t}}\), where \({\breve {{\boldsymbol {\Theta }}}_{t}}\) is obtained by normalizing the columns of Θ _{ t }. In this case, the mutual coherence can be defined as μ(J _{ t } Ψ _{ t })=μ(Θ _{ t })= maxl,k; l≠k[D _{ t }]_{ l,k } [39]. So, given a set \({\mathcal {U}}\) of U sparsifying basis matrices, the minimal coherence basis matrix at time t can be selected by solving the following optimization problem,
Note that to optimally select the basis, we need to solve this optimization problem on every snapshot as the matrix J _{ t } is recomputed at every time step t. In our simulations, we specifically use \({\mathcal {U}}=\{{\mathbf {C}}, {\mathbf {U}}\}\).
Selection of the tuning parameter (λ _{ t })
The sparsity regulating parameter λ _{ t } in the optimization problem of (18) can be adapted dynamically. It can also be kept fixed for multiple snapshots for monitoring a short period of rainfall, within which the sparsity pattern can be assumed to be fixed. An upper bound on λ _{ t } is given by \(\lambda _{t}=\lambda _{t}^{\text {max}}\), which gives the sparsest solution, i.e., \({\hat {\mathbf {z}}}_{t}={\mathbf {0}}_{N}\) or \({\hat {\mathbf {u}}}_{t}={\boldsymbol {\Psi }}_{t}{\hat {\mathbf {z}}}_{t}={\mathbf {0}}_{N}\). Note that the cost function of (18) is nondifferentiable but convex for λ _{ t }>0. So, following the methodologies of [15] and [40], we use a subdifferential based approach to compute \(\lambda _{t}^{\text {max}}\). The subgradient of the nondifferentiable cost of (18) with respect to z _{ t } can be written as
where \({\tilde {\nabla }}_{{\mathbf {z}}_{t}}\) is the subgradient operator towards z _{ t }. Using the firstorder optimality condition, we have
where j=1,…,N. Now, we consider the case z _{ t }=0. Substituting this in the above equation, the optimal value of \(\lambda _{t}^{\text {max}}\) can be selected as \(\lambda _{t}^{\text {max}}=\2({\boldsymbol {\Psi }}_{t}^{T}{\mathbf {Q}}^{1}{\hat {\mathbf {u}}}_{tt1}+{\boldsymbol {\Psi }}_{t}^{T}{\mathbf {J}}_{t}^{T}{\mathbf {R}}_{t}^{1}{\mathbf {y}}_{t})\_{\infty }\). So, a useful range for λ _{ t } is given by \([0,\lambda _{t}^{\text {max}})\).
Traditional approaches to select the tuning parameter for an ℓ _{1}penalized regression problem are crossvalidation and generalized crossvalidation (GCV) [41]. Recent methods suggest information theoretic approaches like Mallow’s Cp type criterion [42], Akaike information criterion (AIC) [43], and Bayesian information criterion (BIC) [44] to find an optimal λ _{ t }. In all of these approaches, the optimal tuning parameter is selected that minimizes a cost function which depends upon the estimate of u _{ t } using a set of \(\{{\lambda }_{t}^{k}\}_{k=1}^{K}\), where the length of the search grid for λ _{ t } is K. In this case, we need to solve the optimization problem of (18) K times in every iteration and select the optimal \({\lambda }_{t}^{k}\) that minimizes any of these aforementioned model selection criteria. After that, (18) needs to be solved again with the selected \({\lambda }_{t}^{k}\), in order to estimate u _{ t }. This seems to be computationally unrealistic for an online application of the dynamic rainfall monitoring over a large service area (large N). To circumvent this problem, the K optimization problems can be solved once and the selected λ _{ t } can be used for multiple snapshots for a short term monitoring application.
It is clear that increasing the value of λ _{ t }, i.e., the term ∥z _{ t }∥_{1} becomes smaller and vice versa. So, if an approximation of z _{ t }, i.e., \({\mathbf {z}}_{t}^{\text {approx}}\), is available, it can be related to the tuning parameter by \(\lambda _{t} \propto 1/\{\mathbf {z}}_{t}^{\text {approx}}\_{1}\). Following this, a coarse but relatively fast approach to dynamically tune λ _{ t } could be selecting \(\lambda _{t} = \nu (\{\boldsymbol {\Psi }}_{t}^{T}{\hat {\mathbf {u}}}_{tt1}\_{1})^{1}\), where \({\hat {\mathbf {u}}}_{tt1}\) can be regarded as an approximation of \({\hat {\mathbf {u}}}_{tt}\) and ν>0 is a proportionality constant. In our simulations, we use ν=1.
For the sake of completeness, we summarize the steps of the two proposed dynamic rainfall monitoring algorithms. In algorithm 1, we follow the standard steps of dynamic state estimation, but we do not update the secondorder statistics of the estimate. In algorithm 2, we use the approximate approach, where we use the standard Kalman covariance update (unaware of sparsity and nonnegativity).
The performance of both these algorithms strongly depends upon the initialization \({\hat {\mathbf {u}}}_{00}\). One should avoid initializations like an all zero vector or an \({\hat {\mathbf {u}}}_{00}\) that consists of negative elements. If we consider the initialization \({\hat {\mathbf {u}}}_{00}={\mathbf {0}}_{N}\), it will produce J _{ t }=0 _{ M×N }, as \({\hat {\mathbf {u}}}_{10}={\mathbf {H}}_{t}{\hat {\mathbf {u}}}_{00}={\mathbf {0}}_{N}\). It is mentioned in [22], that the standard values for b mainly lie in the interval of 0<b<2. It should also be noted that, for b<1 (for frequencies in the range 1−3 GHz, or frequencies above 40 GHz [22, Table II]) the Jacobian \([{\mathbf {J}}_{t}]_{ij}=abl_{ij} [{\hat {\mathbf {u}}}_{tt1}]_{j}^{b1}\) is undefined if \([{\hat {\mathbf {u}}}_{tt1}]_{j}=0\), if we have l _{ ij }≠0. This problem can be circumvented by replacing the 0 rainfall scenario by a very small value (close to zero) like in the order of 10^{−4} mm denoting a no rainfall event and the nonnegativity constraint can be replaced by u _{ t }≥10^{−4} 1 _{ N } in the optimization problems. However, in our simulations, we use b>1.
Simulation results
In this section, we present some simulation results to test the developed methodologies to dynamically monitor the rainfall in a given area. Here, we perform numerical experiments for three scenarios. In the first case, we assume that the dynamics/state model, i.e., H _{ t } is perfectly known through the parameters α, D, and w _{ t }. In the second case, we consider that the dynamics are not perfectly known and we assume that the state model is a Gaussian random walk. In the third case, we consider the scenario where we do not have any information regarding the state model/dynamics. The simulations for these three scenarios are presented below.
Ground truth
We consider two sets of ground truth/true values of the rainfall vector \(\{{\mathbf {u}}_{t}\}_{t=1}^{8}\). In the first case, we assume that the state model is perfectly known. In the next case, we assume that the state model is not perfectly known.
Ground truth with known dynamics
The ground truth is used from a practical rainfall event in an area of 25×25 km ^{2} in Amsterdam, the Netherlands^{1}. We take one spatial map of 15 min gauge adjusted radar rainfall depth (mm) of the same area of the day June 11, 2011, which is shown as the first state, i.e., u _{1} in Fig. 2.
We assume that the state transition matrix, i.e., H _{ t } and the process noise covariance matrix, i.e., Q _{ t }=Q is perfectly known in this case. The parameters of the state transition matrix H _{ t }=H are given as w _{ t }=w=[1,0]^{T}, α=0.33, and D=I _{2} (isotropic diffusion) for all t=2,…,T snapshots, where T=8. We assume the temporal sampling interval, i.e., δ _{ t }=15 min. The parameter w represents a constant advective displacement in 15 min.
The covariance matrix of the state noise, i.e., Q is assumed to have an exponential structure given as \([\!{\mathbf {Q}}]_{ij}={\sigma _{s}^{2}} \exp \left (\frac {\{\mathbf {x}}_{i}{\mathbf {x}}_{j}\_{2}}{\gamma }\right)\), with \({\sigma _{s}^{2}}=10^{3}\) and γ=3.33. The state noise vector q _{ t } at every snapshot is generated from the distribution \({\mathbf {q}}_{t}\sim {\mathcal {N}}({\mathbf {0}}_{N},{\mathbf {Q}})\). After generating the states of u _{ t } using the state model of (6), we set the negative elements of u _{ t } to 0. This is a modelling approximation adopted to avoid the generation of the negative rainfall values for very low rainfall intensities.
The total number of pixels is given as N=25×25=625, each of size 1 km ^{2}. Using this, we generate the states u _{2}…u _{8} using the state model mentioned in (6). Based on these parameters, the spacetime evolution of rainfall over t=1,…,8 snapshots (each of 15 min, i.e., in total 120 min) is shown in Fig. 2. The unit of the rainfall field is millimeter (mm).
Ground truth with approximate dynamics
In this section, we consider eight consecutive snapshots of 15min radar rainfall depths of the same day and area as mentioned in the previous section. The eight snapshots of the true gauge adjusted radar rainfall depths are shown in Fig. 3. Here, we assume that the state model is a Gaussian random walk, i.e., H _{ t }=I. The process noise covariance matrix Q is assumed to be the same as before.
Measurements
In this work, we simulate the measurements using the locations of the microwave links from a network of 151 microwave links. The total number of measurements for all t=1,…,8 is M=151. The locations of the microwave links in the service area along with the (1 km ^{2}) pixels, where we would like to estimate the rainfall intensities, are shown in Fig. 4.
We would like to mention that most of the microwave links in the Netherlands (specially in the urban areas) are operated at 38 GHz. In that case, b≈1, i.e., the measurement model becomes linear. To check the estimation performance of the developed algorithms in a nonlinear measurement framework, we intentionally choose b≠1. In this case, we assume the rain temperature to be 20°, which corresponds to [22, Table II]. We select the operating frequency to be 15 GHz, with a=3.28×10^{−2} and b=1.173. Using these, we simulate the measurements at eight snapshots, i.e., \(\{{\mathbf {y}}_{t}\}_{t=1}^{8}\), using the nonlinear measurement model of (2), where u _{ j,t }’s are the true values for j=1,…,625, and t=1,…8 as mentioned in the previous section.
Here, we generate two different sets of measurements. The first set of measurements is for the seven snapshots, i.e., \(\{{\mathbf {y}}_{t}\}_{t=2}^{8}\). These measurements are computed using the ground truth where the dynamics, i.e., H _{ t }=H for t=2,…,8 are perfectly known (Fig. 2). The second set of measurements are computed using the exact radar rainfall maps (Fig. 3) for eight snapshots, i.e., \(\{{\mathbf {y}}_{t}\}_{t=1}^{8}\) whose dynamics are unknown.
The parameters L _{ i } and l _{ ij } are known from the geometry of the links as shown in Fig. 4. It is assumed that the M measurements are collected in every 15min interval which are corrupted by additive white Gaussian noise characterized by \({\mathbf {e}}_{t}\sim {\mathcal {N}}({\mathbf {0}}_{M},{\sigma _{e}^{2}}{\mathbf {I}}_{M})\).
Dynamic rainfall monitoring
The noisy sets of measurements are used to estimate the rainfall depths at N=625 pixels over T=8 snapshots. In this section, we perform simulations for three different scenarios which are perfectly known dynamics, a Gaussian random walk dynamics, and no dynamics.
Perfectly known dynamics
The measurements \(\{{\mathbf {y}}_{t}\}_{t=2}^{8}\) are computed using the true values shown in Fig. 2, i.e., \(\{{\mathbf {u}}_{t}\}_{t=2}^{8}\). Here, we use \({\sigma _{e}^{2}}=10^{3}\). These measurements are used to estimate the states \(\{{\hat {\mathbf {u}}}_{t}\}_{t=2}^{8}\). The parameters of the spherical variogram model are computed for the particular day of the year, i.e., June 11, 2011 [29], to compute Σ _{ u }. The sill (S _{0}) and the range (d) parameters are computed using (11) of [29], whose parameters are taken from [29, Table 5]. The 15min time interval is rescaled in hourly scales, i.e., 0.25 h. We assume that the nugget is N _{0}=0. The value of the range (d) is 17.4675 km, and the sill (S _{0}) is 5.3328 mm^{2}.
Based on the predictions and the available link locations, it is seen that the DCT matrix exhibits minimal coherence with J _{ t }, in every iteration. We initialize \({\hat {\mathbf {u}}}_{11}={\tilde {\mu }}{\mathbf {1}}_{N}\), where \({\tilde {\mu }}\) is computed by empirically averaging the ground truth of the first state u _{1} over N pixels for both algorithms. In a real application, an appropriate initialization can be computed using the trend of the rainfall field, which is generally available from the climatological information of the area. For algorithm 2, we initialize M _{11}=I _{ N }. We use the software CVX [45] (parser CVX, solver SeDuMi [46]) to solve the convex optimization problems (i.e., (18) for algorithm 1 and (15) for algorithm 2).
In Fig. 5, we show the reconstructed spatial rainfall map for the states \({\hat {\mathbf {u}}}_{2}\) and \({\hat {\mathbf {u}}}_{8}\) using the algorithm 1. The same estimates are shown in Fig. 6 using the algorithm 2.
We plot the pixelwise comparisons of the estimates with the true values for all the seven snapshots, i.e., a total of 625×7=4375 pixels for the algorithms 1 and 2 in Fig. 9 a and Fig. 9 b respectively. The dark black lines in Fig. 9 a and Fig. 9 b, represent the y=x line. It is observed that algorithm 2, i.e., using the standard Kalman covariance update exhibits better estimation performance than algorithm 1, where the second order statistics are not updated.
Gaussian random walk
In this section, the state model is considered to be a Gaussian random walk, i.e., H _{ t }=I for t=1,…,8. The process noise statistics are considered to be the same as before. The measurements for eight snapshots, i.e., \(\{{\mathbf {y}}_{t}\}_{t=1}^{8}\) are generated using the true radar rainfall depths shown in Fig. 3, using the measurement model of (2) with the same a,b coefficients as the previous case. The measurements are different from the previous known dynamics case as the true values of \(\{{\mathbf {u}}_{t}\}_{t=1}^{8}\), i.e., the true radar rainfall depths (as shown in Fig. 3) are different. In this case, the measurement noise variance is reduced to \({\sigma _{e}^{2}}=10^{5}\). Due to better estimation performance (as seen in Section 5.3.1), we select algorithm 2 to estimate the states \(\{{\hat {\mathbf {u}}}_{t}\}_{t=1}^{8}\) using the measurements generated by the true radar rainfall depths.
Now, as the predictions using the sate model are not accurate in this case, we do not perform the tuning of λ _{ t } based on the predictions. However, the tuning of λ _{ t }, in this case can be performed using the standard methods mentioned in Section 4.2. In the current setup, to exploit the sparsity prior on every snapshot, we fix λ _{ t }=λ=2 for the sake of simplicity. The initializations are given by \({\hat {\mathbf {u}}}_{00}={\tilde {\mu }}{\mathbf {1}}_{N}\) and M _{00}=1 _{ N }.
In Fig. 7, we show the estimated spatial rainfall maps of the states \({\hat {\mathbf {u}}}_{1}\) and \({\hat {\mathbf {u}}}_{8}\) assuming that the state model is a Gaussian random walk.
In Fig. 10 a, we compare the estimation performance of the estimates of the 625 pixels over eight snapshots, i.e., a total of 625×8=5000 pixels with the true gauge adjusted radar rainfall depths.
No dynamics
In the last case, we assume that we do not use any prior information regarding the dynamics. On every snapshot, we estimate \({\hat {\mathbf {u}}}_{t}\) using the measurements and exploiting the prior information regarding the sparsity and the nonnegativity. In this case, for the sake of simplicity, we assume that the measurement model is linear, i.e., the case when the links are operated around 38 GHz. Here, we use a,b=1.
The linear measurement model is given by y _{ t }=Φ u _{ t }+e _{ t }, where \({y}_{i,t}=\sum _{j=1}^{N}{u}_{j,t} l_{ij} + e_{i,t}\), where i=1,…,M. On every snapshot, we solve the sparsityaware nonnegativity constrained optimization problem given as
However, this can be easily extended to a nonlinear measurement model by adopting an iterative linearization with respect to a suitable initial guess. Like in the previous case, here, we also fix λ=2 and the used basis is DCT matrix on every t. However, an upper bound on λ can be easily computed in this case by using the same methodology discussed in Section 4.2.
In Fig. 8, we show the estimated states \({\hat {\mathbf {u}}}_{1}\) and \({\hat {\mathbf {u}}}_{8}\) by solving the optimization problem of Eq. (26). The measurement noise variance is set as \({\sigma _{e}^{2}}=10^{5}\).
In Fig. 10 b, we compare the estimation performance of the estimates of total 625×8=5000 pixels with the true gauge adjusted radar rainfall depths.
Analysis of the results
The following inferences can be drawn from the aforementioned simulation studies.

The estimation performances in the first two cases are highly dependent on the accuracy of the state model, availability of the measurements in any region, and the initialization of the algorithm. In Fig. 6 b, the estimate of the state \({\hat {\mathbf {u}}}_{8}\) is much better than the same estimate in Fig. 7 b, as the dynamics are perfectly known in the first case. Also, the estimation performance is improved with time as the state error is minimized with temporal iterations (Fig. 6 b).

As seen in Fig. 4, there are many regions without any microwave links/measurements but where a rainfall field is present. In these regions, the estimates are mainly dependent on the predictions. On the other hand, if there is no rainfall over any link, the rainfall can be overestimated or underestimated in those regions, due to the effects of the measurement noise. This effect severely impairs the estimation performance in the case when we do not have an accurate prediction (or no prediction).

There is always a tradeoff between the estimation performance and the “availability” of the measurements and/or the “accuracy” of the predictions.

The reasons behind the scatter plots being not very symmetric are due to the biased estimates in the measurementvoid regions and the rainfallvoid links.

In the case of an imperfectly known or unknown state model, the prior knowledge of sparsity and nonnegativity help to achieve a stable solution (Figs. 7, 8, 10 a, b).
Performance metrics
To compare the estimation performances of the developed methods, we use some performance metrics, which are described in the following part of this section. The performance of a rainfall monitoring method can be quantified by root mean square error (rmse in mm), the mean bias (mb in mm), and the correlation coefficient (ρ) [7]. We quantify the overall estimation performances of all the above scenarios for N pixels over T snapshots using the aforementioned metrics. If the true value and the estimate of the rainfall field at any t are given by u _{ t } and \({\hat {\mathbf {u}}}_{t}\), respectively, then the performance metrics can be defined in the following ways
where \({\hat {\mu }}=\frac {1}{NT}\sum _{t=1}^{T}\sum _{j=1}^{N}[{\hat {\mathbf {u}}}_{t}]_{j}\) and \({\mu }=\frac {1}{NT}\sum _{t=1}^{T}\sum _{j=1}^{N}\) [u _{ t }]_{ j } are the sample means of the estimated and the true values of rainfall for N pixels over T snapshots.
The performance metrics are computed for the estimates using algorithm 2 for the scenarios of perfectly known dynamics and Gaussian random walk dynamics. For both of these scenarios, we also estimate the rainfall depths using a simple EKF without any sparsity and nonnegativity constraint. To avoid the negative estimates produced by the EKF, we set the negative estimates to 0. While computing the performance metrics, we fix the process and the measurement noise variances to \({\sigma _{s}^{2}}=10^{4}\) and \({\sigma _{e}^{2}}=10^{3}\), respectively, in all the cases.
When the dynamics are perfectly known, then the performance metrics are computed for the estimates of 625 pixels for seven snapshots and averaged over 20 different measurement noise realizations. In Table 1, we present the performance metrics computed for algorithm 2 and a simple EKF (with the thresholding) for the perfectly known dynamics case.
In Table 2, we present the aforementioned performance metrics computed for the estimates of 625 pixels for eight snapshots using algorithm 2 (with fixed λ _{ t }=λ=2) and an EKF (with the thresholding), where the state model is assumed to be a Gaussian random walk. Here, we also average the performance metrics for 20 different measurement noise realizations. In all of these realizations, the measurements are generated using the true radar rainfall depths as shown in Fig. 3.
Analysis of the performance metrics

For a perfectly known state model, extra information like sparsity and nonnegativity does not play any significant role in terms of estimation accuracy. This is clear from Table 1 where it is shown that the performance improvement over a simple EKF (with setting the negative estimates to 0) is negligible.

When the information regarding the state model is unknown and approximated as a Gaussian random walk model, then the performance of a simple EKF is very poor. In this case, the sparsity and nonnegativity information along with the measurements improve the estimation performance (Table 2).

The last mentioned observation is quite useful in practical cases, where the availability of an accurate state model is scarce.
The computation times for both algorithm 1 and algorithm 2 including the basis selection part is less than a minute for N=625 pixels using the aforementioned off the shelf solvers. The computation time is increased if N is higher than 625. Algorithm 1 is computationally simpler than algorithm 2 because there is no covariance update state. But the price we pay is in terms of the estimation performance. However, the speed of the developed algorithms can be increased by using a projected subgradient method [47].
Conclusions
We have developed a generalized dynamic rainfall monitoring algorithm from limited nonlinear attenuation measurements by utilizing the spatial sparsity and nonnegativity of the rainfall field. We have formulated the dynamic rainfall monitoring algorithm as a constrained convex optimization problem. The performance of the developed algorithm is compared with the standard approaches like an EKF for the scenarios, where we have both perfect knowledge about the state model and an approximate state model. Numerical experiments show that the developed approach outperforms a simple EKF in scenarios, where the state model is not perfectly known. The proposed methodology can be equivalently implemented for dynamic field tracking in tomographic applications like MRI and microwave tomography, where we have pathintegrated Gaussian measurements.
However, tackling more complicated dynamics of the rain (possibly nonlinear and highly timevarying), and nonGaussian measurement noise could be possible future extensions of this work. In that case, both the state and the measurement models are nonlinear. This triggers one to use an unscented Kalman filter (UKF), particle filtering based algorithms, or other heuristic approaches. Estimation of the underlying dynamics of rainfall from the available ground truth and using it for realtime dynamic monitoring is also a part of the future research. A realtime selection of the most informative attenuation measurements from the available links could be interesting in order to reduce the processing time and computational complexity.
Endnote
^{1} Data courtesy: Royal Netherlands Meteorological Institute (KNMI), The Netherlands.
References
 1
O Sendik, H Messer, A new approach to precipitation monitoring: a critical survey of existing technologies and challenges. IEEE Signal Proc. Mag. 32(3), 110–122 (2015).
 2
H Messer, A Zinevich, P Alpert, Environmental monitoring by wireless communication networks. Science. 312(5774), 713–713 (2006).
 3
H Messer, A Zinevich, P Alpert, Environmental sensor networks using existing wireless communication systems for rainfall and wind velocity measurements. IEEE Instrum. Meas. Mag. 15(2), 32–38 (2012).
 4
D Giuli, A Toccafondi, GB Gentili, A Freni, Tomographic reconstruction of rainfall fields through microwave attenuation measurements. J. Appl. Meteorol. 30(9), 1323–1340 (1991).
 5
D Giuli, L Facheris, S Tanelli, Microwave tomographic inversion technique based on stochastic approach for rainfall fields monitoring. IEEE Trans. Geosci. Remote Sens. 37(5), 2536–2555 (1999).
 6
O Goldshtein, H Messer, A Zinevich, Rain rate estimation using measurements from commercial telecommunications links. IEEE Trans. Signal Process. 57(4), 1616–1625 (2009).
 7
A Zinevich, P Alpert, H Messer, Estimation of rainfall fields using commercial microwave communication networks of variable density. Adv. Water Resour. 31(11), 1470–1480 (2008).
 8
A Overeem, H Leijnse, R Uijlenhoet, Countrywide rainfall maps from cellular communication networks. Proc. Natl. Acad. Sci. 110(8), 2741–2745 (2013).
 9
Y Liberman, H Messer, in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Accurate reconstruction of rain field maps from commercial microwave networks using sparse field modeling (Florence, Italy, 2014), pp. 6786–6789.
 10
V Roy, S Gishkori, G Leus, in Proc. IEEE Global Conference on Signal and Information Processing (GlobalSIP). Spatial rainfall mapping from pathaveraged rainfall measurements exploiting sparsity (Atlanta, USA, 2014), pp. 321–325.
 11
A Zinevich, H Messer, P Alpert, Frontal rainfall observation by a commercial microwave communication network. J. Appl. Meteorol. Climatol. 48(7), 1317–1334 (2009).
 12
Y Liberman, in Proc. 22nd European Signal Processing Conference (EUSIPCO). Object tracking extensions for accurate recovery of rainfall maps using microwave sensor network (EURASIPLisbon, Portugal, 2014), pp. 1322–1326.
 13
EJ Candès, MB Wakin, An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008).
 14
A Charles, MS Asif, J Romberg, C Rozell, in Proc. 45th Annual Conference on Information Sciences and Systems (CISS). Sparsity penalties in dynamical system estimation (Baltimore, USA, 2011), pp. 1–6.
 15
S Farahmand, GB Giannakis, G Leus, Z Tian, Tracking target signal strengths on a grid using sparsity. EURASIP J. Adv. Signal Process. 2014(1), 1–17 (2014).
 16
AS Charles, CJ Rozell, Dynamic filtering of sparse signals using reweighted ℓ_{1}, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (Vancouver, BC, Canada, 2013).
 17
AS Charles, CJ Rozell, in Proc. IEEE Global Conference on Signal and Information Processing (GlobalSIP). Convergence of basis pursuit denoising with dynamic filtering (Atlanta, USA, 2014), pp. 374–378.
 18
A Carmi, P Gurfil, D Kanevsky, Methods for sparse signal recovery using kalman filtering with embedded pseudomeasurement norms and quasinorms. IEEE Trans. Signal Process. 58(4), 2405–2409 (2010).
 19
W Dai, D Sejdinovic, O Milenkovic, in Proc. International Conference on Sampling Theory and Application(SampTA). Gaussian dynamic compressive sensing (Singapore, 2011).
 20
F Sigrist, HR Künsch, WA Stahel, A dynamic nonstationary spatiotemporal model for short term prediction of precipitation. Ann. Appl. Stat. 6(4), 1452–1477 (2012).
 21
K Xu, CK Wikle, NI Fox, A kernelbased spatiotemporal dynamical model for nowcasting weather radar reflectivities. J. Am. Stat. Assoc. 100(472), 1133–1144 (2005).
 22
R Olsen, D Rogers, DB Hodge, The a r ^{b} relation in the calculation of rain attenuation. IEEE Trans. Antennas Propag. 26(2), 318–329 (1978).
 23
D Atlas, C Ulbrich, Pathand areaintegrated rainfall measurement by microwave attenuation in the 1–3 cm band. J. Appl. Meteorol. 16(12), 1322–1331 (1977).
 24
A Jameson, A comparison of microwave techniques for measuring rainfall. J. Appl. Meteorol. 30(1), 32–54 (1991).
 25
H Leijnse, R Uijlenhoet, J Stricker, Microwave link rainfall estimation: effects of link length and frequency, temporal sampling, power resolution, and wet antenna attenuation. Adv. Water Resour. 31(11), 1481–1493 (2008).
 26
A Berne, R Uijlenhoet, Pathaveraged rainfall estimation using microwave links: Uncertainty due to spatial rainfall variability. Geophys. Res. Lett. 34(7) (2007).
 27
N Cressie, Statistics for spatial data: Wiley Series in Probability and Statistics (WileyInterscience, New York, 1993).
 28
N Cressie, CK Wikle, Statistics for spatiotemporal data (John Wiley & Sons, 2011).
 29
C van de Beek, H Leijnse, P Torfs, R Uijlenhoet, Seasonal semivariance of dutch rainfall at hourly to daily scales. Adv. Water Resour. 45:, 76–85 (2012).
 30
PE Brown, GO Roberts, KF Kåresen, S Tonellato, Blurgenerated nonseparable spacetime models. J. R. Stat. Soc. Ser. B Stat Methodol. 62(4), 847–860 (2000).
 31
J Stroud, P Muller, B Sansó, Dynamic models for spatiotemporal data. J. R. Stat. Soc. Ser. B Stat Methodol. 63(4), 673–689 (2001).
 32
SM Kay, Fundamentals of statistical signal processing: estimation theory (PTR PrenticeHall, Upper Saddle River, NJ, 1993).
 33
E Morin, DC Goodrich, RA Maddox, X Gao, HV Gupta, S Sorooshian, Spatial patterns in thunderstorm rainfall events and their coupling with watershed hydrological response. Adv. Water Resour. 29(6), 843–860 (2006).
 34
M Tanaka, T Katayama, Robust fixedlag smoother for linear systems including outliers in the system and observation noises. Int. J. Syst. Sci. 19(11), 2243–2259 (1988).
 35
DL Donoho, X Huo, Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory. 47(7), 2845–2862 (2001).
 36
DL Donoho, M Elad, VN Temlyakov, Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans. Inf. Theory. 52(1), 6–18 (2006).
 37
S Yan, C Wu, W Dai, M Ghanem, Y Guo, in Proceedings of the sixth international workshop on knowledge discovery from sensor data. Environmental monitoring via compressive sensing (ACMBeijing, China, 2012), pp. 61–68.
 38
R Rubinstein, AM Bruckstein, M Elad, Dictionaries for sparse representation modeling. Proc. IEEE. 98(6), 1045–1057 (2010).
 39
M Elad, Optimized projections for compressed sensing. IEEE Trans. Signal Process. 55(12), 5695–5702 (2007).
 40
SJ Kim, K Koh, M Lustig, S Boyd, D Gorinevsky, An interiorpoint method for largescale ℓ _{1}regularized least squares. IEEE J. Sel. Top. Sign. Process. 1(4), 606–617 (2007).
 41
R Tibshirani, Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Methodol. 58(1), 267–288 (1996).
 42
CL Mallows, Some comments on Cp. Technometrics. 15(4), 661–675 (1973).
 43
H Akaike, Information theory and an extension of the maximum likelihood principle (Springer, 1998).
 44
G Schwarz, Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978).
 45
M Grant, S Boyd, Y Ye, CVX: Matlab software for disciplined convex programming (2008).
 46
JF Sturm, Using sedumi 1.02, a Matlab toolbox for optimization over symmetric cones. Optim. Methods Softw. 11(1–4), 625–653 (1999).
 47
D Bertsekas, JN Tsitsiklis, Parallel and distributed computation: numerical methods, vol. 23 (Englewood Cliffs, NJ: Prentice hall, 1989).
Acknowledgements
This work has been supported by the SHINE project, the flagship project of DIRECT (Delft Institute for Research on ICT) at the Delft University of Technology.
Author information
Affiliations
Corresponding author
Additional information
Competing interests
The authors declare that they have no competing interests.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Roy, V., Gishkori, S. & Leus, G. Dynamic rainfall monitoring using microwave links. EURASIP J. Adv. Signal Process. 2016, 77 (2016). https://doi.org/10.1186/s1363401603676
Received:
Accepted:
Published:
Keywords
 Rainfall monitoring
 Sparsity
 Field estimation