Sequential Monte Carlo for inference of latent ARMA time-series with innovations correlated in time

We are interested in the study of latent time-series with diverse memory properties observed via nonlinear functions. To systematically accommodate observed and hidden variables, we adopt the state-space methodology. For the state, we use a flexible formulation that allows for short- and long-memory processes. As for the space equation, we make minimal assumptions.

Specifically, the state dynamics follow an ARMA(p,q) model with non-i.i.d. innovations. That is, the driving noise for the ARMA process is correlated in time. The only restriction is that the innovation process must be stationary. For the observation equation, we consider nonlinear functions of the state.

where \(x_{t}\in \mathbb {R}\) (x _t =0, ∀t<0) is the state of the latent process at t; a ₁,a ₂,⋯,a _p are the autoregressive (AR) parameters; and b ₁,b ₂,⋯,b _q are the moving average (MA) parameters of the ARMA model.

The symbol u _t represents a zero-mean Gaussian innovation process that is correlated in time. That is, it is fully characterized by its first (i.e., mean) and second moments (i.e., an arbitrary autocovariance function γ _u (τ)). Note that stationarity is enforced (the mean is constant and the autocovariance depends only on the time lag τ), but no restriction on the form of the autocovariance function is imposed.

We denote with \(y_{t}\in \mathbb {R}\) the observation at time t. We consider that the observation can be expressed by any generic function h(x _t ,v _t ) and that the observation noise v _t can have any distribution for as long as the likelihood function f(y _t |x _t ) is computable up to a proportionality constant.

Our goal is to sequentially estimate the posterior distribution of x _t , given the observations y _1:t ≡{y ₁,y ₂,⋯,y _t }, that is, f(x _t |y _1:t ). We want to update f(x _t |y _1:t ) to f(x _t+1|y _1:t+1), for each new acquired observation y _t+1.

The analytical solution to this integral is only possible for the case of Gaussian densities and linear functions, i.e., the celebrated Kalman Filter [16]. Since we do not restrict ourselves to such assumptions in this paper, we resort to SMC methods, widely popular since the seminal publication of [23]. These methods provide suboptimal solutions, as they compute probability random measure approximations of the densities of interest, while guaranteeing convergence under suitable conditions.

We detail in Section 3 the Bayesian analysis of the studied time-series (i.e., ARMA(p,q) models with correlated innovations), before delving into the intricacies of sequential Monte Carlo methods in Section 4.

We provide a Bayesian analysis of an ARMA(p,q) model driven by an innovation process that is not i.i.d. We emphasize the importance of considering the full ARMA(p,q) with correlated innovations model, as it allows for an accurate description of time-series with a wide range of memory properties within the same mathematical formulation.

Typical ARMA(p,q) models (i.e., with i.i.d. innovations) possess exponentially decaying memory properties. Regardless of the parameterization of the autoregressive or the moving average components, the time-series forgets its past at a rate proportional to c ^−τ for a constant c∈(0,1). On the contrary, by considering correlated innovations u _t , we allow for the time-series to exhibit a wide range of decaying memory properties. That is, the modeled time-series can forget their past values sub-exponentially.

For example, if the innovations are fractional Gaussian processes, then the memory dependency is proportional to τ ^−c for a constant c∈(0,1) (e.g., see Fig. 1). Note that such diverse memory characteristics cannot be modeled with ARMA models and uncorrelated innovations. Furthermore, any of the particular subcases (AR(p), MA(q), or ARMA(p,q), with or without correlated noise) are covered within the proposed generic formulation.

All in all, we study a generic ARMA(p,q) model with non-i.i.d. innovations. We are interested in the statistical properties of this stochastic process and, thus, in the joint distribution of the time-series up to time instant t, i.e., f(x _1:t ). This distribution contains all the relevant information of the time-series, from which any marginal and conditional of interest can be derived.

where the time subscript t also indicates matrix dimensionality.

Note the symmetric Toeplitz structure of the matrix, where the only required elements are the normalized autocovariance values ρ _u (τ), for lags τ=0,1,⋯,t−1.

Based on these sufficient statistics of the correlated innovation process and the formulation of the time-series as in Eq. (7), we derive the densities of interest for our model in Eq. (1).

Our objective is to sequentially infer the evolution of a latent time-series with correlated innovations, as we acquire new observations. In Bayesian terminology, one is interested in updating the filtering density from f(x _t |y _1:t ) to f(x _t+1|y _1:t+1) as new data are observed. We do so by using Eq. (2).

As previously pointed out, the analytical solution to such equation is intractable for the most interesting cases (models with nonlinearities and non-Gaussianities) and thus, we resort to sequential Monte Carlo methods. We briefly provide an overview of SMC methods in subsection 4.1, before explaining in detail our proposed method in subsection 4.2.

We now illustrate the applicability of the proposed SMC methods and evaluate their performance. We do so by considering the stochastic log-volatility (SV) state-space framework. That is, the observations are a zero-mean process with time-varying log-variance that one wants to estimate.

The SV model is popular in the study of nonlinear state-space models (due to the estimation challenges that it presents [15, 48–50]) and is of interest in finance (due to its applicability in the study of stock returns [17, 51–53]).

It has been established that Kalman filter (KF)-based methods fail to accurately estimate the latent state for SV models. In principle, for the nonlinearities in the SV model observation equation, extensions to the popular KF, such as the extended Kalman filter (EKF) [19], the unscented Kalman filter (UKF) [54], and other Sigma-Point Kalman filters [55] should be applicable. However, as reported in [56], these methods fail when addressing the SV model since they are unable to update their prior beliefs for such model (the Kalman gain is always null). Alternatives based on transformations of the model have been suggested ([15, 50]) but, as reported in [33], they fall short when compared to SMC methods.

Furthermore, the SV model has been in use in econometrics for a long time [53], as it is of interest in estimating the risk involved in financial transactions. There, the observations describe the price evolution of an asset, for which estimating its volatility is critical. This is not an easy task, and many efforts have been reported, where the risk is described with diverse memory properties [7, 14, 57].

Motivated by its practical application and the challenges it poses to the inference problem, we focus on the SV model, where the log-volatility is described by a latent ARMA(p,q) model with correlated innovations.

Without loss of generality, we focus on ARMA models with fractional Gaussian noise. With this modeling, we accommodate a wide range of memory properties: from uncorrelated to long-memory processes. This is a natural extension of the classical ARMA model, where instead of i.i.d. Gaussian innovations, the ARMA(p,q) filters a fractional Gaussian process with Hurst parameter H. The properties of such model are equivalent to those of the FARIMA(p,d,q), when \(d=H-\frac {1}{2}\).

which is parameterized by the Hurst parameter H and variance \(\sigma _{u}^{2}\). When H=0.5, the process is uncorrelated, while the memory of the innovations increases as H→1. We illustrate in Fig. 1 the dependency of the normalized autocovariance function \(\left (\text {i.e.,}\ \sigma _{u}^{2}=1\right)\) with respect to the parameter H.

We evaluated the proposed method in this nonlinear model, first under the known ARMA parameter case, and then, under unknown parameters. We show in Fig. 2 how the proposed method is able to accurately track different latent processes with diverse memory properties, even when the innovation variance \(\sigma _{u}^{2}\) is unknown. The SMC methods were run with M=1000 particles for different values of the Hurst parameter.

Note that the unknown \(\sigma _{u}^{2}\) case induces a slight loss in accuracy. However, the estimation performance is comparable. The justification relies on the form of the derived marginalized density. As more data are observed, the transition density for the unknown variance case (i.e., Eq. (17)) becomes very similar to the one in the known case (i.e., Eq. (16)). This occurs because a Student tdistribution with high degrees of freedom is very similar to a Gaussian distribution. Thus, the proposal densities in both SMC methods become almost identical with time.

To get further insight on the impact of not knowing the innovation variance \(\sigma _{u}^{2}\), we study the evolution of the scale factor in Eq. (17) over time. We plot (see Fig. 3) the scale factor

$$ \frac{\nu_{0}\sigma_{0}^{2}+x_{1:t}^{\top} \Sigma_{t}^{-1}x_{1:t}}{\nu_{0}+t} $$

(56)

as we get more data and observe that the estimate approaches the true \(\sigma _{u}^{2}\) value. The estimation accuracy improves with time for all the evaluated ARMA parameterizations and memory properties of the innovation process.

We now turn our attention to the more challenging scenario where the ARMA parameters are unknown. We evaluate both proposed approaches, i.e., the DA- and IS-based SMC methods from subsections 4.2.2 and 4.2.3, respectively.

The state filtering results allow us to conclude that both proposed approaches are suitable solutions for inference of latent processes with unknown ARMA parameters. Besides, it is noticeable that the impact on state estimation accuracy of not knowing the parameters a and b is more pronounced than not knowing the innovation variance \(\sigma _{u}^{2}\).

We also observe a slightly better filtering performance of the DA-SMC when compared to the IS-SMC. However, this improved state tracking accuracy comes with a cost, as the estimation of the unknown parameters is worse for the DA-based SMC. This effect is illustrated with some parameter estimation realizations of an ARMA(1,1) process with unknown parameters a ₁ and b ₁, with known innovation variance \(\sigma _{u}^{2}\) in Fig. 4, and unknown variance \(\sigma _{u}^{2}\) in Fig. 5.

We note that, for both proposed SMC methods, estimation of the AR parameters is more accurate than of the MA parameters. Recall that, due to how these parameters are used in our computations (inversion and multiplication of matrices involved when evaluating the sufficient statistics), identifiability and numerical issues may arise. In the proposed method, particles that are result of numerical instabilities are automatically discarded by their corresponding weights, as they become negligible. In such cases, we observe a reduced effective particle size, but the method is still able to track the state.

Furthermore, the DA-based SMC overestimates the unknown variance (see Fig. 5). Although this might seem irrelevant for the filtering problem, variance overestimation is critical when predicting future instances of the time-series, as the density becomes too wide to be informative.

When applying IS, one explicitly computes weights based on both the state and parameter samples, while with the DA approach, one hopes that the best state particles are associated with good parameters too (although this is not explicitly accounted for). As a result of the parameter-explicit weight computation in IS-based methods, the number of particles with non-negligible weights is much reduced at every time instant (as one looks for both good state and parameter samples). Consequently, the effective particle size of the IS-based SMC method is quite low, and thus, the obtained results much more volatile. Averaged effective particle sizes for the proposed SMC methods are provided in Table 3.