This section presents a method for error degree estimation in numerical weather prediction using MKDA-based ordinal regression. In this paper, we define prediction error degrees as several ordered discrete ranks. Specifically, we divide the axis of the prediction error into several intervals and assign a symbol, which corresponds to the prediction error degree, for each interval as shown in the example in Figure 1. As shown in Figure 2, the proposed method tries to estimate the unknown prediction error degree that will occur in the forecast in each area from known observed meteorological data. Specifically, from observed data obtained several hours ago, the prediction error degree is estimated for each area based on MKDA-based ordinal regression. In order to perform training of MKDA, we have to provide pairs of features extracted from the observed meteorological data and known prediction error degrees. Therefore, those training pairs are prepared from past observation times, e.g., a few days before the target day for which prediction error degrees are estimated.
Section 2.1 presents details of feature extraction from meteorological data, and Section 2.2 presents an algorithm for estimating the prediction error degree based on ordinal regression using MKDA.
2.1 Feature extraction from meteorological data
This subsection presents details of feature extraction from meteorological data. Using the proposed method, we try to estimate the prediction error degree in each area by using features calculated from ‘previously observed errors of a target meteorological element and some other related elements’ and ‘their time variations’ in the same area and its neighboring areas. Only features of neighboring areas in which the atmospheres move to the target area affect the prediction error degree estimation of the target area.
Suppose that prediction error degree estimation of a target meteorological element F0 is performed in area p at time t. Furthermore, it is assumed that the prediction errors of several related meteorological elements including F0, F
l
(l=0,1,⋯,L; L+1 being the number of meteorological elements used for calculating features) at time t−s Δ t (s=0,1,⋯,S) are known, where the index s is used for referring to the current or past time steps. The proposed method calculates time average and maximum and minimum values of the prediction errors e
l
(p,t), , , , respectively, and the average of their time variations between time t−(s+1)Δ t and t−s Δ t for each meteorological element F
l
in each area p as feature values. In this section, e
l
(p,t) represents the prediction error of meteorological element F
l
in area p at time t. In detail, , , , and can be obtained as follows:
(1)
(2)
(3)
(4)
where
(5)
For calculating the four features , , , and of the target area p for meteorological element F
l
at time t, the errors e
l
(p,t),e
l
(p,t−Δ t),e
l
(p,t−2Δ t),⋯,e
l
(p,t−S Δ t) are used. This means that errors in the same area p at current and past times are used for the calculation. Note that in Equations 1 to 5, the index l is used for referring to the l th meteorological element F
l
.
Furthermore, in order to use the prediction errors that are propagated from neighboring areas to the target area based on atmospheric movements, motion vectors representing atmospheric movements between time t−Δ t and time t are obtained by using observed wind velocities for each area. Then, the feature of the prediction error for meteorological element F
l
propagated from the neighboring areas to the target area p is obtained by the following equation:
(6)
where average[·] is an operator calculating the average values. Furthermore, R(p,t) represents a set of areas in which atmospheres move to the target area p from time t−Δ t to time t. Specifically, by denoting the atmospheric movement of area p∗ from time t−Δ t to time t as v(p∗,t−Δ t), R(p,t) can be represented as follows:
(7)
By using the atmospheric movements, we can select areas in which atmospheres move to the target area p from time t−Δ t to time t. As shown in Equation 7, we define the neighbor of p. In this equation, the neighbor is a set of areas p∗ satisfying p∗+v(p∗,t−Δ t)≈p. This means ∥p∗+v(p∗,t−Δ t)−p∥
c
<ε is satisfied for a small positive constant ε. Note that ∥·∥
c
represents the chessboard distance. In this study, we set ε to 5 km. Since the distance between the most neighboring areas is 5 km in the dataset used in the experiments, we set ε to 5 km. Therefore, if the distance between the most neighboring areas changes, we should also change ε. Furthermore, it is well known that if the distance between the most neighboring areas becomes smaller, the performance of the numerical weather prediction also becomes better. Similarly, it can be expected that the performance of the prediction error degree estimation becomes better if the distance between the most neighboring areas becomes smaller. As shown in the example in Figure 3, the atmospheres of six areas, i.e., yellow areas, move to the target area p. Therefore, these six areas are regarded as neighbors. Then, the prediction errors e
l
(p∗,t−Δ t) of these six areas p∗ in the previous time step t−Δ t are averaged, and the feature is obtained. If none of the areas are moving into the target area p, is set to zero in our method. By using the features in Equation 6, the influence of prediction errors propagated to the target area p can be considered.
From the features , and (l=0,1,⋯,L), we can define a feature vector for each area p at time t. Note that these five features in area p at time t for meteorological element F
l
can be calculated from several isometric surfaces in the proposed method. Therefore, given the number of isometric surfaces as J, 5J features are obtained in area p at time t for each meteorological element F
l
. Then, for each area p at time t, a total of d(=5J×(L+1)) features is obtained, i.e., a d-dimensional feature vector is finally obtained.
2.2 Algorithm for estimation of prediction error degree
This subsection presents an algorithm for estimating prediction error degrees using MKDA-based ordinal regression. The proposed method estimates class labels, i.e., prediction error degrees at each area p, from the features obtained as described in the previous subsection.
First, we denote a set of training data (x
i
,y
i
)∈Rd×R (i=1,2,⋯,M; M being the number of training samples) as . Each x
i
∈Rd is a d-dimensional (d being the number of features shown in the previous subsection) input feature vector, and y
i
∈{1,2,⋯,K} is the corresponding ordered class label, where K is the number of classes. This label can be obtained by quantizing the known prediction error of target meteorological element F0 into K ranks as shown in the example in Figure 1. In Figure 1, K is set to seven. Furthermore, the proposed method maps x
i
into the feature space via a nonlinear map [26], and is obtained. In the proposed method, ϕ(·) is a nonlinear map which projects an input vector to high-dimensional feature space. Furthermore,
represents this high-dimensional feature space. Note that its dimension depends on the definition of the corresponding kernel function of the nonlinear map ϕ(·). Since ϕ(x
i
) is high-dimensional or infinite-dimensional, it may not be possible to calculate them directly. Fortunately, it is well known that the following computational procedures depend only on the inner products in the feature space, which can be obtained from a suitable kernel function [26]. Given two vectors x
m
and x
n
∈Rd, our method uses the following multiple kernel functions:
(8)
where γ
l
represents the weight of the l th kernel κ
l
(·,·) and satisfies γ
l
≥0 and . The superscript ′ denotes the vector/matrix transpose in this paper. Furthermore, E
l
in Equation 8 is a diagonal matrix whose diagonal elements are zero or one, and it enables the extraction of features of meteorological element F
l
that are used for the l th kernel κ
l
(·,·). Specifically, the dimension of vectors x
m
and x
n
is d(=5J×(L+1)) as defined in the previous subsection, and each extraction matrix E
l
extracts 5J features of each meteorological element F
l
. In the proposed method, the multiple kernel scheme is applied to different meteorological elements. Since we use L+1 kinds of meteorological elements, L+1 kernels are linearly combined in Equation 8. As shown in the previous subsection, various kinds of meteorological elements can be used for estimation of the prediction error degree. Thus, the proposed method extends the kernel function to a multiple kernel version as shown in Equation 8. Then, by successfully determining the parameters γ
l
(l=0,1,⋯,L) in the multiple kernel function, the features can be mapped into the optimal feature space, enabling accurate ordinal regression. It is important to successfully determine the parameters γ
l
, and details of their determination are shown in Section 2.2.2. Note that κ
l
(·,·) of each meteorological element F
l
is set to the well-known Gaussian kernel in our method.
2.2.1 Sampling of training data
Note that when the kernel method is adopted, direct use of MKDA, for which computation depends only on the inner products in the feature space, becomes difficult due to the large amount of training data.
Therefore, the proposed method uses a sampling scheme.
Specifically, we regard the error data e
l
(p,t) at time t as two-dimensional signals and perform their downsampling.
Then, the new sampled training data (x
i
,y
i
) (i=1, 2,⋯,N; N<M) can be obtained, where (x
i
,y
i
) is redefined, and N is the number of new training samples. In the following explanations of training of MKDA, we use these training data (x
i
,y
i
) (i=1,2,⋯,N). Also, we denote a set of these sampled training data as .
By reducing the number of training samples, the performance of the KDA-based ordinal regression tends to become worse. Note that in the proposed method, we regard the error data e
l
(p,t) as two-dimensional signals and perform their downsampling. Generally, neighboring areas in meteorological data tend to have similar features, and it seems that the distribution of training data is not drastically changed by the sampling. Thus, performance degradation tends to be avoided. Furthermore, in the proposed method, the remaining training data in , which are removed by the sampling, can be used for estimating γ
l
(l=0,1,⋯,L) in Equation 8. Fortunately, by using these remaining data, we can improve the performance of the error degree estimation based on the multiple kernel scheme. The details are shown in 2.2.3.
2.2.2 Derivation of MKDA
The objective of the discriminant analysis is to find a projection w from which different classes can be well separated. Specifically, we first define within-class and between-class scatter matrices as follows:
(9)
(10)
where
(11)
In the above equations, corresponds to x
i
(i=1,2,⋯,N) belonging to the k th class (i.e., y
i
=k), and mk denotes the mean vector of belonging to the k th class as shown in Equation 11. Furthermore, , and is a centering matrix satisfying Hk′=Hk and (Hk)2=Hk, where Ik is the Nk×Nk identity matrix, and 1k=[1,1,⋯,1]′ is an Nk×1 vector. The vector in Equation 10 stands for the mean vector of all samples x
i
(i=1,2,⋯,N). Generally, the objective of the discriminant analysis can be achieved by the following equation:
(12)
In the proposed method, we perform ordinal regression with K ordered classes. Therefore, the goal of our method is to find the optimal projection w satisfying the following two points:
-
1.
The projection w should minimize the within-class distance and maximize the between-class distance simultaneously.
-
2.
The projection w should ensure ordinal information of different classes, i.e., the average projection of samples from higher rank classes should be larger than that of lower rank classes.
Therefore, the formulation for the MKDA-based ordinal regression is derived from [13] as follows:
(13)
where C>0 represents a penalty coefficient. The above equation minimizes the within-class distances. Furthermore, instead of using the between-class scatter matrix directly, the above equation tries to maximize the distance between the two projected means from the closest pair of classes. Specifically, Equation 12 tries to minimize the within-class distance and maximize the between-class distance simultaneously. On the other hand, Equation 13 reformulates the problem of Equation 12. The within-class distance is minimized from w′S
w
w in J(w,ρ). Furthermore, instead of directly maximizing the between-class distance, a new constraint w′(mk+1−mk)≥ρ (k=1,2,⋯,K−1) is introduced. In this way, our MKDA-based ordinal regression tries to estimate the projection w minimizing within-class distance with the constraint of ordinal information. Thus, the distribution direction can be considered by using the within-class scatter matrix S
w
in Equation 13.
In MKDA, the projection w is high-dimensional or infinite-dimensional and cannot be calculated directly. Thus, the projection w is written as follows:
(14)
where β
i
(i=1,2,⋯,N) is a linear coefficient, and β=[β1,β2,⋯,β
N
]′. In addition, Ξ = [ϕ(x1),ϕ(x2),⋯, ϕ(x
N
)]. In the proposed method, we derived Equation 14 by using the method in [13]. In several kernel methods such as KDA and KPCA, the projection is represented by a linear combination of the samples. Therefore, we adopt the above derivation. However, derivation of the representer theorem in a multiple kernel case is not so straightforward. In the proposed method, we approximately use the derivation of Equation 14. Since the theoretical analysis of this approximation cannot be shown in this paper, this will be a future work of our study.
By using Equations 9, 11, and 14 and J(w,ρ) and w′(mk+1−mk) in Equation 13 are respectively rewritten as follows:
(15)
(16)
where
(18)
(19)
The problem of w in Equation 13 can be rewritten as that of β as follows:
(20)
In order to solve Equation 20, we define the following Lagrangian equation:
(21)
where α=[α1,α2,⋯,αK−1]′ represents a vector containing Lagrange multipliers. Then, by calculating and , the following equations are respectively obtained:
(22)
(23)
From the above equations, the optimization problem in Equation 20 is turned into
(24)
The proposed method estimates the optimal result of α by using the penalty method [27].
2.2.3 MKDA-based ordinal regression and determination of kernels’ contributions
As shown in the above explanation, we can obtain the optimal projection of w from β obtained by α.
From the obtained optimal projection w, the rank of an unseen input feature vector x, i.e., the prediction error degree at each area p, can be estimated by the following decision rule:
(25)
where b
k
is defined as
(26)
Then, from Equations 14 and 26, Equation 25 is rewritten as follows:
(27)
The above equation enables ordinal regression for estimating prediction error degrees.
Note that since our method adopts a multiple kernel algorithm, we also have to determine γ=[γ0,γ1,⋯,γ
L
]′ in Equation 8. Some methods such as simple MKL [21] have been proposed for determining γ. However, if the simple MKL is used for estimating the parameters of the linear combination of kernels, successful performance of the error degree estimation is not possible. We guess when using the simple MKL, the result of γ that provides w optimal for Equation 13 is obtained from the sampled training data, and the generalization characteristic becomes worse, and then a phenomenon similar to overfitting occurs. This means that the fitting of γ tends to strongly depend on the sampled training data. Furthermore, it has been reported that the MKL approach does not always outperform a single kernel-based approach [19]. Thus, it is important to determine γ in such a way that the performance can be guaranteed for other new samples to keep the robustness.
Fortunately, we have the other remaining training data (x
i
,y
i
) included in , which are removed by the sampling scheme. Therefore, we use them and verify the estimation performance of the error degrees, and the best result of γ is determined by an exhaustive search. Generally, the statistical characteristics of the test data tend to be slightly different from that of the training data. Therefore, in order to make the proposed method robust to this difference, we use the other remaining training data included in . Specifically, the proposed method changes the values of γ
l
(l=0,1,2,⋯,L) as 0.1,0.2,⋯,1.0 with the constraints of γ
l
≥0 (l=0,1,2,⋯,L) and Then, the problem in Equation 28 is optimized with α. Furthermore, for the data (x
i
,y
i
) included in , which are the remaining training data after the sampling, we perform the estimation of prediction error degrees and calculate their mean absolute errors.
Specifically, the errors between the estimated error degree f(x
i
) in Equation 31 and the true rank y
i
are calculated. Finally, we output the optimal result of γ that provides the most accurate estimation results, i.e., the minimum mean absolute error. By using this exhaustive search procedure, the proposed method enables determination of the multiple kernel function. Note that in the same way as γ, the proposed method estimates the parameter of the Gaussian kernel for each meteorological element, where the searching interval is set to 0.5.
In this way, we can perform prediction error degree estimation by using ordinal regression based on MKDA. In numerical weather prediction, various observed inputs, i.e., various input feature vectors, are obtained. Since discriminant analysis can consider the global information of the data with the distribution of the classes, the proposed method adopts it for the estimation. Furthermore, the proposed method adopts sampling of the training data and effectively uses the remaining data for estimating the optimal parameters of the multiple kernel scheme. This is the biggest difference between the proposed method and the conventional KDA-based method.