Video Frames Reconstruction Based on Time-Frequency Analysis and Hermite Projection Method

Abstract

A method for temporal analysis and reconstruction of video sequences based on the time-frequency analysis and Hermite projection method is proposed. The S-method-based time-frequency distribution is used to characterize stationarity within the sequence. Namely, a sequence of DCT coefficients along the time axes is used to create a frequency-modulated signal. The reconstruction of nonstationary sequences is done using the Hermite expansion coefficients. Here, a small number of Hermite coefficients can be used, which may provide significant savings for some video-based applications. The results are illustrated with video examples.

1. Introduction

Video signal exchange and storage are very important in multimedia applications. For this purpose, different kinds of video processing techniques are needed, such as video compression algorithms, video denoising methods, and scene analysis [1–4]. Depending on the video quality and bit-rate constraints, various compression algorithms have been developed [5–10]. These algorithms commonly employ motion-compensated differential coding (known as P and B frames), that is the interframe prediction based on the reference frames (I-frames). I-frames are set at user-defined intervals (e.g., 1 key frame for every 5 frames, or 15 frames, etc.). Thus, the algorithm compares two images and sends only the parts of the following images (B- and P-frames) that differ from the reference image [5]. For example, such algorithms are MPEG-2 compression and its improved version MPEG-4 [6]. A good implementation of MPEG-4 can additionally reduce the bit rate for approximately 15%, but it requires high processing power. Furthermore, the H.264 standard improves compression in comparison to MPEG-4 [6–8]. It offers many additional but optional tools, so that the compression ratio will significantly vary for different implementations. The most popular Baseline Profile provides a bit rate reduction of 10%–30% over MPEG-4, but it requires almost twice the CPU power. An overly simple H.264 implementation may produce worse results than an MPEG-4 implementation while the Main Profile is computationally heavy. Finally, some applications use the Moving-JPEG (MJPEG) multimedia format, where video frames are separately compressed as JPEG images [9]. It does not include interframe prediction, which results in lower compression ratio. However, it has been commonly used by digital still cameras for the unified treatment of still and video compression. Also, it has been used for IP-based video cameras via HTTP streams.

Here, we propose a method for video sequence reconstruction based on the time-frequency analysis and Hermite projections. The main goal of this paper is not to provide a specific compression solution for video applications, but rather an auxiliary tool for other video processing algorithms, such as video surveillance, motion tracking, and video compression. Combined with the existing compression algorithms, this approach can additionally reduce the amount of data required for high-quality video reconstruction. It does not use the exhaustive search procedures for motion estimation, spatial or temporal prediction, or the computationally demanding advanced options included in other approaches. The proposed procedure can be applied to the coefficients of raw video format or the reference frames (I frames) of coded video, or to the coefficients within the sequence of JPEG images. Therefore, the possibility to merge it with the existing techniques could be interesting for researchers and could provide additional improvements of compression ratio.

The procedure consists of two parts. The first one employs the time-frequency analysis to examine the temporal stationarity/non-stationarity of the coefficients over time. When observing a sequence of video frames, one may distinguish between stationary scene regions that do not change over time and dynamic scene regions containing moving objects (nonstationary regions). Video sequences usually contain noise, causing coefficients to vary, even in the absence of moving objects. In order to reduce the noise influence, here we propose a time-frequency-based procedure for temporally stationary and nonstationary coefficients characterization. Various time-frequency distributions have been used for the analysis of noisy nonstationary signals with different instantaneous frequency laws [11, 12]. Here, we focus on the use of computationally efficient quadratic distribution called the S-method [13, 14]. To characterize temporal behaviour, the sequence of coefficients at the position is analyzed by using the S-method.

The second part of the proposed procedure deals with the high-quality reconstruction of the coefficients. The reconstruction of a stationary sequence is based on its first coefficient. On the other hand, the efficient reconstruction of nonstationary sequences of coefficients is obtained by using the Hermite projection method [15]. Namely, by using a certain number of Hermite coefficients, nonstationary sequence can be reconstructed. This number could be quite smaller than the length of original sequence. Although, the quality of reconstructed video depends on the number of Hermite functions, significant savings can be achieved even if a high video quality is required.

The paper is organized as follows. Section 2 describes the theory behind the time-frequency analysis and its application for characterizing the temporal stationarity. In Section 3, the reconstruction procedure based on the Hermite projection method is proposed. In Section 4, the proposed method is applied to the examples. Concluding remarks are given in Section 5.

2. Theoretical Background

A brief theoretical background on the S-method-based time-frequency analysis and the Hermite projection method is presented in this Section. The time-frequency analysis will be used to characterize the stationarity of video coefficients over time while the Hermite projection method reduces the amount of data for high-quality video reconstruction.

2.1. Time-Frequency Analysis—the S-Method

Time-frequency representations have been used to analyze the time-varying spectral properties of nonstationary signals. The commonly used approaches are obtained by introducing the time dependency into the Fourier analysis using the time-windowing technique. Hence, the short time Fourier transform (STFT) is defined as follows [12]:

(1)

where is a signal, and w(t) is a window function. The spectrogram is the energetic version of STFT and it is defined as . The main drawback of the spectrogram is a low time-frequency resolution. Therefore, the quadratic distributions are introduced to improve time-frequency concentration. An efficient quadratic time-frequency distribution is obtained by the S-method. It is defined as follows [13]:

(2)

where is a finite frequency domain window. The S-method preserves the autocomponents concentration as in the Wigner distribution but significantly reduces or removes the cross-terms. Unlike the Wigner distribution, the oversampling in time domain is not necessary because the aliasing components will be removed in the same way as the cross-terms. The discrete form of the S-method can be written as follows:

(3)

where n and k denote discrete time and frequency, respectively, while the rectangular window P(l) is assumed. Parameter L determines the frequency window width which is . Windowing the product in the convolution through the narrow window , the cross-terms will be reduced or even removed. Thus, by choosing an appropriate value of , the sharpness of the Wigner distribution can be preserved while avoiding the cross-terms. Namely, high autoterms concentration is obtained with only a few summation terms due to the fast convergence within . Hence, in many practical applications is a suitable choice (e.g., ). Also, as shown in the sequel, a lower L value requires a fewer number of computations.

The S-method is computationally less demanding in comparison with other quadratic distributions. It requires complex multiplications and complex additions (N is the number of samples within the window), unlike the Wigner distribution which requires complex multiplications and complex additions. Also, the S-method allows simple and efficient hardware realization that has already been done [14].

2.2. Fast Hermite Projection Method

The Hermite projection method has been introduced in various image and speech processing applications [15–19]. Namely, it has been shown that this method could be efficient in image database retrieval, image filtering, texture analysis, text-independent speaker indentification, and so forth. The expansion into Hermite functions provides good localization in both signal and transform domain. Although the computation of Hermite functions seems to be a demanding task, they could be easily obtained using recursive realization as follows:

(4)

The first step in the Hermite projection method is to remove the baseline since:

(5)

The baseline is defined as follows:

(6)

where is a two-dimensional signal, and while the baseline is for a fixed . Further, the baseline is subtracted from the original values as follows:

(7)

The decomposition into N Hermite functions is defined as:

(8)

where holds for a fixed , while the coefficients of Hermite expansion are

(9)

Fast Hermite projection method uses the Gauss-Hermite quadrature to calculate the Hermite expansion coefficients as follows [15, 16]:

(10)

where are zeros of Hermite polynomials

(11)

The constants are obtained using the Hermite functions as follows:

(12)

3. Video Analysis and Reconstruction Using Time-Frequency Representations and Fast Hermite Projection Method

3.1. Analysis of Temporal Stationarity within the Video Sequence

By observing a video scene over time, usually there are some blocks that do not change (the box marked by 1 in Figure 1) while the others vary, for example, due to the presence of moving objects (the box marked by 2 in Figure 1). These two types of blocks will be referred to as stationary and nonstationary blocks, respectively. For example, a temporal sequence of pixels belonging to the stationary block should represent a constant amplitude signal, unlike the sequence of pixels from nonstationary block. The same holds when a sequence of frequency coefficients, for example Discrete Cosine Transform (DCT) coefficients, is observed instead of pixels. Thus, in order to analyze the stationarity/non-stationarity within the sequence of frames, a procedure described in the sequel can be applied to different coefficients. We focus on the DCT coefficients, since they are usually employed in image and video processing algorithms.

Figure 1

An illustration of stationary and nonstationary blocks in a sequence of frames (box 1-stationary block, box 2-nonstationary block).

The video frames are split in blocks and DCT coefficients are calculated. Further, the sequence of DC coefficients within the K consecutive frames is considered as follows:

(13)

where block position is determined by the position of its first coefficient while indicate frames' numbers.

The temporal sequence of coefficients may contain the nonstationarities due to the motion, noise, or luminance variations. Thus, the stationary sequence becomes slightly nonstationary even in the presence of a small amount of noise. The comparison between consecutive coefficients may lead to an incorrect conclusion. Consequently, cannot be used to indicate whether a sequence is stationary or not. In order to eliminate the influence of noise, the time-frequency analysis is employed. Therefore, the examination of stationarity is performed by using the time-frequency-based instantaneous frequency estimation. It is estimated as a position of the time-frequency distribution maxima as explained below.

Based on , a frequency-modulated signal is created as follows [17]:

(14)

where while μ is a constant that controls time-frequency resolution and t is a time vector. Thus, for each block, 64 frequency-modulated signals are created. Further, for the signal , the time-frequency distribution is obtained by using the S-method as follows:

(15)

One may note that

(16)

Therefore, if , the block at the position is stationary and will remain unaltered within K consecutive frames. Otherwise, the observed block is nonstationary.

The AC components (the alternating components, that is, the remaining 63 components in the DCT block) within the stationary block are stationary as well. The AC components within the nonstationary block should be analyzed separately. The S-method of a sequence of DC components belonging to nonstationary and stationary block are given in Figures 2(a) and 2(b), respectively. Also, time-frequency representations of two AC components are included.

Figure 2

Time-frequency representations of coefficients belonging to: (a) nonstationary block, (b) stationary block.

The time-frequency representation of stationary sequence should be robust to certain amount of noise, meaning that it should be flat even in the presence of noise. Otherwise, the nonstationarities caused by the noise may be interpreted as nonstationarities due to the motion. Note that additive noise within the sequence becomes multiplicative one after the frequency-modulated signal is formed (according to (14)). The performance of time-frequency distributions in the presence of multiplicative noises has been studied in the literature [20–23], where various analyses and optimality conditions have been derived. Here, numerous experiments have been performed to prove good characteristics of the proposed approach in a noisy environment.

It has been shown, (in Figure 3), that the proposed method can be robust in the presence of some additional Gaussian (zero mean and variance up to 0.001) and impulse noise (noise density up to 0.002) added to the video frames. In particular, three cases are observed for a stationary sequence:

• (i) Figure 3(a)—no additional noise (just the noise caused by luminance variations),

• (ii) Figure 3(b)—with Gaussian noise,

• (iii) Figure 3(c)—with impulse noise.

Figure 3

(a) without additional noise, (b) with Gaussian noise (zero mean and variance 0. 001), (c) with impulse noise (noise density 0.002).

In each case, one sample frame is illustrated (left), as well as the noisy sequence of DC coefficients and its time-frequency representation (right), which is flat even in the presence of noise.

In order to speed up the procedure, the S-method can be calculated for several components at the same time. Namely, a frequency-modulated signal can be modified into multicomponent signal as follows:

(17)

where is an AC component within the block. The S-method provides a cross-term free representation, but the components have to be spaced from each other by using the constants Namely, these constants are used to shift the components up and down from the central frequency, so that they do not overlap. They are integers whose values depend on the window width and can be chosen experimentally.

3.2. Hermite Projection-Based Temporal Reconstruction of Nonstationary Pixels within the Sequence of Video Frames

The Hermite functions are used as the basis functions for the video sequence expansion method due to their favorable properties. They represent an independent set of orthogonal functions, with good localization. Therefore, they can provide a unique representation of signals, while the coefficients of expansion are easily computed. Hence, the Hermite functions-based transform has been used in many applications for different types of signals, especially for images [15, 16]. Beside the Hermite functions, some other possible basis functions with desirable properties are Legendre polynomial, Laguerre polynomials, Bessel functions, and so forth [18]. For instance, the Legendre polynomials are defined on normalized intervals and their Fourier transform has infinite spread. Thus, there are difficulties to determine the expansion coefficients when the original signal is not explicitly given. The uncertainty inequalities for Laguerre polynomials cannot be easily reduced to a form that involves only expansion coefficients. In the case of Bessel function, the derivation of the coefficients from explicit or implicit information about the signal is very complicated [18].

Furthermore, by using the Hermite expansion, the signal energy is approximated by the numerical integral of the Gauss-Hermite type and converges more rapidly than the rectangle rule in the case of the DCT [19]. Therefore, the Hermite functions allow for a higher concentration of signal energy at lower frequencies and lead to better compression.

Consider the pixels , whose intensity varies over time. For frames, we can observe a nonstationary sequence in the following form:

(18)

where represents a pixel value in the k th frame. The sequence can be decomposed into N Hermite functions: A sequence of K elements can be reconstructed even by a small number of Hermite coefficients c _p , that is, for An error, depending on the value of N, is introduced by the reconstruction. Thus, with a suitable choice of N, a sequence with K pixels can be represented using smaller number (N) of coefficients without significant quality degradation.

Instead of pixels, one can reconstruct DCT coefficients within the blocks. For instance, a temporal sequence of DC components from the blocks whose central pixels are on the position is

(19)

The original nonstationary sequence V_DC for video frames is illustrated in Figure 4(a). Its time-frequency representation is given in Figure 2(a) (frames from 224 to 584). The two reconstructed sequences with and Hermite coefficients are illustrated in Figures 4(b) and 4(c), respectively. An additional moving average smoothing procedure is applied as well

(20)

where denotes the k th element of sequence reconstructed by N Hermite coefficients. Namely, the moving average smoothing is used to reduce the errors introduced by the reconstruction when the number of Hermite coefficients is significantly lower than the number of the original coefficients, such as

Figure 4

(a) Original sequence with 360 DC components, (b) the sequence reconstructed using 240 Hermite coefficients, (c) the sequence reconstructed using 180 Hermite coefficients.

Therefore, in the case with , the sequence is reconstructed by using a number of Hermite coefficients that is half the number of original coefficients, that is, . In the second case, the saving rate is .

The previously described procedure should be done for all AC components, as well.

4. Examples

Example 1.

A video sequence with 1200 frames (48 seconds) is considered. It is recorded by the video surveillance camera in the shopping center. It is split into three parts in order to illustrate different moving objects. Several frames for each of them are merged in Figure 5.

First, the temporal stationarity of blocks is analyzed. For this purpose, the frames are divided into blocks and the DCT is performed. Then, the DC sequences are obtained for

In the time-frequency analysis, the window width influences the resolution in the time-frequency domain. A narrow window produces good time resolution while a wide window produces good frequency resolution. In practical applications, the window width should be chosen to provide a good tradeoff between resolutions along the two axes. Here, the window widths of 32, 64, and 128 samples are analyzed and it has been shown experimentally that the width of 64 samples is the most appropriate for the considered sequence length. Thus, the stationarity of a DC sequence is analyzed by using the S-method with window width of 64 samples while . An appropriate value of is chosen to produce a smoothed representation of stationary coefficients, keeping the variations of nonstationary (dynamic) coefficients still intensive.

Here, three representative cases are observed as follows:

• (i) stationary block (e.g., box 1 in Figure 5),

• (ii) partly nonstationary (e.g., box 2), and

• (iii) nonstationary block (e.g., box 3).

The blocks with DC sequences producing constant value in the time-frequency domain (Figure 6(a)) are stationary over the considered time and could be reconstructed from the first frame. Therefore, a temporally stationary sequence of DC components is reconstructed over time by a single coefficient. The same holds for AC components from the stationary block.

Furthermore, we have considered a sequence which is a combination of stationary and nonstationary ones. Namely, a sequence of blocks that is mostly stationary over time and has just a couple of short nonstationary parts (Figure 6(b)) will be called partly nonstationary. Here, we assume that a partly nonstationary sequence has at least 2/3 of stationary coefficients over time (800 out of 1200 coefficients). In other words, the time-frequency representation of partly nonstationary sequence is linear along 2/3 of the sequence length. For instance, the partly nonstationary sequence presented by the S-method in Figure 6(b) can be reconstructed as follows:

• (i) stationary part 1:360-1 coefficient,

• (ii) nonstationary part 361:450-60 Hermite coefficients, that is,

• (iii) stationary part 451:900-1 coefficient,

• (iv) nonstationary part 901:1200-200 Hermite coefficients, that is, .

Thus, the total number of coefficients, required for the reconstruction of partly nonstationary sequence (Figure 6(b)) of length 1200, is 262. Note that two coefficients should be added for the baseline calculation of each nonstationary part. However, they do not have significant influence to the total number of coefficients.

The block whose DC sequence is mostly made of nonstationary segments is called a nonstationary block. An illustrative example is given in Figure 6(c). Due to its complexity and dynamics, the reconstruction of such a sequence requires a higher number of coefficients:

• (i) nonstationary part 1:360-257 Hermite coefficients

• (ii) stationary part 361:460-1 coefficient,

• (iii) nonstationary part 461:520-42 coefficients

• (iv) stationary part 521:690-1 coefficient,

• (v) nonstationary part 691:1100-230 coefficients,

• (vi) stationary part 1101:1200-1 coefficient.

The total number of coefficients is 532 (without the baseline ones). For the three observed sequences, the average number of Hermite coefficients, required for the reconstruction, is 265 per sequence. It provides the average saving ratio .

Note that, if the DC component is nonstationary, most of the AC components are also nonstationary. The S-method obtained for a few AC components within the nonstationary block is shown in Figure 7(a)–7(d). In the case of AC components reconstruction, a high quality is achieved with . Although the block is nonstationary, some coefficients (e.g., AC (4, 4) in Figure 7(d)) can be partly nonstationary and require just a partial reconstruction with Hermite coefficients.

The total number of stationary, partly nonstationary, and nonstationary blocks within the 1200 frames of the observed sequences is given in Table 1. For the sake of simplicity, it is assumed that all 64 components within the block have almost the same temporal behavior. Nevertheless, there could be slight variations for some of the AC components.

From the presented statistics, we can calculate the total number of coefficients for video reconstruction, which is approximately 20% of the number of original coefficients.

Some of the reconstructed and original nonstationary blocks are illustrated in Figure 8. Each row presents a reconstructed block (left) versus its original version (right). The blocks are chosen randomly from different frames to illustrate the quality of reconstruction. Note that the difference between the original and reconstructed blocks is imperceptible. Additionally, an original and corresponding reconstructed frame is shown in Figure 9. It can be seen that the reconstructed frame preserves the quality of the original one.

The peak signal to noise ratio (PSNR) is calculated and it is approximately around 47 dB, which is significantly higher than in the other compression algorithms [10]. As previously estimated, the proposed method requires approximately 20% of the original coefficients for such a high-quality reconstruction, entailing the compression ratio 5 : 1. Thus, if combined with the existing algorithms it may significantly improve the total compression ratio, without degrading the quality. The estimated compression ratio can be further increased which will produce a lower PSNR.

Note that the number of Hermite coefficients N, used for the reconstruction, has been set empirically, based on a large number of tests. Namely, in the experiments we have with PSNR ≈ 47 dB. By increasing the ratio K/N, PSNR between the original and reconstructed frames slowly decreases (e.g., = 1.8 ⇒ PSNR ≈ 43 dB, = 2.2 ⇒ PSNR ≈ 40 dB, etc.).

Table 1 The number of stationary and nonstationary blocks within the considered video sequence.

Figure 5

An illustration of test video sequence.

Figure 6

The S-method of: (a) a stationary DC sequence, (b) a partly nonstationary DC sequence, (c) a nonstationary DC sequence.

Figure 7

The S-method of AC components on the positions (a) (2,1), (b) (1,2), (c) (3,3), (d) (4,4).

Figure 8

Zoomed reconstructed (left) and original blocks (right) from randomly chosen frames.

Figure 9

(a) Original frame, (b) Reconstructed frame.

Example 2.

This example aims to show that the proposed method can be performed even on a set of nonconsecutive frames, such as I frames in the MPEG sequence. For this purpose, we made a new sequence of frames that will be called I sequence by selecting each 13th frame from the starting video sequence (we assumed that the I frame rate is set at every 13 frames). However, without loss of generality, we can also use each 5th, 12th, or 15th frame, depending on I frame refreshing rate which can be user defined. The total number of frames within the sequence is 126. Due to a smaller number of coefficients than in the previous case, the window width is 42 samples for the calculation of the S-method.

In order to optimize the processing time, the S-method is calculated for several components at once. The illustrations are given in Figure 10, where the multicomponent time-frequency representation is given for four DCT components from two image blocks. Note that, the DCT components within the first block (Figure 10(a)) are mostly stationary, unlike the components from the second block.

The reconstruction procedure is performed for each coefficient as described in the previous example. The stationary segments are reconstructed by a single coefficient, the nonstationary parts of DC components with ratio , while the ratio for nonstationary segments of AC sequences is An example with the original and corresponding reconstructed sequence is shown in Figure 11. The reconstructed and the corresponding original blocks from different frames are zoomed in Figure 12. The same blocks from Example 1 are observed. Although the I sequence contains significant discontinuities comparing to the case when each frame is used, the proposed approach again provides a high-quality reconstruction, with a slightly lower PSNR than in the previous example.

Figure 10

The S-method calculated for a few DCT coefficients (a) mostly stationary, (b) nonstationary coefficients.

Figure 11

Original and reconstructed I sequence.

Figure 12

Zoomed reconstructed (left column) and original blocks (right column) from randomly chosen frames.

Example 3 (Performance comparison with MJPEG).

In this example, we discuss one simple solution for combining the proposed approach with the Motion JPEG algorithm in order to improve the compression ratio. A part of a video sequence having 126 JPEG frames (as a basis of MJPEG format) of total size 1.38 MB is used. The frame size is while the average number of bits per block is .

The proposed approach classifies DCT blocks into stationary (S) and nonstationary (NS) ones. In the considered sequence, the number of S blocks is , while . All the coefficients from the S blocks are constant over time and can be reconstructed from the corresponding first frame's blocks. Thus, while the set of 126 JPEG frames requires No 126 bits, the proposed approach needs  bits to represent the coefficients of S blocks.

Each NS block of DCT coefficients during 126 frames forms a matrix of the size . Using the proposed approach, it is represented by the matrix of Hermite coefficients of the size . In other words, instead of 126 DCT -blocks, we have 70 -blocks of Hermite coefficients. The blocks of Hermite coefficients (rounded to the integer values), look very similar to the quantized DCT blocks, having the same range and distribution of values. Thus, they can be treated and coded in the same way as DCT blocks in the JPEG algorithm (zigzag scan, lossless entropy coding, etc.). The total number of bits (for the observed sequence) can be calculated as follows:

• (i) for Motion JPEG: ,

• (ii) for the combined (proposed + MJPEG) approach: .

In this example, the combined approach leads to 10 times smaller size of video sequence.

5. Conclusion

The proposed method for video sequence reconstruction employs two different signal processing techniques: the time-frequency analysis and the Hermite projection method. The time-frequency distribution provides an efficient analysis of temporal variations of coefficients. In that sense, it is used to distinguish stationary and nonstationary coefficients. Temporally nonstationary coefficients are reconstructed using a smaller number of Hermite expansion coefficients. The results have shown that the high-quality video reconstruction can be achieved by using significantly reduced number of coefficients. An additional improvement can be obtained by using the JPEG compression to reduce the number of AC components that should be reconstructed. The future works could include the time-frequency-based analysis of temporal stationarity in video surveillance applications to detect the appearance of moving objects. For instance, the surveillance system may ignore nonstationarities of short duration (e.g., bird flyover) while the attention should be paid when nonstationary segments last longer (meaning that significant movements appear). To make the proposed method faster for possible real time applications, it would be necessary to develop a special purpose hardware implementation.