A New Frame Memory Compression Algorithm with DPCM and VLC in a 4 × 4 Block

Frame memory compression (FMC) is a technique to reduce memory bandwidth by compressing the video data to be stored in the frame memory. This paper proposes a new FMC algorithm integrated into an H.264 encoder that compresses a 4 × 4 block by di ﬀ erential pulse code modulation (DPCM) followed by Golomb-Rice coding. For DPCM, eight scan orders are predeﬁned and the best scan order is selected using the results of H.264 intra prediction. FMC can also be used for other systems that require a frame memory to store images in RGB color space. In the proposed FMC, RGB color space is transformed into another color space, such as YCbCr or G, R-G, B-G color space. The best scan order for DPCM is selected by comparing the e ﬃ ciency of all scan orders. Experimental results show that the new FMC algorithm in an H.264 encoder achieves 1.34 dB better image quality than a previous MHT-based FMC for HD-size sequences. For systems using RGB color space, the transform to G, R-G, B-G color space makes most e ﬃ cient compression. The average PSNR values of R, G, and B colors are 46.70 dB, 50.80 dB, and 44.90 dB, respectively, for 768 × 512-size images.


Introduction
Frame memory size and bandwidth requirements often limit the performance of a video processor designed for implementing a video compression standard such as MPEG-2, MPEG-4, and H.263 or H.264/AVC [1][2][3][4]. Frame memory compression (FMC) is a technique to reduce frame memory size by compressing the data to be stored in frame memory. Memory bandwidth requirement is also reduced by FMC because data access requirements are reduced. Figure 1 shows a video processor in which the encoder and decoder of an FMC algorithm are integrated inside the processor. A reference frame is, in general, stored in an off-chip memory. When the video processor stores the reference frame in the off-chip memory, the FMC encoder compresses the data. To access the reference frame from the off-chip memory, the video processor fetches compressed data from the off-chip memory and the FMC decoder decompresses and restores the original data.
Three properties, low latency, random accessibility, and low image quality degradation, are required for an efficient FMC algorithm. Video processor performance is significantly affected by the speed of the external memory, and FMC algorithm latency delays the access of external memory. Therefore, low latency in the FMC algorithm is required to minimize performance drop-off. Image compression algorithms like JPEG2000 are not suitable for FMC because they are too complex for low latency implementation, although their compression efficiency is high. The second property, random accessibility, is needed because frame memory can be accessed at an arbitrary address. Finally, FMC algorithms, in general, adopt lossy compression to maintain relatively high compression efficiency. Lossy compression typically degrades image quality, and therefore, additional image quality degradation may limit the practical use of FMC algorithms.
Extensive research efforts have been made to reduce the size and bandwidth requirements of frame memory [5][6][7][8][9]. A popular technique for FMC is a transform-based approach in which a frame is decomposed into small blocks that are transformed into a frequency domain by a simple transform, such as discrete cosine transform (DCT) [  Transform or its variations [7]. The frequency domain coefficients are then compressed by quantization followed by variable length encoding, such as Golomb-Rice coding. A transform-based approach achieves an efficient compression when the block size for a transform is large. For example, the block size in the algorithm in [6] is 16 × 16. As the transform block size increases, the hardware complexity of a transform as well as the compression latency also increases.
Another approach is a spatial domain FMC that requires a relatively small amount of computation [8,9]. The FMC in [8] is a variable-ratio compression which achieves an average of 40% memory reduction. The FMC in [9] is a DPCMbased approach which achieves 50%-constant compression with pattern matching and selective quantization. This FMC is implemented in software, but it is not verified in hardware. Due to the sequential nature of the pattern decision, a large latency is expected if this algorithm is implemented in hardware. Frame memory compression techniques for specific applications have been proposed [10,11]. For LCD, it is often the case that data are over-driven to compensate for the slow response time of an LCD panel. To detect the difference between the current and the previous frames, the previous frame is stored in a frame memory. FMC is used to reduce the frame memory space and aggressive techniques are employed at the sacrifice of the image quality because a reconstructed image is used only to detect the difference and slight image quality degradation is tolerable. Another example use of FMC is texture compression in graphics rasterization [12,13]. In general, a slight image quality degradation is allowed in texture image rasterization. Therefore, texture compression often uses the dictionarybased approach that aims an aggressive compression ratio at the sacrifice of image quality. Both algorithms for the LCD over-drive and texture compression allows image quality degradation, and consequently, they may not be suitable for image compression integrated in an H.264 compression chip. This paper proposes a new FMC algorithm that compresses frame data efficiently by using intraprediction information provided by an H.264/AVC encoder. The proposed algorithm divides an image frame into 4 × 4 blocks and compresses each block independently by a 50% constant compression ratio. For each 4 × 4 block, DPCM is performed along a predefined scan order. To achieve high compression efficiency, eight DPCM scan orders are predefined on an analog of the eight 4 × 4 intraprediction modes (excluding the DC prediction mode) for an H.264/AVC encoder. To select the best scan order, the FMC algorithm uses the information provided by H.264/AVC intraprediction because those predictions evaluate the correlations among neighboring pixels and provide information about the direction between highly correlated pixels. Once H.264 intraprediction mode is selected, the scan order is selected from the intraprediction mode, and DPCM is performed. The DPCM results are further compressed by Golomb-Rice coding. If the compression ratio does not reach 50%, the 4 × 4 block pixel data are quantized by 1-bit right shifting, repeat DPCM, and entropy coding.
Frame memory is used not only in a video compression processor but also in an LCD driver [10,11] or a 2D/3D graphics processing chip [14]. A 50% compression of a reference frame can also be used for these chips to save the frame memory bandwidth and space. However, these chips do not include an intraprediction module, so that the best scan mode must be decided by the FMC algorithm itself. Furthermore, video compression standards usually employ the YCbCr 4 : 2 : 0 color format in the frame memory, but other chips often employ the RGB 4 : 4 : 4 color format. Therefore, the FMC algorithm for a video processor is not directly applicable for an LCD driver or a 2D/3D graphics chip. The second part of this paper modifies the FMC algorithm proposed for an H.264/AVC encoder to be used for the frame memory compression for these chips. This modification includes the transform of the RGB color space to another color space efficient for compression. Other modifications are the inclusion of the step to select the best scan mode and the combined packetization of three color components.
The paper is organized as follows. In Section 2, the proposed FMC algorithm is described. Then the FMC algorithm for RGB color space is presented in Section 3. Section 4 explains the hardware implementation of the proposed FMC algorithm. Section 5 compares image quality degradation of the proposed FMC algorithm with a previous algorithm. Conclusions are presented in Section 6.

FMC with H.264/AVC Video Compression
This section proposes an FMC algorithm that can be used to reduce frame memory for an H.264/AVC encoder.

Basic Idea.
The proposed FMC algorithm was designed to compress a 4 × 4 block by 50% and generate a 64-bit packet. To achieve this aim, the proposed algorithm employs DPCM, which calculates differences between successively scanned data and uses those differences to represent the data. For efficient DPCM compression, the differences between successive data should be small so that the data can be represented by a small number of bits. The magnitude of the difference depends on the image contents as well as the scan order. For example, if a 4 × 4 block includes vertical stripes, a DPCM scan along the vertical direction results in a smaller difference than that along the horizontal direction. Therefore, it is important to select a scan order that minimizes the differences between data. To this end, the proposed FMC algorithm uses eight scan modes (see Figure 2). The eight modes are based on an analog of the    Figure 2 because Mode 2 did not provide information useful for scan order selection. An advantage resulting from Mode 2 exclusion is that only three bits are needed to represent the remaining eight modes. The eight modes in Figure 2 cover various image types for DPCM scans. For example, Mode 0 is suitable for an image with vertical stripes while Mode 1 is suitable for horizontal stripes, and an image with diagonal stripes may be best suited to one of the other modes.

2.2.
Algorithm. The flowchart of the proposed algorithm is shown in Figure 3. A single 4 × 4 block is the input of the algorithm and the output is a 64-bit packet. As this FMC is designed to reduce frame memory for H.264/AVC compression, the H.264/AVC compression operations, including intraprediction, are performed with FMC. To select two scan modes from among the 8 modes shown in Figure 2, the 4 × 4 intraprediction result is assessed by the algorithm. The first mode is the same as that determined by intraprediction, excluding the DC mode. The horizontal and vertical modes, in general, produce efficient FMC results. Thus, one of these two modes is always selected as the second mode. For example, if modes 1, 3, 5, or 7 are selected first by H.264 intraprediction, then mode 0 is selected as the second mode, while if modes 0, 4, 6, or 8 are selected first, mode 1 is selected second. If the DC mode is selected by intraprediction, modes 0 and 1 are selected as the first and second modes, respectively.
The two selected scan orders are provided to the next step, which performs DPCM operations along the selected scan orders. The input 4 × 4 pixels are quantized with the quantization parameter (QP). For quantization, the input data are right shifted by QP times. For example, if QP = 2, then the input data are shifted to the right twice. During this shift operation, the left most bit is replaced by 0. The quantization parameter is initially set to 0 and incremented later, if required. The DPCM results are compressed by Golomb-Rice coding and the required number of bits for a single packet is calculated. If this number is less than the limit (i.e., 64 bits), then the result of Golomb-Rice coding is packed into a 64-bit packet. Since two scan modes are selected and Golomb-Rice coding is performed for both modes, the one requiring the smaller number of bits is selected. If the Golomb-Rice coding result requires a larger number of bits than the limit, the QP is incremented by 1 and quantization, DPCM, and Golomb-Rice coding are performed a second time. The Golomb-Rice coding and packetizing steps are explained next. In order to match the desired bit-rate, the proposed algorithm prequantizes the input pixels and then applies DPCM. However, in lossy DPCM usually, there is a feedback loop, and quantization is applied during (and not before) the prediction. For a uniform quantizer, if the quantization step size Δ (= 2 QP ) is sufficiently small, it is reasonable to assume that the quantization error is uniformly distributed in interval [−Δ/2, Δ/2]. Note that the QP value used in the proposed FMC is small (see Figure 20). Therefore, the quantization error is likely to be distributed uniformly. This implies that the quantization errors in both the feedback loop and prequantization approaches have similar distribution of quantization error and consequently the coding errors of the two DPCMs do not differ significantly.
On the other hand, the hardware complexity of the prequantization is just about a half of that required by the conventional feedback-loop approach because the conventional approach requires two adders in addition to the dequantizer for an encoder whereas the prequantization requires just a single adder. In summary, the prequantization DPCM is adopted in this paper because computational complexity is about a half of the feedback-loop DPCM although the prequantization DPCM increases slightly the coding error.

Golomb-Rice
Coding. The Golomb-Rice coding [15,16] accepts only a nonnegative number as input. However, a DPCM result can be negative. Therefore, for Golomb-Rice coding input, a negative DPCM result is converted into a nonnegative number by where diff represents a DPCM result and source represents the input to the Golomb-Rice coding. For Golomb-Rice coding, source is divided by 2 k and the division quotient is represented in unary notation that represents a nonnegative integer, n, with n zeros, followed by a single one. The quotient and remainder in conventional k bit binary notation are then concatenated to form a Golomb-Rice codeword. The length of a Golomb-Rice codeword is For a small source, a smaller k results in a smaller Golomb-Rice codeword length. As source increases, a larger k may produce a smaller code length. Thus, the choice of k depends on the value of source. For example, if k = 0, the length increase is too large for a large source. On the other hand, if k > 2, the length is too large for a small source, and a k greater than 2 is unacceptable for 50% compression because the minimum number of bits assigned to each pixel is 4. Therefore, the chosen value of k is either 1 or 2. For the eight modes shown in Figure 2, a difference along the dotted line is encoded with k = 2 while a difference along the solid line is encoded with k = 1. DPCM results along dotted lines may be large because the dotted lines cross edges. In this case, a large k may lead to a smaller number of bits to represent this large difference. By assigning the large k (k = 2) to the dotted line and the small k (k = 1) to the rest, the total number of bits generated by Golomb-Rice coding for all 16 pixels are, in general, reduced. 2.4. Packetization. The Golomb-Rice codewords are packetized as a 64-bit packet. Figure 4 shows the packet format. The 8 scan modes are coded with 3 bits and stored in the leftmost position and the 3-bit QP is stored next. The first pixel requires (8−QP) bits stored next to the QP and the remaining bits store the Golomb-Rice codewords for the remaining 15 pixels. Video compression standards, such as H.264/AVC, employ the 4 : 2 : 0 format in the YCbCr color space to represent an image. In general, the three color components are stored in separate spaces in frame memory. One reason for separate memory allocation is because the three components are not always accessed at the same time. For example, motion estimation in the H.264/AVC requires only the Y color component. Another reason is the difference in the amount of data in the Y and Cb (or Cr) color components. In the 4 : 2 : 0 format, Y color data are assigned to each pixel, while single Cb (or Cr) data are assigned to every 2 × 2 pixels [17]. Thus, the amount of data for the Cb (or Cr) color component is one fourth of that for the Y color component. As a result, the Y color component requires four times larger memory space than the Cb (or Cr) color components. As the three colors are stored separately and accessed independently, they are also compressed independently. Thus, the FMC algorithm in Figure 3 is performed independently three times for Y, Cb, and Cr colors. Figure 5(a), and assume that the intraprediction mode resulting from H.264/AVC is 1. Thus, the first scan order selected is mode 1 and the second scan order is mode 0. QP = 0 requires 91 bits for mode 1 and 212 bits for mode 0. Thus, QP = 0 is not acceptable for both modes. For QP = 1, mode 1 scans data as denoted by the arrow shown in Figure 5(b). The pixel values quantized with QP = 1 (i.e., shifted once to the right) are also shown in Figure 5 Table 1 shows the Golomb-Rice codewords for the DPCM results. For example, the fourth DPCM result, DPCM [4], is −9. From (1), the source for this value is 17. From k = 2, the quotient and remainder are 4 and 1, respectively. The quotient in unary notation is 00001 and the remainder in k-bit binary notation is 01. The final codeword is the concatenation of the quotient and remainder, that is, 0000101. Table 1 shows the codewords of all DPCM results. Fifty bits were required for all the words. In addition to these bits, 6 bits are necessary to store the mode and QP and 7 bits are required for the first datum.  1. As mode 1 requires fewer bits than mode 0, it is chosen as the best scan mode. Figure 6 shows the FMC result. In Figure 6, the first three bits (001) and the next three bits (001) represent mode 1 and QP, respectively. The next seven bits represent the first datum quantized by QP = 1. The remaining bits are the Golomb-Rice codewords of the next 15 DPCM results.

FMC of Frame Memory in RGB Color Space
There exist a number of applications other than H.264/AVC video compression that store video data in frame memory. For instance, an LCD display driver needs frame memory to store its output video [10,11]. For another example, a 2D or 3D graphics processor also requires frame memory [14]. The FMC algorithm proposed in Section 2 is not directly applicable to these other applications because they cannot use the information obtained by H.264/AVC intraprediction. Moreover, these other applications, in general, store video in the RGB color space while the algorithm in Section 2 is developed for video in the YCbCr color space. This section extends the algorithm proposed in the previous section and proposes the FMC algorithm suitable for video in the RGB color space.

FMC in the 4 : 4 : 4 Format and Combined Packetization.
In an LCD display driver or 2D/3D graphic processor, an image is stored in the RGB 4 : 4 : 4 format in which each pixel is represented by R, G, and B color components. Unlike the YCbCr colors in the 4 : 2 : 0 format, RGB color components in the 4 : 4 : 4 format are, in general, accessed at the same time [10][11][12][13][14]. Thus, an effective memory access is possible by storing three color components for one pixel in consecutive memory addresses. As three color components are stored consecutively and accessed at the same time, these components can also be compressed at the same time to be packetized into a single combined packet. The combined packet allows more efficient compression than the separate packet because the scan mode and QP can be shared by these three colors. The format of the combined packet is shown in Figure 7. The 4 × 4 block in the 4 : 4 : 4 format consists of 16 pixels of three colors, so that total 384 bits are required to store a single 4 × 4 block. By 50% compression, the compressed packet size is less than or equal to 192 bits.  Figure 6: The packetized result of the example shown in Figure 5 and Table 1.  The scan mode and QP are stored in the leftmost 6 bits. Note that only one scan mode and QP are required for three colors.
The first pixel data of three colors are stored next followed by remaining pixels. For the compression of the remaining data, it is observed experimentally that the Exp-Golomb coding is more efficient than the Golomb-Rice coding (see details in the next subsection).

Exp-Golomb Coding.
Golomb-Rice codewords used in Section 2 are efficient when the value of source is not large.
Recall that the length of a Golomb-Rice codeword increases in proportion to its value. On the other hand, another entropy coding, the length of an Exp-Golomb codeword, is The details about Exp-Golomb coding are presented in [17]. The length of an Exp-Golomb codeword increases in proportion to log(source). Therefore, Exp-Golomb coding generates a shorter codeword than Golomb-Rice coding when the value of source is large. It is observed by experiments that Exp-Golomb coding is more efficient than Golomb-Rice coding for combined packetization (see Figure 20). Similar to Golomb-Rice coding, a large k generates a short codeword when the value of source is large. On the other hand, a small k is preferable for a small source. Thus, the value of k is chosen in the same manner as for Golomb-Rice coding in Section 2; that is, 2 is chosen for the source (DPCM results) represented by the dotted line's shown in Figure 2 while 1 is chosen for the rest DPCM results.

Scan Mode Decision.
Among the eight possible scan modes shown in Figure 2, the mode that generates the smallest packet size must be selected. In Section 2, two candidate scan modes are determined from the intraprediction mode in H.264/AVC. Then, the results of the two modes are compared and the best mode is selected between the two candidate modes by comparing their packet sizes. For the FMC in the RGB color space, the information from H.264/AVC is not available. Thus, all eight scan modes are compared and the best mode is selected among them. To this end, the parameter QP is set to 0 and the lengths of fifteen sources (DPCM results) are evaluated and then added to obtain the packet size. The packet size must be evaluated for the whole eight scan modes, so that a large amount of computation is required for the selection of the best scan mode.
The computation for best mode selection is reduced by taking advantage of the fact that there exist many DPCM results that are shared by multiple scan modes. For instance, in Figure 2, the first DPCM results of mode 1 and 2 are identical (i.e., they are the difference between the leftmost top pixel and its next pixel to the right). For the eight scan modes with fifteen DPCM results each, the code lengths of 120 DPCM results need to be evaluated. Among these 120 DPCM results, 57 results are shared by more than one scan modes. Thus, 63 DPCM results in total are necessary for the evaluation of the code lengths for eight scan modes.
To obtain the accurate packet size, the evaluation of the lengths of sources must be repeated until the packet size is less than 192. However, the repeated evaluations require too much computation. Therefore, only the evaluation with QP = 0 is used to choose the best scan mode. Experiments show that the order of the packet size chosen with QP = 0 is almost the same as the order with the best QP.

Color Transform.
With experiments, it is observed that the compression efficiency is improved when the RGB color space is first transformed into the YCbCr color space and then the FMC is applied to the image in the YCbCr space (see Section 5 for details on the experimental results). Note that the transformed image in this case is in the 4 : 4 : 4 format instead of the 4 : 2 : 0 format as in Section 2. Thus, all three color components are available for each pixel, and they are packetized in the same format shown in Figure 7. One of the reasons why the YCbCr color space is more efficient than the RGB color space is because the data in the Cb and Cr colors vary more slowly than those in the R and B colors, respectively. As a result, the DPCM results in the Cb and Cr colors are smaller than those in the R and B colors, respectively. The combined packetization of Y, Cb, and Cr colors allow increased bits assigned to the Y color thanks to the reduced bits assigned for Cb and Cr colors. The increased bits assigned to the Y color decreases the error in the Y color, and consequently, the error in the Cb and Cr is also reduced because Y is used to derive Cb and Cr. Moreover, Y color affects the subjective quality greater than Cb or Cr color. As a result, image quality is, in general, improved by the color space transform. The transform coefficients between the RGB color space and the YCbCr color space are given by ITU-R recommendation BT.601 [18].
One source of quality degradation with the YCbCr color space transform is the round-off error in the transform. For instance, consider the pixel with its value {R For the FMC in the RGB color space, it is not mandatory to use the YCbCr color space. In the JPEG2000 standard for image compression, a modified YCbCr color space is used for the removal of the transform error [19]. The FMC algorithm can be applied to the JPEG2000 YCbCr color space just in the same way as the original YCbCr color space. The transformation error is reduced because the transformation is reversible. In JPEG2000, 9 bits are used to store each of Cb and Cr components so that no error is created by the transform. Thus, the image quality with the JPEG YCbCr space is better than that with the original YCbCr space.
A number of demosaicing algorithms [20][21][22][23] as well as digital display interface such as low-voltage differential signaling (LVDS) adopt the color space consisting of G, R-G, B-G instead of the YCbCr color space. One of the main advantages is a simple transformation from/to the RGB color space because only subtraction operations are needed for the transformation. Another advantage comes from the fact that the error in R-G or B-G does not affect the G color space so that the error in the G color space is less than that in R-G or B-G color space. This property can reduce the quality degradation by color transform because human eyes are more sensitive to the G color than the R or B color. For simplicity, Dr and Db are used hereafter to denote R-G and B-G spaces, respectively. Instead of the original YCbCr color space, the JPEG2000 color space or GDbDr color space can also be used for FMC. Section 5 presents experimental comparisons among these color spaces.
In the packet shown in Figure 7, the first three pixels are stored from the 7th bit. In the RGB color space, (8 -QP) bits are necessary to store one color component of the first pixel. Thus, to store three colors, 3 · (8 − QP) bits are required. For the original YCbCr color space, 8 bits are required to represent Y, Cb, Cr color components. Thus, 3 · (8 − QP) bits are necessary to store the first pixel in the packet shown in Figure 7. In the JPEG2000 YCbCr color space, (8 -QP) bits are needed for the first pixel. On the other hand, (8 -QP + 1) bits are needed for the Cb color of the first pixel because they include the sign bit. Similarly, Cr also requires (8 -QP + 1) bits. Therefore, (8 − QP) + 2 · (8 − QP) bits are required to store Y, Cb, and Cr colors of the first pixel. For the GDbDr color space, G requires 8 bits while Dr or Db requires 9 bits. Thus, (8 − QP) + 2 · (8 − QP) bits are also necessary for the first pixel. Figure 8 shows the flow chart of the FMC algorithm discussed in this section. This algorithm processes three color components in the YCbCr or GDbDr space simultaneously, so that the number of bits for the input 4 × 4 pixels is 384 and that for the output packet is reduced to 192 by 50% compression. When compared with the algorithm shown in Figure 3, the first step is added to the transform from the RGB color space to YCbCr (or GDbDr) color space. The scan mode decision step is different from that in Figure 3 because the best scan mode is decided by comparing all 8 scan modes. The Golomb-Rice coding is replaced by Exp-Golomb Coding, and Quantization, DPCM steps are the same as those in Figure 3.

Algorithm.
3.6. FMC by 75%. The data in the RGB color space can be compressed by 75% with the combination of color transform, subsampling, and the FMC proposed in Section 2. Subsampling from the 4 : 4 : 4 format to the 4 : 2 : 0 format achieves 50% compression. Recall that the FMC algorithm in Section 2 is applied to the subsampled data in the 4 : 2 : 0 format to achieve another 50% compression. Color transform to another color space like YCbCr is necessary because the subsampling of the Cb and Cr colors does not severely deteriorate the visual quality of an image because human eyes are more sensitive to the Y color than Cb and Cr colors. The original YCbCr color space may create a roundoff error. To reduce this error, the JPEG2000 YCbCr color space or the GDbDr color space is also considered as the target color space. The effectiveness of three color spaces are evaluated by experiments as presented in Section 5.

Hardware Implementation
This section explains the hardware implementation of the proposed FMC algorithm in Section 2.

Encoder.
The pipeline architecture of the FMC encoder is shown in Figure 9(a). To increase the throughput, the encoder operation is pipelined in four stages. In pipeline Stages 1 and 2, quantization, DPCM, and Golomb-Rice  encoding are performed for codeword generation. Initially, QP is chosen 0 and the codeword is generated. In Stage 3, if the codeword size is less than or equals to 64 bits, the pipeline moves to the next stage. Otherwise, the QP is incremented and Stages 1, 2 and 3 are repeated. The codeword generation and QP increment are repeated until the codeword size is less than or equal to 64. Five cycles are needed to complete a single iteration of Stages 1, 2, and 3. The total execution time is 5(QP + 1) + 1 cycles because Stages 1, 2, and 3 take 5 cycles. If QP is 0, a new 4 × 4 block is processed at every 5 cycle. The gate count of the FMC encoder is 19.8 K.

Decoder.
In general, the execution time of an FMC encoder is not critical because the compressed data are not used immediately but they are stored in a frame memory for use in some time later. However, the execution time of an FMC decoder is critical because its result is immediately used. Therefore, an optimized hardware design is needed to minimize the execution time of a decoder. Figure 9(b) shows the proposed pipelined architecture of an FMC decoder. In Stage 1, a 64-bit packet is read from the frame memory. The proposed FMC decoder needs 5 cycles to complete one 4 × 4 block and processes a new 4 × 4 block for every 3 cycles. Assuming that the memory bandwidth is allowed to transmit 32 bits per a cycle, the throughput of the FMC decoder is larger than that of the frame memory. Therefore, the memory bandwidth is the bottleneck of the overall throughput and the addition of the FMC decoder does not decrease the data access throughput. The gate count of the FMC decoder is 11.3 K.

Complexity Comparison.
The complexity of the proposed algorithm is compared with the previous work based on Modified Hadamard Transform [7]. Table 2 shows the numbers of additions (or subtractions) and shifts required for both encoding and decoding operations of FMC. For the proposed FMC, N represents the number of iterations. The  the communication between modules. One AHB bus is mainly used for the control of the hardware modules by ARM7TDMI processor and the other AHB bus is mainly used for data communication between hardware modules and external memory. The FMC encoder and decoder are placed between the AHB bus and the memory controller. Figure 11 shows the layout and the chip photograph of the H.264/AVC encoder shown in Figure 10. The die area of the H.264/AVC encoder is 5 mm × 5 mm using the Dongbu 1P6M 0.13 um CMOS technology.

FMC Algorithm in an H.264 Encoder.
Software implementation of the proposed algorithm in Section 2 is integrated with H.264/AVC JM reference software version 13.2 [25] so that the reference frame is compressed by the proposed FMC. Previous work, based on Modified Hadamard Transform (MHT) proposed in [7], is also implemented and the results are compared. The two algorithms are evaluated with three CIF-size (352 × 288) video sequences: Foreman, Mobile Calendar, and Table Tennis; as well as with two HD-size (1920 × 1080) sequences: Blue sky and Pedestrian area. For every sequence, 100 frames are used and the encoding speed is 30 frames per second. For experiments, the test sequence is encoded as a Baseline profile stream with the intraframe interval of 10, 3 reference frames for motion estimation, deblocking filter turned on, ratedistortion optimization also turned on, and four QP values, 20, 24, 28, and 32. The rate distortion performances for Y component are shown in Figure 12. The average PSNR degradations, by the FMC algorithms, are measured and shown in Table 3. These values are obtained by Bjontegaard's method presented in [26]. For the three CIF-size sequences, the average PSNR degradations are 0.77 dB and 2.39 dB by the proposed and MHT-based FMCs, respectively. For the two HD-size sequences, the average PSNR degradations are 0.38 dB and 1.72 dB by the proposed and MHT-based FMCs, respectively. For both CIF-size and HD-size video sequences, the proposed FMC makes a significant improvement over the previous MHT-based FMC. The results also show that quality degradation of HD-size video is less than that of CIF-size video. This is because spatial correlation of a 4 × 4 block generally increases as image size increases, so that compression with minimal loss of information is possible. The simulation also evaluates the efficiency of the scan mode decision step in Figure 3. The mode selected by the scan mode decision step may not always be the scan mode that maximizes the compression efficiency. Thus, all 8 modes are used by the FMC algorithm and the best scan mode is then selected. In Figure 12, "8-mode FMC" presents the results when the best scan mode is selected from among all 8 modes. Another simulation uses the scan mode selected by the H.264 intraprediction, "1-mode FMC" (Figure 12). The computational complexity of 1mode is half of that using the proposed algorithm because only one mode is evaluated while the proposed algorithm evaluates two modes. The 1-mode quality degradation is larger than that using the proposed algorithm. Comparing the average of the three CIF-size sequences, the 8-mode algorithm was 0.20 dB better than the proposed algorithm while the 1-mode algorithm is 0.34 dB worse than the  proposed algorithm. For the two HD-size sequences, the 8mode and 1-mode algorithms average 0.11 dB better and 0.26 dB worse, respectively, than the proposed algorithm. These results show that the proposed algorithm produces a reasonable trade-off between complexity and quality. Figure 13 shows the subjective quality comparison. As shown in the figure, the MHT-based FMC suffers from the blur around the numbers while the number blurring is significantly reduced by the proposed FMC.
Within the 60 frames of the Foreman sequence, the PSNR of each frame is shown in Figure 14. Three lines show the proposed FMC, the MHT-based FMC, and the original H.264 encoder with no FMC. An intraframe is inserted once in every 10 frames, and the peaks in the graph represent the intraframes. The MHT-based FMC significantly drops the PSNR for all frames while the proposed algorithm produces notably less quality degradation.
Since the frame compression is lossy, this raises the issue of drift, as there may be a mismatch between the encoded frame written in the compressed file, and the decoded frame stored in the memory and used later for the prediction of successive frames. The decrease of PSNR is observed in Figure 14 as the PSNR of a frame distant from an intraframe is less than that close to the intraframe not only with the proposed FMC but also with the H.264 encoder. This result shows that the drift by the propose FMC does not affect significantly the PSNR drop. In order to precisely measure the additional PSNR drop caused by the proposed FMC, the PSNR difference between the original H.264 encoder without the FMC and the integrated H.264 encoder with the FMC is shown in Figure 15. As shown in this figure, the PSNR difference does not vary significantly regardless of the distance from an intraframe. This result also shows that the additional PSNR drop caused by the proposed FMC is not very significant. This experiment is performed with various intervals of intraframe period, and the results are similar to that shown in Figure 15. Thus, the additional experimental results are not presented in this paper.
The eight scanning modes given in Figure 2 are employed based on an analog of the 4 × 4 intraprediction modes for an H.264 encoder. Among the eight scanning modes, the best mode is selected to minimize the DPCM error. For the selected scanning mode, the scan along the solid line is the major scanning direction whereas the scan along the dotted line is, in general, perpendicular to the major scanning direction. Therefore, the difference along the solid line is likely to be smaller than that along the dotted line. For example, consider the case when a 4 × 4 block includes a virtual stripe pattern so that scanning mode 0 is selected. In this case, the scan along the dotted line crosses the vertical stripe and the chance is very high that the difference along the dotted line is larger than that along the solid line. Therefore, the "source" along the dotted line is expected to have a large value.
The expectation is supported by experimental results given in Table 4. The numbers given in this table are the ratios of the average difference along the dotted line over that along the solid line. This table shows that the difference along the dotted line is about 153.4% of that along the solid line.
In an H.264 encoder, deblocking filter is the only module that stores the reference frame. Figure 16 shows a 16 × 16 macroblock (lightly shaded blocks) that is the current macroblock to be filtered. To perform deblock filtering, the 4 × 16 pixels (dark shaded blocks) above the current macroblock and 16 × 4 pixels in the left of the current macroblock are necessary. Note that the 4 × 16 pixels are already processed by the above macroblock and they are compressed before they are stored. Then, for the current macroblock, the above 4 × 16 pixels are read again from the reference memory and filtered and then written back again. Thus, these pixels are stored into reference memory twice. As they are compressed whenever they are stored into reference memory, they are compressed twice. The successive compressions increase the PSNR degradation.
One way to reduce the PSNR degradation is to store the data without compression for the first write of the 4 × 16 pixels. These 4×16 pixels are read again and then compressed in the second write. As the second write finally stores the reference frame which is to be used by the next frame, the goal of memory size reduction is achieved even though only the second write is compressed. Table 5 shows the BD-PSNR difference between the two approaches. The numbers in the table show the BD-PSNR drop (i.e., the difference in the BD-PSNR between the original H.264 encoder and the integrated H.264 encoder with the proposed FMC). The first column shows test video sequences and the second column shows the case when both the first and second writes compress the 4 × 16 data whereas the third column shows the BD-PSNR drop when only the second write by deblocking filter is compressed.      Figure 16: The pixels to be written twice for deblocking filter. By storing the first write without compression, an average of about 0.12 dB improvement is achieved for CIF-size videos and an average of 0.09 dB improvement for HD size videos. The H.264 Encoder has four modules that access the external memory. They are image sensor interface, video input module, motion estimation, and deblocking filter. The image sensor module receives pixel data from an image sensor in the YUV 4 : 2 : 0 format and stores it in the external memory. The video input module reads the input data to process H.264 encoding. The memory bandwidth to access the input data is as follows: where Wand H represents width and height of a frame. For the reference frame, both deblocking filter and motion estimation modules access the reference frame. The bandwidth required by deblocking filter is as follows: EURASIP Journal on Advances in Signal Processing

13
The memory bandwidth requirement by motion estimation depends on the search range, search algorithm and data reuse scheme. The H.264 encoder in this paper adopts the full search algorithm and level C data reuse scheme [27]. Thus, the required memory bandwidth is where SR H and SR V represent the horizontal and vertical search ranges, respectively, and f ref is the number of reference frame. The memory requirement for chrominance components by motion estimation is as follows [28]: Thus, the total memory requirement is BW total = BW current frame(store/load) +BW DB store +BW DB load +BW ME luma +BW ME chroma ) × FrameRate.
(8) Figure 17 shows the required memory bandwidth that depends on the frame size and search range. The frame rate is 30 frames per second and all frames are encoded as P-frame. The bar graphs given in Figure 17 show To support this memory bandwidth, the required operating frequency is Freq min = BW total memory bus bit width×memory bus utilization .
Assuming that memory bus bit width is 32 and the memory bus utilization is 100%, the line graphs show the required operating clock frequency of the external memory.  which is in the range of the normal operating frequency of an SDRAM. The reduction of memory traffic also makes decreases the power consumption.

FMC for the RGB Color
Space. This subsection presents the experimental results to evaluate the FMC algorithm for the RGB color space proposed in Section 3. To this end, twenty-three images of size 768 × 512 in the RGB color space shown in Figure 18 Figure 18: Test RGB bitmap images.  As the human eyes are more sensitive to the G color than the B color, it is reasonable to choose the GDbDr-based FMC rather than the JPEG2000 YCbCr-based FMC. Note that the average PSNRs for R, G, and B achieved by the GDbDrbased FMC are 46.70 dB, 50.80 dB, and 44.90 dB, respectively. As the PSNR is very large, the quality degradation is hardly observed. Among the twenty-three test images, three images with the lowest PSNR degradation are chosen and the images with the GDbDr-based FMC are compared with their original images. Figure 19 shows these images and it is very hard to distinguish the original image from the compressed image.
The FMC algorithm proposed in Section 3.6 is also evaluated with the twenty-three RGB images. Recall that this algorithm achieves 75% compression by combining the 50% FMC algorithm and another 50% compression by color transform and subsampling from RGB 4 : 4 : 4 format into YCbCr (or GDbDr) 4 : 2 : 0 format. Table 7 shows PSNR values when the images are compressed by 75%. For the transform into the standard YCbCr color space and the JPEG YCbCr color space are evaluated and compared with the GDbDr color space. As shown in Table 7, the FMC with GDbDr color space achieves the best quality for the G and R color component while the FMC with JPEG2000 YCbCr color space achieves the best quality for B color components. The FMC with the standard YCbCr color space does not outperform for any image. When compared with the image quality by the 50% FMC algorithm presented in Table 6, the image quality is significantly degraded as PSNR is much less than that in Table 6. The quality degradation is caused by large compression ratio. Figure 20 shows the efficiency of the combined packetization which is explained in Section 3.1. Figure 20(a) shows the average QPs for the combined and separate packetizations. Note that QP + 1 corresponds to the number of iteration in FMC encoding. To compare the efficiencies of Golomb-Rice coding and Exp-Golomb Coding, both are used for the compression and their results are compared. In this figure, GR and EG stand for Golomb-Rice and Exp-Golomb codings, respectively. The combined packetization reduces the average QP of G color, but it increases the average QP of Dr and Db colors. This implies that the degradation of G color is substantially reduced while those of R and B colors may increase. Figure 20(b) compares the PSNR of combined packetization with that of separated packetization. As shown in this figure, the PSNR of G colors as well as R and B colors increase when combined packetization is adopted. It is because the error reduction of the G color also affects the error reduction of R and B colors, consequently resulting in the improvement of PSNR for all three color components. It is also shown that Golomb-Rice coding is efficient for separate packetization while Exp-Golomb coding is efficient for combined packetization. This is shown in Figure 20(a) because the QP of Exp-Golomb coding for combined packetization is slightly less than that of Golomb-Rice coding while the QP is substantially increased by Exp-Golomb coding for separate packetization. In Figure 20(b), it is shown that Exp-Golomb coding achieves better PSNR than Golomb-Rice coding for combined packetization, but less PSNR for separate packetization.

Conclusions
This paper proposes an FMC algorithm that compresses video data to be stored into frame memory. The proposed FMC algorithm achieves lower image degradation than other transform based algorithms. The computational complexity is relatively small because the information given by H.264/AVC video encoding is used. By using the pixel correlation information, which comes from the H.264/AVC intraprediction, an efficient DPCM scan order is selected without a significant increase in the amount of computation. The proposed algorithm performs the compression in a 4 × 4 block, so that both horizontal and vertical correlations are exploited. As a result, higher compression efficiency is achieved than with an MHT-based algorithm, which exploits only the horizontal correlations. As a result, compared to an MHT-based algorithm, image quality is improved by an average of 1.62 dB and 1.34 dB for CIF-size and HD-size images, respectively. The proposed FMC algorithm is modified for the system without an H.264/AVC encoder. As the intraprediction result from H.264/AVC is not available, an additional step to select the best scan order is necessary. This system, in general, stores RGB colors instead of YCbCr colors as in H.264/AVC compression. For improved compression efficiency, the RGB color space is transformed into another color space and then compression algorithm is performed for the transformed domain. Experiments with various color spaces show that the most efficient result is obtained with the G, R-G, B-G color space.