A deblocking algorithm based on color psychology for display quality enhancement

Abstract

This article proposes a post-processing deblocking filter to reduce blocking effects. The proposed algorithm detects blocking effects by fusing the results of Sobel edge detector and wavelet-based edge detector. The filtering stage provides four filter modes to eliminate blocking effects at different color regions according to human color vision and color psychology analysis. Experimental results show that the proposed algorithm has better subjective and objective qualities for H.264/AVC reconstructed videos when compared to several existing methods.

Introduction

Many video coding techniques have been developed for various kinds of applications, including multimedia communication, videophone, videoconferencing, video streaming, and high-definition television [1–4]. The goal of video coding techniques is to reduce transmission bitrates and storage data while maintaining a certain quality. The Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG (Video Coding Experts Group) has developed a new standard for video coding. In December 2001, VCEG and MPEG (Moving Picture Experts Group) formed a JVT to finalize the draft of new coding standard for formal approval submission as H.264/AVC [5] in March 2003. H.264/AVC provides several enhancements to improve coding efficiency.

The discrete cosine transform (DCT) and motion estimation efficiently reduce spatial and temporal redundancies in most video coding standard. H.264/AVC divides an image into 4 × 4 or 8 × 8 block of pixels, and transforming each block from the spatial domain to the frequency domain by DCT. At low bit-rate coding, there may be only one DC and few AC coefficients to represent an image block. Hence, coarse quantization of transform coefficients cause noticeable discontinuities at block boundaries called blocking artifacts. In other words, blocking effects are the grid noise along block boundaries in a relatively homogeneous region.

Several previous studies [6–10] have been proposed for the design of deblocking filters, which can largely be categorized into three groups. The first group uses a low pass filter around block boundaries to remove blocking artifacts [6, 7]. The second group applies maximum a posterior probability technique to eliminate blocking artifacts [8]. The third group uses the set-theoretic reconstruction method, which defines constraint sets by using observed data or a prior knowledge of the solution [9, 10]. The intersection of all constraint sets gives the feasible set. All samples outside the feasible set are illegal and the projection onto convex sets [11] technique is used to project them to be legal. The success of set-theoretic method depends on the constraint sets highly. In [12], the classification algorithm classifies a block into three types: PLANE, EDGE, and TEXTURE, according to statistical characteristic of an image in the pixel domain. Each of these types has its corresponding filter to remove blocking artifacts. Similar to that in [12], the study in [13] classifies block type in the transform domain and proposes a wavelet-based block analysis. In [14, 15], some characteristics in human visual system are utilized to detect blocking artifacts; the appropriate filter is used to remove these artifacts.

In order to eliminate blocking effects, many deblocking filter algorithms have been proposed to provide nice solutions in still images [16–18]. The low-pass filter is also employed to eliminate blocking artifacts in video compression standards [19]. Tai et al. [20] proposed a deblocking algorithm based on five filtering modes by considering the activity across block boundaries. These five filtering modes include three frequency-related modes (the smooth mode, the intermediate mode, and the complex mode), one special mode, and a refined mode. Tai et al. [21] proposed a new method to remove blocking artifacts in the low bit-rate block-based video coding in H.264/AVC and MPEG-4. However, it is necessary to modify the filtered block size in order to suit different codecs. Then, Chen and her coauthors [22] proposed a general method which is suitable for various video compression standards. Nevertheless, when edges cross block boundaries, Chen’s method may blur real edge.

H.264/AVC in-loop filter performs simple operations to detect and analyze artifacts on coded blocking boundaries and attenuates those by applying a selected filter [23]. The H.264/AVC loop filter is an adaptive filter. The amount of filtering performed each position of the block edge depends on some factors. The filter can be divided into three levels: slice level, block level, and pixel level. The filter strength, i.e., the amount of filtering is computed based on the help of parameter boundary strength (BS), depending on the current quantizer, the macroblock type, the motion vector, gradient of the image samples across the blocking boundary, and other parameters. However, the filtering algorithm is highly complex and is account for 33% of the total decode time in the study of the baseline profile decoder [24].

This article proposes a deblocking filtering algorithm that eliminates the blocking artifacts from H.264/AVC encoded videos. The preliminary results are published in [25]. The theoretic analysis and experiments of this article are more solid. In the detection process, the proposed “Sobel-Wavelet” method can clearly reveal the positions of artifacts, and the proposed filter eliminates blocking effects in RGB domain, instead of YCbCr domain, considering the effects of human color vision and color psychology. In objective evaluation, the PSNR of the proposed method is approximated to the H.264/AVC loop filter. The proposed method has better performance in GBIM than H.264/AVC loop filter and other existing methods. In subjective evaluation, our proposed method has significant improvement when compared to other existing methods. Our method is also close to the H.264/AVC loop filter. In conclusion, the proposed method has the advantage of the post-processing-based method in terms of simplify and flexibility, but the performance is very close to H.264/AVC.

The rest of this article is organized as follows. Section “Blocking artifacts detection” describes the proposed blocking artifacts detection. Then, the proposed deblocking algorithm is explained in Section “Deblocking artifact algorithm”. Experimental results are shown in Section “Experimental results” in order to evaluate the performance of the proposed scheme. Finally, concluding remarks and some future work are given.

Blocking artifacts detection

Because most blocking artifacts occur at 4 × 4 or 8 × 8 block boundaries in H.264/AVC encoded video, the filtering should focus on pixels around these block boundaries. If filtering modifies all block boundaries, it will blur images. Hence, it is important to detect blocking boundaries and exclude real edges to avoid over-smoothing image textures and keep objective quality. The proposed blocking artifacts detection is based on Sobel edge detector and wavelet-based edge detector. According to our observation and analysis, Sobel edge detector detects all strong and coherent edges including real edges and blocking boundaries. Under noise condition, Sobel detector may fail to detect the real edges of an image. Wavelet-based edge detection has better detection especially in noise conditions, while Sobel equally detects both blocking artifacts and real edges. Combining the results of Sobel and Wavelet detectors together, the proposed “Sobel–Wavelet” method can clearly reveal artifacts among real edges avoid over-smoothing problem. Figure 1 illustrates the flowchart of our proposed detection scheme.

Figure 1

Flowchart of blocking artifacts detection.

Sobel edge detector

Sobel edge detector is a well-known tool of edge detection in image processing. 3 × 3 Sobel operator acts locally on an image and only detects edges at small scales. The Sobel edge detector is sensitive to high frequency; hence, Sobel can detect textures in an image. In general, Sobel edge detector uses two 3 × 3 kernels, H and V, to detect horizontal and vertical edges shown in Figure 2.

Figure 2

Sobel edge detector.

Two directional edge maps, horizontal and vertical edge information, can be determined by Equations (1) and (2), respectively.

$$S_h=H*F,S_h:\text{horizontal edge map,}$$

(1)

$$S_v=V*F,S_v:\text{vertical edge map,}$$

(2)

where F is an image with the size M × N, and * means convolution operation. When four values $S_{h1}$, $S_{h2}$, $S_{h3}$, and $S_{h4}$ are all greater than zero, these pixels belong to edges. And the same situation applies to $S_{v1}$, $S_{v2}$, $S_{v3}$, and $S_{v4}$. Otherwise, they just belong to textures in an image. Figure 3 illustrates the positions of four continuous pixels on the vertical and horizontal block boundaries. The decision function is given as

$$s_h\left(4m+i\text{,}4n\right)=\{\begin{array}{cc}\hfill 1,\hfill & \hfill S_{h1}\&S_{h2}\&S_{h3}\&S_{h4}>0\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\text{,}0\le i<4\text{,}$$

(3)

$$s_v\left(4m\text{,}4n+j\right)=\{\begin{array}{cc}\hfill 1,\hfill & \hfill S_{v1}\&S_{v2}\&S_{v3}\&S_{v4}>0\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\text{,}0\le j<4\text{,}$$

(4)

for $0\le m\le M/4-1$ and $0\le n\le N/4-1$, where $S_h=1$ and $S_v=1$ represent that the corresponding pixels belong to edge and smooth regions, respectively. In Sobel edge detector, both real edges and blocking artifacts are detected and shown in the edge map. In Figure 4, the black lines indicate all strong and coherent edges detected by Sobel edge detector. In Figure 5, we can see that the black line indicates the original horizontal direction edge of the image.

Figure 3

Edge map across 4 × 4 block boundaries.

Figure 4

Results of edge detection by Sobel operator for the first frame of TableTennis by H.264/AVC at 140 kbps. (a) Vertical direction. (b) Horizontal direction.

Figure 5

Details of the blue region. (a) TableTennis video sequence by H.264/AVC at 140 kbps and (b) horizontal direction edge detection.

Proposed wavelet-based edge detector

The proposed algorithm uses the well-known wavelet basis, Haar wavelet [26]. In the one-dimensional case, the Haar wavelet’s mother wavelet function $\psi \left(t\right)$ can be represented as

$$\psi \left(t\right)=\{\begin{array}{cc}\hfill \begin{array}{c}\hfill 1,\hfill \\ \hfill -1,\hfill \\ \hfill 0,\hfill \end{array}\hfill & \hfill \begin{array}{c}\hfill 0\le t\le 1/2\hfill \\ \hfill 1/2\le t\le 1\hfill \\ \hfill \text{otherwise}\hfill \end{array}\hfill \end{array}\text{,}$$

(5)

and its scaling function $\phi \left(t\right)$ can be defined as

$$\phi \left(t\right)\{\begin{array}{cc}\hfill 1,\hfill & \hfill 0\le t<1\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\text{.}$$

(6)

Two-scale wavelet decomposition with mother wavelet function $\psi \left(t\right)$ and scaling functions $\phi \left(t\right)$ generates four coefficients, $W_1^H$, $W_2^H$, $W_3^H$, and $W_4^H$. The positions of these four coefficients are shown in Figure 6. According to Equations (7) and (8), four wavelet coefficients are used to generate Equations (9) and (10). The blocking effects in an image have horizontal and vertical directional discontinuities. By multi-resolution analysis, cross-scale wavelet coefficients, which refers to the wavelet coefficients that have different scales but have the same orientation information such as $W_1^H$ and $W_2^H$, are employed to generate two directional maps as

$$E^H=W_1^H\times W_2^H\text{,}$$

(7)

$$E^V=W_1^V\times W_2^V\text{,}$$

(8)

Figure 6

Illustration of wavelet coefficient positions of $W_{\mathbf{1}}^{\mathbf{H}}$,$W_2^H$,$W_1^V$, and$W_2^V$.

Next, two thresholds, $T{h}_1$ and $T{h}_2$, the average edge strengths in horizontal and vertical directions of one frame are used for generating edge maps in Equations (11) and (12). If $E^H\left(m\text{,}n\right)$ is equal to or greater than $T{h}_1$, $e^H\left(m\text{,}n\right)$ belongs to an edge pixel; otherwise it belongs to blocking artifact one. In this part, the edge maps, $e^H$ and $e^V$, show real edges and suppress the appearance of blocking artifacts.

$$e^H\left(m\text{,}n\right)=\{\begin{array}{cc}\hfill 1,\hfill & \hfill if{E}^H\left(m\text{,}n\right)\ge T{h}_1\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\text{,}$$

(9)

$$e^V\left(m\text{,}n\right)=\{\begin{array}{cc}\hfill 1,\hfill & \hfill if{E}^V\left(m\text{,}n\right)\ge T{h}_2\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\text{,}$$

(10)

$$T{h}_1=\frac{4}{M\ast N}\left\{\sum _{m=0}^{\frac{M}{4}-1}\sum _{n=0}^{N-1}\left|F\left(4m\text{,}n\right)-F\left(4m-1\text{,}n\right)\right|\right\}\text{,}$$

(11)

$$T{h}_2=\frac{4}{M\ast N}\left\{\sum _{m=0}^{M-1}\sum _{n=0}^{\frac{N}{4}-1}\left|F\left(m,4n\right)-F\left(m,4n-1\right)\right|\right\}\text{.}$$

(12)

Sobel-wavelet detector

The proposed “Sobel–Wavelet” scheme combines the detected results of Sobel edge detector and Wavelet-based edge detector through the following equations:

$$L_h\left(m\text{,}n\right)=S_h\left(m\text{,}n\right)-e^H\left(m\text{,}n\right)\text{,}$$

(13)

$$L_v\left(m\text{,}n\right)=S_v\left(m\text{,}n\right)-e^V\left(m\text{,}n\right)\text{.}$$

(14)

If the pixel belongs both to the edge in the Sobel set and the true edge in the wavelet set, this pixel is determined as true edge. $L_h\left(m\text{,}n\right)=0$ and $L_v\left(m\text{,}n\right)=0$ mean that $F\left(m\text{,}n\right)$ belongs to true edge in horizontal and vertical directions, respectively. Figure 7a,b shows edge maps $S_h$ and $S_v$. The black lines in Figure 7c,d shows blocking effects. These two figures show blocking artifacts obtained by jointly considering the detected results of Wavelet-based edge detector that has high probability to exclude true edges. Figure 8 is the details of the red region of Figure 7.

Figure 7

Illustration of edge maps (the first reconstructed frame of Foreman by H.264/AVC at 105 kbps). (a)$S_h$, (b)$S_v$, (c)$L_h$, (d)$L_v$.

Figure 8

Details of the red region of Figure7. (a)$S_h$, (b)$S_v$, (c)$L_h$, (d)$L_v$.

Deblocking artifact algorithm

Our proposed blocking artifacts detection has high probability to find blocking boundaries and exclude real edges compared to the conventional schemes. In this section, a new deblocking filter is proposed to eliminate blocking artifacts detected by the proposed “Sobel–Wavelet” scheme. Blocking artifacts exists both in luminance and chrominance components; however, most of the conventional schemes only focus on luminance for deblocking process. Not only in luminance, the study in [19] removes the blocking artifacts both in luminance and chrominance components and obtains better results. The proposed algorithm removes blocking artifacts in the RGB domain instead of YCbCr; therefore, it can simultaneously eliminate blocking artifacts from two components. This is because three components of filtering RGB have effect on both luma and chrominance components of the YCbCr domain. Figure 9 presents the detailed flowchart of the proposed deblocking filter based on the above concept.

Figure 9

Flowchart of the proposed deblocking algorithm.

Some features from human vision [27] and color psychology [28] are used to design the proper filter to eliminate blocking effects. Four modes including complex mode, R mode, G mode, and B mode are used in our filtering stage. If the pixels belong to inverse colors across blocking boundary, this case is the complex mode. Note that inverse color is those on the opposite sides of the color wheel shown in Figure 10[29]. Otherwise, the pixel’s major color, which has the largest amplitude among R, G, and B components, is used to determine which mode it belongs to. For example, if the pixel’s major color is R component, we deem this case as R mode.

Figure 10

Illustration of color wheel [[26]].

Complex mode

The complex mode is shown in Equation (15). According to color psychology analysis, the pixels which across blocking boundary have different fundamental colors get more attention [28]. Hence, the proposed algorithm determines this case is the complex mode. Figure 11 shows the locations of the pixels and “c” and “d” across blocking boundary. In the complex mode, the proposed filter gives different filters depending on human vision system (Equation 15). Because human vision system has more attention in the complex mode, the proposed filter only update two pixels, “c” and “d” across the blocking boundary in the complex mode.

Figure 11

Locations of pixels “a”–“f” in the blocking boundary.

Since human beings have more sensibility on inverse colors than close colors based on color psychology, the number of updated pixels is less than other modes. In the complex mode, the deblocking algorithm is defined as

$$\left[\begin{array}{c}\hfill \begin{array}{c}\hfill r_c\hfill \\ \hfill r_d\hfill \\ \hfill g_c\hfill \end{array}\hfill \\ \hfill \begin{array}{c}\hfill g_d\hfill \\ \hfill b_c\hfill \\ \hfill b_d\hfill \end{array}\hfill \end{array}\right]=\left[\begin{array}{cccccc}\hfill 1-\frac{1}{32}\hfill & \hfill \frac{1}{32}\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 1+\frac{1}{32}\hfill & \hfill \frac{-1}{32}\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1-\frac{1}{16}\hfill & \hfill \frac{1}{16}\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1+\frac{1}{16}\hfill & \hfill \frac{-1}{16}\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1-\frac{1}{8}\hfill & \hfill \frac{1}{8}\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1+\frac{1}{8}\hfill & \hfill \frac{-1}{8}\hfill \end{array}\right]\left[\begin{array}{c}\hfill \begin{array}{c}\hfill R_c\hfill \\ \hfill R_d\hfill \\ \hfill G_c\hfill \end{array}\hfill \\ \hfill \begin{array}{c}\hfill G_d\hfill \\ \hfill B_c\hfill \\ \hfill B_d\hfill \end{array}\hfill \end{array}\right]\text{,}$$

(15)

where $r_c$, $r_d$, $g_c$, $g_d$, $b_c$, and $b_d$ are updated pixels, and the suffixes, c and d, are the location of pixels. R, G, and B represent the pixel’s R, G and B values, respectively. For most parts, blocking artifacts are caused by block-based transform coding. Coding errors in block boundaries are larger than that in the interior of the block [23]. This property is a good indicator to design the filter. Together with the BS, different filters are applied to each pixel position for objective and subjective quality improvement through a heuristic rule.

R, G, and B modes

Except the complex mode, we design other three modes, R, G, and B modes, according to color analysis. In the R mode, the position of the pixels is depicted in Figure 12. At the same time, we also need considering that pixel’s value in the R, G, and B components. Because this area belongs to red color, the R component needs to be filtered with more pixels. If $R_c$ and $R_d$ are not both less than 128, the proposed algorithm will determine that is a light color and the filter of the R component is shown Equation (16).

$$\left[\begin{array}{c}\hfill r_b\hfill \\ \hfill r_c\hfill \\ \hfill r_d\hfill \\ \hfill r_e\hfill \end{array}\right]=\left[\begin{array}{cccc}\hfill 1\hfill & \hfill \frac{-1}{4}\hfill & \hfill \frac{1}{4}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{1}{2}\hfill & \hfill \frac{1}{2}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{1}{2}\hfill & \hfill \frac{1}{2}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{1}{4}\hfill & \hfill \frac{-1}{4}\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{c}\hfill R_b\hfill \\ \hfill R_c\hfill \\ \hfill R_d\hfill \\ \hfill R_e\hfill \end{array}\right]\text{,}$$

(16)

Figure 12

Position of the pixels in R mode.

If $R_c$ and $R_d$ are both less than 128, the updated pixels are not only four pixels but also include the adjacent two pixels as

$$\left[\begin{array}{c}\hfill r_a\hfill \\ \hfill r_f\hfill \end{array}\right]=\left[\begin{array}{cccc}\hfill 1\hfill & \hfill \frac{-1}{8}\hfill & \hfill \frac{1}{8}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{1}{8}\hfill & \hfill \frac{-1}{8}\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{c}\hfill R_a\hfill \\ \hfill R_c\hfill \\ \hfill R_d\hfill \\ \hfill R_f\hfill \end{array}\right]\text{.}$$

(17)

In the G component, the filter will update less number of pixels than the R component. If $G_c$ and $G_d$ are not both less than 128, the filter is defined as

$$\left[\begin{array}{c}\hfill g_c\hfill \\ \hfill g_d\hfill \end{array}\right]=\left[\begin{array}{cc}\hfill \frac{1}{2}\hfill & \hfill \frac{1}{2}\hfill \\ \hfill \frac{1}{2}\hfill & \hfill \frac{1}{2}\hfill \end{array}\right]\left[\begin{array}{c}\hfill G_c\hfill \\ \hfill G_d\hfill \end{array}\right]\text{.}$$

(18)

If $G_c$ and $G_d$ are both less than 128, the updated pixels not only two pixels but also include the adjacent two pixels as

$$\left[\begin{array}{c}\hfill g_b\hfill \\ \hfill g_e\hfill \end{array}\right]=\left[\begin{array}{cccc}\hfill 1\hfill & \hfill \frac{-1}{4}\hfill & \hfill \frac{1}{4}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{1}{4}\hfill & \hfill \frac{-1}{4}\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{c}\hfill G_b\hfill \\ \hfill G_c\hfill \\ \hfill G_d\hfill \\ \hfill G_e\hfill \end{array}\right]\text{.}$$

(19)

The number of updated pixels in the B component and that in the G component are the same. If $B_c$ and $B_d$ are not both less than 128, the filter is defined as

$$\left[\begin{array}{c}\hfill b_c\hfill \\ \hfill b_d\hfill \end{array}\right]=\left[\begin{array}{cc}\hfill \frac{1}{2}\hfill & \hfill \frac{1}{2}\hfill \\ \hfill \frac{1}{2}\hfill & \hfill \frac{1}{2}\hfill \end{array}\right]\left[\begin{array}{c}\hfill B_c\hfill \\ \hfill B_d\hfill \end{array}\right]\text{.}$$

(20)

If $B_c$ and $B_d$ are both less than 128, the updated pixels not only two pixels include the adjacent two pixels as

$$\left[\begin{array}{c}\hfill b_b\hfill \\ \hfill b_e\hfill \end{array}\right]=\left[\begin{array}{cccc}\hfill 1\hfill & \hfill \frac{-1}{4}\hfill & \hfill \frac{1}{4}\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill \frac{1}{4}\hfill & \hfill \frac{-1}{4}\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{c}\hfill B_b\hfill \\ \hfill B_c\hfill \\ \hfill B_d\hfill \\ \hfill B_e\hfill \end{array}\right]\text{.}$$

(21)

The deblocking process of the G mode and B mode are very similar to R mode shown in Figures 13 and 14. In summary, the number of updated pixels of the certain component is more than other two components. Moreover, the proposed algorithm further designs the luminance filter to avoid over-smooth the image texture for enhancing subjective quality. Here, Y is the luminance component. The designed filter is shown in the following equation:

$$Y=\frac{y_{\text{block}}+y}{2}\text{,}$$

(22)

where $y_{\text{block}}$ represents the luminance component of the image without deblocking filter, and y means that the luminance component of the image processed by the proposed deblocking filter. When the difference between $y_{\text{block}}$ and y is greater than the predefined threshold, the filtered values are recalculated.

Figure 13

Position of the pixels in G mode.

Figure 14

Position of the pixels in B mode.

Experimental results

Several H.264/AVC-coded sequences, including Foreman, HallMonitor, Mother&Daughter, and TableTennis with different bitrates, are used in our experiments. The parameters setting in the reference software JM 12.3 high profile are shown in Table 1.

Table 1 Parameters setting in JM 12.3

Subjective quality comparison

We compare the subjective quality of Tai’s [21] algorithm, Chen’s algorithm [22], and H.264/AVC loop filter [23] with that of the proposed algorithm. Tai’s algorithm is designed for blocking effects occurred at 8 × 8 bock boundaries, but blocking effects may occur at 4 × 4 block boundaries in H.264/AVC. Hence, this algorithm should be modified to be able to process 4 × 4 block boundaries. The mode decision step and the filter step on each mode of the algorithm are not modified in the simulation. Figures in this section show the original frame (a), non-filtered frame (b), filtered by Tai’s [21] 4 × 4 filter (c), filtered by Chen’s [22] filter (d), filtered by H.264/AVC loop filter [23] (e), and filtered by proposed filter (f). The filtered results are shown in Figures 15, 16, 17, and 18. Furthermore, the details of the blue region of Figures 15, 16, 17, and 18 are shown in Figures 19, 20, 21, and 22. As can be seen in Figure 19c,d Tai’s algorithm does not filter the chrominance and Chen’s method filters the chrominance, but the blocking effects still occur. In Foreman’s eyes, the proposed deblocking filter can filter blocking boundaries based on human vision system and color psychology. Our method determines that Foreman’s eyes are a complex area; therefore, this region is not over-smoothed. In Figure 20c,d Tai’s and Chen’s methods cannot differentiate between blocking artifacts and true edges in an image. Hence, they may blur the true edges and decrease the subjective quality. The proposed method has the better subjective quality than Tai’s and Chen’s methods. Similar results can be seen in Figures 17, 18, 21, and 22.

Figure 15

Deblocking results of Foreman encoded by H.264 (CIF, Frame #1, target bit-rate: 105 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 16

Deblocking results of HallMonitor sequence encoded by H.264 (CIF, Frame #1, target bit-rate: 58 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 17

Deblocking results of Mother&Daughter encoded by H.264 (CIF, Frame #1, target bit-rate: 36 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 18

Deblocking results of TableTennis encoded by H.264 (CIF, Frame #1, target bit-rate: 140 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 19

Details of the blue region in Foreman (CIF, Frame #1, target bit-rate: 105 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 20

Details of the blue region in HallMonitor (CIF, Frame #1, target bit-rate: 58 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 21

Details of the blue region in Mother&Daughter (CIF, Frame #1, target bit-rate: 36 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Figure 22

Details of the blue region in TableTennis (CIF, Frame #1, target bit-rate: 140 kbps). (a) The original frame, (b) non-filtered frame, (c) filtered by Tai’s [18] 4 × 4 filter, (d) filtered by Chen’s [19] filter, (e) filtered by H.264/AVC loop filter [20], and (f) filtered by the proposed algorithm.

Objective quality comparison

Four measurements, ${\text{PSNR}}_{\text{Total}}$, ${\text{PSNR}}_{\text{Y}}$, the generalized block-edge impairment metric (GBIM) [30], and structural similarity index (SSIM) are used to evaluate objective quality. Here, ${\text{PSNR}}_{\text{Total}}$ is also used in objective quality comparison due to the design of the chrominance filter is considered in Chen’s method, loop filter [23], and the proposed algorithm. The function of ${\text{PSNR}}_{\text{Total}}$ is shown as

$${\text{PSNR}}_{\text{Total}}=\frac{4\times {\text{PSNR}}_{\text{Y}}+{\text{PSNR}}_{\text{Cb}}+{\text{PSNR}}_{\text{Cr}}}{6}\text{,}$$

(23)

In this equation, the weight of ${\text{PSNR}}_{\text{Y}}$ is 4, because the sampling pattern of our tested sequences is 4:2:0. A GBIM measures the horizontal and vertical differences, ${\text{M}}_{\text{h\_GBIM}}$ and ${\text{M}}_{\text{v\_GBIM}}$, between the columns and rows at all block boundaries of the reconstructed frame. GBIM is done by averaging ${\text{M}}_{\text{h\_GBIM}}$ and ${\text{M}}_{\text{v\_GBIM}}$,

$$\text{GBIM}=\frac{{\text{M}}_{\text{hG BIM}}+{\text{M}}_{\text{vG BIM}}}{2}\text{.}$$

(24)

The major advantage of GBIM is that it calculates the blockiness of a frame without the original frame. The purpose of this measurement is to detail the state of frame blockiness. The higher the GBIM value is, the greater the severity of the blocking effects become. We also use SSIM as one of objective quality assessment metrics. SSIM is based on the study by Wang et al. [31]. The main difference of SSIM compared to PSNR is that it does not estimate perceived errors to quantify image degradations but considers them as perceived changes in structural information variation. To evaluate the deblocking results, considering all PSNR, GBIM and SSIM seems better than only one of them.

Four different sequences are used to observe their objective qualities, including Foreman, HallMonitor, Mother&Daughter, and TableTennis with 100 frames in different bitrates: 105 k and 84 k, 58 k and 45 k, 36 k and 27 k, 140 k and 108 k, respectively. The results of ${\text{PSNR}}_{\text{Total}}$ and GBIM are shown in Tables 2, 3, and 4. The larger ${\text{PSNR}}_{\text{Total}}$ value means the better image quality, but the GBIM value is as smaller as better. In Tables 2, 3, 4, and 5, the ${\text{PSNR}}_{\text{Total}}$ of the proposed algorithm is higher than Tai’s algorithm by 0.03 to 0.3 dB. Because Tai’s algorithm does not filter the chrominance component, there are still some blocking artifacts in the Cb and Cr components. The ${\text{PSNR}}_{\text{Total}}$ of the proposed algorithm is higher than Chen’s algorithm by 0.01 to 0.03 dB. Because Chen’s algorithm may filter true edges at block boundaries, which blurs true edges and decreases the visual quality. The proposed algorithm’s ${\text{PSNR}}_{\text{Total}}$ is lower than that of H.264/AVC loop filter. The reason is that the H.264/AVC loop filter uses the system information of the codec, so it can find out blocking effects more accurately than post-processing-based method. For GBIM results, the proposed algorithm is better than other algorithms. Especially, the GBIM of the proposed scheme is much smaller than that of H.264/AVC loop filter, Chen’s filter and Tai’s filter. This is because the proposed algorithm filtered with luminance and chrominance components, and we also refine Y component to avoid over-smoothing original texture. In summary, the proposed method has better performance in terms of subjective and objective qualities when compared to other well-known methods, Chen’s filter and Tai’s filter. The proposed method has very close objective quality to H.264/AVC loop filter but the better GBIM. For SSIM and ${\text{PSNR}}_{\text{Y}}$ comparisons, the H.264/AVC loop filter still has the best performance among all algorithms because it is in-loop filter and has complete coding information. In SSIM, other post-processing-based algorithms’ performances are very similar. In ${\text{PSNR}}_{\text{Y}}$, all the outcomes of our proposed method are better than that of non-filter while some outcomes of Tai’s and Chen’s are worse than that of non-filter.

Table 2 Objective quality comparison for “ Foreman ”

Table 3 Objective quality comparison for “ HallMonitor ”

Table 4 Objective quality comparison for “ Mother&Daughter ”

Table 5 Objective quality comparison for “ TableTennis ”

Evaluation of the proposed "Sobel-Wavelet" edge detector

Table 6 shows the efficiency of the proposed Sobel–Wavelet detector. The test frame is the first frame of Foreman sequence with the CIF resolution at 105 k bitrate. The ground truth of real edge and artifact for the test frame is determined by human directly. The detection results of wavelet-based edge detector for real edge and blocking artifact are 97.18 and 2.82%, respectively. Wavelet-based edge detector is robust for blocking artifacts to reveal real edge. Sobel edge detector equally detects real edge and blocking artifact. The proposed Sobel–Wavelet detector can clearly reveal blocking artifacts and detection accuracy reaches up to 99.1%.

Table 6 Efficiency of proposed Sobel–Wavelet detector

Computational complexity comparison

The comparisons of the computational complexity and the processing time are listed in Tables 7 and 8. The computational operators are addition, shift, comparison, and division. Table 8 shows the processing time comparisons of H.264/AVC encoded CIF Foreman sequence at 105 k. Experimental results show that our proposed algorithm has the similar complexity to the Chen’s filter [22], but higher than the Tai’s 4 × 4 filter [21]. The proposed algorithm filters in the RGB domain and Chen’s method filters in both Y and CbCr components, but the Tai’s 4 × 4 filter [21] just aims at Y component. For this reason, the computational complexity is a little higher than the Tai’s 4 × 4 filter [21], similar to Chen’s filter [22], and lower than the H.264/AVC loop filter [23].

Table 7 Computational complexity comparison

Table 8 Processing time comparison

Conclusions

This article proposes a deblocking algorithm to remove blocking artifacts for enhancement of the subjective and objective qualities of H.264/AVC encoded videos. The proposed detection algorithm differentiates between true edges and blocking artifacts to avoid over-smoothing the original texture in an image. Furthermore, the proposed deblocking filter determines the filtering stage according to color psychology analysis and gives different filters based on human vision system. As for objective quality, the proposed algorithm has higher PSNR than Tai’s and Chen’s methods; though the PSNR it is a little bit lower H.264/AVC loop filter, it has better GBIM. In subjective quality, the proposed algorithm is better than Tai’s and Chen’s methods and is close to H.264/AVC loop filter. Experimental results demonstrate the effectiveness of our proposal method. To make the proposed method more feasible, the computational complexity reduction is the major direction of our future work.