A concatenation scheme of LDPC codes and source codes for flash memories

Abstract

Recently, low-density parity-check (LDPC) codes have been applied in flash memories to correct errors. However, as verified in this article, their performance degrades rapidly as the number of stuck cells increases. Thus, this paper presents a concatenation reliability scheme of LDPC codes and source codes, which aims to improve the performance of LDPC codes for flash memories with stuck cells. In this scheme, the locations of stuck cells is recorded by source codes in the write process such that erasures rather than wrong log-likelihood ratios on these cells are given in the read process. Then, LDPC codes correct these erasures and soft errors caused by cell-to-cell interferences. The analyses of channel capacity and compression rates of source codes with side information show that the memory cost of the proposed scheme is moderately low. Simulation results verify that the proposed scheme outperforms the traditional scheme with only LDPC codes.

Introduction

In flash memories, as programming and erasure (P/E) cycles increase, some cells may be stuck at a voltage level not related to the data being written in, resulting in stuck cells or defective memory cells [1, 2]. At present, a class of hard-decision decoding error correction codes (ECC), Bose Chaudhuri Hocquenghem (BCH) codes, are employed to correct such stuck errors caused by stuck cells as well as soft errors [3, 4], induced by cell-to-cell interferences. Moreover, if the number of stuck cells in one page exceeds a certain limit, the whole page will be discarded and masked.

As scaling is becoming more and more aggressive, reliability is also becoming one of the most important challenges in multi-level cell (MLC) flash memories now and future. Therefore, a class of more powerful soft-decision decoding ECCs, low-density parity-check (LDPC) codes, [5–7], are introduced to improve the reliability of flash memories [3, 4, 8]. Different from hard-decision decoders, stuck errors are very harmful to soft-decision decoders, since their log-likelihood ratios (LLRs) could have a large magnitude such that soft-decision decoders may mistakenly consider them correct with very high reliability. From our simulation results, even a small number of these stuck errors may trap soft-decision message-passing decoders of LDPC codes. It causes severe performance degradation, such as worse waterfall thresholds and slower slope of bit error rate (BER) curves.

In this article, we present a concatenation reliability scheme of LDPC codes and source codes to alleviate the performance degradation. The key idea is to erase the wrong LLRs of stuck cells for message-passing decoders according to their locations provided by source codes. The scheme consists of two parts, namely the write and the read parts. When the controller is writing data into flash memories, it is also designed to detect stuck cells. The locations of these stuck cells are compressed by source codes with side information to efficiently reduce the memory cost. In the read part, the compressed location information is recovered and the wrong LLRs related to these locations are erased, i.e., set their LLRs to zero. Then, LDPC decoders correct both these erasures and soft errors.

We also compare the proposed concatenation scheme with the traditional one from the angle of error performance and memory cost. On one hand, it can significantly outperforms the traditional one, since it removes the wrong LLRs for LDPC decoders. On the other hand, capacity and compression rate analyses indicate that it moderately requires a small percentage of memory cost. Simulation results show that it achieves significantly better error performance than the traditional one, even if the traditional one uses LDPC codes with lower rate. Because the proposed scheme can tolerate more stuck errors, it is also able to prolong the life of flash memories, besides reliability.

The rest of the article consists of four sections. In “Introduction” section, we introduce some backgrounds on channel models of flash memories and LDPC codes. The proposed concatenation scheme is presented in “Concatenation scheme of LDPC codes and source codes” section. Analyses and simulation are given in “Analyses and simulation” section. This article is concluded in the last section.

Background

Channel models for different types errors

In flash memories, each cell uses ^2svoltage levels $L_1,L_2,\dots ,L_{2^s}$ to represent s stored bits. If s=1, they are single-level cell flash memories; if s>1, they are ^2s MLC flash memories. The writing processing, i.e., voltage level programming, is realized by incremental step pulse program with an incremental program step voltage Δ _Vpp [4]. Under the ideal condition, the distributions of voltage levels except the first one are uniform distributions with the width of Δ _{V pp} . The distribution of the first level is a wide Gaussian-like distribution. Since these distributions do not overlap each other, no errors occur. However, because of inevitable cell-to-cell interferences in real systems, all the voltage distributions change to Gaussian-like distributions and may overlap (Figure 1). Thus, soft errors may occur. Cell-to-cell interferences [3, 4] can be modeled as an additive white Gaussian noise (AWGN) channel with pulse-amplitude modulation (PAM). In other words, the voltage level _Li has the output

$$R_j=L_i+N_i$$

(1)

for the j th flash cell through the AWGN channel, where _Ni is the additive Gaussian noise with zero mean and variance ${\sigma }_i^2$.

Figure 1

Voltage distributions with cell-to-cell interferences. Each level counts 22k cells.

In addition, since P/E cycles gradually degrade flash memory medium, some cells may permanently be stuck with one voltage level [1, 2], named stuck errors or stuck-at-faults. In this article, we mainly consider cells stuck at the highest level which is a common type of stuck errors after many P/E cycles. Suppose that there are u% stuck cells in the flash memory. Such errors can be described by a stuck channel with a transition probability p = 0.01 u as shown in Figure 2. Each level _Li has the same probability p to be stuck at the highest level $L_{2^s}$, noted _{⊖ p}{ _Li }.

Figure 2

Model of stuck channel in flash memories.

LDPC codes

A binary regular (n, k) LDPC code [6] is defined as the null space of an m × n sparse parity-check matrix H, m  ≥ n − k, over GF(2) with the following structural properties: (1) each row has constant weight ρ; (2) each column has constant weight γ; and (3) no two rows (or two columns) can have more than one position where they both have 1-components. H is said to be (γ ρ)-regular and the code given by the null space of H is called a (γ ρ)-regular LDPC code. Property (3) is referred to as the row-column (RC) constraint. Most LDPC codes investigated in flash memories are regular due to high code rate and low error-floor requirements.

Consider a binary message vector m of length k and an (n, k) binary LDPC code. Its corresponding codeword c of length n is m · G and satisfies c · H^T = 0, where G is the generator matrix of the LDPC code.

Soft-decision decoding algorithms for LDPC codes are mostly iterative in nature and devised based on message-passing. The most well known ones [6] are the sum-product algorithm and its simplified version, the min-sum (MS) algorithm [9].

Concatenation scheme of LDPC codes and source codes

Channel model for both soft errors and stuck errors

Two types of errors in flash memories are introduced in “Channel models for different types errors” section. Soft errors caused by cell-to-cell interferences are described by an AWGN channel. Stuck errors caused by stuck cells are described by a stuck sub-channel. In existing researches, the two types of errors from the two channels are separately considered for correction [3, 4, 10]. However, both of them exist in real systems at the same time, especially at middle-late stage of memory life. Thus, we consider a channel model consisting of two sub-channels, the AWGN sub-channel for soft errors and the stuck channel for stuck errors. Furthermore, it can be time-varying according to P/E cycles thus more realistic. As a result, the reliability scheme based on the proposed channel model will be more efficient in applications and make it possible to extend the life of memory cells.

The block diagram of the channel model is shown in Figure 3. Consider ^2s level flash memories and a binary data vector c of length n = s^n′as the input. By Gray mapping, it is stored into a voltage level vector V of length of ^n′. After the first stuck sub-channel, the vector of V becomes

$$\mathbf{S}={\ominus }_p\left\{\mathbf{V}\right\},$$

(2)

where _Sj = _{⊖ p}{ _Vj } for the j th cell, j = 0,1,…,^n′− 1. If _Sj is not stuck, it will pass the second AWGN sub-channel and the voltage level _Sj will change to _Rj = _Li + _Ni ; otherwise, _Rj = _Li .

Figure 3

Channel model composed of AWGN sub-channel and stuck sub-channel.

The proposed concatenation scheme

Until now only ECCs, e.g., BCH codes or LDPC codes, are used to correct errors in practice, since the AWGN sub-channel dominants the performance in the early stage (i.e., few P/E cycles) of flash memories. However, as the number of P/E cycles increases, the stuck sub-channel will bring significant effects, especially harmful to probability decoders like message-passing decoders of LDPC codes, and will finally kill the memories.

If we only consider the reading process in ECC schemes, the controller can only receive voltage levels R which may be mixed with stuck errors. From these voltage levels, LLRs are calculated based on the AWGN sub-channel with PAM modulation. Since these LLRs are mixed with wrong magnitudes and signs, even the maximum likelihood codeword based on these LLRs may not be correct. In “Analyses and simulation” section, we will show that a small number of stuck cells will cause significant performance degradation.

However, if we also consider both the writing process, the decoder will be able to distinguish normal cells and stuck cells by the information provided by the controller.

As described in [4], MLC flash memories are written by a program and verify scheme. If some cells are stuck, their voltages keep unchanged such that they can be detected by a well-designed controller. As shown in Figure 4, we propose that the controller detects stuck cells and then records their locations into a vector l of length ^n′, in the writing process, where _lj = 1 if j th cell is stuck; otherwise _lj = 0. To reduce the memory cost, the location vector l is compressed into a vector z. In the next section, we will present the compression approach and its memory cost.

Figure 4

The write part of the reliability concatenation scheme of LDPC codes and sources codes.

In the reading process, as shown in Figure 5, the controller receives voltage levels R from the current page and the compressed vector from the compressed location page. The vector r of LLRs [11] is calculated based on the AWGN sub-channel with PAM modulation from R. After the location information l is recovered from the compressed vector z and the side information v[12] (see “Memory cost of source codes” section), the LLRs on the stuck locations are set to zero, i.e., _{rsj + t}= 0 if _lt = 0, where t = 0,1,…,s − 1. Then, the LLRs are sent to LDPC decoders to output the decoded data.

Figure 5

The read part of the concatenation reliability scheme of LDPC codes and source codes.

Analyses and simulation

In this section, we first would like to analyze the capacity of the flash channel models. Then, based on the analyses of channel capacity and compression rates of source codes, the memory cost of source codes and the overall code rate of the proposed scheme are presented. Last, simulation results verify that the proposed scheme achieves better performance than the traditional one, even if the traditional one deploys lower rate LDPC codes.

Capacity analysis

It is interesting to know the capacity C of the proposed channel model. First, it is clear that the channel consisting of two sub-channels is a Markov Chain. Thus, its capacity is upper bounded by the capacity of the AWGN sub-channel _Ca and the capacity of the stuck sub-channel _Cs [13],

$$C\le \text{min}\{C_a,C_s\}.$$

(3)

Consider 4-level flash memories, i.e., ^{2 s}= 4. The capacity of the stuck sub-channel _Cs is

$$C_s=(1-p)\underset{2}{\text{log}}(p^{-p}+3(1-p\left)\right)+p\underset{2}{\text{log}}\frac{p^{(1-p)}+3p(1-p)}{p^{(1-p)}+(1-p)p^{-p}}.$$

(4)

The derivation of _Cs is given in the Appendix.

By the proposed scheme, stuck errors are transferred to erasures. Thus, the channel can be viewed as the concatenation of the AWGN sub-channel and the ^{2 s}-ary erasure sub-channel (Figure 6). It is well known that the capacity of the erasure sub-channel _Ce with erasure probability p is s (1 − p) [13]. If s = 2, then the capacity of the erasure sub-channel is

$$C_e=2(1-p).$$

(5)

Figure 6

Channel model with source codes with side information.

In the next section, the capacity of the stuck sub-channel (4) and the erasure sub-channel (5) is used to derive the memory cost of source codes.

Memory cost of source codes

In this article, we propose to deploy lossless source codes to compressed the location vector l into the compressed vector z. Moreover, as shown in Figure 5, side information [12, 14] from voltage levels is used to further reduce the memory cost of compression.

It is well known in information theory that compression rates of lossless source codes can efficiently approach [15] the entropy of the original data. For the stuck location vector, its entropy is

$$n^{\prime }H\left(\mathbf{l}\right)=n^{\prime }H\left(l_i\right)=n^{\prime }(-p\underset{2}{\text{log}}p-(1-p\left)\underset{2}{\text{log}}p\right).$$

(6)

In addition, according to Slepian-Wolf theorem [12], the theoretical limit of lossless compression of vector l is improved to the conditional entropy H (l|v), with the help of side information v, where l and v are correlated with Pr( _li ≠ _vi ) < 0.5. Since stuck cells stay in the highest level as shown in Figure 2, the voltage level vector R from data pages provides side information to improve the efficiency of source codes. In other words, if the ${\widehat{R}}_j\ne L_{2^s}$, the j th cell is not stuck, i.e., _lj = 0, where ${\widehat{R}}_j$ is the hard-decision of _Rj [11]. Let us describe the side information by the vector $\mathbf{v}=[v_0,v_1,\dots ,v_{n^{\prime }-1}]$, where _vi = 0 if ${\widehat{R}}_j\ne L_{2^s}$; else _vi = 1. Then, the cell is stuck, i.e., _lj = 1 only if _vi = 1. Suppose that the data bit is discrete uniform distribution in ‘0’ and ‘1’. Recall that the transition probability is p. The probability of stuck errors is p (^{2 s}−1)/^2s, since there is a probability 1/^2s that stuck cells are coincidentally written into $L_{2^s}$ such that no errors occur in these cells. Then, the probability that _vi = 1 is $p+(1-p)/2^s=\frac{1+p(2^s-1)}{2^s}$ and the conditional probabilities are Pr ( _li = 0| _vi = 0) = 1, Pr ( _li = 1| _vi = 0) = 0, $\mathrm{Pr}(l_i=0|v_i=1)=\frac{1}{1+p(2^s-1)}$, and $\mathrm{Pr}(l_i=1|v_i=1)=\frac{p(2^s-1)}{1+p(2^s-1)}$. Thus, the conditional entropy H (l|v)

$$\begin{array}{l}\text{lllH}\left(\mathbf{l}\right|\mathbf{v})=n^{\prime }H(l_i\left|v_i\right)\\ =n^{\prime }\left\{\frac{1+p(2^s-1)}{2^s}\left[\mathrm{Pr}(l_i=1|v_i=1)\underset{2}{\text{log}}\frac{1}{\mathrm{Pr}(l_i=1|v_i=1)}\right.\right.\\ \left.\left.+\mathrm{Pr}(l_i=0|v_i=1)\underset{2}{\text{log}}\frac{1}{\mathrm{Pr}(l_i=0|v_i=1)}\right]\right\},\\ =-\frac{n^{\prime }(1+p(2^s-1\left)\right)}{2^s}\left[\frac{p(2^s-1)}{1+p(2^s-1)}\underset{2}{\text{log}}\frac{p(2^s-1)}{1+p(2^s-1)}\right.\\ \left.+\frac{1}{1+p(2^s-1)}\underset{2}{\text{log}}\frac{1}{1+p(2^s-1)}\right].\end{array}$$

(7)

Thus, the compression rate _Rsc of source codes is about

$$R_{\mathrm{sc}}\ge \frac{H\left(\mathbf{l}\right|\mathbf{v})}{s{n}^{\prime }}.$$

(8)

Suppose that the code rate of the LDPC code is _Rcc , then the overall code rate of the concatenation scheme is

$$R_{\mathrm{or}}=R_{\mathrm{cc}}/(1+R_{\mathrm{sc}}).$$

(9)

The stuck location changes much slower, since stuck cells occur only after many P/E cycles on data pages. In other words, P/E cycles happen much less often in the compressed location pages than data pages. Thus, we could consider that compressed location pages contain few errors before the flash memories die. A high rate and short ECC is strong enough to guarantee the reliability, which has a small impact on the overall code rate. From brevity, we does not involve the error correcting issue of compressed location pages in this article.

It is clear that the proposed scheme takes about H (l|v) bits to record the stuck locations. On the other hand, the stuck sub-channel is replaced by the erasure sub-channel, which has larger channel capacity. Recall the capacity of the erasure sub-channel _Cs (4) and the capacity of the erasure sub-channel (5). We define the memory cost of source codes as

$$C_{\mathrm{mc}}=1-\frac{C_e/(1+R_{\mathrm{sc}})}{C_s}=1-\frac{C_e}{C_s(1+R_{\mathrm{sc}})}.$$

(10)

In Figure 7, we show that the memory cost is moderately low and grows almost linearly with the transition probability p when p is less than 0.02. It is also worth to mention that the combination of the BEC sub-channel and the AWGN channel is inherent better the combination of the stuck sub-channel and the AWGN channel for the soft-decision decoders, since the harmful wrong LLRs have been removed with the help of source codes. As we will show in the next section, the proposed scheme outperforms the traditional ECC only scheme.

Figure 7

Memory cost of the concatenation scheme.

Simulation results

In this section, we will show the performance degradation of LDPC codes caused by stuck cells. Then, the proposed scheme is compared with the traditional scheme with only one ECC.

In our simulation, we consider 2-bit (4-level) flash memories, i.e., s=2. Gray mapping is used to map 2 bits of data to a voltage level _Li , where _{L 1}, _{L 2}, _{L 3}, and _{L 4} represent ‘00’, ‘01’, ‘11’, and ‘10’, respectively. Consider cell-to-cell interferences as the AWGN sub-channel, their means are all zero and their variances ${\sigma }_i^2$ are 4^{σ 2}, ^{σ 2}, ^{σ 2}, and 2^{σ 2}, where ^{σ 2} is measured by

$$V_{{\sigma }^2}\triangleq 10\underset{10}{\text{log}}\frac{{(L_4-L_1)}^2}{{\sigma }^2}.$$

Clearly, the larger $V_{{\sigma }^2}$, the less cell-to-cell interferences. A similar definition can be found in [16]. Moreover, two percentages, 0.5 and 1%, of stuck cells are investigated. From (8), the compression rates of source codes for 0.5 and 1% are about 0.0161 and 0.0245, respectively.

In the proposed scheme, a (16000,15000) LDPC code with constant column weight 4 and code rate 15/16 is used. The source code costs about 0.0141 × 16000 ≈ 226 bits to record the location vector l if p = 0.005 and 0.0245 × 16000≈ 392 bits if p = 0.01. Thus, their overall rates (9) are 0.924 if p = 0.05 and 0.915 if p = 0.01. To be fair, we add the same overhead into the traditional scheme such that a (16226,15000) LDPC code with constant column weight 4 and a (16392,15000) LDPC code with constant column weight 4 are used in the traditional scheme for p = 0.005 and p = 0.01, respectively. All the LDPC codes are decoded by the MS algorithm [9] with 50 iterations.

The simulation results are shown in Figure 8. First, we show the severe performance degradation caused by a small number of stuck errors. Consider the traditional scheme, if there are 0.5% of stuck cells in the flash memory, the performance degradation between no stuck cells and 0.5% stuck cells is about 1.8 dB at BER 1^{0 − 6}. Moreover, the slope of the BER curve with stuck cells obviously reduces. If the percentage of stuck cells increases to 1%, the performance degradation is 2.5 dB at BER 1⁰⁻³. In addition, the BER curve with stuck cells becomes flat. Second, we compare the traditional scheme with the concatenation scheme. If there are 0.5% stuck cells, the concatenation scheme provides about 1.3 dB at BER 1^{0 − 6}over the traditional scheme with the (16226,15000) LPDC code. Moreover, the proposed scheme outperforms the traditional scheme with the (16500,15000) LPDC code 0.4 dB at BER 1^{0 − 6}, whose code rate 0.909 is lower than 0.924. If the percentage increases to 1%, the performance gain is much larger, about 1.7 dB at BER 1⁰⁻³. It is worth mentioning that though the concatenation scheme suffers from stuck cells, the slopes of its BER curves are as well as LDPC codes without stuck errors.

Figure 8

BER of different schemes with different percentages of stuck cells.

Conclusion

Cell-to-cell interferences and stuck cells are the two most important reliability issues in flash memories. In this article, we present a concatenation reliability scheme of LDPC codes and source codes with side information to enhance the reliability. The proposed scheme erases stuck cells according to their recorded stuck locations such that the performance of soft-decision decoding algorithms for LDPC codes is significantly improved. Simulation results indicate that the proposed scheme greatly outperforms the traditional scheme and does not have reduction in the slopes of BER curves. Moreover, the memory cost of recording stuck locations is moderately low and grows almost linearly of the transition probability p.

Last we would like to give some remarks on the concatenation scheme:

• (1) Due to the asymmetric property of the stuck sub-channel, Figure 2, it is not optimal to set LLRs to zero for erased cells. Thus, we set the LLR to ∓ α log2 instead of zero in the simulation, where α is a scaling factor. However, no obvious coding gain is observed, since the amount of stuck cells is relatively small and little bias on them will not have obvious impacts on LDPC decoders. Thus, this article sets zero for simplicity.

• (2) In this article, side information from the asymmetric property, Figure 2, is used to reduce the memory cost of source codes. Other side informations are also amicable to improve the source codes’ efficiency, as long as they are correlated to the stuck location vector l with Pr ( _li ≠ _vi )<0.5 [14].

Appendix

Capacity of 4-level stuck sub-channel

The channel capacity of the stuck sub-channel is defined as follows,

$$C_s=\underset{p\left(V_i\right)}{\text{max}}\left[I(\mathbf{V};\mathbf{S})\right].$$

By constructing a new function with the Lagrange multiplier λ,

$$F=I(\mathbf{V};\mathbf{S})-\lambda \sum _{\mathbf{V}}p\left(V_i\right),$$

we obtain a function system

$$\left\{\begin{array}{c}\frac{\partial F}{\partial p\left(V_i\right)}=\frac{\partial \left[I(\mathbf{V};\mathbf{S})-\lambda \sum _{\mathbf{V}}p\left(V_i\right)\right]}{\partial p\left(V_i\right)}=0\hfill \\ \sum _{\mathbf{V}}p\left(V_i\right)=1\hfill \end{array}\right..$$

As is known, the mutual information of the channel is

$$I(\mathbf{V};\mathbf{S})=\sum _{i=1}^s\sum _{j=1}^{s}p\left(V_i\right)p\left(S_j\right|V_i)\underset{2}{\text{log}}\frac{p\left(S_j\right|V_i)}{p\left(S_j\right)},$$

(11)

where p ( _Vi )’s represent the probability contribution of the input while the probability of the output is given by

$$p\left(S_j\right)=\sum _{i=1}^{s}p\left(V_i\right)p\left(S_j\right|V_i).$$

The functions are transformed as below,

$$\left\{\begin{array}{c}\sum _{j=1}^{s}p\left(S_j\right|V_i)\underset{2}{\text{log}}\frac{p\left(S_j\right|V_i)}{p\left(S_j\right)}=\underset{2}{\text{log}}e+\lambda \hfill \\ \sum _{\mathbf{V}}p\left(V_i\right)=1\hfill \end{array}\right.,$$

where e represents the Euler number. For the 4-level stuck-sub-channel,

$$\left\{\begin{array}{c}p\left(S_1\right|V_1)\underset{2}{\text{log}}\frac{p\left(S_1\right|V_1)}{p\left(S_1\right)}+p(S_4\left|V_1\right)\underset{2}{\text{log}}\frac{p\left(S_4\right|V_1)}{p\left(S_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ p\left(S_2\right|V_2)\underset{2}{\text{log}}\frac{p\left(S_2\right|V_2)}{p\left(S_2\right)}+p(S_4\left|V_2\right)\underset{2}{\text{log}}\frac{p\left(S_4\right|V_2)}{p\left(S_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ p\left(S_3\right|V_3)\underset{2}{\text{log}}\frac{p\left(S_3\right|V_3)}{p\left(S_3\right)}+p(S_4\left|V_3\right)\underset{2}{\text{log}}\frac{p\left(S_4\right|V_3)}{p\left(S_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ p\left(S_4\right|V_4)\underset{2}{\text{log}}\frac{p\left(S_4\right|V_4)}{p\left(S_4\right)}=\underset{2}{\text{log}}e+\lambda ,\\ p\left(V_1\right)+p\left(V_2\right)+p\left(V_3\right)+p\left(V_4\right)=1,\end{array}\right.$$

where log₂e + λ equals to the channel capacity. Since p(_{S 4}|_{V 1}) = p (_{S 4}|_{V 2}) = p (_{S 4}|_{V 3}) = p, p(_{S 1}|_{V 1}) = p (_{S 2}|_{V 2}) = p (_{S 3}|_{V 3}) = 1 − p = q, and p (_{S 4}|_{V 4}) = 1,

$$\left\{\begin{array}{c}q\underset{2}{\text{log}}\frac{1}{p\left(V_1\right)}+p\underset{2}{\text{log}}\frac{p}{p+\mathrm{qp}\left(V_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ q\underset{2}{\text{log}}\frac{1}{p\left(V_2\right)}+p\underset{2}{\text{log}}\frac{p}{p+\mathrm{qp}\left(V_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ q\underset{2}{\text{log}}\frac{1}{p\left(V_3\right)}+p\underset{2}{\text{log}}\frac{p}{p+\mathrm{qp}\left(V_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ \underset{2}{\text{log}}\frac{1}{p+\mathrm{qp}\left(V_4\right)}=\underset{2}{\text{log}}e+\lambda ,\hfill \\ p\left(V_1\right)+p\left(V_2\right)+p\left(V_3\right)+p\left(V_4\right)=1.\hfill \end{array}\right.$$

From Equations (12) – (16), we obtain

$$\begin{array}{c}-q\underset{2}{\text{log}}p\left(V_1\right)+p\underset{2}{\text{log}}p-p\underset{2}{\text{log}}(p+\mathrm{qp}(V_4\left)\right)\hfill \\ =-\underset{2}{\text{log}}(p+\mathrm{qp}(V_4\left)\right).\hfill \end{array}$$

Therefore,

$$p\left(V_1\right)=\frac{1}{p^{-p}+3q}=p\left(V_2\right)=p\left(V_3\right)=\frac{q\left(V_4\right)}{3}.$$

(17)

If p (_{V 4}) ≥ 0, (17) is the optimum distribution of the input. Thus,

$$\begin{array}{c}{C}_s=I\left(\mathbf{V};\mathbf{S}\left|p\right(V_1),p(V_2),p(V_3),p(V_4)\right)\hfill \\ =q\underset{2}{\text{log}}(p^{-p}+3q)+p\underset{2}{\text{log}}\frac{p^q+3\mathrm{pq}}{p^q+q{p}^{-p}}.\hfill \end{array}$$

However, if p (_{V 4}) < 0, we set p (_{V 4}) to 0 to solve Equations (12) – (16) again and obtain

$$p\left(V_1\right)=p\left(V_2\right)=p\left(V_3\right)=\frac{1}{3}.$$

In this case,

$$C_s=q\underset{2}{\text{log}}3.$$