The present section addresses the design of the order constellation with fixed (under the assumption that the WL MMSE equalizer is used) and it is organized as follows. In Section 3.1, we address the optimum constellation design for the WL MMSE receiver by extending the results of [2] to the case of additive rotationally variant disturbance. In Section 3.2, we propose a suboptimum strategy based on the rhombic transformation of a given constellation. Such a strategy allows one to reduce both the computational complexity of the optimization procedure and the amount of information required at the transmitting side in order to adapt the constellation.
The results in the previous section allow one to state that, by using a realvalued constellation () instead of a complexvalued nonredundant () one, a performance gain can be achieved in terms of the MMSE at the equalizer output. On the other hand, not always an MSE gain provided by the WL equalizer leads to a SER gain [19]. In fact, for a fixed expended average energy per bit, the reduction of the minimum distance between the constellation points, due to the adoption of onedimensional constellations rather than twodimensional ones (e.g., when we adopt the PAM rather than the QAM) leads to a potential increase in the SER. Therefore, we address the constellation design minimizing the SER at the WL MMSE equalizer output by accounting for its rotationally variant properties.
In the literature (e.g., [2, 20]), most of the constellations employed by the transmission stage are circularly symmetric (), while statistically redundant constellations are confined to the realvalued ones. Moreover, in [2], with reference to the transmission over a time nondispersive channel () affected by circularly symmetric noise, a procedure for constellation optimization has been proposed, showing also that, for large signaltonoise ratios (SNR), the performance of the conventional QAM maximumlikelihood (ML) receiver is invariant with respect to rhombic transformations of the complex plane. However, it is important to point out that a rhombic transformation of a circular constellation makes it rotationally variant and, for some values of (e.g., ), the procedure in [2] provides a rotationally variant constellation. On the other hand, the WL equalizer is equivalent to the linear equalizer over the nondispersive channel considered in [2] and, therefore, optimizing the circularity degree of the constellation does not provide any performance advantage. On the other hand, when a timedispersive channel is considered, the WL MMSE equalizer is sensitive to the rotationally variant properties of the transmitted signal and, therefore, we propose a transceiver structure (see Figure 1) where (i) the transmitter can switch between the available constellations of order ; (ii) the WL MMSE receiver accounts for the CSI and informs the transmitter, by means of a feedback channel, about which constellation has to be adopted to minimize the SER.
The use of a feedback channel in order to improve the bitrate could also be exploited for choosing the constellation size rather than its circularity degree when the signaltonoise ratio of each channel realization is not previously known. For example, the problem of the constellation choice has been addressed in [21, 22] with reference to the discrete multitone (DMT) transceiver and to multipleinput multipleoutput transceiver, respectively. The two parameters of the constellations (size and circularitydegree) could also be jointly optimized by generalizing the procedures here proposed.
3.1. Constellation Optimization in the Presence of Gaussian Rotationally Variant Noise
In order to optimize over the constellation choice we need to first derive a performance analysis of the considered equalizer. Approximated evaluations of the performance of the WL receiver are available in [11] for a QAM constellation and in [3] for a PAM constellation in the presence of a PAM cochannel interference. Moreover, such performance analysis is generalized in [9] for IIR WL filters. Here, we derive an approximation of the equalizer performance suited for successive optimization over transmitter constellation.
With no loss of generality, assume that and rewrite the output of the FIR equalizer as follows:
where is the transmitted symbol drawn from the complexvalued constellation with and , and is the residual disturbance that includes the intersymbol interference and the noise terms after the WL equalizer filtering. The circularly symmetric model for the additive disturbance is inadequate since the output of a WL filter is, in general, rotationally variant. Therefore, we model as rotationally variant, that is, , , and . Moreover, in order to make the constellation design analytically tractable, we approximate as Gaussian. For the sake of clarity, let us note that, if symbols and noise are circularly symmetric (), then the additive disturbance and the equalizer output will be circularly symmetric too; on the other hand, if is rotationally variant, then will be rotationally variant too, but nothing can be stated about the circularity properties of also when .
The sample is the input of the decision device which performs the symbolbysymbol ML detection of the transmitted symbol. By defining the following eigenvalue decomposition (the dependence on at the righthandside is omitted for simplicity):
with being the eigenvector matrix and having on the diagonal the eigenvalues, it can be verified that the pairwise error probability [20], that is, the probability of transmitting and deciding (at the receiver) in favor of when the transmission system uses only and , is given by
where denotes + where and , and, for and ,
When , and analogously for and . By utilizing (11), assuming that the symbols are equally probable, and resorting to both the union bound and Chernoff bound techniques, the SER is upperbounded as follows:
and, therefore, the optimum constellation can be approximated with the solution of the following problem:
Unfortunately, it is difficult to find the closedform expression of the solution of such an optimization problem. For such a reason, we propose to find a local solution by means of numerical algorithms (e.g., a projected gradient method). To this aim, we can exploit the gradient of with respect to , while we resort to numerical approximation of the gradient with respect to since it is difficult to obtain its analytical expression.
Before proceeding, let us discuss the property of the locally optimum constellation for a fixed . The th component of the gradient of is given by
By zeroing the gradient of the Lagrangian
one has that the locally optimum satisfies the following equation:
with
Condition (17) generalizes the result of [2] to the case of rotationally variant (i.e., or ) and with a constrained pseudocorrelation. (.) In fact, (17) with (i.e., no constraint is imposed on the pseudocorrelation) requires that is proportional to the weighted sum (with weights ) of , , as found in [2]. For the sake of clarity, let us note that the procedure proposed in [2] does not allow one to exploit the potential advantage of a rotationally variant constellation when the WL MMSE receiver is employed. For example, when a linear MMSE equalizer is employed for in high signaltonoise ratio, the minimum of the SER is equivalently achieved [2] by both the conventional 4QAM constellation and the rhombic constellations with the same perimeter, that is, the perimeter of the largest convex polygon consisting of the lines (see [1] for further details). On the other hand, when a WL MMSE equalizer is employed, a rhombic constellation, which is rotationally variant, is not equivalent to the conventional 4QAM since the achieved MMSE is dependent on as shown in (8).
3.2. A Suboptimum Procedure Based on Rhombic Transformations
In this section, we propose a suboptimum constellationdesign procedure for the WL MMSE equalizer. The method is based on the exploitation of a rhombic transformation that operates on a circularly symmetric constellation making it rotationally variant. Such a transformation depends on two parameters and allows one to control the pseudocorrelation of the obtained constellation; consequently, the optimization procedure is simplified since the SER in (13) is a function of only two parameters, instead of parameters.
Assume that is a unitpower circularlysymmetric complexvalued constellation and define the complexvalued constellation as follows:
or, more compactly (the compact expression is introduced for notation simplicity whereas the matrix form is utilized to understand the physical meaning),
with and . When (), is stretched along the inphase (quadrature) component and it becomes onedimensional for ; when , a correlation between and is introduced and for , even if it is twodimensional, can be reduced to a onedimensional constellation by a simple rotation. For symmetry, in the following we consider only the positive values of and . It is easily verified that, if is drawn from , then
The method proposed here assumes that the informationbearing symbol sequence, say , is drawn from a fixed constellation (e.g., the optimum constellation provided by [2]) whereas the possibly rotationally variant channel input is obtained by resorting to the zeromemory precoding defined by the rhombic transformation (19). Clearly, such a strategy is suboptimum since it assumes that the channel input can be drawn from only those constellations resulting from a rhombic transformation of the chosen . However, the main advantages of such a method in comparison with the optimum one are

(1)
the huge reduction of the computational complexity of the constellation optimization procedure when ; in fact, the SER becomes a function of only two variables ( and ), regardless of the constellation order ;

(2)
the reduced implementation complexity of the transmitter stage; in fact, the symbolmapping is implemented by means of the linear transformation (19);

(3)
the decrease of the information amount to be transmitted on the feedback channel; in fact, only the values of two parameters (instead of ) have to be sent to the transmitter.
According to such a choice, the constellation optimization is carried out by solving the minimization problem
with
where denotes , denotes − , and denotes + − − − , and where (23) follows from (13) and (19), and the dependence of the disturbance parameters on has been replaced by the dependence on and . Since finding the closedform expression of and is a difficult problem, here we propose to approximate with a function, say , whose minimization can be carried out by evaluating it only over a very limited set of points. In the sequel, such an approximation is derived for a QAM constellation , though it can be analogously determined for denser constellations.
First, we approximate the cost function (23) by assuming that the components of the residual disturbance are uncorrelated, that is, . By means of some tedious but simple algebra operations, it can be shown that is lower bounded by
where
where denotes and denotes . Since the righthand side of (24) is minimized by large values of , we propose to approximate the solution of (22) with the following one:
where is the curve corresponding to the maximum value of for a fixed (or, equivalently, to the maximum value of for a fixed ). Of course, the restriction to leads to a significant decrease in the computational complexity. Let us point out that, interestingly, such a restricted optimization procedure accounts for the possible transmission of the conventional 4PAM: in fact, it can be easily verified that when , .
This also suggests an extreme simplification obtained by choosing just between the 4PAM and 4QAM constellation (twochoice procedure), that is, one can resort to an architecture that switches between the 4QAM and the 4PAM constellations according to the following rule:
Three remarks about the suboptimum procedure (26) follow.
Remark 1.
The results carried out here with reference to the QAM constellation can be easily generalized to higherorder constellations. More specifically, the SERbound approximations (analogous to the one in (24)) can be obtained by assuming that the inner summation in (23) is restricted to those constellation points closest to the th one. Moreover, it can be shown that the conventional square QAM constellations (with ) can be transformed by (19) into the conventional uniform PAM. Note, however, that such a property is not satisfied by the constellations of any order; for example, as also shown in Section 4, when using the rectangular QAM (see Figure 2(g)) the rhombic transformation allows one to obtain the nonuniform PAM reported in Figure 2(i).
Remark 2.
The optimum transmission strategy proposed here requires that the receiver sends on the feedback channel the whole optimum constellation. If the suboptimum procedure is used, the transmitter architecture can be simplified. In fact, a unique symbol mapper for the alphabet is needed and the constellation is adapted by adjusting the zeromemory WL filter (19). Unfortunately, the main disadvantage in terms of the computational complexity of the receiver remains the adaptation of the decision mechanism for the constellation .
Remark 3.
When the proposed suboptimum strategy is used, the channel input is obtained by performing a zeromemory WL filtering of the informationbearing sequence . For such a reason, it is reasonable to consider an alternative receiver structure that performs the WL MMSE equalization of the received signal in order to estimate , instead of . After some matrix manipulations, it can be verified that such WL MMSE equalizer is the cascade of the WL MMSE equalizer in (4) and (5) and the WL zeromemory filter performing the inverse of the transformation (19) (note that (19) is not invertible for every value of and , e.g., when a realvalued constellation is adopted (), however, in such a case, an ad hoc inverse transformation can be easily defined). This allows one to use a unique symbol demapper and the standard decision mechanism for the constellation . The MMSE achieved by such a structure is
It can be easily shown that (a) if , then , unless , and (b) since . Such results show that the minimumdistance decision based on the WL MMSE estimation of outperforms the (computationally simpler) minimumdistance decision based on the WL MMSE estimation of .