# Nyquist Filters with Alternative Balance between Time- and Frequency-Domain Parameters

- Marek Bobula
^{1}Email author, - Aleš Prokeš
^{1}and - Karel Daněk
^{2}

**2010**:903980

https://doi.org/10.1155/2010/903980

© Marek Bobula et al. 2010

**Received: **20 March 2010

**Accepted: **1 October 2010

**Published: **28 October 2010

## Abstract

Designing matched pulse shaping filters with their cascade satisfying the Nyquist condition for minimum intersymbol interference constitutes an important task for almost all digital data radio transceivers processing an incoming data signal on the sample-by-sample basis. Despite their practical importance, there are only few sets of Nyquist filter definitions and design techniques to devise digital filter coefficients available for a designer. In this paper we propose a set of Nyquist filters that balance the time- and frequency-domain parameters in favor of a filter stop-band attenuation and residual intersymbol interference compared with the already existing Nyquist filter sets. Using a number of filter examples, this paper shows that the proposed Nyquist filters can be a good option for applications that need to fulfill strict limits of adjacent and alternate channel power attenuation while providing a low level of residual intersymbol interference and group delay of the digital filter.

## Keywords

## 1. Introduction

*intersymbol interference*(ISI). To lower the amount of ISI to zero, the overall transmitter-channel-receiver frequency response should fulfill the first Nyquist criterion defined in the frequency domain as [1–3]

For a minimum double-sided signal bandwidth equal to the Nyquist frequency
, where
is the modulation symbol period, the Nyquist condition yields a unique solution with its time representation being a sinc pulse. For the signal bandwidth exceeding the Nyquist frequency, innumerable filter characteristics should exist [1]. However, to the best of the present authors' knowledge, there are only two sets of filter frequency responses explicitly defined by their continuous piecewise function definitions [4, 5]. The first one is the well-known *raised cosine* (RC) filter. In spite of their practical importance there are only few design techniques that can be used to directly devise a matched combination of digital Nyquist filters. The Matlab software and its Digital Filter Design application [6] provide Nyquist filters based on a truncated version of the (square root) raised cosine filter. Such filters create a single balance between the time- and frequency-domain parameters. Additional weighting window can be used to compromise the out-of-band suppression for the excess bandwidth; however the coefficients of square root filter version cannot be explicitly calculated and for most of the windows, the minimum excess bandwidth is limited by the equivalent noise bandwidth of the window being used [7]. When the same design approach is used to calculate the coefficients of the square root RC filter directly, the residual ISI is increasing rapidly, since the design technique significantly violates the first Nyquist condition. Other techniques for designing the pulse shaping filters that have been reported are based on optimization techniques with a specific objective function such as maximizing the robustness against timing jitter [2, 3, 8–10] minimizing the duration of the impulse response [11–13] or minimizing the *peak to average power ratio* (PAPR) at the transmitter filter output [14]. Besides the complex way of calculating the filter coefficients, filters that are obtained by optimization techniques are usually of nonlinear phase [11, 12] or they cannot be directly partitioned into matched filter pairs [2, 3, 8–10, 15–17] having linear phase characteristic. The same holds true for the IIR Nyquist filters as discussed in [17, 18]. The most recent methods for designing Nyquist filters are based on linear programming technique [8–10, 16]. The latest and a well-written reference along this line concerning with the problem of designing matched pairs of digital Nyquist (
) filters with balanced performance and linear phase characteristic can be found in [19]. The method is based on numerical calculation and does not provide an exact definition of the filter characteristic, which complicates the design of matched filter pairs with different values of the group delay parameter and oversampling ratio. However, it offers a wide range of communication filters and will also be used for comparison in this paper.

If the time- and frequency-domain parameters of the raised cosine filter are taken as a standard, the aim of this paper is to formulate an alternative Nyquist filter definition that reasonably balances the parameters in favor of filter stop-band attenuation, which is directly related to an adjacent channel power as one of the most critical parameters of the several practical applications. Giving a number of filter examples, this paper also provides a detailed comparison of the proposed filters and the existing Nyquist filter sets.

## 2. Nyquist Filters Design Analysis

*vestigial symmetry*around the frequency of . In addition to these conditions, the overall Nyquist filter is typically split into its transmitter (TX) and receiver (RX) parts to fulfill the matched filtering condition for maximizing the signal-to-noise ratio at the decision stage [1]. The condition can be written in the frequency domain as follows:

- (i)
The length of the impulse response should be kept as small as possible to minimize the implementation cost and the propagation delay of the DSP part of the radio transceiver.

- (ii)
The Nyquist criterion should be satisfied as closely as possible to minimize intersymbol interference.

- (iii)
At the transmitter side, the primary issue is the transmission bandwidth defined by the adjacent channel power requirements. Thus the adjacent channel and alternate channel power limits dictate the stop-band attenuation of the transmitter filter .

- (iv)
The lower the magnitude of the impulse response side-lobes the better is the eye opening, hence the better timing jitter immunity can be reached [3].

In several applications, there is also a demand to minimize the peak-to-average power ratio of the transmitted pulses [14, 19]. Such an optimization of the transmitter and receiver filters yields a filter pair that is no longer mutually matched and therefore it is not considered in this work.

*raised cosine*(RC) frequency response defined by

*excess bandwidth*parameter usually called

*roll-off factor*. The corresponding continuous time impulse response of the raised cosine filter can be written as

The lower the roll-off factor, the narrower is the occupied bandwidth of the signal. However, with a low roll-off factor the original modulation (modulated) signal becomes more vulnerable to timing jitter as the eye pattern of the received signal narrows down and becomes practically almost undetectable for the roll-off factor equaling zero. The second negative effect resulting from lowering of the roll-off factor is related to the practical aspects of the digital filter design and that is the length of the filter impulse response or the decay of its tails to zero. The lower the roll-off factor, the longer the impulse response of the raised cosine filter has to be taken into account when designing a digital filter with comparable stop-band attenuation.

The excess bandwidth parameter can be used to trade off the occupied bandwidth and the time duration of the pulse. In both definitions (6) and (7) the excess bandwidth is assumed to be lower than 100%. Although there are Nyquist filters with the excess bandwidth exceeding 100% [20], they are not a primary concern of this work.

where is the number of samples per one modulation symbol.

## 3. Nyquist -Filter Formulation

A closer analysis of (5) unveils the fact that the frequency response of the RC filter is a convolution of a rectangular pulse of duration and a raised cosine window modified by the roll-off parameter.

*Blackmann*,

*exact-Blackmann*,

*Blackmann-Harris,*and windows by changing the values of , , and in its definition

*Nyquist*

*-filter*defined in the frequency domain can thus be written as follows:

*,*, and equal , , and , respectively, the equations can be effectively reduced to

Thus, when comparing both filters (Section 5) the RC filter will have an equivalent excess bandwidth (15) adjusted by the value of the parameter.

and is a train of unit impulses. In order to keep the notation simple the coefficients of the Nyquist -filter impulse response have been denoted .

can be used for the transmitter and receiver filter directly, since the impulse response is symmetrical and therefore linear phase.

It is important to note that all the definitions of the Nyquist and square root Nyquist filters either in the frequency or the time domain were given in their noncausal form and thus zero phase characteristics were implicitly assumed.

## 4. Nyquist Filter Evaluation Criteria

## 5. Matched Nyquist Filters Evaluation

0.2 | 0.25 | 0.3 | 0.35 | 0.4 | 0.45 | 0.5 | ||

0.027 | 0.034 | 0.040 | 0.047 | 0.054 | 0.061 | 0.068 | ||

0.227 | 0.284 | 0.340 | 0.397 | 0.454 | 0.511 | 0.568 | ||

5 | 0.1 | 5.9 | 10.9 | 8.8 | 8.0 | 8.6 | 8.4 | |

−4.2 | 5.9 | 6.7 | 0.6 | 7.8 | 5.2 | 2.6 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.7 | ||

6 | 4.2 | 10.8 | 8.5 | 8.2 | 8.6 | 7.5 | 6.8 | |

3.9 | 6.8 | 4.6 | 6.6 | 4.3 | 5.9 | 11.3 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.7 | ||

7 | 9.9 | 8.9 | 8.3 | 8.4 | 7.2 | 6.8 | 5.8 | |

10.4 | 2.0 | 6.7 | 3.8 | 11.4 | 10.4 | 6.6 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.8 | ||

8 | 10.2 | 8.0 | 8.6 | 7.2 | 6.8 | 5.2 | 4.5 | |

5.0 | 7.6 | 4.3 | 10.9 | 10.6 | 7.1 | 8.2 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.8 |

0.2 | 0.25 | 0.3 | 0.35 | 0.4 | 0.45 | 0.5 | ||

0.027 | 0.034 | 0.040 | 0.047 | 0.054 | 0.061 | 0.068 | ||

0.227 | 0.284 | 0.340 | 0.397 | 0.454 | 0.511 | 0.568 | ||

4 | 4.6 | 10.8 | 8.4 | 8.3 | 8.6 | 7.4 | 6.8 | |

4.3 | 6.5 | 5.8 | 6.4 | 4.1 | 7.1 | 10.9 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.8 | ||

5 | 4.2 | 10.8 | 8.5 | 8.2 | 8.6 | 7.5 | 6.8 | |

3.9 | 6.8 | 4.6 | 6.6 | 4.3 | 5.9 | 11.3 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.7 | ||

6 | 4.1 | 10.8 | 8.6 | 8.2 | 8.6 | 7.6 | 6.8 | |

3.7 | 7.0 | 3.7 | 6.7 | 4.4 | 5.1 | 11.5 | ||

−0.7 | −0.9 | −1.2 | −1.5 | −1.9 | −2.3 | −2.7 |

*"*

*"*filters in contrast to the Nyquist -filters move a balance in favor of the time domain parameters and thus the difference in stop-band attenuation and residual ISI is even greater than in the case of the RC filter described before. Since all the values are given in their relative form, one can easily make a comparison between the RC and the " " filter.

0.2 | 0.25 | 0.3 | 0.35 | 0.4 | 0.45 | 0.5 | ||

0.027 | 0.034 | 0.040 | 0.047 | 0.054 | 0.061 | 0.068 | ||

0.227 | 0.284 | 0.340 | 0.397 | 0.454 | 0.511 | 0.568 | ||

5 | 8.2 | 15.4 | 19.5 | 18.2 | 19.6 | 19.9 | 19.1 | |

3.2 | 8.8 | 6.9 | 12.8 | 13.4 | 8.3 | 10.1 | ||

−2.2 | −2.9 | −3.8 | −4.9 | −6.1 | −7.3 | −8.3 | ||

6 | 13.7 | 15.4 | 18.3 | 19.9 | 19.5 | 19.4 | 13.0 | |

5.5 | 12.3 | 14.7 | 11.1 | 7.4 | 15.1 | 17.5 | ||

−2.2 | −1.5 | −3.9 | −5.0 | −6.1 | −7.3 | −6.6 | ||

7 | 18.8 | 18.3 | 13.1 | 19.3 | 19.8 | 19.6 | 18.5 | |

11.9 | 12.2 | 11.7 | 8.4 | 18.9 | 14.8 | 17.9 | ||

−2.2 | −3.0 | −2.4 | −5.0 | −6.2 | −7.4 | −8.4 |

Numerical values of [dB], [dB], [dB]. Square root Nyquist ( ) filter versus square root Nyquist -filter. The parameter is fixed and the group delay parameter of the filter varies; and are the design parameters of the iteration algorithm [19].

0.5 | 1 | 2 | 10 | 0.5 | 1 | 2 | 10 | ||

0.25 0.034 | −1.2 | −0.5 | −0.3 | 0.0 | −0.1 | 0.1 | 0.1 | 0.2 | |

1.4 | −2.6 | −5.0 | −6.6 | 3.5 | −1.3 | −4.6 | −6.9 | ||

0.0 | 0.1 | 0.2 | −0.2 | −0.3 | −0.3 | −0.2 | −0.2 | ||

0.5 | 1 | 2 | 10 | 0.5 | 1 | 2 | 10 | ||

0.25 0.034 | −2.6 | −2.2 | −2.1 | −1.0 | 1.1 | 0.6 | 0.3 | 0.9 | |

−0.1 | −3.1 | −4.1 | −5.2 | 8.5 | 3.2 | −1.1 | −5.0 | ||

0.2 | 0.2 | 0.2 | 0.1 | −0.4 | −0.1 | 0.0 | 0.0 | ||

0.5 | 1 | 2 | 10 | 0.5 | 1 | 2 | 10 | ||

0.25 0.034 | −9.6 | −9.3 | −9.3 | −4.9 | 4.6 | 4.7 | 4.6 | 3.6 | |

−1.4 | −2.2 | −2.5 | −4.1 | 11.8 | 6.4 | 1.6 | −5.0 | ||

0.4 | 0.4 | 0.4 | 0.3 | −0.9 | −0.4 | −0.3 | −0.1 |

## 6. Conclusion

As has been shown in this paper, the conflicting requirements placed on the design of the digital matched Nyquist filter pair make the task complicated. From this point of view, the typical truncated raised cosine filter represents a single balance between the time- and frequency-domain parameters. The Nyquist -filter set, which was defined throughout this paper, enlarges the family of Nyquist filters, and when compared with the truncated square root raised cosine filter cascade, it strikes the balance towards the frequency domain parameters while reaching a low level of residual intersymbol interference. It does so at the cost of a higher side lobes level of the impulse response. An exact symbolic definition of the Nyquist -filter in the frequency domain gives the designer scope to choose freely the arbitrary filter parameters such as equivalent excess bandwidth, group delay and oversampling parameters and derive coefficients for either the "normal" or the square root filter variants. The filters generated are of linear-phase having symmetrical impulse responses, which directly contribute to efficient hardware realization structures.

## Declarations

### Acknowledgments

This work was supported by the Czech Science Foundation under project no. 102/08/0851.

## Authors’ Affiliations

## References

- Burr A:
*Modulation and Coding for Wireless Communications*. Pearson Education Ltd., Harlow, UK; 2001.Google Scholar - Alagha NS, Kabal P: Generalized raised-cosine filters.
*IEEE Transactions on Communications*1999, 47(7):989-997. 10.1109/26.774849View ArticleGoogle Scholar - Farhang-Boroujeny B: Nyquist filters with robust performance against timing jitter.
*IEEE Transactions on Signal Processing*1998, 46(12):3427-3431. 10.1109/78.735318View ArticleGoogle Scholar - Farhang-Boroujeny B:
*Signal Processing Techniques for Software Radios*. ECE Dep. University of UTAH, Salt Lake City, Utah, USA; http://www.lulu.com - Beaulieu NC, Tan CC, Damen MO: A "better than" Nyquist pulse.
*IEEE Communications Letters*2001, 5(9):367-368. 10.1109/4234.951379View ArticleGoogle Scholar - Mathworks Inc : Digital Filter Design DSP Blockset Description. January 2010, http://www.mathworks.com/access/helpdesk_r13/help/toolbox/dspblks/digitalfilterdesign.html
- Harris FJ: On the use of windows for harmonic analysis with the discrete Fourier transform.
*Proceedings of the IEEE*1978, 66(1):51-83.View ArticleGoogle Scholar - Boonyanant P, Tantaratana S: Design and hybrid realization of FIR Nyquist filters with quantized coefficients and low sensitivity to timing jitter.
*IEEE Transactions on Signal Processing*2005, 53(1):208-221.MathSciNetView ArticleGoogle Scholar - Coleman JO: Linear programming of data communication pulses tolerant of timing jitter or multipath.
*Procedings of the 5th Annual Conference on Wireless Communications, July 1993, Calgary, Canada*http://jocoleman.info/pubs/papers/wireless93.pdfGoogle Scholar - Muravchik CH, Guisantes JR: Optimized signaling waveforms to reduce the effects of intersymbol interference and timing jitter.
*IEEE Transactions on Communications*1995, 43(1):11-14. 10.1109/26.385949View ArticleGoogle Scholar - Kragh F, Nga C, Hermes DJ, Robertson RC: Spectrally efficient digital modulation using new pulse shapes. In
*Proceedings of the Military Communications Conference (MILCOM '07), October 2007*. IEEE; -5.Google Scholar - Hermes DJ, Kragh FE: A bandwidth efficient constant envelope modulation.
*Proceedings of the 40th Asilomar Conference on Signals, Systems, and Computers (ACSSC '06), November 2006*488-492.Google Scholar - Zhang X, Yoshikawa T: Design of FIR Nyquist filters with low group delay.
*IEEE Transactions on Signal Processing*1999, 47(5):1454-1458. 10.1109/78.757243View ArticleGoogle Scholar - Châtelain B, Gagnon F: Peak-to-average power ratio and intersymbol interference reduction by nyquist pulse optimization.
*Proceedings of the IEEE 60th Vehicular Technology Conference (VTC '04), September 2004*954-958.Google Scholar - Tuncer TE: Optimum Nyquist filters with controlled time and frequency characteristics.
*Electronics Letters*2002, 38(21):1292-1293. 10.1049/el:20020880View ArticleGoogle Scholar - Davidson TN: Efficient design of waveforms for robust pulse amplitude modulation.
*IEEE Transactions on Signal Processing*2001, 49(12):3098-3111. 10.1109/78.969517View ArticleGoogle Scholar - Nakayama K, Mizukami T: A new IIR Nyquist filter with zero intersymbol interference and its frequency response approximation.
*IEEE transactions on circuits and systems*1982, 29(1):23-24. 10.1109/TCS.1982.1085078View ArticleGoogle Scholar - Maeng SJ, Lee BGI: Design of linear-phased IIR Nyquist filters.
*IEEE Journal on Selected Areas in Communications*1995, 13(1):167-175. 10.1109/49.363135View ArticleGoogle Scholar - Farhang-Boroujeny B: A universal square-root Nyquist (M) filter design for digital communication systems.
*Proceedings of the SDR Forum Technical Conference (SDR '06), November 2006, Orlando, Fla, USA*http://www.sdrforum.org/pages/sdr06/sdr06_papers/2.1/2.1-3.pdf or http://www.ece.utah.edu/~farhang/Nyquist_M_r1.pdfGoogle Scholar - Benjebbour A, Asai T, Yoshino H: Nonparametric interference suppression using cyclic wiener filtering: pulse shape design and performance evaluation.
*EURASIP Journal on Wireless Communications and Networking*2008, 2008:-14.Google Scholar - Jan J:
*Digital Signal Filtering, Analysis and Restoration*. VUTIUM BUT, Brno, Czech Republic; 2002.Google Scholar - Proakis J:
*Digital Communication*. 4th edition. McGraw-Hill Higher Education, Singapore; 2000.Google Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.