A Computationally Efficient Method for Polyphonic Pitch Estimation

—This paper presents a computationally efficient method for polyphonic pitch estimation. The method employs the Fast Resonator Time-Frequency Image (RTFI) as the basic time-frequency analysis tool. The approach is composed of two main stages. First, a preliminary pitch estimation is obtained by means of a simple peak-picking procedure in the pitch energy spectrum. Such spectrum is calculated from the original RTFI energy spectrum according to harmonic grouping principles. Then the incorrect estimations are removed according to spectral irregularity and knowledge of the harmonic structures of the music notes played on commonly-used music instruments. The new approach is compared with a variety of other frame-based polyphonic pitch estimation methods, and results demonstrate the high performance and computational efficiency of the approach


I. INTRODUCTION
Polyphonic pitch estimation plays an important role in music signal analysis.It can be essentially used for the detection of musically relevant features such as melody and harmony [1].In the case of content-based music retrieval, the "automatic" extraction of melody information is a crucial element for any music retrieval system [2].Another potential application is assisting the structured audio coding [3,4].
A number of approaches have been proposed in the literature.Klapuri proposed a polyphonic pitch estimation algorithm based on an iterative method [5], which was further explored for music transcription [6].In such method, first the predominant pitch of concurrent musical sound is estimated.Then the spectrum of the sound with the predominant pitch is estimated and subtracted from the mixture.The estimation and subtraction is repeated iteratively on the residual signal.
Recognizing a note in note-mixtures is a typical pattern recognition problem.Therefore, some approaches transform the polyphonic pitch estimation into a pattern recognition problem, which is then solved by employing machine learning methods such as neural networks [7,8] and support vector machines [9,10].Other methods such as Bayesian inference [11,12,13], sparse coding [14], and nonnegative matrix factorization [15] have also been investigated.More detailed reviews on the state of the art of polyphonic pitch estimation can also be found in [16].
The aim of this article is to describe a computationally efficient method for polyphonic pitch estimation.The method consists of time-frequency analysis and post-process phases.For both phases, novel techniques are used to increase computational efficiency.In the post-process phase, neither iterative processing nor machine learning is needed.First, a preliminary estimation is used to find all possible pitch candidates, which may include extra estimations.Then the incorrect estimations are removed according to the spectral irregularity and knowledge of the harmonic structures.The postprocess phase mainly involves pick-peaking, addition and subtraction operations, and the computational overload is negligible.Accordingly, the computational cost of the method chiefly depends on the time-frequency analysis part.The constant-Q Fast Resonator Time-Frequency Image (RTFI) has been selected as the basic time-frequency analysis tool.RTFI is employed here mainly because it can be implemented by the simplest filter banks.In addition, fast implementations of such filter banks can also further improve the computational efficiency.
As a result, the overall approach is 3 times faster than real time on a standard PC equipped with a 2.0 GHz Pentium processer.The method was also evaluated in the multiple fundamental frequency frame level estimation task of MIREX 2007 [17].The achieved results demonstrate the high performance and computational efficiency of the new approach.The method was the fastest and ranks third place in overall performance of the 16 submitted systems.Compared to the state-of-the-art approaches, it is more than 13 times faster, and has only slightly worse performance.(the accuracy of state-of-the-art method is 60.5%, whereas our method's accuracy is 58.2%).
The paper is organized as follows.Section II briefly introduces a new time-frequency analysis tool called Resonator Time-Frequency Image (RTFI), and the motivation to select Fast RTFI constant-Q analysis.Section III describes a new polyphonic pitch estimation method.Notably, Subsection III.C explains the novelty of the proposed method.Section IV describes the experimental setup and reports the performance evaluation, and Subsection IV.F compares the method with other state-of-the-art methods evaluated in MIREX 2007.Finally, Section V summarizes the main results and discusses possible extensions and future work.

A. Frequency-dependent Time-frequency Analysis
A Frequency-Dependent Time-Frequency (FDTF) analysis may be defined as follows: Unlike the STFT, the window function w of an FDTF may depend on the analytical frequency ω.
This means that time and frequency resolutions can be tuned according to the analytical frequency.
Equation (1) can also be expressed as: where Equation ( 1) is more suitable to express a transform-based implementation, whereas equation (2) leads to a straightforward implementation of a filter bank with impulse response functions expressed by equation (3).
A novel time-frequency representation, known as the Resonator Time-Frequency Image (RTFI), has been developed.Its main feature is that it selects a first-order complex resonator filter bank to implement a frequency-dependent time-frequency analysis.This was chosen due to the flexibility with regards to time and frequency resolution, and the simplicity and computational efficiency of an implementation based on first order filters.

B. Resonator Time-Frequency Image
The Resonator Time-Frequency Image (RTFI) can be described as follows: In the above equations, I R denotes the impulse response of the first-order complex resonator filter with oscillation frequency ω, and the factor r(ω) before the integral in the equation ( 4) is used to normalize the gain of the frequency response when the resonator filter's input frequency is the oscillation frequency.The decay factor r is dependent on the frequency ω and determines the exponent window length and the time resolution.It also determines the bandwidth (i.e. the frequency resolution).
Since the RTFI has a complex spectrum, it may be expressed as follows: where A(t,ω) and φ(t,ω) are real functions.The energy of the signal may then be given by 2 ) , ( ) , ( In this work, it is proposed to use the first order complex resonator digital filter bank to implement a discrete RTFI.To reduce the memory requirements needed to store the RTFI values, the RTFI is separated into different time frames and the average RTFI values are calculated in each frame.Finally the average RTFI energy is used to track the time-frequency characteristics of the music signal.The average RTFI energy spectrum can be expressed as follows: where M is the number of sample in the time frame, g is the index of frame, dB() converts the value to decibels and the ratio of M to sampling rate is the duration time of the frame in the averaging process.RTFI(n, f k ) denotes the value of the discrete RTFI at sampling point n and frequency f k , and J g denotes the frame which begins at the J g th sample of the analyzed signal.

C. Multi-resolution Fast RTFI
The Fast RTFI is used to reduce the redundancy in computation.In some cases it is not necessary to keep the same sampling frequency of the input for every filter in the filter bank.For the filters with lower center frequencies, the sampling rate can be decreased.In the fast implementation, the filter bank is separated into different octave frequency bands.The inputs of the filter banks in the same frequency band maintain the same sampling rate.The input signal is recursively low-pass filtered and down sampled by a factor of 2 from the highest to the lowest frequency band according to the scheme depicted in Figure 1.
This section has briefly introduced the basic idea behind RTFI analysis.A more detailed description of the discrete RTFI and its fast implementation can be found in [18,19].

D. Motivation for Selecting Constant-Q Time-Frequency Analysis
Resolution is a key factor of any time-frequency analysis.In the following, it is explained how it may be reasonable to select a nearly constant-Q resolution for a general-purpose music analysis system.Using the MIDI (Music Instrument Digital Interface) note numbers, the fundamental frequency and corresponding partials of a music note k' can be described as: Supposing that the energy of every music note is mainly distributed over the first 10 partials, thus 0 ) ( ' ≈ m k f Energy for m≥11, the frequency ratio between the partials of one note and the fundamental frequencies of other notes can be expressed as follows: This means that the first 10 partials always overlap with another fundamental frequency.Since the fundamental frequencies follow an exponential law (9), most of the energy is concentrated in frequency bins that are evenly spaced on a logarithmic axis.This is the reason for which the required resolution is constant-Q.

E. Motivation for Selecting Fast RTFI to Implement Constant-Q Time-Frequency Analysis
The proposed method is manly used for polyphonic pitch tracking, where a joint time-frequency analysis is first needed.Either filter bank or constant-Q transform can be used to compute constant-Q time-frequency spectrum.As RTFI is implemented by the simplest filter bank, it is faster than any other filter-bank-based implementation.The Fast RTFI is also compared with transform-based implementations as follows.
So as to use a constant-Q transform for a joint time-frequency analysis, the time signal needs to be cut into different frames and then a constant-Q transform is performed in each frame [20].It is assumed that the pitch tracking can report pitches every 10ms, so the time interval between two successive frames is set as 10ms.To perform a constant-Q time-frequency analysis for a 1-second signal, the constant-Q transform needs to be calculated 100 times, and the required number of complex multiplies can be expressed as: where Q is the constant ratio of frequency to resolution, f s is the sampling rate, f min is the lowest analytical frequency, N 1 is the number of octave bands, and N 2 is the number of frequency components in one octave band.A fast constant-Q transform has been proposed in [21].It employs an FFT to calculate constant-Q transform.When the fast constant-Q transform is used for time-frequency analysis of a 1-second signal, the required number of complex multiplies can be roughly expressed as: ) log( 100 For the Fast RTFI analysis of a 1-second signal, the required number of complex multiplies can be roughly obtained as: In the proposed method, the constant-Q factor Q is set as 17, the lowest analysis frequency f min is 26 Hz, the number of octave bands N 1 is 9, and the number of frequency components in one octave band is equal to 120.Accordingly, for constant-Q analysis of a 1-second signal, Fast RTFI implementation needs approximately 240*f s complex multiplies, constant-Q transform implementation needs approximately 24900*f s , and fast constant-Q transform implementation needs approximately 2000*f s .
The comparison clearly suggests that Fast RTFI implementation is also much faster than transformbased implementation for a constant-Q time-frequency analysis.

A. System Overview
ARTFI denotes the input RTFI average energy spectrum, k = 1, 2, 3, … is the frequency index on the logarithmic scale, the second term in the right hand part of the equation denotes the moving average of ARTFI , and M 1 is the length of the window for calculating the moving average.
Similarly, preliminary estimates of the possible multiple pitches are found by a simple peak-picking procedure in a relative pitch energy spectrum, which is obtained from the RTFI average energy spectrum.Then a confidence measure is employed to remove pitch candidates whose harmonic Fig. 2. System overview of new polyphonic pitch estimation method components are not strongly represented.Finally, the pitches are found by investigating the spectral irregularity of the remaining candidates.These five steps are described in detail in the following subsections.

B. Detailed Description 1) Time-frequency Processing Based on the RTFI Analysis:
In the first step, the Fast RTFI is used to analyze the input music signal and to produce a timefrequency energy spectrum.The input sample is a monaural music signal frame at a sampling rate of 44.1 kHz.All 1080 filters are used.The centre frequencies are set on a logarithmic scale.The centre frequency difference between two neighbouring filters is equal to 0.1 semitone, and the analyzed frequency range is from 26 Hz up to 13 kHz.Then, the time-frequency energy spectrum of the input frame is used to obtain an RTFI average energy spectrum according to equation ( 8).This RTFI average energy spectrum is used as the only input vector for later processing.An integer k is used to denote the frequency index on a logarithmic scale, and f k denotes the corresponding frequency value expressed in Hz in the equation: Equation ( 14) has been derived from the fundamental frequencies of musical notes on the western music scale.One example for the input RTFI average energy spectrum of a piano note is provided in Figure 3.

2) Extraction of Harmonic Components:
In the second step, the input RTFI average energy spectrum is first transformed into the relative energy spectrum according to the expression (13).
Figure 3 shows the RTFI energy spectrum and its moving average.The relative energy spectrum RES(f k ) is a measure of the energy spectrum for the k th frequency bin, relative to the energy spectrum over a frequency range near the k th frequency bin.
If there is a peak in the relative energy spectrum at the k th frequency index and the value RES(f k ) is larger than a threshold A 1 , it is likely that there is a harmonic component at the frequency index k.The corresponding value RES(f k ) is assumed to be a measure of confidence in the existence of the harmonic component.

3) Preliminary Estimations of Pitch Candidates:
In the third step, based on the harmonic grouping principle, the input RTFI average energy spectrum is first transformed into the pitch energy spectrum (PES) and the relative pitch energy spectrum (RPES) as follows: where M 2 is the length of the window for calculating the moving average, and L is a parameter that denotes how many low harmonic components are together considered as important evidence for Fig. 3.The input RTFI energy spectrum, moving average and the corresponding relative energy spectrum of a piano polyphonic note consisting of two concurrent notes with fundamental frequencies 82 Hz and 466 Hz determining the existence of a possible pitch.Similar techniques have been proposed for pitch estimations by some researchers.In [22], the authors propose a polyphonic pitch estimation approach by summing harmonic amplitudes.There are two main differences between the method described in this paper and the approach introduced in reference [22].First, the reference approach is based on the STFT spectrum, whereas the proposed method employs an RTFI constant-Q spectrum.Secondly, the reference approach directly sums harmonic amplitudes and does not use a decibel scale, whereas the new method produces a pitch energy spectrum by summing the harmonic energies on a decibel scale.Our experiments demonstrate that directly summing the harmonic energies yields lower estimation performances.
In practical implementations, instead of using equation ( 15), the pitch energy spectrum on a logarithmic scale can easily be approximated by the following expression (here L is less than 10):  1, the deviation between the approximate and ideal values of the pitch energy spectrum can be considered negligible for practical purposes.
There are two assumptions made when determining a preliminary estimate of the possible pitches from the relative pitch energy spectrum.If there is a pitch with fundamental frequency f k, in the input signal, there should be a peak centred around the frequency f k in the relative pitch energy spectrum, and the peak value should exceed a threshold A 2 .Both assumptions are consistent with real music examples when a suitable threshold A 2 is selected.Only the low-pitch notes may have very faint first harmonic components that cannot be reliably extracted.Based on these observations, some assumptions concerning the extracted harmonic components can be made for determination of whether an extracted pitch is correct.For example, if there is a pitch with a fundamental frequency higher than 82 Hz, either the lowest three harmonic components or the lowest three odd harmonic components of this pitch should all be present in the extracted harmonic components.If there is a pitch with a fundamental frequency lower than 82 Hz, four of the lowest six harmonic components should be present in the extracted harmonic components.
In two typical cases, the extra estimated pitches can be removed based on the above assumptions.
In the first case, the extra pitch estimation is caused by a noise peak in the preliminary pitch estimation.In the second case, the harmonic components of an extra estimated pitch are partly overlapped by the harmonic components of the true pitches.In such a case, the non-overlapped harmonic components become important clues to check the existence of the extra estimated pitch.If a polyphonic set of notes contains two concurrent music notes C5 and G5, for example, the fundamental frequency ratio of the two notes is nearly 2/3.Then, it is probable that there is an extra pitch estimation on the C4 note, because its even harmonics are overlapped by the odd harmonics of C5, and the C4 note's third, sixth, ninth, … harmonic components are nearly overlapped by the G5 note's odd harmonics.However, the C4's first, fifth, and seventh harmonic components are not overlapped, so the extra C4 estimation can be easily identified by checking the existence of the first harmonic component based on the above assumption.

5) Determining the Existence of the Pitch Candidate by the Spectral Irregularity:
By means of the previous steps, the extra incorrect estimations centered around the pitches whose note intervals are 12, 19, or 24 semitones higher than the identified true pitches.In such a case, the fundamental frequencies of the extra estimated pitches are placed 2, 3 or 4 times the frequency of a true pitch, and the harmonic components of each extra pitch are completely overlapped by the true pitch.For example, consider when two of the estimated pitch candidates are the notes with fundamental frequencies F 0 and 3F 0 .Here the difficulty is to determine if the note with the fundamental frequency 3F 0 is an incorrect extra estimation caused by the overlapped frequency components of the lower frequency music note.This is the most difficult case in the polyphonic pitch estimation problem.However, such a problem can be solved by investigating spectral irregularity.
The spectral value difference between two neighbouring harmonic components is small and random in most cases.But when a music note with the fundamental frequency F 0 is mixed with another note with the higher integer ratio fundamental frequency nF 0 , then the corresponding spectral value of every n th harmonic component will become clearly larger than the neighbouring harmonic components.If there are two estimated pitch candidates that have fundamental frequencies of F 0 and F' 0 ( F' 0 ≈ nF 0 ), and a frequency ratio that is approximately an integer n, then the proposed method employs the following two steps to determine if the higher pitch with the fundamental F' 0 occurs.
First, the energy spectrum of the first 10n corresponding harmonic components with the fundamental frequency F 0 is calculated by an RTFI analysis with uniform resolution.The RTFI average energy spectrum of the harmonic components can be expressed as ARTFI H (k), k=1, 2, 3, …, (10n), where k denotes the harmonic component index.
The second step is composed of the following operations.The Spectral Irregularity (SI) is calculated using the expression: )) 2 According to our observations, if two of the estimated pitch candidates have the fundamental frequencies, F 0 and F' 0 for which (F' 0 ≈ nF 0 ) and if the higher pitch does not occur, then SI(n) is usually small.On the other hand, if the higher pitch does occur, then the overlapped harmonic components are often strengthened so that SI(n) results in a larger value.When SI(n) is smaller than a given threshold, the overlapped higher pitch candidate is removed.The threshold is determined by experiments on a training database.In practical examples, most incorrect extra estimates caused by the overlapping of harmonic components are placed at a low integer multiple of the frequency of the true pitch.Consequently, the new method proposed in this paper only consider cases for which the fundamental frequency ratio of two pitch candidates is equal to 2, 3 or 4.

C. Novelty of the Proposed Method
In this subsection, the novelty and promising features of the proposed method is outlined.In the time-frequency processing part, the Fast RTFI constant-Q time-frequency analysis is first employed for polyphonic pitch tracking.As explained in Section II.E, it is much more computationally efficient than other implementations.
In the post-process phase, the developed method first estimates pitch candidates by peak-picking from the relative pitch energy spectrum.Since the sounds with integer fundamental frequency ratio can produce very similar peak patterns in a pitch energy spectrum, usually an extra incorrect estimation has an integer ratio to the fundamental frequencies of an identified pitch.This problem mainly arises from the coinciding frequency partials between Western polyphonic music notes.
The state-of-the-art method solves the problem by employing iterative estimation and cancelation schema [5].The basic idea is to first find a predominant pitch, and estimate the spectrum of the predominant pitch.Then the estimated spectrum is cancelled from the mixture and produces residual signals before the next estimation.The estimation and cancellation is repeated iteratively on the residual signal.It may also involve the process of estimating the polyphonic number of the analyzed sound.
So as to solve the problem of coinciding frequency partials, the basic idea of the new proposed method is completely different from the state-of-art approach introduced above.The proposed method provides a much simpler solution to the problem and does not require to implement an iterative procedure or to estimate the polyphonic number.In the new method, the preliminary estimation finds all possible pitch candidates.Then some pitch candidates are removed if their harmonic components are not enough represented in the energy spectrum.Finally, if fundamental frequencies between any two pitch candidates have an integer ratio, the spectral irregularity is calculated to remove the pitch candidate, which is considered to be an error estimation caused by coinciding frequency partials from a lower pitch.
By employing these new techniques, the proposed method is more computationally efficient, but presenting comparable performance with the other state-of-art methods.

A. Performance Evaluation Criteria
Three criteria were used to evaluate the performance of the polyphonic pitch estimation methods; "Precision", "Recall", and "F-measure".Given a reference fundamental frequency, if there is an estimation that is equal to or presents an error of no more than 3% deviation from the reference fundamental frequency, it is considered to be a correct detection.Otherwise, it is considered as a false negative (FN).Any estimation that deviates by more than 3% from all reference fundamental frequencies is considered to be a false positive (FP).Precision, Recall, and F-measure can be defined according to the following expressions: where N CD , N FP , and N FN denote the total number of correct detections, false positives and false negatives, and P and R denote the values of precision and recall, respectively.In addition, the Overall Accuracy, as defined in [9], is also used for the performance comparison with other state-of-art methods.

B. Setting the Method Parameters
The real performance of an estimation method may be overestimated when parameters have been optimally selected to fit the test data.So as to prevent such occurrence, separate training and test datasets have been constructed.
It is quite difficult to record a large number of polyphonic samples from different musical instruments and label their polyphony content.A preferred method is to produce the polyphonic samples by mixing real recorded monophonic samples of different music instruments.
In these experiments, two different monophonic sample sets were used to create the training and test dataset.The monophonic sample set I consisted of a total of 755 monophonic samples from 19 different instruments, such as piano, guitar, winds, strings, and brass, etc.To obtain fairer evaluation results of practical cases, the monophonic sample set II was used to generate the test dataset.
Compared to set I, the monophonic samples in Set II, for the same type of instrumentation as samples in Set I, were played by different performers and instruments from different instrument manufacturers.Set II included 23 different instrument types, a total of 690 monophonic samples in the five octave pitch range of 48 Hz to 1500 Hz.
All the monophonic samples in Set I and Set II were selected from the RWC instrument sound database [23].Every instrument sample was recorded at three levels of dynamics (forte, mezzo, piano) across the total range of that instrument.Generally speaking, different instruments play with different strengths.Accordingly, instead of being normalized, the natural amplitudes of the monophonic samples were kept in order to construct polyphonies by different energy ratios.The high number of polyphonic samples was generated by randomly mixing these different monophonic samples.These polyphonic samples were generated by first selecting an instrument and then a random note from the instrument's playing range.Based on the monophonic sample set I, a total of 11,000 polyphonic samples with the polyphony from two to six note mixtures were generated for the training dataset.
Similarly, monophonic set II was used to generate 11,000 polyphonies for the test dataset.The size of every polyphonic subset in the training and test datasets is described in Table 2.All the following test experiments were performed on the whole test dataset, which was classified into five different subsets according to the polyphony number of the mixed polyphonic samples.
The described method has eight different parameters: L, M 1 , M 2 A 1 , A 2 and the thresholds of spectral irregularity.These parameters were tuned on the training dataset.The different parameter values were selected by a heuristic method.Table 3 reports the values, which were tried for different parameters.
About 15,000 parameter combinations were tried.Values that yielded the best average F-Measure on the training dataset were selected, and parameters were fixed when the method was evaluated on the test dataset.

C. Performance and Robustness
The method was tested on the test dataset and achieved F-measures of 89%, 87%, 84%, 81%, and 78% respectively on polyphonic mixtures ranging from two to six simultaneous sounds.In order to test the robustness, pink noise was added into the polyphonic mixtures with different Signal-to-Noise ratios.The pink noise was generated in the frequency range of 50 Hz to 10K Hz.The Signal-to-Noise refers to the ratio between the clean input signal power and the added pink noise power.
Figure 6 shows the F-measure of the new method with different levels of added pink noise, where a value of 1 for the F-measure indicates optimal performance.In general, the method is robust, even in cases of severe noise levels.The tested samples were classified into five different sample subsets according to the polyphony number of the mixed polyphonic samples.For example, in Figure 6, the F-measure corresponding to the polyphony number 2 denotes the F-measure value estimated on the sample subset, in which every polyphonic sample consists of a two-note mixture.

D. Comparison Experiments with/without Applying Relative Spectrums
In the described method, the relative spectrums (relative energy spectrum and relative pitch energy spectrum) have been used.A comparison experiment has been made to evaluate how the application of relative spectrums improves the method's performance.The method was tested for every polyphony sample subset of the test dataset.The test results of the method with or without applying the relative spectrum are reported in Table 4.The results demonstrate that the application of the relative spectrum improves the method's performance.

E. Tradeoff between Recall and Precision
Precision is the percentage of the transcribed notes that are correct, and Recall is the percentage of all the notes that are found.There is inherent tradeoff between Precision and Recall.Depending on applications, better Precision or better Recall is preferred.For example, in some music transcription systems, the extra incorrect estimations in the result are very harmful, so better Precision is preferred.
However, if the output result will be used for further improvement with the combination of some higher level knowledge, better Recall is preferred.
The tradeoff between Precision and Recall can be controlled by adjusting the thresholds A 1 , A 2 and the thresholds of spectral irregularity.In this method, harmonic components need to be extracted from the relative energy spectrum by peak-picking.Although the peaks with larger values have higher probability to represent harmonic components, there may still be some large peaks which represent noise.Thus, only the peaks with values larger than the threshold A 1 are considered to represent harmonic components.When A 1 is set to a small value, more true harmonic components may be extracted, but more noise peaks are also incorrectly assumed to be harmonic components.As a result, more true notes may be found, but the incorrect estimation are also increased.Therefore, when A 1 is set low, the method will get better Recall at the cost of lower Precision.Similarly, if thresholds A 2 and the thresholds of spectral irregularity are set low, estimation performance will probably have better Recall.Otherwise, the estimation performance will have better Precision.Figure 7 shows the estimation performance (F-measure, Recall, Precision) of this method with two different parameter sets.Compared with the left image (small parameter values), the Precision shown in the right image (large parameter values) increases at the price of a lower Recall.

F. MIREX 2007 Results -Performance Comparison to Other State-of-art Methods
In order to compare our technique with other state-of-art approaches, the new method was submitted to the multiple fundamental frequency frame level estimation task of MIREX 2007 [17].In this evaluation task, there were 28 test files, each of which had a 30-second duration.These files consisted of 20 real recordings, 8 synthesized from RWC samples.The summary results of the first 8 methods in the rank are reported in Table 5.In the evaluation, our method (labeled as team 'ZR') was ranked third in the 16 submitted approaches.However the difference of results between our method and the best method (team 'RK') was really minor, whereas our method was approximately 13 times faster than the best method (team 'RK').The algorithm has been implemented as Matlab M-files and MEX-files.The execution time on a 2 GHz Pentium processor is about one third of the time duration of a monaural audio recording.

V. CONCLUSION AND FUTURE WORK
In this article, a computationally efficient and robust method has been proposed to estimate pitches in real polyphonic music.Compared to the state-of-art approach, the proposed method is conceptually simple and much faster, and presents comparable performance.In the method, the pitch estimation process can be separated into three consecutive stages.In order to show how each stage improves the performance, the method was run on the test dataset, and the result in each stage is reported.First, the preliminary estimation aims to find all possible pitch candidates.About 95% of true notes were successfully found.Then the method removes the pitch candidates, which do not have sufficient harmonic components in the energy spectrum.In this stage, the total performance F-measure is improved from 33% to 63%.Finally, possible remaining ambiguities (such as an integer ratio between fundamental frequencies) are partially solved by investigating the spectral irregularity.The final stage increases the F-measure from 63% to 83%.
Approximately 30% of all errors are octave errors, and about 18% of all errors are due to confusion between notes with the fundamental frequency ratio of 1/3.These errors are mainly caused by coinciding frequency partials from a lower pitch.This result demonstrates that the coinciding issue can be further investigated by combining the method with analysis of temporal features which were not yet exploited.The harmonic components from the same instrument sound source often present similar temporal features, such as a common onset time, amplitude modulation and frequency modulation.The harmonic relative frequency components with similar temporal features should have a higher probability of representing the same note than those with different temporal features.
For example, a polyphonic note combination may consist of two notes, A3 and A4, where A3 is played by piano and the note A4 is played by violin.It is very difficult to make a polyphonic estimation for this case, because the harmonic components of A4 are completely overlapped by the even harmonic components of A3.However, such a difficult case may be resolved by using temporal features.As shown in Figure 8, the blue lines denote the first four odd harmonic components of A3, and the red/magenta lines denote the first four even harmonic components of the note A3.It can be clearly seen that the energy spectrums of the first four even harmonic components have different temporal features than the first four odd harmonic components.This difference indicates that the even harmonic components are probably shared with another musical note.The remaining errors are mainly related to the fact that the timbres of the instruments may differ greatly from each other.Therefore, assumptions concerning the spectral harmonic characteristics are unlikely to be suitable for all instruments.Further improvements to the approach could be achieved by developing efficient instrument recognition algorithms.The recognition algorithms could first automatically estimate the dominant instrument type for the music signal being analyzed; then the method can use the known spectral harmonic characteristics of that instrument type.In this situation, instrument recognition need only be sufficiently accurate as to place the musical signal in the class of an instrument with similar spectral harmonic characteristics.

Figure 2
Figure2provides an overview of the new polyphonic pitch estimation method.It can be

Figure 4 4 )
Figure 4 illustrates the relative pitch energy spectrum of a violin example, which consists of four

Fig. 4 .
Fig.4.Relative Pitch Energy Spectrum of a violin example consisting of four concurrent notes with the fundamental frequencies 266Hz, 299Hz, 353Hz and 403Hz.

Figure 5 Fig. 5 .
Figure 5 illustrates the RTFI average energy spectrum of the first 30 harmonic components of two

Fig. 6 .
Fig.6.F-Measure of test results of the proposed method with a clean signal or various levels of added noise.

Fig. 7 .
Fig.7.F-Measure, Recall and Precision results for the proposed method with different parameters.

Figure 8
Figure 8 Energy Changes of Harmonic Components of a Polyphonic Note with Two Polyphonies A3 and A4

Table 5 :Results of Multiple Fundamental Frequency Frame Level Estimation Task of MIREX 2007
issue is only partially solved by identifying the spectral irregularity.In future work, this frequency