Efficient methods for joint estimation of multiple fundamental frequencies in music signals

2.1 Preprocessing

The analysis is performed in the frequency domain, computing the magnitude spectrogram using a 93 ms Hanning windowed frame with a 9.28 ms hop size. This is the frame size typically chosen for multiple f₀ estimation of music signals in order to achieve a suitable frequency resolution, and it experimentally showed to be adequate. The selected frame overlap ratio may seem high from a practical point of view, but it was required to compare the method with other studies in MIREX (see 4.3).

To get a more precise estimation of the lower frequencies, zero padding is used multiplying the original window size by a factor z to complete it with zeroes before computing the FFT.

In order to increase the efficiency, many unnecessary spectral bins are discarded for the subsequent analysis using a simple peak picking algorithm to extract the hypothetical partials. At each frame, only those spectral peaks with an amplitude higher than a threshold μ are selected, removing the rest of spectral information and obtaining this way a sparse representation containing a subset of spectral bins. It is important to note that this thresholding does not have a significant effect on the results, as values of μ are quite low, but the efficiency of the method importantly increases.

2.2 Candidate selection

The evaluation of all possible f₀ combinations in a mixture is computationally intractable, therefore a reduced subset of candidates must be chosen before generating their combinations. For this, candidates are first selected from the spectral peaks within the range [f_min, f_max] corresponding to the musical pitches of interest. Harmonic sounds with missing fundamentals are not considered, although they seldom appear in practical situations. A minimum spectral peak amplitude ε for the first partial (f₀) can also be assumed in this stage.

The spectral magnitudes at the candidate partial positions are considered as a criterion for candidate selection as described next.

2.2.1 Partial search

Slight harmonic deviations from ideal partial frequencies are common in music sounds, therefore inharmonicity must be considered for partial search. For this, a constant margin around each harmonic frequency f _h ± f _r is set. If there are no spectral peaks within this margin, the harmonic is considered to be missing. Besides considering a constant margin, frequency dependent margins were also tested assuming that partial deviations in high frequencies are larger than those in low frequencies. However, results decreased, mainly because many false positive harmonics (most of them corresponding to noise) can be found in high frequencies.

Different strategies were also tested for partial search, and finally, like in [30], the harmonic spectral location and spectral interval principles [31] were chosen in order to take inharmonicity into account. The ideal frequency f _h of the first harmonic is initialized to f _h = 2f₀. The next ones are searched at f_h+1= (f _x + f₀) ± f _r , where f _x = f _i if the previous harmonic h was found at the frequency f _i , or f _x = f _h if the previous partial was missing.

In many studies, the closest peak to f _h within a given region is identified as a partial. A novel variation which experimentally slightly increased (although not significantly) the proposed method performance is the inclusion of a triangular window. This window, centered in f _h with a bandwidth 2f _r and a unity amplitude, is used to weight the partial magnitudes within this range (see Figure 2). The spectral peak with maximum weighted value is selected as a partial. The advantage of this scheme is that low amplitude peaks are penalized and, besides the harmonic spectral location, intensity is also considered to correlate the most important spectral peaks with partials.

2.3 Generation of candidate combinations

All the possible combinations of the F selected candidates are calculated and evaluated, and the combination with highest score is yielded at the target frame. The combinations consist of different number of fundamental frequencies. In contrast to studies like [26], there is not need for a priori estimation of the number of concurrent sounds before detecting the fundamental frequencies, and the polyphony is implicitly calculated in the f₀ estimation stage, choosing the combination with highest score independently from the number of candidates.

Therefore, N combinations are evaluated at each frame, so the adequate selection of F and P is critical for the computational efficiency of the algorithm. An experimental discussion on this issue is presented in Sec. 4.2.

2.4 Evaluation of combinations

In order to evaluate a combination ${\mathcal{C}}_i\in \mathcal{C}\left(t\right)$, a hypothetical pattern is first estimated for each of its candidates. Then, these patterns are evaluated in terms of their intensity and smoothness, assuming that music sounds have a perceivable intensity and their spectral shapes are smooth, like it occurs for most harmonic instruments. The combination $\widehat{\mathcal{C}}\left(t\right)$ which patterns maximize these measures is yielded at the target frame t.

2.4.1 Inference of hypothetical patterns

The intention of this stage is to infer harmonic patterns for the candidates. This is performed taking into account the interactions with other candidates in the analysed combination, assuming that they have smooth spectral envelopes. A pattern (HPS) is a vector p _c estimated for each candidate $c\in \mathcal{C}$ consisting of the hypothetical harmonic amplitudes of the first H harmonics:

${\mathbf{p}}_c={\left(p_{c,1},p_{c,2},\dots ,p_{c,h},\dots ,p_{c,H}\right)}^{\mathsf{\text{T}}}$

(2)

where p_c,his the amplitude for the h harmonic of the candidate c. The partials are searched the same way as previously described for the candidate selection stage. If a particular harmonic is not found within the search margin, then the corresponding value p_c,his set to zero. As in music sounds the first harmonics are usually the most representative and they contain most of the sound energy, only the first H partials are considered to build the patterns. Once the partials of a candidate are identified, the HPS values are estimated taking into account the hypothetical source interactions. For this task, their harmonics are identified and labeled with the candidate they belong to (see Figure 3). After the labeling process, some harmonics will only belong to one candidate (non-overlapped harmonics), whereas others will belong to more than one candidate (overlapped harmonics).

Assuming that interactions between non-coincident partials (beating) do not alter significantly the original spectral amplitudes, the non-overlapped amplitudes are directly assigned to the HPS. However, the contribution of each source to an overlapped partial amplitude must be estimated.

Getting an accurate estimate of the amplitudes of colliding partials is not reliable only with the spectral magnitude information. In this study, the additivity of linear spectrum is assumed as in most approaches in the literature. Assuming additivity and spectral smoothness, the amplitudes of overlapped partials can be estimated similarly to [26, 32] by linear interpolation of the neighboring non-overlapped partials, as shown in Figure 3 (bottom).

If there are two or more consecutive overlapped partials, then the interpolation is done the same way using the available non-overlapped values. For instance, if harmonics 2 and 3 of a pattern are overlapped, then the amplitudes of harmonics 1 and 4 are used to estimate them by linear interpolation.

After the interpolation, the estimated contribution of each partial to the mixture is subtracted before processing the next candidates. This calculation (see Figure 3) is done as follows:

• If the interpolated (expected) value is greater than the corresponding overlapped harmonic amplitude, then p_c,his set as the original harmonic amplitude, and the spectral peak is completely removed from the residual, setting it to zero for the candidates that share that partial.

• If the interpolated value is smaller than the corresponding overlapped harmonic amplitude, then p_c,his set as the interpolated amplitude, and this value is linearly subtracted for the candidates that share the harmonic.

The residual harmonic amplitudes after this process are iteratively analysed for the rest of the candidates in the combination in ascending frequency order.

2.4.2 Candidate evaluation

The intensity l(c) of a candidate c is a measure of the strength of a source obtained by summing its HPS amplitudes:

$l\left(c\right)=\sum _{h=1}^H{p}_{c,h}$

(3)

Assuming that a pattern should have a minimum loudness, those combinations having any candidate with a very low absolute (l(c) <η) or relative $\left(l\left(c\right)<\gamma L_{\mathcal{C}},\mathsf{\text{being}}{L}_{\mathcal{C}}={\text{max}}_{{\forall }_c}\left\{l\left(c\right)\right\}\right)$ intensity are discarded.

The underlying hypothesis assumes that a smooth spectral pattern is more probable than an irregular one. This is assessed through a novel smoothness measure s(c) which is based on Gaussian smoothing.

To compute it, the HPS of a candidate is first normalized dividing the amplitudes by its maximum value, obtaining $\bar{\mathbf{p}}$. The aim is to compare $\bar{\mathbf{p}}$ with a smooth model $\tilde{\mathbf{p}}$ built from it, in such a way that the similarity between $\bar{\mathbf{p}}$ and $\tilde{\mathbf{p}}$ will give an estimation of the smoothness.

For this purpose, $\bar{\mathbf{p}}$ is smoothed using a truncated normalized Gaussian window ${\mathcal{N}}_{0,1}$, which is convolved with the HPS to obtain $\tilde{\mathbf{p}}$:

${\tilde{\mathbf{p}}}_c={\mathcal{N}}_{0,1}*{\bar{\mathbf{p}}}_c$

(4)

Only three components were chosen for the Gaussian window of unity variance, ${\mathcal{N}}_{0,1}={\left(0.21,0.58,0.21\right)}^{\mathsf{\text{T}}}$, due to the small size of p _c , which is limited by H. Typical values for H are within the range H ∈ [5, 20], as only the first harmonics contain most of the energy of a harmonic source.

Then, as shown in Figure 4, a roughness measure r(c) is computed by summing up the absolute differences between $\tilde{\mathbf{p}}$ and the actual normalized HPS amplitudes:

$r\left(c\right)=\sum _{h=1}^H\left|{\tilde{\mathbf{p}}}_{c,h}-{\bar{\mathbf{p}}}_{c,h}\right|$

(5)

The roughness r(c) is normalized into $\bar{r}\left(c\right)$ to make it independent of the intensity:

$\bar{r}\left(c\right)=\frac{r\left(c\right)}{1-{\mathcal{N}}_{0,1}\left(\bar{x}\right)}$

(6)

And finally, the smoothness s(c) ∈ [0, 1] of a HPS is calculated as:

$s\left(c\right)=1-\frac{\bar{r}\left(c\right)}{H_c}$

(7)

where H _c is the index of the last harmonic found for the candidate. This factor was introduced to prevent that high frequency candidates that have less partials than those at low frequencies will have higher smoothness. This way, the smoothness is considered to be more reliable when there are more partials to estimate it.

A candidate score is computed taking into account the HPS smoothness and intensity:

$S\left(c\right)=l\left(c\right)\cdot s^{\kappa }\left(c\right)$

(8)

where κ is a factor that permits to balance the smoothness contribution experimentally.

3.1 Temporal smoothing

A simple and effective novel technique is presented in order to smooth the detection across time. Instead of selecting the most likely combination at isolated frames, adjacent frames are also analysed to get the score of each combination.

The method aims to enforce the pitch continuity in time. For this, the fundamental frequencies of each combination $\mathcal{C}$ are mapped into music pitches, obtaining a pitch combination ${\mathcal{C}}^{\prime }$. For instance, the combination ${\mathcal{C}}_i=\left\{261\mathsf{\text{Hz}},416\mathsf{\text{Hz}}\right\}$ is mapped into ${\mathcal{C}}_i^{\prime }=\left\{{\mathsf{\text{C}}}_4,\mathsf{\text{G}}{♯}_{\mathsf{\text{4}}}\right\}$.

If there is more than one combination with the same pitches (for instance, ${\mathcal{C}}_1=\left\{260\mathsf{\text{Hz}}\right\}$ and ${\mathcal{C}}_2=\left\{263\mathsf{\text{Hz}}\right\}$ are both ${\mathcal{C}}^{\prime }=\left\{{\mathsf{\text{C}}}_4\right\}$), it is removed, and the unique combination with the highest score value is only kept.

where ${\mathcal{C}}_i^{\prime }$ are the combinations that appear at least once in the 2K + 1 frames considered. Note that the score values for the same combination are summed in the 2K frames around t to obtain $\tilde{S}\left({\mathcal{C}}_i^{\prime }\left(t\right)\right)$. An example of this procedure is displayed in Figure 5 for K = 1. If ${\mathcal{C}}_i$ is missing for any t - K <j <t + K, it does not contribute to the sum.

If ${\widehat{\mathcal{C}}}^{\prime }\left(t\right)$ does not contain any combination because there are no valid candidates in the frame t, then a rest is yielded without evaluating the adjacent frames.

When the temporal evolution of the smoothed intensity $\tilde{l}\left(c^{\prime }\left(t\right)\right)$ of the winner combination candidates is plotted in a three-dimensional representation (see Figures 6 and 7), it can be seen that the correct estimates usually show smooth temporal curves. An abrupt change (a sudden note onset or offset, represented by a vertical cut in the smoothed intensities 3D plot) means that the energy of some harmonic components of a given candidate were suddenly improperly assigned to another candidate in the next frame. Therefore, vertical lines in the plot usually indicate errors in assigning harmonic components.

3.2 Pitch tracking

A basic pitch tracking method is introduced in order to favor smooth transitions of $\tilde{l}\left(c^{\prime }\left(t\right)\right)$. The proposed technique aims to increase the temporal coherence using a layered weighted directed acyclic graph (wDAG).

The idea is to minimize abrupt changes in the intensities of the pitch estimates. For that, a graph layered by frames is built with the pitch combinations, where the weights consider the differences in the smoothed intensities for the candidates in adjacent frames and their combination scores. Let G = (V, v _I , E, w,t) be a layered wDAG, with vertex set V, initial vertex v _I , edge set E, and edge function w, where w(v _i ,v _j ) is the weight of the edge from the vertex v _i to v _j . The position function t: V → {0,1, 2,..., T} associates each node with an input frame, being T the total number of frames. Each vertex v _i ∈ V represents a combination ${\mathcal{C}}_i^{\prime }$. The vertices are organized in layers (see Figure 8), in such a way that all vertices in a given layer have the same value for t(v) = τ, and they represent the M most likely combinations at a time frame τ.

Using this scheme, the transition weight between two combinations considers the score of the target combination and the differences between the candidate intensities.

Once the graph is generated, the shortest path that minimizes the sum of weights from the starting node to the final state across the wDAG is found using the Dijkstra [33] algorithm. The vertices that belong to the shortest path are the pitch combinations yielded at each time frame.

Building the wDAG for all possible combinations at all frames could be computationally intractable, but considering only the M most likely combinations at each frame keeps almost the same runtime than without performing tracking for small values of M.

4.1 Evaluation metrics

Different metrics for multiple f₀ estimation can be found in the literature. The evaluation can be done both at frame by frame and note levels. The first mode evaluates the correct estimation in a frame by frame basis, whereas note tracking also considers the temporal coherence of the detection, adding more restrictions for a note to be considered correct. For instance, in the MIREX note tracking contest, a note is correct if its f₀ is closer than half a semitone to the ground-truth pitch and its onset is within a ± 50 ms range of the ground truth note onset.

A false positive (FP) is a detected pitch (or note, if evaluation is performed at note level) which is not present in the signal, and a false negative (FN) is a missing pitch. Correctly detected pitches (OK) are those estimates that are also present in the ground-truth at the detection time.

An alternative metric based on the speaker diarization error score from NIST^a was proposed by Poliner and Ellis [34] to evaluate multiple f₀ estimation methods. The NIST metric consists of a single error score which takes into account substitution errors (mislabeling an active voice, E _subs ), miss errors (when a voice is truly active but results in no transcript, E _miss ), and false alarm errors (when an active voice is reported without any underlying source, E _fa ).

This metric avoids counting errors twice as classical metrics do in some situations. For instance, using accuracy, if there is a C₃ pitch in the reference ground-truth but the system reports a C₄, two errors (a false positive and a false negative) are counted. However, if no pitch was detected, only one error would be reported.

Poliner and Ellis [34] state that, as in the universal practice in the speech recognition community, this is probably the most adequate measure, since it gives a direct feel for the quantity of errors that will occur as a proportion of the total quantity of notes present.

4.2 Parameterization

A data set of random pitch combinations, also used in the evaluation of Klapuri [35] method, was used to tune up the algorithm parameters. The data set consists on 4000 mixtures with polyphony^b 1, 2, 4, and 6. The 2842 audio samples from 32 music instruments used to generate the mixtures are from the McGill University master samples collection^c, the University of Iowa^d, IRCAM studio online^e, and recordings of an acoustic guitar. In order to respect the copyright restrictions, only the first 185 ms of each mixture^f were used for evaluation. In this dataset, the range of valid pitches is [f_min = 38 Hz, f_max = 2100 Hz], and the maximum polyphony is P = 6.

The values for the free parameters of the method were experimentally evaluated. Their impact on the performance and efficiency can be seen on Figures 9 and 10, and it is extensively analysed in [4, pp. 141-156]. In these figures, the cross point represents the values selected for the parameters. Lines represent the impact tuning individual parameters when keeping the selected values for the rest of parameters.

In the parameterization stage, the selected parameter values were not those that achieved the highest accuracy in the test set, but those that obtained a good trade-off between accuracy and low computational cost.

4.3 Evaluation and comparison with other methods

The core method was externally evaluated and compared with other approaches in MIREX 2007 [22] multiple f₀ estimation and tracking contest, whereas the extended method was submitted to MIREX 2008 [23]. The data set used in both MIREX editions were essentially the same, therefore the results can be directly compared. The details of the evaluation and the ground-truth labeling are described in [36]. Accuracy, precision, recall and E _tot were reported for frame by frame estimation, whereas precision, recall and F-measure were used for the note tracking task.

The core method (PI1-07) was evaluated using the parameters specified in Table 1. For this contest, a final postprocessing stage was added. Once the fundamental frequencies were estimated, they were converted into music pitches, and pitch series shorter than d = 56 ms were removed to avoid some local discontinuities.

The extended method was submitted with pitch tracking (PI1-08) and without it (PI2-08) for comparison. In the non-tracking case, a similar procedure than in the core method was adopted, removing notes shorter than a minimum duration and merging note with short rests between them. Using pitch tracking, the methodology described in Sec. 3.2 was performed instead, increasing the temporal coherence of the estimate with the wDAG using M = 5 combinations at each layer.

In the review from Bay et al. [36], the results of the algorithms evaluated in both MIREX editions are analysed. As shown in Figure 11, the proposed methods achieved a high overall accuracy and the highest precision rates. The extended method also obtained the lowest error rates E _tot from all the methods submitted in both editions (see Figure 12).

In the evaluation of note tracking considering only onsets, the proposed approaches showed lower accuracies (Figure 13), as only the extended method can perform pitch tracking. The inclusion of the tracking stage did not improve the results for frame by frame estimation, but in the note tracking task the results outperformed those obtained for the same method without tracking. The proposed methods were also very efficient respect to the other state of the art algorithms presented (see Table 3), especially considering that they are based on a joint estimation scheme.

While the proposed approaches achieved the lowest E _tot score, there were very few false alarms compared to miss errors. On the other hand, the methods from Ryynänen and Klapuri [17] and Yeh et al. [37] had a better balanced precision, recall, as well as a good balance in the three error types, and as a result, high accuracies.

Quoting Bay et al. [36], "Inspecting the methods used and their performances, we can not make generalized claims as to what type of approach works best. In fact, statistical significance testing showed that the top three methods (YRC, PI, and RK) were not significantly different."