Music recommendation according to human motion based on kernel CCA-based relationship

Abstract

In this article, a method for recommendation of music pieces according to human motions based on their kernel canonical correlation analysis (CCA)-based relationship is proposed. In order to perform the recommendation between different types of multimedia data, i.e., recommendation of music pieces from human motions, the proposed method tries to estimate their relationship. Specifically, the correlation based on kernel CCA is calculated as the relationship in our method. Since human motions and music pieces have various time lengths, it is necessary to calculate the correlation between time series having different lengths. Therefore, new kernel functions for human motions and music pieces, which can provide similarities between data that have different time lengths, are introduced into the calculation of the kernel CCA-based correlation. This approach effectively provides a solution to the conventional problem of not being able to calculate the correlation from multimedia data that have various time lengths. Therefore, the proposed method can perform accurate recommendation of best matched music pieces according to a target human motion from the obtained correlation. Experimental results are shown to verify the performance of the proposed method.

1 Introduction

With the popularization of online digital media stores, users can obtain various kinds of multimedia data. Therefore, technologies for retrieving and recommending desired contents are necessary to satisfy the various demands of users. A number of methods for content-based multimedia retrieval and recommendation^a have been proposed. Image recommendation [1–3], music recommendation [4–6], and video recommendation [7, 8] have been intensively studied in several fields. It should be noted that most of these previous works had the constraint of query examples and returned results to be recommended being of the same type. However, due to diversification of users' demands, there is a need for a new type of multimedia recommendation in which the media types of query examples and the returned results can be different. Thus, several recommendation methods [9–12] for realizing these recommendation schemes have been proposed. Generally, they are called cross-media recommendation. In the conventional methods of the cross-media recommendation, the query examples and recommended results need not to be of the same media types. For example, users can search music pieces by submitting either an image example or a music example.

Among the conventional methods of cross-media recommendation, Li et al. proposed a method for recommendation between images and music pieces by comparing their features directly using a dynamic time warping algorithm [9]. Furthermore, Zhang et al. proposed a method for cross-media recommendation between multimedia documents based on a semantic graph [11, 12]. A multimedia document (MMD) is a collection of co-existing heterogeneous multimedia objects that have the same semantics. For example, an educational web page with instructive text, images and audio is an MMD. By these conventional methods, users can search for their desired contents more flexibly and effectively.

It should be noted that the above-conventional methods concentrate on recommendation between different types multimedia data. Thus, in this scheme, users are forced to provide query multimedia data, although they do not have a limitation of media types. This means that users must make some decisions to provide queries, and this causes difficulties for reflecting their demands. If recommendation of some multimedia data from features directly obtained from users is realized, one feasible solution can be provided to overcome the limitation. Specifically, we show the following two example applications: (i) background music selection from humans' dance motions for non-edited video contents^b and (ii) presentation of music information from features of target music pieces or dance motions. In the first example, using the relationship obtained between dance motions and music pieces in a database, we can obtain/find matched music pieces from human motions in video contents, and vice versa. This should be useful for creating a new dance program with background music and a music promotional video with dance motions. For example, given human motions of a classic ballet program, we can assign music pieces matched to the target human motions, and this example will be shown in the verification in the experiment section. Next, in the second example, this can present to users information of music that they are listening to, i.e., song title, composer, etc. Users can use sounds of music pieces or the user's own dance motion associated with the music as the query for obtaining information on the music. As described above, the application can also use the relationship between human motions and music pieces, and it can be a more flexible information presentation system than the conventional ones. In this way, information directly obtained from users, i.e., users' motions can retain the potential to get various benefits. These schemes are cross-media recommendation schemes and they remove barriers between users and those multimedia contents.

In this article, we deal with recommendation of music pieces from features obtained from users. Among the features, human motions have high-level semantics, and their use is effective for realizing accurate recommendation. Therefore, we try to estimate suitable music pieces from human motions. This is because we consider that correlation extraction between human motions and music pieces becomes feasible using some specific video contents such as dance and music promotional videos. This benefit is also useful in performance verification. Then, we assume that the meaning of "suitable" is emotionally similar. Specifically, in our purpose, the recommendation of suitable music pieces according to human motions is that the recommended music pieces are emotionally similar to the query human motions.

In this article, we propose a new method for cross-media recommendation of music pieces according to human motions based on kernel canonical correlation analysis (CCA) [13]. We use video contents in which video sequences and audio signals contain human motions and music pieces, respectively, as training data for calculating their correlation. Then, using the obtained correlation, estimation of the best matched music piece from a target human motion becomes feasible. It should be noted that several methods of cross-media recommendation have previously been proposed. However, there have been no methods focused on handling data that have various time lengths, i.e., human motions and music pieces. Thus, we propose a cross-media recommendation method that can effectively use characteristics of time series, and we assume that this can be realized using kernel CCA and our defined kernel functions. From the above discussion, the main contribution of the proposed method is handling data that have various time lengths for cross-media recommendation.

In this approach, we have to consider the differences in time lengths. In the proposed method, new kernel functions of human motions and music pieces are introduced into the CCA-based correlation calculation. Specifically, we newly adopt two types of kernel functions, which can represent similarities by effectively using human motions or music pieces having various time lengths, for the kernel CCA-based correlation calculation. First, we define a longest common subsequence (LCSS) kernel for using data having different time lengths. Since the LCSS [14] is commonly used for motion comparison, the LCSS kernel should be suitable for our purpose. It should be noted that kernel functions must satisfy Mercer's theorem [15], but our newly defined kernel function does not necessarily satisfy this theorem. Therefore, we also adopt another type of kernel function, spectrum intersection kernel, that satisfies Mercer's theorem. This function introduces the p-spectrum [16] and is based on the histogram intersection kernel [17]. Since the histogram intersection kernel is known as a function that satisfies Mercer's theorem, the spectrum intersection kernel also satisfies this theorem.

Actually, there have been kernel functions that do not satisfy Mercer's theorem, and there have also been several proposed methods that use such kernel functions. The effectiveness of the above-described methods has also been verified. Thus, we should also verify the effectiveness of our defined kernel function, which does not satisfy Mercer's theorem, i.e., the LCSS kernel. In addition, we should also compare our two newly defined kernel functions experimentally. Therefore, in this article, we introduce two types of kernel functions. Using these two types of kernel functions, the proposed method can directly compare multimedia data that have various time lengths, and this is the main advantage of our method. Thus, the use of these kernel functions effectively provides a solution to the problem of not being able to simply apply sequential data such as human motions and music pieces to cross-media recommendation. Consequently, effective modeling of the relationship using music and human motion data that have various time lengths is realized, and successful music recommendation can be expected.

This article is organized as follows. First, in Section 2, we briefly explain the kernel CCA used for calculating the correlation between human motions and music pieces. Next, in Section 3, we describe our two newly defined kernel functions. Kernel CCA-based music recommendation according to human motion is proposed in Section 4. Experimental results that verify the performance of the proposed method are shown in Section 5. Finally conclusions are given in Section 6.

2 Kernel canonical correlation analysis

In this section, we explain kernel CCA. First, two variables x and y are transformed into Hilbert space H _x and H _y via non-linear maps ϕ _x and ϕ _y . From the mapped results ϕ _x (x) ∈ H _x and ϕ _y (y) ∈ H _y ,^c the kernel CCA seeks to maximize the correlation

$$\rho =\frac{E\left[uv\right]}{\sqrt{E\left[u^2\right]E\left[v^2\right]}}$$

(1)

between

$$u=⟨a,{\varphi }_x\left(x\right)⟩$$

(2)

and

$$v=⟨b,{\varphi }_y\left(y\right)⟩$$

(3)

over the projection directions a and b. This means that kernel CCA finds the directions a and b that maximize the correlation $E\left[uv\right]$ of corresponding projections subject to $E\left[u^2\right]=1$ and $E\left[v^2\right]=1$.

The optimal directions a and b can be found by solving the Lagrangian

$$\mathcal{L}=E\left[uv\right]-\frac{{\lambda }_1}{2}\left(E\left[u^2\right]-1\right)-\frac{{\lambda }_2}{2}\left(E\left[v^2\right]-1\right)+\frac{\eta }{2}\left(\left|\right|a|{|}^2+|\left|b\right|{|}^2\right),$$

(4)

where η is a regularization parameter. The above-computation scheme is called regularized kernel CCA [13]. By taking the derivatives of Equation 4 with respect to a and b, λ₁ = λ₂(= λ) is derived, and the directions a and b maximizing the correlation ρ (= λ) can be calculated.

3 Kernel function construction

Construction of new kernel functions is described in this section. The proposed method constructs two types of kernel functions for human motions and music pieces, respectively. First, we introduce an LCSS kernel as a kernel function that does not satisfy Mercer's theorem. This function is based on the LCSS algorithm [18], which is commonly used for motion or temporal music signal comparison since the LCSS algorithm can compare two temporal signals even if they have different time lengths. Therefore, it seems that this kernel function is suitable for our recommendation scheme. On the other hand, we also introduce a spectrum intersection kernel that satisfies Mercer's theorem. This function is based on the p-spectrum [16], which is generally used for text comparison. The p-spectrum uses the continuity of words. This property is also useful for analyzing the structure of temporal sequential data, i.e., human motions. Thus, the spectrum intersection kernel is also suitable for our recommendation scheme.

For the following explanation, we prepare pairs of human motions and music pieces extracted from the same video contents and denote each pair as a segment. The segments are defined as short terms of video contents that have various time lengths. From the obtained segments, we extract human motion features and music features of the j th (j = 1, 2,..., N) segment as $V_j=\left[v_j\left(1\right),v_j\left(2\right),...,v_j\left(N_{v_j}\right)\right]$ and $M_j=\left[m_j\left(1\right),m_j\left(2\right),...,m_j\left(N_{M_j}\right)\right]$, where $N_{v_j}$ and $N_{M_j}$ are the numbers of components of V _j and M _j , respectively, and N is the number of segments. In V _j and $M_j,v_j\left(l_v\right)\left(l_v=1,2,...,N_{v_j}\right)$ and $m_j\left(l_m\right)\left(l_m=1,2,...,N_{M_j}\right)$ correspond to optical flows [19] and chroma vectors [20], respectively. The optical flow is a simple and representative feature that represents motion characteristics between two successive frames in video sequences and is commonly used for motion comparison. Thus, we adopt the optical flow as temporal components of human motion features. Furthermore, the chroma vector represents tone distribution of music signals at each time. The chroma vector can represent the characteristics of a music signal robustly if it is extracted in a short time. In addition, due to the simplicity of the implementation, we adopted these features in our method. More details of these features are given in Appendices A.1 and A.2.

3.1 Kernel function for human motions

3.1.1 LCSS kernel

In order to define kernel functions for human motions having various time lengths, we firstly explain the LCSS kernel for human motions that uses an LCSS-based similarity in [14]. An LCSS is an algorithm that enables calculation of the longest common part and its length (LCSS length) between two sequences.

Figure 1 shows an example of a table produced by LCSS length of two sequences X = 〈B, D, C, A, B〉 and Y = 〈A, B, C, B, A, B〉. In this figure, the highlighted components represent the common components in two different sequences and LCSS length between X and Y becomes four.

Figure 1

An example of a table based on LCSS length of the sequences X = 〈 B , D , C , A , B 〉 and Y = 〈 A , B , C , B , A , B 〉.

Here, we show the definition of similarity between human motion features. For the following explanations, we denote two human motion features as $V_a=\left[v_a\left(1\right),v_a\left(2\right),...,v_a\left(N_{v_a}\right)\right]$ and $Vb=\left[v_b\left(1\right),v_b\left(2\right),...,v_b\left(N_{v_b}\right)\right]$, where $v_a\left(l_a\right)\left(l_a=1,2,...,N{v}_a\right)$ and $v_b\left(l_b\right)\left(l_b=1,2,...,N_{v_b}\right)$ are components of V _a and V _b , respectively, and $N_{v_a}$ and $N_{v_b}$ are the numbers of components in V _a and V _b , respectively. In addition, v _a (l _a ) and v _b (l _b ) correspond to optical flows extracted in each frame in each video sequence. Note that $N_{v_a}$ and $N_{v_b}$ depend on the time lengths of their segments; that is, they depend on the number of frames of their video sequences. The similarity between V _a and V _b is defined as follows:

$$Si{m}_v\left(V_a,V_b\right)=\frac{LCSS\left(V_a,V_b\right)}{min\left(N_{v_a},N_{v_b}\right)},$$

(5)

where LCSS(V _a ,V _b ) is the LCSS length of V _a and V _b , and it is recursively defined as

$$LCSS\left(V_a,V_b\right)=R_{V_a{V}_b}\left(l_a,l_b\right){|}_{l_a=N_{v_a,l_b=N_{v_b}}},$$

(6)

$$R_{V_a{V}_b}\left(l_a,l_b\right)=\left\{\begin{array}{cc}\hfill 0\hfill & \hfill \mathsf{\text{if}}{N}_{V_a}=0\mathsf{\text{or}}{N}_{V_b}=0,\hfill \\ \hfill 1+R_{V_a{V}_b}\left(l_a-1,l_b-1\right)\hfill & \hfill \mathsf{\text{if}}c\left(v_a\left(l_a\right)\right)=c\left(v_b\left(l_b\right)\right),\hfill \\ \hfill max\left\{R_{V_a{V}_b}\left(l_a-1,l_b\right),R_{V_a{V}_b}\left(l_a,l_b-1\right)\right\}\hfill & \hfill \mathsf{\text{otherwise}},\hfill \end{array}\right.$$

(7)

where c(·) is a cluster number of optical flow. In the proposed method, we apply a k-means algorithm [21] for all optical flows obtained from all segments, and the obtained cluster numbers assigned to the belonging optical flows c(·) are used for easy comparison of two different optical flows. For this purpose, some kinds of quantization or labeling of the temporal variation of the time series seem to be available. In the proposed method, we adopt k-means clustering for its simplicity.

We then define this similarity measure as the LCSS kernel for human motions ${\kappa }_v^{LCSS}\left(\cdot ,\cdot \right)$ as follows:

$${\kappa }_V^{LCSS}\left(V_a,V_b\right)=Si{m}_V\left(V_a,V_b\right).$$

(8)

The above-kernel function can be used for time series having various time lengths. Not only our LCSS kernel but also other kernel functions are known as non-positive semi-definite. Therefore, these do not strictly satisfy Mercer's theorem [15]. Fortunately, kernel functions that do not satisfy Mercer's theorem have been verified to be effective for classification of sequential data using a kernel function in [18].

Furthermore, several methods using kernel functions that do not satisfy the theorem have been proposed in [22, 23]. Also, a sigmoid kernel has been commonly used and is well known as a kernel function which does not satisfy Mercer's theorem. We therefore briefly discuss implications and problems that might emerge using a kernel function that does not satisfy the theorem. In order to satisfy Mercer's theorem, a gram matrix whose elements correspond to values of a kernel function is required to be a positive semi-definite and symmetric matrix. Not only our defined kernel function but also other kernel functions that do not satisfy Mercer's theorem have symmetric and non-positive semi-definite gram matrices. Thus, for the solution based on such kernel functions, several methods have modified eigenvalues of the gram matrices to be greater than or equal to zero. It should be noted that we used our defined kernel functions directly in the proposed method.

3.1.2 Spectrum intersection kernel

Next, we explain the spectrum intersection kernel for human motions. In order to define the spectrum intersection kernel for human motions, we firstly calculate p-spectrum-based features. The p-spectrum [16] is the set of all p-length (contiguous) subsequences that it contains. The p-spectrum-based features on string $\mathcal{X}$ are indexed by all possible subsequences ${\mathcal{X}}_s$ of length p and defined as follows:

$$r_p\left(\mathcal{X}\right)={\left(r_{{\mathcal{X}}_s}\left(\mathcal{X}\right)\right)}_{{\mathcal{X}}_s\in {\mathcal{A}}^p},$$

(9)

where

$$r_{{\mathcal{X}}_s}\left(\mathcal{X}\right)=\mathsf{\text{number}}\mathsf{\text{of}}\mathsf{\text{times}}{\mathcal{X}}_s\mathsf{\text{occurs}}in\mathcal{X},$$

(10)

and $\mathcal{A}$ is the set of characters in strings. For human motion features, we cannot apply the p-spectrum directly since human motion features are defined as sequences of vectors. Therefore, we apply the p-spectrum to sequences of cluster numbers of optical flows as that done for the LCSS kernel. We use the histogram intersection kernel [17] for constructing the spectrum intersection kernel. The histogram intersection kernel κ^HI(·, ·) is a useful kernel function for classification of histogram-shaped features and is defined as follows:

$${\kappa }^{HI}\left(h_a,h_b\right)=\sum _{{}^{i}h=1}^{N^h}min\left\{h_a\left(i_h\right),h_b\left(i_h\right)\right\},$$

(11)

where h _a and h _b are histogram-shaped features, h _a (i _h ) and h _b (i _h ) are the i _h th element (bin) values of h _a and h _b , respectively, and N^his the numbers of bins of histogram-shaped features. Furthermore, ${\sum }_{{}^{i}h=1}^{N^h}{h}_a\left(i_h\right)=1$ and ${\sum }_{{}^{i}h=1}^{N^h}{h}_b\left(i_h\right)=1$ are required to apply the histogram intersection kernel into h _a and h _b . The p-spectrum-based features also have histogram shapes, and they can be applied to the histogram intersection kernel. Note that the sums of elements have to be normalized in the same way as that done for histogram-shaped features. After that, we define this kernel function as the spectrum intersection kernel for human motions ${\kappa }_v^{SI}\left(\cdot ,\cdot \right)$ shown as follows:

$${\kappa }_V^{SI}\left(V_a,V_b\right)={\kappa }^{HI}\left(r_p\left(V_a\right),r_p\left(V_b\right)\right).$$

(12)

The above-kernel function can consider statistical characteristics of human motion features. Since the histogram intersection kernel is positive semi-definite [17], the spectrum intersection kernel can satisfy Mercer's theorem [15]. Note that the above-kernel function is equivalent to the spectrum kernel defined in [16] if we use the simple inner product of p-spectrum-based features instead of the histogram intersection in Equation 12.

3.2 Kernel function for music pieces

3.2.1 LCSS kernel

The kernel functions for music pieces are defined in the same way as those of human motions. First, we show the definition of the LCSS kernel for music pieces. For the following explanations, we denote two music features as $M_a=\left[m_a\left(1\right),m_a\left(2\right),...,m_a\left(N_{M_a}\right)\right]$ and $M_b=\left[m_b\left(1\right),m_b\left(2\right),...,m_b\left(N_{M_b}\right)\right]$, where M _a and M _b are chromagrams [24] and are extracted from segments, $m_a\left(l_a\right)\left(l_a=1,2,...,N_{M_a}\right)$ and $m_b\left(l_b\right)\left(l_b=1,2,...,N_{M_b}\right)$ are components of M _a and M _b , and $N_{M_a}$ and $N_{M_b}$ are the numbers of components of M _a and M _b , respectively. In addition, m _a (l _a ) and m _b (l _b ) are chroma vectors [20] that have 12 dimensions. Since $N_{M_a}$ and $N_{M_b}$ depend on the time lengths of their segments, the similarity between music features is also defined on the basis of the LCSS algorithm. Note that it is desirable that the similarity between an original music piece and its modulated version becomes high since they have similar melodies, base lines, or harmonics. Therefore, we define similarity considering the modulation of music. In the proposed method, we use temporal sequences of chroma vectors, i.e., chromagrams defined in [24], as music features. One of the advantages of the use of 12-dimensional chroma vectors in the chromagrams is that the transposition amount of modulation can be naturally represented only by the amount ζ by which its 12 elements are shifted (rotated). Therefore, the proposed method effectively uses the above characteristic for measuring similarities between chromagrams. For the following explanation, we define the modulated chromagram $M_b^{\zeta }=\left[m_b^{\zeta }\left(1\right),m_b^{\zeta }\left(2\right),..,m_b^{\zeta }\left(N_{M_b}\right)\right]$. Note that $m_b^{\zeta }\left(l_b\right)\left(l_b=1,2,...,N_{M_b}\right)$ represents a modulated chroma vector whose elements are shifted by amount ζ.

The similarity between M _a and M _b is defined as follows:

$$Si{m}_M\left(M_a,M_b\right)=\underset{\zeta }{max}\left\{\frac{LCSS\left(M_a,M_b^{\zeta }\right)}{min\left(N_{M_a},N_{M_b}\right)}\right\},$$

(13)

where $LCSS\left(M_a,M_b^{\zeta }\right)$ is recursively defined as

$$LCSS\left(M_a,M_b^{\zeta }\right)=R_{M_a{M}_b^{\zeta }}\left(l_a,l_b\right){|}_{l_a=N_{M_a,}{l}_b=N_{M_b}},$$

(14)

$$R_{M_a{M}_b^{\zeta }}\left(l_a,l_b\right)=\left\{\begin{array}{cc}\hfill 0\hfill & \hfill \mathsf{\text{if}}{l}_a=0\mathsf{\text{or}}{l}_b\mathsf{\text{ = 0,}}\hfill \\ \hfill 1+R_{M_a{M}_b^{\zeta }}\left(l_a-1,l_b-1\right)\hfill & \hfill \mathsf{\text{if}}Si{m}_{\tau }\left\{m_a\left(l_a\right),m_b^{\zeta }\left(l_b\right)\right\}>T_h,\hfill \\ \hfill max\left\{R_{M_a{M}_b^{\zeta }}\left(l_a-1,l_b\right),R_{M_a{M}_b^{\zeta }}\left(l_a,l_b-1\right)\right\}\hfill & \hfill \mathsf{\text{otherwise}}\mathsf{\text{.}}\hfill \end{array}\right.$$

(15)

$$si{m}_{\tau }\left\{m_a\left(l_a\right),m_b^{\zeta }\left(l_b\right)\right\}=1-\frac{\left|{\tilde{m}}_a\left(l_a\right){\tilde{m}}_b^{\zeta }\left(l_b\right)\right|}{\sqrt{12}}$$

(16)

$${\tilde{m}}_a\left(l_a\right)=\frac{m_a\left(l_a\right)}{\underset{\tau }{max}{m}_{a,\tau }\left(l_a\right)},$$

(17)

$${\tilde{m}}_b^{\zeta }\left(l_b\right)=\frac{m_b^{\zeta }\left(l_b\right)}{\underset{\tau }{max}{m}_{b,\tau }^{\zeta }\left(l_b\right)},$$

(18)

where T _h (= 0.8) is a positive constant for determining the fitness between two different chroma vectors, Sim _τ {·, ·} is a similarity between chroma vectors defined in [20], ${\tilde{m}}_a\left(l_a\right)$ and ${\tilde{m}}_b^{\zeta }\left(l_b\right)$ are normalized chroma vectors, m_{a, τ}(l _a ) and $m_{b,\tau }^{\zeta }\left(l_b\right)$ are elements of the chroma vectors, and τ corresponds to tone, i.e., "C", "D#", "G#", etc. Note that the effectiveness of Sim _τ {·, ·} is verified in [20]. We then define this similarity as the LCSS kernel for music pieces ${\kappa }_M^{LCSS}\left(\cdot ,\cdot \right)$ described as follows:

$${\kappa }_M^{LCSS}\left(M_a,M_b\right)=Si{m}_M\left(M_a,M_b\right).$$

(19)

3.2.2 Spectrum intersection kernel

Next, we explain the spectrum intersection kernel for music pieces. In order to define the spectrum intersection kernel for music pieces, we firstly calculate p-spectrum-based features in the same way as those of human motions. It should be noted that the proposed method cannot calculate the p-spectrum from music features directly since the music features are defined as sequences of vectors. Therefore, we transform all of the vector components of music features into characters, such as alphabetic letters or numbers, based on hierarchical clustering algorithms, where the characters correspond to cluster numbers. For clustering the vector components, the modulation of music should also be considered in the same way as the LCSS kernel for music pieces. Therefore, clustering considering modulation is necessary. The procedures of this scheme are shown as follows.

Step 1: Calculation of optimal modulation amounts between music features

First, the proposed method calculates the optimal modulation amounts ζ^abbetween two music features M _a and M _b . This scheme is based on LCSS-based similarity and is defined as follows:

$${\zeta }^{ab}=\underset{\zeta }{argmax}\left\{\frac{LCSS\left(M_a,M_b^{\zeta }\right)}{min\left(N_{M_a},N_{M_b}\right)}\right\}.$$

(20)

The optimal modulation amount ζ^abis calculated for all pairs.

Step 2: Similarity measurement between chroma vectors using the obtained optimal modulation amounts

Similarity between vector components, which is that between chroma vectors, is calculated using the obtained optimal modulation amounts. For example, the similarity between chroma vectors m _a (l _a ) and m _b (l _b ), which are the l _a th and l _b th components of two arbitrary music features M _a and M _b , respectively, is calculated using the obtained optimal modulation amount ζ^aband Equation 16 as follows:

$$Si{m}_c\left\{m_a\left(l_a\right),m_b\left(l_b\right)\right\}=1-\frac{|{\tilde{m}}_a\left(l_a\right)-{\tilde{m}}_b^{{\zeta }^{ab}}\left(l_b\right)|}{\sqrt{12}}.$$

(21)

The above similarity is calculated between two different chroma vectors for all music features.

Step 3: Clustering chroma vectors based on the obtained similarities

Using the obtained similarities, the two most similar chroma vectors are assigned to the same cluster for clustering chroma vectors. This scheme is based on the single linkage method [25]. The merging scheme is recursively performed until the number of clusters becomes less than K _M .

Using the clustering results, the proposed method calculates transformed music features $m_j^{*}={\left[m_j^{*}\left(1\right),m_j^{*}\left(2\right),\dots ,m_j^{*}\left({N_M}_{{}_j}\right)\right]}^{\prime }\left(j=1,2,\dots ,N\right)$, where $m_j^{*}\left(l_M\right)\left(l_M=1,2,\dots ,{N_M}_{{}_j}\right)$ is a cluster number assigned to a corresponding chroma vector. Note that vector/matrix transpose is denoted by the superscript ' in this article. The proposed method then calculates p-spectrum-based features from $m_j^{*}$. For the following explanations, we denote two transformed music features as $m_a^{*}={\left[m_a^{*}\left(1\right),m_a^{*}\left(2\right),\dots ,m_a^{*}\left(N_{M_a}\right)\right]}^{\prime }$ and $m_b^{*}={\left[m_b^{*}\left(1\right),m_b^{*}\left(2\right),\dots ,m_b^{*}\left(N_{M_b}\right)\right]}^{\prime }$, where $m_a^{*}$ and $m_b^{*}$ are vectors transformed from M _a and M _b , respectively, and $m_a^{*}\left(l_a\right)\left(l_a=1,2,\dots ,N_{M_a}\right)$ and $m_b^{*}\left(l_b\right)\left(l_b=1,2,\dots ,N_{M_b}\right)$ are the cluster numbers assigned to m _a (l _a ) and m _b (l _b ), respectively. Then, the spectrum intersection kernel for music pieces is calculated in the same way as that for human motions and is defined as follows:

$${\kappa }_M^{SI}\left(m_a,m_b\right)={\kappa }^{HI}\left(r_p\left(m_a^{*}\right),r_p\left(m_b^{*}\right)\right).$$

(22)

4 Kernel CCA-based music recommendation according to human motion

A method for recommending music pieces suitable for human motions is presented in this section. An overview of the proposed method is shown in Figure 2. In our cross-media recommendation method, pairs of human motions and music pieces that have a close relationship are necessary for effective correlation calculation. Therefore, we prepare these pairs extracted from the same video contents as segments. From the obtained segments, we extract human motion features and music features. More details of these features are given in Appendices A.1 and A.2. By applying kernel CCA to the features of human motions and music pieces, the proposed method calculates their correlation. In this approach, we define new kernel functions that can be used for data having various time lengths and introduce them into the kernel CCA.

Figure 2

Overview of the proposed method. The left and right parts in this figure represent the correlation calculation phase and the recommendation phase, respectively, in the proposed method.

Therefore, the proposed method can calculate the correlations by considering their sequential characteristics. Then, effective modeling of the relationship using human motions and music pieces having various time lengths is realized, and successful music recommendation can be expected.

First, we define the features of V _j and M _j (j = 1, 2,..., N) in the Hilbert space as ϕ _v (vec[V _j ]) and ϕ _M (vec[M _j ]), where vec[·] is the vectorization operator that turns a matrix into a vector. Next, we find features

$$s_j=A^{\prime }\left({\varphi }_V\left(\mathsf{\text{vec}}\left[V_j\right]\right)-{\bar{\varphi }}_V\right),$$

(23)

$$t_j=B^{\prime }\left({\varphi }_M\left(\mathsf{\text{vec}}\left[M_j\right]\right)-{\bar{\varphi }}_M\right),$$

(24)

$$A=\left[a_1,a_2,\dots ,a_D\right],$$

(25)

$$B=\left[b_1,b_2,\dots ,b_D\right],$$

(26)

where ${\bar{\varphi }}_V$ and ${\bar{\varphi }}_M$ are mean vectors of ϕ _v (vec[V _j ]) and ϕ _M (vec[M _j ]) (j = 1, 2,..., N), respectively. The matrices A and B are coefficient matrices whose columns a _d and b _d (d = 1, 2,..., D), respectively, correspond to the projection directions in Equations 2 and 3, where the value D is the dimension of A and B. Then, we define a correlation matrix Λ whose diagonal elements are the correlation coefficients λ _d (d = 1,2,..., D). The details of the calculation of A, B, and Λ are shown as follows.

In order to obtain A, B, and Λ, we use the regularized kernel CCA shown in the previous section. Note that the optimal matrices A and B are given by

$$A={\Xi }_{v}H{E}_v,$$

(27)

$$B={\Xi }_{M}H{E}_M,$$

(28)

$${\Xi }_V=\left[{\varphi }_V\left(\mathsf{\text{vec}}\left[V_1\right]\right),{\varphi }_V\left(\mathsf{\text{vec}}\left[V_2\right]\right),\dots ,{\varphi }_V\left(\mathsf{\text{vec}}\left[V_N\right)\right]\right],$$

(29)

$${\Xi }_M=[{\varphi }_M(\text{vec}[M_1]),{\varphi }_M(\text{vec[}{M}_2]),\dots ,{\varphi }_M(\text{vec[}{M}_N])],$$

(30)

where $E_V=\left[e_{V_1},e_{V_2},\dots ,e_{V_D}\right]$ and $E_M=\left[{e_M}_{{}_1},{e_M}_{{}_2},\dots ,{e_M}_{{}_D}\right]$ are N × D matrices. Furthermore,

$$H=I-\frac{1}{N}11^{\prime }$$

(31)

is a centering matrix, where I is the N × N identity matrix, and 1 = [1,..., 1]' is an N × 1 vector. From Equations 27 and 28, the following equations are satisfied:

$$a_d={\Xi }_{V}H{e}_{V_d},$$

(32)

$$b_d={\Xi }_{M}H{e}_{M_d}.$$

(33)

Then, by calculating the optimal solution ${e_V}_{{}_d}$ and $e_{M_d}\left(d=1,2,\dots ,D\right)$, A and B are obtained. In the same way as Equation 4, we calculate the optimal solution ${e_V}_{{}_d}$ and $e_{M_d}$ that maximizes

$$\mathcal{L}={e^{\prime }}_{V}L{e}_M-\frac{\lambda }{2}\left({e^{\prime }}_{V}M{e}_V-1\right)-\frac{\lambda }{2}\left({e^{\prime }}_{M}P{e}_M-1\right),$$

(34)

where e _V , e _M , and λ correspond to ${e_V}_{{}_d},e_{M_d}$, and λ _d , respectively. In the above equation, L, M, and P are calculated as follows:

$$L=\frac{1}{N}H{K}_{V}HH{K}_{M}H,$$

(35)

$$M=\frac{1}{N}H{K}_{V}HH{K}_{V}H+{\eta }_{1}H{K}_{V}H,$$

(36)

$$P=\frac{1}{N}H{K}_{M}HH{K}_{M}H+{\eta }_{2}H{K}_{M}H.$$

(37)

Furthermore, η₁ and η₂ are regularization parameters, and $K_V\left(={\Xi }_V^{\prime }{\Xi }_V\right)$ and $K_M\left(={\Xi }_M^{\prime }{\Xi }_M\right)$ are matrices whose elements are defined as values of the corresponding kernel functions defined in Section 3. By taking derivatives of Equation 34 with respect to e _V and e _M , optimal e _V , e _M , and λ can be obtained as solutions of following eigenvalue problems:

$$M^{-1}L{P}^{-1}{L}^{\prime }{e}_V={\lambda }^2{e}_V,$$

(38)

$$P^{-1}{L}^{\prime }{M}^{-1}L{e}_M={\lambda }^2{e}_M,$$

(39)

where λ is obtained as an eigenvalue, and the vectors e _V and e _M are, respectively, obtained as eigenvectors. Then, the d th (d = 1, 2,..., D) eigenvalue of λ becomes λ _d , where λ₁ ≥ λ₂ ≥ ... ≥ λ _D . Note that the dimension D is set to a value for which the cumulative proportion obtained from λ _d (d = 1,2,...,D) becomes larger than a threshold. Furthermore, the eigenvectors e _V and e _M corresponding to λ _d become ${e_V}_{{}_d}$ and $e_{M_d}$, respectively.

From the obtained matrices A, B, and Λ, we can estimate the optimal music features from given human motion features, i.e., we can select the best matched music pieces according to human motions. An overview of music recommendation is shown in Figure 3. When a human motion feature V _in is given, we can select the predetermined number of music pieces according to the query human motion that minimize the following distances:

Figure 3

Overview of music recommendation according to human motion.

$$d=\parallel t_{in}-{\widehat{t}}_i{\parallel }^2\left(i=1,2,\dots ,M_t\right),$$

(40)

where t _in and ${\widehat{t}}_i$ are, respectively, the query human motion feature and music features in the database ${\widehat{M}}_i\left(i=1,2,\dots ,M_t\right)$ transformed into the same feature space shown as follows:

$$\begin{array}{ll}\hfill {\widehat{t}}_i& =B^{\prime }\left({\varphi }_M\left(\mathsf{\text{vec}}\left[{\widehat{M}}_i\right]\right)-{\bar{\varphi }}_M\right)\\ =E_M^{\prime }\left({\kappa }_{{\widehat{M}}_i}-\frac{1}{N}{K}_{M}1\right),\end{array}$$

(41)

$$\begin{array}{ll}\hfill t_{in}& =\Lambda A^{\prime }\left({\varphi }_V\left(\mathsf{\text{vec}}\left[V_{in}\right]\right)-{\bar{\varphi }}_V\right)\\ ={\Lambda E}_V^{\prime }\left({{\kappa }_V}_{{}_{in}}-\frac{1}{N}{K}_{V}1\right),\end{array}$$

(42)

and M _t is the number of music pieces in the database. Note that ${{\kappa }_V}_{{}_{in}}$ is an N × 1 vector whose q th elements are ${\kappa }_V^{LCSS}\left(V_{in},V_q\right)$ or ${\kappa }_V^{SI}\left(V_{in},V_q\right)$, and ${\kappa }_{{\widehat{M}}_i}$ is an N × 1 vector whose q th elements are ${\kappa }_M^{LCSS}\left({\widehat{M}}_i,M_q\right)$ or ${\kappa }_M^{SI}\left({\widehat{M}}_i,M_q\right)$.

As described above, we can estimate the best matched music pieces according to the human motions. The proposed method calculates the correlation between human motions and music pieces based on the kernel CCA. Then, the proposed method introduces the kernel functions that can be used for time series having various time lengths based on the LCSS or p-spectrum. Therefore, the proposed method enables calculation of the correlation between human motions and music pieces that have various time lengths. Furthermore, effective correlation calculation and successful music recommendation according to human motion based on the obtained correlation are realized.

5 Experimental results

The performance of the proposed method is verified in this section. For the experiments, 170 segments were manually extracted. In the experiments, we used video contents of three classic ballet programs. Of the 170 segments, 44 were from Nutcracker, 54 were from Swan Lake, and 72 were from Sleeping Beauty. Each segment consisted of only one human motion and the background music did not change in the segment. In addition, camera change was not included in the segment. The audio signals in each segment were mono channel, 16 bits per sample and were sampled at 44.1 [kHz]. Human motion features and music features were extracted from the obtained segments.

For evaluation of the performance of our method, we used videos of classic ballet programs. However, there were some differences between motions extracted from classic ballet programs and those extracted in our daily life. In cross-media recommendation, we have to consider whether or not we should recommend contents that have the same meanings as those of queries. For example, when we recommend music pieces from the user's information, recommendation of sad music pieces is not always suitable if the user seems to be sad. Our approach also has to consider the above point. In this article, we focus on extraction of the relationship between human motions and music pieces and perform the recommendation based on the extracted relationship. In addition, we have to prepare some ground truths for evaluation of the proposed method. Therefore, we used videos of classic ballet programs since the human motions and music pieces extracted from the same videos of classic ballet programs had strong and direct relationships.

In order to evaluate the performance of our method, we also prepared five datasets #1 to #5 that were pairs of 100 segments for training (training segments) and 70 segments for testing (testing segments), i.e., a simple cross-validation scheme. It should be noted that we randomly divided the 170 segments into five datasets. The reason for dividing the 170 segments into five datasets was to perform various verifications by changing the combination of test segments and training segments. Then, the number of datasets (five) was simply determined. Furthermore, the training segments and testing segments were obtained from the above prepared 170 segments. For the experiments, 12 kinds of tags representing expression marks in music shown in Table 1 were used. We examined whether each tag could be used for labeling human motions and music pieces. Thus, tags that seemed to be difficult to use for these two media types were removed in this process. Then, we could obtain the above 12 kinds of tags. One suitable tag was manually selected and annotated to each segment for performance verification. In the experiments, one person with musical experience annotated the label that was the best matched to each segment. Generally, annotation should be performed by several people. However, since labels, i.e., expression marks in music, were used in the experiment, it was necessary to have the ground truths made by a person who had knowledge of music. Thus, in the experiment, only one person annotated the labels.

Table 1 Description of expression marks

First, we show the recommended results (see Additional file 1). In this file, we show original video contents and recommended video contents. The background music pieces of recommended video contents are not original but are music pieces recommended by our method. These results show that our method can recommend a suitable music piece for a human motion.

Additional file 1:Recommended results. Additional file 1.mov; Description of data: This video content shows our recommendation results. In this video content, original video contents and recommended results, whose video contents' background music are music pieces recommended by our method, are shown. (MOV 6 MB)

Next, we quantitatively verify the performance of the proposed method. In this simulation, we verify the effectiveness of our kernel functions. In the proposed method, we define two types of kernel functions, LCSS kernel and spectrum intersection kernel, for human motions and music pieces. Thus, we experimentally compare our two newly defined kernel functions. Using combinations of the kernel functions, we prepared four simulations "Simulation 1"-"Simulation 4", as follows:

• Simulation 1 used the LCSS kernel for both human motions and music pieces.

• Simulation 2 used the spectrum intersection kernel for both human motions and music pieces.

• Simulation 3 used the spectrum intersection kernel for human motions and the LCSS kernel for music pieces.

• Simulation 4 used the LCSS kernel for human motions and the spectrum intersection kernel for music pieces.

These simulations were performed to verify the effectiveness of our two newly defined kernel functions for human motions and music pieces. For the following explanations, we denote the LCSS kernel as "LCSS-K" and the spectrum intersection kernel as "SI-K". In addition, for the experiments, we used the following criterion:

$$\mathsf{\text{Accuracy Score}}=\frac{{\sum }_{i_1=1}^{70}{Q}_{i_1}^1}{70},$$

(43)

where the denominator corresponds to the number of testing segments. Furthermore, $Q_{i_1}^1\left(i_1=1,2,\dots ,70\right)$ is one if the tags of three recommended music pieces include the tag of the human motion query.

Otherwise, $Q_{i_1}^1$ is zero. It should be noted that the number of recommended music pieces (three) was simply determined. We next explain how the number of recommended music pieces affects the performance of our method. For the following explanation, we define the terms "over-recommendation" and "mis-recommendation". Over-recommendation means that the recommended results tend to contain music pieces that are not matched to the target human motions as well as matched music pieces, and mis-recommendation means that music pieces that are matched to the target human motions tend not to be correctly selected as the recommendation results. There is a tradeoff relationship between over-recommendation and mis-recommendation. That is, if we increase the number of recommended results, over-recommendation increases and mis-recommendation decreases. On the other hand, if we decrease the number of recommended results, over-recommendation decreases and mis-recommendation increases. Furthermore, we evaluate the recommendation accuracy according to the above criterion. Figure 4 shows that the accuracy score of simulation 1 was higher than accuracy scores of the other simulations. This is because the LCSS kernel can effectively compare human motions and music pieces respectively having different time lengths. Note that in these simulations, we used bi (p = 2)-gram for calculating p-spectrum-based features shown in Equation 9, the number of clusters for chroma vectors is set to K _M = 500 and the parameters in our method are shown in Tables 2, 3, 4 and 5. All of these parameters are empirically determined, and they are set to values that provide the highest accuracy. More details of parameter determination are given in Appendix.

Figure 4

Accuracy scores in each simulation. #1 to #5 are dataset numbers and "AVERAGE" is the average value of the accuracy scores for the datasets.

Table 2 Description of parameters used in Simulation 1

Table 3 Description of parameters used in Simulation 2

Table 4 Description of parameters used in Simulation 3

Table 5 Description of parameters used in Simulation 4

In the following, we discuss the results obtained. First, we discuss the influence of our human motion features. The features used in our method are based on optical flow and extracted between two regions that contain a human corresponding to two successive frames. This feature can represent movements of arms, legs, hands, etc. However, this feature cannot represent global human movements. This is an important factor for representing motion characteristics of classic ballet. For accurate relationship extraction between human motions and music pieces, it is necessary to improve human motion features into features that can also represent global human movement. This can be complemented using information obtained by much more accurate sensors such as kinect.^d

Next, we discuss the experimental conditions. In the experiments with the proposed method, we used tags, i.e., expression marks in music, as ground truths. This was annotated to each segment. However, this annotation scheme does not consider the relationship between tags. For example, in Table 1, "agitato" and "appassionato" have similar meanings. Thus, the choice of the 12 kinds of tags might be not suitable. It might be necessary to reconsider the choice tags. Also, we found that it is more important to introduce the relationship between tags into our defined accuracy criteria. However, it is difficult to quantify the relationship between them. Thus, we used only one tag for each segment. This can also be expected by the results of subjective evaluation in next experiment.

We also used comparative methods for verifying performance of the proposed method. For the comparative method, we exchanged the kernel functions into gaussian kernel ${\kappa }^{\mathsf{\text{G - K}}}\left(x,y\right)=exp\left(-\frac{\parallel x-y{\parallel }^2}{2{\sigma }^2}\right)\left(\mathsf{\text{G - K}}\right)$, sigmoid kernel κ^S-K(x, y) = tanh(α x'y + β) (S-K), and linear kernel κ^L-K(x, y) = x'y (L-K). In this experiment, we set parameters σ(= 5.0), α(= 5.0), and β(= 3.0). It should be noted that these kernel functions cannot be applied to our human motion features and music features directly since the features have various dimensions. Therefore, we simply used the time average of optical flow-based vectors, $v_j^{\mathsf{\text{avg}}}$, for human motion features and the time average of chroma vectors, $m_j^{\mathsf{\text{avg}}}$, for music features. Then, we applied the above three types of kernel functions to the obtained features. Figure 5 shows the results of comparison for each kernel function. These results show that our kernel functions are more effective than other kernel functions. The results also show that it is important to consider the temporal characteristic of data, and our kernel function can successfully consider this characteristic. Note that in this comparison, we used parameters that provide the highest accuracy. The parameters are shown in Tables 6, 7 and 8.

Figure 5

Accuracy comparison in each kernel. #1 to #5 are dataset numbers and "AVERAGE" is the average value of the accuracy scores for the datasets.

Table 6 Description of parameters used in gaussian kernel

Table 7 Description of parameters used in sigmoid kernel

Table 8 Description of parameters used in linear kernel

Finally, we show results of subjective evaluation for our recommendation method. We performed subjective evaluation using 15 subjects (User1-User15). Table 9 shows the profiles of the subjects. In the evaluation, we used video contents which consisted of video sequences and music pieces. In the video contents, each video sequence included one human motion, and each music piece was a recommended result by the proposed method according to the human motion. The tasks of the subjective evaluation were as follows:

Table 9 Profiles of the subjects

• 1. Subjects watched each video content, whose video sequence was a target classic ballet scene and whose music was recommended by the proposed method.

• 2. Subjects determined whether the target classic ballet scene and the recommended music pieces were matched or not. Specifically, they answered yes or no.

• 3. Procedures 1 and 2 were repeated for 210 video contents.

In the subjective evaluation, we used the recommended results obtained by Simulation 1 in the above-described experiment. We also used datasets #1 and #2 for the subjective evaluation. In the evaluation, we showed the top three recommended results for each query human motion (query segment). Then, 70 query segments were examined and 210 recommended results were obtained for each dataset.

Furthermore, we used two criteria, "Accuracy Score 2" and "Accuracy Score 3", for verifying the performance. Accuracy Score 2 is defined as follows:

$$\mathsf{\text{Accuracy}}\mathsf{\text{Score}}2=\frac{{\sum }_{i_{2=1}}^{70}{Q}_{i_2}^2}{70},$$

(44)

where the denominator corresponds to the number of query segments. $Q_{i_2}^2\left(i_2=1,2\dots ,70\right)$ is one if one or some of the recommended three music pieces at least subjects determined the query human motion and its music piece were matched. Otherwise, $Q_{i_2}^2$ is zero. In addition, Accuracy Score 3 is the ratio of assessment results for 210 music pieces and is defined as follows:

$$\mathsf{\text{Accuracy Score }}3=\frac{{\sum }_{i_{3=1}}^{210}{Q}_{i_3}^3}{210},$$

(45)

where the denominator corresponds to the number of recommended music pieces. Furthermore, $Q_{i_3}^3\left(i_3=1,2,\dots ,210\right)$ is one if subjects determined the query human motion and its music piece matched. Otherwise, $Q_{i_3}^3$ is zero. Table 10 shows the results of each score in the subjective evaluation. From the results, both scores show higher recommendation accuracy than that of the quantitative evaluation. Therefore, the results of the subjective evaluation confirmed the effectiveness of our method.

Table 10 Accuracy of subjective evaluation of each user in Dataset #1 and Dataset #2

6 Conclusions

In this article, we have presented a method for music recommendation according to human motion based on the kernel CCA-based relationship. In the proposed method, we newly defined two types of kernel functions. One is a sequential similarity-based kernel function that uses the LCSS algorithm, and the other is a statistical characteristic-based kernel function that uses the p-spectrum. Using these kernel functions, the proposed method enables calculation of the correlation that can consider their sequential characteristics. Furthermore, based on the obtained correlation, the proposed method enables accurate music recommendation according to human motion.

In the experiments, recommendation accuracy was sensitive to the parameters. It is desirable that these parameters be adaptively determined from the datasets. Thus, we need to complement this determination algorithm. Feature selection of the human motions and music pieces is also needed for more accurate extraction of the relationship between human motions and music pieces. These topics will be the subjects of subsequent studies.

Endnotes

^aIn this article, we simply denote "retrieval and recommendation" as recommendation hereafter. ^bIn this article, video sequences are defined as data that contain only visual signals, and video contents are defined as data that contain both visual signals and audio signals. ^cIn this section, we assume that $E\left[{\varphi }_x\left(x\right)\right]=0$ and $E\left[{\varphi }_y\left(y\right)\right]=0$ for brief explanation, where $E\left[\cdot \right]$ denotes the sample average of the random variates. ^dhttp://www.xbox.com/en-US/Kinect.

Appendix A: Feature extraction

In this article, we use human motion features and music features. Here, each feature extraction is explained in detail. Segments are extracted from video contents, i.e., video contents are separated into some segments S _j (j = 1,2,..., N). Then, human motion features and music features are extracted from each segment. In this appendix, we explain methods for extraction of human motion features and music features in A.1 and A.2, respectively.

A.1 Extraction of human motion features

First, the proposed method separates segments S _j into frames $f_j^k\left(k=1,2,\dots ,N_j\right)$, where N _j is the number of frames in segment S _j . Furthermore, a rectangular region including one human is clipped from each frame, and they are regularized to the same size. In this article, we assume that this rectangular region has previously been obtained. Deciding the rectangular regions including humans might be difficult. However, there are several methods for extracting/deciding human regions from video sequences [26, 27]. These methods achieved accurate human region detection by combining visual information and sensor information such as kinect,^d using a stereo-camera, or using a camera for which position is calibrated. Although we extract the rectangular region manually for simplicity, we consider that a certain precision can be guaranteed using these methods.

Next, we show the calculation of optical flow-based vectors. For calculating optical flows from segments, we firstly divide regions of frame $f_j^k$ into blocks ${\mathcal{B}}_j^b\left(b=1,2,\dots ,N^{{\mathcal{B}}_j}\right)$, where $N^{{\mathcal{B}}_j}\left(=1600\right)$ is the number of blocks in each frame. Then, based on the Lucas-Kanade Algorithm [19], the optical flow in each block ${\mathcal{B}}_i^b$ is calculated between two successive regions from $f_j^{k+1}$ to $f_j^k$ for all segments S _j . Then, we obtain optical flow-based vectors $v_j\left(k\right)\left(k=1,2\dots ,N_{V_j}\right)$ containing vertical and horizontal direction optical flow values for all blocks. Then, $N_{v_j}$ corresponds to N _j -1.

In this article, the human motion feature V _j of segment S _j is obtained as the sequence of the optical flow-based vector v _j (k). The features obtained by the above procedure represent the temporal characteristics of human motions.

A.2 Extraction of music features

The proposed method uses chromagrams [24]. A chromagram represents the temporal sequence of chroma vectors over time and is calculated from each segment. Furthermore, the chroma vector represents magnitude distribution on the chroma that is assigned into 12 pitch classes within an octave, and thus the chroma vector has 12 dimensions. The 12-dimensional chroma vector m(t) is extracted from the magnitude spectrum Ψ _τ (f _Hz , t), which is calculated using short-time Fourier transform (STFT), where f _Hz is frequency and t is time in an audio signal. The τ(τ = 1, 2,...,12)th element of m(t) corresponds to a pitch class of equal temperament and is defined as follows:

$$m_{\tau }\left(t\right)=\sum _{h=Oc{t}_L}^{Oc{t}_H}{\int }_{-\infty }^{\infty }BP{F}_{\tau ,h}\left(f_{Hz}\right)\Psi \left(f_{Hz},t\right)d{f}_{Hz},$$

where Oct _H and Oct _L represent the highest and lowest octave positions, respectively. Furthermore, BPF _τ,h (f _Hz ) is a bandpass filter that passes the signal at the log-scale frequency F _τ,h (in cents) of pitch class τ (chroma) in octave position h (height) as shown in the following equation:

$$F_{\tau ,h}=1200h+100\left(\tau -1\right).$$

We define a chromagram that represents a temporal sequence of 12-dimensional chroma vectors extracted by the above procedure in segment S _j as the music features $M_j=\left[m_j\left(1\right),m_j\left(2\right),\dots ,m_j\left(N_{M_j}\right)\right]$, where $N_{M_j}$ is the number of components of M _j . Details of the chroma vector and the chromagram are shown in [20].

Appendix B: Parameter determination

In this section, we explain the parameter determination. First, for the determination of parameters, we performed experiments to show the relationship between the accuracy score and the parameters. Figure 6 shows the relationships between the accuracy score and parameters in Simulation 1. From the obtained results, it can be seen that the kernel CCA-based approach tends to be sensitive for the parameters. It should be noted that in the dataset used for the experiments, there are quite different types of pairs of human motions and music pieces. Then, for similar pairs of human motions and music pieces, we will be able to use fixed parameters and obtain accurate results. Therefore, it can be seemed that stable recommendation accuracy scores are achieved using parameters that are determined from datasets that have similar characteristics. This means that for stable recommendation, some schemes performing clustering and classification of the contents become necessary as pre-procedures. The other simulations and other database are also sensitive the same as the shown results. For the above reasons, we used the parameters that provided the highest accuracy. Thus, the parameters were not determined by cross-validation. However, we recognized that such parameter should be determined by the cross-validation. This is our future work.

Figure 6

Relationships between η ₁ , η ₂ , K _c , and Accuracy Score in Simulation 1. (a) Relationship between η₁ and Accuracy Score, (b) Relationship between η₂ and Accuracy Score, and (c) Relationship between K _c and Accuracy Score. For examining each parameter, the other parameters are fixed. Then, in (b), datasets #3, #4 and #5 have almost the same characteristics.