A dependent multi-label classiﬁcation method derived from the k -nearest neighbor rule

In multi-label classiﬁcation, each instance in the training set is associated with a set of labels, and the task is to output a label set whose size is unknown a priori for each unseen instance. The most commonly-used approach for multi-label classiﬁcation is where a binary classiﬁer is learned independently for each possible class. However, multi-labeled data generally exhibit relationships between labels, and this approach fails to take such relationships into account. In this paper, we describe an original method for multi-label classiﬁcation problems derived from a Bayesian version of the k -Nearest Neighbor ( k -NN) rule. The method developed here is an improvement on an existing method for multi-label classiﬁcation, namely multi-label k -NN, which takes into account the dependencies between labels. Experiments on simulated and benchmark datasets show the usefulness and the eﬃciency of the proposed approach as compared to other existing methods.


Introduction
Traditional single-label classification assigns an object to exactly one class, from a set of Q disjoint classes. Multi-label classification is the task of assigning an instance simultaneously to one or multiple classes. In other words, the target classes are not exclusive: an object may belong to an unrestricted set of classes, rather than to exactly one class. For multi-labeled data, an instance may belong to more than one class not because of ambiguity (fuzzy membership), but because of multiplicity (full membership) [1]. Note that traditional supervised learning problems (binary or multi-class) can be regarded as special cases of the problem of multi-label learning, where instances are restricted to belonging to a single class.
Recently, multi-label classification methods have been increasingly required by modern applications where it is quite natural for some instances to belong to several classes at once. Typical examples of multi-label problems are text categorization, functional genomics and scene classification. In text categorization, each document may belong to multiple topics, such as arts and humanities [11] [17] [6] [8]; in gene functional analysis, each gene may be associated with a set of functional classes, such as energy, metabolism and cellular biogenesis [7]; in natural scene classification, each image may belong to several image types at the same time, such as sea and sunset [1].
A common approach to a multi-label learning problem is to transform it into one or more single-label problems. The best known transformation method is the binary relevance (BR) approach [21]. This approach transforms a multi-label classification problem with Q possible classes into Q single-label classification problems. The qth single-label classification problem (q ∈ {1, . . . , Q}) consists in separating the instances belonging to class ω q from the others. This problem is solved by training a binary classifier (0/1 decision) where each instance in the training set is considered to be positive if it belongs to ω q , and negative otherwise. The output of the multi-label classifier is determined by combining the decisions provided by the different binary classifiers. The BR approach tacitly assumes that labels can be assigned independently: when one label provides information about another, the binary classifier fails to capture this effect. For example, if a news article belongs to a "music" category, it is very likely that it also belongs to an "entertainment" category. Although the BR approach is generally criticized for its assumption of label independencies [15] [2], it is a simple, intuitive approach that has the advantage of having low computational complexity. In [29], the authors present a Bayesian multi-label k-nearest neighbor (MLkNN) approach where, in order to assign a set of labels to a new instance, a decision is made separately for each label by taking into account the number of neighbors containing the label to be assigned. This method therefore fails to take into account the dependency between labels.
In this paper, we present a generalization of the MLkNN-based approach to multi-label classification problems where the dependencies between classes are considered. We call this method DMLkNN, for Dependent Multi-Label k-Nearest Neighbor. The principle of the method is as follows. For each unseen instance, we identify its k-NNs in the training set. According to the class membership of neighboring instances, a global maximum a posteriori (MAP) principle is used in order to assign a set of labels to the new unseen instance. Note that unlike MLkNN, in order to decide whether a particular label should be included among the unseen instance's labels, the global MAP rule takes into account the numbers of different labels in the neighborhood, instead of considering only the number of neighbors having the label in question.
Note that this paper is an extension of a previously published conference paper [27]. Here, the method is more thoroughly interpreted and discussed. Extensive comparisons on several real world datasets and with some stateof-the-art methods are added in the experimental section. In addition, we provide an illustrative example on a simulated dataset, where we explain step by step the principle of our algorithm.
The remainder of the paper is organized as follows. Section 2 presents related work. Section 3 describes the principle of multi-label classification and the notion of label dependencies. Section 4 introduces the DMLkNN method and its implementation. Section 5 presents some experiments and discusses the results. Finally, Section 6 summarizes this work and makes concluding remarks.

Related work
Several methods have been proposed in the literature for multi-label learning. These methods can be categorized into two groups. A first group contains the indirect methods that transform a multi-label classification problem into binary classification problems (a binary classifier for each class or pairwise classifiers) [2] [1] or into multi-class classification problem (each subset of classes is considered as a new class) [21]. A second group consists in extending common learning algorithms and making them able to manipulate 3 hal-00655629, version 1 -31 Dec 2011 multi-label data directly [26]. Some multi-label classification methods are briefly described below.
In [28], an adaptation of the traditional radial basis function (RBF) neural network for multi-label learning is presented. It consists of two layers of neurons: a first layer of hidden neurons representing basis functions associated with prototype vectors, and a second layer of output neurons related to all possible classes. The proposed method, named MLRBF, first performs a clustering of the instances corresponding to each possible class; the prototype vectors of the first-layer basis functions are then set to the centroids of the clustered groups. In a second step, the weights of the second-layer are fixed by minimizing a sum-of-squares error function. The output neuron of each class is connected to all input neurons corresponding to the prototype vectors of the different possible classes. Therefore, information encoded in the prototype vectors of all classes is fully exploited when optimizing the connection weights and predicting the label sets for unseen instances.
In [7], a multi-label ranking approach based on support vector machines (SVM) is presented. The authors define a cost function and a special multilabel margin and then propose an algorithm named RankSVM based on a ranking system combined with a label set size predictor. The set size predictor is computed from a threshold value that separates the relevant from the irrelevant labels. The value is chosen by solving a learning problem. The goal is to minimize a ranking loss function while having a large margin. RankSVM uses kernels rather than linear dot products, and the optimization problem is solved via its dual transformation.
In [26], an evidence-theoretic k-NN rule for multi-label classification is presented. This rule is based on an evidential formalism for representing uncertainties in the classification of multi-labeled data and handling imprecise labels, described in detail in [4]. The formalism extends all the notions of Dempster-Shafer theory [19] to the multi-label case, with only a moderate increase in complexity as compared to the classical case. Under this formalism, each piece of evidence about an instance to be classified is represented by a pair of sets: a set of classes that surely apply to the unseen instance, and a set of classes that surely do not apply to this instance.
A distinction should be made between multi-label and multiple-label learning problems. Multiple-label learning [9] is a semi-supervised learning problem for single-label classification where each instance is associated with a set of labels, but where only one of the candidate labels is the true label for the given instance. For example, this situation occurs when the training data is labeled by several experts and, owing to conflicts and disagreements between the experts, a set of labels, rather than exactly one label, is assigned to some instances. The set of labels of an instance contains the decision (the assigned label) made by each expert about this instance. This means that there is an ambiguity in the class labels of the training instances.
Another learning problem is multi-instance multi-label learning, where each object is described by a bag of instances and is assigned a set of labels [18]. Different real-world applications can be handled under this framework. For example, in text categorization, each document can be represented by a bag of instances, with each instance representing a section of the document in question, while the document may deal with several topics at the same time, such as culture and society.
In [20], dynamic conditional random fields (DCRFs) are presented for representing and handling complex interaction between labels in sequence modeling, such as when performing multiple, cascaded labeling tasks on the same sequence. DCRFs are a generalization of conditional random fields. Inference in DCRFs can be done using approximate methods, and training can be done by maximum a posteriori estimation.

Principle
Let X = R d denote the domain of instances and let Y = {ω 1 , ω 2 , . . . , ω Q } be the finite set of labels. The multi-label classification problem can be formulated as follows. Given a set D = {(x 1 , Y 1 ), (x 2 , Y 2 ), . . . , (x n , Y n )} of n training examples, independently drawn from X × 2 Y , and identically distributed, where x i ∈ X and Y i ⊆ Y, the goal of the learning system is to build a multi-label classifier H : X −→ 2 Y in order to assign a label set to each unseen instance. As for standard classification problems, we can associate with the multi-label classifier H a scoring function f : X × Y −→ R, which assigns a real number to each instance/label combination (x, ω) ∈ X × Y. The score f (x, ω) corresponds to the probability that instance x belongs to class ω. Given any instance x with its known set of labels Y ⊆ Y, the scoring function f is assumed to give larger scores for labels in Y than it does for those not in Y . In other words, f (x, ω q ) > f (x, ω r ) for any ω q ∈ Y and ω r ∈ Y . The scoring function f allows us to rank the different labels according to their scores. For an instance x, the higher the rank of a label ω, the 5 hal-00655629, version 1 -31 Dec 2011 larger the value of the corresponding score f (x, ω). Note that the multi-label classifier H(·) can be derived from the function f (·, ·) via thresholding: where t is a threshold value.

Label dependencies in multi-label applications
In multi-label classification, the assignment of class ω to an instance x may provide information about that instance's membership of other classes. Label dependencies exist when the probability of an instance belonging to a class depends on its membership of other classes. For example, a document with the topic politics is unlikely to be labeled as entertainment, but the probability that the document belongs to the class economic is high.
In general, relationships between labels are high-order or even full-order, i.e., there is a relation between a label and all remaining labels, but these relations are more difficult to represent than second-order relations, that is to say relations that exist between pairs of labels. The dependencies between labels can be represented in the form of a contingency matrix mat that allows us to express only second-order relations between labels. Let H q 1 denote the hypothesis that instance x belongs to class ω q ∈ Y. Given a multi-labeled dataset D with Q possible labels, mat[q][r] = Pr(H q 1 |H r 1 ), where q and r ∈ {1, . . . , Q} with q = r, indicates the second-order relationship between labels ω q and ω r . Pr(H q 1 |H r 1 ) represents the proportion of data in D belonging to ω r to which the label ω q is also assigned. mat[q][q] = Pr(H q 1 ) indicates the frequency of label ω q in the dataset D. Figures 1 show the contingency matrices for the emotion dataset (Q = 6) used in the experiments in Section 5. In this dataset, each instance represents a song and is labeled by the emotions evoked by this song. We can see in Figure 1 that mat [1][4] = Pr(H 1 1 |H 4 1 ) = 0, meaning that labels ω 1 and ω 4 cannot occur together. This is easily interpretable, as ω 1 corresponds to "amazed-surprised" while ω 4 corresponds to "quiet-still", and these two emotions are clearly opposite. We can also see that mat [5][4] = Pr(H 5 1 |H 4 1 ) = 0.6 , which means that ω 5 , representing "sad-lonely", frequently coexists in the label sets with ω 4 . We can see from these examples that labels in multi-labeled datasets are often mutually dependent, and exploiting relationships between labels will be very helpful in improving classification performance.

The DMLkNN method for multi-label classification
We use the same notation as in [29] in order to facilitate comparison with the MLkNN method. Given an instance x and its associated label set Y ⊆ Y, let N k x denote the set of the k closest training examples of x in the training dataset D according to a distance function d(., .), and let y x be the Q-dimensional category vector of x whose qth component indicates whether x belongs to class ω q : Let us represent by c x the Q-dimensional membership counting vector of x, the qth component of which indicates how many examples amongst the k-NNs of x belong to class ω q : 7 hal-00655629, version 1 -31 Dec 2011

MAP principle
Let x now denote an instance to be classified. Like in all k-NN based methods, when classifying a test instance x, the set N k x of its k nearest neighbors should first be identified. Under the multi-label assumption, the counting vector c x is computed. As mentioned before, let H q 1 denote the hypothesis that x belongs to class ω q , and H q 0 the hypothesis that x should not be assigned to ω q . Let E q j (j ∈ {0, 1, . . . , k}) denote the event that there are exactly j instances in N k x belonging to class ω q . To determine the qth component of the category vector y x for instance x, the MLkNN algorithm uses the following MAP [29]: while for the DMLkNN algorithm, the following MAP is used: In contrast to decision rule (3), we can see from Equation (4) that the assignment of label ω q to the test instance x depends not only on the event that there are exactly c x (q) instances having label ω q in N k x , i.e., E q cx(q) , but also on ω l ∈Y\{ωq} E l cx(l) , which is the event that there are exactly c x (l) instances having label ω l in N k x , for each ω l ∈ Y\{ω q }. Thus, it is clear that label correlation is taken into account in (4), since all the components of the counting vector c x are involved in the assignment or not of label ω q to x, which is not the case in Equation (3).

Posterior probability estimation
Regarding the counting vector c x , the number of possible events is upper bounded by (k +1) Q . This means that, in addition to the complexity problem, the estimation of (4) from a relatively small training set will not be accurate. To overcome this difficulty, we will adopt a fuzzy approximation for (4). This approximation is based on the event F l j , j ∈ {0, 1, . . . , k}, which is the event that there are approximately j instances in N k x belonging to class 8 ω l , i.e., F l j , denotes the event that the number of instances in N k x that are assigned label ω l is in the interval [j − δ; j + δ], where δ ∈ {0, . . . , k} is a fuzziness parameter. As a consequence, we can derive a fuzzy MAP rule: To remain closer to the initial formulation and for comparison with MLkNN, (5) will be replaced by the following rule: For large values of δ, the results of our method will be similar to those of MLkNN. In fact, for δ = k, the MLkNN algorithm is a particular case of the DMLkNN algorithm, where ω l ∈Y\{ωq} F l cx(l) will be certain, because for each ω l ∈ Y\{ω q }, the number of instances in N k x belonging to class ω l will surely be in the interval [j − k; j + k]. For small values of δ, the assignment or not of label ω q to test instance x will not only depend on the number of instances in N k x that belong to label ω q , but also on the number of instances in N k x belonging to the remaining labels. Using Bayes' rule, Equations (3) and (6) can be written as follows: To rank labels in Y, a Q-dimensional real-valued vector r x can be calculated. The qth component of r x is defined as the posterior probabil-9 hal-00655629, version 1 -31 Dec 2011 : For comparison, the real-valued vector r ′ x for MLkNN has the following expression: .
In order to determine the category vector y x and the real-valued vector r x of instance x, we need to determine the prior probabilities Pr(H l b ) and the likelihoods Pr(E q cx(q) , , for each q ∈ {1 · · · Q}, and b ∈ {0, 1}. These probabilities are estimated from a training dataset D.
Given an instance x to be classified, the output of the DMLkNN method for multi-classification is determined as follows: %Computing the prior probabilities and the number of instances belonging to each class %Counting the training instances whose membership counting vectors satisfy the constraints (13) Identify N (xi) and cx i 9.

11
hal-00655629, version 1 -31 Dec 2011 first calculated and the instances belonging to each label are counted (steps 1 to 3): Recall that n is the number of training instances. u(q) is the number of instances belonging to class ω q , and u ′ (q) indicates the number of instances not having ω q in their label sets: For test instance x, the k-NNs are identified and the membership counting vector c x is determined (step 4). To decide whether or not to assign the label ω q to x, we must determine the likelihoods Pr(E q cx(q) , 1}, using the training instances such that their corresponding membership counting vectors satisfy the following constraints: This is illustrated in steps 5 to 12. The number of instances from the training set satisfying these constraints and belonging to class ω q is stored in v(q). The number of remaining instances satisfying the previous constraints and not having ω q in their sets of labels is stored in v ′ (q). The likelihoods Pr(E q cx(q) , , are then computed: where s is a smoothing parameter [13]. Smoothing is commonly used to avoid zero probability estimates. When s = 1, it is called Laplace smoothing. Finally, the category vector y x and the real-valued vector r x to rank labels in Y are calculated using equations (8) and (10) (13). MLkNN and DMLkNN should therefore not necessarily be compared using the same smoothing parameter.

Illustration on a simulated dataset
In this section, we illustrate the behavior of the DMLkNN and MLkNN methods using simulated data.
The simulated dataset contains 1019 instances in R 2 belonging to three possible classes, Y = {ω 1 , ω 2 , ω 3 }. The data were generated from seven The number of instances in each class is chosen arbitrarily (see Table 1). Taking into account the geometric distribution of the Gaussian data, the instances of each set were respectively assigned to label(s)  Figure 3 shows the neighboring training instances and the estimated label set for a test instance x using DMLkNN and MLkNN. For both methods, k was set to 8, and Laplace smoothing (s = 1) was used. For DMLkNN, δ was set to 1. Below we describe the different steps in the estimation of the label set of x using the DMLkNN and MLkNN algorithms applied to the test data. For the sake of clarity we recall some definitions of events already given above. The membership counting vector of the test instance is c x = (7, 3, 2). Using the DMLkNN method, in order to estimate the label set of x, the following probabilities need to be computed from Equation (8): We recall that E 1 7 is the event that there are seven instances in N k x which have label ω 1 , and F 2 3 is the event that the number of instances in N k x belonging to label ω 2 is in the interval [3 − δ; 3 + δ] = [2,4]. In contrast, for estimating the label set of the unseen instance using the MLkNN method, the following probabilities must be computed from Equation (7): First, the prior probabilities are computed from the training set according to Equation (11) Thus, the estimated label set for test instance x given by the DMLkNN method is Y = {ω 1 , ω 2 }, while that given by MLkNN is Y ′ = {ω 1 }. The true label set for x is Y = {ω 1 , ω 2 }. Here we can see that no error has occurred when estimating the label set of x using the DMLkNN method, while for MLkNN the estimated label set is not identical to the ground truth label set. Seven training instances in N k x have class ω 1 in their label sets, while only three instances belong to ω 2 . In fact, the existence of class ω 1 in the neighborhood of x gives some information about the existence or not of class ω 2 in the label set of x. If we take a look at the training dataset, we remark that 14,7% of instances belong to ω 1 , 15.9% to ω 2 , and 29.8% to ω 1 and ω 2 simultaneously. Thus, the probability that an instance belongs to both classes ω 1 and ω 2 is approximately twice the probability that it belongs to only one of the two classes. DMLkNN is able to capture the relationship between classes ω 1 and ω 2 , while MLkNN is not able to capture this relation. This example shows that the DMLkNN method, which takes into account the dependencies between labels, may improve the classification performance, and estimate the label sets of test instances with greater accuracy.

Experiments
In this section, we report a comparative study between DMLkNN and some state-of-the-art methods on several datasets collected from real world applications, and using different evaluation metrics.

Evaluation metrics
There exist a number of evaluation criteria that evaluate the performance of a multi-label learning system, given a set D = {(x 1 , Y 1 ), . . . , (x n , Y n )} of n test examples. We now describe some of the main evaluation criteria used in the literature to evaluate a multi-label learning system [17] [21]. The evaluation metrics can be divided into two groups: prediction-based and ranking-based metrics. Prediction-based metrics assess the correctness of the label sets predicted by the multi-label classifier H, while ranking-based metrics evaluate the label ranking quality depending on the scoring function f . Since not all multi-label classification methods compute a scoring function, prediction-based metrics are of more general use.

Prediction-based metrics
Accuracy. The accuracy metric is an average degree of similarity between the predicted and the ground truth label sets of all test examples: where Y i = H(x i ) denotes the predicted label set of instance x i .
F1-measure. The F1-measure is defined as the harmonic mean of two other metrics known as Precision (P rec) and Recall (Rec) [24]. The former computes the proportion of correct positive predictions while the latter calculates the proportion of true labels that have been predicted as positives. These metrics are defined as follows: and, Hamming loss. This metric counts prediction errors (an incorrect label is predicted) and missing errors (a true label is not predicted): where ∆ stands for the symmetric difference between two sets. Note that the values of the prediction-based evaluation criteria are in the interval [0, 1]. Larger values of the first four metrics correspond to higher classification quality, while for the Hamming loss metric, the smaller the symmetric difference between predicted and true label sets, the better the performance [21] [24]. 17 hal-00655629, version 1 -31 Dec 2011

Ranking-based metrics
As stated before, this group of criteria is based on the scoring function f (., .) and evaluates the ranking quality of the different possible labels [7] [29]. Let rank f (., .) be the ranking function derived from f and taking values in {1, . . . , Q}. For each instance x i , the label with the highest scoring value has rank 1, and if f ( One-error. The one-error metric evaluates how many times the top-ranked label, i.e. the label with the highest score, is not in the true set of labels of the instance: where for any proposition H, H equals to 1 if H holds and 0 otherwise. Note that, for single-label classification problems, the one-error is identical to ordinary classification error.
Coverage. The coverage measure is defined as the average number of steps needed to move down the ranked label list in order to cover all the labels assigned to a test instance: Ranking loss. This metric calculates the average fraction of label pairs that are reversely ordered for an instance: Average precision. This criterion was first used in information retrieval and was then adapted to multi-label learning problems in order to evaluate the effectiveness of label ranking. This metric measures the average fraction of labels ranked above a particular label y ∈ Y i which actually are in Y i :

hal-00655629, version 1 -31 Dec 2011
For the ranking-based metrics, smaller values of the first three metrics correspond to better label ranking quality, while AvP rec(f, D) = 1 means that the labels are perfectly ranked for all test examples [7].

Benchmark datasets
Given a multi-labeled dataset Several real datasets 2 were used in our experiments. The datasets used are from different application domains, namely text categorization, bioinformatics and multimedia applications (music and image).
• The emotion dataset, presented in [10], consists of 593 songs annotated by experts according to the emotions they generate. The emotions are: amazed-surprise, happy-pleased, relaxing-calm, quiet-still, sad-lonely and angry-fearful. Each emotion corresponds to a class. Consequently there are 6 classes, and each song was labeled as belonging to one or several classes. Each song was also described by 8 rhythmic features and 64 timbre features, resulting in a total of 72 features. The number of distinct label sets is equal to 27, the label cardinality is 1.868, and the label density is 0.311.
• The scene dataset consists of 2407 images of natural scenery. For each image, spatial color moments are used as features. Images are divided into 49 blocks using a 7 × 7 grid. The mean and variance of each band are computed corresponding to a low-resolution image and to computationally inexpensive texture features, respectively [1]. Each image is then transformed into a 49 × 3 × 2 = 294-dimensional feature vector. A label set is manually assigned to each image. There are 6 different semantic types: beach, sunset, field, fall-foliage, urban and mountain. The average number of labels per instance is 1.074, thus the label density is 0.179. The number of distinct sets of labels is equal to 15.
• The yeast dataset contains data regarding the gene functional classes of the yeast Saccharomyces cerevisiae [7]. It includes 2417 genes, each of which is represented by 103 features. A gene is described by the concatenation of micro-array expression data and a phylogenetic profile, and is associated with a set of functional classes. There are 14 possible classes and there exist 198 distinct label combinations. The label cardinality is 4.237, and the label density is 0.303.
• The medical dataset consists of 978 examples, each example represented by 1449 features. This dataset was provided by the Computational Medicine Center as part of a challenge task involving the automated processing of free clinical text, and is the dataset used in [15]. The average cardinality is 1.245, and the label density is 0.028 with 94 distinct label sets.
• The Enron email dataset consists of 1702 examples, each represented by 1001 features. It comprises email messages belonging to users, mostly senior management of the Enron Corp. This dataset was used in [15]. 753 distinct label combinations exist in the dataset. The label cardinality is 3.378 and the label density is 0.064. Table 2 summarizes the characteristics of the emotion, scene, yeast, medical and Enron datasets.
• The webpage categorization dataset was investigated in [22] [29]. The data were collected from the "yahoo.com" domain. Eleven different webpage categorization subproblems are considered, corresponding to 11 different categories: Arts and Humanities, Business and Economy,  Computers and Internet, Education, Entertainment, Health, Recreation and Sports, Reference, Science, Social and Science, and Society and Culture. Each subproblem consists of 5000 documents. Over the 11 subproblems, the number of categories varies from 21 to 40 and the instance dimensionality varies from 438 to 1,047. Table 3 shows the statistics of the different subproblems within the webpage dataset.

Experimental Results
The DMLkNN method was compared to two other binary relevancebased approaches, namely, MLkNN and BRkNN. The model parameters for DMLkNN are the number of neighbors k, the fuzziness parameter δ, and the smoothing parameter s. Parameter tuning can be done via cross-validation. For fair comparison, k was set to 10 for the three methods, and s was set to 1, as in [29]. Note that as stated in Section 4.2, the parameter δ should be set to a small value. When k is set to 10, extensive experiments have shown that the value δ = 2 generally gives good classification performances for DMLkNN.
In addition to the two k-NN based algorithms, our method was compared to two other state-of-the-art multi-label classification methods that have been 21 hal-00655629, version 1 -31 Dec 2011 shown to have good performances: MLRBF [28], derived from radial basis function neural networks, and RankSVM [7], based on the traditional support vector machine. As used in [28], the fraction parameter for MLRBF was set to 0.01 and the scaling factor to 1. For RankSVM, polynomial kernel was used as reported in [7].
For all k-NN based algorithms, the Euclidean distance was used. Note that usually, when feature variables are numeric and are not of comparable units and scales, i.e., there are large differences in the ranges of values encountered (such as in the emotion dataset), the distance metric implicitly assigns greater weight to features with wide ranges than to those with narrow ranges. This may affect the nearest neighbors search. In such cases, feature normalization is recommended to approximately equalize the ranges of features so that they will have the same effect on distance computation [5]. In addition, we may remark that in the cases of the medical and Enron datasets, the dimensions of feature vectors are very large as compared to the number of training instances (see Table 2). We applied the χ 2 statistic approach for feature selection on these two datasets, and we retained 20% of the most relevant features [25].
Five iterations of ten-fold cross-validation were performed on each dataset. Tables 4 through 8 report the detailed results in terms of the different evaluation metrics for the emotion, scene, yeast, medical and Enron datasets, respectively. On the webpage dataset, ten-fold cross validation was performed on each subproblem, and Table 9 reports the average results.
For each method, the mean values of the different evaluation criteria, as well as the standard deviations (std) are given in the tables. A two-tailed paired t-test at 5% significance level was performed in order to determine the statistical significance of the obtained results in comparison with the best performances indicated in bold. In addition, for each dataset, the methods were ranked in decreasing order of performance. The average ranks over the different evaluation criteria are reported in the tables.
To give some idea about the computational complexity of the different algorithms, Table 10 provides the corresponding runtime statistics (in seconds) on the different datasets, using train/test experiments. All the algorithms were implemented in Matlab 7.4 and executed on the same computer (Intel Core Duo 2.13 GHz, 2 Go RAM).
Using the Sign test, a statistical comparison between the classifiers was made over the different datasets cited above. Table 11 reports the average ranking on each evaluation metric.

22
From the experimental results presented, the following observations can be made: • DMLkNN performs better than MLkNN with respect to all evaluation metrics and on all datasets. In addition, DMLkNN performs better than BRkNN and is very competitive with the remaining methods that are based on more sophisticated classifiers (SVM and RBF). When the results obtained on the different datasets are averaged, DMLkNN gives the best performances with respect to all evaluation metrics apart from One-error and Average-precision. The next best results are obtained from MLRBF.
• The experimental results show that, generally, DMLkNN performs better with respect to predicted-based metrics than with respect to rankingbased metrics, as for example on the emotion and scene datasets. In fact, for each instance to be classified, the output of DMLkNN is determined by combining binary decisions made about that instance's membership of the different classes. Thus, this method is concerned more with the pertinence of the predicted sets of labels than with the ranking of all labels. A ranking-based method, such as RankSVM, on the other hand, will normally tend to be more competitive with other methods as regards ranking-based metrics. This may be explained by the fact that ranking-based methods operate by ranking the labels according to their relevance to a given instance to be classified, and then splitting the ordered set of labels into subsets of relevant and irrelevant labels for that instance.
• DMLkNN has good performances and is more competitive with the other algorithms on datasets with high label-density, such on the emotion and yeast datasets. The proposed method is intended primarily to take into account the dependencies between labels, and DMLkNN tends to perform better on datasets with high label multiplicity, in which labels may be potentially more interdependent.
• MLkNN is computationally faster than DMLkNN. In fact, in the MLkNN method, the likelihoods Pr(E q cx(q) |H q b ), b ∈ {0, 1}, are calculated from the training set, stored, and then used only when predicting the label set of each query instance. In contrast, when DMLkNN is used, the number of likelihoods Pr(E q cx(q) ,  [14]. The complexity of MLRBF depends primarily on the complexity of the K-means algorithm used for clustering the instances belonging to each possible class. MLRBF therefore has a linear complexity with respect to the number of classes, the number of clusters, the number of instances, and the dimensionality of the corresponding features vectors.

Conclusion
In this paper, we proposed an original multi-label learning algorithm derived from the k-NN rule, where the dependencies between labels are taken into account. Our method is based on the binary relevance approach, that is frequently criticized for ignoring possible correlations between labels [15], but here, this disadvantage is overcome. The classification of an instance is accomplished through the use of local information given by the k nearest neighbors, and by using the maximum a posteriori rule. This method, referred to as DMLkNN, generalizes the MLkNN algorithm presented in [29].
The proposed method is particularly useful in practical situations where data are significantly multi-labeled. Using a variety of benchmark datasets and different evaluation criteria, the experimental results clearly demonstrate this, and confirm the usefulness and the effectiveness of DMLkNN as compared to other state-of-the-art multi-label classification methods.
Note that, when the number of classes grows, more data are required in order to compute the posterior probabilities for DMLkNN. On limited datasets, it will be hard to capture relationships between labels. In addition, the performances of the proposed method depend on the distance computation metric used to determine the nearest neighbors.