- Research
- Open Access

# LDA boost classification: boosting by topics

- La Lei
^{1}Email author, - Guo Qiao
^{1}, - Cao Qimin
^{1}and - Li Qitao
^{1}

**2012**:233

https://doi.org/10.1186/1687-6180-2012-233

© Lei et al.; licensee Springer. 2012

**Received: **24 April 2012

**Accepted: **19 October 2012

**Published: **2 November 2012

## Abstract

AdaBoost is an efficacious classification algorithm especially in text categorization (TC) tasks. The methodology of setting up a classifier committee and voting on the documents for classification can achieve high categorization precision. However, traditional Vector Space Model can easily lead to *the curse of dimensionality* and *feature sparsity* problems; so it affects classification performance seriously. This article proposed a novel classification algorithm called LDABoost based on boosting ideology which uses Latent Dirichlet Allocation (LDA) to modeling the feature space. Instead of using words or phrase, LDABoost use latent topics as the features. In this way, the feature dimension is significantly reduced. Improved Naïve Bayes (NB) is designed as the weaker classifier which keeps the efficiency advantage of classic NB algorithm and has higher precision. Moreover, a two-stage iterative weighted method called Cute Integration in this article is proposed for improving the accuracy by integrating weak classifiers into strong classifier in a more rational way. Mutual Information is used as metrics of weights allocation. The voting information and the categorization decision made by basis classifiers are fully utilized for generating the strong classifier. Experimental results reveals LDABoost making categorization in a low-dimensional space, it has higher accuracy than traditional AdaBoost algorithms and many other classic classification algorithms. Moreover, its runtime consumption is lower than different versions of AdaBoost, TC algorithms based on support vector machine and Neural Networks.

## Keywords

- Latent Dirichlet Allocation
- Topics
- Boosting
- Two-procedure iterative weighting
- Text classification

## 1. Introduction

Text categorization (TC) has received unprecedented focus in recent years. A TC system can rescue people from tremendous amount of information in this era of information explosion. In addition, text classification is the foundation of many popular information processing technologies such as information retrieval, machine Q & A and sentiment analysis. Since a high percentage of information in the network is textual information [1], the precision of text classification will largely determines the ability of people in information utilization, in other words, the quality of our life.

*F*:

*D*×

*C*→ {0, 1}, where

*D*is a set of documents and

*C*is the set of predefined categories:

The approximating function *M*:*D* × *C* → {0, 1} is called a classifier. The task is to build a classifier that produces results as close as possible to the true category assignment function *F*.

*corpus*is a very important word in a scientific literature retrieval system, but it would not be chosen in a corpus database system. An example of feature selection in a sports news classification system is shown in Figure 1.

Since feature selection is the basis of TC, it has aroused extensive attention from scholars. Feature representation models such as Bag-of-words, Vector Space Model (VSM), Probabilistic Latent Semantic Indexing [2], and Latent Dirichlet Allocation (LDA) [3] have been proposed for selecting features in document set.

In traditional Bag-of-words and VSM, words are selected as features. Word features tend to result in the curse of dimensionality and feature sparsity problems. Feature dimension of a middle-size document set may reach 10^{4} or 10^{5}[4] and extremely increasing the computational and runtime complexity of the task. This is the so-called curse of dimensionality. Feature sparsity means the occurrence probability in a certain document of a feature which belonged to the document set is very low. In other words, in the vector space, most components of a text are zero-vectors. Feature sparsity would greatly reduce the accuracy of classification [5]. To solve problems above, some experts try to use non-continuous phrases [6], concepts [7], and topics [8] as features.

Another pivotal aspect of TC is a classification algorithm design. Although there are also considerable literatures in this area, support vector machine (SVM), Decision Tree, Neural Networks, Naïve Bayes (NB), Rocchio, and voting-based algorithm [9] are the most important methods. The core issue of categorization is kept balance between accuracy and efficiency. Some algorithms have quite good accuracy and high time cost at the same time, such as SVM. Light classification algorithm, for instance, NB, has low time consumption but the precision is not always ideal. Even more, neural networks and some other compromise solutions may lead to bad performance both in accuracy and efficiency. Voting-based categorization algorithms also known as classifier committees can adjust the number and professional level of “experts” in the committees to find a balance between performance and time-computational consumption.

Few researchers place dimension reduction and classification algorithm in the same framework to make a comprehensive consideration. Classification algorithm should be based on feature selection to further improving its performance. In another hand, feature dimension reduction should use classification algorithm to check its effectiveness.

The rest of this article is organized as follows. Section 2 reviews LDA and analyzes its application in text feature selection. Section 3 improves traditional NB as the weak classifier. In Section 4, a two-procedure iterative weighted method is proposed by introducing Mutual Information (MI) criterion in it to integrating a strong classifier. Section 5 then proposed LDABoost based on Sections 3 and 4 which is the first time that LDA is used together with Boosting algorithm to the best of the authors’ knowledge as the final classification framework. The application of the novel classification method is presented and analyzed in Section 6. Finally, Section 7 summarizes the article.

## 2. Feature extraction by LDA

Strictly speaking, dimension reduction algorithms can be categorized into two groups: feature extraction and feature selection. In the former, new features of texts are combined from their original features through algebraic transformation. In the latter, subsets of features are selected directly. Feature extraction is mathematically efficient but with high computational overhead [10]. Feature selection is quite convenient to be implemented in real world. However, there is no theoretically guarantee in optimality for feature selection’s solution. Probabilistic topic model-based dimension reduction algorithms attract more and more attention because it maintains the merit of feature extraction and to some extent overcome the high computational consumption problems.

### 2.1. LDA

In Figure 2, *K* is the number of topics, *M* the number of documents, *N*_{
m
} the number of words in the *m* th document, φ_{
k
} the words distribution in topic *k*, θ_{
m
} the topic distribution in document *m*, φ_{
k
} and θ_{
m
} also the parameters of multinomial distribution which are used to *generating* topics and words, α and β are empirical parameters and usually they are symmetric.

_{ k }and θ

_{ m }follow a Dirichlet allocation as

where 0 ≤ *μ*_{
k
} ≤ 1, ∑ _{
k
}* μ*_{
k
} = 1, *α*_{0} = ∑ _{k = 1}^{
k
}* α*_{
k
} and *Γ* is the Gamma function. Dirichlet distribution is the priori conjugate distribution of multinomial distribution.

*generating*words [12]:

- 1.
Topic sampling by φ

_{ k }~*P*_{Dir}(*β*),*k*∊ [1,*K*]. - 2.
In document

*m*,*m*∊ [1,*M*] make topic probability distribution sampling by θ_{ m }~*P*_{Dir}(*α*). - 3.
Document length sampling by

*N*_{ m }~ Poisson (ξ). - 4.
Select a latent topic

*z*_{m,n}~ Multinomial (θ) for*n*th word in document*m*, where*n*∊ [1,*N*]. - 5.
Generate a word

*w*_{ m,n }~ Multinomial(${\phi}_{{z}_{m,n}}$).

Follow above steps, LDA model aggregates semantically similar words as latent topic. If we make topic selection according to function (2) using them as text features, the feature dimension will greatly be reduced.

### 2.2. Parameter estimation in LDA

Obviously, neither Equation (1) nor Equation (2) can be calculated directly. Therefore, the topic selection problem translated into parameter estimation problem. In LDA, parameters can be estimated by Maximum Entropy, Variational Bayesian Inference [13], Expectation-Propagation [14], Gibbs sampling, etc.

Gibbs sampling is a special case of Markov Chain Monte Carlo, it samples for a component of the joint distribution and keep the value of other components in a time. For the situation of high-dimensional joint distribution, this strategy simplified steps of the algorithm.

_{ k }and θ

_{ m }by using integration. CGS samples topic

*z*for each word

*w*. Once the topic of

*w*is identified, φ

_{ k }and θ

_{ m }can be calculated by frequency statistics. As the analysis above, parameter estimation problem translate into calculate the conditional probability of topic sequence in the condition that word sequence is known as

*w*is a vector constitute by the words end-to-end. Because the sequence of

*z*is usually very long, the possible value growth exponentially with the length of the vector and difficult to be calculated directly. Fortunately, CGS can decompose the problem into several sub-problems, samples a topic in each time. The final sampling function is

Assume *w*_{
i
} = *t*, where *z*_{
i
} represents the topic variation of *i* th word, → *i* means exclude element *i*, *n*_{
k
}^{
t
} is the occurrence time of word *t* in topic *k*, *βt* is the priori of Dirichlet distribution, *n*_{
m(k)
}is the frequency of topic *k* in document *m*, *αk* is the Dirichlet priori of topic *k*.

*k*of word

*w*, parameters φ

_{ k }and θ

_{ m }can be computed as:

LDA builds a statistic model for document set, texts, categories, topics, and words. Using sampling algorithms such as Gibbs sampling can estimate the model’s parameters and achieve document representation in feature space.

### 2.3. Dimension reduction based on LDA

Reasonable feature selection and feature extraction approaches should make documents of the same category have much shorter distance in feature space and documents from different categories have much longer distance. In other words, categorization results based on selected features should have maximum within-class similarity and minimum between-classes similarity.

Where *x* = (*x*_{1}, *x*_{2}, …, *x*_{
n
})^{
T
} is a multi-variable feature vector, the mean of *x* is *μ* = (*μ*_{1}, *μ*_{2}, …, *μ*_{
n
})^{
T
}. Different from Euclidean distance, Mahalanobis distance can reflect the relationship between various of the feature. In addition, it takes features’ characteristics of scale-invariant into account. Therefore, Mahalanobis distance is used to measure the distance of topics and as the reference of classification.

As show in the figure, LDA can decrease the probability of misclassification caused by confusing words. Furthermore, science plenty of words converging into a topic, LDA significantly reduces the dimensionality of feature space. Topics in feature space are quite similar with cluster headers in ad hoc networks. In ad hoc networks, using cluster headers as representation of the web can greatly deduces the complexity of network topology. Similarly, use topics to representing documents can benefit categorization.

- 1.
Input training document set.

- 2.
Preprocessing. Such as word segmentation and Part-of-Speech tagging.

- 3.
Preprocessing. Check the stop words list and remove them out of the document set.

- 4.
Set values for empirical parameters.

- 5.
Call LDA. Synthesize words into latent topics.

- 6.
Calculate Mahalanobis distance of topics and select high weight topics as the feature topics.

Hitherto, a document feature extraction method is proposed. It based on LDA model and can significantly reduce the dimension of feature space by selecting topics as document features. Using the low-dimensional feature set as the foundation can greatly improve the accuracy of TC, moreover, decrease its time and computational consumption.

## 3. Classifier design based on NB

Theoretically, once weak classifiers are more accurate than guess randomly (1/2 in two-class tasks or 1/*n* in multi-class tasks), AdaBoost can integrate them into a strong classifier whose precision could infinitely close to the true category distribution [17]. However, when the precision of weak classifiers is lower, more weak classifiers are needed to construct a strong classifier. Too many weak classifiers in the system sometimes increase its complexity and computational consumption to intolerable level. In another hand, boosting algorithms which use complex base learners based on SVM [18], Neural Networks [19], etc., can certainly achieve higher accuracy but lead to some new problems because they are over sophisticated and thus contrary to the ideology of Boosting algorithm.

Boosting algorithm proposed in this article uses topics supported by LDA as its feature set. According to the analysis in Section 2, topic feature set has parlous lower dimension and features in it have higher discrimination. Therefore, weak classifier based on simple algorithm such as NB can achieve an ideal precision with really low runtime cost.

### 3.1. NB classification

The basic idea of NB is calculates the priori probability of an object, then using Bayesian formula to calculate its posterior probability. Finally, use the posterior probability as the probability of which category the new text should belong to.

*X*= (

*x*

_{1},

*x*

_{2}, …,

*x*

_{ n }) of weather topic features belong to some class can be calculated as:

Where *N*(*z*_{
k
}, *d*_{
l
}) is the frequency of *k* th topic in the *l* th document. |*V*| is the sum of topics, c_{
j
} the *j* th category, and *D* the sum of documents which belong to it.

*P*(

*c*

_{ j }) can be calculated as:

Where *C* is the sum of categories, *n* the number of feature topics in document *d*_{
l
}.

*P*(

*c*

_{ j }|

*d*

_{ l }) of a document in different category condition has the same denominator $\sum _{r=1}^{C}P\left({c}_{r}\right){\displaystyle \prod _{k=1}^{n}P\left({z}_{k}|{c}_{r}\right)}$.Therefore, NB TC finally calculates function below

As shown in Equation (12), NB is quite a light classification method.

### 3.2. Multi-level NB

Features do not have weight in original NB, they are believed to have equal contribution for classification. However, this assumption is seldom suitable in TC. Latent topics from headlines, abstracts, and key words always have significant importance for TC. In addition, first and last paragraph of the document usually summarize the article and therefore may contain much more information for classification. Features selected from other parts of the document sometimes give lower benefit for categorization.

Therefore, topic features can be divided into several levels according to their position in documents. Give different weight for different level so that features from different levels can play different roles in categorization.

*k*of levels can be set by empirical values. However, empirical values need human experience and thus increase labor costs. Actually,

*k*can be adjusted adaptively by sampling and comparing the relative entropy of features in different level. When the relative entropy of two levels is lower than system’s lower bound, emerge the levels, when it is higher than upper bound, split them into more levels. The flow chart of multi-level NB is shown in Figure 4.

Following steps in Figure 4, a multi-level NB categorization algorithm is constructed. It uses topics extracted by LDA instead of feature words in traditional VSMs to improving its classification ability and maintaining the runtime consumption. Furthermore, a multi-level strategy is introduced in NB to ensure it use topics in a more effective way.

## 4. Cute integration (CI): the way strong classifier generated

Whether strong classifier has a good performance depends largely on how weak classifiers are combined. To build a powerful strong classifier, basis classifiers which have higher precision must take more responsibility in categorization process. Therefore, categorization system should distinguish between the performances of weak classifiers and give them different weights according to their capabilities. Moreover, ambiguous texts should be identified and pay more attention on them by allocating them higher weights. Using these weights, Boosting algorithms can integrate weak classifiers as the strong classifier in a more efficient way and achieve excellent performance.

### 4.1. Weighting mechanism of classic AdaBoost review

where *h*_{
t
}(*x*) is a basis classifier, *αt* is a coefficient, and *H*(*x*) the final strong classifier.

*x*

_{1},

*y*

_{1}), (

*x*

_{2},

*y*

_{2}), …, (

*x*

_{ m },

*y*

_{ m }),

*x*

_{ i }∊

*X*, and

*y*

_{ i }= ± 1. The strong classifier can be constructed as [20]

- 1.
Initialize weight

*D*_{1}(*i*) = 1/*m*, for*t*= 1, 2, …,*T*. - 2.Select a weak classifier with the smallest weighted error:${h}_{t}=arg\underset{{h}_{j}\in H}{min}\phantom{\rule{0.5em}{0ex}}{\epsilon}_{j}={\displaystyle \sum _{i=1}^{m}{D}_{t}\left(i\right)({y}_{i}\ne {h}_{j}\left({x}_{i}\right)}$(15)
Where

*ɛ*_{ j }is the error rate. - 3.
Prerequisite:

*ɛ*_{ t }< 1/2, otherwise stop. - 4.
Upper bounded

*ɛ*_{ t }by ${\epsilon}_{t}\left(H\right)\le {\displaystyle \prod _{t=1}^{T}{Z}_{t}}$, where*Z*_{ t }is a normalization factor. - 5.
Select

*α*_{ t }to greedily minimize*Z*_{ t }(*α*) in each step. - 6.
Optimizing:

Where ${r}_{t}={\displaystyle \sum _{i=1}^{m}{D}_{t}\left(i\right){h}_{t}\left({x}_{i}\right){y}_{i}}$ by using the constraint ${Z}_{t}=2\sqrt{{\epsilon}_{t}\left(1-{\epsilon}_{t}\right)}\le 1$.

- 7.Reweighting as${\alpha}_{t}=\frac{1}{2}log\left(\frac{1+{r}_{t}}{1-{r}_{t}}\right)$(16)$\begin{array}{l}{D}_{t+1}\left(i\right)=\frac{{D}_{t}\left(i\right)exp\left(-{\alpha}_{t}{y}_{i}{h}_{t}\left({x}_{i}\right)\right)}{{Z}_{t}}\\ \phantom{\rule{3em}{0ex}}\phantom{\rule{1em}{0ex}}=\frac{exp\left(-{y}_{i}{\displaystyle \sum _{q=1}^{t}{\alpha}_{q}{h}_{q}({x}_{i}}\right))}{m{\displaystyle \prod _{q=1}^{t}{Z}_{q}}}\end{array}$(17)$exp\left(-{\alpha}_{t}{y}_{i}{h}_{t}\left({x}_{i}\right)\right)\{\begin{array}{l}<1,\phantom{\rule{1em}{0ex}}{y}_{i}={h}_{t}\left({x}_{i}\right)\\ >1,\phantom{\rule{1em}{0ex}}{y}_{i}\ne {h}_{t}\left({x}_{i}\right)\end{array}$(18)

In the above algorithm, the definition of *better classification performance* is not reasonable. Only using the classification error subset of former classifiers to training later classifiers is not enough. We called the documents which are classified incorrectly *difficult* document. The later classifiers will be evaluated whether they have the ability to rightly classifying difficult documents. However, the former classifiers have not been trained by the error subset of later classifiers.

This classifiers’ training mechanism has overlooked two basic questions. First, if the document subset *R*_{
i
} which be classified rightly by the classifier *i* is also easy for classifier *i* + 1. Second, if the documents be classified incorrectly by the classifier *j* is also difficult for classifier *j* - 1.

The negligence of above questions makes the weights allocation strategy have no comprehensive consideration of training samples. In addition, in this situation training set could not be fully utilized to generating a more powerful strong classifier.

### 4.2. Two-procedure weighting method

*powerful*for base classifiers by using not only the former part, but also the later part of the training sets. The work step of two-procedure weighting method is shown in Figure 6.

- 1.
**Begin:**initialize documents weights*w*_{ d }(*i*) and weak classifier weights*w*_{ c }(*j*). - 2.
Training first classifier

*C*_{1}with first sample documents subset*D*_{1}, mark the set of documents which be misclassified by*C*_{1}in*D*_{1}as*E*_{1}. - 3.
**Loop**: training*C*_{ i }with*D*_{ i }and*E*_{i−1} - 4.
Calculation: calculating weights of base classifiers according to first round of loops (trainings).

- 5.
Reverse iterative: training

*c*_{1}with*D*_{ n }. - 6.
**Loop**: training*c*_{ i }with*D*_{ i }and*E*_{n−i}. - 7.
Calculation: calculating weights of weak classifiers according to second round of loops (trainings).

- 8.
Calculate final weights of base classifiers according to steps 4 and 7.

- 9.
Cascade: combine base classifiers according to their final weights and construct strong classifier.

- 10.
**End**.

Above steps ensure the full use of training sets and generate weight in each procedure.

### 4.3. Judgment for measuring the error

Most previous boosting-based algorithm only records the number of incorrectly classified documents. However, error numbers sometime cannot faithfully reflect the performance of weak classifiers because the severity of the error is not always the same.

Image the situation in Figure 3: make misclassification that put a film review about *Titanic* in the Ocean category is not as serious as put an Oscar Academy Awards in the Ocean category. In order to improve system’s ability of distinguish between base classifiers’ performance, some judgment should be used to evaluating the severity of errors.

Distance between the category which a document should belong to and the category which the document be classified incorrectly probably is the most intuitive reference to determine how serious an error is. However, the distance between text categories could not be measured directly like what scientist has done in physical world. In this article, we use MI as the judgment.

*X*and

*Y*are a pair of discrete random variable where

*X*,

*Y*~

*P*(

*x*,

*y*), the joint entropy of

*X*and

*Y*defined as

As shown in Figure 7, greater MI of two categories means they contain more similar information, thereby the distance between them is shorter. Obviously, it is less serious to misclassifying a document to a category which has large MI with its true category. Assume *C*_{
i
} is the true class of document *i*, *C*_{
i
}’ is the error class of *i*. We can use *I*(*C*_{
i
}; *C*_{
i
}’) as the severity judgment of classification error.

*D*= (

*d*

_{1},

*d*

_{2}, …,

*d*

_{ m }) is the document set of category

*C*,

*D*’ = (

*d*’

_{1},

*d*’

_{2}, …,

*d*’

_{ n }) is the document set of category

*C’*, the MI of them can be calculated as

*t*into account, it is easy to learn each misclassification corresponds to two categories, in other words, corresponds to a MI value. We can use the following function as the weight definition of classifier i.

### 4.4. CI algorithm: strong classifier construction

Strong classifier can be generated by integrating weak classifiers based on the strategies proposed in Sections 4.2 and 4.3. The strong classifier construction algorithm in this article called CI.

Using Equation (24) directly is the simplest but not the best way to weighting classifiers. Note that some basis classifiers may have a very high weight both in the first and second procedures. It means these classifiers have global high categorization ability and should play a more important role in classification process instead of using the average weight simply. In this case, an upper bound value is set as the final weight of significantly powerful classifiers. In another hand, some classifiers may have a very low weight in both two iterative loops. The utility of these classifiers must be limited by using a lower bound value to enhance system’s accuracy.

Moreover, some weak classifiers may have a very high weight in one procedure but a very low weight in another iterative step. The system should consider the weak classifiers as noise-oversensitive and deduce its weight. In this article, we use min(*w*_{
j
}, *w*_{
j
}’) as the final weight of noise-oversensitive classifier.

The runtime complexity of MI calculation is *O*(*m* · *n*) [21]. Therefore, the time consumption of CI algorithm is *O*(*m* · *n*^{2}), where *m* the number of base classifiers and *n* the number of training documents.

As analysis above, the computational complexity is proportional to the number of weak classifiers. In addition, when the number of classification objects increase, the time consumption would increase quadratic. Therefore, the algorithms avoid index explosion problem and have an acceptable runtime complexity. In addition, there is no condition missing and the weight’s value of every classifier is non-infinite. Therefore, CI algorithm is convergence.

In the figure, *E*_{
i
} is the error set of the *i* th basis classifier, *w*_{
i
} the weight of the *i* th classifier in the first weighting procedure, *w*_{
i
}’ the weight of the *i* th classifier in the second weighting step, *α* the lower threshold of weight, *w*_{MIN} the lower bound, *β* the upper threshold of weight, *w*_{MAX} the upper bound, *T* the upper threshold of the difference between *w*_{
i
} and *w*_{
i
}’ , and *W* the final weight of the *i* th classifier.

Hitherto, the categorization performance of base classifiers could be measured accurately with a low time and computational overhead. The evaluation could be used for generating strong classifier in most reasonable way. Furthermore, the usage effectiveness of the training set is maximized by the CI. Theoretically, above algorithm should have better precision and higher efficiency than other boosting algorithms.

## 5. The final form of LDABoost

Combining works in previous sections together we can get the final framework of the novel TC system. It called LDABoost in this article.

Feature dimensionality reduction is the foundation of LDABoost. LDABoost uses LDA to modeling documents. Gibbs sampling method is used for estimating LDA’s parameters and LDA uses the estimated parameters to generating topics. Most representative topics are extracted by evaluating them with Mahalanobis distance to form the feature set. Improved multi-level NB method works on the feature set as weak learns. Weak learns vote on the category which document belonged to. Document sets are input twice in different order and the weights of base classifiers are calculated by introducing MI for performance judgment in each procedure. An adaptive strategy is used to calculating the final weight of a classifier according to the weights generated in the two-weighting procedure. Finally, the strong classifier is constructed similar with AdaBoost according to base classifiers’ weight.

- 1.
Input document set.

- 2.
Document set modeling.

- 3.
Model simplification and LDA parameters estimation.

- 4.
Topics features extraction.

- 5.
Train multi-level NB by training set.

- 6.
Weak classifiers formed a committee.

- 7.
Weak classifiers voting.

- 8.
Additional voting by input training samples in reverse order.

- 9.
Base classifiers’ classification performance evaluation according to MI.

- 10.
Weight allocation based on Steps 7–9

- 11.
According to the weights of weak classifiers to generate a strong classifier.

- 12.
Input test set.

- 13.
Text classification using LDABoost.

- 14.
Output category.

Follow the steps above, the object set of text will be classified in a high accuracy and high efficient way.

## 6. System application and analysis

The novel text classification tool which called LDABoost in this article is fully proposed in the former sections. To evaluating its performance in real world we made large number of tests to measure LDABoost’s precision and time consumption. In addition, we also deployed several experimental control groups and referenced a lot of related literatures to make our conclusion about the performance of LDABoost. We use same training sets and same test set downloaded from same corpora. What’s more, all experiments were done on the same platform. Therefore, the only variable is the classification tools.

**Hardware and software environments of the experiment**

Item | Product | Edition/Indicator |
---|---|---|

CPU | Intel Core 2 Duo | 2.93GHz |

Memory | Kingston DDR3 1333 | 2G |

Hard disk | Seagate ST500DM002 | 500G |

OS | Windows XP | Professional SP3 |

IDE | Eclipse | 3.4 |

Simulation tool | Matlab | 7.0 |

We use texts download from standard corpora. For evaluating its performance in different language, Reuters 21578 Classic text categorization corpus and CCL (Peking University modern Chinese corpus) are used.

### 6.1. Efficiency test and analysis

Huge time consumption is the major reason of why some theoretically high-accuracy classification algorithms could not get out of the laboratory. Therefore, when appraising a TC tool, the runtime complexity must be taken into account.

*k*-nearest neighbors, etc. We have chosen most four representative algorithms of them: neural networks, NB, SVM, and AdaBoost for the comparative experiment with LDABoost. Two hundred thousands of English texts were downloaded from Reuters 21587 and two hundred thousands of Chinese texts were downloaded from CCL. In each language, we use a hundred thousands of texts as the training set and the others as the test set. For controlling the number of variables, each text is 2 KB.txt document. For ease of display, logarithmic axe is used to indicating that the amount of documents. The results of test are shown in Figure 10.

In the above figure, the unit of *X*-axis is second (s) and the unit of *Y*-axis is log_{10} (number of documents). We choose different modes to evaluate the efficiency contribution of different strategies in LDABoost. LDABoost is the original LDABoost algorithm proposed in this article. LDABoost.1 uses VSM model and words for feature representation. LDABoost.2 uses LDA to reducing dimensionality but give up CI which is the smart mechanism of strong classifier generation.

Open source toolkits: JGibbLDA [22], svmcls 2.0 [23], ParzenPNN, and CLIF_NB [24] are used for the test. In order to meet the requirements of this article, we made some modifications to the source code.

Figure 10 reveals that NB has the highest efficiency and SVM needs the longest classification time. The time consumption of LDABoost is much lower than neural networks and SVM. In addition, it is more effective than original AdaBoost.

In all editions of LDABoost, LDABoost.2 has the best efficiency. That probably because CI leads to additional time overhead. However, the difference of time consumption between LDABoost and LDABoost.2 is small. The reason probably is CI has low runtime complexity. Using LDA for feature extraction can significantly improve efficiency according to the experiment result. Approximately 10% of time costs are saving by using topic features.

It is interesting to note that various tools have roughly the same efficiency in English and Chinese documents processing. Original AdaBoost is exception which has a bit higher time cost than neural works when classifying Chinese texts.

### 6.2. Experiment for precision analysis

Precision is the most important criterion for evaluating the performance of TC system. Since the most data in internet are textual information, the precision of TC will largely determine the extent of our information utilization, even affect our life quality.

**Precision of different algorithms in English TC**

Category algorithm | Society | Economics | Science | Politics | Military | Culture |
---|---|---|---|---|---|---|

NB | 0.781 | 0.769 | 0.774 | 0.799 | 0.772 | 0.773 |

Neural networks | 0.818 | 0.830 | 0.829 | 0.815 | 0.815 | 0.806 |

SVM | 0.859 | 0.868 | 0.863 | 0.868 | 0.864 | 0.871 |

AdaBoost | 0.864 | 0.860 | 0.853 | 0.854 | 0.867 | 0.866 |

LDABoost | 0.904 | 0.897 | 0.901 | 0.911 | 0.892 | 0.912 |

LDABoost.1 | 0.854 | 0.866 | 0.863 | 0.870 | 0.867 | 0.871 |

LDABoost.2 | 0.863 | 0.871 | 0.858 | 0.864 | 0.870 | 0.855 |

**Precision of different algorithms in Chinese TC**

Category algorithm | Society | Economics | Science | Politics | Military | Culture |
---|---|---|---|---|---|---|

NB | 0.785 | 0.769 | 0.771 | 0.794 | 0.769 | 0.772 |

Neural networks | 0.817 | 0.832 | 0.825 | 0.807 | 0.808 | 0.803 |

SVM | 0.847 | 0.867 | 0.852 | 0.862 | 0.860 | 0.871 |

AdaBoost | 0.856 | 0.848 | 0.851 | 0.855 | 0.861 | 0.859 |

LDABoost | 0.899 | 0.896 | 0.904 | 0.907 | 0.879 | 0.910 |

LDABoost.1 | 0.851 | 0.868 | 0.855 | 0.858 | 0.867 | 0.869 |

LDABoost.2 | 0.863 | 0.859 | 0.877 | 0.866 | 0.866 | 0.872 |

As shown in above tables, standard LDABoost has higher accuracy than other algorithm. The performance of LDABoost is far beyond NB and neural networks. In addition, the novel algorithm has better performance than SVM and original AdaBoost. That because boosting itself is a powerful ideology, LDA and CI further improve its performance. Comparative data of LDABoost.1 and LDABoost.2 proved the contribution of LDA and CI. Without both of them, LDABoost will be similar with original AdaBoost. Therefore, the performances of LDABoost.1 and LDABoost.2 are better than AdaBoost and worse than LDABoost.

We use 2,000, 4,000, 6,000, 8,000, 10,000 and 20,000 texts as the training sets. Figure 11 reveals that the precision of LDABoost increases very slowly while the size of training set increases largely. Although the algorithm proposed in this article is not absolutely size-independent, the correlation between algorithm’s accuracy and size of training set is low enough for building a high performance classification with very little manual cost.

Moreover, experimental results shown that there is no significant different between the English and the Chinese texts classification precisions. System can be considered as language-insensitive.

In a word, LDABoost is an excellent tool for TC, it achieves really high accuracy while control the runtime complexity in a very low degree. That because the feature extraction based on LDA improves the efficiency and accuracy, the two-procedure MI based strong classifier generation mechanism further enhances the precision.

## 7. Conclusion and future work

An improved boosting algorithm is proposed in this article. It uses LDA as the dimension reduction tool to extracting topic features. This method largely decreased the feature dimensionality. To the best of the authors’ knowledge, it is the first time LDA be introduced into boosting algorithm, this innovation enhance accuracy and efficiency at the same time. A multi-level NB algorithm is designed as weak classifiers. It keeps the advantage of high efficiency in original NB and has higher accuracy. Furthermore, different with AdaBoost, a two-procedure weighting algorithm which uses MI as the judgment of base classifiers’ performance is used to construct the final strong classifier. Experimental result shown that the novel algorithm has lower time consumption and higher efficiency than many other categorization tools. In addition, LDABoost is proved language-insensitive and not large training set dependent.

However, probably the parameters of LDA could be estimated in a more efficient and accurate way. Furthermore, LDABoost based on other weak classifiers such as C4.5, kNN, or SVM may achieve higher precision or lower runtime complexity. The utility of LDABoost in other classification tasks such as image processing and speaker identification should be tested. This will be undertaken as future works on this topic.

## Declarations

### Acknowledgement

The authors have partially been supported by the China Association for Science and Technology.

## Authors’ Affiliations

## References

- Thorleuchter D, Van den Poel D, Prinzie A: Mining ideas from textual information.
*Expert Syst Appl*2010, 37(10):7182-7188. 10.1016/j.eswa.2010.04.013View ArticleGoogle Scholar - Dinga C, Lib T, Peng W: On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic Indexing.
*Comput Stat Data An*2008, 52(8):3913-3927. 10.1016/j.csda.2008.01.011View ArticleMathSciNetGoogle Scholar - Blei DM, Ng AY, Jordan MI: Latent Dirichlet allocation.
*J Mach Learn Res*2003, 3(1):993-1022.MATHGoogle Scholar - Kim H, Howland P, Park H: Dimension reduction in text classification with support vector machines.
*J Mach Learn Res*2006, 6(1):37-53.MathSciNetMATHGoogle Scholar - Paris S, Raj B, Madhusudana S: Sparse and shift-invariant feature extraction from Non-negative data.
*Int Conf Acoust Spee*2008, 2069-2072.Google Scholar - Stark E: Indefiniteness and specificity in Old Italian texts.
*J Semitic Stud*2002, 19(3):315-332.Google Scholar - Claudia P, Foster P: Aggregation-based feature invention and relational concept classes.
*Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining*2003, 167-176.Google Scholar - Cui W, Liu S, Tan L, Conglei S, Yangqiu S, Zekai G, Huamin Q, Xin T: IEEE transactions on visualization and computer graphics. 2011, 17(12):2412-2421.View ArticleGoogle Scholar
- Feldman R, Sanger J:
*The Text Mining Handbook- Advanced Approaches in Analyzing Unstructured Data [M]*. Post &Telecom Press, Beijing; 2009:4-7.Google Scholar - Qiang C, Chengjie ZH: Selecting maximally separable feature subset for multiclass classification with applications to high-dimensional data.
*Lect Notes Comput Sc*2001, 8(2):1217-1233.Google Scholar - Newman D, Asuncion A, Smyth P, Welling M: Distributed inference for latent Dirichlet allocation.
*Adv Neural Inf Syst*2007, 16(4):1-9.Google Scholar - Xing D, Girolami M: Employing LatentDirichletAllocation for fraud detection in telecommunications.
*Pattern Recogn Lett*2007, 28(13):1727-1734. 10.1016/j.patrec.2007.04.015View ArticleGoogle Scholar - Chappell MA, Groves AR, Whitcher B, Woolrich MW: Variational Bayesian inference for a nonlinear forward model.
*IEEE T Signal Proces*2009, 57(1):223-236.MathSciNetView ArticleGoogle Scholar - Dong X, Gonzalez Ballester MA, Zheng G: Automatic extraction of femur contours from calibrated X-Ray images using statistical information.
*J of Mult*2007, 2(5):46-54.Google Scholar - Heinrich G: Parameter estimation for text analysis. http://www.arbylon.net/publication/text-est.pdf
- García-Serrano A, Benavent X, Granados R, Goñi-Menoyo JS: Some results using different approaches to merge visual and text-based features in CLEF’08 photo collection.
*9th Workshop of the Cross-Language Evaluation Forum*2009, 568-571.Google Scholar - Mitéran J, Matas J, Bourennane E, Paindavoine M, Dubois J: Automatic hardware implementation tool for a discrete adaboost-based decision algorithm.
*Eurasip J Appl Sig P*2005, 7: 1035-1046.View ArticleMATHGoogle Scholar - Tiantan C, Hongwei L, Shuisheng Z: Large scale classification with local diversity AdaBoost SVM algorithm.
*J Syst Engin El*2009, 20(6):1344-1350.Google Scholar - Schwenk H, Bengio Y: Boosting neural networks.
*Neural Comput*2000, 12(8):1869-1887. 10.1162/089976600300015178View ArticleGoogle Scholar - Rätsch G, Onoda T, Müller K-R: Soft Margins for AdaBoost.
*Mach Learn*2001, 42(3):287-320. 10.1023/A:1007618119488View ArticleMATHGoogle Scholar - Peng H, Long F, Ding C: Feature selection based on mutual information: criteria of Max-dependency, Max-relevance, and Min-redundancy.
*IEEE Trans. Pattern Anal. Mach. Intell*2005, 27(8):1226-1238.View ArticleGoogle Scholar - http://jgibblda.sourceforge.net
- Shaojun W, Qi L, Peng Y, Xiyuan P: CLS-SVM: a local modeling method for time series forecasting.
*Chinese J Scien Instrum*2011, 32(8):1824-1829.Google Scholar - Liu L, Song H, Lu Y: Method of CLIB_NB text classification learning based on naive bayes.
*Mini-Micro Syst*2005, 28(9):1575-1577.Google Scholar - Rish I: An empirical study of the naive Bayes classifier.
*IJCAI-01 Workshop on Empiracal Method in Artificial Intelligence*2001, 41-46.Google Scholar - Ruiz ME, Srinivasan P: Hierarchical text categorization using neural networks.
*Information Retrieval*2002, 5(1):87-118. 10.1023/A:1012782908347View ArticleMATHGoogle Scholar - Arun Kumar M, Gopal M: A comparison study on multiple binary-class SVM methods for unilabel text categorization.
*Pattern Recogn Lett*2010, 34(11):1437-1444.View ArticleGoogle Scholar - Romero E, Marquez L, Carreras X: Margin maximization with feed-forward neural networks: a comparative study with SVM and AdaBoost.
*Neurocomputing*2004, 57: 313-344.View ArticleGoogle Scholar

## Copyright

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.