Multiple Classifier Systems

Cluster analysis is an un-supervised learning technique that is widely used in the process of topic discovery from text. The research presented here proposes a novel un-supervised learning approach based on aggregation of clusterings produced by different clustering techniques. By examining and combining two different clusterings of a document collection, the aggregation aims at revealing a better structure of the data rather than imposing one that is imposed or constrained by the clustering method itself. When clusters of documents are formed, a process called topic extraction picks terms from the feature space (i.e. the vocabulary of the whole collection) to describe the topic of each cluster. It is proposed at this stage to re-compute terms weights according to the revealed cluster structure. The work further investigates the adaptive setup of the parameters required for the clustering and aggregation techniques. Finally, a topic accuracy measure is developed and used along with the F-measure to evaluate and compare the extracted topics and the clustering quality (respectively) before and after the aggregation. Experimental evaluation shows that the aggregation can successfully improve the clustering quality and the topic accuracy over individual clustering techniques.

We recently introduced the idea of solving cluster ensembles using a Weighted Shared nearest neighbors Graph (WSnnG). Preliminary experiments have shown promising results in terms of integrating different clusterings into a combined one, such that the natural cluster structure of the data can be revealed. In this paper, we further study and extend the basic WSnnG. First, we introduce the use of fixed number of nearest neighbors in order to reduce the size of the graph. Second, we use refined weights on the edges and vertices of the graph. Experiments show that it is possible to capture the similarity relationships between the data patterns on a compact refined graph. Furthermore, the quality of the combined clustering based on the proposed WSnnG surpasses the average quality of the ensemble and that of an alternative clustering combining method based on partitioning of the patterns' co-association matrix.

In this paper, we propose a cluster-based cumulative representation for cluster ensembles. Cluster labels are mapped to incrementally accumulated clusters, and a matching criterion based on maximum similarity is used. The ensemble method is investigated with bootstrap re-sampling, where the k-means algorithm is used to generate high granularity clusterings. For combining, group average hierarchical metaclustering is applied and the Jaccard measure is used for cluster similarity computation. Patterns are assigned to combined meta-clusters based on estimated cluster assignment probabilities. The cluster-based cumulative ensembles are more compact than co-association-based ensembles. Experimental results on artificial and real data show reduction of the error rate across varying ensemble parameters and cluster structures.

This paper presents a probabilistic model for combining cluster ensembles utilizing information theoretic measures. Starting from a co-association matrix which summarizes the ensemble, we extract a set of association distributions, which are modelled as discrete probability distributions of the object labels, conditional on each data object. The key objectives are, first, to model the associations of neighboring data objects, and second, to allow for the manipulation of the defined probability distributions using statistical and information theoretic means. A Jensen-Shannon Divergence based Clustering Combination (JSDCC) method is proposed. The method selects cluster prototypes from the set of association distributions based on entropy maximization and maximization of the generalized JS divergence among the selected prototypes. The method proceeds by grouping association distributions by minimizing their JS divergences to the selected prototypes. By aggregating the grouped association distributions, we can represent empirical cluster conditional probability distributions of the object labels, for each of the combined clusters. Finally, data objects are assigned to their most likely clusters, and their cluster assignment probabilities are estimated. Experiments are performed to assess the presented method and compare its performance with other alternative co-association based methods.

In this paper we propose a Multiple Classifier System (MCS) for classifying breast lesions in Dynamic Contrast Enhanced-Magnetic Resonance Imaging (DCE-MRI). The proposed MCS combines the results of two classifiers trained with dynamic and morphological features respectively. Twenty-one malignant and seventeen benign breast lesions, histologically proven, were analyzed. Volumes of Interest (VOIs) have been automatically extracted via a segmentation procedure assessed in a previous study. The performance of the MCS have been compared with histological classification. Results indicated that with automatic segmented VOIs 90% of test-set lesions were correctly classified.

Machine learning algorithms are spreading into many essential decision-making procedures of intelligent systems. These algorithms rely on big data which contain several sensitive features regarding the training instances such as gender, age, and race. In some contexts, these features might be useful and relevant like in health or other physiology-related frameworks. However, there are some fields where these features should be ignored like legal, financial, or educational incentivizing decisions. In complex decision-making procedures, it is important to guarantee indifference to these sensitive features. Choosing suitable algorithms or designing decision-making procedures that take this aspect of fairness into account is of crucial importance; especially from the reliability of upcoming intelligent systems which directly affects the impact of the new technology. In this study sample fairness assessment, experiments are performed in three different relevant contexts by applying two different ensemble building methods. It is shown how fairness can be assessed without hurting the accuracy. Furthermore, guidelines are provided for experimental design in order to derive more generic observations regarding incorporating the fairness notion into major classification algorithms 1 .

In this paper, we present biometric person recognition experiments in a real-world car environment using speech, face, and driving signals. We have performed experiments on a subset of the in-car CIAIR corpus collected at the Nagoya University, Japan. We have used Mel-frequency cepstral coefficients (MFCC) for speaker recognition. For face recognition, we have reduced the feature dimension of each face image through principal component analysis (PCA). As for modeling the driving behavior, we have employed features based on the pressure readings of acceleration and brake pedals and their timederivatives. For each modality, we use a Gaussian mixture model (GMM) to model each person's biometric data for classification. GMM is the most appropriate tool for audio and driving signals. For face, even though a nearestneighbor-classifier is the preferred choice, we have experimented with a single mixture GMM as well. We use background models for each modality and also normalize each modality score using an appropriate sigmoid function. At the end, all modality scores are combined using a weighted sum rule. The weights are optimized using held-out data. Depending on the ultimate application, we consider three different recognition scenarios: verification, closed-set identification and open-set identification. We show that each modality has a positive effect on improving the recognition performance.

Sensitivity indices are used to rank the importance of input design variables or components by estimating the degree of uncertainty of output variable influenced by the uncertainty generated from input variables or components. With the advent of highly complex engineering simulation models that describe the relationship between input variables and output response, the need for an efficient and effective sensitivity analysis is more demanding. Traditional importance measures either requires extensive random number generations or unable to measure variables interaction effects. In this article, a generalized approach that can provide efficient and accurate global sensitivity indices is developed. The approach consists of two steps; running an orthogonal array based experiment using moment-matched levels of the input variables followed by a variance contribution analysis. The benefits of the approach are demonstrated through different real life examples.

Combining classifiers by majority voting (MV) has recently emerged as an effective way of improving performance of individual classifiers. However, the usefulness of applying MV is not always observed and is subject to distribution of classification outputs in a multiple classifier system (MCS). Evaluation of MV errors (MVE) for all combinations of classifiers in MCS is a complex process of exponential complexity. Reduction of this complexity can be achieved provided the explicit relationship between MVE and any other less complex function operating on classifier outputs is found. Diversity measures operating on binary classification outputs (correct/incorrect) are studied in this paper as potential candidates for such functions. Their correlation with MVE, interpreted as the quality of a measure, is thoroughly investigated using artificial and real-world datasets. Moreover, we propose new diversity measure efficiently exploiting information coming from the whole MCS, rather than its part, for which it is applied.

The social media has made the world a global world and we, in addition to, as part of physical society, are now part of the virtual society as well. There has been the generation of a large amount of information over the social web. By way of increasing online information, new opportunities emerged, and diverse issues have been raised, which have attracted researchers to address these research problems. In this current age, where online business and e-commerce are part of our daily lives, recommender systems (RSs) are very effective for information filtering. RSs play a significant role in our lives by assisting users in recommending items and services what they may be interesting in to purchase or avail. In this research work, our goal is to predict the users’ ratings for various items, which are an active research area in collaborative filtering (CF). In this work, we have explored various similarity measures based on user-user and item-item rating predictions on different dataset...

The goal of this work is to introduce an architecture to automatically detect the amount of stress in the speech signal close to real time. For this an experimental setup to record speech rich in vocabulary and containing different stress levels is presented. Additionally, an experiment explaining the labeling process with a thorough analysis of the labeled data is presented. Fifteen subjects were asked to play an air controller simulation that gradually induced more stress by becoming more difficult to control. During this game the subjects were asked to answer questions, which were then labeled by a different set of subjects in order to receive a subjective target value for each of the answers. A recurrent neural network was used to measure the amount of stress contained in the utterances after training. The neural network estimated the amount of stress at a frequency of 25 Hz and outperformed the human baseline.

ABSTRACT: The specific objective of this paper was to provide a comparative analysis of three automatic classification algorithms: Quinlan's C4. 5 and two robust probabilistic classifiers like Support Vector Machine (SVM) and AdaBoost (a short for “adaptive boosting”). This work is part of a wider project whose general objective is to develop a methodology for the automatic classification, based on CORINE land-cover (CLC) classes, of high resolution multispectral IKONOS images. The dataset used for the comparison is ...

We detail an exploratory experiment aimed at determining the performance of stochastic vector quantisation as a purely fusion methodology, in contrast to its performance as a composite classification/fusion mechanism. To achieve this we obtain an initial pattern space for which a simulated PDF is generated: a well-factored SVQ classifier then acts as a composite classifier/classifier fusion system in order to provide an overall representation rate. This performance is then contrasted with that of the individual classifiers (constituted by the factored code-vectors) acting in combination via conventional combination mechanisms. In this way, we isolate the performance of networked-SVQs as a purely combinatory mechanism for the base classifiers.

A novel technique is being proposed to prune the
weights of artificial neural networks (ANNs) while training
with backpropagation algorithm. Iterative update of weights
through gradient descent mechanism does not guarantee
convergence in a specified number of epochs. Pruning of
non-relevant weights not only reduces the computational
complexity but also improves the classification performance.
This algorithm first defines the “relevance” of initialized
weights in a statistical sense by introducing a coefficient of
dominance for each weight converging on a hidden node and
subsequently employing the concept of complexity penalty.
Based upon complexity penalty for each weight, a decision
has been taken to either prune or retain the weight. It has
been shown analytically that a weight with higher complexity
penalty has a high degree of Fisher information which
further implies its ability to capture the variations in the
input set for better classification. Simulation experiments
performed with five benchmark data sets reveal that ANNs
trained after being pruned using the proposed technique
exhibit higher convergence, lower execution time and higher
success rate in the test phase and yields substantial reduction
in computational resources. For complex architectures, early
convergence was found to be directly correlated with percentage
of weights pruned. The efficacy of the technique has
been validated on several benchmark datasets having large
diversity of attributes.

The "one against one" and the "one against all" are the two most popular strategies for multi-class SVM; however, according to the literature review, it seems impossible to conclude which one is better for handwriting recognition. Thus, we compared these two classical strategies on two different handwritten character recognition problems. Several post-processing methods for estimating posterior probability were also evaluated and the results were compared with the ones obtained using MLP. Finally, the "one against all" strategy appears significantly more accurate for digit recognition, while the difference between the two strategies is much less obvious with upper-case letters. Besides, the "one against one" strategy is substantially faster to train and seems preferable for problems with a very large number of classes. To conclude, SVMs allow significantly better estimation of probabilities than MLP, which is promising from the point of view of their incorporation into handwriting recognition systems.

The "one against one" and the "one against all" are the two most popular strategies for multi-class SVM; however, according to the literature review, it seems impossible to conclude which one is better for handwriting recognition. Thus, we compared these two classical strategies on two different handwritten character recognition problems. Several post-processing methods for estimating posterior probability were also evaluated and the results were compared with the ones obtained using MLP. Finally, the "one against all" strategy appears significantly more accurate for digit recognition, while the difference between the two strategies is much less obvious with upper-case letters. Besides, the "one against one" strategy is substantially faster to train and seems preferable for problems with a very large number of classes. To conclude, SVMs allow significantly better estimation of probabilities than MLP, which is promising from the point of view of their incorporation into handwriting recognition systems.

Recently, ensemble techniques have also attracted the attention of Genetic Programing (GP) researchers. The goal is to further improve GP classification performances. Among the ensemble techniques, also bagging and boosting have been taken into account. These techniques improve classification accuracy by combining the responses of different classifiers by using a majority vote rule. However, it is really hard to ensure that classifiers in the ensemble be appropriately diverse, so as to avoid correlated errors. Our approach tries to cope with this problem, designing a framework for effectively combine GP-based ensemble by means of a Bayesian Network. The proposed system uses two different approaches. The first one applies a boosting technique to a GP-based classification algorithm in order to generate an effective decision trees ensemble. The second module uses a Bayesian network for combining the responses provided by such ensemble and select the most appropriate decision trees. The Bayesian network is learned by means of a specifically devised Evolutionary algorithm. Preliminary experimental results confirmed the effectiveness of the proposed approach.

In many real world applications, the cost of a wrong decision may be much higher than the benefit of having a high classification rate. This kind of classification problems represent very challenging tasks, because they require highly reliable systems and may benefit of introducing a reject option. To this purpose, many classification systems have been proposed, among which classifier ensembles represent a successful example. This approach aims at combining classifiers making uncorrelated errors. In this framework, the Random Forest (RF) represents a case of special interest. A RF is made of a suitable ensemble of decision trees and has proved to be very effective in several fields. In this paper, the RF classification reliability is experimentally analyzed with reference to the case of handwriting recognition. The aim is to verify if such reliability can be effectively used to introduce a new reject option, which considers, for each unknown sample, a subset of few classes including with high probability the correct one. The whole system operates as a pre-classification stage and the reject option should allow us to obtain a low error rate without significantly affecting the recognition rate. Experiments, carried out on two real world datasets, have shown the effectiveness of the proposed method.

Most of the methods for combining classifiers rely on the assumption that the experts to be combined make uncorrelated errors. Unfortunately, this theoretical assumption is not easy to satisfy in practical cases, thus effecting the performance obtainable by applying any combination strategy. We tried to solve this problem by explicitly modeling the dependencies among the experts through the estimation of the joint probability distributions among the outputs of the classifiers and the true class. In this paper we propose a new weighted majority vote rule, that uses the joint probabilities of each class as weights for combining classifier outputs. A Bayesian Network automatically infers the joint probability distribution for each class. The final decision is made by taking into account both the votes received by each class and the statistical behavior of the classifiers. The experimental results confirmed the effectiveness of the proposed method.

Vehicle classification has been a highly focused topic amongst the scientific community due to its application in automatic tolling systems, road surveillance, traffic monitoring, etc. Image-based classification using computer vision techniques provides a non-invasive, cost effective, automated option to these systems. In traffic video surveillance systems vehicle detection and classification help provide more accurate and detailed statistics that can be used in intelligent transport systems. We implemented a non-invasive vision based vehicle classification system able to operate in both multilane free flow environments and in toll stations. Additionally, it can be used in surveillance cameras present in various motorways to provide not only quantitative information about flowing traffic but also qualitative. In this work we implement three hog-based descriptors as well as an additional edge points groups SIFT-based descriptor. These were tested on four distinct datasets captured us...

One of the main factors affecting the effectiveness of ECOC methods for classification is the dependence among the errors of the computed codeword bits. We present an extensive experimental work for evaluating the dependence among output errors of the decomposition unit of ECOC learning machines. In particular, we compare the dependence between ECOC Multi Layer Perceptrons (ECOC monolithic), made up by a single MLP, and ECOC ensembles made up by a set of independent and parallel dichotomizers (ECOC PND), using measures based on mutual information. In this way we can analyze the relations between performances, design and dependence among output errors in ECOC learning machines. Results quantitatively show that the dependence among computed codeword bits is significantly smaller for ECOC PND, pointing out that ensembles of independent dichotomizers are better suited for implementing ECOC classification methods.

Statistical classification of hyperspectral data is challenging because the input space is high in dimension and correlated, but labeled information to characterize the class distributions is typically sparse. The resulting classifiers are often unstable and have poor generalization. A new approach that is based on the concept of random forests of classifiers and implemented within a multiclassifier system arranged as a binary hierarchy is proposed. The primary goal is to achieve improved generalization of the classifier in analysis of hyperspectral data, particularly when the quantity of training data is limited. The new classifier incorporates bagging of training samples and adaptive random subspace feature selection with the Binary Hierarchical Classifier (BHC), such that the number of features that is selected at each node of the tree is dependent on the quantity of associated training data. Classification results from experiments on data acquired by the Hyperion sensor on the NASA EO-1 satellite over the Okavango Delta of Botswana are superior to those from our original best basis BHC algorithm, a random subspace extension of the BHC, and a random forest implementation using the CART classifier.

Training a parametric classi er involves the use of a training set of data with known la-beling to estimate or \learn" the parameters of the chosen model. A test set, consisting of patterns not previously seen by the classi er, is then used to determine the classi cation performance. ...

Recently, ensemble techniques have also attracted the attention of Genetic Programing (GP) researchers. The goal is to further improve GP classification performances. Among the ensemble techniques, also bagging and boosting have been taken into account. These techniques improve classification accuracy by combining the responses of different classifiers by using a majority vote rule. However, it is really hard to ensure that classifiers in the ensemble be appropriately diverse, so as to avoid correlated errors. Our approach tries to cope with this problem, designing a framework for effectively combine GP-based ensemble by means of a Bayesian Network. The proposed system uses two different approaches. The first one applies a boosting technique to a GP-based classification algorithm in order to generate an effective decision trees ensemble. The second module uses a Bayesian network for combining the responses provided by such ensemble and select the most appropriate decision trees. The Bayesian network is learned by means of a specifically devised Evolutionary algorithm. Preliminary experimental results confirmed the effectiveness of the proposed approach.

This paper proposes a new ensemble method that constructs an ensemble of tree-structured classifiers using multi-view learning. We are motivated by the fact that an ensemble can outperform its members providing that these classifiers are diverse and accurate. In order to construct diverse individual classifiers, we assume that the object to be classified is described by multiple feature sets (views). We aim to construct different tree classifiers using different views to improve the accuracy of the multi-class recognition task. For the decision fusion of the binary classifiers within each tree classifier, Dempster's unnormalized rule of combination is applied and an evidence-theoretic decision profile is proposed to combine the decisions of different trees. Numerical experiments have been performed on two real-world data sets: a data set of handwritten digits, and another data set of 3D visual objects. The results indicate that the proposed forest efficiently integrates multi-view data and outperforms the individual tree classifiers.

For the last few decades, learning based on multiple kernels, such as the ensemble kernel regressor and the multiple kernel regressor, has attracted much attention in the field of machine learning. Although its efficacy was revealed numerically in many works, its theoretical ground is not investigated sufficiently. In this paper, we discuss regression problems with a class of kernels whose corresponding reproducing kernel Hilbert spaces have a common subspace with an invariant metric and show that the ensemble kernel regressor (the mean of kernel regressors with those kernels) gives a better learning result than the multiple kernel regressor (the kernel regressor with the sum of those kernels) in terms of the generalization ability of a model space.

For the last few decades, learning based on multiple kernels, such as the ensemble kernel regressor and the multiple kernel regressor, has attracted much attention in the field of machine learning. Although its efficacy was revealed numerically in many works, its theoretical ground is not investigated sufficiently. In this paper, we discuss regression problems with a class of kernels whose corresponding reproducing kernel Hilbert spaces have a common subspace with an invariant metric and show that the ensemble kernel regressor (the mean of kernel regressors with those kernels) gives a better learning result than the multiple kernel regressor (the kernel regressor with the sum of those kernels) in terms of the generalization ability of a model space.

AdaBoost and other ensemble methods have successfully been applied to a number of classi cation tasks, seemingly defying problems of over tting. AdaBoost performs gradient descent i n a n e r r o r function with respect to the margin, asymptotically concentrating on the patterns which are hardest to learn. For very noisy problems, however, this can be disadvantageous. Indeed, theoretical analysis has shown that the margin distribution, as opposed to just the minimal margin, plays a crucial role in understanding this phenomenon. Loosely speaking, some outliers should be tolerated if this has the bene t of substantially increasing the margin on the remaining points. We propose a new boosting algorithm which a llows for the possibility of a pre-speci ed fraction of points to lie in the margin area or even on the wrong side of the decision boundary.

We present an approach for the automatic classification of Nuclear Magnetic Resonance Spectroscopy data of biofluids with respect to drug induced organ toxicities. Classification is realized by an Ensemble of Support Vector Machines, trained on different subspaces according to a modified version of Random Subspace Sampling. Features most likely leading to an improved classification accuracy are favored by the determination of subspaces, resulting in an improved classification accuracy of base classifiers within the Ensemble. An experimental evaluation based on a challenging, real task from pharmacology proves the increased classification accuracy of the proposed Ensemble creation approach compared to single SVM classification and classical Random Subspace Sampling.

A communication model for the Hypothesis Boosting (HB) problem is proposed. Under this model, AdaBoost algorithm can be viewed as a threshold decoding approach for a repetition code. Generalization of such decoding view under theory of theory of Recursive Error Correcting Codes allows the formulation of a generalized class of low-complexity learning algorithms applicable in high dimensional classification problems. In this paper, an instance of this approach suitable for High Dimensional Features Spaces (HDFS) is presented.

In this paper, a novel hybrid kernel machine ensemble is proposed for abnormal ECG beat detection to facilitate long-term monitoring of heart patients. A binary SV M is trained using ECG beats from different patients to adapt to the reference values based on the general patient population. A one-class SV M is trained using only normal ECG beats from a specific patient to adapt to the specific reference value of the patient. Trained using different data sets, these two SV M s usually perform differently in classifying ECG beats of that specific patient. Therefore, integration of the two types of SV M s is expected to perform better than using either of them separately and that improving the generalization. Experimental results using MIT/BIH arrhythmia ECG database show good performance of our proposed ensemble and support its feasibility in practical clinical application.

In this paper, a novel hybrid kernel machine ensemble is proposed for abnormal ECG beat detection to facilitate long-term monitoring of heart patients. A binary SV M is trained using ECG beats from different patients to adapt to the reference values based on the general patient population. A one-class SV M is trained using only normal ECG beats from a specific patient to adapt to the specific reference value of the patient. Trained using different data sets, these two SV M s usually perform differently in classifying ECG beats of that specific patient. Therefore, integration of the two types of SV M s is expected to perform better than using either of them separately and that improving the generalization. Experimental results using MIT/BIH arrhythmia ECG database show good performance of our proposed ensemble and support its feasibility in practical clinical application.

Recently, the problem of imbalanced data is the focus of intense research of machine learning community. Following work tries to utilize an approach of transforming the data space into another where classification task may become easier. Paper contains a proposition of a tool, based on a photographic metaphor to build a classifier ensemble, combined with a random subspace approach. Developed solution is insensitive to a sample size and robust to dimension increase, which allows a regularization of feature space, reducing the impact of biased classes. The proposed approach was evaluated on the basis of the computer experiments carried out on the benchmark and synthetic datasets.

This paper provides a three-step framework to predict user assessment of the suitability of movies for an inflight viewing context. For this, we employed classifier stacking strategies. First of all, using the different modalities of training data, twenty-one classifiers were trained together with a feature selection algorithm. Final predictions were then obtained by applying three classifier stacking strategies. Our results reveal that different stacking strategies lead to different evaluation results. A considerable improvement can be found for the F1-score when using the label stacking strategy.

This paper provides a three-step framework to predict user assessment of the suitability of movies for an inflight viewing context. For this, we employed classifier stacking strategies. First of all, using the different modalities of training data, twenty-one classifiers were trained together with a feature selection algorithm. Final predictions were then obtained by applying three classifier stacking strategies. Our results reveal that different stacking strategies lead to different evaluation results. A considerable improvement can be found for the F1-score when using the label stacking strategy.

Multiple modalities present potential difficulties for kernel-based pattern recognition in consequence of the lack of intermodal kernel measures. This is particularly apparent when training sets for the differing modalities are disjoint. Thus, while it is always possible to consider the problem at the classifier fusion level, it is conceptually preferable to approach the matter from a kernel-based perspective. By interpreting the aggregate of disjoint training sets as an entire data set with missing inter-modality measurements to be filled in by appropriately chosen substitutes, we arrive at a novel kernel-based technique, the neutral-point method. On further theoretical analysis, it transpires that the method is, in structural terms, a kernel-based analog of the well-known sum rule combination scheme. We therefore expect the method to exhibit similar error-canceling behavior, and thus constitute a robust and conservative strategy for the treatment of kernel-based multi-modal data.

In the field of machine learning, the ensemble has been employed as a common methodology to improve the performance upon multiple base classifiers. However, the true predictions are often canceled out by the false ones during consensus due to a phenomenon called "curse of correlation" which is represented as the strong interferences among the predictions produced by the base classifiers. In addition, the existing practices are still not able to effectively mitigate the problem of imbalanced classification. Based on the analysis on our experiment results, we conclude that the two problems are caused by some inherent deficiencies in the approach of consensus. Therefore, we create an enhanced ensemble algorithm which adopts a designed rank-based chain-mode consensus to overcome the two problems. In order to evaluate the proposed ensemble algorithm, we employ a well-known benchmark data set NSL-KDD (the improved version of dataset KDDCup99 produced by University of New Brunswick) to make comparisons between the proposed and 8 common ensemble algorithms. Particularly, each compared ensemble classifier uses the same 22 base classifiers, so that the differences in terms of the improvements toward the accuracy and reliability upon the base classifiers can be truly revealed. As a result, the proposed rank-based chain-mode consensus is proved to be a more effective ensemble solution than the traditional consensus approach, which outperforms the 8 ensemble algorithms by 20% on almost all compared metrices which include accuracy, precision, recall, F1-score and area under receiver operating characteristic curve. Keywords-Consensus, curse of correlation, imbalanced classification, rank-based chain-mode ensemble. I. INTRODUCTION N the field of Machine Learning (ML), ensemble is an effective approach to improve the accuracy upon multiple base classifiers. Significant efforts on ensemble algorithms have been made in recent years [1], which have been applied to various domains such as classification [2], regression [3] and clustering [4]-[6]. The overwhelming majority of the practices are adopting the same framework named Multiple Classifier System (MCS) [7] which consists of a defective procedure called consensus. Since the consensus is responsible for making the final prediction (i.e., labeling) via combining multiple mutualconflicted predictions made by the base classifiers, such a combination process will introduce four issues in practice: (1)

There are two different stages to consider when constructing multiple classifier systems: The Meta-Classifier Stage that is responsible for the combination logic and basically treats the ensemble members as black boxes, and the Classifier Stage where the functionality of members is in focus. Furthermore, on the upper stage-also called voting strategy stage-the method of combining members can be done by fusion and selection of classifiers. In this paper, we propose a novel procedure for building the meta-classifier stage of MCSs, using an oracle of three-level voting strategy. This is a dynamic, half fusion-half selection type method for ensemble member combination, which is midway between the extremes of fusion and selection. The MCS members are weighted and combined with the help of the oracle, which is founded on a voting strategy of three levels: (1) The Local Implicit Confidence (LIC), (2) The Global Explicit Confidence (GEC), and (3) The Local Explicit Confidence (LEC). The first confidence segment is dependent of classifier construction, via the implicit knowledge gathered simultaneously with training. Since this strongly depends on the internal operation of the classifier, it can not always be obtained, for example, when using some particularly complex classification methods. We used several, known classifier algorithms (Decision Trees, Neural Networks, Logistic Regression, SVM) where it is possible to extract this information. The second goodness index is calculated on the validation partition of the labeled train data. It is used to obtain the general accuracy of a single classifier using a data set independent of the training partition. And finally, the third part of the confidence triplet depends also on the unlabeled objects yet to be classified. Due to this, it can only be calculated in classification time.

One of the challenges in machine learning is the classification of datasets with ambiguous instances. In this paper we study specifically datasets with examples that have overlapping feature values for different classes. In these circumstances there is a bound on the classification performance. While there seems to be a race for accuracy, very little has been done to understand and solve the issues related to ambiguous data where the possible classification performance is limited. We discuss the use of SVMs in a proposed scheme to handle classification in such problem domains. A new approach is offered that tries to separate ambiguous data from the data that are much simpler to classify in order to prevent their influence on the classification process. We demonstrate that by separating the ambiguous data, although we lose some data, the performance of the classification increases significantly. In contrast to previous findings with some other classifiers, our experimental results sh...

Combining the predictions of a set of classifiers has been shown to be an effective way to create composite classifiers that are more accurate than any of the component classifiers. Increased accuracy has been shown in a variety of real-world applications, ranging from protein sequence identification to determining the fat content of ground meat. Despite such individual successes, the answers are not known to fundamental questions about classifier combination, such as "Can classifiers from any given model class be combined to create a composite classifier with higher accuracy?" or "Is it possible to increase the accuracy of a given classifier by combining its predictions with those of only a small number of other classifiers?". The goal of this dissertation is to provide answers to these and closely related questions with respect to a particular model class, the class of nearest neighbor classifiers. We undertake the first study that investigates in depth the combination of nearest neighbor classifiers. Although previous research has questioned the utility of combining nearest neighbor classifiers, we introduce algorithms that combine a small number of component nearest neighbor classifiers, where each of the components stores a small number of prototypical instances. In a variety of domains, we show that these algorithms yield composite classifiers that are more accurate than a nearest neighbor classifier that stores all training instances as prototypes. The research presented in this dissertation also extends previous work on prototype selection for an independent nearest neighbor classifier. We show that in many domains, storing a very small number of prototypes can provide classification accuracy greater than or equal to that of a nearest neighbor classifier that stores all training instances. We extend previous work by demonstrating that algorithms that rely primarily on random sampling can effectively choose a small number of prototypes. v

Documentation of software architecture is a good approach to understand the architecture of a software system and to match it with the changes needed during the software maintenance phase. In several systems like legacy or older systems these documents are not available or if available; they are not up-to-date and usable. So, reconstruction of architecture in order to maintain these systems is a necessary activity. When we talk about software architecture, usually we are looking for a modular view of architecture with low coupling and high cohesion. In this paper, we try to improve the algorithm of machine-learning which is presented before for architecture recovery, and we propose using it for architecture reconstruction in order to obtain optimum modularity in the architecture with low coupling and high cohesion. The proposed algorithm is evaluated in a case study, and its results are presented.

In recent times, the manufacturing processes are faced with many external or internal (the increase of customized product rescheduling , process reliability,..) changes. Therefore, monitoring and quality management activities for these manufacturing processes are difficult. Thus, the managers need more proactive approaches to deal with this variability. In this study, a proactive quality monitoring and control approach based on classifiers to predict defect occurrences and provide optimal values for factors critical to the quality processes is proposed. In a previous work (Noyel et al. 2013), the classification approach had been used in order to improve the quality of a lacquering process at a company plant; the results obtained are promising, but the accuracy of the classification model used needs to be improved. One way to achieve this is to construct a committee of classifiers (referred to as an ensemble) to obtain a better predictive model than its constituent models. However, the selection of the best classification methods and the construction of the final ensemble still poses a challenging issue. In this study, we focus and analyze the impact of the choice of classifier types on the accuracy of the classifier ensemble; in addition, we explore the effects of the selection criterion and fusion process on the ensemble accuracy as well. Several fusion scenarios were tested and compared based on a real-world case. Our results show that using an ensemble classification leads to an increase in the accuracy of the classifier models. Consequently, the monitoring and control of the considered real-world case can be improved.

Hyperspectral images are one of the most important data source for land cover analysis. These images encode information about the earth surface expressed in terms of spectral bands, allowing us to precisely classify and identify materials of interest. An approach that has been widely used is the combination of various classification methods in order to produce a more accurate thematic map based on classification of hyperspectral images. Our multi-objective remote sensed hyperspectral image classifier combiner (MORSHICC) approach uses a genetic algorithm-based strategy for choosing the best subset of classifiers, that is, the one which provides higher accuracy with the fewest possible amount of classifiers. We propose to use combiners that linearly weigh each classification approach through Genetic Algorithm (WLC-GA) and Integer Linear Programming (WLC-ILP). For building the combiners, we used three data representations and four learning algorithms, producing twelve classification approaches such that the multi-objective approach can select the best subset. Experimental results on well-known datasets show that the MORSHICC approach with WLC-GA and WLC-IP not only produces combiners with fewer classifier approaches but also improves the final accuracy rates. Therefore, these combiners may produce more accurate thematic maps for real and large datasets in a short time.

In this paper, a novel hybrid kernel machine ensemble is proposed for abnormal ECG beat detection to facilitate long-term monitoring of heart patients. A binary SV M is trained using ECG beats from different patients to adapt to the reference values based on the general patient population. A one-class SV M is trained using only normal ECG beats from a specific patient to adapt to the specific reference value of the patient. Trained using different data sets, these two SV M s usually perform differently in classifying ECG beats of that specific patient. Therefore, integration of the two types of SV M s is expected to perform better than using either of them separately and that improving the generalization. Experimental results using MIT/BIH arrhythmia ECG database show good performance of our proposed ensemble and support its feasibility in practical clinical application.

We present a methodology to analyze Multiple Classifiers Systems (MCS) performance, using the disagreement concept. The goal is to define an alternative approach to the conventional recognition rate criterion, which usually requires an exhaustive combination search. This approach defines a Distance-based Disagreement (DbD) measure using an Euclidean distance computed between confusion matrices and a soft-correlation rule to indicate the most likely candidates to the best classifiers ensemble. As case study, we apply this strategy to two different handwritten recognition systems. Experimental results indicate that the method proposed can be used as a low-cost alternative to conventional approaches.

The accuracy of predictions is better if the combinations of the different approaches are used. Currently in collaborative filtering research, the linear blending of various methods is used. More accurate classifiers can be obtained by combining less accurate ones. This approach is called ensembles of classifiers. Different collaborative filtering methods uncover the different aspects of the dataset. Some of them are good at finding out local relationships; the others work for the global characterization of the data. Ensembles of different collaborative filtering algorithms can be created to provide more accurate recommender systems.

Membrane protein prediction is a significant classification problem, requiring the integration of data derived from different sources such as protein sequences, gene expression, protein interactions etc. A generalized probabilistic approach for combining different data sources via supervised selective kernel fusion was proposed in our previous papers. It includes, as particular cases, SVM, Lasso SVM, Elastic Net SVM and others. In this paper we apply a further instantiation of this approach, the Supervised Selective Support Kernel SVM and demonstrate that the proposed approach achieves the top-rank position among the selective kernel fusion variants on benchmark data for membrane protein prediction. The method differs from the previous approaches in that it naturally derives a subset of "support kernels" (analogous to support objects within SVMs), thereby allowing the memory-efficient exclusion of significant numbers of irrelevant kernel matrixes from a decision rule in a manner particularly suited to membrane protein prediction.

We have previously introduced, in purely theoretical terms, the notion of neutral point substitution for missing kernel data in multimodal problems. In particular, it was demonstrated that when modalities are maximally disjoint, the method is precisely equivalent to the Sum rule decision scheme. As well as forging an intriguing analogy between multikernel and decision-combination methods, this finding means that the neutral-point method should exhibit a degree of resilience to class misattribution within the individual classifiers through the relative cancelling of combined estimation errors (if sufficiently decorrelated). However, the case of completely disjoint modalities is unrepresentative of the general missing data problem. We here set out to experimentally test the notion of neutral point substitution in a realistic experimental scenario with partially-disjoint data to establish the practical application of the method. The tested data consists in multimodal Biometric measurements of individuals in which the missing-modality problem is endemic. We hence test a SVM classifier under both the modal decision fusion and neutral point-substitution paradigms, and find that, while error cancellation is indeed apparent, the genuinely multimodal approach enabled by the neutral-point method is superior by a significant factor.