Novel ensemble methods for regression via classification problems

2015

This paper reviewed the current state-of-the-art of optimization of ensemble methods so as to provide us with a better direction of how we will conduct our research in the future. The primary aim of ensemble method is to integrate a set of models that are used for solving different tasks so as to come up with enhanced composite global model, which produces higher accuracy and reliable estimate than what can be achieved through a single model. Diversity, combination strategies, number of based classifiers, types of ensemble, and performance measures are the key factors to be considered in the build of committees. When the numbers of base classifiers become huge, ensemble methods incurred high storage space and computational time, selective ensemble is proposed by most literatures to solve these problems. In terms of optimization techniques, multi-objectives techniques have become the better ones to use due to their efficiency in terms of optimization process and they provide a set of...

The modern technologies, which are characterized by cyber-physical systems and internet of things expose organizations to big data, which in turn can be processed to derive actionable knowledge. Machine learning techniques have vastly been employed in both supervised and unsupervised environments in an effort to develop systems that are capable of making feasible decisions in light of past data. In order to enhance the accuracy of supervised learning algorithms, various classification-based ensemble methods have been developed. Herein, we review the superiority exhibited by ensemble learning algorithms based on the past that has been carried out over the years. Moreover, we proceed to compare and discuss the common classification-based ensemble methods, with an emphasis on the boosting and bagging ensemble-learning models. We conclude by out setting the superiority of the ensemble learning models over individual base learners.

Ensemble methods have been called the most influential development in Data Mining and Machine Learning in the past decade. They combine multiple models into one usually more accurate than the best of its components. Ensembles can provide a critical boost to industrial challenges -from investment timing to drug discovery, and fraud detection to recommendation systems -where predictive accuracy is more vital than model interpretability.

Data mining techniques like classification is effectively for used for prediction. Due to technological up gradation, the datasets which are large are distributed over different locations and classification has become a difficult task. The single classifier models are not sufficient for these types of datasets. So the recent research concentrates on combination of various classifiers and creates models. Ensemble methods combine multiple models and are useful in both supervised and unsupervised learning. This paper discusses the framework of ensemble and two types of ensemble models. A review of various algorithms of these two models is given. Combination methods which are used for combining outputs and few applications where it can be used effectively are also discussed. Key TermsData mining techniques like classification is effectively for used for prediction. Due to technological up gradation, the datasets which are large are distributed over different locations and classification has become a difficult task. The single classifier models are not sufficient for these types of datasets. So the recent research concentrates on combination of various classifiers and creates models. Ensemble methods combine multiple models and are useful in both supervised and unsupervised learning. This paper discusses the framework of ensemble and two types of ensemble models. A review of various algorithms of these two models is given. Combination methods which are used for combining outputs and few applications where it can be used effectively are also discussed.

IEEE Access

The recent advances in computing power and machine learn-13 ing (ML) have given rise to several innovations and devel-14 opments in many areas of research and human lives. In the 15 last few years, advancements in machine learning, a subset 16 of artificial intelligence (AI), have transformed and inher-17 ently changed almost every area of our lives [1], [2]. Par-18 ticularly, ML has been applied in disease diagnosis [3], 19 fraud detection [4], text classification [5], and image recog-20 nition [6], among others. Unlike humans, ML algorithms 21 consider several factors when making decisions, and they 22 are not prone to fatigue or prejudice. Meanwhile, learn-23 ing can sometimes be challenging, especially when learning 24 from high-dimensional and imbalanced datasets [7],[8], [9]. 25 Research has shown that conventional ML algorithms tend 26 to underperform when trained with imbalanced datasets [10]. 27 Therefore, researchers have frequently resorted to new and 28 improved learning approaches, such as ensemble and deep 29 learning. 30 weighted majority voting is presented in [57]. Meanwhile, 230 there are several combination rules in the literature. In [58], 231 some combination rules were introduced, including the min-232 imum, maximum, product, median, and sum rules. 233 B. ENSEMBLE SELECTION 234 Ensemble selection is a technique used to build ensemble 235 classifiers from a set of base models. It is a vital topic in 236 ensemble learning because selecting a suitable subset of base 237 models could lead to better performance than when all the 238 models are used to construct the ensemble classifier [59]. 239 Since the base models are developed using various ML algo-240 rithms or different subsets of the training data, their perfor-241 mances would be different; while some would have good 242 performance, others might have poor performance. Instead 243 of combining the good and bad models, selecting only a 244 subset of models with good performance might be beneficial, 245 which would enhance the overall ensemble performance [60]. 246 An ensemble selection strategy is employed to select the 247 optimal subset of base classifiers, and they are usually guided 248 by a scoring function [61]. Caruana et al. [60] developed the 249 foremost forward model selection strategy to extract the best 250 performing subset of base models, and its basic procedure is 251 outlined as follows: 252 i. Begin with the empty ensemble. 253 ii. Select the base classifier from the library that maximizes 254 the ensemble's performance using a validation set. 255 iii. Repeat step II for a predefined number of iterations or 256 until all the models in the library have been examined. 257 iv. Return the subset of models that produces the best per-258 formance on the validation set. 259 The forward model selection strategy is fast and efficient 260 but occasionally overfits, leading to poor ensemble perfor-261 mance. Hence, several ensemble selection strategies have 262 been recently proposed; for example, Sun and Pfahringer [61] 263 developed the bagging ensemble selection, which combines 264 bagging and ensemble selection. Furthermore, the ensemble 265 selection strategies can be divided into static and dynamic 266 methods. The static strategy selects a single subset of base 267 models during model training and applies it to predict all 268 the unseen instances. The static selection methods can be 269 grouped into ordering-based techniques and optimization-270 based techniques. 271 As the name implies, ordering-based techniques attempt to 272 order the base models with respect to specific criteria, and 273 only the top models are chosen as the optimal subset. Some 274 criteria for ordering the base models include validation error 275 and kappa measure [59]. Guo et al. [62] recently developed 276 a method to order the base models via an evaluation metric 277 that takes the margin and diversity into consideration. Mean-278 while, optimization-based techniques formulate the selec-279 tion process as an optimization problem that can be solved 280 using mathematical programming or heuristic optimization. 281 Static methods limits the flexibility of the ensemble selection process. 843 the study employed the SMOTE Tomek link (SMOTETomek) 844 to create new datasets with even class distribution since 845 all four datasets are imbalanced. Secondly, the study uses 846 the isolation forest (iForest) algorithm [135] to detect and 847 remove outliers in the datasets. Meanwhile, the iForest is an 848 ensemble implementation that creates isolation trees using 849 the given dataset. The isolation trees are repeatedly devel-850 oped by splitting the training set until all the samples are 851 isolated or the specified tree height is obtained. The exper-852 imental results show that the proposed approach achieved 853 superior classification performance than other methods and 854 prior research works. The proposed ensemble obtained 855 96.7%, 85.8%, 75.8%, and 100% accuracy when predicting 856 type 2 diabetes, hypertension, prehypertension, and CKD, 857 respectively. 858 Kazemi and Mirroshandel [136] proposed a novel ensem-859 ble method for the early detection of kidney stones, a common 860 disease affecting people globally. The study employed several 861 ML algorithms, including decision trees, naïve Bayes, and 862 artificial neural networks (ANN), to learn the relationships 863 between some biological features related to kidney stone 864 disease. Furthermore, the study developed a novel method 865 to combine the different classifiers. The combination method 866 involved assigning weights to the different classifiers via a 867 genetic algorithm (GA) based computation. Meanwhile, the 868 data used in the research was obtained between 2012 and 869 2016 from 936 patients with kidney stone disease. The pro-870 posed ensemble achieved a classification accuracy of 97.1%.

Springer eBooks, 2020

Ensemble learning has been shown to significantly improve predictive accuracy in a variety of machine learning problems. For a given predictive task, the goal of ensemble learning is to improve predictive accuracy by combining the predictive power of multiple models. In this paper, we present an ensemble learning algorithm for regression problems which leverages the distribution of the samples in a learning set to achieve improved performance. We apply the proposed algorithm to a problem in precision medicine where the goal is to predict drug perturbation effects on genes in cancer cell lines. The proposed approach significantly outperforms the base case.

2011

Computational Statistics & Data Analysis, 2007

A robust classification procedure is developed based on ensembles of classifiers, with each classifier constructed from a different set of predictors determined by a random partition of the entire set of predictors. The proposed methods combine the results of multiple classifiers to achieve a substantially improved prediction compared to the optimal single classifier. This approach is designed specifically for high-dimensional data sets for which a classifier is sought. By combining classifiers built from each subspace of the predictors, the proposed methods achieve a computational advantage in tackling the growing problem of dimensionality. For each subspace of the predictors, we build a classification tree or logistic regression tree. Our study shows, using four real data sets from different areas, that our methods perform consistently well compared to widely used classification methods. For unbalanced data, our approach maintains the balance between sensitivity and specificity more adequately than many other classification methods considered in this study.

Bagging, boosting and random subspace methods are well known re-sampling ensemble methods that generate and combine a diversity of learners using the same learning algorithm for the base-regressor. In this work, we built an ensemble of bagging, boosting and random subspace methods ensembles with 8 sub-regressors in each one and then an averaging methodology is used for the final prediction. We performed a comparison with simple bagging, boosting and random subspace methods ensembles with 25 sub-regressors, as well as other well known combining methods, on standard benchmark datasets and the proposed technique had better correlation coefficient in most cases.

2014

If we look a few years back, we will find that ensemble classification model has outbreak many research and publication in the data mining community discussing how to combine models or model prediction with reduction in the error that results. When we ensemble the prediction of more than one classifier, more accurate and robust models are generated. We have convention that bagging, boosting with neural network etc. are the most popular method of combining different models and are realized in many data mining software but there are variation and alternative to bagging and boosting. This survey paper will give insight into various newly proposed ensemble classification models based on different methodologies.