MetaBags: Bagged Meta-Decision Trees for Regression

2011

An ensemble consists of a set of individually trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble is often more accurate than any of the single classifiers in the ensemble. Bagging (Breiman, 1996c) and Boosting (Freund and Shapire, 1996; Shapire, 1990) are two relatively new but popular methods for producing ensembles. In this paper we evaluate these methods on 23 data sets using both neural networks and decision trees as our classification algorithm. Our results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is sometimes much less accurate than Boosting. On the other hand, Boosting can create ensembles that are less accurate than a single classifier -- especially when using neural networks. Analysis indicates that the performance of the Boosting methods is dependent on the characteristic...

IEEE Access

The recent advances in computing power and machine learn-13 ing (ML) have given rise to several innovations and devel-14 opments in many areas of research and human lives. In the 15 last few years, advancements in machine learning, a subset 16 of artificial intelligence (AI), have transformed and inher-17 ently changed almost every area of our lives [1], [2]. Par-18 ticularly, ML has been applied in disease diagnosis [3], 19 fraud detection [4], text classification [5], and image recog-20 nition [6], among others. Unlike humans, ML algorithms 21 consider several factors when making decisions, and they 22 are not prone to fatigue or prejudice. Meanwhile, learn-23 ing can sometimes be challenging, especially when learning 24 from high-dimensional and imbalanced datasets [7],[8], [9]. 25 Research has shown that conventional ML algorithms tend 26 to underperform when trained with imbalanced datasets [10]. 27 Therefore, researchers have frequently resorted to new and 28 improved learning approaches, such as ensemble and deep 29 learning. 30 weighted majority voting is presented in [57]. Meanwhile, 230 there are several combination rules in the literature. In [58], 231 some combination rules were introduced, including the min-232 imum, maximum, product, median, and sum rules. 233 B. ENSEMBLE SELECTION 234 Ensemble selection is a technique used to build ensemble 235 classifiers from a set of base models. It is a vital topic in 236 ensemble learning because selecting a suitable subset of base 237 models could lead to better performance than when all the 238 models are used to construct the ensemble classifier [59]. 239 Since the base models are developed using various ML algo-240 rithms or different subsets of the training data, their perfor-241 mances would be different; while some would have good 242 performance, others might have poor performance. Instead 243 of combining the good and bad models, selecting only a 244 subset of models with good performance might be beneficial, 245 which would enhance the overall ensemble performance [60]. 246 An ensemble selection strategy is employed to select the 247 optimal subset of base classifiers, and they are usually guided 248 by a scoring function [61]. Caruana et al. [60] developed the 249 foremost forward model selection strategy to extract the best 250 performing subset of base models, and its basic procedure is 251 outlined as follows: 252 i. Begin with the empty ensemble. 253 ii. Select the base classifier from the library that maximizes 254 the ensemble's performance using a validation set. 255 iii. Repeat step II for a predefined number of iterations or 256 until all the models in the library have been examined. 257 iv. Return the subset of models that produces the best per-258 formance on the validation set. 259 The forward model selection strategy is fast and efficient 260 but occasionally overfits, leading to poor ensemble perfor-261 mance. Hence, several ensemble selection strategies have 262 been recently proposed; for example, Sun and Pfahringer [61] 263 developed the bagging ensemble selection, which combines 264 bagging and ensemble selection. Furthermore, the ensemble 265 selection strategies can be divided into static and dynamic 266 methods. The static strategy selects a single subset of base 267 models during model training and applies it to predict all 268 the unseen instances. The static selection methods can be 269 grouped into ordering-based techniques and optimization-270 based techniques. 271 As the name implies, ordering-based techniques attempt to 272 order the base models with respect to specific criteria, and 273 only the top models are chosen as the optimal subset. Some 274 criteria for ordering the base models include validation error 275 and kappa measure [59]. Guo et al. [62] recently developed 276 a method to order the base models via an evaluation metric 277 that takes the margin and diversity into consideration. Mean-278 while, optimization-based techniques formulate the selec-279 tion process as an optimization problem that can be solved 280 using mathematical programming or heuristic optimization. 281 Static methods limits the flexibility of the ensemble selection process. 843 the study employed the SMOTE Tomek link (SMOTETomek) 844 to create new datasets with even class distribution since 845 all four datasets are imbalanced. Secondly, the study uses 846 the isolation forest (iForest) algorithm [135] to detect and 847 remove outliers in the datasets. Meanwhile, the iForest is an 848 ensemble implementation that creates isolation trees using 849 the given dataset. The isolation trees are repeatedly devel-850 oped by splitting the training set until all the samples are 851 isolated or the specified tree height is obtained. The exper-852 imental results show that the proposed approach achieved 853 superior classification performance than other methods and 854 prior research works. The proposed ensemble obtained 855 96.7%, 85.8%, 75.8%, and 100% accuracy when predicting 856 type 2 diabetes, hypertension, prehypertension, and CKD, 857 respectively. 858 Kazemi and Mirroshandel [136] proposed a novel ensem-859 ble method for the early detection of kidney stones, a common 860 disease affecting people globally. The study employed several 861 ML algorithms, including decision trees, naïve Bayes, and 862 artificial neural networks (ANN), to learn the relationships 863 between some biological features related to kidney stone 864 disease. Furthermore, the study developed a novel method 865 to combine the different classifiers. The combination method 866 involved assigning weights to the different classifiers via a 867 genetic algorithm (GA) based computation. Meanwhile, the 868 data used in the research was obtained between 2012 and 869 2016 from 936 patients with kidney stone disease. The pro-870 posed ensemble achieved a classification accuracy of 97.1%.

Bagging, boosting and random subspace methods are well known re-sampling ensemble methods that generate and combine a diversity of learners using the same learning algorithm for the base-regressor. In this work, we built an ensemble of bagging, boosting and random subspace methods ensembles with 8 sub-regressors in each one and then an averaging methodology is used for the final prediction. We performed a comparison with simple bagging, boosting and random subspace methods ensembles with 25 sub-regressors, as well as other well known combining methods, on standard benchmark datasets and the proposed technique had better correlation coefficient in most cases.

Neural Networks, 2019

Regression is a very relevant problem in machine learning, with many different available approaches. The current work presents a comparison of a large collection composed by 77 popular regression models which belong to 19 families: linear and generalized linear models, generalized additive models, least squares, projection methods, LASSO and ridge regression, Bayesian models, Gaussian processes, quantile regression, nearest neighbors, regression trees and rules, random forests, bagging and boosting, neural networks, deep learning and support vector regression. These methods are evaluated using all the regression datasets of the UCI machine learning repository (83 datasets), with some exceptions due to technical reasons. The experimental work identifies several outstanding regression models: the M5 rule-based model with corrections based on nearest neighbors (cubist), the gradient

IRJET, 2021

Ensemble methods are successful in machine learning applications. Applications of ensemble methods have been used in a broad range of industries, such as air traffic controllers which uses ensembles to minimize airplanes arrival delay time. This paper describes mainly about three ensemble algorithms. They are stacking, bagging and boosting. These three ensemble methods which are also known as metaalgorithms approaches which mainly focuses on combining various machine learning algorithms into a single model through which predictions can be done to decrease the variance, which is also called bagging and boosting to decrease bias and improving predictive force using stacking.

International Journal of Machine Learning and Cybernetics, 2016

Selecting the a right classification algorithm is an important step for the success of data mining project. Run time can be used to assess efficienciency of a classification algorithm of interest. Experimenting with several algorithms can increase the cost of data mining project. Using the idea of meta-learning, we present an approach to estimate the run time of a particular classification algorithm on an arbitrary dataset. Our work provides a way to evaluate the choice of an algorithm without experimental execution. We demonstrate that the use of Multivariate Adaptive Regression Splines (MARS) significantly outperforms other regression models, even a `state-of-the-art' algorithm such as Support Vector Regression (SVR).

Journal of Computer Science, 2011

Problem statement: Regression via Classification (RvC) is a method in which a regression problem is converted into a classification problem. A discretization process is used to covert continuous target value to classes. The discretized data can be used with classifiers as a classification problem. Approach: In this study, we use a discretization method, Extreme Randomized Discretization (ERD), in which bin boundaries are created randomly to create ensembles. Results: We show that the proposed ensemble method is useful for RvC problems. We show theoretically that the proposed ensembles for RvC perform better than RvC with the equal-width discretization method. We also show the superiority of the proposed ensemble method experimentally. Experimental results suggest that the proposed ensembles perform competitively to the method developed specifically for regression problems. Conclusion: As the proposed method is independent of the choice of the classifier, various classifiers can be used with the proposed method to solve the regression method.

Scientific Reports 7, 2017

Ensemble generation is a natural and convenient way of achieving better generalization performance of learning algorithms by gathering their predictive capabilities. Here, we nurture the idea of ensemble-based learning by combining bagging and boosting for the purpose of binary classification. Since the former improves stability through variance reduction, while the latter ameliorates overfitting, the outcome of a multi-model that combines both strives toward a comprehensive net-balancing of the bias-variance trade-off. To further improve this, we alter the bagged-boosting scheme by introducing collaboration between the multi-model's constituent learners at various levels. This novel stability-guided classification scheme is delivered in two flavours: during or after the boosting process. Applied among a crowd of Gentle Boost ensembles, the ability of the two suggested algorithms to generalize is inspected by comparing them against Subbagging and Gentle Boost on various real-world datasets. In both cases, our models obtained a 40% generalization error decrease. But their true ability to capture details in data was revealed through their application for protein detection in texture analysis of gel electrophoresis images. They achieve improved performance of approximately 0.9773 AUROC when compared to the AUROC of 0.9574 obtained by an SVM based on recursive feature elimination. Machine learning has been transforming the world by improving our understanding of artificial intelligence 1-3 and by providing solutions for some outstanding problems such as multi-modal parcellation of human cerebral cortex 4 and materials discovery 5. A learning algorithm generalizes if, given access to some training set, it returns a hypothesis whose empirical error is close to its true error 6. There are three main approaches to institute generalization guarantees: (1) by providing bounds of various notions of functional space capacity-most notably, using the VC-dimension 7 ; (2) by establishing connections between the stability of a learning algorithm and its ability to generalize 8-10 , and (3) by considering the compression-scheme method 11. Here we describe an effective way to fuse boosting and bagging ensembles in which algorithmic stability directs a novel process of collaboration between the resulting ensemble's weak/strong components that outperforms best-case boosting/bagging for a broad range of applications and under a variety of scenarios. The algorithms were assessed on various realistic datasets, showing improved performance in all cases, on average of slightly below 40%, compared to the best-case boosting/bagging counterparts. Furthermore, in a medical setting for protein detection in texture analysis of gel electrophoresis images 12 , our approach exhibits surpassing performance of approximately 0.9773 area under the ROC curve (AUROC), compared to three machine-learning feature selection approaches: Multiple Kernel Learning, Recursive Feature Elimination with different classifiers and a Genetic Algorithm-based approach with Support Vector Machines (SVMs) as decision functions, having 0.9574 or less AUROCs. Moreover, when collaboration is effectuated with weak components, our algorithm runs up to more than five times faster than the underlying boosting algorithm. We anticipate our approach to be a starting point for more sophisticated models for generating stability-guided collaborative learning approaches, not necessarily limited to boosting. Ensemble techniques 13-15 show improved accuracy of predictive analytics and data mining applications. In a typical ensemble method, the base inducers and diversity generators are responsible for generating diverse clas-sifiers which represent the generalized relationship between the input and the target attributes. A strong classifier 1

2011

There are a number of learning methods that provide solutions to classification and regression problems, including Linear Regression, Decision Trees, KNN, and SVMs. These methods work well in many applications, but they are challenged for real world problems that are noisy, nonlinear or high dimensional. Furthermore, missing data (e.g., missing historical features of companies in stock data), is not managed well by current approaches. We present an implementation of a hybrid learning system that combines an ensemble of decision trees (Random Forest) with of Linear Regression. Linear Regression (LR) is fast but not accurate because it assumes linearity, while Random Forests are not as fast as LR but have been shown to be accurate for high dimensional and large data sets. By combining these approaches we address the weaknesses of each approach and exploit their strengths both in terms of real time performance and accuracy. In this thesis, we evaluate a hybrid Random Forest and Linear ...

IEEE Access, 2020

Ensemble methods based on k-NN models minimise the effect of outliers in a training dataset by searching groups of the k closest data points to estimate the response of an unseen observation. However, traditional k-NN based ensemble methods use the arithmetic mean of the training points' responses for estimation which has several weaknesses. Traditional k-NN based models are also adversely affected by the presence of non-informative features in the data. This paper suggests a novel ensemble procedure consisting of a class of base k-NN models each constructed on a bootstrap sample drawn from the training dataset with a random subset of features. In the k nearest neighbours determined by each k-NN model, stepwise regression is fitted to predict the test point. The final estimate of the target observation is then obtained by averaging the estimates from all the models in the ensemble. The proposed method is compared with some other state-of-the-art procedures on 16 benchmark datasets in terms of coefficient of determination (R 2), Pearson's product-moment correlation coefficient (r), mean square predicted error (MSPE), root mean squared error (RMSE) and mean absolute error (MAE) as performance metrics. Furthermore, boxplots of the results are also constructed. The suggested ensemble procedure has outperformed the other procedures on almost all the datasets. The efficacy of the method has also been verified by assessing the proposed method in comparison with the other methods by adding non-informative features to the datasets considered. The results reveal that the proposed method is more robust to the issue of non-informative features in the data as compared to the rest of the methods.