Reduction Techniques for Instance-Based Learning Algorithms

2008

For many real world problems we must perform classification under widely varying amounts of computational resources. For example, if asked to classify an instance taken from a bursty stream, we may have anywhere from several milliseconds to several minutes to return a class prediction. For such problems an anytime algorithm may be especially useful. In this work we show how we convert the ubiquitous nearest neighbor classifier into an anytime algorithm that can produce an instant classification, or if given the luxury of additional time, can continue computations to increase classification accuracy. We demonstrate the utility of our approach with a comprehensive set of experiments on data from diverse domains. We further show the utility of our work with two deployed applications, in classifying and counting fish, and in classifying insects.

Medical Physics, 2009

Ensemble classifiers have been shown efficient in multiple applications. In this article, the authors explore the effectiveness of ensemble classifiers in a case-based computer-aided diagnosis system for detection of masses in mammograms. They evaluate two general ways of constructing subclassifiers by resampling of the available development dataset: Random division and random selection. Furthermore, they discuss the problem of selecting the ensemble size and propose two adaptive incremental techniques that automatically select the size for the problem at hand. All the techniques are evaluated with respect to a previously proposed information-theoretic CAD system ͑IT-CAD͒. The experimental results show that the examined ensemble techniques provide a statistically significant improvement ͑AUC= 0.905Ϯ 0.024͒ in performance as compared to the original IT-CAD system ͑AUC= 0.865Ϯ 0.029͒. Some of the techniques allow for a notable reduction in the total number of examples stored in the case base ͑to 1.3% of the original size͒, which, in turn, results in lower storage requirements and a shorter response time of the system. Among the methods examined in this article, the two proposed adaptive techniques are by far the most effective for this purpose. Furthermore, the authors provide some discussion and guidance for choosing the ensemble parameters.

Lecture Notes in Computer Science, 2006

Parametric models such as linear regression can provide useful, interpretable descriptions of simple structure in data. However, sometimes such simple structure does not extend across an entire data set and may instead be confined more locally within subsets of the data. Nonparametric regression typically involves local averaging. In this study, local averaging estimator is coupled with a machine learning technique -boosting. In more detail, we propose a technique of local boosting of decision stumps. We performed a comparison with other well known methods and ensembles, on standard benchmark datasets and the performance of the proposed technique was greater in most cases.

Lecture Notes in Computer Science, 2013

Case-Base Maintenance (CBM) has two important goals. On the one hand, it aims to reduce the size of the case-base. On the other hand, it has to improve the accuracy of the CBR system. CBM can be represented as a multi-objective optimization problem to achieve both goals. Multi-Objective Evolutionary Algorithms (MOEAs) have been recognised as appropriate techniques for multi-objective optimisation because they perform a search for multiple solutions in parallel. In the present paper we introduce a fitness function based on the Complexity Profiling model to perform CBM with MOEA, and we compare its results against other known CBM approaches. From the experimental results, CBM with MOEA shows regularly good results in many case-bases, despite the amount of redundant and noisy cases, and with a significant potential for improvement.

Lecture Notes in Computer Science, 2011

A solid building process and a good evaluation of the knowledge base are essential in the clinical application of Case-Based Reasoning Systems. Unlike other approaches, each piece of the knowledge base (cases of the case memory) is knowledge-complete and independent from the rest. Therefore, the main issue to build a case memory is to select which cases must be included or removed. Literature provides a wealth of methods based on instance selection from a database. However, it can be also understood as a multiobjective problem, maximising the accuracy of the system and minimising the number of cases in the case memory. Most of the efforts done in this evaluation of case selection methods focus on the number of registers selected, providing an evaluation of the system based on its accuracy. On the one hand, some case selection methods follow a non deterministic approach. Therefore, a rough evaluation could entail to inaccurate conclusions. On the other hand, specificity and sensitivity are critical values to evaluate tests in the medical field. However, these parameters are hardly ever included in the case selection evaluation. In order to partially solve this problem, we propose an evaluation methodology to obtain the best case selection method for a given memory case. We also propose a case selection method based on multiobjective constrained optimisation for which Evolutionary Algorithms are used. Finally, we illustrate the use of this methodology by evaluating classic and the case selection method proposed, in a particular problem of Burn Intensive Care Units.

Journal of Medical Systems, 2014

Obesity is becoming one of the serious problems for the health of worldwide population. Social interactions on mobile phones and computers via internet through social enetworks are one of the major causes of lack of physical activities. For the health specialist, it is important to track the record of physical activities of the obese or overweight patients to supervise weight loss control. In this study, acceleration sensor present in the smartphone is used to monitor the physical activity of the user. Physical activities including Walking, Jogging, Sitting, Standing, Walking upstairs and Walking downstairs are classified. Time domain features are extracted from the acceleration data recorded by smartphone during different physical activities. Time and space complexity of the whole framework is done by optimal feature subset selection and pruning of instances. Classification results of six physical activities are reported in this paper. Using simple time domain features, 99 % classification accuracy is achieved. Furthermore, attributes subset selection is used to remove the redundant features and to minimize the time complexity of the algorithm. A subset of 30 features produced more than 98 % classification accuracy for the six physical activities.

Lecture Notes in Computer Science, 2010

Pattern Analysis and Applications, 2014

Although the k-NN classifier is a popular classification method, it suffers from the high computational cost and storage requirements it involves. This paper proposes two effective cluster-based data reduction algorithms for efficient k-NN classification. Both have low preprocessing cost and can achieve high data reduction rates while maintaining k-NN classification accuracy at high levels. The first proposed algorithm is called Reduction through Homogeneous Clusters (RHC) and is based on a fast pre-processing clustering procedure that creates homogeneous clusters. The centroids of these clusters constitute the reduced training set. The second proposed algorithm is a dynamic version of RHC that retains all its properties and, in addition, it can manage datasets that cannot fit in main memory and is appropriate for dynamic environments where new training data are gradually available. Experimental results, based on fourteen datasets, illustrate that both algorithms are faster and achieve higher reduction rates than four known methods, while maintaining high classification accuracy.

ACM SIGKDD Explorations Newsletter, 2004

There are several aspects that might influence the performance achieved by existing learning systems. It has been reported that one of these aspects is related to class imbalance in which examples in training data belonging to one class heavily outnumber the examples in the other class. In this situation, which is found in real world data describing an infrequent but important event, the learning system may have difficulties to learn the concept related to the minority class. In this work we perform a broad experimental evaluation involving ten methods, three of them proposed by the authors, to deal with the class imbalance problem in thirteen UCI data sets. Our experiments provide evidence that class imbalance does not systematically hinder the performance of learning systems. In fact, the problem seems to be related to learning with too few minority class examples in the presence of other complicating factors, such as class overlapping. Two of our proposed methods deal with these conditions directly, allying a known over-sampling method with data cleaning methods in order to produce better-defined class clusters. Our comparative experiments show that, in general, over-sampling methods provide more accurate results than under-sampling methods considering the area under the ROC curve (AUC). This result seems to contradict results previously published in the literature. Two of our proposed methods, Smote + Tomek and Smote + ENN, presented very good results for data sets with a small number of positive examples. Moreover, Random over-sampling, a very simple over-sampling method, is very competitive to more complex over-sampling methods. Since the over-sampling methods provided very good performance results, we also measured the syntactic complexity of the decision trees induced from over-sampled data. Our results show that these trees are usually more complex then the ones induced from original data. Random over-sampling usually produced the smallest increase in the mean number of induced rules and Smote + ENN the smallest increase in the mean number of conditions per rule, when compared among the investigated over-sampling methods.

Journal of Intelligent Systems, 2000

In this paper we present a new noise filtering method, called soft decision tree noise filter (SDTNF), to identify and remove mislabeled data items in a data set. In this method, a sequence of decision trees are built from a data set, in which each data item is assigned a soft class label (in the form of a class probability vector). A modified decision tree algorithm is applied to adjust the soft class labeling during the tree building process. After each decision tree is built, the soft class label of each item in the data set is adjusted using the decision tree's predictions as the learning targets. In the next iteration, a new decision tree is built from a data set with the updated soft class labels. This tree building process repeats iteratively until the data set labeling converges. This procedure provides a mechanism to gradually modify and correct mislabeled items. It is applied in SDTNF as a filtering method by identifying data items whose classes have been relabeled by decision trees as mislabeled data. The performance of SDTNF is evaluated using 16 data sets drawn from the UCI data repository. The results show that it is capable of identifying a substantial amount of noise for most of the tested data sets and significantly improving performance of nearest neighbor classifiers at a wide range of noise levels. We also compare SDTNF to the consensus and majority voting methods proposed by Friedl [1996, 1999] for noise filtering. The results show SDTNF has a more efficient and balanced filtering capability than these two methods in terms of filtering mislabeled data and keeping non-mislabeled data. The results also show that the filtering capability of SDTNF can significantly improve the performance of nearest neighbor classifiers, especially at high noise levels. At a noise level of 40%, the improvement on the accuracy of nearest neighbor classifiers is 13.1% by the consensus voting method and 18.7% by the majority voting method, while SDTNF is able to achieve an improvement by 31.3%.

Lecture Notes in Computer Science, 2010

Attempts to extract logical rules from data often lead to large sets of classification rules that need to be pruned. Training two classifiers, the C4.5 decision tree and the Non-Nested Generalized Exemplars (NNGE) covering algorithm, on datasets that have been reduced earlier with the EkP instance compressor leads to statistically significantly lower number of derived rules with nonsignificant degradation of results. Similar results have been observed with other popular instance filters used for data pruning. Numerical experiments presented here illustrate that it is possible to extract more interesting and simpler sets of rules from filtered datasets. This enables a better understanding of knowledge structures when data is explored using algorithms that tend to induce a large number of classification rules.

Lecture Notes in Computer Science, 2010

Performance and accuracy of a neural network are strongly related to its design. Designing a neural network involves topology (number of neurons, number of layers, number of synapses between layers, etc.), training synapse weights, and parameter selection. Radial basis function neural networks (RBFNNs) could additionally require some other parameters, for example, the means and standard deviations if the activation function of neurons in the hidden layer is a Gaussian function. Commonly, Genetic Algorithms and Evolution Strategies have been used for automatically designing RBFNNs In this work, the use of prototype selection methods for designing a RBFNN is proposed and studied. Experimental results show the viability of designing RBFNNs using prototype selection.

Lecture Notes in Computer Science, 2008

In Pattern recognition, the supervised classifiers use a training set T for classifying new prototypes. In practice, not all information in T is useful for classification therefore it is necessary to discard irrelevant prototypes from T. This process is known as prototype selection, which is an important task for classifiers since through this process the time in the training and/or classification stages could be reduced. Several prototype selection methods have been proposed following the Nearest Neighbor (NN) rule; in this work, we propose a new prototype selection method based on the prototype relevance and border prototypes, which is faster (over large datasets) than the other tested prototype selection methods. We report experimental results showing the effectiveness of our method and compare accuracy and runtimes against other prototype selection methods.

International Journal of Applied Mathematics and Computer Science, 2011

Application of agent-based simulated annealing and tabu search procedures to solving the data reduction problem The problem considered concerns data reduction for machine learning. Data reduction aims at deciding which features and instances from the training set should be retained for further use during the learning process. Data reduction results in increased capabilities and generalization properties of the learning model and a shorter time of the learning process. It can also help in scaling up to large data sources. The paper proposes an agent-based data reduction approach with the learning process executed by a team of agents (A-Team). Several A-Team architectures with agents executing the simulated annealing and tabu search procedures are proposed and investigated. The paper includes a detailed description of the proposed approach and discusses the results of a validating experiment.

Artificial Intelligence Review, 2013

The k-NN classifier is a widely used classification algorithm. However, exhaustively searching the whole dataset for the nearest neighbors is prohibitive for large datasets because of the high computational cost involved. The paper proposes an efficient model for fast and accurate nearest neighbor classification. The model consists of a non-parametric cluster-based preprocessing algorithm that constructs a two-level speed-up data structure and algorithms that access this structure to perform the classification. Furthermore, the paper demonstrates how the proposed model can improve the performance on reduced sets built by various data reduction techniques. The proposed classification model was evaluated using eight real-life datasets and compared to known speed-up methods. The experimental results show that it is a fast and accurate classifier, and, in addition, it involves low pre-processing computational cost.

Entropy

Building an accurate prediction model is challenging and requires appropriate model selection. This process is very time consuming but can be accelerated with meta-learning-automatic model recommendation by estimating the performances of given prediction models without training them. Meta-learning utilizes metadata extracted from the dataset to effectively estimate the accuracy of the model in question. To achieve that goal, metadata descriptors must be gathered efficiently and must be informative to allow the precise estimation of prediction accuracy. In this paper, a new type of metadata descriptors is analyzed. These descriptors are based on the compression level obtained from the instance selection methods at the data-preprocessing stage. To verify their suitability, two types of experiments on real-world datasets have been conducted. In the first one, 11 instance selection methods were examined in order to validate the compression-accuracy relation for three classifiers: k-nearest neighbors (kNN), support vector machine (SVM), and random forest. From this analysis, two methods are recommended (instance-based learning type 2 (IB2), and edited nearest neighbor (ENN)) which are then compared with the state-of-the-art metaset descriptors. The obtained results confirm that the two suggested compression-based meta-features help to predict accuracy of the base model much more accurately than the state-of-the-art solution.

Applied Sciences

Instance selection and construction methods were originally designed to improve the performance of the k-nearest neighbors classifier by increasing its speed and improving the classification accuracy. These goals were achieved by eliminating redundant and noisy samples, thus reducing the size of the training set. In this paper, the performance of instance selection methods is investigated in terms of classification accuracy and reduction of training set size. The classification accuracy of the following classifiers is evaluated: decision trees, random forest, Naive Bayes, linear model, support vector machine and k-nearest neighbors. The obtained results indicate that for the most of the classifiers compressing the training set affects prediction performance and only a small group of instance selection methods can be recommended as a general purpose preprocessing step. These are learning vector quantization based algorithms, along with the Drop2 and Drop3. Other methods are less effi...

Lecture Notes in Computer Science, 2005

The KNN rule has been widely used in many pattern recognition problems, but it is sensible to noisy data within the training set, therefore, several sample edition methods have been developed in order to solve this problem. A. Franco, D. Maltoni and L. Nanni proposed the Reward-Punishment Editing method in 2004 for editing numerical databases, but it has the problem that the selected prototypes could belong neither to the sample nor to the universe. In this work, we propose a modification based on selecting the prototypes from the training set. To do this selection, we propose the use of the Fuzzy C-means algorithm for mixed data and the KNN rule with similarity functions. Tests with different databases were made and the results were compared against the original Reward-Punishment Editing and the whole set (without any edition).

Computational Intelligence [Working Title]

Classification is a data mining task. It aims to extract knowledge from large datasets. There are two kinds of classification. The first one is known as complete classification, and it is applied to balanced datasets. However, when it is applied to imbalanced ones, it is called partial classification or a problem of classification in imbalanced datasets, which is a fundamental problem in machine learning, and it has received much attention. Considering the importance of this issue, a large amount of techniques have been proposed trying to address this problem. These proposals can be divided into three levels: the algorithm level, the data level, and the hybrid level. In this chapter, we will present the classification problem in imbalanced datasets, its domains of application, its appropriate measures of performances, and its approaches and techniques.

Progress in Artificial Intelligence, 2012

Class imbalance problems have lately become an important area of study in machine learning and are often solved using intelligent resampling methods to balance the class distribution. The aim of this work is to show that balancing the class distribution is not always the best solution when intelligent resampling methods are used, i.e. there is often a class distribution other than 50 % that improves the results. This paper presents a methodology to find a (near-)optimal class distribution. Experiments have been performed using 29 databases, 8 different resampling methods and two learning algorithms, and have been evaluated using AUC performance metric and statistical tests. The results go beyond the original objective, because they show that although intelligent resampling methods are mostly used to solve class imbalance problems they can also be used to improve results, using the class distribution found in data sets that are not imbalanced. As a consequence, we can state that when solving any machine learning problem it is always worth looking for the optimal class distribution.

Open Geosciences

This study examines rainfall forecasting for the Perfume (Huong) River basin using the machine learning method. To be precise, statistical measurement indicators are deployed to evaluate the reliability of the actual accumulated data. At the same time, this study applied and compared two popular models of multi-layer perceptron and the k-nearest neighbors (k-NN) with different configurations. The calculated rainfall data are obtained from the Hue, Aluoi, and Namdong hydrological stations, where the rainfall demonstrated a giant impact on the downstream from 1980 to 2018. This study result shows that both models, once fine-tuned properly, enjoyed the performance with standard metrics of R_squared, mean absolute error, Nash–Sutcliffe efficiency, and root-mean-square error. In particular, once Adam stochastic is deployed, the implementation of the MLP model is significantly improving. The promising forecast results encourage us to consider applying these models with future data to help...

2021

Algorithms and Architectures for Parallel Processing

Linear discriminant analysis is considered as current techniques in feature extraction so, LDA, by discriminant information which obtains in mapping space, does the classification act. When the classes' distribution is not normal, LDA, to perform classification, will face problem and will resulted the poor performance of criteria in performing the classification act. One of the proposed ways is the use of other measures, such as Chernoff's distance so, by using Chernoff's measure LDA has been spreading to its heterogeneous states and LDA in this state, in addition to use information among the medians, uses the information of the classes' Covariance matrices. By defining scattering matrix , based on Boundary and non-Boundary samples and using these matrices in Chernoff's criteria, the decrease of the classes' overlapping in the mapping space in as result, the rate of classification correctness increases. Using Boundary and non-Boundary samples in scattering matrices causes improvement over the result. In this article, we use a new discovering multi-stage Algorithm to choose Boundary and non-Boundary samples so, the results of the conducted experiments shows promising performance of the proposing method.

2017

The presence of useless information and the huge amount of data generated by telecommunication services can affect the efficiency of traditional Intrusion Detection Systems (IDSs). This fact encourage the development of data preprocessing strategies for improving the efficiency of IDSs. On the other hand, improving such efficiency relying on the data reduction strategies, without affecting the quality of the reduced dataset (i.e. keeping the accuracy during the classification process), represents a challenge. Also, the runtime of commonly used strategies is usually high. In this paper, a novel hybrid data reduction strategy is presented. The proposed strategy reduces the number of features and instances in the training collection without greatly affecting the quality of the reduced dataset. In addition, it improves the efficiency of the classification process. Finally, our proposal is favorably compared with other hybrid data reduction strategies.

Companion Proceedings of The 2019 World Wide Web Conference on - WWW '19, 2019

For a product of interest, we propose a search method to surface a set of reference products. The reference products can be used as candidates to support downstream modeling tasks and business applications. The search method consists of product representation learning and fingerprint-type vector searching. The product catalog information is transformed into a high-quality embedding of low dimensions via a novel attention auto-encoder neural network, and the embedding is further coupled with a binary encoding vector for fast retrieval. We conduct extensive experiments to evaluate the proposed method, and compare it with peer services to demonstrate its advantage in terms of search return rate and precision. CCS CONCEPTS • Computing methodologies → Natural language processing; Search methodologies; Machine learning.

Soft Computing, 2013

The k-nearest neighbors classifier is a widely used classification method that has proven to be very effective in supervised learning tasks. In this paper, a fuzzy rough set method for prototype selection, focused on optimizing the behavior of this classifier, is presented. The hybridization with an evolutionary feature selection method is considered to further improve its performance, obtaining a competent data reduction algorithm for the 1-nearest neighbors classifier. This hybridization is performed in the training phase, by using the solution of each preprocessing technique as the starting condition of the other one, within a cycle. The results of the experimental study, which have been contrasted through nonparametric statistical tests, show that the new hybrid approach obtains very promising results with respect to classification accuracy and reduction of the size of the training set.

AI Communications, 2016

One of the important issues related with all types of data analysis, either statistical data analysis, machine learning, data mining, data science or whatever form of data-driven modeling, is data quality. The more complex the reality to be analyzed is, the higher the risk of getting low quality data. Unfortunately real data often contain noise, uncertainty, errors, redundancies or even irrelevant information. Useless models will be obtained when built over incorrect or incomplete data. As a consequence, the quality of decisions made over these models, also depends on data quality. This is why pre-processing is one of the most critical steps of data analysis in any of its forms. However, pre-processing has not been properly systematized yet, and little research is focused on this. In this paper a survey on most popular pre-processing steps required in environmental data analysis is presented, together with a proposal to systematize it. Rather than providing technical details on specific pre-processing techniques, the paper focus on providing general ideas to a non-expert user, who, after reading them, can decide which one is the more suitable technique required to solve his/her problem.

Complexity

In the paper, several data reduction techniques for machine learning from big datasets are discussed and evaluated. The discussed approach focuses on combining several techniques including stacking, rotation, and data reduction aimed at improving the performance of the machine classification. Stacking is seen as the technique allowing to take advantage of the multiple classification models. The rotation-based techniques are used to increase the heterogeneity of the stacking ensembles. Data reduction makes it possible to classify instances belonging to big datasets. We propose to use an agent-based population learning algorithm for data reduction in the feature and instance dimensions. For diversification of the classifier ensembles within the rotation also, alternatively, principal component analysis and independent component analysis are used. The research question addressed in the paper is formulated as follows: does the performance of a classifier using the reduced dataset be imp...

Intelligent Decision Technologies 2017

Integrated machine learning is understood as integration of the data reduction with the learning process. Such integration allows to introduce adaptation mechanisms within the learning process by modification of the data with a view to finding its better representation from the point of view of the learning performance criterion. Data modification can be carried out through data reduction in both dimensions, i.e. the feature and the instance ones producing the set of prototypes. Currently, data reduction has become a crucial technique for big data analysis and improvement of the machine learning process results. In this paper the stacking technique has been proposed for improving the process of the integrated machine classification and to assure diversification among prototypes. To validate the proposed approach we have carried-out computational experiment. The paper includes the description of the approach and the discussion of the validating experiment results.

Pattern Analysis and Applications, 2008

Artificial immune system (AIS)-based pattern classification approach is relatively new in the field of pattern recognition. The study explores the potentiality of this paradigm in the context of prototype selection task that is primarily effective in improving the classification performance of nearest-neighbor (NN) classifier and also partially in reducing its storage and computing time requirement. The clonal selection model of immunology has been incorporated to condense the original prototype set, and performance is verified by employing the proposed technique in a practical optical character recognition (OCR) system as well as for training and testing of a set of benchmark databases available in the public domain. The effect of control parameters is analyzed and the efficiency of the method is compared with another existing techniques often used for prototype selection. In the case of the OCR system, empirical study shows that the proposed approach exhibits very good generalization ability in generating a smaller prototype library from a larger one and at the same time giving a substantial improvement in the classification accuracy of the underlying NN classifier. The improvement in performance has been statistically verified. Consideration of both OCR data and public domain datasets demonstrate that the proposed method gives results better than or at least comparable to that of some existing techniques.

Artificial Intelligence and Soft Computing, 2020

Due to the increasing size of the datasets, prototype selection techniques have been applied for reducing the computational resources involved in data mining and machine learning tasks. In this paper, we propose a density-based approach for selecting prototypes. Firstly, it finds the density peaks in each dimension of the dataset. After that, it builds clusters of objects around these peaks. Finally, it extracts a prototype that represents each cluster and selects the most representative prototypes for including in the final reduced dataset. The proposed algorithm can deal with some crucial weak points of approaches that were previously proposed regarding the setting of parameters and the capability of dealing with high-dimensional datasets. Our method was evaluated on 14 well-known datasets used in a classification task. The performance of the proposed algorithm was compared to the performances of 8 prototype selection algorithms in terms of accuracy and reduction rate. The experimental results show that, in general, the proposed algorithm provides a good trade-off between reduction rate and the accuracy with reasonable time complexity.

Mathematical Problems in Engineering, 2019

The imbalance data refers to at least one of its classes which is usually outnumbered by the other classes. The imbalanced data sets exist widely in the real world, and the classification for them has become one of the hottest issues in the field of data mining. At present, the classification solutions for imbalanced data sets are mainly based on the algorithm-level and the data-level. On the data-level, both oversampling strategies and undersampling strategies are used to realize the data balance via data reconstruction. SMOTE and Random-SMOTE are two classic oversampling algorithms, but they still possess the drawbacks such as blind interpolation and fuzzy class boundaries. In this paper, an improved oversampling algorithm based on the samples’ selection strategy for the imbalanced data classification is proposed. On the basis of the Random-SMOTE algorithm, the support vectors (SV) are extracted and are treated as the parent samples to synthesize the new examples for the minority ...

European Journal of Wildlife Research, 2017

Many wildlife populations are declining due to anthropologic and environmental pressures. Indicators can facilitate detection of and reaction to population declines. We tested spatial population pattern variables as indicators for the future fate of capercaillie populations in Slovenia. This population was surveyed from 1979 to 1983, 1998 to 2000, and 2008 to 2011. We tested the following indicators of spatial population patterns: the proportion of occupied leks in the neighbourhood of each lek, the distance to the nth nearest occupied lek, the distance to the centre of the cluster and the distance to the closest neighbouring cluster. The AUC and conditional and marginal R2 were used to evaluate the strength of the indicators. There was a steep decline in the number of leks in the Dinaric Mountains during the period 1980-2010, while the number of leks in the Eastern Slovenian Alps and Julian Alps decreased until 2000 and then stabilized. There was an increase in the distance to the nearest occupied leks over the years, but there were differences in this indicator between the clusters. A combination of the distance to the fourth nearest occupied lek and the occupancy of the first nine leks best predicted the future of a lek, showing that the indicators used in the study can be used to predict the future fate of a lek. Our results can help guide future conservation and management decisions to conserve capercaille populations.

Cheminformatics and its Applications [Working Title], 2019

The drug discovery process from hit-to-lead has been a challenging task that requires simultaneously optimizing numerous factors from maximizing compound activity, efficacy to minimizing toxicity and adverse reactions. Recently, the advance of artificial intelligence technique enables drugs to be efficiently purposed in silico prior to chemical synthesis and experimental evaluation. In this chapter, we present fundamental concepts of artificial intelligence and their application in drug design and discovery. The emphasis will be on machine learning and deep learning, which demonstrated extensive utility in many branches of computer-aided drug discovery including de novo drug design, QSAR (Quantitative Structure-Activity Relationship) analysis, drug repurposing and chemical space visualization. We will demonstrate how artificial intelligence techniques can be leveraged for developing chemoinformatics pipelines and presented with real-world case studies and practical applications in drug design and discovery. Finally, we will discuss limitations and future direction to guide this rapidly evolving field.

Lecture Notes in Computer Science, 2009

Instance selection is a pre-processing technique for machine learning and data mining. The main problem is that previous approaches still suffer from the difficulty to produce effective samples for training classifiers. In recent research, a new sampling technique, called Progressive Border Sampling (PBS), has been proposed to produce a small sample from the original labelled training set by identifying and augmenting border points. However, border sampling on multi-class domains is not a trivial issue. Training sets contain much redundancy and noise in practical applications. In this work, we discuss several issues related to PBS and show that PBS can be used to produce effective samples by removing redundancies and noise from training sets for training classifiers. We compare this new technique with previous instance selection techniques for learning classifiers, especially, for learning Naïve Bayes-like classifiers, on multi-class domains except for one binary case which was for a practical application.

Computers, materials & continua, 2022

Association rules' learning is a machine learning method used in finding underlying associations in large datasets. Whether intentionally or unintentionally present, noise in training instances causes overfitting while building the classifier and negatively impacts classification accuracy. This paper uses instance reduction techniques for the datasets before mining the association rules and building the classifier. Instance reduction techniques were originally developed to reduce memory requirements in instance-based learning. This paper utilizes them to remove noise from the dataset before training the association rules classifier. Extensive experiments were conducted to assess the accuracy of association rules with different instance reduction techniques, namely: Decremental Reduction Optimization Procedure (DROP) 3, DROP5, ALL K-Nearest Neighbors (ALLKNN), Edited Nearest Neighbor (ENN), and Repeated Edited Nearest Neighbor (RENN) in different noise ratios. Experiments show that instance reduction techniques substantially improved the average classification accuracy on three different noise levels: 0%, 5%, and 10%. The RENN algorithm achieved the highest levels of accuracy with a significant improvement on seven out of eight used datasets from the University of California Irvine (UCI) machine learning repository. The improvements were more apparent in the 5% and the 10% noise cases. When RENN was applied, the average classification accuracy for the eight datasets in the zero-noise test enhanced from 70.47% to 76.65% compared to the original test. The average accuracy was improved from 66.08% to 77.47% for the 5%-noise case and from 59.89% to 77.59% in the 10%-noise case. Higher confidence was also reported in building the association rules when RENN was used. The above results indicate that RENN is a good solution in removing noise and avoiding overfitting during the construction of the association rules classifier, especially in noisy domains.

Applied Sciences, 2019

The broad context of this literature review is the connected manufacturing enterprise, characterized by a data environment such that the size, structure and variety of information strain the capability of traditional software and database tools to effectively capture, store, manage and analyze it. This paper surveys and discusses representative examples of existing research into approaches for feature set reduction in the big data environment, focusing on three contexts: general industrial applications; specific industrial applications such as fault detection or fault prediction; and data reduction. The conclusion from this review is that there is room for research into frameworks or approaches to feature filtration and prioritization, specifically with respect to providing quantitative or qualitative information about the individual features in the dataset that can be used to rank features against each other. A byproduct of this gap is a tendency for analysts not to holistically ge...

The Knowledge Engineering Review, 2019

A large variety of issues influence the success of data mining on a given problem. Two primary and important issues are the representation and the quality of the dataset. Specifically, if much redundant and unrelated or noisy and unreliable information is presented, then knowledge discovery becomes a very difficult problem. It is well-known that data preparation steps require significant processing time in machine learning tasks. It would be very helpful and quite useful if there were various preprocessing algorithms with the same reliable and effective performance across all datasets, but this is impossible. To this end, we present the most well-known and widely used up-to-date algorithms for each step of data preprocessing in the framework of predictive data mining.

2005

Two extensions of the original Wilson's editing method are introduced in this paper. These new algorithms are based on estimating probabilities from the k-nearest neighbor patterns of an instance, in order to obtain more compact edited sets while maintaining the classification rate. Several experiments with synthetic and real data sets are carried out to illustrate the behavior of the algorithms proposed here and compare their performance with that of other traditional techniques.

Lecture Notes in Computer Science, 2002

This paper presents some ideas about automatic procedures to implement a system with the capability of detecting patterns arising from classes not represented in the training sample. The procedure aims at incorporating automatically to the training sample the necessary information about the new class for correctly recognizing patterns from this class in future classification tasks. The Nearest Neighbor rule is employed as the central classifier and several techniques are added to cope with the peril of incorporating noisy data to the training sample. Experimental results with real data confirm the benefits of the proposed procedure.

Ecological Informatics, 2020

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

International Journal on Artificial Intelligence Tools, 2005

Algorithms reducing the storage requirement of the nearest neighbor classifier (NNC) can be divided into three main categories: Fast searching algorithms, Instance-based learning algorithms and Prototype based algorithms. We propose an algorithm, LVQPRU, for pruning NNC prototype vectors and a compact classifier with good performance is obtained. The basic condensing algorithm is applied to the initial prototypes to speed up the learning process. The learning vector quantization (LVQ) algorithm is utilized to fine tune the remaining prototypes during each pruning iteration. We evaluate LVQPRU on several data sets along with 12 other algorithms using ten-fold cross-validation. Simulation results show that the proposed algorithm has high generalization accuracy and good storage reduction ratios.

Lecture Notes in Computer Science

In this paper we present a non parametric density-based data reduction technique designed to be used in robust parameter estimation problems. Existing approaches are focused on reducing the amount of data preserving the density function. In our case the reduction is oriented to automatically remove the samples that are considered non interesting while taking into account those that are meaningful, those that have a high density associated. We use this filtering process to simplify the data sets in order to improve the performance of robust parameter estimators. We show its results when used along an existing estimator on synthetic and real LADAR data.

Progress in Artificial Intelligence, 2019

A natural way of handling imbalanced data is to attempt to equalise the class frequencies and train the classifier of choice on balanced data. For two-class imbalanced problems, the classification success is typically measured by the geometric mean (GM) of the true positive and true negative rates. Here we prove that GM can be improved upon by instance selection, and give the theoretical conditions for such an improvement. We demonstrate that GM is non-monotonic with respect to the number of retained instances, which discourages systematic instance selection. We also show that balancing the distribution frequencies is inferior to a direct maximisation of GM. To verify our theoretical findings, we carried out an experimental study of 12 instance selection methods for imbalanced data, using 66 standard benchmark data sets. The results reveal possible room for new instance selection methods for imbalanced data.

Lecture Notes in Computer Science, 2011

A k nearest neighbor (kNN) classifier classifies a query instance to the most frequent class of its k nearest neighbors in the training instance space. For imbalanced class distribution, a query instance is often overwhelmed by majority class instances in its neighborhood and likely to be classified to the majority class. We propose to identify exemplar minority class training instances and generalize them to Gaussian balls as concepts for the minority class. Our k Exemplar-based Nearest Neighbor (kENN) classifier is therefore more sensitive to the minority class. Extensive experiments show that kENN significantly improves the performance of kNN and also outperforms popular re-sampling and costsensitive learning strategies for imbalanced classification.

Autonomous Search, 2011

Lecture Notes in Business Information Processing, 2022

Predictive process monitoring is a subfield of process mining that aims to estimate case or event features for running process instances. Such predictions are of significant interest to the process stakeholders. However, state-of-the-art methods for predictive monitoring require the training of complex machine learning models, which is often inefficient. This paper proposes an instance selection procedure that allows sampling training process instances for prediction models. We show that our sampling method allows for a significant increase of training speed for next activity prediction methods while maintaining reliable levels of prediction accuracy.

Neural Computing and Applications, 2016

Adaptive human-computer interfaces (HCIs) are fundamental to designing adaptive websites and adaptive decision support systems. Integrating these intelligent systems with modern eye trackers provides more effective ways to exploit eye fixation data and offers improved services to users. We develop an exemplar-based classifier using the tabu search algorithm to predict which decision strategy may underlie an empirical search behavior. Our algorithm reduces the size of decision concept representations to find the best exemplars for each concept. Experimental results show that our classifier is highly accurate in classifying the sequence of empirical eye fixations, demonstrating the promise of integrating adaptive HCIs with modern eye trackers.

Engineering Science and Technology, an International Journal

In online advertising, the user-clicks dataset based fraudulent publishers' classification models exhibit poor performance due to high skewness in class distribution of the publishers. The nearest-neighbor based classification techniques are popularly used to reduce the impact of class skewness on performance. The Nearest-Neighbor techniques use Prototype Selection (PS) methods to select promising samples before classifying them for reducing the size of training data. Although Nearest-Neighbor techniques are simple to use and reduce the negative impact of the loss of potential information, they suffer from higher storage requirements and slower classification speed when applied on datasets with skewed class distributions. In this paper, we propose a Quad Division Prototype Selection-based k-Nearest Neighbor classifier (QDPSKNN) by introducing quad division method for handling uneven class distribution. The quad-division divides the data into four quartiles (groups) and performs controlled under-sampling for balancing class distribution. It reduces the size of the training dataset by selecting only the relevant prototypes in the form of nearest-neighbors. The performance of QDPSKNN is evaluated on Fraud Detection in Mobile Advertising (FDMA) user-click dataset and fifteen other benchmark imbalanced datasets to test its generalizing behaviour. The performance is also compared with one baseline model (k-NN) and four other prototype selection methods such as NearMiss-1, NearMiss-2, NearMiss-3, and Condensed Nearest-Neighbor. The results show improved classification performance with QDPSKNN in terms of precision, recall, f-measure, g-mean, reduction rate and execution time, compared to existing prototype selection methods in the classification of fraudulent publishers as well as on other benchmark imbalanced datasets. Wilcoxon signed ranked test is conducted to demonstrate significant differences amid QDPSKNN and state-of-the-art methods.

Lecture Notes in Computer Science, 2013

k-Nearest Neighbors is surely one of the most important and widely adopted non-parametric classification methods in pattern recognition. It has evolved in several aspects in the last 50 years, and one of the most known variants consists in the usage of prototypes: a prototype distills a group of similar training points, diminishing drastically the number of comparisons needed for the classification; actually, prototypes are employed in the case the cardinality of the training data is high. In this paper, by using the dominant set clustering framework, we propose four novel strategies for the prototype generation, allowing to produce representative prototypes that mirror the underlying class structure in an expressive and effective way. Our strategy boosts the k-NN classification performance; considering heterogeneous metrics and analyzing 15 diverse datasets, we are among the best 6 prototype-based k-NN approaches, with a computational cost which is strongly inferior to all the competitors. In addition, we show that our proposal beats linear SVM in the case of a pedestrian detection scenario.

International Journal of Pattern Recognition and Artificial Intelligence, 2005

The excessive computational resources required by the Nearest Neighbor rule are a major concern for a number of specialists and practitioners in the Pattern Recognition community. Many proposals for decreasing this computational burden, through reduction of the training sample size, have been published. This paper introduces an algorithm to reduce the training sample size while preserving the original decision boundaries as much as possible. Consequently, the algorithm tends to obtain classification accuracy close to that of the whole training sample. Several experimental results demonstrate the effectiveness of this method when compared to other reduction algorithms based on similar ideas.

Journal of Intelligent Systems, 2021

Decision trees learning is one of the most practical classification methods in machine learning, which is used for approximating discrete-valued target functions. However, they may overfit the training data, which limits their ability to generalize to unseen instances. In this study, we investigated the use of instance reduction techniques to smooth the decision boundaries before training the decision trees. Noise filters such as ENN, RENN, and ALLKNN remove noisy instances while DROP3 and DROP5 may remove genuine instances. Extensive empirical experiments were conducted on 13 benchmark datasets from UCI machine learning repository with and without intentionally introduced noise. Empirical results show that eliminating border instances improves the classification accuracy of decision trees and reduces the tree size, which reduces the training and classification times. In datasets without intentionally added noise, applying noise filters without the use of the built-in Reduced Error ...

Journal of Intelligent Information Systems

Predictive process monitoring is a subfield of process mining that aims to estimate case or event features for running process instances. Such predictions are of significant interest to the process stakeholders. However, most of the state-of-the-art methods for predictive monitoring require the training of complex machine learning models, which is often inefficient. Moreover, most of these methods require a hyper-parameter optimization that requires several repetitions of the training process which is not feasible in many real-life applications. In this paper, we propose an instance selection procedure that allows sampling training process instances for prediction models. We show that our instance selection procedure allows for a significant increase of training speed for next activity and remaining time prediction methods while maintaining reliable levels of prediction accuracy.

Knowledge and Information Systems, 2018

Instance reduction techniques are data preprocessing methods originally developed to enhance the nearest neighbor rule for standard classification. They reduce the training data by selecting or generating representative examples of a given problem. These algorithms have been designed and widely analyzed in multi-class problems providing very competitive results. However, this issue was rarely addressed in the context of one-class classification. In

Neural Computing and Applications, 2017

Computer-aided sleep monitoring system can effectively reduce the burden of experts in analyzing the large volume of electroencephalogram (EEG) recordings corresponding to sleep stages. In this paper, a new technique for automated classification of sleep stages based on iterative filtering of EEG signals is presented. In order to perform sleep stages classification, the EEG signals are decomposed using iterative filtering method. The modes obtained from iterative filtering of EEG signal can be considered as amplitude-modulated and frequency-modulated (AM-FM) components. The discrete energy separation algorithm (DESA) is applied to the modes to determine amplitude envelope and instantaneous frequency functions. The extracted amplitude envelope and instantaneous frequency functions have been used to compute Poincare ´plot descriptors and statistical measures. The Poincare ´plot descriptors and statistical measures are applied as input features for different classifiers in order to classify sleep stages. The classifiers namely, naı ¨ve Bayes, k-nearest neighbor, multilayer perceptron, C4.5 decision tree, and random forest are applied in order to classify the EEG epochs corresponding to various sleep stages. The experimental study has been performed on online available Sleep-EDF database for two-class to six-class classification of sleep stages based on EEG signals. The two-class to six-class classification problems are formulated by taking different combinations of EEG signals corresponding to various sleep stages. The comparison of the results is presented for different multi-class classification problems with the other recently proposed methods. The results show that the proposed method has provided better tenfold crossvalidation classification accuracy than other existing methods.

International Journal of Pattern Recognition and Artificial Intelligence, 2009

Evolutionary prototype selection has shown its effectiveness in the past in the prototype selection domain. It improves in most of the cases the results offered by classical prototype selection algorithms but its computational cost is expensive. In this paper, we analyze the behavior of the evolutionary prototype selection strategy, considering a complexity measure for classification problems based on overlapping. In addition, we have analyzed different k values for the nearest neighbour classifier in this domain of study to see its influence on the results of PS methods. The objective consists of predicting when the evolutionary prototype selection is effective for a particular problem, based on this overlapping measure.

Proceedings of the 2009 ACM symposium on Applied Computing, 2009

Instance-based classifiers that compute similarity between instances suffer from the presence of noise in the training set and from overfitting. In this paper we propose a new type of distance-based classifier that instead of computing distances between instances computes the distance between each test instance and the classes. Both are represented by patterns in the space of the frequent itemsets. We ranked the itemsets by metrics of itemset significance. Then we considered only the top portion of the ranking that leads the classifier to reach the maximum accuracy. We have experimented on a large collection of datasets from UCI archive with different proximity measures and different metrics of itemsets ranking. We show that our method has many benefits: it reduces the number of distance computations, improves the classification accuracy of state-of-the art classifiers, like decision trees, SVM, k-nn, Naive Bayes, rule-based classifiers and association rule-based ones and outperforms the competitors especially on noise data.

Memetic Computing, 2012

One issue in data classification problems is to find an optimal subset of instances to train a classifier. Training sets that represent well the characteristics of each class have better chances to build a successful predictor. There are cases where data are redundant or take large amounts of computing time in the learning process. To overcome this issue, instance selection techniques have been proposed. These techniques remove examples from the data set so that classifiers are built faster and, in some cases, with better accuracy. Some of these techniques are based on nearest neighbors, ordered removal, random sampling and evolutionary methods. The weaknesses of these methods generally involve lack of accuracy, overfitting, lack of robustness when the data set size increases and high complexity. This work proposes a simple and fast immune-inspired suppressive algorithm for instance selection, called SeleSup. According to self-regulation mechanisms, those cells unable to neutralize danger tend to disappear from the organism. Therefore, by analogy, data not relevant to the learning of a classifier are eliminated from the training process. The proposed method was compared with three important instance selection algorithms on a number of data sets. The experiments showed that our mechanism