Linear Classification: Perceptron vs WINNOW

Artificial Intelligence, 1997

We give an adversary strategy that forces the Perceptron algorithm to make a(kN) mistakes in learning monotone disjunctions over N variables with at most k literals. In contrast, Littlestone's algorithm Winnow makes at most 0(k log N) mistakes for the same problem. Both algorithms use thresholded linear functions as their hypotheses. However, Winnow does multiplicative updates to its weight vector instead of the additive updates of the Perceptron algorithm. In general, we call an algorithm additive if its weight vector is always a sum of a fixed initial weight vector and some linear combination of already seen instances. Thus, the Perceptron algorithm is an example of an additive algorithm. We show that an adversary can force any additive algorithm to make (N + k-1) /2 mistakes in learning a monotone disjunction of at most k literals. Simple experiments show that for k < N, Winnow clearly outperforms the Perceptron algorithm also on nonadversarial random data. @ 1997 Elsevier Science B.V.

A breakdown of the statistical and algorithmic difference between logistic regression and perceptron. The purpose of this abstract is to derive the learning algorithm behind this widely used machine/deep learning algorithm with their scratch python implementation.

1994

DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

Supervised machine learning is an important task for learning artificial neural networks; therefore a demand for selected supervised learning algorithms such as back propagation algorithm, decision tree learning algorithm and perceptron algorithm has been arise in order to perform the learning stage of the artificial neural networks. In this paper; a comparative study has been presented for the aforementioned algorithms to evaluate their performance within a range of specific parameters such as speed of learning, overfitting avoidance, and their accuracy. Besides these parameters we have included their benefits and limitations to unveil their hidden features and provide more details regarding their performance. We have found the decision tree algorithm is the best as compared with other algorithms that can solve the complex problems with a remarkable speed.

We present a detailed experimental comparison of the pocket algorithm, thermal perceptron, and barycentric correction procedure algorithms that most commonly used algorithms for training threshold logic units (TLUs). Each of these algorithms represent stable variants of the standard perceptron learning rule in that they guarantee convergence to zero classification errors on datasets that are linearly separable and attempt to classify as large a subset of the training patterns as possible for datasets that are not linearly ...

Perceptron Multilayer, 2024

This paper provides a comprehensive examination of the perceptron and multilayer perceptron (MLP) models, em- phasizing their historical significance and practical applications in artificial intelligence and machine learning. It begins with an overview of the perceptron, introduced by Frank Rosenblatt in 1958, as a foundational element in neural network research, capable of solving linear classification problems. The paper discusses the limitations of single-layer perceptrons, particularly their inability to address non-linear problems like the XOR problem, which led to the development of multilayer perceptrons and the backpropagation algorithm in the 1980s. The study details the implementation of a simple neural network with one hidden layer using C++, focusing on key com- ponents such as activation functions, weight updates, and training methods. It also explores the integration of the perceptron model with hardware components, specifically using the ESP32 microcontroller to demonstrate real-world applications, including controlling LEDs based on model predictions. Furthermore, the paper evaluates the performance and generalization capabilities of both perceptron and multilayer perceptron models through training and validation datasets. In addition to practical implementations, the paper discusses the evolution of neural network architectures, including convo- lutional and recurrent neural networks, and their relevance in solving complex problems beyond the scope of simpler models. The findings underscore the importance of understanding the perceptron as a stepping stone in the broader context of neural network research and its implications for future advancements in artificial intelligence.

IEEE Transactions on Neural Networks, 1997

Pattern classification using neural networks and statistical methods is discussed. We give a tutorial overview in which popular classifiers are grouped into distinct categories according to their underlying mathematical principles; also, we assess what makes a classifier neural. The overview is complemented by two case studies using handwritten digit and phoneme data that test the performance of a number of most typical neural-network and statistical classifiers. Four methods of our own are included: Reduced kernel discriminant analysis, the learning -nearest neighbors classifier, the averaged learning subspace method, and a version of kernel discriminant analysis. , in 1993. His current research interests include neural networks, pattern recognition, as well as computational and mathematical statistics, in particular nonparametric function estimation. His earlier work has also included functional analysis, geometric modeling, and computer aided design.

We investigate the comparative performance of SVMs and MLPs in terms of average balanced and unbalanced error rates, and their generalization, on practical classification tasks. For that purpose we carried out repeated classification experiments on 35 public real-world datasets using SVMs with RBF kernel and MLPs with four risk functionals. These included the classical mean square error (MSE), the cross-entropy (CE), and two unconventional risks proposed in recent years: EXP, a generalized exponential risk, and HS, the Shannon entropy of the classifier's output error.

Proceedings of the 2010 SIAM International Conference on Data Mining, 2010

State-of-the-art learning algorithms accept data in feature vector format as input. Examples belonging to different classes may not always be easy to separate in the original feature space. One may ask: can transformation of existing features into new space reveal significant discriminative information not obvious in the original space? Since there can be infinite number of ways to extend features, it is impractical to first enumerate and then perform feature selection. Second, evaluation of discriminative power on the complete dataset is not always optimal. This is because features highly discriminative on subset of examples may not necessarily be significant when evaluated on the entire dataset. Third, feature construction ought to be automated and general, such that, it doesn't require domain knowledge and its improved accuracy maintains over a large number of classification algorithms. In this paper, we propose a framework to address these problems through the following steps: (1) divide-conquer to avoid exhaustive enumeration; (2) local feature construction and evaluation within subspaces of examples where local error is still high and constructed features thus far still do not predict well; (3) weighting rules based search that is domain knowledge free and has provable performance guarantee. Empirical studies indicate that significant improvement (as much as 9% in accuracy and 28% in AUC) is achieved using the newly constructed features over a variety of inductive learners evaluated against a number of balanced, skewed and high-dimensional datasets. Software and datasets are available from the authors.

2014

The Direct Kernel Perceptron (DKP) ) is a very simple and fast kernelbased classifier, related to the Support Vector Machine (SVM) and to the Extreme Learning Machine (ELM) , whose α-coefficients are calculated directly, without any iterative training, using an analytical closed-form expression which involves only the training patterns. The DKP, which is inspired by the Direct Parallel Perceptron, , uses a Gaussian kernel and a linear classifier (perceptron). The weight vector of this classifier in the feature space minimizes an error measure which combines the training error and the hyperplane margin, without any tunable regularization parameter. This weight vector can be translated, using a variable change, to the α-coefficients, and both are determined without iterative calculations. We calculate solutions using several error functions, achieving the best trade-off between accuracy and efficiency with the linear function. These solutions for the α coefficients can be considered alternatives to the ELM with a new physical meaning in terms of error and margin: in fact, the linear and quadratic DKP are special cases of the two-class ELM when the regularization parameter C takes the values C = 0 and C = ∞. The linear DKP is extremely efficient and much faster (over a vast collection of 42 benchmark and real-life data sets) than 12 very popular and accurate classifiers including SVM, Multi-Layer Perceptron, Adaboost, Random Forest and Bagging of RPART decision trees, Linear Discriminant Analysis, K-Nearest Neighbors, ELM, Probabilistic Neural Networks, Radial Basis Function neural networks and Generalized ART. Besides, despite its simplicity and extreme efficiency, DKP achieves higher accuracies than 7 out of 12 classifiers, exhibiting small differences with respect to the best ones (SVM, ELM, Adaboost and Random Forest), which are much slower. Thus, the DKP provides an easy and fast way to achieve classification accuracies which are not too far from the best one for a given problem. The C and Matlab code of DKP are freely available. 1