Alex Smola

Carnegie Mellon University, Machine Learning, Faculty Member

Followers

Following

Co-authors

Public Views

Interests

Uploads

Papers by Alex Smola

A Generalized Representer Theorem

Lecture Notes in Computer Science, 2001

Wahba's classical representer theorem states that the solutions of certain risk minimization prob... more Wahba's classical representer theorem states that the solutions of certain risk minimization problems involving an empirical risk term and a quadratic regularizer can be written as expansions in terms of the training examples. We generalize the theorem to a larger class of regularizers and empirical risk terms, and give a self-contained proof utilizing the feature space associated with a kernel. The result shows that a wide range of problems have optimal solutions that live in the finite dimensional span of the training examples mapped into feature space, thus enabling us to carry out kernel algorithms independent of the (potentially infinite) dimensionality of the feature space.

Download

Robust Ensemble Learning for Data Mining

Lecture Notes in Computer Science, 2000

ABSTRACT We propose a new boosting algorithm which similarly to - Support-Vector Classi cation al... more ABSTRACT We propose a new boosting algorithm which similarly to - Support-Vector Classi cation allows for the possibility of a pre-speci ed fraction of points to lie in the margin area or even on the wrong side of the decision boundary. It gives a nicely interpretable way of controlling the trade-o between minimizing training error and capacity. Furthermore, it can act as a lter for nding and selecting informative patterns from a database.

Large Margin Classification for Moving Targets

Lecture Notes in Computer Science, 2002

We consider using online large margin classification algorithms in a setting where the target cla... more We consider using online large margin classification algorithms in a setting where the target classifier may change over time. The algorithms we consider are Gentile's Alma, and an algorithm we call Norma which performs a modified online gradient descent with respect to a regularised risk. The update rule of Alma includes a projectionbased regularisation step, whereas Norma has a weight decay type of regularisation. For Alma we can prove mistake bounds in terms of the total distance the target moves during the trial sequence. For Norma, we need the additional assumption that the movement rate stays sufficiently low uniformly over time. In addition to the movement of the target, the mistake bounds for both algorithms depend on the hinge loss of the target. Both algorithms use a margin parameter which can be tuned to make them mistake-driven (update only when classification error occurs) or more aggressive (update when the confidence of the classification is below the margin). We get similar mistake bounds both for the mistakedriven and a suitable aggressive tuning. Experiments on artificial data confirm that an aggressive tuning is often useful even if the goal is just to minimise the number of mistakes.

Download

Support Vector Regression with Automatic Accuracy Control

Perspectives in Neural Computing, 1998

A new algorithm for Support Vector regression is proposed. For a priori chosen , it automatically... more

Download

Efficient mini-batch training for stochastic optimization

Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014

Download

Learning high-order MRF priors of color images

Proceedings of the 23rd international conference on Machine learning - ICML '06, 2006

String kernels which compare the set of all common substrings between two given strings have rece... more String kernels which compare the set of all common substrings between two given strings have recently been proposed by . Surprisingly, these kernels can be computed in linear time and linear space using annotated suffix trees. Even though, in theory, the suffix tree based algorithm requires O(n) space for an n length string, in practice at least 40n bytes are required --20n bytes for storing the suffix tree, and an additional 20n bytes for the annotation. This large memory requirement coupled with poor locality of memory access, inherent due to the use of suffix trees, means that the performance of the suffix tree based algorithm deteriorates on large strings. In this paper, we describe a new linear time yet space efficient and scalable algorithm for computing string kernels, based on suffix arrays. Our algorithm is a) faster and easier to implement, b) on the average requires only 19n bytes of storage, and c) exhibits strong locality of memory access. We show that our algorithm can be extended to perform linear time prediction on a test string, and present experiments to validate our claims.

Download

Collaborative competitive filtering

Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, 2011

While a user's preference is directly reflected in the interactive choice process between her and... more While a user's preference is directly reflected in the interactive choice process between her and the recommender, this wealth of information was not fully exploited for learning recommender models. In particular, existing collaborative filtering (CF) approaches take into account only the binary events of user actions but totally disregard the contexts in which users' decisions are made. In this paper, we propose Collaborative Competitive Filtering (CCF), a framework for learning user preferences by modeling the choice process in recommender systems. CCF employs a multiplicative latent factor model to characterize the dyadic utility function. But unlike CF, CCF models the user behavior of choices by encoding a local competition effect. In this way, CCF allows us to leverage dyadic data that was previously lumped together with missing data in existing CF models. We present two formulations and an efficient large scale optimization algorithm. Experiments on three real-world recommendation data sets demonstrate that CCF significantly outperforms standard CF approaches in both offline and online evaluations.

Download

Unified analysis of streaming news

Proceedings of the 20th international conference on World wide web, 2011

News clustering, categorization and analysis are key components of any news portal. They require ... more News clustering, categorization and analysis are key components of any news portal. They require algorithms capable of dealing with dynamic data to cluster, interpret and to temporally aggregate news articles. These three tasks are often solved separately. In this paper we present a unified framework to group incoming news articles into temporary but tightly-focused storylines, to identify prevalent topics and key entities within these stories, and to reveal the temporal structure of stories as they evolve. We achieve this by building a hybrid clustering and topic model. To deal with the available wealth of data we build an efficient parallel inference algorithm by sequential Monte Carlo estimation. Time and memory costs are nearly constant in the length of the history, and the approach scales to hundreds of thousands of documents. We demonstrate the efficiency and accuracy on the publicly available TDT dataset and data of a major internet news site.

Download

Support vector machine learning for interdependent and structured output spaces

Twenty-first international conference on Machine learning - ICML '04, 2004

Learning general functional dependencies is one of the main goals in machine learning. Recent pro... more Learning general functional dependencies is one of the main goals in machine learning. Recent progress in kernel-based methods has focused on designing flexible and powerful input representations. This paper addresses the complementary issue of problems involving complex outputs such as multiple dependent output variables and structured output spaces. We propose to generalize multiclass Support Vector Machine learning in a formulation that involves features extracted jointly from inputs and outputs. The resulting optimization problem is solved efficiently by a cutting plane algorithm that exploits the sparseness and structural decomposition of the problem. We demonstrate the versatility and effectiveness of our method on problems ranging from supervised grammar learning and named-entity recognition, to taxonomic text classification and sequence alignment.

Download

Joint Kernel Maps

Discriminative Models

Energy-Based Models

Learning as Search Optimization

This chapter contains sections titled: Introduction, Previous Work, Search Optimization, Experime... more

Efficient Algorithms for Max-Margin Structured Classification

Fair and balanced: Learning to present news stories

Relevance, diversity and personalization are key issues when presenting content which is apt to p... more Relevance, diversity and personalization are key issues when presenting content which is apt to pique a user's interest. This is particularly true when presenting an engaging set of news stories. In this paper we propose an efficient algorithm for selecting a small subset of relevant articles from a streaming news corpus. It offers three key pieces of improvement over past work: 1) It is based on a detailed model of a user's viewing behavior which does not require explicit feedback. 2) We use the notion of submodularity to estimate the propensity of interacting with content. This improves over the classical context independent relevance ranking algorithms. Unlike existing methods, we learn the submodular function from the data. 3) We present an efficient online algorithm which can be adapted for personalization, story adaptation, and factorization models. Experiments show that our system yields a significant improvement over a retrieval system deployed in production.

Download

Introduction to Machine Learning

WWW 2012 Invited Tutorial Overview New Templates for Scalable Data Analysis

v-Arc: Ensemble Learning in the Presence of Outliers

AdaBoost and other ensemble methods have successfully been applied to a number of classification ... more AdaBoost and other ensemble methods have successfully been applied to a number of classification tasks, seemingly defying problems of overfitting. AdaBoost performs gradient descent in an error function with respect to the margin, asymptotically concentrating on the patterns which are hardest to learn. For very noisy problems, however, this can be disadvantageous. Indeed, theoretical analysis has shown that the margin distribution, as opposed to just the minimal margin, plays a crucial role in understanding this phenomenon. Loosely speaking, some outliers should be tolerated if this has the benefit of substantially increasing the margin on the remaining points. We propose a new boosting algorithm which allows for the possibility of a pre-specified fraction of points to lie in the margin area Or even on the wrong side of the decision boundary.

Download

Gaussian Process Belief Propagation

The framework of graphical models is a cornerstone of applied Statistics, allowing for an intuiti... more The framework of graphical models is a cornerstone of applied Statistics, allowing for an intuitive graphical specification of the main features of a model, and providing a basis for general Bayesian inference computations though belief propagation (BP). In the latter, messages are passed between marginal beliefs of groups of variables. In parametric models, where all variables are of fixed finite dimension, these beliefs and messages can be represented easily in tables or parameters of exponential families, and BP techniques are widely used in this case. In this paper, we are interested in nonparametric models, where belief representations do not have a finite dimension, but grow with the dataset size. In the presence of several dependent domain variables, each of which is represented as a nonparametric random field, we aim for a synthesis of BP and nonparametric approximate inference techniques. We highlight the difficulties in exercising this venture and suggest possible techniques for remedies. We demonstrate our program using the example of semiparametric latent factor models [15], which can be used to model conditional dependencies between multiple responses.

Download

Generalization Bounds and Consistency for Structured Labeling

Alex Smola

Uploads

Papers by Alex Smola

Log In