Modeling score distributions in information retrieval

Encyclopedia of Data Warehousing and Mining, Second Edition, 2009

The goal of a web-based retrieval system is to find data items that meet a user’s request as fast and accurately as possible. Such a search engine finds items relevant to the user’s query by scoring and ranking each item in the database. Swets (1963) proposed to model the distributions of these scores to find an optimal threshold for separating relevant from non-relevant items. Since then, researchers suggested several different score distribution models, which offer elegant solutions to improve the effectiveness and efficiency of different components of search systems. Recent studies show that the method of modeling score distribution is beneficial to various applications, such as outlier detection algorithms (Gao & Tan, 2006), search engines (Manmatha, Feng, & Rath, 2001), information filtering (Zhang & Callan, 2001), distributed information retrieval (Baumgarten, 1999), video retrieval (Wilkins, Ferguson, & Smeaton, 2006), kernel type selection for image retrieval (Doloc-Mihu & R...

2009

Comparative evaluations of information retrieval systems are based on a number of key premises, including that representative topic sets can be created, that suitable relevance judgements can be generated, and that systems can be sensibly compared based on their aggregate performance over the selected topic set. This paper considers the role of the third of these assumptions -that the performance of a system on a set of topics can be represented by a single overall performance score such as the average, or some other central statistic. In particular, we experiment with score aggregation techniques including the arithmetic mean, the geometric mean, the harmonic mean, and the median. Using past TREC runs we show that an adjusted geometric mean provides more consistent system rankings than the arithmetic mean when a significant fraction of the individual topic scores are close to zero, and that score standardization (Webber et al., SIGIR 2008) achieves the same outcome in a more consistent manner.

Information retrieval techniques deal with different units of information such as terms, topics or documents. There usually exist explicit or implicit link structures between different items of each unit or between items across different units. For example hyperlinks between pages in a hypertext collection are explicit structures, while the links between terms in a co-occurrence network are implicit structures. Many of the traditional information retrieval methods only use the content information of the items for retrieval purposes and overlook the link structures. Those that use the link structures also do not fully exploit the discrimination power of contents as well as all useful link information. In this thesis, we propose a general probabilistic score propagation framework for combining content and link information, which can fully take advantage of content information and the link

2008

In this work 1 , we analyze the popular KLdivergence ranking function in information retrieval. We uncover the generative distribution, namely the Smoothed Dirichlet distribution, underlying this ranking function and show that this distribution captures term occurrence distribution much better than the multinomial, thus offering, for the first time, a reason behind the success of the KLdivergence ranking function. We present theoretically motivated approximations to the distribution that lead to a closed form maximum likelihood solution, much like the multinomial, making it ideal for online IR tasks. We use the new distribution to construct a new, well-motivated ad-hoc retrieval algorithm. Our experiments show that this algorithm performs at least as well as similar algorithms that employ cross-entropy ranking. It also provides additional flexibility, e.g. in handling scenarios like a mixture of true and pseudo relevance feedback, due to a consistent generative framework.

String Processing and Information Retrieval, 2015

This paper investigates an approach for estimating the effectiveness of any IR system. The approach is based on the idea that a set of documents retrieved for a specific query is highly relevant if there are only a small number of predominant topics in the retrieved documents. The proposed approach is to determine the topic probability distribution of each document offline, using Latent Dirichlet Allocation. Then, for a retrieved set of documents, a set of probability distribution shape descriptors, namely the skewness and the kurtosis, are used to compute a score based on the shape of the cumulative topic distribution of the respective set of documents. The proposed model is termed DeShaTo, which is short for Describing the Shape of cumulative Topic distributions. In this work, DeShaTo is used to rank retrieval systems without relevance judgments. In most cases, the empirical results are better than the state of the art approach. Compared to other approaches, DeShaTo works independently for each system. Therefore, it remains reliable even when there are less systems to be ranked by relevance.

2012

Inferring the distributions of relevant and nonrelevant documents over a ranked list of scored documents returned by a retrieval system has a broad range of applications including information filtering, recall-oriented retrieval, metasearch, and distributed IR. Typically, the distribution of documents over scores is modeled by a mixture of two distributions, one for the relevant and one for the nonrelevant documents, and expectation maximization (EM) is run to estimate the mixture parameters.

… of the 28th annual international ACM …, 2005

ACM Transactions on Information Systems, 2008

Although the seminal proposal to introduce Language Modeling in Information Retrieval was based on a multi-variate Bernoulli model, the predominant modeling approach is now centered on Multinomial models. Language modeling for retrieval based on multi-variate Bernoulli distributions is seen to be inefficient and believed to be less effective than the Multinomial model. In this paper, we examine the multi-variate Bernoulli model with respect to its successor and examine its role in future retrieval systems. In the context of Bayesian Learning both modeling approaches are described, contrasted and compared theoretically and computationally. We show that the query likelihood following a multi-variate Bernoulli distribution introduces interesting retrieval features which may be useful for specific retrieval tasks such as sentence retrieval. Then, we address the efficiency aspect and show that algorithms can be designed to perform retrieval efficiently for multi-variate Bernoulli models, before performing an empirical comparison to study the behaviorial aspects of the models. A series of comparisons is then conducted on a number of test collections and retrieval tasks to determine the empirical and practical differences between the different models. Our results indicate that for sentence retrieval the multi-variate Bernoulli model can significantly outperform the Multinomial model. However, for the other tasks the Multinomial model provides consistently better performance (and in most cases significantly so). An analysis of the various retrieval characteristics reveals that the multi-variate Bernoulli model tends to promote long documents whose non-query terms are informative. While this is detrimental to the task of document retrieval (documents tend to contain considerable non-query content), it is valuable for other tasks such as sentence retrieval, where the retrieved elements are very short and focused.

… of the 17th annual international ACM …, 1994

The 2-Poisson model for term frequencies is used to suggest ways of incorporating certain variables in probabilistic models for information retrieval. The variables concerned are within-document term tkequency, document length, and within-query term frequency. Simple weighting functions are developed, and tested on the TREC test collection. Considerable performance improvements (over simple inverse collection frequency weighting) are demonstrated. Search