PART OF SPEECH N-GRAMS AND INFORMATION RETRIEVAL

Journal of the Association for Information Science and Technology, 2016

Purpose: The article discusses on a general methodological level different methods that have been used for management of single key word form variation in information retrieval during the history of textual information retrieval. The paper offers the reader an overall practical guide for choosing between different methods to be used for different types of European languages. Methods being compared in the paper include stemming, lemmatization, truncation, syllabification, unsupervised morphological methods, character n-gramming and generation of inflected word forms. Methodology/Approach: Based on the empirical findings and results achieved by other researchers the paper discusses several pros and cons of different keyword variation management methods in a broader context than usually in IR, where only achieved effectiveness results are normally considered. The study proposes a list of five criteria for comparison of the conflation methods in general and offer a heuristics for choosing a suitable method for conflation of a specific language. Findings: Simpler character-based methods could be preferred in IR instead of very sophisticated linguistic methods. It is also suggested that for morphologically simple languages, such as English, any kind of keyword variation management may be futile, as the increase in IR effectiveness achieved may be very low. Morphologically more complex languages can be conflated with the simple methods quite effectively for present IR search engines.

Proceedings of the sixth conference on Applied natural language processing -, 2000

This paper describes the results of some experiments using a new approach to information access that combines techniques from natural language processing and knowledge representation with a penaltybased technique for relevance estimation and passage retrieval. Unlike many attempts to combine natural language processing with information retrieval, these results show substantial benefit from using linguistic knowledge.

Journal of the American Society for Information Science and Technology, 2007

are developed. The interaction between the syntagm and the paradigm in computational operations on written language in indexing, searching, and retrieval is used to account for transformations of the signified or meaning between documents and their representation and between queries and documents retrieved. Characteristics of the message, and messages for selection for written language, are brought to explain the relative frequency of occurrence of words and multiple word sequences in documents. The examples given in the companion article are revisited and a fuller example introduced. The signified of the sequence stood for, the term classically used in the definitions of the sign, as something standing for something else, can itself change rapidly according to its syntagm. A greater than ordinary discourse understanding of patterns in retrieval is obtained.

International Journal of Electronic Business, 2005

One of the problems of e-Business is to find relevant documents for making correct decisions. The main problem of the Internet is the huge amount of documents that makes it difficult to find the relevant ones, hence the importance of the methods allowing for improving the quality of document retrieval. We discuss some linguistic problems of document retrieval on the Internet related to the following natural language phenomena: (1) morphological processes: e.g., takes, took, taken are grammar forms of take, (2) polysemy and homonymy: most words have several senses, e.g., bank is a financial institution, shore, bench, etc., (3) non-linearity of syntactic relations: in case of a query that contains word combinations, the words forming a word combination can be separated by other words in the documents. Some linguistic-based methods and strategies related to the discussed problems are proposed that improve the quality of document retrieval or show the necessity of application of linguistic methods.