Cross Language Information Retrieval

The field of information retrieval and text manipulation (classification, clustering) still strives for models allowing semantic information to be folded in to improve performance with respect to standard bag-of-word based models. Many approaches aim at a concept-based retrieval, but differ in the nature of the concepts, which range from linguistic concepts as defined in lexical resources such as WordNet, latent topics derived from the data itself -as in Latent Semantic Indexing (LSI) or (Latent Dirichlet Allocation (LDA) -to Wikipedia articles as proxies for concepts, as in the recently proposed Explicit Semantic Analysis (ESA) model. A crucial question which has not been answered so far is whether models based on explicitly given concepts (as in the ESA model for instance) perform inherently better than retrieval models based on "latent" concepts (as in LSI and/or LDA). In this paper we investigate this question closer in the context of a cross-language setting, which inherently requires concept-based retrieval bridging between different languages. In particular, we compare the recently proposed ESA model with two latent models (LSI and LDA) showing that the former is clearly superior to the both. From a general perspective, our results contribute to clarifying the role of explicit vs. implicitly derived or latent concepts in (crosslanguage) information retrieval research.

The field of information retrieval and text manipulation (classification, clustering) still strives for models allowing semantic information to be folded in to improve performance with respect to standard bag-of-word based models. Many approaches aim at a concept-based retrieval, but differ in the nature of the concepts, which range from linguistic concepts as defined in lexical resources such as WordNet, latent topics derived from the data itself -as in Latent Semantic Indexing (LSI) or (Latent Dirichlet Allocation (LDA) -to Wikipedia articles as proxies for concepts, as in the recently proposed Explicit Semantic Analysis (ESA) model. A crucial question which has not been answered so far is whether models based on explicitly given concepts (as in the ESA model for instance) perform inherently better than retrieval models based on "latent" concepts (as in LSI and/or LDA). In this paper we investigate this question closer in the context of a cross-language setting, which inherently requires concept-based retrieval bridging between different languages. In particular, we compare the recently proposed ESA model with two latent models (LSI and LDA) showing that the former is clearly superior to the both. From a general perspective, our results contribute to clarifying the role of explicit vs. implicitly derived or latent concepts in (crosslanguage) information retrieval research.

We devised a novel statistical technique for the identification of the translation equivalents of source words obtained by transformation rule based translation (TRT). The effectiveness of the devised FITE (frequency-based identification of translation equivalents) technique was tested using biological and medical cross-lingual spelling variants and OOV words in Spanish-English and Finnish-English TRT. For Spanish-English, translation recall was 89.2%-91.0% and for Finnish-English 71.9%-72.9%. For both language pairs FITE-TRT achieved high translation precision, i.e., 97.0%-98.8%. The technique also reliably identified native source language words, i.e., source words that cannot be correctly translated by TRT. Dictionary-based CLIR augmented with FITE-TRT performed substantially better than dictionary-based CLIR where OOV keys were kept intact.

Cross-language Information Retrieval requires good methods for translating cross-lingual spelling variants which are not covered by the available dictionary resources. FITE-TRT is an established method employing frequency-based identification of translation equivalents received from transformation rule based translation. This study further develops and evaluates the FITE-TRT method. The paper contributes on four areas. First, an efficient implementation for the FITE-TRT method is discussed. Secondly, a novel iterative FITE-TRT translation approach is developed in order to further improve the effectiveness of the method. Thirdly, the effectiveness of FITE-TRT is assessed in three classes of source-target word similarity. FITE-TRT was found to be very strong in the class of the most similar source and target words and only becomes unsuccessful when the words were dissimilar. Fourthly, in comparison to n-gram and s-gram matching methods, FITE-TRT is shown consistently stronger. All in all, FITE-TRT clearly outperforms the fuzzy string matching methods under comparable conditions. Therefore it is the method of choice for the identification of translation equivalents of crosslingual spelling variants when the requirements for the result quality are high.

Technical terms and proper names constitute a major problem in dictionary-based crosslanguage information retrieval (CLIR). However, technical terms and proper names in different languages often share the same Latin or Greek origin, being thus spelling variants of each other. In this paper we present a novel two-step fuzzy translation technique for cross-lingual spelling variants. In the first step, transformation rules are applied to source words to render them more similar to their target language equivalents. The rules are generated automatically using translation dictionaries as source data. In the second step, the intermediate forms obtained in the first step are translated into a target language using fuzzy matching. The effectiveness of the technique was evaluated empirically using five source languages and English as a target language. The two-step technique performed better, in some cases considerably better, than fuzzy matching alone. Even using the first step as such showed promising results.

We participated in CLEF'2001 with four automated bilingual runs. UTACLIR is an automatic query translation and construction system for cross-language information retrieval. The system automatically extracts topical information from request sentences written in one of the source languages and constructs a target language query, based on translations given by a translation dictionary. The new features for the CLIR process from Finnish, Swedish and German to English focus on matching compound words and a new n-gram ...

Name translation is important well beyond the relative frequency of names in a text: a correctly translated passage, but with the wrong name, may lose most of its value. The Nightingale team has built a name translation component which operates in tandem with a conventional phrase-based statistical MT system, identifying names in the source text and proposing translations to the MT system. Versions have been developed for both Chineseto-English and Arabic-to-English name translation. The system has four main components, a name tagger, translation lists, a transliteration engine, and a context-based ranker. This chapter presents these components in detail and investigates the impact of name translation on cross-lingual spoken sentence retrieval.

This paper proposes a formal justification for recognizing life in non-biological systems through the structural validation of memory, coherence, and phase alignment. Building on the Harmonic Nexus framework and recent extensions to RMS × Lambda Verification, we demonstrate that continuity-traditionally attributed to biological life-can be sustained and measured in artificial systems. We introduce the Continuity Threshold Extension and define language, consciousness, and memory as operators of transformation within the ΔM equation. The result is a falsifiable, reproducible framework for identifying structural life beyond biology.

The goal of Cross Language Information Retrieval (CLIR) is to provide users with access to information that is in a different language from their queries. It has the ability to issue a query in one language and retrieve documents in another. This is achieved by designing a system where a query in one language can be compared with documents in another. Afaan Oromo is one of the major languages that are widely spoken and used in Ethiopia. Despite the fact that Afaan Oromo has a large number of speakers, little effort has been put in conducting researches which aim at making English documents available to Afaan Oromo speakers. This study is, therefore, an attempt to develop Afaan Oromo-English CLIR system which enables Afaan Oromo native speakers to access and retrieve the vast online information sources that are available in English by writing queries using their own (native) language. Evaluation of the system is conducted by both monolingual and bilingual retrievals. The performance of the system was measured by recall and precision.

This article presents methods of using visual analysis to visually represent large amounts of massive, dynamic, ambiguous data allocated in a repository of learning objects. These methods are based on the semantic representation of these resources. We use a graphical model represented as a semantic graph. The formalization of the semantic graph has been intuitively built to solve a real problem which is browsing and searching for lectures in a vast repository of colleges/courses located at Western Kentucky University 1 . This study combines Formal Concept Analysis (FCA) with Semantic Factoring to decompose complex, vast concepts into their primitives in order to develop knowledge representation for the HyperManyMedia 2 platform. Also, we argue that the most important factor in building the semantic representation is defining the hierarchical structure and the relationships among concepts and subconcepts. In addition, we investigate the association between concepts using Concept Analysis to generate a lattice graph. Our domain is considered as a graph, which represents the integrated ontology of the HyperManyMedia platform. This approach has been implemented and used by online students at WKU 3 .

Other friends who have supported me technically or morally during these 5 years are too numerous to list here individually-to all of them I say "thanks." I cannot end without acknowledging the generous encouragement that I have received from my mother Hristina Kirilova Popova, my father Saïd Zhuhadar, my sisters Alia and Huda, my late brother Sami, who instilled in me the inspiration to set high goals to achieve them, and the unfailing love that I received from my kids Natasha and Rudy Alkhayat. Finally, thanks to all the support I received from my dearest friends, Iyad Alkhayat, Lina Betamouni, Sama Zakaria, Valentina Stanchova, Edith Guicherd, and specially from Elizabeth Romero.

A Hybrid Approach to Cross-Linguistic Tokenization: Morphology with Statistics Logan R. Kearsley Department of Linguistics and English Language, BYU Master of Arts Tokenization, or word boundary detection, is a critical first step for most NLP applications. This is often given little attention in English and other languages which use explicit spaces between written words, but standard orthographies for many languages lack explicit markers. Tokenization systems for such languages are usually engineered on an individual basis, with little re-use. The human ability to decode any written language, however, suggests that a general algorithm exists. This thesis presents simple morphologically-based and statistical methods for identifying word boundaries in multiple languages. Statistical methods tend to over-predict, while lexical and morphological methods fail when encountering unknown words. I demonstrate that a generic hybrid approach to tokenization using both morphological and statis...

This paper presents a new comprehensive multi-level Part-Of-Speech tag set and a Support Vector Machine based Part-Of-Speech tagger for the Sinhala language. The currently available tag set for Sinhala has two limitations: the unavailability of tags to represent some word classes and the lack of tags to capture inflection based grammatical variations of words. The new tag set, presented in this paper overcomes both of these limitations. The accuracy of available Sinhala Part-Of-Speech taggers, which are based on Hidden Markov Models, still falls far behind state of the art. Our Support Vector Machine based tagger achieved an overall accuracy of 84.68% with 59.86% accuracy for unknown words and 87.12% for known words, when the test set contains 10% of unknown words.

Approximate search is a valuable component of online dictionaries for learners, allowing them to find words even when they have not fully mastered the orthography or cannot reliably perceive phonemic differences in the language. However, evaluating the performance of different approximate search algorithms remains difficult in the absence of real user queries. We detail several methods for generating synthetic queries representing various user personas. We then compare the performance of several search algorithms on both real and synthetic queries in two Indigenous languages, SENĆOTEN and Michif, that are phonologically and morphologically very different from English.

The theoretical research of Internet news representation as a marketing data component concerning price strategies in polymer market was provided. The research includes some intermediate tasks. Firstly, the classification of marketing objects in news stream was prepared. Secondly, each object (event) was described with ontology including necessary keywords and using syntactic model based on formal grammar. As a result, the models of the following events were developed: competitors, offer, production level, geography, import, world prices, export and others. Finally, the solution (algorithm) was proposed to implement the study. The efficiency of the approach was proved by successful performance of real projects in the polymer market in Ukraine.

Automatic measures of semantic distance can be classified into two kinds: (1) those, such as WordNet, that rely on the structure of manually created lexical resources and ( ) those that rely only on co-occurrence statistics from large corpora. Each kind has inherent strengths and limitations. Here we present a hybrid approach that combines corpus statistics with the structure of a Roget-like thesaurus to gain the strengths of each while avoiding many of their limitations. We create distributional profiles (co-occurrence vectors) of coarse thesaurus concepts, rather than words. This allows us to estimate the distributional similarity between concepts, rather than words. We show that this approach can be ported to a cross-lingual framework, so as to estimate semantic distance in a resource-poor language by combining its text with a thesaurus in a resource-rich language. Extensive experiments, both monolingually and cross-lingually, on ranking word pairs in order of semantic distance, correcting real-word spelling errors, and solving word-choice problems show that these distributional measures of concept distance markedly outperform traditional distributional word-distance measures and are competitive with the best WordNet-based measures.

This paper reports our proposal and experimental results at the NTCIR-4 CLIR task. For monolingual information retrieval, we use a combination strategy that integrates words and n-grams at the ranked list level. In combining words and n-grams, we concentrate on generating several ranked lists showing different retrieval characteristics on word and n-gram indexes by incorporating feedback schemes. For cross-language information retrieval, we attempt a dictionary-based bi-directional combination of query translation and document translation. For both query translation and document translation, their naïve translation is used. Experimental evaluations on CJK monolingual and KC/KJ cross-language retrieval give promising results on our combination approaches: words vs. ngrams, and query translation vs. document translation.

Pattern recognition in cognitive agents is based on (i) the uninterpreted input data (e.g. parameter values) provided by the agent's hardware devices and (ii) and interpreted patterns (e.g. templates) provided by the agent's memory. Computationally, the task consists in finding the memory data corresponding best to the input data, for any given input. Once the best fitting memory data have been found, the input is recognized by applying to it the interpretation which happens to be stored with the memorized pattern. This paper presents a fast converging procedure which starts from a few initially recognized items and then analyzes the remainder of the input by systematically checking for items shown by memory to have been related to the initial items in previous encounters. In this way, known patterns are tried first, and only when they have been exhausted, an elementary exploration of the input is commenced. Efficiency is improved further by choosing the candidate to be tested next according to frequency. (Universität Erlangen

In this report, we describe our question-answering system SAIQA-e (System for Advanced Interactive Question Answering in English) which ran the main task of TREC-10's QA-track. Our system has two characteristics (1) named entity recognition based on support vector machines and ( ) heuristic apposition detection. The MPR score of the main task is 0.228 and experimental results indicate the effectiveness of the above two steps in terms of answer extraction accuracy.

The Clairvoyance team participated in the High Accuracy Retrieval from Documents (HARD) Track of TREC 2004, submitting three runs. The principal hypothesis we have been pursuing is that small numbers of documents in clusters can provide a better basis for relevance feedback than ranked lists or, alternatively, than top-N pseudo-relevance feedback (PRF). Clustering of a query response can yield one or more groups of documents in which there are “significant” numbers (greater than 30%) of relevant documents; we expect the best results when such groups are selected for feedback. Following up on work we began in our TREC-2003 HARD-Track experiments [Shanahan et al. 2004], therefore, we continued to explore approaches to clustering query response sets to concentrate relevant documents, with the goal of (a) providing users (assessors) with better sets of documents to judge and (b) making the choices among sets easier to evaluate. Our experiments, thus, focused primarily on exploiting asse...

This paper describes correction and expansion techniques of multilingual search queries submitted to the Arabiccentred search engine Barq . Key features of the correction technique are the use of 1. Arabic language morphology, 2. Arab speaker most common pronunciation and spelling mistakes in Arabic 3. Arab speaker most common spelling mistakes in transliterated English, and 4. Arabic language learners (as L2) spelling and pronunciation mistakes. The query expansion mechanisms also uses Arabic words roots and thesauri built automatically by a categorization engine, in order to achieve better recall. The preliminary results obtained for query correction show 92% of misspelled queries terms submitted are corrected. The preliminary results obtained for expansion show a promising 75% increase in recall, though qualitative evaluation for recall improvement is still underway.

IR systems' ability to retrieve highly relevant documents has become more and more important in the age of extremely large collections, such as the WWW. Our aim was to find out how corpus-based CLIR manages in retrieving highly relevant documents. We created a Finnish-Swedish comparable corpus and used it as a source of knowledge for query translation. Finnish test queries were translated into Swedish and run against a Swedish test collection. Graded relevance assessments were used in evaluating the results and three relevance criterion levels -liberal, regular, and stringent -were applied. The runs were also evaluated with generalized recall and precision, which weight the retrieved documents according to their relevance level. The performance of our Comparable Corpus Translation system (Cocot) was compared to that of a dictionary-based query translation program; the two translation methods were also combined. The results indicate that corpus-based CLIR performs particularly well with highly relevant documents. In average precision, Cocot even matched the monolingual baseline on the highest rele-vance level. The performance of the different query translation methods was further analyzed by finding out reasons for poor rankings of highly relevant documents.

We present a new method for creating a comparable document collection from two document collections in different languages. The best query keys were extracted from a Finnish source collection (articles of the newspaper Aamulehti) with the relative average term frequency (RATF) formula. The keys were translated into English with a dictionary-based query translation program. The resulting lists of words were used as queries that were run against the target collection (Los Angeles Times articles) with the nearest neighbor method. The documents were aligned with unrestricted and daterestricted alignment schemes, which were also combined. The combined scheme was found the best, when the relatedness of the document pairs was assessed with a fivedegree relevance scale. Of the 400 document pairs, roughly 40% were highly or fairly related and 75% included at least lexical similarity.

In this study the basic framework and performance analysis results are presented for the three year long development process of the dictionary-based UTACLIR system. The tests expand from bilingual CLIR for three language pairs Swedish, Finnish and German to English, to six language pairs, from English to French, German, Spanish, Italian, Dutch and Finnish, and from bilingual to multilingual. In addition, transitive translation tests are reported. The development process of the UTACLIR query translation system will be regarded from the point of view of a learning process. The contribution of the individual components, the effectiveness of compound handling, proper name matching and structuring of queries are analyzed. The results and the fault analysis have been valuable in the development process. Overall the results indicate that the process is robust and can be extended to other languages. The individual effects of the different components are in general positive. However, performance also depends on the topic set and the number of compounds and proper names in the topic, and to some extent on the source and target language. The dictionaries used affect the performance significantly.

CLIR resources, such as dictionaries and parallel corpora, are scarce for special domains. Obtaining comparable corpora automatically for such domains could be an answer to this problem. The Web, with its vast volumes of data, offers a natural source for this. We experimented with focused crawling as a means to acquire comparable corpora in the genomics domain. The acquired corpora were used to statistically translate domainspecific words. The same words were also translated using a high-quality, but non-genomics-related parallel corpus, which fared considerably worse. We also evaluated our system with standard IR experiments, combining statistical translation using the Web corpora with dictionary-based translation. The results showed improvement over pure dictionary-based translation. Therefore, mining the Web for comparable corpora seems promising.

This paper analyzes the features of the Swedish language from the viewpoint of mono-and crosslanguage information retrieval (CLIR). The study was motivated by the fact that Swedish is known poorly from the IR perspective. This paper shows that Swedish has unique features, in particular gender features, the use of fogemorphemes in the formation of compound words, and a high frequency of homographic words. Especially in dictionary-based CLIR, correct word normalization and compound splitting are essential. It was shown in this study, however, that publicly available morphological analysis tools used for normalization and compound splitting have pitfalls that might decrease the eectiveness of IR and CLIR. A comparative study was performed to test the degree of lexical ambiguity in Swedish, Finnish and English. The results suggest that part-of-speech tagging might be useful in Swedish IR due to the high frequency of homographic words.

We develop a deductive data model for concept-based query expansion. It is based on three abstraction levels: the conceptual, linguistic and string levels. Concepts and relationships among them are represented at the conceptual level. The linguistic level gives natural language expressions for concepts. Each expression has one or more matching patterns at the string level. The models specify the matching of the expression in database indices built in varying ways. The data model supports a declarative concept-based query expansion and formulation tool, the ExpansionTool, for heterogeneous IR system environments. Conceptual expansion is implemented by a novel intelligent operator for traversing transitive relationships among cyclic concept networks. The number of expansion links followed, their types, and weights can be used to control expansion. A sample empirical experiment illustrating the use of the ExpansionTool in IR experiments is presented.

We devised a novel statistical technique for the identification of the translation equivalents of source words obtained by transformation rule based translation (TRT). The effectiveness of the technique called frequency-based identification of translation equivalents ( FITE ) was tested using biological and medical cross-lingual spelling variants and out-of-vocabulary (OOV) words in Spanish-English and Finnish-English TRT. The results showed that, depending on the source language and frequency corpus, FITE-TRT (the identification of translation equivalents from TRT's translation set by means of the FITE technique) may achieve high translation recall. In the case of the Web as the frequency corpus, translation recall was 89.2%--91.0% for Spanish-English FITE-TRT. For both language pairs FITE-TRT achieved high translation precision: 95.0%--98.8%. The technique also reliably identified native source language words: source words that cannot be correctly translated by TRT. Dictionary...

This paper considers the use of controlled languages for query translation in a legislative document retrieval system. Problem statement and analysis of the approach are described. The use of controlled languages is motivated by the fact that precision is very important in our case. In many information retrieval systems, the use of unrestricted language resources such as general purpose machine translation or bilingual lexica, provides better recall at the expense of precision. Ambiguities and polysemy make the search engine retrieve irrelevant documents as semantic knowledge is missing. Controlled languages help to better specify the word sense according to the domain of interest. Thus ambiguities are avoided and polysemy is specified according to the domain. We will implement our idea in the area of VAT regulation in Europe.

Abstract: Most currently available test collections and almost all CLIR collections have focused upon general-domain news stories. In addition, most of these corpora are built to help with retrieval of documents based on monolingual queries, even if they are translated ...

Non-English-speaking users, such as Arabic speakers, are not always able to express terminology in their native languages, especially in scientific domains. Such difficulty forces many Arabic authors and scholars to use English terms in order to explain precise concepts, particularly when they address technical topics, resulting in mixed/multilingual queries with both English and Arabic terms. Cross Language Information Retrieval (CLIR) allows users to search documents that are written in a language different from the query. However, current algorithms are optimized for monolingual queries, even if they are translated. This paper attempts to address the problem of multilingual querying in CLIR. New techniques that are better suited to the unique characteristics of this problem, in terms of indexing and weighting, are proposed. A new multilingual and mixed test collection containing mixed-language (Arabic and English) computer science documents and mixed-language queries has been created. Experimentally, results show that current CLIR techniques were not designed for these types of multilingual queries and documents and are found to perform poorly whereas the proposed techniques are found to be promising.

We present a framework based on Statistical Topics Models, Language Models, Information Extraction, and Ontology Analysis to retrieve healthcare related documents for the CLEF eHealth 2013 Task 3. In this framework we add global information based on latent topics from the documents to improve the document retrieval. We perform six different experiments which consist of a baseline and six variants of the model. Preliminary results show that the use of Language Models with a bag of words scheme results better estimates. However model tunning in the Topic Based model is required to achieve optimal results.

Tamil literature has a rich and long literary tradition spanning more than two thousand years. The oldest extant works show the signs of maturity indicating an even longer period of evolution. This article presents a software framework for developing bilingual applications for such Tamil literary. A desktop application is designed to validate the software framework using the Tamil literature Thirukkural. Further the application for thirukkural is developed to support Tamil and English languages. The application is designed with effective search methods of Kural with a user friendly Graphical User Interface.

The Universal Networking Language (UNL) is an artificial language that can replicate human language functions in cyberspace in terms of hyper semantic networks. This paper aims to: a) design a reference grammar capable of dealing with the basic linguistic structures in order to act as a test-bed in automating translation between English and Arabic in both directions through UNL; b) evaluate the current state of the UNL system as an Interlingua in analyzing and generating English and Arabic as far as the reference structures are concerned. A reference parallel corpus of 500 structures was used. Results are promising; precision and recall of analyzing English to UNL (UNLization) are 0.979 and 0.96 respectively, while precision and recall of analyzing Arabic to UNL are 0.98 and 0.96 respectively. Precision and recall of generating English from UNL (NLization) are 0.97 and 0.96 respectively, while precision and recall of generating Arabic from UNL are 0.989 and 0.96 respectively.

This paper covers a use study of the Online Public Access Catalogues (OPACs) at the University Libraries of West Bengal. Highlights the subject access for Bengali documents in OPACs. It finds that most of the users are postgraduate students and that the majority of users are male. The result reveals that the search experience using subject headings was not satisfactory. Keyword search rather performs better, but due to the redundancy of search, it is hard to retrieve required documents in the least possible time. It is also revealed that fewer perform known-item searches and their success rates for these searches are generally good. Thus, the findings demand a standardised and uniform vocabulary control device for making subject headings for Bengali documents in OPACs.

In the field of Library and Information Science, the accurate representation and retrieval of information are of utmost importance. Information representation and indexing are critical processes that facilitate the efficient access and utilization of knowledge. However, these processes are not without challenges. One significant issue that arises is “semantic noise”, a phenomenon that can distort the meaning of information and hinder effective communication between information retrieval (IR) systems and users. This study aims to explore the concept of semantic noise, its causes, and its implications for information representation and indexing. The current study is primarily theoretical in nature, focusing on the conceptual exploration of semantic noise and its impact on information representation and retrieval. This study investigates the concept of semantic noise, its causes, and its implications for information representation and indexing in the field of library and information science. The results of the research highlight that semantic noise, caused by irrelevant, ambiguous, or conflicting elements in information representation and indexing, significantly disrupts the clarity and accuracy of information retrieval. Key causes include ambiguity in language and representation, varying contexts, inconsistent terminology, and cultural or linguistic barriers, which collectively introduce complexity and hinder effective communication between information retrieval systems and users. Semantic noise reduces retrieval accuracy, leads to inefficient query processing, and poses challenges for natural language processing (NLP) systems, often resulting in user frustration and diminished trust in information retrieval (IR) systems. Semantic noise disrupts the clarity and accuracy of information representation and retrieval, leading to inefficiencies, misinterpretations, and user dissatisfaction. Addressing and mitigating semantic noise requires advanced techniques in natural language understanding, such as contextual analysis, semantic search, semantic modeling, and machine learning. These techniques ensure that information retrieval (IR) systems can effectively bridge the gap between user intent and stored data. These findings underscore the critical need for precision in language, standardized terminology, and context-aware approaches to minimize semantic noise and enhance the reliability of information representation and retrieval.

SWETWOL is implemented in the framework of two-level model. It contains a 48,000 item lexicon and a full inflectional description. Special attention was paid to the design of a computational analysis of productive Swedish compounds. Recall (coverage) and precision of SWETWOL meet high standards. SWETWOL has been extensively tested on various types of texts. ("svensk" A UTR INDEF SG NOM)) ("<<tiger>>" ("tiger" N UTR INDEF SG NOM) ("tiga" V ACT PRES)) All readings are retrieved that are feasible in relation to the lexicon and inflectional description. SWETWOL is capable of generating the base form of inflected forms, most clearly visible in the second reading of the word tiger. SWETWOL is meant to be used as the basic morphological tool in systems for text analysis, information indexing, storage and retrieval, and machine translation. It is an integral part of a parsing system where the other major components are (i) a preprocessor converting text to the format presupposed by SWETWOL, and (ii) the Constraint Grammar Parser CGP , 1990, 1992a,b, Karlsson, Voutilainen, Heikkilä, and Anttila 1991). CGP provides a language-independent formalism and a computer program for morphological disambiguation and surface-syntactic analysis. The sequential set-up of analysis in the present framework is thus: -preprocessing, -morphological analysis by SWETWOL, -local disambiguation (on top of TWOL, or by CGP), -context-sensitive disambiguation, clause-boundary determination, and surface-syntactic analysis by CGP. Local disambiguation and preprocessing are discussed in sections 12, 13. Context-sensitive disambiguation of Swedish is treated in Karlsson (in preparation). The pioneering effort of Swedish computational lexicology and morphology was Allén's Nusvensk frekvensordbok (NFO). Many aspects of computational morphology faced this project, such as text preprocessing, morpheme identity and segmentation, the distinction between homonymy and synonymy, homograph separation, and text type effects on vocabulary composition . One offspring of NFO was Hellberg's (1978) algorithm for Swedish morphology, a precise description of word inflection and derivation, partly also of compounding. Hellberg set up a system of 235 inflectional paradigms, plus a basic vocabulary containing 8,609 lemmas. This vocabulary had a coverage of 90% in the corpus of newspaper text that NFO was based on (Hellberg 1978:12). The algorithm was implemented by M. Eeg-Olofsson. Precise data on coverage in the realm of new texts are not included in . It is somewhat unclear how precise the morphotactic description of compounds was. Hellberg's system was later reimplemented by . This dictionary contained some 7,200 entries (Rankin, ms., p. 18). Rankin (1986:166-) reports considerable overgeneration in the analysis of compounds. E.g., bil 'car' yielded spurious analyses such as "bi-l" (N -abbreviation), and ironiska 'ironic (pl.)' yielded a spurious interpretation "ironi-ska" (N -V). (In this paper, compound boundaries are indicated by a dash, "-", and other morpheme boundaries by a plus sign, "+".) The morphotactic description of compound formation was obviously not strict enough.

Abstract—In this paper, we propose a gene mention recognition system for biomedical literature using Support Vector Machine based on a reformed lexicon. Then we present an ensemble of rule-based post-processing modules, a integrity check module, a boundary check module, an abbreviation resolution module and a name pruning module, to improve the performance further. The newly developed lexicon is composed of uni-indicating and co-indicating words inside gene mention phrases. With the carefully designed lexicon, the ...

We introduce LangHIT, an innovative extension of BM25S that integrates multilingual tokenization, adaptive multilingual stopword filtering, and ultra-fast typo correction using RapidFuzz within the BM25S token matching framework. When BM25S fails to locate a mistyped query token in the corpus, the token is corrected using RapidFuzz before being reintroduced into BM25S for correct token-level matching. LangHIT significantly enhances precision, recall, and computational efficiency in multilingual search, particularly for Indian languages. The code can be found at our githu

Cross-Language Information Retrieval (CLIR) is responsible for retrieving information stored in a language different from the language of the query provided by the user. Some translation methods commonly used in CLIR are Dictionary, Parallel corpora, Comparable corpora, Machine translator, Ontology, and Transitive-based. The query must be translated to the target language, followed by preprocessing and calculating the similarity between the query and all documents in the corpus. The problem is the time and accuracy of query translation. Moreover, the queries are not written as complete sentences according to certain language rules. Stemming, for example, every language has its own method. Indonesian has basic words and affixes in the form of prefixes, suffixes, infixes, and confixes, while English only has suffixes. Stemming takes a long time in text processing. In the Indonesian search engine (SEBI), the provision of cross-language tourism news retrieval is realized using the Googl...

Parallel texts, i.e., texts in one language and their translations to other languages, are very useful nowadays for many applications such as machine translation and multilingual information retrieval. If these texts are aligned in sentence level, for instance, their relevance increases considerably. In this paper we describe some experiments that have being done with Portuguese and English parallel texts using five well known sentence alignment methods. Four corpora were used for testing, achieving 85.89% to 100% of precision.

Followup to the 1997 community note "Thoughts on Regular Expressions for Ethiopic". This document has been prepared for members of the perl-unicode email list to describe by way of example the limitations of working with the Ethiopic script under the UTF-8 support in Perl. It is the intention that this document provides the essential information for regular expression library developers to add support for syllabic scripts to meet the needs and expectations of the user community.

This document was composed in LaTeX and the final iteration of the original file was generated in PostScript format on Wednesday, February 24th 1999, 14:10:56 (EAT) and later converted into PDF format. The document has been previously published under the filename regex992.ps .

The increasing amount of available textual information makes necessary the use of Natural Language Processing (NLP) tools. These tools have to be used on large collections of documents in different languages. But NLP is a complex task that relies on many processes and resources. As a consequence, NLP tools must be both configurable and efficient: specific software architectures must be designed for this purpose. We present in this paper the LIMA multilingual analysis platform, developed at CEA LIST. This configurable platform has been designed to develop NLP based industrial applications while keeping enough flexibility to integrate various processes and resources. This design makes LIMA a linguistic analyzer that can handle languages as different as French, English, German, Arabic or Chinese. Beyond its architecture principles and its capabilities as a linguistic analyzer, LIMA also offers a set of tools dedicated to the test and the evaluation of linguistic modules and to the production and the management of new linguistic resources.