Text Indexing

The recent results of the research in the construction of the electronic dictionary of Serbo-Croatian are presented. This research involves the development of methodological and theoretical principles for the construction of the lexicon for a highly inflective language. The problems that emerge on different levels of language standardization such as graphemic, orthographic, dialectic, inflective and derivational are pointed out.

In this paper we describe the resources and tools for the processing of texts written in Serbian. Most of the resources have been developed within the University of Belgrade NLP group located at the Faculty of Mathematics. The main features of these resources, namely available monolingual and multilingual corpora and various e-dictionaries are briefly described. The use of Intex, the main tool of the NLP group, for the recognition of unknown words, text tagging, building local grammars and disambiguation is outlined.

We present an optimal adaptive algorithm for context queries in tagged content. The queries consist of locating instances of a tag within a context specified by the query using patterns with preorder, ancestor-descendant and proximity operators in the document tree implied by the tagged content. The time taken to resolve a query Q on a document tree T is logarithmic in the size of T , proportional to the size of Q, and to the difficulty of the combination of Q with T , as measured by the minimal size of a certificate of the answer. The performance of the algorithm is no worse than the classical worst-case optimal, while provably better on simpler queries and corpora. More formally, the algorithm runs in time O(δk lg(n/δk)) in the standard RAM model and in time O(δk lg lg min(n, σ)) in the Θ(lg(n))-word RAM model, where k is the number of edges in the query, δ is the minimum number of operations required to certify the answer to the query, n is the number of nodes in the tree, and σ is the number of labels indexed.

Digital games, as a popular technology in youth entertainment, constitute a fast-growing field which has been affecting various aspects of education for several years now. The research project “Lexipaignio” focuses on the development of an innovative and state-of-the-art NLP (Natural Language Processing) environment for the creation of digital educational games for Greek students. A variety of simple and easy-to-play mini-games has been specified aiming to improve students’ linguistic competence by developing a better understanding on various grammatical, morphological and vocabulary related phenomena in general, but also in the context of specific subjects (e.g. geology – geography, biology, etc.). In this paper, the main functionalities of the NLP environment will be presented towards the implementation of mini-games for the Greek language.

A major engineering challenge in statistical machine translation systems is the efficient representation of extremely large translation rulesets. In phrase-based models, this problem can be addressed by storing the training data in memory and using a suffix array as an efficient index to quickly lookup and extract rules on the fly. Hierarchical phrasebased translation introduces the added wrinkle of source phrases with gaps. Lookup algorithms used for contiguous phrases no longer apply and the best approximate pattern matching algorithms are much too slow, taking several minutes per sentence. We describe new lookup algorithms for hierarchical phrase-based translation that reduce the empirical computation time by nearly two orders of magnitude, making on-the-fly lookup feasible for source phrases with gaps.

The unidirectional FM index was introduced by Ferragina and Manzini in 2000 and allows to search a pattern in the index in one direction. The bidirectional FM index (2FM) was introduced by Lam et al. in 2009. It allows to search for a pattern by extending an infix of the pattern arbitrarily to the left or right. The method of Lam et al. can conduct one step in time O(σ) while needing space O(σ • n) using constant time rank queries on bit vectors. Schnattinger and colleagues improved this time to O(log σ) while using O(log σ • n) bits of space for both, the FM and 2FM index. This is achieved by the use of wavelet trees. In this paper we introduce a new method for conducting an exact search in a uniand bidirectional FM index in O(1) time per step while using O(log σ •n)+o(log σ •σ •n) bits of space. This is done by replacing the wavelet tree by a new data structure, the Enhanced Prefixsum Rank dictionary (EPR-dictionary). To our knowledge this is the first constant time method for a search step in 2FM indices and a space improvement for FM indices. We implemented this method in the SeqAn C++ library and experimentally validated our theoretical results. In addition we compared our implementation with other freely available implementations of bidirectional indices and show that we are between ≈ 2.7 -4.6 times faster. This will have a large impact for many bioinformatics applications that rely on practical implementations of (2)FM indices e.g. for read mapping.

We introduce a new method for conducting an exact search in a uni- and bidirectional FM index in $\mathcal{O}(1)$ time per step while using $\mathcal{O}(\log \sigma \cdot n) + o(\log \sigma \cdot \sigma \cdot n)$ bits of space. This is done by replacing the wavelet tree by a new data structure, the \emph{Enhanced Prefixsum Rank dictionary} (EPR-dictionary). To our knowledge this is the first constant time method for a search step in 2FM indices and a space improvement for FM indices. We implemented this method in the SeqAn C++ library and experimentally validated our theoretical results. In addition we compared our implementation with other freely available implementations of bidirectional indices and show that we are between $\approx 2.7-4.6$ times faster. This will have a large impact for many bioinformatics applications that rely on practical implementations of (2)FM indices e.g. for read mapping.

For [Formula: see text], define [Formula: see text] as the set of integers [Formula: see text]. Given an integer [Formula: see text] and a string [Formula: see text] of length [Formula: see text] over [Formula: see text], we count the number of times that each one of the [Formula: see text] distinct strings of length [Formula: see text] over [Formula: see text] occurs as a subsequence of [Formula: see text]. Our algorithm makes only one scan of [Formula: see text] and solves the problem in time complexity [Formula: see text] and space complexity [Formula: see text]. These are very close to best possible.

Suffix trees and suffix arrays are two of the most widely used data structures for text indexing. Each uses linear space and can be constructed in linear time for polynomially sized alphabets. However, when it comes to answering queries with worst-case deterministic time bounds, the prior does so in O(m log |Σ|) time, where m is the query size, |Σ| is the alphabet size, and the latter does so in O(m + log n) time, where n is the text size. If one wants to output all appearances of the query, an additive cost of O(occ) time is sufficient, where occ is the size of the output. Notice that it is possible to obtain a worst case, deterministic query time of O(m) but at the cost of super-linear construction time or space usage. We propose a novel way of combining the two into, what we call, a suffix tray. The space and construction time remain linear and the query time improves to O(m + log |Σ|) for integer alphabets from a linear range, i.e. Σ ⊂ {1, • • • , cn}, for an arbitrary constant c. The construction and query are deterministic. Here also an additive O(occ) time is sufficient if one desires to output all appearances of the query. We also consider the online version of indexing, where the text arrives online, one character at a time, and indexing queries are answered in tandem. In this variant we create a cross between a suffix tree and a suffix list (a dynamic variant of suffix array) to be called a suffix trist; it supports queries in O(m + log |Σ|) time. The suffix trist also uses linear space. Furthermore, if there exists an online construction for a linear-space suffix tree such that the cost of adding a character is worst-case deterministic f (n, |Σ|) (n is the size of the current text), then one can further update the suffix trist in O(f (n, |Σ|) + log |Σ|) time. The best currently known worstcase deterministic bound for f (n, |Σ|) is O(log n) time.

We introduce the first index that can be built in o(n) time for a text of length n, and can also be queried in o(q) time for a pattern of length q. On an alphabet of size σ, our index uses O(n √ log n log σ) bits, is built in O(n((log log n) 2 + √ log σ)/ log σ n) deterministic

We propose a new method of extracting texts related to a given keyword from Web pages collected by a search engine. By combining structural pattern matching and text classification, texts related to a given keyword such as reputations of a given restaurant can be extracted automatically from Web pages in unfixed sites, which is impossible by conventional wrappers. According to our cross validation results on extracting reputations of a given Ramen shop from Web pages collected by a search engine, our method achieved 79.3% precision and 56.6% recall by allowing acceptable errors.

The SPIRIT search engine provides a test bed for the development of web search technology that is specialised for access to geographical information. Major components include the user interface, geographical ontology, maintenance and retrieval functions for a test collection of web documents, textual and spatial indexes, relevance ranking and metadata extraction. Here we summarise the functionality and interaction between these components before focusing on the design of the geo-ontology and the development of spatio-textual indexing methods. The geo-ontology supports functionality for disambiguation, query expansion, relevance ranking and metadata extraction. Geographical place names are accompanied by multiple geometric footprints and qualitative spatial relationships. Spatial indexing of documents has been integrated with text indexing through the use of spatio-textual keys in which terms are concatenated with spatial cells to which they relate. Preliminary experiments demonstrate considerable performance benefits when compared with pure text indexing and with text indexing followed by a spatial filtering stage.

Stream computing research is moving from terascale to petascale levels. It aims to rapidly analyze data as it streams in from many sources and make decisions with high speed and accuracy in fields as diverse as security surveillance and financial services including stock trading. We specifically consider real-time text indexing and search with high input data rates (10 GB/s or more) along with small index ageoff(expiry) time. This makes it necessary to have maximal indexing rates for large volumes of data as well as minimal latency for indexing (time between start of indexing for a document and its availability for search) while maintaining very-low search response time. In addition, future massively parallel architectures with storage class memories will enable high speed in-memory real-time indexing, where index can be completely stored in a high capacity storage class memory. In this paper, we present the design of distributed datastructures and distributed real-time text indexing algorithm for parallel systems having large (thousands to hundred thousand) number of cores/processors, while simultaneously providing acceptable search performance [1]. The inherent trade-offs involved in index space, indexing throughput and search response time make this problem particularly challenging. Our algorithm uses group-based index construction and leverages novel index data structures that reduce load imbalance and make text indexing and merge process more scalable and efficient. We show analytically that the asymptotic parallel time complexity of our distributed indexing algorithm, is at least Ω(log(P)) factor better than typical indexing approaches, where P is the number of indexing nodes in a group. We further demonstrate the performance and scalability of our distributed indexing algorithm, on an MPP architecture (Blue Gene/L 1) using actual IBM intranet data. We achieved high indexing throughput of around 312 GB/min on an 8K node Blue Gene/L machine. In comparison with parallel indexing implemented using typical approaches like CLucene 2 , this is 3×-7× better. To the best of our knowledge, this is the first published result on indexing throughput at such a large scale, with sustained search performance. We further show that our approach is scalable 1. http://www.research.ibm.com/bluegene 2. http://www.sourceforge.net/projects/clucene to 128K nodes, giving an estimated indexing throughput of 5 T B/min. We also achieved indexing latency that is around 10× better than typical indexing approaches.

YADDA framework facilitates information exchange between digital document repositories. YaddaWeb, its web-based interface, provides browse and search functionalities. Content providers use DeskLight application to add or modify metadata and content. Internally, YADDA contains flexible repository aggregation mechanisms, multiple hierarchy support and full-text indexing capabilities. YADDA framework is an excellent solution for Open Access paradigm of content exchange. Migration of the Mathematical Collection of Polish Virtual Library of Science to YADDA is currently being prepared.

YADDA2 is an open software platform which facilitates creation of digital library applications. It consists of versatile building blocks providing, among others: storage, relational and full-text indexing, process management, and asynchronous communication. Its loosely-coupled serviceoriented architecture enables deployment of highly-scalable, distributed systems.

In this paper we investigate some properties and algorithms related to a text sparsification technique based on the identification of local maxima in the given string. As the number of local maxima depends on the order assigned to the alphabet symbols, we first consider the case in which the order can be chosen in an arbitrary way. We show that looking for an order that minimizes the number of local maxima in the given text string is an Np-hard problem. Then, we consider the case in which the order is fixed a priori. Even though the order is not necessarily optimal, we can exploit the property that the average number of local maxima induced by the order in an arbitrary text is approximately one third of the text length. In particular, we describe how to iterate the process of selecting the local maxima by one or more iterations, so as to obtain a sparsified text. We show how to use this technique to filter the access to unstructured texts, which appear to have no natural division in words. Finally, we experimentally show that our approach can be successfully used in order to create a space efficient index for searching sufficiently long patterns in a DNA sequence as quickly as a full index.

In theory, speech recognition technology can make any spoken words in video or audio media usable for text indexing, search and retrieval. This article describes the News-on-Demand application created within the Informedia TM Digital Video Library project and discusses how speech recognition is used in transcript creation from video, alignment with closed-captioned transcripts, audio paragraph segmentation and a spoken query interface. Speech recognition accuracy varies dramatically depending on the quality and type of data used. Informal information retrieval test show that reasonable recall and precision can be obtained with only moderate speech recognition accuracy. 1.1. Component Technologies There are three broad categories of technologies we can bring This material is based upon work supported by the National Science Foundation under Cooperative Agreement No. IRI-9411299. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Many applications depend on efficient management of large sets of distinct strings in memory. For example, during index construction for text databases a record is held for each distinct word in the text, containing the word itself and information such as counters. We propose a new data structure, the burst trie, that has significant advantages over existing options for such applications: it uses about the same memory as a binary search tree; it is as fast as a trie; and, while not as fast as a hash table, a burst trie maintains the strings in sorted or near-sorted order. In this paper we describe burst tries and explore the parameters that govern their performance. We experimentally determine good choices of parameters, and compare burst tries to other structures used for the same task, with a variety of data sets. These experiments show that the burst trie is particularly effective for the skewed frequency distributions common in text collections, and dramatically outperforms all ...

In this article we present the MIRTO platform-under development at the University Stendhal of Grenoble-and how it addresses common flaws of CALL software. This platform led to another project: the creation of a pedagogically indexed text base. We introduce here the notion of pedagogical indexation, and confront the particular case of pedagogical indexation for language learning with the existing pedagogical resource description standards, before proposing leads towards the implementation of the former.

This communication is meant to present our project of pedagogically indexed text base. After introducing the notion of pedagogical indexation, which needs to be articulated around the teachers needs, we explain to which extent existing pedagogical resource description standards are inadequate to achieve pedagogical indexation for raw texts for language teaching. We then introduce the notions underlying the creation of our prototype through a fictional study case. The notions, which we introduce, are meant to be able to take into account the pedagogical context of the potential use of the text when giving a value to its pedagogical properties. These notions include text facet, view of a text according to a facet for a given pedagogical context, homogeneous text collection and text visualization. Through the use of these notions our prototype will allow teachers to query for texts depending on pedagogical criteria and provide them with assistance for the actual choice of the text. Com...

Defining pedagogical indexation of texts for language learning as an indexation allowing users to query for texts in order to use them in language teaching requires to take into account the influence of the properties of the teaching situation we define as "pedagogical context". We propose to justify the notions of prisms and facets on which our model rely through the description of material selection in the task of planing a language class as an adaptation of Yinger's model of planing. This interpretation of Yinger's model is closely intertwined with the elaboration of the notion of pedagogical context. The latter provides sounder bases on which to build our model on. This resulted in improvements in the potentialities of the model compared to its first published version.

In this article we present the MIRTO platform-under development at the University Stendhal of Grenoble-and how it addresses common flaws of CALL software. This platform led to another project: the creation of a pedagogically indexed text base. We introduce here the notion of pedagogical indexation, and confront the particular case of pedagogical indexation for language learning with the existing pedagogical resource description standards, before proposing leads towards the implementation of the former.

In today's world e-learning is one of the popular modes of learning and video lectures are more prominent in keeping learners engaged with course. Internet enabled to keep a large number of video lectures on-line. To search for a required topic or subtopic from this huge video repository is becoming very tedious. One way to search for a particular topic is through keyword based search and it is based on extraction of text content available in lecture video files and to achieve it one has to maintain metadata. To maintain the metadata associated with video the frames of video containing text are required to be processed. As the video contains multiple frames per second it is not required to consider each and every frame. So, the frames containing distinct content called key frames need to be identified. The identification of key frames plays crucial role in the lecture video searching process. In this paper, different techniques for key frame identification are experimentally tested and results were compared.

Pesquisa de abordagem qualitativa de carater descritivo-exploratorio, em que foram entrevistados 13 profissionais da equipe de enfermagem, atraves de entrevistas semiestruturadas, bem como observacoes sistematicas nao participantes. Observou-se que a tecnica utilizada para a realizacao dos curativos estava inadequada. Apesar dos profissionais possuirem conhecimento adequado quanto a tecnica asseptica, na pratica, nao atentam para a forma correta de realizacao do procedimento de limpeza ou por vezes ignoram, nao utilizando corretamente os materiais estereis. Tambem ficou evidenciada a falta do uso de protocolo para a realizacao da troca de curativos. Fica evidenciada a necessidade da realizacao de uma educacao continuada com os profissionais da equipe de enfermagem da instituicao. Como estrategia, pode-se incorporar os novos conhecimentos, tecnologias e alternativas disponiveis para utilizacao em suas praticas, deixando claras as responsabilidades e como deve ser a atuacao de cada pr...

The NZDL aims to impose structure on anarchic and uncataloged repositories of information, providing information consumers with effective tools to locate and peruse what they need. Our goal is to produce an easy-to-use digital library system that runs on inexpensive computers at information providers' own sites and offers an information service that information providers themselves maintain. New Zealand's geographical isolation magnifies the benefits of networked digital libraries, in terms of both cost and timeliness of access to information. We are collaborating with the Medoc project in Germany to provide local indexes to German language technical-reports, and with the J Biological Chemistry in the United States to field-test novel browsing techniques. The project rests on five basic planks. First, we avoid manual processing of source material, and avoid making assumptions about the document repositories from which it is collected-e.g. we do not require bibliographic metadata. Second, access is via a full-text index of the entire contents of each document, rather than document surrogates. Third, we are concerned with user interface aspects and the real needs of library users. Fourth, our systems must operate in geographically remote locations with high Internet costs-an environment in which the benefits of networked library technology are especially striking. Finally, we aim to produce a library scheme that operates on small, inexpensive, servers.

A repetitive sequence collection is one where portions of a base sequence of length n are repeated many times with small variations, forming a collection of total length N . Examples of such collections are version control data and genome sequences of individuals, where the differences can be expressed by lists of basic edit operations. Flexible and efficient data analysis on a such typically huge collection is plausible using suffix trees. However, suffix tree occupies O(N log N ) bits, which very soon inhibits in-memory analyses. Recent advances in full-text selfindexing reduce the space of suffix tree to O(N log σ) bits, where σ is the alphabet size. In practice, the space reduction is more than 10-fold, for example on suffix tree of Human Genome. However, this reduction factor remains constant when more sequences are added to the collection. We develop a new family of self-indexes suited for the repetitive sequence collection setting. Their expected space requirement depends only on the length n of the base sequence and the number s of variations in its repeated copies. That is, the space reduction factor is no longer constant, but depends on N/n. We believe the structures developed in this work will provide a fundamental basis for storage and retrieval of individual genomes as they become available due to rapid progress in the sequencing technologies.

Temporal graphs represent binary relationships that change along time. They can model the dynamism of, for example, social and communication networks. Temporal graphs are defined as sets of contacts that are edges tagged with the temporal intervals when they are active. This work explores the use of the Compressed Suffix Array (CSA), a well-known compact and self-indexed data structure in the area of text indexing, to represent large temporal graphs. The new structure, called Temporal Graph CSA (TGCSA), is experimentally compared with the most competitive compact data structures in the state-of-the-art, namely, EdgeLog and CET. The experimental results show that TGCSA obtains a good space-time trade-off. It uses a reasonable space and is efficient for solving complex temporal queries. Furthermore, TGCSA has wider expressive capabilities than EdgeLog and CET, because it is able to represent temporal graphs where contacts on an edge can temporally overlap.

Prior work inspired by compression algorithms has described how the Burrows Wheeler Transform can be used to create a distance measure for bioinformatics problems. We describe issues with this approach that were not widely known, and introduce our new Burrows Wheeler Markov Distance (BWMD) as an alternative. The BWMD avoids the shortcomings of earlier efforts, and allows us to tackle problems in variable length DNA sequence clustering.BWMD is also more adaptable to other domains, which we demonstrate on malware classification tasks. Unlike other compression-based distance metrics known to us, BWMD works by embedding sequences into a fixed-length feature vector. This allows us to provide significantly improved clustering performance on larger malware corpora, a weakness of prior methods.

It seems that there is no limit to the amount of data we need to store in our computers and also send these data to our friends and colleagues. For this purpose people tend to store a lot of files inside their storage. When the storage nears it's limit, then they try to reduce those files size to minimum by using data compression software's. Compression aims to represent an input data with least number of bits. In this paper we describe the Burrows-Wheeler Transform (BWT), a data compression technique which is the basis of some of the best compressor available today.

Consider an input text string T ≡ T [1, N ] drawn from an unbounded alphabet, so text positions can be accessed using comparisons. We study partial computation in suffix-based problems for Data Compression and Text Indexing such as • retrieve any segment of K ≤ N consecutive symbols from the Burrows-Wheeler transform of T , which is at the heart of the bzip2 family of text compressors, and • retrieve any chunk of K ≤ N consecutive entries of the Suffix Array or the Suffix Tree, two popular Text Indexing data structures for T. Prior literature would take O(N log N) comparisons (and time) to solve these problems by solving the total problem of building the entire Burrows-Wheeler transform or Text Index for T , and performing a post-processing to single out the wanted portion. The technical challenge is that the suffixes of interest are potentially of size O(KN) and overlap in intricate ways: we have to use structural properties of these overlaps to avoid rescanning them repeatedly. We introduce a novel adaptive approach to partial computational problems above, and solve both the partial problems in O(K log K + N) comparisons and time, improving the best known running times of O(N log N) for K = o(N). These partial-computation problems are intimately related since they share a common bottleneck: the suffix multi-selection problem, which is to output the suffixes of rank r 1 , r 2 ,. .. , r K under the lexicographic order, where r 1 < r 2 < • • • < r K , r i ∈ [1, N ]. Special cases of this problem are well known: K = N is the suffix sorting problem that is the workhorse in Stringology with hundreds of applications, and K = 1 is the recently studied suffix selection. We show that suffix multi-selection can be solved in Θ   N log N − K j=0 ∆ j log ∆ j + N   time and comparisons, where r 0 = 0, r K+1 = N + 1, and ∆ j = r j+1 − r j for 0 ≤ j ≤ K. This is asymptotically optimal, and also matches the bound in [7] for multi-selection on atomic elements (not suffixes). Matching the bound known for atomic elements for strings is a long running theme and challenge from 70's, which we achieve for the suffix multi-selection problem. The partial suffix problems as well as the suffix multi-selection problem have many applications.

Data sets are growing rapidly and there is an attendant need for tools that facilitate human analysis of them in a timely manner. To help meet this need, column-oriented databases (or "column stores") have come into wide use because of their low latency on analytic workloads. Column stores use a number of techniques to produce these dramatic performance techniques, including the ability to perform operations directly on compressed data. In this paper, we describe how the Tableau Data Engine (an internally developed column store) leverages a number of compression techniques to improve query performance. The approach is simpler than existing systems for operating on compressed data and more unified, removing the necessity for custom data access mechanisms. The approach also uses some novel metadata extraction techniques to improve the choices made by the system's run-time optimizer.

Data sets are growing rapidly and there is an attendant need for tools that facilitate human analysis of them in a timely manner. To help meet this need, column-oriented databases (or "column stores") have come into wide use because of their low latency on analytic workloads. Column stores use a number of techniques to produce these dramatic performance techniques, including the ability to perform operations directly on compressed data. In this paper, we describe how the Tableau Data Engine (an internally developed column store) leverages a number of compression techniques to improve query performance. The approach is simpler than existing systems for operating on compressed data and more unified, removing the necessity for custom data access mechanisms. The approach also uses some novel metadata extraction techniques to improve the choices made by the system's run-time optimizer.

The SPIRIT search engine provides a test bed for the development of web search technology that is specialised for access to geographical information. Major components include the user interface, geographical ontology, maintenance and retrieval functions for a test collection of web documents, textual and spatial indexes, relevance ranking and metadata extraction. Here we summarise the functionality and interaction between these components before focusing on the design of the geo-ontology and the development of spatio-textual indexing methods. The geo-ontology supports functionality for disambiguation, query expansion, relevance ranking and metadata extraction. Geographical place names are accompanied by multiple geometric footprints and qualitative spatial relationships. Spatial indexing of documents has been integrated with text indexing through the use of spatio-textual keys in which terms are concatenated with spatial cells to which they relate. Preliminary experiments demonstrate considerable performance benefits when compared with pure text indexing and with text indexing followed by a spatial filtering stage.

Indexing very large collections of strings, such as those produced by the widespread next generation sequencing technologies, heavily relies on multistring generalization of the Burrows-Wheeler Transform (BWT): large requirements of in-memory approaches have stimulated recent developments on external memory algorithms. The related problem of computing the Longest Common Prefix (LCP) array of a set of strings is instrumental to compute the suffix-prefix overlaps among strings, which is an essential step for many genome assembly algorithms. In a previous paper, we presented an in-memory divide-and-conquer method for building the BWT and LCP where we merge partial BWTs with a forward approach to sort suffixes. In this paper, we propose an alternative backward strategy to develop an external memory method to simultaneously build the BWT and the LCP array on a collection of m strings of different lengths. The algorithm over a set of strings having constant length k has O(mkl) time and I/O volume, using O(k + m) main memory, where l is the maximum value in the LCP array.

Indexing of static and dynamic sets is fundamental to a large set of applications such as information retrieval and caching. Denoting the characteristic vector of the set by B, we consider the problem of encoding sets and multisets to support approximate versions of the operations rank(i) (i.e., computing j≤i B[j]) and select(i) (i.e., finding min{p | rank(p) ≥ i}) queries. We study multiple types of approximations (allowing an error in the query or the result) and present lower bounds and succinct data structures for several variants of the problem. We also extend our model to sliding windows, in which we process a stream of elements and compute suffix sums. This is a generalization of the window summation problem that allows the user to specify the window size at query time. Here, we provide an algorithm that supports updates and queries in constant time while requiring just (1 + o(1)) factor more space than the fixed-window summation algorithms.

We consider the range mode problem where given a sequence and a query range in it, we want to find items with maximum frequency in the range. We give time- and space- efficient algorithms for this problem. Our algorithms are efficient for small maximum frequency cases. We also consider a natural generalization of the problem: the range mode enumeration problem, for which there has been no known efficient algorithms. Our algorithms have query time complexities which is linear to the output size plus small terms.

This paper considers various flavors of the following online problem: preprocess a text or collection of strings, so that given a query string p, all matches of p with the text can be reported quickly. In this paper we consider matches in which a bounded number of mismatches are allowed, or in which a bounded number of "don't care" characters are allowed. The specific problems we look at are: indexing, in which there is a single text t, and we seek locations where p matches a substring of t; dictionary queries, in which a collection of strings is given upfront, and we seek those strings which match p in their entirety; and dictionary matching, in which a collection of strings is given upfront, and we seek those substrings of a (long) p which match an original string in its entirety. These are all instances of an all-to-all matching problem, for which we provide a single solution. The performance bounds all have a similar character. For example, for the indexing problem with n = |t| and m = |p|, the query time for k substitutions is O(m + (c 1 log n) k k! + # matches), with a data structure of size O(n (c 2 log n) k k!) and a preprocessing time of O(n (c 2 log n) k k!), where c1, c2 > 1 are constants. The deterministic preprocessing assumes a weakly nonuniform RAM model; this assumption is not needed if randomization is used in the preprocessing.

Suffix trees and suffix arrays are two of the most widely used data structures for text indexing. Each uses linear space and can be constructed in linear time for polynomially sized alphabets. However, when it comes to answering queries with worst-case deterministic time bounds, the prior does so in O(m log |Σ|) time, where m is the query size, |Σ| is the alphabet size, and the latter does so in O(m + log n) time, where n is the text size. If one wants to output all appearances of the query, an additive cost of O(occ) time is sufficient, where occ is the size of the output. Notice that it is possible to obtain a worst case, deterministic query time of O(m) but at the cost of super-linear construction time or space usage. We propose a novel way of combining the two into, what we call, a suffix tray. The space and construction time remain linear and the query time improves to O(m + log |Σ|) for integer alphabets from a linear range, i.e. Σ ⊂ {1, • • • , cn}, for an arbitrary constant c. The construction and query are deterministic. Here also an additive O(occ) time is sufficient if one desires to output all appearances of the query. We also consider the online version of indexing, where the text arrives online, one character at a time, and indexing queries are answered in tandem. In this variant we create a cross between a suffix tree and a suffix list (a dynamic variant of suffix array) to be called a suffix trist; it supports queries in O(m + log |Σ|) time. The suffix trist also uses linear space. Furthermore, if there exists an online construction for a linear-space suffix tree such that the cost of adding a character is worst-case deterministic f (n, |Σ|) (n is the size of the current text), then one can further update the suffix trist in O(f (n, |Σ|) + log |Σ|) time. The best currently known worstcase deterministic bound for f (n, |Σ|) is O(log n) time.

On-demand string sorting is the problem of preprocessing a set of strings to allow subsequent queries for finding the k lexicographically smallest strings (and afterward the next k etc.) This on-demand variant strongly resembles the search engine queries which give you the best k-ranked pages recurringly. We present a data structure that supports this in O(n) preprocessing time, where n is the number of strings, and answer queries in O(log n) time. There is also a cost of O(N) time amortized over all operations, where N is the total length of the strings. Our data structure is a heap of strings, which supports heapify and delete-mins. As it turns out, implementing a full heap with all operations is not that simple. For the sake of completeness, we propose a heap with full operations based on balanced indexing trees that supports the heap operations in optimal times.

This paper considers various flavors of the following online problem: preprocess a text or collection of strings, so that given a query string p, all matches of p with the text can be reported quickly. In this paper we consider matches in which a bounded number of mismatches are allowed, or in which a bounded number of "don't care" characters are allowed. The specific problems we look at are: indexing, in which there is a single text t, and we seek locations where p matches a substring of t; dictionary queries, in which a collection of strings is given upfront, and we seek those strings which match p in their entirety; and dictionary matching, in which a collection of strings is given upfront, and we seek those substrings of a (long) p which match an original string in its entirety. These are all instances of an all-to-all matching problem, for which we provide a single solution. The performance bounds all have a similar character. For example, for the indexing problem with n = |t| and m = |p|, the query time for k substitutions is O(m + (c 1 log n) k k! + # matches), with a data structure of size O(n (c 2 log n) k k!) and a preprocessing time of O(n (c 2 log n) k k!), where c1, c2 > 1 are constants. The deterministic preprocessing assumes a weakly nonuniform RAM model; this assumption is not needed if randomization is used in the preprocessing.

In this chapter we discuss various bitmap index technologies for efficient query processing in data warehousing applications. We review the existing literature and organize the technology into three categories, namely bitmap encoding, compression and binning. We introduce an efficient bitmap compression algorithm and examine the space and time complexity of the compressed bitmap index on large data sets from real applications. According to the conventional wisdom, bitmap indices are only efficient for low-cardinality attributes. However, we show that the compressed bitmap indices are also efficient for high-cardinality attributes. Timing results demonstrate that the bitmap indices significantly outperform the projection index, which is often considered to be the most efficient access method for multi-dimensional queries. Finally, we review the bitmap index technology currently supported by commonly used commercial database systems and discuss open issues for future research and development.

Bitmap indexes are known as the most effective indexing methods for range queries on append-only data, especially for low cardinality attributes. Recently, bitmap indexes were also shown to be just as effective for high cardinality attributes when certain compression methods are applied. There are many different bitmap indexes in the literature but no definite comparison among them has been made, largely because there is no accurate prediction of their index sizes and search time. This paper presents a systematic evaluation of two large subsets of compressed bitmap indexes that use multi-component and multi-level encodings. We combine extensive analyses with ample experimental results to confirm them, whereas earlier studies of these indexes are either empirical or for uncompressed indexes only. Our analyses provide highly accurate predictions that agree with test measurements. These analyses not only identify the best methods in terms of index size and query processing cost, but also reveal new ways of using multi-level methods that significantly improve their performance. Using the best parameters obtained through analyses, we produce three two-level indexes with the optimal computational complexity. Furthermore, the fastest two-level indexes are predicted and observed to be 5 to 10 times faster than other well-known indexes.

This article presents a survey of techniques for ranking results in search engines, with emphasis on link-based ranking methods and the PageRank algorithm. The problem of selecting, in relation to a user search query, the most relevant documents from an unstructured source such as the WWW is discussed in detail. The need for extending classical information retrieval techniques such as boolean searching and vector space models with link-based ranking methods is demonstrated. The PageRank algorithm is introduced, and its numerical and spectral properties are discussed. The article concludes with an alternative means of computing PageRank, along with some example applications of this new method.

A Scalable Distributed Data Structure (SDDS) allows to store a large scalable file over a distributed RAM. The file scales up transparently for the application over the nodes of a multicomputer, e.g., a local network of PCs. The prototype system termed SDDS 2000, designed by CERIA experiments with this technology for Wintel multicomputers. The application may manipulate data much faster than on local disks. We present the functions we have added to SDDS-2000 to backup the RAM on each storage node onto the local disk. Our goal is to write only the data that has changed since the last backup. We experiment for this purpose with the algebraic signatures. We present our architecture and design choices. Performance measures validate our implementation. It is now available for the noncommercial use as part of new version of our prototype termed SDDS-2002.

In this paper, we present simple and area efficient VLSI architectures for HufFman coding, an industrial standard proposed by MPEG, JPEG, and others. We use a memory of size O(nlogn) bits to store a HuEman code tree, where n is the number of symbols. This storage scheme supports real-time encoding and decoding. In addition, few simple arithmetic operations are performed on the chip for encoding and decoding. Based on our scheme, we show a design for &bit symbols. The proposed design requires 256 X 9 and 64 X U-bit memory modules to process 8-bit symbols. The chip occupies a silicon a m of 3.5 X 3.5 mm' using 1.2 micron CMOSN standard library cells. Compared with the known parallel implementation in [I] which requires upto 65536 PE's, the proposed architecture leads to a single PE design. It requires significantly less area than the known single PE design in 121. Different Huffknan codes can be stored by changing the contents of the memory, without changing the design.

This working notes describe the runs and results obtained by the LIG at ImageCLEFphoto 2008. The submitted runs are: two runs (text only and text+image) without diversification on classes, and two runs (text only and text+image) with class diversification were submitted. The text retrieval is based on language model of Information Retrieval, and the image part is processed using RGB histograms on 9 image blocks with a similarity value based on Jeffrey divergence. Results using text+image are obtained by a linear combination of normalized results on text and image. The diversification is based on clusters, according to the cluster given in the queries. When the cluster name is not directly extracted from the images (like city or country), we apply a visual clustering. Not surprisingly, the cluster recall at 20 (i.e., cr(20)) results are higher for the runs that include diversification. On the other hand, the precision at 20 and the mean average precision results are higher without diversification on our runs, for both text only and image+text results.

Di Yogyakarta masih banyak dapat ditemukan naskah-naskah kuno yang merupakan warisan budaya yang tak ternilai harganya. Maka Apabila naskah-naskah tersebut dapat dikonversikan ke dalam format digital, akan banyak manfaat yang bisa diraih. Penelitian ini mencoba untuk membuat sebuah prototipe sistem pengenalan citra dokumen sastra Jawa. Data masukan untuk sistem diperoleh dari hasil pembacaan dokumen sastra Jawa dengan alat optis, yang kemudian disimpan sebagai file gambar dengan format *.jpg. Selanjutnya dengan mempergunakan berbagai metode untuk pengolahan citra, diperoleh citra karakter Jawa pembentuk citra dokumen masukan. Dengan mempergunakan histogram untuk piksel yang berwarna hitam diperoleh ciri untuk setiap karakter. Ciri ini kemudian disimpan dalam basis data karakter Jawa, untuk kemudian ciri ini dipakai dalam pencocokan ciri apabila terdapat masukan karakter Jawa. Dengan mempergunakan jarak Euclidean akan diperoleh nama-nama Latin setiap karakter Jawa pembentuk citra dokumen. Dari hasil penelitian menunjukkan bahwa untuk mengenal citra dokumen sastra Jawa, dalam hal ini dokumen "Menak Sorangan I, Bab I, Halaman 3 dan 4" diperoleh prosentase keberhasilan pengenalan sebesar 86.53%. Maka dapat disimpulkan bahwa pemilihan metode-metode yang dipergunakan dalam tahap pengenalan citra dokumen pada penelitian ini relatif sudah baik.

The run-length compressed Burrows-Wheeler transform (RLBWT) used in conjunction with the backward search introduced in the FM index is the centerpiece of most compressed indexes working on highly-repetitive data sets like biological sequences. Compared to grammar indexes, the size of the RLBWT is often much bigger, but queries like counting the occurrences of long patterns can be done much faster than on any existing grammar index so far. In this paper, we combine the virtues of a grammar with the RLBWT by building the RLBWT on top of a special grammar based on induced suffix sorting. Our experiments reveal that our hybrid approach outperforms the classic RLBWT with respect to the index sizes, and with respect to query times on biological data sets for sufficiently long patterns.

Let S and S be two strings, having the same length, over a totally-ordered alphabet. We consider the following two variants of string matching. Parameterized Matching: The characters of S and S are partitioned into static characters and parameterized characters. The strings are a parameterized match iff the static characters match exactly, and there exists a one-to-one function which renames the parameterized characters in S to those in S. Order-Preserving Matching: The strings are an order-preserving match iff for any two integers i, j ∈ [1, |S|], S[i] S[j] ⇐⇒ S [i] S [j], where denotes the precedence order of the alphabet. Let P be a collection of d patterns {P 1 , P 2 ,. .. , P d } of total length n characters, which are chosen from a totally-ordered alphabet Σ. Given a text T , also over Σ, we consider the dictionary indexing problem under the above definitions of string matching. Specifically, the task is to index P, such that we can report all positions j (called occurrences) ...