Word Segmentation

The paper presents the development of first publically available Urdu N-grams extracted from different books. For the best representation of Ngrams, large amount of Urdu corpus is collected from books covering different domains. The automatic cleaning of 37 million words corpus is discussed. The domain-wise N-grams are extracted which can be used in different Natural Language Processing and Information Retrieval applications.

Writing systems used in Southeast Asia, such as Mon, Burmese, Khmer, and Thai scripts are phonogramic systems of Indic origin, traced back to Brāhmī script in BC3c which was designed for an Indo‐Aryan language, Prākrit. It is inconceivable that a writing system was transmitted and accepted without being accompanied by the language it notates. The script of Indic origin must have been transmitted to Southeast Asia, conveying sounds of Aryan languages.

The ethnic groups in Southeast Asia who accepted an Indic script early are thought to be Pyu, Mon, Khmer, Cham, and Malay (in its broad sense). Keeping its original use to notate Aryan languages, they also began to divert it to notate their own languages respectively. The sound systems of their languages were fairly different from those of Prākrit (and also of Sanskrit). The inventory of glyphs of the script originally designed for Prākrit was insufficient for most of their languages. Under the situation, there are two options for the ‘phonogramic policy’ that the speakers of such a language could take.

1. Notating sounds of the target language within the framework of the accepted writing system, and
2. Accommodating the writing system to the target language by inventing new glyphs, reinterpreting glyphs, and introducing new combinations of glyphs.

In this talk, I will overview the enterprise of enriching notations sought in mainland Southeast Indic scripts, particularly Burmese and Thai scripts.

A new method for Chinese word segmentation named Conditional F&BMM (Forward and Backward Maximal Matching) which incorporates both bigram statistics (i.e., mutual information and difference of t-test between Chinese characters) and linguistic rules for ambiguity resolution is proposed in this paper. The key characteristics of this model are the use of: (i) statistics which can be automatically derived from any raw corpus, (ii) a rule base for disambiguation with consistency and controlled size to be built up in a systematic way. In Chinese, there are no delimiters, such as spacing in English, to explicitly indicate boundaries between words. Chinese word segmentation, viewed as the first step in any Chinese information processing system, has been intensively studied by the Chinese language computing community in the last decade. Unfortunately, a satisfactory segmentation procedure remains elusive. Ambiguity is one of two main obstacles to progress in Chinese word segmentation research (another is Unknown Word) [I]. We have to face two kinds of fundamental ambiguities: • Type I -Overlapping Ambiguity (OA)

The Arabic language is a Semitic language and it exhibits systematic but complex morphological structure based on root-pattern design. The aim of the present paper is to propose a transfer-based approach using morphological analysis which induces a syntactic symmetry and morphology to improve machine translation system between Arabic and English. Both languages are highly asymmetrical in terms of morphological structures. Our system supposes segmentation the word in the morphologically rich language into the sequence of prefix (es)stem-suffix (es). The system identifies morphemes to be merged or deleted in the morphologically rich language to make the desired morphological and syntactic symmetry. The technique applied aims to improve Arabic-to-English translation quality.

The coexistence of a set of these properties in a given language may promote the perception of stressed syllables in relation to other syllables - yielding stress-timing -, or make all syllables equally salient - yielding syllable-timing. Recently, the properties behind the rhythmic ...

This study sets out to explore the basic processing unit in Chinese sentence reading in an eye movement experiment. In the experiment, four analogous conditions were created: normal Chinese sentence with no highlighting, highlighting that marked words, highlighting that marked the unit under slow reading conditions and highlighting that marked the unit under natural reading conditions. All eye movement measures indicated that sentence processing under the word condition was the most difficult. As a result, it was revealed that the word was not a basic processing unit in Chinese sentence reading. The segmented unit, which is the 'prosodic word' under the slow reading condition, is the smallest prosodic unit that can be accepted by the readers and within reader's perceptual span, and is thus more likely to be the basic processing unit in Chinese sentence reading. Moreover, the duration of pause is an objective marker of the basic processing unit.

Chinese lexical analysis consists of word segmentation and part-of-speech tagging. Most previous studies consider them as two separate tasks. In this paper we formalize the two processes as a unique chunking task on a sequence of morphemes and present an integrated lexical analysis system for Chinese based on lexicalized hidden Markov models. In this way, both contextual lexical information and word-internal morphological features can be statistically explored and further combined for disambiguation and unknown word resolution. Experimental results show that the proposed system outperforms several baselines, illustrating the benefits of the unified lexical chunking method with morphemes as the basic units.

showed that adults were able to segment into words an artificial language that included no pauses or other prosodic cues for word boundaries. We propose an account of their results that requires only limited computational abilities and memory capacity. In this account, parsing emerges as a natural consequence of the on-line attentional processing of the input, thanks to basic laws of memory and associative learning. Our account was implemented in a computer program, PARSER. Simulations revealed that PARSER extracted the words of the language well before exhausting the material presented to participants in the Saffran et al. experiments. In addition, PARSER was able to simulate the results obtained under attention-disturbing conditions (Saffran, Newport, Aslin, Tunick, & Barrueco, 1997) and those collected from 8-month-old infants (Saffran, Aslin, and Newport, 1996a). Finally, the good performance of PARSER was not limited to the trisyllabic words used by Saffran et al., but also extended to a language composed of one-to five-syllable words.

This paper addresses two major problems in closed task of Chinese word segmentation (CWS): tagging sentences interspersed with non-Chinese words, and long named entity (NE) identification. To resolve the former, we apply Kmeans clustering to identify non-Chinese characters, and then adopt a two-tagger architecture: one for Chinese text and the other for non-Chinese text. For the latter problem, we apply postprocessing to our CWS output using automatically generated templates. The experiment results show that, when non-Chinese characters are sparse in the training corpus, our two-tagger method significantly improves the segmentation of sentences containing non-Chinese words. Identification of long NEs and long words is also enhanced by template-based postprocessing. In the closed task of SIGHAN 2006 CWS, our system achieved F-scores of 0.957, 0.972, and 0.955 on the CKIP, CTU, and MSR corpora respectively.

Recent dynamic tokenisation methods operate directly on bytes and pool their latent representations into patches. This bears similarities to computational models of word segmentation that determine lexical boundaries using spikes in an autoregressive model's prediction error. Inspired by this connection, we explore whether grouping predictable bytes-rather than pooling their representations-can yield a useful fixed subword vocabulary. We propose a new information-driven subword tokeniser, ByteSpan, that uses an external byte-level LM during training to identify contiguous predictable byte sequences and group them into subwords. Experiments show that ByteSpan yields efficient vocabularies with higher morphological alignment scores than BPE for English. Multilingual experiments show similar compression and Rényi efficiency for 25 languages.

Invited Keynote Delivered by Dr Fermín Moscoso del Prado Martín

In contrast with a broad corpus of research on the distribution of the frequencies of words, the distribution of the frequencies of phonemic contrasts is remarkably understudied. Using data from over one hundred language varieties, I will discuss that all of them can be modeled a priori by a virtually parameter-free distribution; in other words, just knowing the number of phonemic contrasts in a language variety, one can accurately guess their relative frequencies. Moreover, in situations with extreme undersampling —as would be the case when documenting new, unknown varieties— this knowledge can be used for guessing the expected number of existing contrasts. Finally, I discuss how informational and diachronic constraints give rise to these distributions. Incrementally complex models of phonological change can replicate these distributions with minimum assumptions.

Dzongkha, the national language of Bhutan, is continuous in written form and it fails to mark the word boundary. Dzongkha word segmentation is one of the fundamental problems and a prerequisite that needs to be solved before more advanced Dzongkha text processing and other natural language processing tools can be developed. This paper presents our initial attempt at segmenting Dzongkha sentences into words. The paper describes the implementation of Maximal Matching (Dictionary based Approach) followed by bigram techniques (Non-dictionary based Approach) in segmenting the Dzongkha scripts. Although the used techniques are basic and naive, it provides a baseline of the Dzongkha word segmentation task. Preliminary experimental results show percentage of segmentation accuracy. However, the segmentation accuracy is dependent on the type of document domain and size and quality of the lexicon and the corpus. Some of the related issues for future directions are also discussed.

An image-based approach to document image analysis is presented. The methods are moti- vated by a merged view of shape and textural image properties at multiple scales. The principal binary image operations are morphological and multiresolution. The generalized opening is in- troduced for extraction of both shape and texture from an image. Threshold reduction operations are introduced for performing efficient and controllable shape and texture transformations between resolution levels. Some problems, such as halftone or dark area segmentation, can be in large part solved by a sequence of threshold reductions. Aspects of the approach are illustrated by the problem of identifying italic and bold words in text, using word-level extraction at lowered resolution. The computational costs of the basic operations are given, so that algorithm efficienc y can be estimated, and the importance of operating at the lowest feasable resolution is demonstrated. For example, word segmentation and hal...

Text-line segmentation is considered as a crucial step of document analysis and recognition systems because its output is considered as the input of recognition systems. Due to the reason that the same handwritten image page has different characteristics, we propose in this paper a multilevel segmentation framework for handwritten historical documents. In this framework, one or many segmentation methods are selected according to the input document features. This framework is tested on the IAM historical database (60 images) and on images from the segmentation competition for handwritten document segmentation held at ICFHR 2010. The evaluation of the segmentation framework is based on several evaluation metrics. The tests show that the proposed framework gives promoting results.

Unsupervised methods tend to discover highly speaker-specific representations of speech. We propose a method for improving the quality of posteriorgrams generated from an unsupervised model through partitioning of the latent classes. We do this by training a sparse siamese model to find a linear transformation of the input posteriorgrams to lower-dimensional posteriorgrams. The siamese model makes use of same-category and differentcategory speech fragment pairs obtained by unsupervised term discovery. After training, the model is converted into an exact partitioning of the posteriorgrams. We evaluate the model on the minimal-pair ABX task in the context of the Zero Resource Speech Challenge. We are able to demonstrate that our method significantly reduces the dimensionality of standard Gaussian mixture model posteriorgrams, while still making them more robust to speaker variations. This suggests that the model may be viable as a general post-processing step to improve probabilistic acoustic features obtained by unsupervised learning.

We present a Chinese Named Entity Recognition (NER) system submitted to the close track of Sighan Bakeoff2006. We define some additional features via doing statistics in training corpus. Our system incorporates basic features and additional features based on Conditional Random Fields (CRFs). In order to correct inconsistently results, we perform the postprocessing procedure according to n-best results given by the CRFs model. Our final system achieved a F-score of 85.14 at MSRA, 89.03 at CityU, and 76.27 at LDC.

When infants and toddlers are confronted with sequences of sounds, they are required to segment the sounds into meaningful units to achieve sufficient understanding. Rhythm has been regarded as a crucial cue for segmentation of speech sounds. Although previous intermodal methods indicated that infants and toddlers could detect differences in speech sounds based on stress-timed and syllable-timed units, these methods could not clearly indicate how infants and toddlers perform sound segmentation. Thus, the present study examined whether Japanese infants and toddlers could segment word speech sounds comprising basic morae (i.e., rhythm units similar to syllables), on the basis of concurrent basic mora units within syllable units, using the new intermodal matching procedure. The results indicated that, regardless of their ages and linguistic abilities, Japanese infants and toddlers aged 6–25 months tended to segment Japanese words comprising basic morae sounds on the basis of concurrent...

This interesting volume represents a collection of high quality essays that touch on some of the more recent and "controversial" topics in creole studies in the last 10 years. In some ways, this volume represents the shift away from dominant debates or dichotomies that had previously defined creole studies, e.g., superstrata versus substrata, substrata versus universals (or the language bio-program hypothesis [LBH]). Bickerton's work on the LBH dominated much of the creolistics research of the 1980s-1990s, even if most of the research in print reflected significantly more detractors arguing against the hypothesis than supporters of it. In the same way, McWhorter's 1998 article has come to dominate much of the research in the early 21st century. Most work seems not to support his Prototype Hypothesis (PH), but to argue against it. Several of the articles in the present volume are reactions to/against McWhorter's work. The book is dedicated to the memory of Jacques Arends. In fact, the last article (pp. 223-241) represents one of Jacques's last published works (written in collaboration with his students) since his death in 2005. As many have already written, he and his excellent scholarship will be missed. This volume of 11 papers contains a subset of those that were presented at the "Second International Workshop on the Phonology and Morphology of Creole Languages", held at the University of Siegen in October 2003. As the (sub)title suggests, the book is divided into three parts: "Section 1: segmental aspects" (pp. 3-82); "Section 2: Syllabic aspects" (pp. 85-150); and "Section 3: Morphological aspects" (pp. 153-241). Instead of giving an inventory of all the articles, I will touch upon the chapters that I found the most illuminating or provocative. The first article, by Thomas B. , presents an inventory of phonemes in so-called creole languages. Klein concludes that, as regards vowels and consonants, creole languages are fairly typical (i.e., the typological middle ground) in terms of phonemic inventory among the world's languages. That is, there is nothing reduced or simplified about the phonemes in creoles. He writes: "The vast majority of Creole languages exhibit the typical non-Creole inventory size of 20-37 phonemes" (p. 8). Klein reaches this conclusion by comparing creole

This paper focuses on the segmentation of unconstrained handwritten documents containing bilingual data at the word level. Most of the official documents available in India are bilingual, i.e., in the regional language as well as English language. For the purpose of current research work, the handwritten documents from the academic domain containing Punjabi from Gurumukhi script and English language from Latin script have been considered. A heuristic approach based on projection profiles, statistical and structural properties of script has been designed to segment the textual data of documents. The proposed approach can segment the closed, curved, skewed and touched text lines having large variations in pattern and size. For word segmentation, an end-point detection algorithm has been designed to segment the words with intra word gap. The proposed approaches have been evaluated by designing a domain-based bilingual handwritten dataset after having consultations with academicians, and experts in the field. For text line segmentation, an average accuracy of 95.89% and 92.95% has been achieved for IAM and bilingual dataset respectively. However, for word segmentation, there has been an accuracy of 89.74% and 92.25% respectively for IAM and bilingual dataset. As many as 280 documents with various writing styles and content have been selected for the purpose.

Punctuation plays a vital role in written language, enhancing clarity, structure, and communicative intent. While modern artificial intelligence (AI) models have demonstrated advanced capabilities in natural language processing, their ability to accurately restore and generate punctuation-especially across multiple languages-remains a critical challenge. This study investigates the development of an AI-based punctuation system using the T5 transformer architecture, applied to both English and Burmese. By creating and fine-tuning parallel datasets of unpunctuated and punctuated text, the research evaluates the model's performance using both automated metrics and human judgment. Despite the resource limitations of Burmese, the model achieved high accuracy through transfer learning and syllable-based tokenization. The findings demonstrate that punctuation is not merely a syntactic feature but a culturally embedded linguistic system. This research highlights the potential of multilingual punctuation modeling for improving machine translation, speech-to-text applications, educational tools, and linguistic inclusion, especially in low-resource language contexts.

In this study the perception of word juncture in English and Arabic is investigated. Word juncture is taken as the allophonic, or phonetic, variation at word boundary that is contrastive. It is hypothesized that minimal pairs differentiated by means of juncture ( or boundary features) cannot be identified when heard in isolation and that the sentential context helps identifying the phrases of the pair. Corpuses of English and Arabic minimal pairs of juncture phrases were collected and native speakers of English and Arabic were asked to pronounce these phrases in isolation and then to use them in sentences. Three groups of subjects (a group of 13 Undergraduate students of English, a group of 11 MA students of English and a group of 9 Lecturers, all being native speakers of Arabic) were chosen to carry out the experimental part of the study. Four perception tests were carried out: two on English and two on Arabic. These tests were designated to examine the subjects' precision in t...

For large-vocabulary continuous speech recognition (LVCSR) of highly-inflected languages, selection of an appropriate recognition unit is the first important step. The morpheme-based approach is often adopted because of its high coverage and linguistic properties. But morpheme units are short, often consisting of one or two phonemes, thus they are more likely to be confused in ASR than word units. Generally, word units provide better linguistic constraint, but increases the vocabulary size explosively, causing OOV (out-of-vocabulary) and data sparseness problems in language modeling. In this research, we investigate approaches of selecting word entries by concatenating morpheme sequences, which would reduce word error rate (WER). Specifically, we compare the ASR results of the word-based model and those of the morpheme-based model, and extract typical patterns which would reduce the WER. This method has been successfully applied to an Uyghur LVCSR system, resulting in a significant reduction of WER without a drastic increase of the vocabulary size.

This paper is a report from collective participation in NTCIR-5 Question Answering Challenge between researchers from Mie University, Hokkaido University and Otaru University of Commerce. Although our results were not impressive, we would like to share our experiences with everyone who think about participating in the challenge but is afraid of his or her lack of experience in the field. Understanding the problems of QA from the practical side was very instructive and gave us a stronger base for future trials. We briefly ...

Word Segmentation is the foremost obligatory task in almost all the NLP applications, where the initial phase requires tokenization of input into words. Like other Asian languages such as Chinese, Thai and Myanmar, Urdu also faces word segmentation challenges. Though the Urdu word segmentation problem is not as severe as the other Asian language, since space is used for word delimitation, but the space is not consistently used, which gives rise to both space omission and space insertion errors in Urdu. In this paper we present a word segmentation system for handling space omission problem in Urdu script with application to Urdu-Devnagri Transliteration system. Instead of using manually segmented monolingual corpora to train segmenters, we make use of bilingual corpora and statistical word disambiguation techniques. Though our approach is adapted for the specific transliteration task at hand by taking the corresponding target (Hindi) language into account, the techniques suggested can be adapted to independently solve the space omission Urdu word segmentation problems. The two major components of our system are : identification of merged words for segmentation and proper segmentation of the merged words. The system was tested on 1.61 million word Urdu test data. The recall and precision for the merged word recognition component were found to be 99.29% and 99.38% respectively. The words are correctly segmented with 99.15% accuracy.

This research paper describes a corpus based transliteration system for Punjabi language. The existence of two scripts for Punjabi language has created a script barrier between the Punjabi literature written in India and in Pakistan. This research project has developed a new system for the first time of its kind for Shahmukhi script of Punjabi language. The proposed system for Shahmukhi to Gurmukhi transliteration has been implemented with various research techniques based on language corpus. The corpus analysis program has been run on both Shahmukhi and Gurmukhi corpora for generating statistical data for different types like character, word and n-gram frequencies. This statistical analysis is used in different phases of transliteration. Potentially, all members of the substantial Punjabi community will benefit vastly from this transliteration system.

Word Segmentation is an important prerequisite for almost all Natural Language Processing (NLP) applications. Since word is a fundamental unit of any language, almost every NLP system first needs to segment input text into a sequence of words before further processing. In this paper, Shahmukhi word segmentation has been discussed in detail. The presented word segmentation module is part of Shahmukhi-Gurmukhi transliteration system. Shahmukhi script is usually written without short vowels leading to ambiguity. Therefore, we have designed a novel approach for Shahmukhi word segmentation in which we used target Gurmukhi script lexical resources instead of Shahmukhi resources. We employ a combination of techniques to investigate an effective algorithm by applying syntactical analysis process using Shahmukhi Gurmukhi dictionary, writing system rules and statistical methods based on n-grams models.

This paper describes an algorithm to segment an input Turkish string without any spaces, which may be an output of a speech-to-text application, into words by using morphological analyzer. It is quite possible to use the algorithm on other languages, which has a morphological analysis component, as well. Turkish morphological analyzer is designed and implemented as the linguistic engine of the algorithm. The construction of the analyzer proposes a technique that attempts to achieve group vise morpheme recognition instead of searching suffixes one by one in a word.

A readability formula is obtained that can be used by computer programs for style checking of Japanese texts and need not syntactic or semantic information. The formula is derived as a linear combination of tile surface characteristics of the text that are related to its readability: (1) the average number of characters per sentence, (2) for each type of characters (Roman alphabets, kanzis, hiraganas, katakanas), relative frequencies of rims (maximal swings) that ,:onsists only of that type of characters, (3) the average number of characters per each type of runs, and ( ) tooten (comma) to kuten (period) ratio. To find the proper weighting, principal component analysis (PCA) was appliedto these characteristics taken from 77 sample texts. We have found a component which is related to the readability. Its scores match to the empirical knowledges of reading ease. We have also obtained experimental confirmation that the component is an adequate measure for stylistic ease of reading, by the cloze procedure and by the examination on the average lime taken to fill out one blank of the cloze texts.

Automatic term extraction is the first step towards automatic or semi-automatic update of existing domain knowledge base. Most of the researches applied word segmentation as a preprocessing step to Chinese term extraction. However, segmentation ambiguity is unavoidable, especially in identifying unknown words for Chinese. In this paper, we discuss the effect and limitations of segmentation to Chinese terminology extraction. Detailed study shows that propagated errors caused by word segmentation have great impact on the result of terminology extraction. Based on our analysis and experiments, it is proven that character-based terminology extraction yields much better result than that using segmentation as a preprocessing step.

Word segmentation is the first and obligatory task for every NLP. For inflectional languages like English, French, Dutch,.. their word boundaries are simply assumed to be whitespaces or punctuations. Whilst in various Asian languages, including Chinese and Vietnamese, ...

Osmanlı Metin Arşivi Projesi kapsamında Osmanlı Türkçesi metinlerinin yüklenmesi, ikilileştirilmesi, satır ve kelime bölütlenmesi, etiketlenmesi, tanınması ve testlerinin yapılması amacıyla bir Genel Ag arabirimi geliştirilmiştir. Bu arabirim sayesinde Osmanlı arşivleriyle çalışan araştırmacıların uzmanlık yardımının alınması ve geliştirdigimiz tanıma teknolojilerinin elyazması arşivlere uygulanması mümkün hale gelmiştir.

This paper reports on a study involving the automatic extraction of Chinese legal terms. We used a word segmented corpus of Chinese court judgments to extract salient legal expressions with standard collocation learning techniques. Our method takes the characteristics of Chinese legal terms into account. The extracted terms were evaluated by human markers and compared against a legal term glossary manually compiled from the same set of data. Results show that at least 50% of the extracted terms are legally salient. Hence they may supplement the outcome and lighten the inconsistency of human efforts. Moreover, various types of significant knowledge in the legal context are mined from the data as a by-product.

A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

The main drawback of previous Chinese character error detection systems is the high false alarm rate. To solve this problem, we propose a system that combines a statistic method and template matching to detect Chinese character errors. Error types include pronunciationrelated errors and form-related errors. Possible errors of a character can be collected to form a confusion set. Our system automatically generates templates with the help of a dictionary and confusion sets. The templates can be used to detect and correct errors in essays. In this paper, we compare three methods proposed in previous works. The experiment results show that our system can reduce the false alarm significantly and give the best performance on fscore.

Chinese texts do not contain spaces as word separators like English and many alphabetic languages. To use Moses to train translation models, we must segment Chinese texts into sequences of Chinese words. Increasingly more software tools for Chinese segmentation are populated on the Internet in recent years. However, some of these tools were trained with general texts, so might not handle domain-specific terms in patent documents very well. Some machine-learning based tools require us to provide segmented Chinese to train segmentation models. In both cases, providing segmented Chinese texts to refine a pre-trained model or to create a new model for segmentation is an important basis for successful Chinese-English machine translation systems. Ideally, high-quality segmented texts should be created and verified by domain experts, but doing so would be quite costly. We explored an approach to algorithmically generate segmented texts with parallel texts and lexical resources. Our scores in NTCIR-10 PatentMT indeed improved from our scores in NTCIR-9 PatentMT with the new approach.

This paper presents the application of a multi-scale paradigm to Chinese spoken document retrieval (SDR) for improving retrieval performance. Multi-scale refers to the use of both words and subwords for retrieval. Words are basic units in a language that carry lexical meaning and subword units (such as phonemes, syllables or characters) are building components for words. Retrieval using subword indexing units is found to perform better than words because of the robustness of subword units to out-of-vocabulary (OOV) words during speech recognition and ambiguities in word segmentation. Experimental results have demonstrated that subword bigrams can bring improvement in retrieval performance over words (~9.56%). Application of multi-scale fusion to SDR aims at combining the lexical information of words and the robustness of subwords. This work presents the first detailed investigation for a Cantonese broadcast news retrieval task using two different multi-scale fusion approaches: pre-retrieval fusion and post-retrieval fusion. Multi-scale retrieval using both words and syllable bigrams achieve improvement in retrieval performance (~1.90%) over retrieval on the composite scales.

This paper presents the application of a multi-scale paradigm to Chinese spoken document retrieval (SDR) for improving retrieval performance. Multi-scale refers to the use of both words and subwords for retrieval. Words are basic units in a language that carry lexical meaning and subword units (such as phonemes, syllables or characters) are building components for words. Retrieval using subword indexing units is found to perform better than words because of the robustness of subword units to out-of-vocabulary (OOV) words during speech recognition and ambiguities in word segmentation. Experimental results have demonstrated that subword bigrams can bring improvement in retrieval performance over words (~9.56%). Application of multi-scale fusion to SDR aims at combining the lexical information of words and the robustness of subwords. This work presents the first detailed investigation for a Cantonese broadcast news retrieval task using two different multi-scale fusion approaches: pre-retrieval fusion and post-retrieval fusion. Multi-scale retrieval using both words and syllable bigrams achieve improvement in retrieval performance (~1.90%) over retrieval on the composite scales.

The possibility that during Chinese reading information is extracted at the beginning of the current fixation was examined in this study. Twenty-four participants read for comprehension while their eye movements were being recorded. A pretarget-target two-character word pair was embedded in each sentence and target word visibility was manipulated in two time intervals (initial 140 ms or after 140 ms) during pretarget viewing. Substantial beginning-and end-of-fixation preview effects were observed together with beginning-of-fixation effects on the pretarget. Apparently parafoveal information at least at the character level can be extracted relatively early during ongoing fixations. Results are highly relevant for ongoing debates on spatially distributed linguistic processing and address fundamental questions about how the human mind solves the task of reading within the constraints of different writing systems.

Word segmentation is one of the most important tasks in NLP. This task, within Vietnamese language and its own features, faces some challenges, especially in words boundary determination. To tackle the task of Vietnamese word segmentation, in this paper, we propose the WS4VN system that uses a new approach based on Maximum matching algorithm combining with stochastic models using part-of-speech information. The approach can resolve word ambiguity and choose the best segmentation for each input sentence. Our system gives a promising result with an F-measure of 97%, higher than the results of existing publicly available Vietnamese word segmentation systems.

This paper deals with an Optical Character Recognition (OCR) system for printed Oriya script. The development of OCR for this script is difficult because a large number of character shapes in the script have to be recognized. In the proposed system, the document image is first captured using a flat-bed scanner and then passed through different preprocessing modules like skew correction, line segmentation, zone detection, word and character segmentation etc. These modules have been developed by combining some conventional techniques with some newly proposed ones. Next, individual characters are recognized using a combination of stroke and run-number based features, along with features obtained from the concept of water overflow from a reservoir. The feature detection methods are simple and robust, and do not require preprocessing steps like thinning and pruning. A prototype of the system has been tested on a variety of printed Oriya material, and currently achieves 96.3% character level accuracy on average.

COMUNICA is a voice QA system for Brazilian Portuguese with search capabilities for consulting both structured and unstructured datasets. One of the goals of this work is to help address digital inclusion by providing an alternative way to accessing written information, which users can employ regardless of available computational resources or computational literacy.

Abstract: There are immense efforts to design a complete OCR for most of the world's leading languages, however, multilingual documents either of handwritten or of printed form. As a united attempt, Unicode based OCRs were studied mostly with some positive ...

For languages that have no explicit word boundary such as Thai, Chinese and Japanese, correcting words in text is harder than in English because of additional ambiguities in locating error words. The traditional method handles this by hypothesizing that every substrings in the input sentence could be error words and trying to correct all of them. In this paper, we propose the idea of reducing the scope of spelling correction by focusing only on dubious areas in the input sentence. Boundaries of these dubious areas could be obtained approximately by applying word segmentation algorithm and finding word sequences with low probability. To generate the candidate correction words, we used a modified edit distance which reflects the characteristic of Thai OCR errors. Finally, a part-ofspeech trigram model and Winnow algorithm are combined to determine the most probable correction.

This paper describes Forst's approach to university entrance examinations at NTCIR-11 QA-Lab Task. Our system consists of two types of modules: dedicated modules for each question format and common modules called by the dedicated modules as necessary. Our system uses Basic Element in order to more exactly grasp and reflect the import of questions. We also tackled short-essay questions in the secondary examinations.

Given that nouns rarely appear in isolation in French, infants acquiring the language must often retrieve the underlying representation of vowel-initial lexical forms from liaison contexts that provide conflicting information about the initial phoneme. Given this ambiguity, how do learners represent these nouns in their lexicons, and how do these representations change as learners' knowledge of liaison and the lexicon become more adult-like? To explore this question, we analyze the types of errors children make, in both naturalistic and elicited speech, and how these are affected by input frequency. In doing so, we evaluate two major proposals for how children's early representations of liaison develop. The first model, couched in a constructionist framework, predicts relatively late mastery of liaison (age five or older) and heavy dependence on the contexts in which a particular noun appears in the input. The second model takes an approach to liaison development that integrates it more closely with general phonological development and predicts relatively early mastery (by age three). The results of a corpus study reveal that by age three children are correctly producing liaison in the nominal domain and that their production errors are consistent with a phonological model of liaison acquisition. An elicitation task demonstrates that threeyear-olds succeed at learning and correctly apply their knowledge of liaison to new nouns following brief exposure, though their productions continue to be influenced by nouns' input distributions. Taken together, our findings suggest that by age three children are well on their way to adult-like representations of liaison. A phonologically based model, incorporating the effect of distributional context on early errors, provides a better overall fit to the data we present.

Given that nouns rarely appear in isolation in French, infants acquiring the language must often retrieve the underlying representation of vowel-initial lexical forms from liaison contexts that provide conflicting information about the initial phoneme. Given this ambiguity, how do learners represent these nouns in their lexicons, and how do these representations change as learners' knowledge of liaison and the lexicon become more adult-like? To explore this question, we analyze the types of errors children make, in both naturalistic and elicited speech, and how these are affected by input frequency. In doing so, we evaluate two major proposals for how children's early representations of liaison develop. The first model, couched in a constructionist framework, predicts relatively late mastery of liaison (age five or older) and heavy dependence on the contexts in which a particular noun appears in the input. The second model takes an approach to liaison development that integrates it more closely with general phonological development and predicts relatively early mastery (by age three). The results of a corpus study reveal that by age three children are correctly producing liaison in the nominal domain and that their production errors are consistent with a phonological model of liaison acquisition. An elicitation task demonstrates that threeyear-olds succeed at learning and correctly apply their knowledge of liaison to new nouns following brief exposure, though their productions continue to be influenced by nouns' input distributions. Taken together, our findings suggest that by age three children are well on their way to adult-like representations of liaison. A phonologically based model, incorporating the effect of distributional context on early errors, provides a better overall fit to the data we present.

Numerous experimentations are performed in the area of optical character recognition. The overall process of optical character recognition can be assured successful only when there is skew free document input considered at the time of pre-processing. The automatic skew detection and correction of document images is one of the important operation to be focused in this regard, since it increases the chances of obtaining overall accuracy of entire optical character recognition system. In this paper we have devised a skew detection and correction methodology using multiple projection profiles and skew angles. The document is subjected to computation of vertical profiles and difference is black pixel density is used as parameter to identify the tie points of a particular text block and skew angle. The identified text block is then rotated in opposite direction to get a skew eliminated image. The algorithm has given an accuracy of over 90% in many documents experimented..

For the computer, word forms in an online text are simply letter sequences between blanks. A rule-based automatic language analyses presupposes, however, that the computer can recognize the individual word forms. This includes assigning the base form (lemmatization) and determining the morphosyntactic properties (categorization). It is shown that there are three principled methods of automatic word form recognition, based on word forms, morphemes, and allomorphs, respectively. After describing these different methods using the notions of traditional morphology, they are compared with regards to their handling of neologisms, time efficiency, and space requirements.

Word segmentation skills emerge during infancy, but it is unclear to what extent this ability is shaped by experience listening to a specific language or language type. This issue was explored by comparing segmentation of bi-syllabic words in monolingual and bilingual 7.5-month-old learners of French and English. In a native-language condition, monolingual infants segmented bi-syllabic words with the predominant stress pattern of their native language. Monolingual French infants also segmented in a different dialect of French, whereas both monolingual groups failed in a cross-language test, i.e. English infants failed to segment in French and vice versa. These findings support the hypothesis that word segmentation is shaped by infant sensitivity to the rhythmic structure of their native language. Our finding that bilingual infants segment bi-syllabic words in two native languages at the same age as their monolingual peers shows that dual language exposure does not delay the emergenc...

Morphologically rich languages are notoriously challenging to process for downstream NLP applications. This paper presents a new pretrained language model, ByT5-Sanskrit, designed for NLP applications involving the morphologically rich language Sanskrit. We evaluate ByT5-Sanskrit on established Sanskrit word segmentation tasks, where it outperforms previous data-driven approaches by a considerable margin and matches the performance of the current best lexicon-based model. It is easier to deploy and more robust to data not covered by external linguistic resources. It also achieves new state-of-the-art results in Vedic Sanskrit dependency parsing and OCR post-correction tasks. Additionally, based on the Digital Corpus of Sanskrit, we introduce a novel multitask dataset for the joint training of Sanskrit word segmentation, lemmatization, and morphosyntactic tagging tasks. We fine-tune ByT5-Sanskrit on this dataset, creating a versatile multitask model for various downstream Sanskrit applications. We have used this model in Sanskrit linguistic annotation projects, in information retrieval setups, and as a preprocessing step in a Sanskrit machine translation pipeline. We also show that our approach yields new best scores for lemmatization and dependency parsing of other morphologically rich languages. We thus demonstrate that byte-level pretrained language models can achieve excellent performance for morphologically rich languages, outperforming tokenizer-based models and presenting an important vector of exploration when constructing NLP pipelines for such languages.