Identification of language in a cross linguistic environment

Recurrent Neural Networks are a type of Artificial Neural Networks which are adept at dealing with problems which have a temporal aspect to them. These networks exhibit dynamic properties due to their recurrent connections. Most of the advances in deep learning employ some form of Recurrent Neural Networks for their model architecture. RNN's have proven to be an effective technique in applications like computer vision and natural language processing. In this paper, we demonstrate the effectiveness of RNNs for the task of English to Hindi Machine Translation. We perform experiments using different neural network architectures-employing Gated Recurrent Units, Long Short Term Memory Units and Attention Mechanism and report the results for each architecture. Our results show a substantial increase in translation quality over Rule-Based and Statistical Machine Translation approaches.

International Journal for Research in Applied Science and Engineering Technology, 2021

This paper introduces a new text-to-speech presentation from end-to-end (E2E-TTS) using toolkit called ESPnet-TTS, which is an open source extension. ESPnet speech processing tools kit. Various models come under ESPnet TTS TacoTron 2, Transformer TTS, and Fast Speech. This also provides recipes recommended by the Kaldi speech recognition tool kit (ASR). Recipes based on the composition combined with the ESPnet ASR recipe, which provides high performance. This toolkit also provides pre-trained models and samples of all recipes for users to use as a base .It works on TTS-STT and translation features for various indicator languages, with a strong focus on English, Marathi and Hindi. This paper also shows that neural sequence-to-sequence models find the state of the art or near the effects of the art state on existing databases. We also analyze some of the key design challenges that contribute to the development of a multilingual business translation system, which includes processing bilingual business data sets and evaluating multiple translation methods. The test result can be obtained using tokens and these test results show that our models can achieve modern performance compared to the latest LJ Speech tool kit data. Terms of Reference-Open source, end-to-end, text-to-speech I. INTRODUCTION Language translation and transliteration is increasingly being used in all aspects of daily life, this because the access to communication is faster, cheaper and readily available, enabling people all over the world to communicate directly with each other also can officially understand one another, with the meaning distinctly clear and succinct. [1] TTS or Text-to-Speech technology translates text into spoken speech. TTS can make life effortless and make you more efficient. It helps illiterate students and other disabled learners to learn by removing the pressure to read and present information appropriately. This project aims to tackle the issue with translating the contents of the page in the selected language. It also has a speech to text feature, where the browser speaks in the selected language. Speech is basically, natural form of communication. Speech is very common way for humans to interact because it is the most effective way to communicate, it can be also extended further to interact with the system. If any part of the society is not able to communicate because of Speech disability, technology can be utilized to make this happen. Besides Speech Recognition can be adapt for the individual having Speech disability, If Speech Technologies is a built-in communication system, it can help individuals, who can use a common type of communication to produce the right Speech. Speech Recognition is the capability of machine/program to recognize words and phrases in spoken language and convert them into machine-readable format ,it is a field of research aimed at converting audio into a series of related words. Transliteration-the translation of precise nouns from one orthographic system to another-is an important function in multilingual text processing, useful for programs such as online mapping and as part of typewriters. It is a challenge to translate words and technical terms from user input into all languages in different alphabets and phonics lists. One of the most common problems translators have to deal with is translating the correct words and technical terms into a user's input. These objects are often translated, that is, they are replaced by phonetic equivalent. For example, the local name "क ं चनपु र" in the Devanagari language is translated "Kanchanpur" into English. Translating such material from English into Devanagari is even more challenging, as the translated material creates a vast amount of text that is not available in bilingual dictionaries. II. LITERATURE REVIEW This paper proposes the idea of having Text-to-Speech and Speech-to-Text for some Indian languages. Mihaela Rosco and Thomas Breuel [15] has proposed a sequence to sequence neural network models for transliteration in which model based on epsilon insertions , CTC alignment and attentional sequence to sequence model used in end to end machine translation also achieved high performance on cross script transliteration

Indonesian Journal of Electrical Engineering and Computer Science, 2022

Transliteration is the task of translating text from source script to target script provided that the language of the text remains the same. In this work, we perform transliteration on less explored Devanagari to Roman Hindi transliteration and its back transliteration. The neural transliteration model in this work is based on a sequence-to-sequence neural network that is composed of two major components, an encoder that transforms source language words into a meaningful representation and the decoder that is responsible for decoding the target language words. We utilize gated recurrent units (GRU) to design the multilayer encoder and decoder network. Among the several models, the multilayer model shows the best performance in terms of coupon equivalent rate (CER) and word error rate (WER). The method generates quite satisfactory predictions in Hindi-English bilingual machine transliteration with WER of 64.8% and CER of 20.1% which is a significant improvement over existing methods.

Proceedings of the 10th International Conference on Agents and Artificial Intelligence, 2018

The goal of similar Language IDentification (LID) is to quickly and accurately identify the language of the text. It plays an important role in several Natural Language Processing (NLP) applications where it is frequently used as a pre-processing technique. For example, information retrieval systems use LID as a filtering technique to provide users with documents written only in a given language. Although different approaches to this problem have been proposed, similar language identification, in particular applied to short texts, remains a challenging task in NLP. In this paper, a method that combines word vectors representation and Long Short-Term Memory (LSTM) has been implemented. The experimental evaluation on public and well-known datasets has shown that the proposed method improves accuracy and precision of language identification tasks.

The progression of social media contents, similar like Twitter and Facebook messages and blog post, has created, many new opportunities for language technology. The user generated contents such as tweets and blogs in most of the languages are written using Roman script due to distinct social culture and technology. Some of them using own language script and mixed script. The primary challenges in process the short message is identifying languages. Therefore, the language identification is not restricted to a language but also to multiple languages. The task is to label the words with the following categories L1, L2, Named Entities, Mixed, Punctuation and Others This paper presents the AmritaCen_NLP team participation in FIRE2015-Shared Task on Mixed Script Information Retrieval Subtask 1: Query Word Labeling on language identification of each word in text, Named Entities, Mixed, Punctuation and Others which uses sequence level query labelling with Support Vector Machine.

Journal of Engineering Science and Technology Review, 2024

People have become part of the digital era with the advent of the Web. They actively create, share, a variety of content on the web. Unlike earlier days, people widely use different social platforms to talk about their interests, hobbies, reviews on movies, and purchased items in natural language. Processing such natural languages with mixed language tasks is challenging. A sizable proportion communicates in regional language but using code-mixed and script like Roman and Devnagari for English and Marathi language. These texts are generally informal, causal, short length, non-standard spelling alteration etc are prime challenges in language processing. Language identification in mixed text is challenging, since the Romanized string of several languages is comparable. Mixed text is essential to transform into native script for further processing like Information Retrieval, machine translation, Question Answering etc. Due to the lack of orthography of Latin script in Marathi, language modelling, and identification of mixed text is a challenging issue. Many NLP (Natural Language Processing) applications ranging from machine translation and information retrieval uses Machine transliteration as input mechanism for non-roman script. In this paper, different techniques and various approaches presented by the researchers for code-mixed language, Indian regional languages processing are discussed. The tasks like language identification, transliteration, Named Entity Recognition are reviewed with respect to Statistical, Rule and Neural based approaches.

In this paper, we propose a recognition scheme for the Indian script of Hindi. Recognition accuracy of Hindi script is not yet comparable to its Roman counterparts. This is mainly due to the complexity of the script, writing style etc. Our solution uses a Recurrent Neural Network known as Bidirectional Long Short Term Memory (BLSTM). Our approach does not require word to character segmentation, which is one of the most common reason for high word error rate. We report a reduction of more than 20% in word error rate and over 9% reduction in character error rate while comparing with the best available OCR system.

2018

Text language Identification is a Natural Language Processing task of identifying and recognizing a given language out of many different languages from a piece of text. This paper describes our submission to the ILI 2018 shared-task, which includes the identification of 5 closely related Indo-Aryan languages. We developed a word-level LSTM(Long Short-term Memory) model, a specific type of Recurrent Neural Network model, for this task. Given a sentence, our model embeds each word of the sentence and convert into its trainable word embedding, feeds them into our LSTM network and finally predict the language. We obtained an F1 macro score of 0.836, ranking 5th in the task.

The language identification problem refers to the task of deciding in which natural language a given text is written this is the one of the major challenge in the Natural Language Processing. Once the language being identified next translation is to be carried out. Translation is the communication of the meaning of a source-language text by means of an equivalent target-language text. Whereas interpreting undoubtedly antedates writing, translation began only after the appearance of written literature. In this paper we are presenting the modules for identification and translation of English to Indian languages or Indian Interlingual Languages here particularly for English, Kannada and Telugu. We have analyzed the Morphology of each language for translating a text from source language to target language. To test these modules we have build our own morphological analyzer and generator, part of speech tagger, and designed virtual keyboards for Transliteration and Translation with the help of BUBShabdasagar-2011 Bilingual dictionary which consist around 20,000 words.

International Journal of Engineering and Advanced Technology, 2020

In this era of globalization, it is quite likely to come across people or community who do not share the same language for communication as us. To acknowledge the problems caused by this, we have machine translation systems being developed. Developers of several reputed organizations like Google LLC, have been working to bring algorithms to support machine translations using machine learning algorithms like Artificial Neural Network (ANN) in order to facilitate machine translation. Several Neural Machine Translations have been developed in this regard, but Recurrent Neural Network (RNN), on the other hand, has not grown much in this field. In our work, we have tried to bring RNN in the field of machine translations, in order to acknowledge the benefits of RNN over ANN. The results show how RNN is able to perform machine translations with proper accuracy.