Word Level English to Hindi Neural Machine Translation

Machine Translation, 2021

In this paper we explore neural machine translation (NMT) for Indian languages. Reported work on Indian language Statistical Machine Translation (SMT) demonstrated good performance within the Indo-Aryan family, but relatively poor performance within the Dravidian family as well as between the two families. Interestingly, by common observation NMT generates more fluent output than SMT. This led us to investigate NMT's potential for translation involving Indian languages. The current practice in NMT is to train the models with subword units. Among subwording methods, byte pair encoding (BPE) is a popular choice. We conduct extensive experiments with BPE-based NMT models for Indian languages. An interesting outcome of our study is the finding that the optimal value for BPE merge for Indian language pairs seems to be falling in the range of 0-5000 which is fairly low compared to that observed for European Languages. Additionally, we apply other techniques such as phrase table injection and linguistic feature based enhancements on corpora, plus BERT augmented NMT to boost performance. To the best of our knowledge, this is the first comprehensive study on Indian language NMT (ILNMT) covering major languages in India. As an empirical paper, we expect this work could serve as a benchmark for ILNMT research.

Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), 2019

With the extensive use of Machine Translation (MT) technology, there is progressively interest in directly translating between pairs of similar languages. Because the main challenge is to overcome the limitation of available parallel data to produce a precise MT output. Current work relies on the Neural Machine Translation (NMT) with attention mechanism for the similar language translation of WMT19 shared task in the context of Hindi-Nepali pair. The NMT systems trained the Hindi-Nepali parallel corpus and tested, analyzed in Hindi ⇔ Nepali translation. The official result declared at WMT19 shared task, which shows that our NMT system obtained Bilingual Evaluation Understudy (BLEU) score 24.6 for primary configuration in Nepali to Hindi translation. Also, we have achieved BLEU score 53.7 (Hindi to Nepali) and 49.1 (Nepali to Hindi) in contrastive system type.

Indonesian journal of electrical engineering and computer science, 2024

IJETRM, 2025

Neural Machine Translation (NMT) has transformed the field of language translation by utilizing deep learning techniques to produce accurate and fluent translations. Machine Translation (MT), a core area of Natural Language Processing (NLP), deals with the automatic conversion of text from one language to another. With the increasing need for multilingual interaction, NMT has become a powerful solution to enhance translation quality. This study introduces an English-Hindi NMT system based on the Transformer architecture, which replaces recurrent connections with self-attention mechanisms to effectively model long-range dependencies in text sequences. Unlike traditional Recurrent Neural Network (RNN)-based approaches, the Transformer significantly improves parallelization, boosts accuracy, and lowers computational overhead. The system incorporates a robust preprocessing pipeline involving tokenization, text normalization, and subword segmentation using Byte-Pair Encoding (BPE). Implementation is carried out using PyTorch and Hugging Face libraries, and the model performance is assessed through the BLEU score metric. Experimental evaluation indicates a noticeable improvement in BLEU score compared to previous models, making it highly suitable for low-resource languages such as Hindi. Future work aims to extend the system for multilingual support and enable real-time deployment as a web-based translation tool.

arXiv (Cornell University), 2023

Language is an effective means of communication that enables people to express their thoughts, feelings, desires, and other emotions. The method of automatically converting information from one language to another without human intervention is known as Machine Translation (MT). People are creating and using MT models to overcome these language barriers. Multilingual neural machine translation (MNMT), which builds a single model for multiple languages, is beneficial since it decreases the training time and improves low-resource translation. Lowresource regional languages, like Indian Languages (ILs), oversee the lack of good-quality MT models. In this paper, Indic-to-Indic (IL-IL) MNMT baseline models for 11 ILs are implemented on the Samanantar corpus and analyzed on the Flores-200 corpus. All the models are evaluated using the BLEU score. In addition, the languages are classified under 3 groups namely East Indo-Aryan (EI), Dravidian (DR), and West Indo-Aryan (WI). The effect of language relatedness on MNMT model efficiency is studied. Owing to the presence of large corpora from English (EN) to ILs, MNMT IL-IL models using EN as a pivot are also built and examined. To achieve this, English-Indic (EN-IL) models are also developed, with and without the usage of related languages. Results reveal that using related languages is beneficial for the WI group only, while it is detrimental for the EI group and shows an inconclusive effect on the DR group, but it is useful for EN-IL models. Thus, related language groups are used for the development of pivot MNMT models. Furthermore, the IL corpora are transliterated from the corresponding scripts to a modified ITRANS script, and the best MNMT models from the previous approaches are built on the transliterated corpus. It is observed that the usage of pivot models greatly improves MNMT baselines with AS-TA achieving the minimum BLEU score and PA-HI achieving the maximum score. Among languages, AS, ML, and TA achieve the lowest BLEU score whereas HI, PA, and GU perform the best. Transliteration also helps the models with few exceptions. The best increment of scores is observed in ML, TA, and BN and the worst average increment is observed in KN, HI, and PA, across all languages. The best model obtained is the PA-HI language pair trained on PA-WI transliterated corpus which gives 24.29 BLEU.

To provide approaches for effective Hindi-to-English Machine Translation (MT) that can be helpful in inexpensive and ease implementation of and MT systems. Methods/Statistical Analysis: Structure of the Hindi and English languages have been studied thoroughly. The possible steps towards the Natural languages have also been studied. The methods, rules, approaches, tools, resources etc. related to MT have been discussed in detail. Findings: MT is an idea for automatic translation of a language. India is the country with full of diversity in culture and languages. More than 20 regional languages are spoken along with several dialects. Hindi is a widely spoken language in all the states of country. A lot of literature, poetries and valuable texts are available in Hindi which gives opportunities to retranslate into English. However, new generation is learning English rapidly and also showing keenness to learn it in simplified lucid manner. Several efforts have been made in this direction. A large number of approaches and solutions exist for MT still there is a huge scope. The paper addresses the challenges of MT and solution efforts made in this direction. This motivates researchers to implement new Hindi-to-English Machine translation systems. Application/Improvements: Efficient, inexpensive and ease translation for available Hindi literature, poetries and other valuable texts into English. Children can easily learn the culture through the poetries and literatures hence the Machine Translation of these will bring wonderful impact.

2020

In this manuscript, we (team name is NLPRL) describe systems description that was submitted to the translation shared tasks at WAT 2020. We describe our model as transformer based NMT by using byte-level based BPE (BBPE). We used the OdiEnCorp 2.0 parallel corpus provided by the shared task organizer where the training, validation, and test data contain 69370, 13544, and 14344 lines of parallel sentences, respectively. The evaluation results show the BLEU score of English-to-Oria below the Organizer (1.34) and Oria-to-English direction shows above the Organizer (11.33).

Information

Most Indian languages lack sufficient parallel data for Machine Translation (MT) training. In this study, we build English-to-Indian language Neural Machine Translation (NMT) systems using the state-of-the-art transformer architecture. In addition, we investigate the utility of back-translation and its effect on system performance. Our experimental evaluation reveals that the back-translation method helps to improve the BLEU scores for both English-to-Hindi and English-to-Bengali NMT systems. We also observe that back-translation is more useful in improving the quality of weaker baseline MT systems. In addition, we perform a manual evaluation of the translation outputs and observe that the BLEU metric cannot always analyse the MT quality as well as humans. Our analysis shows that MT outputs for the English–Bengali pair are actually better than that evaluated by BLEU metric.

2019 IEEE Conference on Information and Communication Technology, 2019

In recent years, technological advances have played a very important role in our society, as well as in different educational contexts, including Education for Sustainable Development. Information and Communication Technologies (ICTs), which have made it possible to improve the quality of life of many people. The aim of this study is to review the scientific evidence on the use of ICTs for Sustainable Development. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyzes (PRISMA) guidelines, a bibliographic search was carried out using the Web of Science and Scopus bibliographic repositories, with a total of 19 articles comprising the final sample for review. The results show that the most preeminent strategies used are mobile learning and distance education, which are related to the achievement of the sustainable development goals. Included among the conclusions of this study is the need to investigate the different strategies that are being carried out, with ICTs in school aimed at developing a more sustainable environment standing out.

2021

This paper describes our team’s submission of the EACL DravidianLangTech-2021’s shared task on Machine Translation of Dravidian languages.We submitted our translations for English to Malayalam , Tamil , Telugu and also Tamil-Telugu language pairs. The submissions mainly focus on having adequate amount of data backed up by good preprocessing of it to produce quality translations,which includes some custom made rules to remove unnecessary sentences. We conducted several experiments on these models by tweaking the architecture,Byte Pair Encoding (BPE) and other hyperparameters.