Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Pre-Training Objectives
Data and Tokenization
Memory Aids for Dementia Patients
Interpretability and Safety by Design
Conclusion
References
All Topics
Computer Science
Artificial Intelligence

From Transformers to LLMs

Profile image of Arsalan AslamArsalan Aslam

2025, Elsevier

https://doi.org/10.48550/ARXIV.2506.01934
visibility

…

description

12 pages

link

1 file

Abstract

Large Language Models (LLMs) have catalyzed a paradigm shift in Natural Language Processing (NLP). From the introduction of the Transformer architecture to the development of massive generative models such as GPT-3.5, LLaMA2-7B, and PaLM, these architectures demonstrate emergent behaviors allowing for zero-shot and few-shot learning, grounded generation, and cross-modal understanding. This paper provides a comprehensive review of LLMs, covering their architectural foundations, training paradigms, emergent abilities, evaluation benchmarks, and diverse applications-including mental health monitoring, assistive technologies for the visually impaired, and novel regularization techniques. We also highlight current challenges such as interpretability, bias, and resource efficiency, and discuss future directions for research and responsible deployment.

Related papers

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Elton Zhang

2022

Pretrained general-purpose language models can achieve state-of-the-art accuracies in various natural language processing domains by adapting to downstream tasks via zero-shot, few-shot and fine-tuning techniques. Because of their success, the size of these models has increased rapidly, requiring high-performance hardware, software, and algorithmic techniques to enable training such large models. As the result of a joint effort between Microsoft and NVIDIA, we present details on the training of the largest monolithic transformer based language model, Megatron-Turing NLG 530B (MT-NLG), with 530 billion parameters. In this paper, we first focus on the infrastructure as well as the 3D parallelism methodology used to train this model using DeepSpeed and Megatron. Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model. Finally, we discuss various evaluation results, as wel...

View PDFchevron_right
Large Language Models: Machine Learning Down to Earth
Paulo Leocadio

Brainz Magazine, 2023

The business marketplace has been flooded with waves of technology trends that periodically surface and become present on every other sales pitch from technology vendors and build up as the ultimate necessity in the minds of many CIOs. There is a variety of examples to mention: downsizing, rightsizing, outsourcing, offshoring, consumerization of IT, and the adoption of Cloud technologies, among others. And like virtual tsunamis, these trends come impacting, sometimes disrupting, and even influencing the performance or changing the perception of a corporation (or government) in the eyes of the consumer. It is not different today: tailgating the Cloud adoption, and the spreading of a Big Data mindset to solve problems and build solutions comes Artificial Intelligence (AI) and Machine Learning (ML). Vendors quickly packaged “AI” offerings, often pre-existing call-center-related scripts and chatbots, the door to the CIOs’ IT team was opened, and the offerings began to mature beyond call centers and the diverse medical and micro-biological use. University and Academic AI-related research were reinforced by highly specialized and well-funded dedicated groups in companies like Google, Facebook, and IBM. Smaller organizations, collectives, and startups are also receiving funding and investments. Open AI became an important presence under many aspects and circumstances. Anyone can interact online and in real-time with its chatbot, obtaining real-life value as a work tool for many areas. When asked “please explain what Large Language Models are?”, the OpenAI chatbot responded (OpenAI, 2023): “Large Language Models (LLMs) are a type of artificial intelligence (AI) model that is designed to understand and generate human-like language. These models are trained on massive amounts of text data, such as books, articles, and web pages, and use advanced algorithms to learn the patterns and structures of language. LLMs can perform a wide range of natural language processing (NLP) tasks, such as language translation, sentiment analysis, text classification, and question-answering. They can also generate human-like text, such as short stories, news articles, and even poetry. Some examples of LLMs include GPT-3 (Generative Pre-trained Transformer 3), BERT (Bidirectional Encoder Representations from Transformers), and T5 (Text-to-Text Transfer Transformer). These models have achieved significant breakthroughs in NLP and are widely used in industry and academia for various applications.”(grammar errors found in the AI-generated text were left unchanged on purpose).

View PDFchevron_right
RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts
Marwa Solayman

Birmingham City University, ICACIn 2024 Conference, 2024

Large language models (LLMs) have significantly advanced natural language processing, excelling in areas like text generation, summarization, and question-answering. Despite their capabilities, these models face challenges when fine-tuned on small, domain-specific datasets, often struggling to generalize and deliver accurate results with unfamiliar inputs. To tackle this issue, we introduce RIRO, a novel two-layer architecture designed to improve performance in data-scarce environments. The first layer leverages advanced prompt engineering to reformulate inputs, ensuring better alignment with training data, while the second layer focuses on refining outputs to minimize inconsistencies. Through fine-tuning models like Phi-2, Falcon 7B, and Falcon 1B, with Phi-2 outperforming the others. Additionally, we introduce a benchmark using evaluation metrics such as cosine similarity, Levenshtein distance, BLEU score, ROUGE-1, ROUGE-2, and ROUGE-L. While these advancements improve performance, challenges like computational demands and overfitting persist, limiting the potential of LLMs in data-scarce, high-stakes environments such as healthcare, legal documentation, and software testing.

View PDFchevron_right
Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality
Huy Vu

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021

In human-level NLP tasks, such as predicting mental health, personality, or demographics, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within modern transformer-based language models, limiting the ability to effectively leverage transformers. Here, we provide a systematic study on the role of dimension reduction methods (principal components analysis, factorization techniques, or multi-layer auto-encoders) as well as the dimensionality of embedding vectors and sample sizes as a function of predictive performance. We first find that fine-tuning large models with a limited amount of data pose a significant difficulty which can be overcome with a pre-trained dimension reduction regime. RoBERTa consistently achieves top performance in humanlevel tasks, with PCA giving benefit over other reduction methods in better handling users that write longer texts. Finally, we observe that a majority of the tasks achieve results comparable to the best performance with just 1 12 of the embedding dimensions.

View PDFchevron_right
COMPARATIVE ANALYSIS OF TRANSFORMER BASED LANGUAGE MODELS
Computer Science & Information Technology (CS & IT) Computer Science Conference Proceedings (CSCP)

Natural language processing (NLP) has witnessed many substantial advancements in the past three years. With the introduction of the Transformer and self-attention mechanism, language models are now able to learn better representations of the natural language. These attention-based models have achieved exceptional state-of-the-art results on various NLP benchmarks. One of the contributing factors is the growing use of transfer learning. Models are pre-trained on unsupervised objectives using rich datasets that develop fundamental natural language abilities that are fine-tuned further on supervised data for downstream tasks. Surprisingly, current researches have led to a novel era of powerful models that no longer require fine-tuning. The objective of this paper is to present a comparative analysis of some of the most influential language models. The benchmarks of the study are problem-solving methodologies, model architecture, compute power, standard NLP benchmark accuracies and shortcomings.

View PDFchevron_right
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges
Jubaer Ahmad

Large Language Models (LLMs) recently demonstrated extraordinary capability, including natural language processing (NLP), language translation, text generation, question answering, etc. Moreover, LLMs are a new and essential part of computerized language processing, having the ability to understand complex verbal patterns and generate coherent and appropriate replies for the situation. Though this success of LLMs has prompted a substantial increase in research contributions, rapid growth has made it difficult to understand the overall impact of these improvements. Since a lot of new research on LLMs is coming out quickly, it is getting tough to get an overview of all of them in a short note. Consequently, the research community would benefit from a short but thorough review of the recent changes in this area. This article thoroughly overviews LLMs, including their history, architectures, transformers, resources, training methods, applications, impacts, challenges, etc. This paper be...

View PDFchevron_right
End-to-End Transformer-Based Models in Textual-Based NLP
Abir Rahali

AI

Transformer architectures are highly expressive because they use self-attention mechanisms to encode long-range dependencies in the input sequences. In this paper, we present a literature review on Transformer-based (TB) models, providing a detailed overview of each model in comparison to the Transformer’s standard architecture. This survey focuses on TB models used in the field of Natural Language Processing (NLP) for textual-based tasks. We begin with an overview of the fundamental concepts at the heart of the success of these models. Then, we classify them based on their architecture and training mode. We compare the advantages and disadvantages of popular techniques in terms of architectural design and experimental value. Finally, we discuss open research, directions, and potential future work to help solve current TB application challenges in NLP.

View PDFchevron_right
N-Grammer: Augmenting Transformers with latent n-grams
Aurko Roy

arXiv (Cornell University), 2022

Transformer models have recently emerged as one of the foundational models in natural language processing, and as a byproduct, there is significant recent interest and investment in scaling these models. However, the training and inference costs of these large Transformer language models are prohibitive, thus necessitating more research in identifying more efficient variants. In this work, we propose a simple yet effective modification to the Transformer architecture inspired by the literature in statistical language modeling, by augmenting the model with n-grams that are constructed from a discrete latent representation of the text sequence. We evaluate our model, the N-Grammer on language modeling on the C4 data-set as well as text classification on the Su-perGLUE data-set, and find that it outperforms several strong baselines such as the Transformer and the Primer. We open-source our model for reproducibility purposes in Jax 1 .

View PDFchevron_right
Overview of the Transformer-based Models for NLP Tasks
anthony gillioz

Proceedings of the 2020 Federated Conference on Computer Science and Information Systems, 2020

In 2017, Vaswani et al. proposed a new neural network architecture named Transformer. That modern architecture quickly revolutionized the natural language processing world. Models like GPT and BERT relying on this Transformer architecture have fully outperformed the previous state-of-theart networks. It surpassed the earlier approaches by such a wide margin that all the recent cutting edge models seem to rely on these Transformer-based architectures. In this paper, we provide an overview and explanations of the latest models. We cover the auto-regressive models such as GPT, GPT-2 and XLNET, as well as the auto-encoder architecture such as BERT and a lot of post-BERT models like RoBERTa, ALBERT, ERNIE 1.0/2.0.

View PDFchevron_right
The Recent Large Language Models in NLP
Avtandil Teshebaev

2023

Over the past few years, Natural Language Processing (NLP) has evolved significantly thanks to the development of large Language Models (LMs). In this paper, we present a survey of four recent language models that we believe have had a significant importance in the NLP field lately: BERT (Google), ELMo (Allen Institute), GPT-3 (OpenAI), and LLaMA (Meta AI). For each model, we analyse its architecture, the dataset on which it was trained, its performance evaluation, as well as the strengths and challenges faced by each. Our paper compares the recent Language Models and their contributions to the field of NLP, and discusses future extensions.

View PDFchevron_right

References (49)

  1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008, 2017.
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pages 4171-4186, 2019.
  3. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 2019. URL: https://cdn.openai.com/better-language-models/language_models_ are_unsupervised_multitask_learners.pdf
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
  5. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Anil Raghavan, Nikolaus Parulian, Marc-Alexandre Côté, Raphael Lachaux, Evangelos Panagiotopoulou, Andrei Sabbag, Claire Welleck, Noushad Shafique, Quentin Lhoest, Arthur Mensch, Thomas Scialom, Teven Le Scao, Sylvain Gelly, Yacine Jernite, Hervé Delaunay, and Edouard Grave. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Logesh Advani, Adam Hutchinson, Andy Jones, Hen-Wei Huang, Ja- cob Steele, Eric Noland, Kevin Patton, Evan Wang, Sasha Plibble, Maria Rodriguez, Sharon Zhou, Rewon Child, Hyeontaek Lim, Mohit Iyyer, Manan Patel, Deepthi Gopa, Joseph Poon, Allison Lyle, Harsh Mehta, Alex Shao, Jason Gray, Stas Bekman, Manuel Gomez, Arjun Karthik, Yutong He, Johann Faulkner, Ian Leyens, Gayle Ross, Emily Chen, Shuohang Wang, Yiming Yang, and Barret Zoph. PaLM: Scaling language mod- eling with Pathways. arXiv preprint arXiv:2204.02311, 2022.
  7. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, and Jakob Uszkoreit. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
  8. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bow- man. GLUE: A multi-task benchmark and analysis platform for natural language un- derstanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, 2018.
  9. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32, pages 3261-3275, 2019.
  10. Hee Suk Wang, Mingda Chen, Maarten Bosma, Yoav Artzi, Thomas Kipf, and Denny Zhou. OmniNet: Efficient Vision-Language Foundation Models via Decomposed Mul- timodal Iteration. arXiv preprint arXiv:2305.08824, 2023.
  11. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Edouard Grave, Yu- long Guo, Sparsh Mittal, Daniel Lowe, Jack W. Rae, Christopher Hallinan, Anirudh Srinivasan, Neel Nanda, Sandhini Agarwal, Mikolaj Binkowski, Aaron Hoffman, Yequan Wang, Fazle F. Karim, Brandon Roberts, Benedetta Trovato, Lucas Dixon, Myle Ott, Franziska Mika, Yunkai Zhang, Angela Fan, and Jeff Dean. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
  12. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey. arXiv preprint arXiv:2309.01029, 2023.
  13. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shorish. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623, 2021.
  14. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy consid- erations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, 2019.
  15. Shachar Maidenbaum, Shira Levy-Tzedek, Daniel-Robert Chebat, and Amir Amedi. The "eyecane," a new electronic travel aid for the blind: Technology, behavior & swift learning. Restorative Neurology and Neuroscience, 32(6):813-824, 2014.
  16. Jian Bai, Shiguo Lian, Zhen Liu, Ke Wang, and Dong Liu. A CNN-based wear- able assistive system for visually impaired people walking outdoors. Applied Sciences, 11(21):10026, 2021.
  17. Shahid Munir Shah, Syeda Anshrah Gillani, Mirza Samad Ahmed Baig, Muhammad Aamer Saleem, and Muhammad Hamzah Siddiqui. Advancing depression detection on social media platforms through fine-tuned large language models. arXiv preprint arXiv:2409.14794, 2024.
  18. Mirza Samad Ahmed Baig, Syeda Anshrah Gillani, Shahid Munir Shah, Mahmoud Al- jawarneh, Abdul Akbar Khan, and Muhammad Hamzah Siddiqui. AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating real-time object recog- nition and contextual understanding using large vision-language models. arXiv preprint arXiv:2412.20059, 2024.
  19. Mirza Samad Ahmed Baig, Syeda Anshrah Gillani, Abdul Akbar Khan, and Shahid Munir Shah. AttentionDrop: A novel regularization method for Transformer models. arXiv preprint arXiv:2504.12088, 2025.
  20. Danxia Liu, Xing Lin Feng, Farooq Ahmed, Muhammad Shahid, and Jing Guo. De- tecting and measuring depression on social media using a machine learning approach: Systematic review. JMIR Mental Health, 9(3):e28968, 2022.
  21. Muhammad Rizwan, Muhammad Farooq Mushtaq, Usman Akram, Adeel Mehmood, Imran Ashraf, and Beatriz Sahelices. Depression classification from tweets using small deep transfer learning language models. IEEE Access, 10:129176-129189, 2022.
  22. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Trans- actions of the Association for Computational Linguistics, 8:64-77, 2020.
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715-1725, 2016.
  24. Taku Kudo and John Richardson. SentencePiece: A simple and language indepen- dent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, 2018.
  25. Jack Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  26. He Suk Wang, Yinfei Yang, Jared Kaplan, Yi Tay, and Colin Raffel. Scaling mixture-of- experts to 1.2 trillion parameters. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1873-1887, 2022.
  27. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
  28. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. arXiv preprint arXiv:1701.06538, 2017.
  29. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  30. Roy Schwartz, Maarten Sap, Nazneen Fatema Rajani, David A. Ferrucci, and Yejin Choi. The right to explanation for automated decisions: What is it and when does it apply? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 537-549, 2020.
  31. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehen- sion systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, 2017.
  32. Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
  33. Jean-Baptiste Alayrac, Tom Duerig, Alexey Romanov, Natasha Jaques, Gabriel Syn- naeve, Jakob Foerster, and Ivan Krasin. Flamingo: A Visual Language Model for Few- Shot Learning. Advances in Neural Information Processing Systems, 35:23778-23794, 2022.
  34. Valerio Lomonaco, Lorenzo Pellegrini, Luigi Gallo, and Davide Maltoni. Continual learning for robotics: Definition, framework, and evaluation strategies. IEEE Robotics and Automation Letters, 6(2):1951-1958, 2021.
  35. Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, 10(2):12:1-12:19, 2019.
  36. Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318, 2016.
  37. John Cotra, Fashina Atiba, Jamie Putnam, and Alex Cowen. Self-explainable neural networks for interpretability. arXiv preprint arXiv:2302.07962, 2023.
  38. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Chelsea Nash, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Scott Gray, and Ilya Sutskever. Robust speech recog- nition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022.
  39. Wei Xu, Chris Callison-Burch, and Bill Dolan. Problems in current text simplification research: New data can help. In Proceedings of the 9th International Conference on Language Resources and Evaluation, pages 452-459, 2015.
  40. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. DialogPT: Large-scale generative pre- training for conversational response generation. arXiv preprint arXiv:1911.00536, 2020.
  41. Necati Cihan Camgöz, Oscar Camgöz, Florian Metze, and Raymond Mooney. Sign language translation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 7-8, 2020.
  42. Amila Madzharevic, Armin Gulan, and Admir Bajramovic. Health chatbot for med- ication adherence. In Proceedings of the 14th International Conference on Ubiquitous Information Management and Communication, pages 1-6, 2021.
  43. Jaideep Singh, Pradeep Kumar, and Karan Singh. Deep-learning-based personalized education using GPT architectures. IEEE Transactions on Learning Technologies, 15(2):202-213, 2022.
  44. Paolo Bordone, Lucia Ferroni, Luca Rossi, and Silvia Torta. Assistive conversational agents for memory support in dementia care. International Journal of Human-Computer Interaction, 38(7):651-663, 2022.
  45. Yichen Liu, Jinman Kim, and Lina Zeng. AML-aware fall detection using multimodal sensing and natural language prompts. Sensors, 23(3):1124, 2023.
  46. Shanying Lin, Rik Koncel-Kedziorski, Pearl Pu, Premkumar Natarajan, and William W. Cohen. CommonGen: A constrained text generation challenge for generative com- monsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1680-1692, 2020.
  47. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, 2016.
  48. Amanpreet Singh, Yejin Choi, Percy Liang, and Bo Pang. Beyond the I.I.D. assumption: Inducing generalization in language models through adversarial domain adaptation. arXiv preprint arXiv:2205.08461, 2022.
  49. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.

Related papers

Large Language Models: Advances, Challenges, and Future Directions
WARSE The World Academy of Research in Science and Engineering

International Journal of Advanced Trends in Computer Science and Engineering , 2025

Large Language Models (LLMs) are artificial intelligence (AI) tools transforming Natural Language Processing (NLP) and allowing unparalleled abilities in text generation, translation, summarisation, and so on. Models such as Generative Pre-training Transformer (GPT-4), Gemini, Bidirectional Encoder Representations from Transformer (BERT), Claude, Mixtral, Falcon, Pathways Language Model (PaLM), Large Language Model Meta AI (LLaMA) and DeepSeek-R1 have shown outstanding performance in diverse tasks, propelled by advances in architecture, scale, and training methodologies Though, the development of LLMs imposed critical challenges such as ethical concerns, computational costs, and reasoning and generalisation restrictions. This paper exposes a comprehensive overview of current advances in LLMs, debates main challenges, and suggest future directions for research and development. By giving insight from recent studies, we aim to highlight the potential of LLMs by addressing the serious issues that must be resolved to protect responsible and effective deployment.

View PDFchevron_right
Overview of the progression of state-of-the-art language models
TELKOMNIKA JOURNAL

TELKOMNIKA Telecommunication Computing Electronics and Control, 2024

This review provides a concise overview of key transformer-based language models, including bidirectional encoder representations from transformers (BERT), generative pre-trained transformer 3 (GPT-3), robustly optimized BERT pretraining approach (RoBERTa), a lite BERT (ALBERT), text-to-text transfer transformer (T5), generative pre-trained transformer 4 (GPT-4), and extra large neural network (XLNet). These models have significantly advanced natural language processing (NLP) capabilities, each bringing unique contributions to the field. We delve into BERT's bidirectional context understanding, GPT-3's versatility with 175 billion parameters, and RoBERTa's optimization of BERT. ALBERT emphasizes model efficiency, T5 introduces a text-to-text framework, and GPT-4, with 170 trillion parameters, excels in multimodal tasks. Safety considerations are highlighted, especially in GPT-4. Additionally, XL-Net's permutation-based training achieves bidirectional context understanding. The motivations, advancements, and challenges of these models are explored, offering insights into the evolving landscape of large-scale language models. This is an open access article under the CC BY-SA license.

View PDFchevron_right
An Exhaustive Analysis Of Large Language Models
Tejesvi A Prasad

Large Language Models (LLMs) have revolutionized artificial intelligence by achieving human-level performance in language understanding, generation, and reasoning. This paper provides a comprehensive technical analysis of LLMs, covering their architectural foundations, training methodologies, evaluation metrics, ethical implications, and applications. We explore the evolution from statistical models to transformer-based architectures, emphasizing breakthroughs such as multi-head attention, parameter scaling, and efficiency innovations like sparse attention. Critical challenges, including bias, energy consumption, and factual hallucinations, are analyzed alongside solutions such as neuro-symbolic integration and quantization. The paper concludes with strategic recommendations for researchers and practitioners to address scalability, fairness, and sustainability.

View PDFchevron_right
Comprehensive Review of Large Language Models and its Applications
Pranjali Bahalkar

Nanotechnology Perfection, 2024

In the realm of contemporary academic discussions and scholarly conversations, it has become increasingly evident that Large Language Models (LLMs), which are sophisticated artificial intelligence systems designed for understanding and generating human language, have showcased remarkable and exceptional levels of proficiency across a wide variety of complex and intricate natural language processing (NLP) tasks that are crucial to various applications. This manuscript, which serves as a comprehensive and thorough examination of the field of Large Language Models (LLMs), aims to delve deeply into and investigate the latest and most significant innovations, advancements, and breakthroughs that have emerged within this rapidly evolving discipline and area of study. To begin this exploration, we will carefully elucidate and clarify the fundamental principles and core concepts that serve as the foundational underpinnings of these advanced models, and then we will proceed to categorize them systematically according to their unique architectural compositions and structural characteristics. Furthermore, a detailed and comparative analysis of the dominant methodologies employed in this field will be conducted, which will involve a careful delineation of their respective advantages and disadvantages, particularly in relation to their architectural design features and the empirical results they yield in practical applications. In addition to this, we will investigate and explore prospective avenues for further research and potential domains that could benefit from future inquiry and exploration within this fascinating and significant area of study. Finally, the manuscript will critically evaluate and assess the practical implications of the performance and assessment of LLMs, while also considering their environmental impact and the broader implications of their deployment in real-world scenarios.

View PDFchevron_right
Large Margin Neural Language Model
Jiaji Huang

Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

Neural language models (NLMs) are generative, and they model the distribution of grammatical sentences. Trained on huge corpus, NLMs are pushing the limit of modeling accuracy. Besides, they have also been applied to supervised learning tasks that decode text, e.g., automatic speech recognition (ASR). By re-scoring the n-best list, NLM can select grammatically more correct candidate among the list, and significantly reduce word/char error rate. However, the generative nature of NLM may not guarantee a discrimination between "good" and "bad" (in a task-specific sense) sentences, resulting in suboptimal performance. This work proposes an approach to adapt a generative NLM to a discriminative one. Different from the commonly used maximum likelihood objective, the proposed method aims at enlarging the margin between the "good" and "bad" sentences. It is trained end-to-end and can be widely applied to tasks that involve the re-scoring of the decoded text. Significant gains are observed in both ASR and statistical machine translation (SMT) tasks. Perplexity (PPL) is a commonly adopted metric to measure the quality of an LM. It is exponentiated per-symbol negative log-likelihood, PPL def == exp {−E [log p(s i |s i−1 , s i−2 ,. .. , s 0)]} , where the expectation E is taken with respect to all the symbols. A good language model has a small PPL, being able to assign higher likelihoods to sentences that are more likely to appear. N-gram models (Chen & Goodman, 1996) assume that each symbol depends on the previous N − 1 symbols. This restrictive assumption is also seen in LMs that are based on feed forward network (Bengio et al., 2003). To model longer-term dependencies, recurrent neural networks (e.g., Mikolov et al., 2010) are adopted. Recurrent neural language models (NLMs) often achieve smaller PPLs than N-gram models (

View PDFchevron_right

Related topics

  • Artificial Intelligence
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved