Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction 1
Model/Methodology
Overview
Results
Dataset
Datasets
References
All Topics
Computer Science
Artificial Intelligence

A survey of evolution of image captioning techniques

Profile image of Shivali GoelShivali Goel

2018, International Journal of Hybrid Intelligent Systems

Abstract

Automatic captioning of Images has been explored extensively in the past 10 to 15 years. It is one of the elementary problems in Computer Vision and Natural Language Processing and has vast array of applications in the real world. In this survey, we aim to study different approaches used for the generation of image captions in a chronological manner starting from the basic template based caption generation model to using Neural Networks combined with external world knowledge. We review existing models in detail, highlighting the involved methodologies and improvements in the same that have occurred in time. We gave an overview to the standard image datasets and the evaluation measures developed to discern the quality of generated image captions. Apart from the basic benchmarks we also note speed and accuracy improvements in all the different approaches. Finally, we investigate further possibilities in automatic image caption generation.

Related papers

Review on latest approaches used in Natural Language Processing for generation of Image Captioning
Madhuri Bhalekar

International Journal of Computer Science and Engineering, 2017

Recently the area of image captioning has received a lot of attention from researchers and academia. Image caption generation area has received attentions since the development of Deep Learning. Automatically generating caption from an image is done by integrating the domain of computer vision and natural language processing. Describing the content of an image is inherently a natural language processing and computer vision task. Many image captioning systems have shown that it is possible to describe the most salient information conveyed by images with accurate and meaningful sentences. This paper gives a survey about different recent approaches that have been used for image captioning with discussing the datasets been used.

View PDFchevron_right
An Extensive Analysis of Image Captioning Models, Evaluation Measures, and Datasets
sushma jaiswal

International journal of computer engineering in research trends, 2023

The difficult interdisciplinary endeavour of creating insightful and detailed captions for images is known as "image captioning," and it lies at the nexus of computer vision and natural language processing. We give a comprehensive examination of datasets, evaluation metrics, and image captioning models in this paper. We present a thorough review of popular image captioning models, from conventional methods to the most recent developments utilizing deep learning and attention mechanisms. We examine the design, underlying assumptions, and capabilities of these models, emphasizing how they help produce logical and contextually appropriate captions. Furthermore, we analyses in detail well-known evaluation metrics as BLEU, METEOR, ROUGE, and CIDEr, clarifying their importance in evaluating generated caption quality against ground truth references. Additionally, we talk about the critical role that datasets play in image captioning research, with particular attention to prominent datasets like as COCO, Flickr30k, and Conceptual Captions. We investigate these dataset's diversity, volume, and annotations, emphasizing their impact on model evaluation and training. Our objective is to furnish scholars, professionals, and amateurs with an invaluable tool for comprehending the state of image captioning, so facilitating the creation of inventive models and enhanced assessment techniques.

View PDFchevron_right
A Survey of Various Image Captioning Methods
Abhipsha Das

Image Captioning is an important area of research that lies at the intersection of computer vision and natural language processing. It is important because it provides insight for humans to better comprehend the perception that machines develop when they decipher an image to generate descriptions of it. It encompasses a machine’s understanding of both: • which features (or pixels) represent which objects in the image, and • what the network should derive about the context in which the objects in the image are represented. The latter is more scientifically termed as the language grounding problem, because an object, while differing in function in different circumstances, should be understood at a common level without revolving around its specifics. Establishing this grounding for machines is what is known as the language grounding problem. If this problem is successfully solved, then it will explain a lot of the decision making process that a machine undergoes to develop an understanding of the scene represented in the image. This holds valid because an object, while being classified, must also be described in relation to the other objects present with it. Objects must also be described in a contextual sense for achieving human-like caption generation, in order to get closer to really artificially intelligent machines. Image captioning methods are further adapted into video captioning, which is a natural extension of the former. In this report, we explore and summarize five of the most relevant and popular algorithms developed by researchers over the years towards generating finer and more accurate image captions.

View PDFchevron_right
Capturing the Essence: An In-Depth Exploration of Automatic Image Captioning Techniques and Advancements
sushma jaiswal

International journal of computer engineering in research trends, 2023

This comprehensive overview offers a deep dive into the quickly developing topic of automatic image captioning. It provides a thorough examination of the many approaches, datasets, and assessment criteria applied in the production of written descriptions from visual content. With an emphasis on the extensively used MS COCO dataset, the article explores the critical significance of datasets in addition to exploring the nuances of cutting-edge image captioning algorithms. It also goes over the fundamental function of assessment metrics, stressing the importance of BLEU, METEOR, ROUGE, CIDEr, and SPICE metrics in determining the caliber of created captions. This study is an invaluable tool for scholars and practitioners looking to improve the field because it offers a thorough analysis of the developments and difficulties in this field. To be more precise, we start by quickly going over the earlier traditional works using the retrieval and template. After that, research on deep learning (DL)-based image captioning is concentrated. For a thorough overview, these studies are divided into three categories: the encoder-decoder architecture, the attention techniques, and training techniques based on framework structures and training methods. The publicly accessible datasets, evaluation metrics, and those suggested for particular requirements are then summarized, and the state-of-the-art techniques are then contrasted using the MS COCO dataset, we offer a few talks about unresolved issues and potential avenues for future research.

View PDFchevron_right
Automatic Image Caption Generation Based on Some Machine Learning Algorithms
Daša Manić

Mathematical Problems in Engineering

This paper is dedicated to machine learning, the branches of machine learning, which include the methods for solving this issue, and the practical implementation of the solution to the automatic image description generation. Automatic image caption generation is one of the frequent goals of computer vision. Image description generation models must solve a larger number of complex problems to have this task successfully solved. The objects in the image must be detected and recognized, after which a logical and syntactically correct textual description is generated. For that reason, description generation is a complex problem. It is an extremely important challenge for machine learning algorithms because it represents an impersonation of a complicated human ability to encapsulate huge amounts of highlighted visual pieces of information in descriptive language. The results of the generated descriptions are compared depending on the used pretrained convolutional networks. The BLEU metri...

View PDFchevron_right
A comprehensive survey on automatic image captioning-deep learning techniques, datasets and evaluation parameters
International Journal of Electrical and Computer Engineering (IJECE)

International Journal of Electrical and Computer Engineering (IJECE) , 2025

Automatic image captioning is a pivotal intersection of computer vision and natural language processing, aiming to generate descriptive textual content from visual inputs. This comprehensive survey explores the evolution and state-of-the-art advancements in image caption generation, focusing on deep learning techniques, benchmark datasets, and evaluation parameters. We begin by tracing the progression from early approaches to contemporary deep learning methodologies, emphasizing encoder-decoder based models and transformer-based models. We then systematically review the datasets that have been instrumental in training and benchmarking image captioning models, including MSCOCO, Flickr30k, Flickr8k, and PASCAL 1k, discussing image count, types of scenes, and sources. Furthermore, we delve into the evaluation metrics employed to assess model performance, such as bilingual evaluation understudy (BLEU), metric for evaluation of translation with explicit ordering (METEOR), recall-oriented understudy for gisting evaluation (ROUGE), and consensus-based image description evaluation (CIDEr), analyzing their domains, bases, and measurement criteria. Through this survey, we aim to provide a detailed understanding of the current landscape, identify challenges, and propose future research directions in automatic image captioning.

View PDFchevron_right
Automatic Image and Video Caption Generation With Deep Learning: A Concise Review and Algorithmic Overlap
Soheyla Amirian

IEEE Access

Methodologies that utilize Deep Learning offer great potential for applications that automatically attempt to generate captions or descriptions about images and video frames. Image and video captioning are considered to be intellectually challenging problems in imaging science. The application domains include automatic caption (or description) generation for images and videos for people who suffer from various degrees of visual impairment; the automatic creation of metadata for images and videos (indexing) for use by search engines; general-purpose robot vision systems; and many others. Each of these application domains can positively and significantly impact many other task-specific applications. This article is not meant to be a comprehensive review of image captioning; rather, it is a concise review of both image captioning and video captioning methodologies based on deep learning. This study treats both image and video captioning by emphasizing the algorithmic overlap between the two. INDEX TERMS Deep learning, image captioning, video captioning, long short term memory, generative adversarial network.

View PDFchevron_right
Automated Image Caption Generator Using Deep Learning
IJRASET Publication

International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2025

One of the most important tasks in computer vision and natural language processing is the automatic creation of image captions.. This paper presents an approach to automatically generate descriptive captions for images by combining Convolutional Neural Networks (CNNs) and Inception V3 architecture. The proposed system utilizes a pre-trained Inception V3 model to extract high-level features from input images. These extracted features are then passed to a Recurrent Neural Network (RNN), specifically an LSTM (Long Short-Term Memory) network, to generate coherent and contextually relevant captions. Inception V3, a deep convolutional neural network designed for large-scale image classification, serves as the feature extractor. It helps capture rich spatial hierarchies within the images, making it highly effective for understanding complex visual information. The LSTM network, on the other hand, is used to model the sequence of words in the caption, ensuring grammatical correctness and semantic accuracy. The system is trained on a large dataset of images paired with humangenerated captions, such as the MS-COCO dataset, to ensure robust learning. The proposed method is evaluated based on its performance in generating captions that are semantically and syntactically appropriate. The model's performance is compared to other existing image captioning methods, demonstrating its effectiveness in generating descriptive and accurate captions for unseen images. This work highlights the synergy between CNNs for visual feature extraction and LSTM networks for sequence generation, offering a promising solution for tasks requiring image-to-text conversion, including image retrieval, content-based indexing, and accessibility applications.

View PDFchevron_right
A Hybrid Model for Combining Neural Image Caption and k-Nearest Neighbor Approach for Image Captioning
Seba Susan

2021

A hybrid model is proposed that integrates two popular image captioning methods to generate a text-based summary describing the contents of the image. The two image captioning models are the Neural Image Caption (NIC) and the k-nearest neighbor approach. These are trained individually on the training set. We extract a set of five features, from the validation set, for evaluating the results of the two models that in turn is used to train a logistic regression classifier. The BLEU-4 scores of the two models are compared for generating the binary-value ground truth for the logistic regression classifier. For the test set, the input images are first passed separately through the two models to generate the individual captions. The five-dimensional feature set extracted from the two models is passed to the logistic regression classifier to take a decision regarding the final caption generated which is the best of two captions generated by the models. Our implementation of the k-nearest-n...

View PDFchevron_right
IRJET- Image Caption Generation Methodologies
IRJET Journal

IRJET, 2021

Scene understanding has always been an important task in computer vision, and image captioning is one of the major areas of Artificial intelligence research since it aims to mimic the human ability to compress an enormous amount of visual information in a few sentences. Image caption generation aims to generate a sentence description for an image. The task aims to provide short but detailed caption of the image and requires the use of techniques from computer vision and natural language processing. Recent developments in deep learning and the availability of image caption datasets such as Flickr and COCO have enabled significant research in the area. In this paper, we propose methodologies used such as multilayer Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and a Long Short Term Memory (LSTM) to accurately identify and construct meaningful caption for a given image.

View PDFchevron_right

References (127)

  1. G.A. Miller, WordNet: A lexical database for English, Com- munications of the ACM 38(11) (1995), 39-41.
  2. S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural computation, 1997.
  3. I. Langkilde and K. Knight, Generation that exploits corpus- based statistical knowledge, Proceedings of the 36th ACL, 1998.
  4. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, Gradient- based learning applied to document recognition, Proc IEEE 86(11) (1998), 2278-2324.
  5. E. Reiter and R. Dale, Building Natural Language Genera- tion Systems, Cambridge University Press, 2000.
  6. K. Papineni, S. Roukos, T. Ward and W.-J. Zhu, Bleu: A method for automatic evaluation of machine translation, In ACL, 2002, pp. 311-318.
  7. K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D.M. Blei and M.I. Jordan, Matching words and pictures, JMLR, 2003.
  8. C.-Y. Lin and F.J. Och, Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics, In ACL, 2004, p. 605.
  9. N. Dalal and B. Triggs, Histograms of oriented gradients for human detections, Proceedings of CVPR, 2005.
  10. Y. Bengio, H. Schwenk, J.-S. Senecal, F. Morin and J.-L. Gauvain, Neural probabilistic language models, Innovations in Machine Learning, Springer, 2006.
  11. M. Grubinger, P. Clough, H. Muller and T. Deselaers, The iapr tc-12 benchmark: A new evaluation resource for visual information systems.
  12. L.-J. Li and F.-F. Li, What, where and who? classifying events by scene and object recognition, ICCV, 2007.
  13. L.-J. Li, R. Socher and F.F. Li, Towards total scene un- derstanding: Classification, annotation and segmentation in an automatic framework. In Computer Vision and Pattern Recognition. CVPR. IEEE Conference on, IEEE, 2009, pp. 2036-2043.
  14. S. Gould, R. Fulton and D. Koller, Decomposing a scene into geometric and semantically consistent regions, In Computer Vision, IEEE 12th International Conference on, IEEE, 2009, pp. 1-8.
  15. E. Reiter and A. Belz, An investigation into the validity of some metrics for automatically evaluating natural lan- guage generation systems, Computational Linguistics 35(4) (2009), 529-558.
  16. A. Farhadi, I. Endres, D. Hoiem and D. Forsyth, Describing objects by their attributes, Proceedings of CVPR, 2009.
  17. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. FeiFei, Imagenet: A large-scale hierarchical image database, In Computer Vision and Pattern Recognition (2009). CVPR (2009). IEEE Conference on, IEEE, 2009, pp. 248-255.
  18. A. Aker and R. Gaizauskas, Generating image descriptions using dependency relational patterns, In ACL, 2010.
  19. A. Farhadi, M. Hejrati, M.A. Sadeghi, P. Young, C.
  20. Rashtchian, J. Hockenmaier and D. Forsyth, Every picture 825 tells a story: Generating sentences from images. In ECCV, 826 2010. 827
  21. C. Rashtchian, P. Young, M. Hodosh and J. Hockenmaier,
  22. R. Socher and F.-F. Li, Connecting modalities: Semi-su- 836 pervised segmentation and annotation of images using un- 837 aligned text corpora, In CVPR, 2010.
  23. A. Farhadi, M. Hejrati, M.A. Sadeghi, P. Young, C.
  24. Rashtchian, J. Hockenmaier and D. Forsyth, Every picture 840 tells a story: Generating sentences from images, In ECCV, 841 2010.
  25. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky'and S. Khu- 843 danpur, Recurrent neural network based language model, In 844 INTERSPEECH, 2010.
  26. C. Callison-Burch and M. Dredze, Creating speech and lan- 846 guage data with Amazons Mechanical Turk. NAACL 2010
  27. 847 Workshop on Creating Speech and Language Data with 848 Amazons Mechanical Turk, 2010.
  28. B.Z. Yao, X. Yang, L. Lin, M.W. Lee and S.-C. Zhu, I2T: 850 Image parsing to text description, Proceedings of IEEE 98(8) 851 (2010), 1485-1508.
  29. 852 [27] l. Petrov, Berkeley parser. GNU General Public License v.2, 853 2010. 854
  30. A. Farhadi, M. Hejrati, M.A. Sadeghi, P. Young, C.
  31. Rashtchian, J. Hockenmaier and D. Forsyth, Every picture 856 tells a story: Generating sentences from images, In ECCV, 857 Springer, 2010, pages 15 to 29.
  32. B.Z. Yao, X. Yang, L. Lin, M.W. Lee and S.-C. Zhu, I2T: 859 Image parsing to text description, Proceedings of the IEEE 860 98(8) (2010), 1485-1508.
  33. C. Rashtchian, P. Young, M. Hodosh and J. Hockenmaier, 862 Collecting image annotations using Amazons mechanical 863 turk, In NAACL HLT Workshop Creating Speech and Lan- 864 guage Data with Amazons Mechanical Turk, 2010.
  34. A. Aker and R. Gaizauskas, Generating image descriptions 866 using dependency relational patterns, In ACL, 2010.
  35. R. Kiros and R.Z.R. Salakhutdinov, Multimodal neural lan- 868 guage models, In NIPS Deep Learning Workshop, 2013.
  36. S. Li, G. Kulkarni, T.L. Berg, A.C. Berg and Y. Choi, Com- 870 posing simple image descriptions using web-scalen-grams, 871 In Conference on Computational Natural Language Learn- 872 ing, 2011.
  37. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A.C. Berg 874 and T.L. Berg, Baby talk: Understanding and generating sim- 875 ple image descriptions, In CVPR, 2011.
  38. Y. Jia, M. Salzmann and T. Darrell, Learning cross-modality 877 similarity for multinomial data. In ICCV, 2011.
  39. Y. Yang, C.L. Teo, H. Daume, III. and Y. Aloimonos, 879 Corpus-guided sentence generation of natural images, In 880 EMNLP, 2011.
  40. V. Ordonez, G. Kulkarni and T.L. Berg, Im2text: Describ- 882 Ing images using 1 million captioned photographs. In NIPS, 883 2011.
  41. I. Sutskever, J. Martens and G.E. Hinton, Generating text 885 with recurrent neural networks, In ICML, 2011.
  42. Amazon, Amazon mechanical turk: Artificial artificial intel- ligence, 2011.
  43. Y. Yang, C.L. Teo, H. Daume, III. and Y. Aloimonos, Corpus-guided sentence generation of natural images, Pro- ceedings of EMNLP, 2011.
  44. S. Li, G. Kulkarni, T.L. Berg, A.C. Berg and Y. Choi, Com- posing simple image descriptions using web-scale n-grams. Proceedings of CoNLL, 2011.
  45. V. Ordonez, G. Kulkarni and T.L. Berg, Im2text:Describing images using 1 million captioned photographs. Proceedings of NIPS, 2011.
  46. Flickr, http://www.flickr.com. Accessed 1, 11 Sep 2011.
  47. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A.C. Berg and T L. Berg, Baby talk: Understanding and generating sim- ple image descriptions. In CVPR, IEEE, 2011, pp. 1601- 1608.
  48. Y. Yang, C.L. Teo, H. Daume, III. and Y. Aloimonos, Corpus-guided sentence generation of natural images, In EMNLP, 2011.
  49. T. Mikolovs, Recurrent neural network based language model.
  50. T. Mikolov, A. Deoras, D. Povey, L. Burget and J. Cernocky, Strategies for training large scale neural network language models. In Automatic Speech Recognition and Understand- ing (ASRU), 2011 IEEE Workshop on, IEEE, 2011, pp 196- 201.
  51. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A.C. Berg and T. Berg, Baby talk: Understanding and generating image descriptions. Proceedings of the 24th CVPR, 2011.
  52. Y. Jia, M. Salzmann and T. Darrell, Learning cross-modality similarity for multinomial data, Proc. IEEE Int.Conference Computer Vision, 2011.
  53. V. Ordonez, G. Kulkarni and T.L. Berg, Im2text: Describ- ing images using 1 million captioned photographs Proc. Ad- vances in Neural Inf. Process. Syst, 2011.
  54. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee and A. Ng, Multimodal deep learning, In ICML, 2011.
  55. P. Kuznetsova, V. Ordonez, A.C. Berg, T.L. Berg and Y. Choi, Collective generation of natural image descriptions, In ACL, 2012.
  56. Y. Feng and M. Lapata, Automatic Caption Generation for News Images, IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (2012), 10.1109/TPAMI.2012.118.
  57. A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickin- son, S. Fidler, A. Michaux, S. Mussman, S. Narayanaswamy, D. Salvi et al., Video in sentences out. arXiv preprint arXiv:1204.2742, 2012.
  58. A. Gupta and P. Mannem, From image annotation to im- age description, In Neural information processing. Springer, 2012.
  59. A. Krizhevsky, I. Sutskever and G.E. Hinton, Imagenet clas- sification with deep convolutional neural networks, In NIPS, 2012.
  60. A. Krizhevsky, I. Sutskever and G.E. Hinton, Imagenet clas- sification with deep convolutional neural networks, In NIPS, 2012.
  61. A. Krizhevsky, I. Sutskever and G.E. Hinton, Imagenet clas- sification with deep convolutional neural networks, In Ad- vances in neural information processing systems, 2012, pp. 1097-1105.
  62. M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos and H. Daume, III., Midge: Generating image descriptions from computer vision
  63. A. Kumar and S. Goel / A survey of evolution of image captioning techniques detections, In EACL, Association for Computational Lin- 950 guistics, 2012, pp. 747-756.
  64. A. Gupta, Y. Verma and C. Jawahar, Choosing linguistics 952 over vision to describe images, In AAAI, 2012.
  65. P. Kuznetsova, V. Ordonez, A.C. Berg, T.L. Berg and Y.
  66. 954 Choi, Collective generation of natural image descriptions, 955 2012.
  67. N. Srivastava and R. Salakhutdinov, Multimodal learning 957 with deep boltzmann machines, In NIPS, 2012, pp. 2222- 958 2230.
  68. D. Elliott and F. Keller, Image description using visual de- 960 pendency representations, In EMNLP, 2013.
  69. A. Graves, Generating sequences with recurrent neural net- 962 works, arXiv:1308.0850, 2013.
  70. M. Hodosh, P. Young and J. Hockenmaier, Framing image 964 description as a ranking task: Data, models and evaluation 965 metrics, JAIR 47 (2013).
  71. R. Kiros and Z.R. Salakhutdinov, Multimodal neural lan- 967 guage models, In NIPS Deep Learning Workshop, 2013.
  72. T. Mikolov, K. Chen, G. Corrado and J. Dean, Efficient es- 969 timation of word representations in vector space, In ICLR, 970 2013. 971
  73. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus 972 and Y. LeCun, Overfeat: Integrated recognition, localization 973 and detection using convolutional networks. arXiv preprint 974 arXiv:1312.6229, 2013.
  74. S. Fidler, A. Sharma and R. Urtasun, A sentence is worth a 976 thousand pixels, In CVPR, 2013.
  75. T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado and J. Dean, 978 Distributed representations of words and phrases and their 979 compositionality, In NIPS, 2013.
  76. M. Hodosh, P. Young and J. Hockenmaier, Framing image 981 description as a ranking task: data, models and evaluation 982 metrics. Journal of Artificial Intelligence Research, 2013.
  77. D. Elliott and F. Keller, Image description using visual de- 984 pendency representations, In EMNLP, 2013, pp. 1292-1302.
  78. A. Frome, G.S. Corrado, J. Shlens, S. Bengio, J. Dean, T. 986 Mikolov et al., Devise: A deep visual-semantic embedding 987 model. In NIPS, 2013.
  79. A. Frome, G.S. Corrado, J. Shlens, S. Bengio, J. Dean, T. 989 Mikolov et al., Devise: A deep visual-semantic embedding 990 model. In Advances in Neural Information Processing Sys- 991 tems, 2013, pp. 2121-2129.
  80. C.L. Zitnick and D. Parikh, Bringing semantics into focus 993 using visual abstraction, In Computer Vision and Pattern 994 Recognition(CVPR), IEEE Conference on, IEEE, 2013, pp. 995 3009-3016.
  81. R. Socher, Q. Le, C. Manning and A. Ng, Grounded com- 997 positional semantics for finding and describing images with 998 sentences, In NIPS Deep Learning Workshop, 2013.
  82. M. Hodosh, P. Young and J. Hockenmaier, Framing image 1000 description as a ranking task: Data, models and evaluation 1001 metrics, J. Artif. Intell. Res. (JAIR) 47 (2013), 853-899.
  83. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. 1003 Tzeng and T. Darrell, Decaf: A deep convolutional activation 1004 feature for generic visual recognition. 1005
  84. N. Kalchbrenner and P. Blunsom, Recurrent continuous 1006 translation models, In EMNLP, 2013.
  85. D. Bahdanau, K. Cho and Y. Bengio, Neural machine 1008 translation by jointly learning to align and translate, 1009 arXiv:1409.0473, 2014.
  86. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. 1011 Schwenk and Y. Bengio, Learning phrase representations us- 1012 ing RNN encoder-decoder for statistical machine translation, In EMNLP, 2014.
  87. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng and T. Darrell, Decaf: A deep convolutional activation feature for generic visual recognition, In ICML, 2014.
  88. Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier and S. Lazebnik, Improving image-sentence embeddings using large weakly annotated photo collections, In ECCV, 2014.
  89. A. Karpathy, A. Joulin and L. Fei-Fei, Deep fragment em- beddings for bidirectional image sentence mapping, NIPS, 2014.
  90. R. Kiros, R. Salakhutdinov and R.S. Zemel, Unifying visual- semantic embeddings with multimodal neural language models, In arXiv:1411.2539, 2014.
  91. Kuznetsova, V. Ordonez, T. Berg and Y. Choi, Treetalk: Composition and compression of trees for image descrip- tions, ACL 2(10) (2014).
  92. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Dollar and C.L. Zitnick, Microsoft coco: Common objects in context, arXiv:1405.0312, 2014.
  93. J. Mao, W. Xu, Y. Yang, J. Wang and A. Yuille, Ex- plain images with multimodal recurrent neural networks, In arXiv:1411090, 2014.
  94. R. Socher, A. Karpathy, Q.V. Le, C.D. Manning and A.Y. Ng, Grounded compositional semantics for finding and de- scribing images with sentences, TACL, 2014.
  95. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A.C. Berg and L. Fei-Fei, Image Net Large Scale Visual Recogni- tion Challenge, 2014.
  96. R. Socher, A. Karpathy, Q.V. Le, C. Manning and A.Y. Ng, Grounded compositional semantics for finding and describ- ing images with sentences, In ACL, 2014.
  97. R.J. Pennington and C. Manning, Glove: Global vectors for word representation, 2014.
  98. O.V. Sutskever and Q.V. Le, Sequence to sequence learning with neural networks, In NIPS, 2014.
  99. P. Young, A. Lai, M. Hodosh and J. Hockenmaier, From im- age descriptions to visual denotations: New similarity met- rics for semantic inference over event descriptions, In ACL, 2014.
  100. W. Zaremba, I. Sutskever and O. Vinyals, Recurrent neural network regularization, In arXiv:1409.2329, 2014.
  101. X. Chen and C.L. Zitnick, Learning a recurrent vi- sual representation for image caption generation, CoRR, abs/1411.5654, 2014.
  102. O. Vinyals, A. Toshev, S. Bengio and D. Erhan, Show and tell: A neural image caption generator, arXiv preprint arXiv:1411.4555, 2014.
  103. H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dol- lar, J. Gao, X. He, M. Mitchell, J. Platt et al., From captions to visual concepts and back, arXiv preprint arXiv:1411.4952, 2014.
  104. P. Sermane, D. Eigen, X. Zhang, M. Mathieu, R. Fergus and Y. LeCun, OverFeat: Integrated recognition, localization and detection using convolutional networks, ICLR, 2014.
  105. C. Szegedy, S. Reed, D. Erhan and D. Anguelov, Scalable, high-quality object detection, arXiv preprint arXiv:1412.1441, 2014.
  106. A. Karpathy, A. Joulin and F.F. Li, Deep fragment embed- dings for bidirectional image sentence mapping in Proc, Ad- vances in Neural Inf. Process. Syst, 2014.
  107. O. Vinyals, A. Toshev, S. Bengio and D. Erhan, Show and u n tell: A neural image caption generator in Proc, IEEE Conf. Comp. Vis. Patt. Recogn, 2014.
  108. R. Girshick, J. Donahue, T. Darrell and J. Malik, Rich fea- ture hierarchies for accurate object detection and semantic segmentation, 2014.
  109. R. Kiros, R. Salakhutdinov and R. Zemel, Multimodal neural language models, In ICML, 2014.
  110. J. Mao, W. Xu, Y. Yang, J. Wang and A.L. Yuille, Explain images with multimodal recurrent neural networks, arXiv preprint arXiv:1410.1090, 2014.
  111. A. Karpathy, A. Joulin and L. Fei-Fei, Deep fragment em- beddings for bidirectional image sentence mapping, arXiv preprint arXiv:1406.5679, 2014.
  112. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama and T. Darrell, Caffe: Convolu- tional architecture for fast feature embedding, arXiv preprint arXiv:1408.5093, 2014.
  113. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Dollar and C.L. Zitnick, Microsoft coco: Common objects in context, In ECCV, 2014.
  114. P. Blunsom, N. de Freitas, E. Grefenstette, K.M. Hermann et al., A deep architecture for semantic parsing, In ACL 2014 Workshop on Semantic Parsing, 2014.
  115. S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, In arXiv:1502.03167, 2015.
  116. R. Vedantam, C.L. Zitnick and D. Parikh, CIDEr: Consensus-based image description evaluation, In arXiv:1411.5726, 2015.
  117. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A.C. Berg and L. Fei-Fei, ImageNet Large Scale Visual Recog- nition Challenge, International Journal of Computer Vision (IJCV), April 2015, pp. 1-2.
  118. X. Chen and C.L. Zitnick, Minds Eye: A Recurrent 1110 Visual Representation for Image Caption Generation in 1111 Proc.IEEEConf. Comp. Vis. Patt. Recogn, 2015. 1112
  119. J. Donahue, L.A. Hendricks, S. Guadarrama, M. Rohrbach, 1113 S. Venugopalan, K. Saenko and T. Darrell, Long-term re-1114 current convolutional networks for visual recognition and 1115 description in Proc. IEEE Conf. Comp. Vis. Patt. Recogn, 1116 2015. 1117
  120. J. Mao, W. Xu, Y. Yang, J. Wang and A. Yuille, Deep Cap-1118 tioning with Multimodal Recurrent Neural Networks (m-1119 RNN) in Proc. Int. Conf. Learn. Representations, 2015. 1120
  121. L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle 1121 and A. Courville, Describing videos by exploiting temporal 1122 structure in Proc. IEEE Int. Conf. Comp. Vis, 2015. 1123
  122. J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. 1124 He, G. Zweig and M. Mitchell, Language models for im-1125 age captioning: The quirks and what works, arXiv preprint 1126 arXiv:1505.01809, 2015. 1127
  123. K. Simonyan and A. Zisserman, Very deep convolutional 1128 networks for large-scale image recognition, in Proc. Int. 1129 Conf. Learn. Representations, 2015. 1130
  124. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. 1131 Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, Go-1132 ing deeper with convolutions, in Proc. IEEE Conf. Comp. 1133 Vis. Patt. Recogn, 2015. 1134
  125. A. Karpathy and F.-F. Li, Deep Visual-Semantic Alignments 1135 for Generating Image Descriptions arXiv:1412.2306v2, 1136 2015. 1137
  126. Q. Wu, P. Wang, C. Shen, A. Dick and A.V.D. Hengel, Ask 1138 Me Anything: Free-form Visual Question Answering Based 1139 on Knowledge from External Sources in Proc. IEEE Conf. 1140
  127. Comp. Vis. Patt. Recogn, 2016.

Related papers

Image Caption Generation Methodologies
IRJET Journal

Scene understanding has always been an important task in computer vision, and image captioning is one of the major areas of Artificial intelligence research since it aims to mimic the human ability to compress an enormous amount of visual information in a few sentences. Image caption generation aims to generate a sentence description for an image. The task aims to provide short but detailed caption of the image and requires the use of techniques from computer vision and natural language processing. Recent developments in deep learning and the availability of image caption datasets such as Flickr and COCO have enabled significant research in the area. In this paper, we propose methodologies used such as multilayer Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and a Long Short Term Memory (LSTM) to accurately identify and construct meaningful caption for a given image.

View PDFchevron_right
Automatic Image Captioning Using Neural Networks
Rama Bastola

Journal of Innovations in Engineering Education

Automatically generating a natural language description of an image is a major challenging task in the field of artificial intelligence. Generating description of an image bring together the fields: Natural Language Processing and Computer Vision. There are two types of approaches i.e. top-down and bottom-up. For this paper, we approached top-down that starts from the image and converts it into the word. Image is passed to Convolutional Neural Network (CNN) encoder and the output from it is fed further to Recurrent Neural Network (RNN) decoder that generates meaningful captions. We generated the image description by passing the real time images from the camera of a smartphone as well as tested with the test images from the dataset. To evaluate the model performance, we used BLEU (Bilingual Evaluation Understudy) score and match predicted words to the original caption.

View PDFchevron_right
Captioning the Images: A Deep Analysis
Satish Devane

Advances in Intelligent Systems and Computing, 2018

Image captioning is one of the fundamental tasks in machine learning since the ability to generate text captions of an image can have a great impact by assisting us in day-today life. However, it is not just an object classification or recognition task, because the model must know the dependencies among the recognized objects and their attributes and encode that knowledge correctly in the caption using a natural language like English. Recently, the internet is overwhelmed with the huge amount of textual and visual data consisting of billions of unstructured images and videos. Meaningful captions will serve as useful keys for retrieval, creative searching, and powerful browsing of these images. In this paper, we present the goal of analysis and classification of the recent state-of-the-art in image captioning and discuss significant differences among them. We provide a comparative review of existing models, techniques with their advantages and disadvantages. Future directions in the field of automatic image caption generation are also explored.

View PDFchevron_right
IRJET- A Survey of Image Captioning Models
IRJET Journal

IRJET, 2021

Image caption generation has been a challenging problem for a long time.Numerous attempts have been made at the difficult task of image captioning, which includes the complexities of both computer vision and natural language processing. Deep Learning models have the capability to perform the intricate task of image captioning. In this survey paper, we aim to give a complete review of the various image captioning techniques that have been implemented till date. We discuss the structure of the various models, their performance, advantages and limitations. The different datasets and evaluation metrics that are frequently used in image captioning models have also been discussed.

View PDFchevron_right
Visuals to Text: A Comprehensive Review on Automatic Image Captioning
IEEE/CAA J. Autom. Sinica

IEEE/CAA Journal of Automatica Sinica, 2022

Image captioning refers to automatic generation of descriptive texts according to the visual content of images. It is a technique integrating multiple disciplines including the computer vision (CV), natural language processing (NLP) and artificial intelligence. In recent years, substantial research efforts have been devoted to generate image caption with impressive progress. To summarize the recent advances in image captioning, we present a comprehensive review on image captioning, covering both traditional methods and recent deep learning-based techniques. Specifically, we first briefly review the early traditional works based on the retrieval and template. Then deep learning-based image captioning researches are focused, which is categorized into the encoder-decoder framework, attention mechanism and training strategies on the basis of model structures and training manners for a detailed introduction. After that, we summarize the publicly available datasets, evaluation metrics and those proposed for specific requirements, and then compare the state of the art methods on the MS COCO dataset. Finally, we provide some discussions on open challenges and future research directions.

View PDFchevron_right

Related topics

  • Cognitive Science
  • Computer Science
  • Artificial Intelligence
  • Image Analysis (Mathematics)
  • Hybrid Intelligent Systems
  • Closed Captioning
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved