Academia.eduAcademia.edu

Outline

Title
Abstract
Key Takeaways
Figures
Introduction
Related Work
Text Localization in Complex Background Documents
Experiments and Results
Dataset
Conclusion and Perspectives
References

Automatic text localisation in scanned comic books

Profile image of Christophe RigaudChristophe RigaudProfile image of Dimosthenis KaratzasDimosthenis KaratzasProfile image of Jean-christophe BurieJean-christophe BurieProfile image of Joost van de WeijerJoost van de Weijer
https://doi.org/10.5220/0004301308140819
visibility

…

description

6 pages

link

1 file

Abstract

Comic books constitute an important cultural heritage asset in many countries. Digitization combined with subsequent document understanding enable direct content-based search as opposed to metadata only search (e.g. album title or author name). Few studies have been done in this direction. In this work we detail a novel approach for the automatic text localization in scanned comics book pages, an essential step towards a fully automatic comics book understanding. We focus on speech text as it is semantically important and represents the majority of the text present in comics. The approach is compared with existing methods of text localization found in the literature and results are presented.

Key takeaways
sparkles

AI

  1. The proposed method achieves over 75.8% recall and 76% precision in text localization.
  2. Adaptive segmentation using Minimum Connected Components Thresholding (MCCT) optimizes text detection in comics.
  3. A new benchmark dataset with 1700 text lines will be publicly available for future research.
  4. Text localization improves recognition accuracy and supports applications like OCR training and image compression.
  5. The focus is on speech text, which constitutes the majority of comic text, for automatic understanding.
Figures (5)
Figure 1: Segmentation at different threshold levels from the lower top-left to the higher bottom-right (threshold = 50, 100, 150, 200). We observe that the number of CC increases when the dark lines are cut and also when background start to appear as salt and paper noise. Source: (Roudier, 2011). Therefore, we propose an adaptive segmentation method. For a single page we assume that the text background brightness is similar around all the char- acters of the same page. However, in our case, the op- timal segmentation threshold differs for every single page of comics depending on the background colour of the text areas. The method is based on the obser- vation that choosing the threshold too low, as well as choosing the threshold too high leads to an excess of connected components (CC), figure 1.
Figure 1: Segmentation at different threshold levels from the lower top-left to the higher bottom-right (threshold = 50, 100, 150, 200). We observe that the number of CC increases when the dark lines are cut and also when background start to appear as salt and paper noise. Source: (Roudier, 2011). Therefore, we propose an adaptive segmentation method. For a single page we assume that the text background brightness is similar around all the char- acters of the same page. However, in our case, the op- timal segmentation threshold differs for every single page of comics depending on the background colour of the text areas. The method is based on the obser- vation that choosing the threshold too low, as well as choosing the threshold too high leads to an excess of connected components (CC), figure 1.
Figure 6: Example of the letter validation method (rule 3). The area of the bounding box of the letter “A” is extended in four directions around the letter “A” to check if any other letter (blue rectangles) overlaps with it. In this case, there are two overlapping components (red dashed rectangles) at the right and the bottom of the letter “A”. The component is thus accepted as a possible letter.
Figure 6: Example of the letter validation method (rule 3). The area of the bounding box of the letter “A” is extended in four directions around the letter “A” to check if any other letter (blue rectangles) overlaps with it. In this case, there are two overlapping components (red dashed rectangles) at the right and the bottom of the letter “A”. The component is thus accepted as a possible letter.
Figure 7: Letter horizontal and vertical positions variables (on the left Jetter,, on the right letter2). The bounding boxes are drawn in blue and the geometric centres (cl,c2) in orange.
Figure 7: Letter horizontal and vertical positions variables (on the left Jetter,, on the right letter2). The bounding boxes are drawn in blue and the geometric centres (cl,c2) in orange.
Figure 8: This figure shows the maximum recall obtained manually (MAX) and the adaptive segmentation (AUTO) for a sample of images of the dataset. Image 10 contains more than 60% of bright text over dark background which is not detected by our algorithm.
Figure 8: This figure shows the maximum recall obtained manually (MAX) and the adaptive segmentation (AUTO) for a sample of images of the dataset. Image 10 contains more than 60% of bright text over dark background which is not detected by our algorithm.

Related papers

Method for Real Time Text Extraction of Digital Manga Comic
Herman Tolle

International Journal of Image Processing (IJIP), 2011

View PDFchevron_right
Text-Independent Speech Balloon Segmentation for Comics and Manga
Jean-christophe Burie

Lecture Notes in Computer Science, 2017

Comics and manga are one of the most popular and familiar forms of graphic content over the world and play a major role in spreading country's culture. Nowadays, massive digitization and digital-born materials allow page-per-page mobile reading but we believe that other usages may be released in the near future. In this paper we focus on speech balloon segmentation which is a key issue for text/graphic association in scanned and digital-born comic book images. Speech balloons are at the interface between text and comic character, they inform the reader about speech tone and the position of its speaker. We present a generic and text-independent segmentation method based on topological and spatial organization of connected component.

View PDFchevron_right
CNN Based Extraction of Panels/Characters from Bengali Comic Book Page Images
Samit Biswas

2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), 2019

Peoples nowadays prefer to use digital gadgets like cameras or mobile phones for capturing documents. Automatic extraction of panels/characters from the images of a comic document is challenging due to the wide variety of drawing styles adopted by writers, beneficial for readers to read them on mobile devices at any time and useful for automatic digitization. Most of the methods for localization of panel/character rely on the connected component analysis or page background mask and are applicable only for a limited comic dataset. This work proposes a panel/character localization architecture based on the features of YOLO and CNN for extraction of both panels and characters from comic book images. The method achieved remarkable results on Bengali Comic Book Image dataset (BCBId) consisting of total 4130 images, developed by us as well as on a variety of publicly available comic datasets in other languages, i.e. eBDtheque, Manga 109 and DCM dataset.

View PDFchevron_right
Deep CNN-based Speech Balloon Detection and Segmentation for Comic Books
Jochen Laubrock

2019

We develop a method for the automated detection and segmentation of speech balloons in comic books, including their carrier and tails. Our method is based on a deep convolutional neural network that was trained on annotated pages of the Graphic Narrative Corpus. More precisely, we are using a fully convolutional network approach inspired by the U-Net architecture, combined with a VGG-16 based encoder. The trained model delivers state-of-the-art performance with an F1-score of over 0.94. Qualitative results suggest that wiggly tails, curved corners, and even illusory contours do not pose a major problem. Furthermore, the model has learned to distinguish speech balloons from captions. We compare our model to earlier results and discuss some possible applications.

View PDFchevron_right
Automatic preview generation of comic episodes for digitized comic search
Keiichiro Hoashi

2011

Abstract This research proposes a novel method to present" thumbnails" of episodes of digitized comics, in order to improve the efficiency of comic search. Comic episode thumbnails are generated based on image analysis technologies developed especially for comic images. Namely, the following procedures are developed for our system: automatic comic frame segmentation, text balloon extraction, and a linear regression based model to calculate the importance score of each extracted frame.

View PDFchevron_right
Recognizing Text Elements for SVG Comic Compression and Its Novel Applications
RAY-I CHANG

2011 International Conference on Document Analysis and Recognition, 2011

SVG (scalable vector graphics) has become the standard format for 2D graphics in HTML5. Although some image-to-SVG conversion systems had been proposed, the sizes of files they produced are still large. In [1], we proposed a new system to convert raster comic images into vector SVG files. The compression ratio is better than the previous methods. However, these methods do not process text in raster images. In this paper, we improve our system to recognize text elements in the comic and use these text elements to provide better compression and novel applications. The proposed method uses SCW (sliding concentric windows) and SVM (support vector machine) to identify text regions. Then, OCR (optical character recognition) is applied to recognize text elements in those regions. Instead of encoding the text regions as vectors, the text elements are embedded in the SVG file along with their coordinate values. Experimental results show that we can reduce the file sizes to about 52% of the original SVG files. Using these text elements, we can translate comics into other languages to provide multilingual services easily. Text/content-based image search can be supported efficiently. It can also provide a novel application system for story teller.

View PDFchevron_right
Automatic Photo to Ideophone Manga Matching
Tony Dunnigan

ArXiv, 2020

Photo applications offer tools for annotation via text and stickers. Ideophones, mimetic and onomatopoeic words, which are common in graphic novels, have yet to be explored for photo annotation use. We present a method for automatic ideophone recommendation and positioning of the text on photos. These annotations are accomplished by obtaining a list of ideophones with English definitions and applying a suite of visual object detectors to the image. Next, a semantic embedding maps the visual objects to the possible relevant ideophones. Our system stands in contrast to traditional computer vision-based annotation systems, which stop at recommending object and scene-level annotation, by providing annotations that are communicative, fun, and engaging. We test these annotations in Japanese and find they carry a strong preference and increase enjoyment and sharing likelihood when compared to unannotated and object-based annotated photos.

View PDFchevron_right
Text Block Segmentation in Comic Speech Bubbles
Jean-christophe Burie

2020

Comics and manga text recognition are attracting an increasing research and industrial interest. Also, the state of the art text detection and OCR performances is starting to be mature enough to provide automatic text recognition for a variety of comics and manga writing styles. However, comics text layout sometimes prevents usual text line detection to be applied successfully, even within speech bubbles. In this paper, we propose a domain specific text block detection method able to detect single and multiple text block regions inside speech bubbles, in order to enhance OCR transcription and further post-processing. This approach presents very satisfactory results on all tested bubble styles from Latin and non-Latin scripts.

View PDFchevron_right
Comics page structure analysis based on automatic panel extraction
Jean-christophe Burie

Comic books represent an important cultural heritage in many countries. However, few researches have been done in order to analyse the content of comics such as panels, speech balloons or characters. At first glance, the structure of a comics page may appear easy to determine. In practice, the configuration of the page, the size and the shape of the panels can be different from one page to the next. Moreover, authors often draw extended contents (speech balloon or comic art) that overlap two panels or more. In some situations, the panel extraction can become a real challenge. To determine the structure of a comics page, we propose, in this paper, a method based on region growing and mathematical morphology to extract automatically the panels of a comics page. Our approach is compared with other methods find in the literature. Results are presented and discussed.

View PDFchevron_right
Digital Comics Image Indexing Based on Deep Learning
Jean-christophe Burie

Journal of Imaging

The digital comic book market is growing every year now, mixing digitized and digital-born comics. Digitized comics suffer from a limited automatic content understanding which restricts online content search and reading applications. This study shows how to combine state-of-the-art image analysis methods to encode and index images into an XML-like text file. Content description file can then be used to automatically split comic book images into sub-images corresponding to panels easily indexable with relevant information about their respective content. This allows advanced search in keywords said by specific comic characters, action and scene retrieval using natural language processing. We get down to panel, balloon, text, comic character and face detection using traditional approaches and breakthrough deep learning models, and also text recognition using LSTM model. Evaluations on a dataset composed of online library content are presented, and a new public dataset is also proposed.

View PDFchevron_right

References (25)

  1. Arai, K. and Tolle, H. (2011). Method for real time text ex- traction of digital manga comic. International Journal of Image Processing (IJIP), 4(6):669-676.
  2. Clavelli, A. and Karatzas, D. (2009). Text segmentation in colour posters from the spanish civil war era. In Pro- ceedings of the 2009 10th International Conference on Document Analysis and Recognition, ICDAR '09, pages 181-185, Washington, DC, USA. IEEE Com- puter Society.
  3. Epshtein, B., Ofek, E., and Wexler, Y. (2010). Detect- ing text in natural scenes with stroke width transform. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 2963 -2970.
  4. Jung, K., Kim, K. I., and Jain, A. K. (2004). Text informa- tion extraction in images and video: a survey. Pattern Recognition, 37(5):977 -997.
  5. Karatzas, D. and Antonacopoulos, A. (2007). Colour text segmentation in web images based on human percep- tion. Image and Vision Computing, 25(5):564 -577.
  6. Karatzas, D., Mestre, S. R., Mas, J., Nourbakhsh, F., and Roy, P. P. (2011). Icdar 2011 robust reading competi- tion -challenge 1: Reading text in born-digital images (web and email). International Conference on Docu- ment Analysis and Recognition, 0:1485-1490.
  7. Kim, W. and Kim, C. (2009). A new approach for over- lay text detection and extraction from complex video scene. Image Processing, IEEE Transactions on, 18(2):401 -411.
  8. Matas, J., Chum, O., Urban, M., and Pajdla, T. (2002). Ro- bust wide baseline stereo from. In In British Machine Vision Conference, pages 384-393.
  9. Matsui, Y., Yamasaki, T., and Aizawa, K. (2011). In- teractive manga retargeting. In ACM SIGGRAPH 2011 Posters, SIGGRAPH '11, pages 35:1-35:1, New York, NY, USA. ACM.
  10. Meng, Q. and Song, Y. (2012). Text detection in natural scenes with salient region. In Document Analysis Sys- tems (DAS), 2012 10th IAPR International Workshop on, pages 384 -388.
  11. Neumann, L. and Matas, J. (2012). Real-time scene text lo- calization and recognition. Computer Vision and Pat- tern Recognition, pages 1485-1490.
  12. Oliveira, D. M. and Lins, R. D. (2010). Generalizing tableau to any color of teaching boards. In Proceedings of the 2010 20th International Conference on Pattern Recognition, ICPR '10, pages 2411-2414, Washing- ton, DC, USA. IEEE Computer Society.
  13. Otsu, N. (1979). A threshold selection method from gray- level histograms. IEEE Transactions on Systems, Man and Cybernetics, 9(1):62-66.
  14. Rigaud, C., Tsopze, N., Burie, J.-C., and Ogier, J.- M. (2012). Robust frame and text extraction from comic books. Lecture Note for Computer Science GREC2011, 7423(19).
  15. Roudier, N. (2011). LES TERRES CREUSEES, volume Acte sur BD. Actes Sud.
  16. Shivakumara, P., Phan, T., and Tan, C. L. (2009). A robust wavelet transform based technique for video text de- tection. In Document Analysis and Recognition, 2009. ICDAR '09. 10th International Conference on, pages 1285 -1289.
  17. Su, C.-Y., Chang, R.-I., and Liu, J.-C. (2011). Recog- nizing text elements for svg comic compression and its novel applications. In Proceedings of the 2011 International Conference on Document Analysis and Recognition, ICDAR '11, pages 1329-1333, Wash- ington, DC, USA. IEEE Computer Society.
  18. Thotreingam Kasar, Jayant Kumar, and Ramakrishnan, A. G. (2007). Font and Background Color Indepen- dent Text Binarization. In Intl. workshop on Camera Based Document Analysis and Recognition (workshop of ICDAR), pages 3-9.
  19. Tombre, K., Tabbone, S., Plissier, L., Lamiroy, B., and Dosch, P. (2002). Text/graphics separation revisited. In in: Workshop on Document Analysis Systems (DAS, pages 200-211. Springer-Verlag.
  20. Tsopze, N., Guérin, C., Bertet, K., and Revel, A. (2012). Ontologies et relations spatiales dans la lecture d'une bande dessinée. In Ingénierie des Connaissances, pages 175-182, Paris.
  21. Wang, K. and Belongie, S. (2010). Word spotting in the wild. In Daniilidis, K., Maragos, P., and Paragios, N., editors, Computer Vision ECCV 2010, volume 6311 of Lecture Notes in Computer Science, pages 591- 604. Springer Berlin / Heidelberg.
  22. Weinman, J., Learned-Miller, E., and Hanson, A. (2009). Scene text recognition using similarity and a lexi- con with sparse belief propagation. Pattern Analy- sis and Machine Intelligence, IEEE Transactions on, 31(10):1733 -1746.
  23. Wolf, C. and Jolion, J.-M. (2006). Object count/area graphs for the evaluation of object detection and seg- mentation algorithms. Int. J. Doc. Anal. Recognit., 8(4):280-296.
  24. Wright, S. L. (2002). Ibm 9.2-megapixel flat-panel display: Technology and infrastructure.
  25. Yamada, M., Budiarto, R., Endo, M., and Miyazaki, S. (2004). Comic image decomposition for reading comics on cellular phones. IEICE Transactions, 87- D(6):1370-1376.

Related papers

Segmentation-Free Speech Text Recognition for Comic Books
Jean-christophe Burie

2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)

Speech text in comic books is written in a particular manner by the scriptwriter which raises unusual challenges for text recognition. We first detail these challenges and present different approaches to solve them. We compare the performances of pre-trained OCR and segmentation-free approach for speech text of comic books written in Latin script. We demonstrate that few good quality pre-trained OCR output samples, associated with other unlabeled data with the same writing style, can feed a segmentation-free OCR and improve text recognition. Thanks to the help of the lexicality measure that automatically accept or reject the pretrained OCR output as pseudo ground truth for a subsequent segmentation-free OCR training and recognition.

View PDFchevron_right
Robust Frame and Text Extraction from Comic Books
Jean-christophe Burie

Lecture Notes in Computer Science, 2013

Comic books constitute an important heritage in many countries. Nowadays, digitization allows to search directly from content instead of metadata only (e.g. album title or author name). Few studies have been done in this direction. Only frame and speech balloon extraction have been experimented in the case of simple page structure. In fact, the page structure depends on the author which is why many different structures and drawings exist. Despite the differences, drawings have a common characteristic because of design process: they are all surrounded by a black line. In this paper, we propose to rely on this particularity of comic books to automatically extract frame and text using a connectedcomponent labeling analysis. The approach is compared with some existing methods found in the literature and results are presented.

View PDFchevron_right
Panel and Speech Balloon Extraction from Comic Books
Jean-christophe Burie

2012 10th IAPR International Workshop on Document Analysis Systems, 2012

Comic books represent an important cultural heritage in many countries. However, few researches have been done in order to analyse the content of comics such as panels, speech balloons or characters. At first glance, the structure of a comic page may appear easy to determine. In practice, the configuration of the page, the size and the shape of the panels can be different from one page to the next. Moreover, authors often draw extended contents (speech balloon or comic art) that overlap two panels or more. In some situations, the panel extraction can become a real challenge. Speech balloons are other important elements of comics. Full text indexing is only possible if the text can be extracted. However the text is usually embedded among graphic elements. Moreover, unlike newspapers, the text layout in speech balloons can be irregular. Classic text extraction method can fail. We propose, in this paper, a method based on region growing and mathematical morphology to extract automatically the panels of a comic page and a method to detect speech balloons. Our approach is compared with other methods find in the literature. Results are presented and discussed.

View PDFchevron_right
An active contour model for speech balloon detection in comics
Dimosthenis Karatzas, Jean-christophe Burie

2013

Comic books constitute an important cultural heritage asset in many countries. Digitization combined with subsequent comic book understanding would enable a variety of new applications, including content-based retrieval and content retargeting. Document understanding in this domain is challenging as comics are semi-structured documents, combining semantically important graphical and textual parts. Few studies have been done in this direction. In this work we detail a novel approach for closed and non-closed speech balloon localization in scanned comic book pages, an essential step towards a fully automatic comic book understanding. The approach is compared with existing methods for closed balloon localization found in the literature and results are presented.

View PDFchevron_right
An OCR Pipeline and Semantic Text Analysis for Comics
Dr. Alexander Dunst

A. Del Bimbo et al. (Eds.): ICPR 2020 Workshops, LNCS 12666, pp. 213–222, 2021

Optical character recognition has remained a challenge for comics, given the high variability of placement of text on the page, the wide variety of frequently handwritten fonts, and the limited availability and small size of datasets. This paper reports on currently ongoing work on an OCR pipeline that includes text spotting with the help of a U-Net based fully convolutional neural network and OCR training with the open-source software Calamari, which was performed on the "Graphic Narrative Corpus" of book-length graphic novels written in English. Based on the results of the OCR training, we then present an analysis of the textual properties of 129 graphic novels correlated with page length, historical development, and genre affiliation.

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved