EMPIRICAL TEXT MINING FOR GENRE DETECTION

2005

In this report we propose a new set of features for automatic genre and text type identification: linguistic facets. The label linguistic facet has been created to stress the fact that each of the features in this new set highlights a facet, i.e. an aspect in the communicative context that is reflected in the use of language. Linguistic facets subsume two set of features: functional cues and syntactic patterns. Functional cues are not completely new. Syntactic patterns, instead, have never been tried before in any automatic approach. Both functional cues and syntactic patterns are linguistically-motivated features that can be interpreted functionally. The effort is to derive from text-internal linguistic cues the communicative context in which the text has been produced/consumed. With linguistically-motivated and functionally-interpretable features it is possible to combine qualitative analysis and quantitative findings, and get a more accurate text analysis together with genre and ...

Proceedings of the 37th Annual Hawaii International …, 2004

Proceedings of the COLING/ACL on Main …, 2006

In this paper, we propose an implementable characterization of genre suitable for automatic genre identification of web pages. This characterization is implemented as an inferential model based on a modified version of Bayes' theorem. Such a model can deal with genre hybridism and individualization, two important forces behind genre evolution. Results show that this approach is effective and is worth further research.

2004

Work on register for German is rather scarce, compared to English. We narrow this gap by (i) developing a theoretically grounded comparative typology for genre and register analysis, (ii) compiling a corpus of German register and genre out of DeReKo (Kupietz et al., 2010), (iii) developing high accuracy machine learning algorithms for supervised text genre classification, (iv) providing an extensive description of prototypical register classes through the agglomeration of style feature loadings. We thus contribute both to linguistic theory building and to research on genre adaptation of NLP methods within the LiMo project. Register can be understood as the function between situational context and linguistic analysis (Biber & Conrad, 2009) or as functional stylistics i.e. the distribution of style features in communicative areas (Jakobson, 1960). In designing the corpus typology (taxonomy) we make use of pro- totype theory (Rosch, 1973) that affords the embedding of a basic-level category (communicative purpose) within a super-level (social text use) and a sub-genre (topic modelling). We model the prototypical text classes using dimensions such as document size, a ‘natural’ distribution of topics, or the dominant register of a genre. We then develop a machine learning system for register and genre clas- sification. We focus on a high-quality feature extraction pipeline which implements style features through numerous morphosyntactic annotations, lexicons of linguistic nature (i.a. verb classes), psycholinguistic word norms (i.a. concreteness), and topic models. We identify the best models for supervised text genre classification, achieving results comparable to the state-of-the-art for English. Finally, we show that our work is also applicable to English. Here, we are able to identify pervasive functional dimensions using PCA (to wit, interpersonal and narrative dimensions in literary genre). References: • Biber, D. and Conrad, S. (2009). Register, genre, and style. Cambridge University Press. • Jakobson, R. (1960). Closing statement: Linguistics and poetics. Style in language, 350:377. • Kupietz, M., Belica, C., Keibel, H., and Witt, A. (2010).The german reference corpus DeReKo: A primordial sample for linguistic research. In LREC. • Rosch, E. H. (1973). Natural categories. Cognitive psychology, 4(3):328–350.

2018

We report an ongoing study on quantitative characteristics of texts written in different genres. At this stage, we compared Lithuanian and English texts in terms of genres. We used 16 indices which describe frequency structure of text as well as indicate several other characteristics of written texts. Initial study showed significant differences of indices calculated for genre pairs of the same language. Hierarchical clustering revealed possible applications in using them as features for text categorization/classification by genre, though better results were achieved for Lithuanian texts.

Proc. of the JADT, 2006

In this paper, two experiments in automatic genre classification of web pages are presented. These two experiments are designed to highlight three important issues related to genre classification : corpus composition and genre palettes, feature representativeness, and exportability of classification models. Results show the influence of corpus composition and genre palette on classification rates. They also show how well and to what extent feature sets represent genres in a palette, and give an idea of the limitations of the classification models when exported and used for predictive tasks.

In this article, a system is proposed for the automatic style categorisation of text corpora in the Greek language. This categorisation is based to a large extent on the type of language used in the text, for example whether the language used is representative of formal Greek or not. To arrive to this categorisation, the highly inflectional nature of the Greek language is exploited. For each text, a vector of both structural and morphological characteristics is assembled. Categorisation is achieved by comparing this vector to given archetypes using a statistical-based method. Experimental results reported in this article indicate an accuracy exceeding 98% in the categorisation of a corpus of texts spanning different registers.

mantech publications, 2023

Rapid progress in digital data acquisition techniques have led to huge volume of data. More than 80 percent of today's data is composed of unstructured or semi-structured data. The recovery of similar patterns and trends to see the text data from huge volume of data is a big issue. Text mining is a process of extracting interesting and nontrivial patterns from huge amount of text documents. There lie many techniques and tools to mine the text documents and discover the information for future and process in decision making. The choice of selecting the right and appropriate text mining technique helps to recover the speed and slows the time and effort required to get valuable information. This paper briefly discusses and analyzes the text mining techniques and their applications. With the advancement of technology, more and more data is available in digital form. Among which, most of the data (approx. 85%) is in unstructured textual form. Thus, it has become essential to build better techniques and algorithms to get useful and interesting data from the large amount of textual data. Hence, the field of information extraction and text mining became popular areas of research, to get interesting and needful information

repositories. Genre classification, which narrows down the possible structure of a document, is a valuable step in realising the general automatic extraction of semantic metadata essential to the efficient management and use of digital objects. The characterisation of digital objects in terms of genre also associates the object to the objectives that led to its creation, which indicates its relevance to new objectives in information search. In the present report, we present an analysis of word frequencies in different genre classes in an effort to understand the distinction between independent classification tasks. In particular, we examine automated experiments on thirty-one genre classes to determine the relationship between the word frequency metrics and the degree of its significance in carrying out classification in varying environments.