Academia.eduAcademia.edu

Outline

Title
Abstract
Appendix. Linguistic Features Used in the Study
References
All Topics
Computer Science
Artificial Intelligence

Pinning down text complexity

Profile image of Arne JönssonArne Jönsson

2020, Register Studies

https://doi.org/10.1075/RS.19005.SAN
visibility

…

description

45 pages

link

1 file

Abstract

In this article, we present the results of a corpus-based study where we explore whether it is possible to automatically single out different facets of text complexity in a general-purpose corpus. To this end, we use factor analysis as applied in Biber’s multi-dimensional analysis framework. We evaluate the results of the factor solution by correlating factor scores and readability scores to ascertain whether the selected factor solution matches the independent measurement of readability, which is a notion tightly linked to text complexity. The corpus used in the study is the Swedish national corpus, calledStockholm-Umeå Corpusor SUC. The SUC contains subject-based text varieties (e.g., hobby), press genres (e.g., editorials), and mixed categories (e.g., miscellaneous). We refer to them collectively as ‘registers’. Results show that it is indeed possible to elicit and interpret facets of text complexity using factor analysis despite some caveats. We propose a tentative text complexi...

Related papers

Text Complexity and Readability Measures: An Examination of Historical Trends
Laura Clark Briggs

2014

In recent years, much attention has been given to the readability and complexity of texts. With the move toward the use of Common Core State Standards (CCSS) by many states, policymakers and educators have directed attention to the ability of American students to successfully navigate complex text. This concern with text complexity and readability is not new; in fact, the earliest American attempt to use quantitative measures to examine text began in the late 19 th century.

View PDFchevron_right
Text complexity and linguistic features: Their correlation in English and Russian
Boris Iomdin

Russian Journal of Linguistics

Text complexity assessment is a challenging task requiring various linguistic aspects to be taken into consideration. The complexity level of the text should correspond to the reader’s competence. A too complicated text could be incomprehensible, whereas a too simple one could be boring. For many years, simple features were used to assess readability, e.g. average length of words and sentences or vocabulary variety. Thanks to the development of natural language processing methods, the set of text parameters used for evaluating readability has expanded significantly. In recent years, many articles have been published the authors of which investigated the contribution of various lexical, morphological, and syntactic features to the readability level. Nevertheless, as the methods and corpora are quite diverse, it may be hard to draw general conclusions as to the effectiveness of linguistic information for evaluating text complexity due to the diversity of methods and corpora. Moreover,...

View PDFchevron_right
A New Text Complexity Measure for Ranking Fiction Texts
Galbadrakh Enkhbayar, Michael Dalvean

English teachers often have difficulty matching the complexity of fiction texts with students' reading levels. Texts that seem appropriate for students of a given level can turn out to be too difficult. Furthermore, it is difficult to choose a series of texts that represent a smooth gradation of text difficulty. This paper attempts to address both problems by providing a complexity ranking of a corpus of 200 fiction texts consisting of 100 adults' and 100 children's texts. Using machine learning, several standard readability measures are used as variables to create a classifier which is able to classify the corpus with an accuracy of 84%. A classifier created with linguistic variables is able to classify the corpus with an accuracy of 89%. The 'latter classifier is then used to provide a linear complexity rank for each text. The resulting ranking instantiates a fine-grained increase in complexity. This can be used by an English teacher to select a sequence of texts that represent an increasing challenge to a student without there being a frustratingly discrete rise in difficulty.

View PDFchevron_right
Validating Automated Measures of Text Complexity
Kathleen M Sheehan

Automated text complexity measurement tools (also called readability metrics) have been proposed as a way to help teachers, textbook publishers and assessment developers select texts that are closely aligned with the new, more demanding text complexity expectations specified in the Common Core State Standards. This article examines a critical element of the validity arguments presented in support of proposed metrics: the claim that criterion text complexity scores developed from students’ responses to reading comprehension test items are reflective of the difficulties actually experienced by students while reading. Evidence that fails to support this assertion is examined, and implications relative to the goal of obtaining valid, unbiased evidence about the measurement properties of proposed readability metrics are discussed

View PDFchevron_right
“Please ReaderBench this text”: A multi-dimensional textual complexity assessment framework
Danielle McNamara

HAL (Le Centre pour la Communication Scientifique Directe), 2018

A critical task for tutors is to provide learners with suitable reading materials in terms of difficulty. The challenge of this endeavor is increased by students' individual variability and the multiple levels in which complexity can vary, thus arguing for the necessity of automated systems to support teachers. This chapter describes ReaderBench, an open-source multi-dimensional and multi-lingual system that uses advanced Natural Language Processing techniques to assess textual complexity at multiple levels including surface-based, syntax, semantics and discourse structure. In contrast to other existing approaches, ReaderBench is centered on cohesion and makes extensive usage of two complementary models, i.e., Cohesion Network Analysis and the polyphonic model inspired from dialogism. The first model provides an in-depth view of discourse in terms of cohesive links, whereas the second one highlights interactions between points of view spanning throughout the discourse. In order to argue for its wide applicability and extensibility, two studies are introduced. The first study investigates the degree to which ReaderBench textual complexity indices differentiate between high and low cohesion texts. The ReaderBench indices led to a higher classification accuracy than those included in prior studies using Coh-Metrix and TAACO. In the second study, ReaderBench indices are used to predict the difficulty of a set of various texts. Although the high number of predictive indices (50 plus) accounted for less variance than previous studies, they make valuable contributions to our understanding of text due to their wide coverage.

View PDFchevron_right
Modeling Text Complexity using a Multi-Scale Probit
Arne Jönsson

arXiv: Applications, 2018

We present a novel model for text complexity analysis which can be fitted to ordered categorical data measured on multiple scales, e.g. a corpus with binary responses mixed with a corpus with more than two ordered outcomes. The multiple scales are assumed to be driven by the same underlying latent variable describing the complexity of the text. We propose an easily implemented Gibbs sampler to sample from the posterior distribution by a direct extension of established data augmentation schemes. By being able to combine multiple corpora with different annotation schemes we can get around the common problem of having more text features than annotated documents, i.e. an example of the $p>n$ problem. The predictive performance of the model is evaluated using both simulated and real world readability data with very promising results.

View PDFchevron_right
Text Complexity
Timothy Shanahan
View PDFchevron_right
Online Readability and Text Complexity Analysis with TextEvaluator
Kathleen M Sheehan

Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, 2015

We have developed the TextEvaluator system for providing text complexity and Common Core-aligned readability information. Detailed text complexity information is provided by eight component scores, presented in such a way as to aid in the user's understanding of the overall readability metric, which is provided as a holistic score on a scale of 100 to 2000. The user may select a targeted US grade level and receive additional analysis relative to it. This and other capabilities are accessible via a feature-rich front-end, located at http://texteval-pilot.ets.org/TextEvaluator/.

View PDFchevron_right
Generating Automated Text Complexity Classifications That Are Aligned With Targeted Text Complexity Standards
Michael Flor

2010

The Common Core Standards call for students to be exposed to a much greater level of text complexity than has been the norm in schools for the past 40 years. Textbook publishers, teachers, and assessment developers are being asked to refocus materials and methods to ensure that students are challenged to read texts at steadily increasing complexity levels as they progress through school so that all students remain on track to achieve college and career readiness by the end of 12th grade. Although automated text analysis tools have been proposed as one method for helping educators achieve this goal, research suggests that existing tools are subject to three limitations: inadequate construct coverage, overly narrow criterion variables, and inappropriate treatment of genre effects. Modeling approaches developed to address these limitations are described. Recommended approaches are incorporated into a new text analysis system called SourceRater. Validity analyses implemented on an independent sample of texts suggest that, compared to existing approaches, SourceRater’s estimates of text complexity are more reflective of the complexity classifications given in the new standards. Implications for the development of learning progressions designed to help educators organize curriculum, instruction, and assessment in reading are discussed.

View PDFchevron_right
What is Text Complexity?
Thomas DeVere Wolsey, Dana Grisham

Background Text levels have typically been addressed indirectly within past generations of standards documents. For example, a standard might describe fifth graders as needing to identify figurative language in “grade-level” but grade-level text was not defined. Readability estimates were of some help; however, they are limited measures that cannot fully capture the complexity of a novel or expository text.

View PDFchevron_right

References (36)

  1. Adesam, Y., Bouma, G. and Johansson, R. ( ). The Koala part-of-speechand morphological tagset for Swedish. SLTC.
  2. Asención-Delaney, Y., & Collentine, J. ( ). A multidimensional analysis of a written L Spanish corpus. Applied linguistics, ( ), -. Biber, D. ( ). Variation across speech and writing. Cambridge University Press. https://doi.org/./CBO Biber, D. ( ). A typology of English texts. Linguistics, ( ), -.
  3. Biber, D. ( ). Dimensions of register variation: A cross-linguistic comparison. Cambridge University Press. https://doi.org/./CBO Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E. ( ). Longman grammar of spoken and written English. Longman. Biber, D., & Kurjian, J. ( ). Towards a taxonomy of web registers and text types: A multi- dimensional analysis. In Corpus Linguistics and the Web (pp. -). https://doi.org/./_ Biber, D., & Conrad, S. ( ). Register, genre, and style. Cambridge University Press. https://doi.org/./CBO Biber, D., & Egbert, J. ( ). Register variation on the searchable web: A multi-dimensional analysis. Journal of English Linguistics, ( ), -. https://doi.org/./ Björnsson, C. H. ( ). Läsbarhet. Liber.
  4. Cattell, R. B. ( ). The scree test for the number of factors. Multivariate behavioral research, ( ), -. https://doi.org/./smbr_
  5. Collins-Thompson, K. ( ). Computational assessment of text readability: A survey of current and future research. ITL-International Journal of Applied Linguistics, ( ), -. https://, Science, and Technical Subjects. Appendix A: Research Supporting Key Elements of the Standards, Glossary of Key Terms. Pinning down text complexity: An exploratory study [ ]
  6. Cvrček, V., Komrsková, Z., Lukeš, D., Poukarová, P., Řehořková, A., Zasina, A. J., & Benko, V. ( ). Comparing web-crawled and traditional corpora. Language Resources and Evaluation, -.
  7. Dahl, Ö. ( ). The growth and maintenance of linguistic complexity (Vol. ). John Benjamins Publishing. https://doi.org/./slcs. Dale, E., & Chall, J. S. ( ). The concept of readability. Elementary English, ( ), -.
  8. Dell'Orletta, F., Montemagni, S., & Venturi, G. ( ), September). Linguistic pro ling of texts across textual genres and readability levels. An exploratory study on Italian ctional prose. In Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP (pp. -).
  9. Dell'Orletta, F., Montemagni, S., & Venturi, G. ( ). Assessing document and sentence readability in less resourced languages and across textual genres. ITL-International Journal of Applied Linguistics, ( ), -. https://doi.org/./itl...del DiStefano, C., Zhu, M., & Mindrila, D. ( ). Understanding and using factor scores: Considerations for the applied researcher. Practical Assessment, Research & Evaluation, ( ), -.
  10. Fahlborg, D., & Rennes, E. ( ). Introducing SAPIS-an API service for text analysis and simpli cation. In the second national Swe-Clarin workshop: Research collaborations for the digital age, Umeå, Sweden.
  11. Falkenjack, J. ( ). Towards a model of general text complexity for Swedish (Doctoral dissertation, Linköping University Electronic Press).
  12. Falkenjack, J., Mühlenbock, K. H., & Jönsson, A. ( ), May). Features indicating readability in Swedish text. In Proceedings of the th Nordic Conference of Computational Linguistics (NODALIDA ) (pp. -).
  13. Falkenjack, J., Santini, M., & Jönsson, A. ( ). An exploratory study on genre classi cation using readability features. In Proceedings of the Sixth Swedish Language Technology Conference (SLTC ), Umeå, Sweden.
  14. Feng, L. ( ). Automatic readability assessment (Doctoral dissertation, CUNY Academic Works). Field, A. ( ). Discovering statistics using SPSS for Windows. Londra: Sage Publication. Flesch, R. ( ). A new readibility yardstick. Journal of Applied Psychology, ( ): -. https://doi.org/./h
  15. Field, A., Miles, J., & Field, Z. ( ). Discovering statistics using R. Sage publications.
  16. Hayton, J. C., Allen, D. G., & Scarpello, V. ( ). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational research methods, ( ), -. https://doi.org/./
  17. Hiebert, E. H. ( ). Readability and the common core's staircase of text complexity. Text Matters, .
  18. Horn, J. L. ( ). A rationale and test for the number of factors in factor analysis.
  19. Psychometrika , -. https://doi.org/./BF
  20. Housen, A., De Clercq, B., Kuiken, F., & Vedder, I. ( ). Multiple approaches to complexity in second language research. Second Language Research, ( ), -. https://doi.org/./ Jelen, B. ( ). Excel charts and graphs. Que Publishing Company.
  21. Jönsson, S., Rennes, E., Falkenjack, J., & Jönsson, A. ( ). A component based approach to measuring text complexity. In Proceedings of The Seventh Swedish Language Technology Conference (SLTC-).
  22. Marina Santini & Arne Jönsson Kate, R. J., Luo, X., Patwardhan, S., Franz, M., Florian, R., Mooney, R. J., & Welty, C. ( ), August). Learning to predict readability using diverse linguistic features. In Proceedings of the rd international conference on computational linguistics (pp. -). Association for Computational Linguistics.
  23. Källgren, G., Gustafson-Capková, S., & Hartmann, B. ( ). Manual of the Stockholm Umeå Corpus version . . Department of Linguistics, Stockholm University, December. So a Gustafson-Capková and Britt Hartmann (eds.).
  24. Ledesma, R. D., Valero-Mora, P., & Macbeth, G. ( ). The scree test and the number of factors: a dynamic graphics approach. The Spanish journal of psychology, . https://doi.org/./sjp.. Lu, X. ( ). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, ( ), -. https://doi.org/./ijcl...lu Mühlenbock, K. H. ( ). I see what you mean: Assessing readability for speci c target groups. (Doctoral dissertation, University of Gothenburg, Gothenburg, Sweden). Napolitano, D., Sheehan, K. M., & Mundkowsky, R. ( ), June). Online readability and text complexity analysis with Text Evaluator. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations (pp. -).
  25. Nenkova, A., Chae, J., Louis, A., & Pitler, E. ( ). Structural features for predicting the linguistic quality of text. In Empirical methods in natural language generation (pp. -).
  26. Springer, Berlin, Heidelberg. https://doi.org/./----_ Nivre, J. ( ). Inductive dependency parsing (pp. -). Springer Netherlands. https://doi.org/./---_ Pallotti, G. ( ). A simple view of linguistic complexity. Second Language Research, ( ), -. https://doi.org/./ Petersen, S. ( ). Natural language processing tools for reading level assessment and text simpli cation for bilingual education. (Doctoral dissertation, University of Washington, Seattle, WA, USA).
  27. Petersen, S. E., & Ostendorf, M. ( ). A machine learning approach to reading level assessment. Computer Speech & Language, ( ), -. https://doi.org/./j.csl... Pilán, I., Vajjala, S., & Volodina, E. ( ). A readable read: Automatic assessment of language learning materials based on linguistic complexity. arXiv preprint arXiv: . .
  28. Pitler, E., & Nenkova, A. ( ), October). Revisiting readability: A uni ed framework for predicting text quality. In Proceedings of the conference on empirical methods in natural language processing (pp. -).
  29. Rello, L., Baeza-Yates, R., Bott, S., & Saggion, H. ( a). Simplify or help? Text simpli cation strategies for people with dyslexia. In Proceedings of the th International Cross- Disciplinary Conference on Web Accessibility (pp. -).
  30. Rello, L., Baeza-Yates, R., Dempere-Marco, L., and Saggion, H. ( b). Frequent words improve readability and short words improve understandability for people with dyslexia. In IFIP Conference on Human-Computer Interaction (pp. -. Springer.
  31. Saggion, H. ( ). Automatic text simpli cation. Synthesis Lectures on Human Language Technologies, ( ), -. https://doi.org/./SEDVYHLT Pinning down text complexity: An exploratory study [ ]
  32. Santini, M., Danielsson, B., & Jönsson, A. ( ), August). Introducing the Notion of 'Contrast'Features for Language Technology. In International Conference on Database and Expert Systems Applications (pp. -).
  33. Springer, Cham. https://doi.org/./----_ Sardinha, T. B., Kau mann, C., & Acunzo, C. M. ( ). A multi-dimensional analysis of register variation in Brazilian Portuguese. Corpora, ( ), -. https://doi.org/./cor..
  34. Sardinha, T. B., & Pinto, M. V. (Eds.). ( ). Multi-dimensional analysis, years on: A tribute to Douglas Biber (Vol. ). John Benjamins Publishing Company. https://doi.org/./scl. Štajner, S., & Saggion, H. ( ), August). Data-Driven Text Simpli cation. In Proceedings of the th International Conference on Computational Linguistics: Tutorial Abstracts (pp. -).
  35. Vega, B., Feng, S., Lehman, B., Graesser, A., & D'Mello, S. ( ), July). Reading into the text: Investigating the in uence of text complexity on cognitive engagement. In Educational Data Mining .
  36. Wray, D., & Janan, D. ( ). Readability revisited? The implications of text complexity Published in The Curriculum Journal, . https://doi.org/./..

Related papers

The State of the Field: Qualitative Analyses of Text Complexity
Elfrieda H Hiebert

TextProject Reading Research Report #13.01, 2013

The purpose of this review is to examine the function, logic, and impact of qualitative systems, with a focus on understanding their beneits and imperfections. We identify two primary functions for their use: (a) to match texts to reader ability so that readers read books that are within their grasp and (b) to unearth, and then scafold, those features of speciic texts that are likely to present challenges for readers of difering abilities. We examine three approaches to qualitative text analysis (text leveling systems, rubric and exemplar systems, and text mapping systems) relative to these functions. We conclude by strongly advocating the use of qualitative systems, if for no other reason than to prevent the unchecked use of quantitative approaches from promoting invalid applications of text complexity. At the same time, we raise a set of vexing issues that the field must address if these approaches are to be used with even a modicum of confidence.

View PDFchevron_right
What Can Readability Measures Really Tell Us About Text Complexity?
Ruslan Mitkov

This study presents the results of an initial phase of a project seeking to convert texts into a more accessible form for people with autism spectrum disorders by means of text simplification technologies. Random samples of Simple Wikipedia articles are compared with texts from News, Health, and Fiction genres using four standard readability indices (Kincaid, Flesch, Fog and SMOG) and sixteen linguistically motivated features. The comparison of readability indices across the four genres indicated that the Fiction genre was relatively easy whereas the News genre was relatively difficult to read. The correlation of four readability indices was measured, revealing that they are almost perfectly linearly correlated and that this correlation is not genre dependent. The correlation of the sixteen linguistic features to the readability indices was also measured. The results of these experiments indicate that some of the linguistic features are well correlated with the readability measures and that these correlations are genre dependent. The maximum correlation was observed for fiction.

View PDFchevron_right
Towards an Integrated Approach for Evaluating Textual Complexity for Learning Purposes
Philippe Dessus

Understanding a text in order to learn is subject to modeling and is partly dependent to the complexity of the read text. We transpose the evaluation process of textual complexity into measurable factors, identify linearly independent variables and combine multiple perspectives to obtain a holistic approach, addressing lexical, syntactic and semantic levels of textual analysis. Also, the proposed evaluation model combines statistical factors and traditional readability metrics with information theory, specific information retrieval techniques, probabilistic parsers, Latent Semantic Analysis and Support Vector Machines for best-matching all components of the analysis. First results show a promising overall precision (>50%) and near precision (>85%).

View PDFchevron_right
A Two-Stage Approach for Generating Unbiased Estimates of Text Complexity
Diane Napolitano, Michael Flor, Kathleen M Sheehan

Conference Proceeding: NAACL HLT 2013, 2013

Many existing approaches for measuring text complexity tend to overestimate the complexi-ty levels of informational texts while simulta-neously underestimating the complexity levels of literary texts. We present a two-stage esti-mation technique that successfully addresses this problem. At Stage 1, each text is classi-fied into one or another of three possible ge-nres: informational, literary or mixed. Next, at Stage 2, a complexity score is generated for each text by applying one or another of three possible prediction models: one optimized for application to informational texts, one opti-mized for application to literary texts, and one optimized for application to mixed texts. Each model combines lexical, syntactic and discourse features, as appropriate, to best rep-licate human complexity judgments. We dem-onstrate that resulting text complexity predictions are both unbiased, and highly cor-related with classifications provided by expe-rienced educators.

View PDFchevron_right
Using Multiple Sources of Information in Establishing Text Complexity
Elfrieda H Hiebert

TextProject Reading Research Report #11.03, 2011

A focus of the Common Core State Standards/English Language Arts (CCSS/ELA) is that students become increasingly more capable with complex text over their school careers. This focus has redirected attention to the measurement of text complexity. Although CCSS/ELA suggests multiple criteria for this task, the standards offer a single measure of text complexity—Lexiles. In this paper, I propose that additional quantitative measures are available—including the two components of a Lexile rating—that can provide more comprehensive views of text complexity. I apply these two “intra-Lexile” measures and one additional measure, referential cohesion, to sets of exemplars for grade bands 2–3 and 4–5 within the CCSS/ELA. The analyses suggest that conclusions about text complexity vary considerably when multiple quantitative measures are used, rather than a single, omnibus index.

View PDFchevron_right

Related topics

  • Computer Science
  • Artificial Intelligence
  • Multivariate Statistics
  • Natural Language Processing
  • Corpus-based discourse analysis
  • Register Studies
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved