Academia.eduAcademia.edu

Outline

Title
Abstract
FAQs
Introduction
Desired Properties of Assessment Questions
Related Work
NLP-Based Methods
Ontology-Based Methods
Conclusion and Future Research Directions
References
All Topics
Computer Science
Natural Language Processing

A Similarity-based Theory of Controlling MCQ Difficulty

Profile image of Tahani AlsubaitTahani Alsubait

Abstract

—Several attempts have already been made to automate the generation of assessment questions. These attempts were mainly technical and lacked a theoretical backing. We explore psychological and educational theories to support the development of principled methods to generate questions and control their properties. We present a similarity-based theory to control the difficulty of multiple-choice questions and show its practicality and consistency with the psychological and educational theories.

FAQs

sparkles

AI

What key factors influence the difficulty of multiple-choice questions (MCQs)?add

The paper shows that difficulty correlates with the similarity between the key answer and distractors, with distractors requiring deeper knowledge for effective elimination.

How does knowledge structure affect question answering ability?add

Higher mastery students exhibit cohesive knowledge structures, enabling better discrimination among similar concepts, unlike low mastery students with fragmented knowledge.

What role do distractors play in MCQ quality and assessment accuracy?add

Effective distractors should be plausible yet recognizable by knowledgeable students, promoting accurate assessment of student mastery levels.

What are the implications of psychological theories on MCQ question design?add

Psychological theories, like spreading activation, affirm that knowledge organization influences how students retrieve information and solve questions effectively.

How can similarity measures enhance question generation from ontologies?add

Utilizing similarity measures such as Lin's supports the selection of quality distractors, improving the probability of valid assessment outcomes.

Related papers

Improving Multiple-Choice Questions
lurdes babo

2011

A MC (multiple-choice) question can be defined as a question in which students are asked to select one alternative from a given set of alternatives in response to a question stem. The objective of this paper is to analyse if MC questions may be considered as an interesting alternative for assessing knowledge, particularly in the mathematics area, as opposed to the traditional methods, such as open questions. We also present some guidelines with the intent of improving the quality of the MC questions. For that, we illustrate some researchers' opinions in this area. The construction of good MC questions requires special care which needs the writing ability of test-makers and therefore time consuming. We are going to analyse the difficulties in the construction of MC questions as well as some advantages and limitations of this type of questions. We also are showing the frequent critics and worries, since the beginning of this objective format usage. We aim to give some examples of ...

View PDFchevron_right
A computer-aided environment for generating multiple-choice test items
Ruslan Mitkov

Natural Language Engineering, 2006

This paper describes a novel computer-aided procedure for generating multiple-choice test items from electronic documents. In addition to employing various Natural Language Processing techniques, including shallow parsing, automatic term extraction, sentence transformation and computing of semantic distance, the system makes use of language resources such as corpora and ontologies. It identifies important concepts in the text and generates questions about these concepts as well as multiple-choice distractors, offering the user the option to post-edit the test items by means of a user-friendly interface. In assisting test developers to produce items in a fast and expedient manner without compromising quality, the tool saves both time and production costs.

View PDFchevron_right
PolyAQG Framework: Auto-generating assessment questions
Zaharin Yusoff

2021 IEEE International Conference on Computing (ICOCO), 2021

Designing and setting assessment questions for examinations is always a necessary task for educators. In this article, we identify the research gaps in (semi-) automatically generating questions by evaluating all the available approaches developed thus far. We then propose a framework that puts together previous approaches and suggests ways to fill in their gaps. One hundred and thirteen pieces of literature relevant to question generation approaches have been reviewed and compared. For each of the approaches, the uniqueness of the techniques is explained. The PolyAQG Framework is presented with an explanation of how it would contribute to the solution of the problem, by improving the variety of the questions, increasing the total number of possible choices of question selections, as well as providing a better quality of questions. Apart from the framework, another novelty in this work is the innovative way a domain ontology is used to generate a wider variety of questions.

View PDFchevron_right
Automatic Multiple Choice Question Generation From Text: A Survey
DR. DHAWALESWAR RAO CH

IEEE Transactions on Learning Technologies, 2019

Automatic Multiple Choice Question (MCQ) generation from a text is a popular research area. MCQs are widely accepted for large-scale assessment in various domains and applications. However, manual generation of MCQs is expensive and time-consuming. Therefore, researchers were attracted towards automatic MCQ generation since the late 90's. Since then, many systems have been developed for MCQ generation. We perform a systematic review of those systems. This paper presents our findings on the review. We outline a generic workflow for an automatic MCQ generation system. The workflow consists of six phases. For each of these phases, we find and discuss the list of techniques adopted in the literature. We also study the evaluation techniques for assessing the quality of the system generated MCQs. Finally, we identify the areas where the current research focus should be directed toward enriching the literature.

View PDFchevron_right
Knowledge Externalization Based on Differences of Solutions for Automatic generation of Multiple-choice Question
Tsukasa Hirashima

nectec.or.th

We study about a method to describe knowledge for generating an adequate number of multiple-choice questions. We focus on the fact that instructors can easily explain concrete multiple-choice questions, even if they cannot describe the domain knowledge completely. We propose the Environment of Abstract Rule (EAR), which enables an instructor to describe abstract knowledge by explaining the difference among some concrete multiple-choice questions, and by representing the knowledge in the form of a diagram. In this paper, we describe the concept of EAR and report the results of a preliminary experiment in which we explored whether instructors could describe knowledge easily. As a result, we suggest that the EAR has the potential for generating appropriate knowledge.

View PDFchevron_right
Computer Aids for Designing Effective Multiple Choice Questions
Julian Scher, Marilyn Tremaine

With minimal training in assessment design, professors must generate high quality examinations that fairly assess student accomplishment of course objectives. Multiple choice questions are frequently used for this assessment, but often do not address learning objectives, are misinterpreted, and if reused, may challenge the integrity of some students. It is proposed that computer aids for generating multiple choice tests might be of considerable use to an instructor. This study presents a first step in building such aids, that of demonstrating via a case study, the inadequacy of current multiple choice test item generation methods. The case study also uncovers the functionality to build into a computer test generation aid for instructors. Bloom's taxonomy of learning objectives, discriminant analysis and student evaluations of questions is used to analyze the question data used in this study.

View PDFchevron_right
GUIDELINES FOR THE CONSTRUCTION OF MULTIPLE CHOICE QUESTIONS TESTS
grem lin

Multiple Choice Questions (MCQs) are generally recognized as the most widely applicable and useful type of objective test items. They could be used to measure the most important educational outcomes -knowledge, understanding, judgment and problem solving. The objective of this paper is to give guidelines for the construction of MCQs tests. This includes the construction of both "single best option" type, and "extended matching item" type. Some templates for use in the "single best option" type of questions are recommended.

View PDFchevron_right
Notes From the Field: Automatic Item Generation, Standard Setting, and Learner Performance in Mastery Multiple-Choice Tests
Ara Tekian

Evaluation & the Health Professions, 2020

Mastery learning assessments have been described in simulation-based educational interventions; however, studies applying mastery learning to multiple-choice tests (MCTs) are lacking. This study investigates an approach to item generation and standard setting for mastery learning MCTs and evaluates the consistency of learner performance across sequential tests. Item models, variables for question stems, and mastery standards were established using a consensus process. Two test forms were created using item models. Tests were administered at two training programs. The primary outcome, the test-retest consistency of passfail decisions across versions of the test, was 94% (k ¼ .54). Decision-consistency classification was .85. Item-level consistency was 90% (k ¼ .77, SE ¼ .03). These findings support the use of automatic item generation to create mastery MCTs which produce consistent pass-fail decisions. This technique broadens the range of assessment methods available to educators that require serial MCT testing, including mastery learning curricula.

View PDFchevron_right
A practical discussion to avoid common pitfalls when constructing multiple choice questions items
Eiada al-Faris, Eiad Al-faris

Journal of Family and Community Medicine, 2010

This paper is an att empt to produce a guide for improving the quality of Multiple Choice Questions (MCQs) used in undergraduate and postgraduate assessment. Multiple Choice Questions type is the most frequently used type of assessment worldwide. Well constructed, context rich MCQs have a high reliability per hour of testing. Avoidance of technical items fl aws is essential to improve the validity evidence of MCQs. Technical item fl aws are essentially of two types (i) related to testwiseness, (ii) related to irrelevant diffi culty. A list of such fl aws is presented together with discussion of each fl aw and examples to facilitate learning of this paper and to make it learner friendly. This paper was designed to be interactive with self-assessment exercises followed by the key answer with explanations.

View PDFchevron_right
Automatic generation of analogy questions for student assessment: an Ontology-based approach
Tahani Alsubait

Different computational models for generating analogies of the form ''A is to B as C is to D'' have been proposed over the past 35 years. However, analogy generation is a challenging problem that requires further research. In this article, we present a new approach for generating analogies in Multiple Choice Question (MCQ) format that can be used for students' assessment. We propose to use existing high-quality ontologies as a source for mining analogies to avoid the classic problem of hand-coding concepts in previous methods. We also describe the characteristics of a good analogy question and report on experiments carried out to evaluate the new approach.

View PDFchevron_right

References (51)

  1. L. Aiken. Writing multiple-choice items to measure higher-order educational objectives. Educational and Psychological Measurement, 42(3):803-806, 1982.
  2. M. Al-Yahya. Ontoque: A question generation engine for educational assessment based on domain ontologies. In 11th IEEE International Conference on Advanced Learning Technologies, 2011.
  3. T. Alsubait, B. Parsia, and U. Sattler. Next generation of e-assessment: automatic generation of questions. International Journal of Technology Enhanced Learning, 4(3/4):156-171, 2012.
  4. J. R. Anderson. A spreding activation theory of memory. Journal of Verbal Learning and Verbal Behavior, 22:261-295, 1983.
  5. J. R. Anderson. Cognitive psychology and its implications. New York: W. H. Freeman, 4 edition, 1995.
  6. F. Baader, D. Calvanese, D. L. McGuinness, D. Nardi, and P. F. (eds.) Patel-Schneider. The Description Logic Handbook: Theory, Implemen- tation and Applications. Cambridge University Press, second edition edition, 2007.
  7. D. A. Balota. Automatic semantic activation and episodic memory encoding. Journal of Verbal Learning and Verbal Behavior, 22:88-104, 1983.
  8. D. A. Balota and R. F. Lorch. Depth of automatic spreading Mediated priming effects in pronunciation but not in lexical decision. Journal of Experimental Psychology: Learning, Memory, Cognition, 12:336-345, 1986.
  9. B. S. Bloom and D. R. Krathwohl. Taxonomy of educational objectives: The classification of educational goals by a committee of college and university examiners. Handbook 1. Cognitive domain. New York: Addison-Wesley, 1956.
  10. J. Brown, G. Firshkoff, and M. Eskenazi. Automatic question generation for vocabulary assessment. In Proceedings of HLT/EMNLP, pages 819- 826, Vancuver, Canada, 2005.
  11. J. Carneson, G. Delpierre, and K. Masters. Designing and Managing Multiple Choice Questions. 1996.
  12. M.T.H. Chi, R. Glaser, and E. Rees. Expertise in problem-solving. In R.J. Sternberg (Ed.), Advances in the psychology of human intelligence (Volume 1). Hillsdale, NJ: Erlbaum, 1982.
  13. M.T.H. Chi and R.D. Koeske. Network representation of a childs dinosaur knowledge. Developmental Psychology, 19:2939, 1983.
  14. G. Chung, D. Niemi, and W. L. Bewley. Assessment applications of ontologies. In Paper presented at the Annual Meeting of the American Educational Research Association, 2003.
  15. A. Collins and E. Loftus. A spreading-activation theory of semantic processing. Psychological review, 82(6):407-428, 1975.
  16. A. Collins and M. Quillian. Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Behavior, 8:240-248, 1969.
  17. W. Crawford. Item difficulty as related to the complexity of intellectual processes. Journal of Educational Measurement, 5(2):103-107, 1968.
  18. M. Cubric and M. Tosic. Semcq protg plugin for automatic ontology- driven multiple choice question tests generation. In 11th Intl. Protg Conference, Poster and Demo Session, 2009.
  19. B. B. Davis. Tools for Teaching. San Francisco, CA: Jossey-Bass, 2001.
  20. C. dAmato, S. Staab, and N. Fanizzi. On the inuence of description logics ontologies on conceptual similarity. In EKAW '08 Proceedings of the 16th international conference on Knowledge Engineering: Practice and Patterns, 2008.
  21. H. Fisher-Hoch and S. Hughes. What makes mathematics exam questions difficult. British Educational Research Association, University of Lancaster, England, 1996.
  22. H. Fisher-Hoch, S. Hughes, and T. Bramley. What makes gcse exami- nation questions difficult? outcomes of manipulating difficulty of gcse questions. 1994.
  23. C. Gobbo and M. T. H. Chi. How knowledge is structured and used by expert and novice children. Cognitive Development, 1:221-237, 1986.
  24. GRESampleQuestions. Best sample questions. retrieved march 10, 2012, from http://www.bestsamplequestions.com/gre-questions/analogies/.
  25. L. Guttman. Image theory for the structure of quantitative variates. Psychometrica, 18(4):277-296, 1953.
  26. T.M. Haladyna and S.M. Downing. How many options is enough for a multiple choice test item? Educational & Psychological Measurement, 53(4):999-1010, 1993.
  27. M. Heilman. Automatic Factual Question Generation from Text. PhD thesis, Language Technologies Institute, School of Computer Science, Carnegie Mellon University, 2011.
  28. E. Holohan, M. Melia, D. McMullen, and C. Pahl. The generation of e-learning exercise problems from subject ontologies. In Proceedings of the Sixth IEEE International Conference on Advanced Learning Technologies, pages 967-969, 2006.
  29. A. Hoshino and H. Nakagawa. Real-time multiple choice question generation for language testing: a preliminary study. In Proceedings of the Second Workshop on Building Educational Applications using Natural Language Processing, pages 17-20, Ann Arbor, US, 2005.
  30. K. Janowicz. Sim-dl: Towards a semantic similarity measurement theory for the description logic alcnr in geographic information retrieval. In SeBGIS 2006, OTM Workshops 2006, pages 1681-1692, 2006.
  31. J. Kehoe. Basic item analysis for multiple-choice tests. Practical Assessment, Research & Evaluation, 4(10), 1995.
  32. K. Lehmann and A. Turhan. A framework for semantic-based similarity measures for elh-concepts. JELIA 2012, pages 307-319, 2012.
  33. D. Lin. An information-theoretic definition of similarity. In In: Proc. of the 15th International Conference on Machine Learning, page 296304, San Francisco, CA, 1998. Morgan Kaufmann.
  34. C. Liu, C. Wang, Z. Gao, and S. Huang. Applications of lexical information for algorithmically composing multiple-choice cloze items. In Proceedings of the Second Workshop on Building Educational Ap- plications using Natural Language Processing, pages 1-8, Ann Arbor, US, 2005.
  35. E. F. Loftus. Activation of semantic memory. American Journal of Psychology, 86:331-337, 1974.
  36. J. Lowman. Mastering the Techniques of Teaching (2nd ed.). San Francisco: Jossey-Bass, 1995.
  37. M. Miller, R. Linn, and N. Gronlund. Measurement and Assessment in Teaching, Tenth Edition. Pearson, 2008.
  38. R. Mitkov, L. An Ha, and N. Karamani. A computer-aided environment for generating multiple-choice test items.cambridge university press. Natural Language Engineering, 12(2):177-194, 2006.
  39. R. Mitkov and L.A. Ha. Computer-aided generation of multiple-choice tests. In Proceedings of the First Workshop on Building Educational Ap- plications using Natural Language Processing, pages 17-22, Edmonton, Canada, 2003.
  40. R. Mitkov, L.A. Ha, A. Varga, and L. Rello. Semantic similarity of distractors in multiple-choice tests: Extrinsic evaluation. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 49-56, Athens, Greece, March 2009. Association for Computa- tional Linguistics.
  41. A. Papasalouros, K. Kotis, and K. Kanaris. Automatic generation of multiple-choice questions from domain ontologies. In IADIS e-Learning 2008 conference, Amsterdam, 2008.
  42. M. Paxton. A linguistic perspective on multiple choice questioning. Assessment & Evaluation in Higher Education, 25(2):109-119, 2001.
  43. R. Rada, H. Mili, E. Bicknell, and M. Blettner. Development and application of a metric on semantic nets. In In: IEEE Transaction on Systems, Man, and Cybernetics, volume 19, page 1730, 1989.
  44. P. Resnik. Using information content to evaluate semantic similarity in a taxonomy. In In Proceedings of the 14th international joint conference on Artificial intelligence (IJCAI95), volume 1, pages 448-453, 1995.
  45. G. M. Seddon. The properties of Bloom's taxonomy of educational objectives for the cognitive domain. Review of Educational Research, 48(2):pp. 303-323, 1978.
  46. F. Sharifian and R. Samani. Hierarchical spreading of activation. In In F. Sharifian (Ed.) Proc. of the Conference on Language, Cognition, and Interpretation, pages 1-10, 1997.
  47. J. T. Sidick, G. V. Barrett, and D. Doverspike. Three-alternative multiple- choice tests: An attractive option. Personnel Psychology, 47:829-835, 1994.
  48. R. Smith. An empirical investigation of complexity and process in multiple-choice items. Journal of Educational Measurement, 7(1):33- 41, 1970.
  49. J. W.Pellegrino, N. Chudowsky, and R. Glaser, editors. Knowing what Students Know: The Science and Design of Educational Assessment. NATIONAL ACADEMY PRESS, Washington, DC, 2001.
  50. Z. Wu and MS. Palmer. Verb semantics and lexical selection. In In: Proceedings of the 32nd. Annual Meeting of the Association for Computational Linguistics (ACL 1994), page 133138, 1994.
  51. B. Zitko, S. Stankov, M. Rosic, and A. Grubisic. Dynamic test generation over ontology-based knowledge representation in authoring shell. Expert Systems with Applications: An International Journal, 36(4):8185-8196, 2008.

Related papers

Controlling item difficulty for automatic vocabulary question generation
Hiroyuki Obari

Research and Practice in Technology Enhanced Learning

The present study investigates the best factor for controlling the item difficulty of multiple-choice English vocabulary questions generated by an automatic question generation system. Three factors are considered for controlling item difficulty: (1) reading passage difficulty, (2) semantic similarity between the correct answer and distractors, and (3) the distractor word difficulty level. An experiment was conducted by administering machine-generated items to three groups of English learners. The groups were determined based on their standardised English test scores. In total, 120 items, generated using combinations of the above three factors, were tested. The results reveal that the distractor word difficulty level had the greatest impact on item difficulty, but this tendency changed depending on the proficiency of the test takers. These results will be of use when implementing a fully automatic system for administrating tests.

View PDFchevron_right
Computer-aided generation of multiple-choice tests
Lệ Hà

Proceedings of the HLT-NAACL 03 workshop on Building educational applications using natural language processing -, 2003

This paper describes a novel computer-aided procedure for generating multiple-choice tests from electronic instructional documents. In addition to employing various NLP techniques including term extraction and shallow parsing, the program makes use of language resources such as a corpus and WordNet. The system generates test questions and distractors, offering the user the option to post-edit the test items.

View PDFchevron_right
QuesGen: Clarifying the Dimensions of Multiple-Choice Question Quality
Marilyn Tremaine

understanding of these factors remains a challenge. Yet, classroom instructors must understand the dimensions of question quality, else how will they know whether or not they have constructed good questions? An experiment was conducted to test the effectiveness of QuesGen-a webbased system designed to help instructors write better MCQs. Whereas in the professional testing industry advanced metrics based on item response theory (IRT) exist to measure question quality, in classroom settings, IRT-based metrics are impractical. This paper reports the results of the experiment that have an impact on the understanding of the dimensions of MCQ quality, and suggests designs for software that simultaneously support teaching and deepen practice.

View PDFchevron_right
Improving the Output from Software that Generates Multiple Choice Question (MCQ) Test Items Automatically using Controlled Rhetorical Structure Theory
Robert Foster

Student ReSeaRch WoRkShop

A combination of established theories [1].[2].[3].[4] are applied in an attempt to improve the output from a system [5],[6] which automatically generates MCQ (Multiple Choice Question) test items from source documents. The literature observes that NLG (Natural Language Generation) system evaluation is non-trivial [7] and so the method is evaluated using a process suited to the featured domain [8]. The experiment intersperses 38 MCQ test items whose question stems have been generated using Controlled Rhetorical Structure Theory (CRST) with 62 manually created MCQ test items to form an item bank. A usability score is assigned to each item by a domain expert and these scores are used in the evaluation of the effectiveness of the method. The results provide some evidence to support the incorporation of CRST into future versions of the software. .

View PDFchevron_right
Integrating automatic question generation with computerised adaptive test
Yuni Susanti

Research and Practice in Technology Enhanced Learning, 2020

The present study focuses on the integration of an automatic question generation (AQG) system and a computerised adaptive test (CAT). We conducted two experiments. In the first experiment, we administered sets of questions to English learners to gather their responses. We further used their responses in the second experiment, which is a simulation-based experiment of the AQG and CAT integration. We proposed a method to integrate them with a predetermined item difficulty that enables to integrate AQG and CAT without administering the items in a pretesting. The result showed that all CAT simulations performed better than the baseline, a linear test, in estimating the test taker’s true proficiency.

View PDFchevron_right

Related topics

  • Assessment
  • Semantic similarity
  • Question Generation
  • MCQ-Creation
  • MCQ Generation
  • Automatic Item Generation
    • 580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved