Academia.eduAcademia.edu

Outline

Title
Abstract
FAQs
Figures
Introduction
Background on Ontologies
The Question Generation Task
Experimental Evaluation
Participants and Method
Results
Discussion
Recommendations
Conclusion and Future Work
References
All Topics
Computer Science
Artificial Intelligence

Ontology-Based Multiple Choice Question Generation

Profile image of Maha Al-YahyaMaha Al-Yahya

2014, The Scientific World Journal

https://doi.org/10.1155/2014/274949
visibility

…

description

10 pages

link

1 file

Abstract

With recent advancements in Semantic Web technologies, a new trend in MCQ item generation has emerged through the use of ontologies. Ontologies are knowledge representation structures that formally describe entities in a domain and their relationships, thus enabling automated inference and reasoning. Ontology-based MCQ item generation is still in its infancy, but substantial research efforts are being made in the field. However, the applicability of these models for use in an educational setting has not been thoroughly evaluated. In this paper, we present an experimental evaluation of an ontology-based MCQ item generation system known as OntoQue. The evaluation was conducted using two different domain ontologies. The findings of this study show that ontology-based MCQ generation systems produce satisfactory MCQ items to a certain extent. However, the evaluation also revealed a number of shortcomings with current ontology-based MCQ item generation systems with regard to the education...

FAQs

sparkles

AI

What factors influence the quality of ontology-based MCQs generated by OntoQue?add

The evaluation noted an average quality score of 8.8 out of 10, with distracter plausibility being a significant factor affecting quality. Specifically, rule violations identified include issues with distracters being too easy or not plausible, particularly in the HistOnto ontology.

How do different ontologies affect the educational significance of generated MCQs?add

The study found that MCQs generated from the SemQ ontology were considered more educationally significant than those from the HistOnto ontology, which had nearly half classified as irrelevant. This was attributed to the kind of domain knowledge represented, with factual knowledge being less useful.

What were the main methodologies used for MCQ generation in this study?add

The study utilized class-based, individual-based, and property-based strategies for MCQ generation in the OntoQue system. Each strategy involved deriving stems and distracters from RDF statements within ontological structures.

What recommendations were made for improving ontology-based MCQ generation systems?add

The paper recommends incorporating cognitive levels and learning objectives to enhance educational relevance and developing strategies to create plausible distracters. Additionally, it suggests integrating linguistic knowledge to improve item language and structure.

Which evaluation metrics were used to assess the quality of generated MCQs?add

Key metrics included the total number of worthy MCQ items, modification measures, and adherence to MCQ design rules, with non-revision rates signaling item quality.

Figures (11)
Finally, Section 8 provides our conclusions and highlights avenues for future research. the classes “book” and “journal” and the relations “has- author” and “has-publication-date.” It may also state that “book” and “journal” are types of “publications.” Moreover, the relationships in the ontology may define certain con- straints, such as “a book must have at least one author.’ With regard to assertional knowledge, the “library” ontology may assert the fact “Book: A Tale of Two Cities has-author: Charles Dickens” Ontology entities translated into assertional and terminological knowledge about a domain represent a rich resource from which MCQ items can be automatically generated. They represent asserted facts about a specific domain in a machine-understandable way. Table 1 shows a number of facts and axioms from the bibtex [2] ontology represented in natural language.
Finally, Section 8 provides our conclusions and highlights avenues for future research. the classes “book” and “journal” and the relations “has- author” and “has-publication-date.” It may also state that “book” and “journal” are types of “publications.” Moreover, the relationships in the ontology may define certain con- straints, such as “a book must have at least one author.’ With regard to assertional knowledge, the “library” ontology may assert the fact “Book: A Tale of Two Cities has-author: Charles Dickens” Ontology entities translated into assertional and terminological knowledge about a domain represent a rich resource from which MCQ items can be automatically generated. They represent asserted facts about a specific domain in a machine-understandable way. Table 1 shows a number of facts and axioms from the bibtex [2] ontology represented in natural language.
TABLE 1: Sample facts and axioms from the bibtex ontology. for the entry book in the bibtex ontology [2]. According to OWL ontology, a statement is composed of three major elements, subject, predicate, and object.
TABLE 1: Sample facts and axioms from the bibtex ontology. for the entry book in the bibtex ontology [2]. According to OWL ontology, a statement is composed of three major elements, subject, predicate, and object.
FrGure 1: Worthy versus unworthy items.
FrGure 1: Worthy versus unworthy items.
TABLE 2: Ontology statistics. These statements then form the base from which the stem is generated. Subjects and objects are always related through a particular property, and the engine contains algorithms for constructing items from property axioms (characteristics), such as transitive properties.
TABLE 2: Ontology statistics. These statements then form the base from which the stem is generated. Subjects and objects are always related through a particular property, and the engine contains algorithms for constructing items from property axioms (characteristics), such as transitive properties.
FiGurRE 2: Modification level required.
FiGurRE 2: Modification level required.
TABLE 4: MCQ design rules. TABLE 5: Results of the MCQ generation task.
TABLE 4: MCQ design rules. TABLE 5: Results of the MCQ generation task.
TABLE 6: Reasons for MCQ major modifications.
TABLE 6: Reasons for MCQ major modifications.
TABLE 3: Number of MCQ items generated.
TABLE 3: Number of MCQ items generated.
than one correct answer. These are all due to technical issues (program bugs) and can be resolved in future versions of the system. Distracter plausibility is an important aspect of an MCQ item. It is considered one of the major challenges facing instructors when constructing MCQ items. Although the system uses algorithms and techniques to identify options that are similar to the key as distracters, results indicate that this is not sufficient. A plausible distracter makes students think; it is compelling, but at the same time, confusing. One possible approach for creating plausible distracters is to consider students’ frequent areas of misunderstanding or their common errors.
than one correct answer. These are all due to technical issues (program bugs) and can be resolved in future versions of the system. Distracter plausibility is an important aspect of an MCQ item. It is considered one of the major challenges facing instructors when constructing MCQ items. Although the system uses algorithms and techniques to identify options that are similar to the key as distracters, results indicate that this is not sufficient. A plausible distracter makes students think; it is compelling, but at the same time, confusing. One possible approach for creating plausible distracters is to consider students’ frequent areas of misunderstanding or their common errors.

Related papers

Automatic generation of multiple-choice questions from domain ontologies
Konstantinos I Kotis

2008

ABSTRACT The aim of this paper is to present an innovative approach for generating multiple choice questions in automatic way. Although other approaches have been already reported in the literature, the approach presented in this paper is based on domain specific ontologies and it is independent of lexicons such as WordNet or other linguistic resources.

View PDFchevron_right
Evaluating the quality of the ontology-based auto-generated questions
mona demaidi

Smart Learning Environments, 2017

An ontology is a knowledge representation structure which has been used in Virtual Learning Environments (VLEs) to describe educational courses by capturing the concepts and the relationships between them. Several ontology-based question generators used ontologies to auto-generate questions, which aimed to assess students' at different levels in Bloom's taxonomy. However, the evaluation of the questions was confined to measuring the qualitative satisfaction of domain experts and students. None of the question generators tested the questions on students and analysed the quality of the auto-generated questions by examining the question's difficulty, and the question's ability to discriminate between high ability and low ability students. The lack of quantitative analysis resulted in having no evidence on the quality of questions, and how the quality is affected by the ontology-based generation strategies, and the level of question in Bloom's taxonomy (determined by the question's stem templates). This paper presents an experiment carried out to address the drawbacks mentioned above by achieving two objectives. First, it assesses the auto-generated questions' difficulty, discrimination, and reliability using two statistical methods: Classical Test Theory (CTT) and Item Response Theory (IRT). Second, it studies the effect of the ontology-based generation strategies and the level of the questions in Bloom's taxonomy on the quality of the questions. This will provide guidance for developers and researchers working in the field of ontology-based question generators, and help building a prediction model using machine learning techniques.

View PDFchevron_right
Evaluation of an automatic question generation approach using ontologies
Noor Teo

2017

Advancements in Semantic Web techniques have led to the emergence of ontology based question generation. Ontologies are used to represent domain knowledge in the form of concepts, instances and their relationships as their elements. Many research strategies for generating questions using ontologies have been proposed but little work has been done on investigating whether an ontology is an appropriate source of data for question generation. Since there is no standard guideline for developing an ontology, the representation of ontology elements might vary in many ways, and this paper aims to investigate how the naming of ontology elements would affect the questions generated. In order to achieve this aim, two research questions will be investigated which are: how many correct questions can be generated from an ontology, and what are the reasons for incorrect questions being generated. Categorized question templates and a set of question strategies for mapping templates with a concept ...

View PDFchevron_right
Towards automatic generation of e-assessment using semantic web technologies
Milorad Tosic

2011

Semantic web technologies have been increasingly used as a tool for generating, organizing and personalizing e-learning content, including e-assessment. In this paper we discuss and extend an innovative approach to automated generation of computer-assisted assessment (CAA) from semantic web-based domain ontologies. We expand the work previously done in this area in two important directions: first, we add new ontology elements (annotations), to the meta-ontology used for generating questions; second, we add a semantic interpretation of the mapping between the ‘domain' ontology and the target ‘question' ontology. The semantic interpretation is based on the notion of ‘question templates' that are founded on Bloom's taxonomy of educational objectives, but can be sourced equally well in any other pedagogical premise applicable to question design and content (e.g. Kolb's learning theory). We use the examples obtained from the prototype implementation to show how that w...

View PDFchevron_right
Ontologies for a Semantic Quiz Architecture
Ioan Roxin

2009

In this paper we propose an architecture for a semantic quiz. In the context of a semantic LCMS we introduce two ontologies: the quiz ontology and the LM ontology. The quiz ontology is associated with the LM ontology that is related with the field of higher education in Europe. We present an application that uses these two ontologies. With the quiz ontology we build semantics metadata that express the question; thus each question to be identifiable and reusable. This solution may be a supplementary one to the classical pedagogical methods, in order to constitute the student profile and learning path, knowing that each question can contribute to the validation of the student competences.

View PDFchevron_right
OeLE: Applying ontologies to support the evaluation of open questions-based tests
Dagoberto CASTELLANOS NIEVES

2005

The use of Semantic Web Technologies in E-learning turns to be more and more significant in the last years. In this paper, an approach that makes use of semantic web technologies to support evaluation processes in e-learning courses is described. In this work, ontologies are used to model the knowledge of the courses and to annotate the knowledge contained in the examination tests. Moreover, semantic web and E-learning standards are included in this evaluation approach, which emphasizes on open questionsbased evaluation tests.

View PDFchevron_right
Ontology-Based Generation of Medical, Multi-term MCQs
ghader kurdi

International Journal of Artificial Intelligence in Education, 2019

Designing good multiple choice questions (MCQs) for education and assessment is time consuming and error-prone. An abundance of structured and semi-structured data has led to the development of automatic MCQ generation methods. Recently, ontologies have emerged as powerful tools to enable the automatic generation of MCQs. However, current question generation approaches focus on knowledge recall questions. In addition, questions that have so far been generated are, compared to manually created ones, simple and cover only a small subset of the required question complexity space in the education and assessment domain. In this paper, we focus on addressing the limitations of previous approaches by generating questions with complex stems that are suitable for scenarios beyond mere knowledge recall. We present a novel ontology-based approach that exploits classes and existential restrictions to generate case-based questions. Our contribution lies in: (1) the specification of procedure for generating case-based questions which involve (a) assembling complex stems, (b) selecting suitable options, and (c) providing explanations for option correctness/incorrectness, (2) an implementation of the procedure using a medical ontology and (3) and evaluation of our generation technique to test question quality and their suitability in practise. We implement our approach as an application for a medical education scenario on top of a large knowledge base in the medical domain. We generate more than 3 million questions for four physician specialities and evaluate our approach in a user study with 15 medical experts. We find that using a stratified random sample of 435 questions out of which 316 were rated by two experts, 129 (30%) are considered appropriate to be used in exams by both experts and a further 216 (50%) by at least one expert.

View PDFchevron_right
EAOnto: a heavyweight ontology for supporting e-assessment generation
Lucila Romero, Milagros Gutierrez

2012

Different approaches that use ontology as a tool for generating, organizing and personalizing e-learning content can be found in literature. In the context of e-assessment, the notion of ontology has been used for different purposes. One of them is the use of ontologies as a mean of providing a structure to guide the automated design of assessments. Despite of the effort done in this direction, there is still a lot of work to be done since the most of the approaches have proposed lightweight ontologies that model a part of the assessment domain without considering pedagogical aspects. This paper presents a heavyweight ontology, called EAOnto, that conceptualizes the eassessment domain including a set of rules representing pedagogical principles that are worth to follow. In addition, an empirical evaluation is discussed.

View PDFchevron_right
Measuring the Quality of Assessment Using Questions Generated from the Semantic Web
BEATRIZ BARROS

Lecture Notes in Computer Science, 2018

This article describes a new feature of the adaptive assessment system SIETTE that allows for the static and dynamic generation of questions from tables of data for knowledge assessment. Almost the same approach can be used to generate questions from data collected in a spreadsheet, a database query, or a semantic web query using SPARQL. The main problem faced with question generation is ensuring that the questions are valid for assessment. For this reason, most of the existing systems propose to use this mechanism only for low-stakes assessments. In this paper, we propose a methodology to control question generation quality and measure the impact of potential invalid instances on the final score as well as recommend some strategies to overcome these problems.

View PDFchevron_right
OeLE: Applying ontologies to support the evaluation of open questions-based
Pedro Vivancos

The use of Semantic Web Technologies in E-learning turns to be more and more significant in the last years. In this paper, an approach that makes use of semantic web technologies to support evaluation processes in e-learning courses is described. In this work, ontologies are used to model the knowledge of the courses and to annotate the knowledge contained in the examination tests. Moreover, semantic web and E-learning standards are included in this evaluation approach, which emphasizes on open questionsbased evaluation tests.

View PDFchevron_right

References (30)

  1. M. Al-Yahya, "OntoQue: a question generation engine for edu- cational assesment based on domain ontologies, " in Proceedings of the 11th IEEE International Conference on Advanced Learning Technologies (ICALT '11), pp. 393-395, Athens, Ga, USA, July 2011.
  2. N. Knouf, "Transformation of bibTeX into an OWL ontology, " 2003, http://oaei.ontologymatching.org/2004/Contest/301/on- to.html.
  3. A. Papasalouros, K. Kanaris, and K. Kotis, "Automatic genera- tion of multiple choice questions from domain ontologies, " in Proceedings of the IADIS e-Learning Conference, pp. 427-434, Amsterdam, The Netherlands, July 2008.
  4. J. C. Brown, G. A. Frishkoff, and M. Eskenazi, "Automatic question generation for vocabulary assessment, " in Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 819-826, Strouds- burg, Pa, USA, October 2005.
  5. R. Mitkov, L. A. Ha, and N. Karamanis, "A computer-aided environment for generating multiple-choice test items, " Natural Language Engineering, vol. 12, no. 2, pp. 177-194, 2006.
  6. J. H. Wolfe, "An Aid to Independent Study through Automatic Question Generation (AUTOQUEST), " October 1975.
  7. T. Hirashima, "Metadata editor of exercise problems for intel- ligent e-learning, " in Proceedings of the Workshop Notes of Applications of Semantic Web Technologies for E-Learning at (ISWC '04), pp. 99-101, Hirosjima, Japan, 2004.
  8. D. Gates, "Generating look-back strategy questions from expos- itory texts, " in Proceedings of the Workshop on the Question Generation Shared Task and Evaluation Challenge, Arlington, Va, USA, 2008.
  9. A. N. Kumar, "Generation of problems, answers, grade, and feedback-case study of a fully automated tutor, " ACM Journal on Educational Resources in Computing, vol. 5, no. 3, 2005.
  10. S. Ou, C. Orasan, D. Mekhaldi, and L. Hasler, "Automatic ques- tion pattern generation for ontology-based question answer- ing, " in Proceedings of the 21th International Florida Artificial Intelligence Research Society Conference (FLAIRS '08), pp. 183- 188, Menlo Park, Calif, USA, May 2008.
  11. E. Hovy, L. Gerber, U. Hermjakob, C. Lin, and D. Ravichandran, "Toward Semantics-Based Answer Pinpointing, " 2001.
  12. C. Fellbaum, WordNet, MIT Press, 1998.
  13. A. Corbett and J. Mostow, "Automating comprehension ques- tions: lessons from a reading tutor, " in Proceedings of the Work- shop on the Question Generation Shared Task and Evaluation Challenge, Arlington, Va, USA, 2008.
  14. R. Mitkov, L. A. Ha, and N. Karamanis, "A computer-aided environment for generating multiple-choice test items, " Natural Language Engineering, vol. 12, no. 2, pp. 177-194, 2006.
  15. N. Silveira, "Towards a framework for question generation, " in Proceedings of the Workshop on the Question Generation Shared Task and Evaluation Challenge, Arlington, Va, USA, 2008.
  16. L. Stanescu, C. S. Spahiu, A. Ion, and A. Spahiu, "Question generation for learning evaluation, " in Proceedings of the Inter- national Multiconference on Computer Science and Information Technology (IMCSIT '08), pp. 509-513, Wisla, Poland, October 2008.
  17. B. Cecila Alves, J. Mark Gierl, and Lai Hollis, "Using automated item generation to promote principled test design and devel- opment, " in Proceedings of the American Educational Research Association (AERA '10), Denver, Colo, USA, 2010.
  18. M. J. Gierl, J. Zhou, and C. Alves, "Developing a taxonomy of item model types to promote assessment engineering, " Journal of Technology, Learning, and Assessment, vol. 7, no. 2, pp. 1-50, 2008.
  19. E. Holohan, M. Melia, D. McMullen, and C. Pahl, "Adaptive E- learning content generation based on semantic web technol- ogy, " in Proceedings of the International Workshop on Applica- tions of Semantic Web Technologies for E-learning, Amsterdam, The Netherlands, 2005.
  20. M. Tosic and M. Cubric, "SeMCQ-Protégé plugin for auto- matic ontology-driven multiple choice question tests genera- tion, " in Proceedings of the 11th International Protégé Conference, Amsterdam, The Netherlands, 2009.
  21. H. Knublauch, R. Fergerson, N. Noy, and M. Musen, "The Protégé OWL plugin: an open development environment for semantic web applications, " in The Semantic Web-ISWC 2004, vol. 3298, pp. 229-243, Springer, Berlin, Germany, 2004.
  22. B. Žitko, S. Stankov, M. Rosić, and A. Grubišić, "Dynamic test generation over ontology-based knowledge representation in authoring shell, " Expert Systems with Applications, vol. 36, pp. 8185-8196, 2009.
  23. I. Leshcheva, D. Gorovaya, and D. Leshchev, "Ontology-based assessment technique, " in Proceedings of the 2nd International Workshop on Semantic Web Applications in Higher Education (SemHE '10), Southampton, UK, 2010.
  24. C. D. Bailey, J. N. Karcher, and B. Clevenger, "A comparison of the quality of multiple-choice questions from CPA exams and textbook test banks, " The Accounting Educators' Journal, vol. 10, no. 2, 1998.
  25. P. Buitelaar, P. Cimiano, P. Haase, and M. Sintek, "Towards linguistically grounded ontologies, " in Proceedings of the 6th European Semantic Web Conference on The Semantic Web: Research and Applications, pp. 111-125, Heraklion, Crete, Greece, 2009.
  26. L. R. Aiken, "Testing with multiple-choice items, " Journal of Research & Development in Education, vol. 20, no. 4, pp. 44-58, 1987.
  27. L. C. Jacobs and C. I. Chase, Developing and Using Tests Effectively: A Guide For Faculty, Jossey-Bass, San Francisco, Calif, USA, 1992.
  28. K. Woodford and P. Bancroft, "Multiple choice questions not considered harmful, " in Proceedings of the 7th Australasian Conference on Computing Education(ACE '05), A. Young and D. Tolhurst, Eds., vol. 42, pp. 109-116, Australian Computer Society, Darlinghurst, Australia, 2005.
  29. G. Francopoulo, M. George, N. Calzolari et al., "Lexical Markup Framework (LMF), " in Proceedings of the Language Resources and Evaluation Conference (LREC '06), Genoa, Italy, 2006.
  30. N. E. Gronlund, Measurement and Evaluation in Teaching, Macmillan, 1985.

Related papers

Generating Multiple Choice Questions From Ontologies: How Far Can We Go?
Tahani Alsubait

Ontology-based Multiple Choice Question (MCQ) generation has a relatively short history. Many attempts have been carried out to develop methods to generate MCQs from ontologies. However, there is a still a need to understand the applicability of these methods in real educational settings. In this paper, we present an empirical evaluation of ontology-based MCQ generation. We examine the feasibility of applying ontology-based MCQ generation methods by educators with no prior experience in ontology building. The findings of this study show that this is feasible and can result in generating a reasonable number of educationally useful questions with good predictions about their difficulty levels.

View PDFchevron_right
Generating Multiple Choice Questions From Ontologies: Lessons Learnt
Tahani Alsubait

Ontologies with potential educational value are available in di↵erent domains. However, it is still unclear how such ontologies can be exploited to generate useful instructional content, e.g., assessment questions. In this paper, we present an approach to automatically generate multiple-choice questions from OWL ontologies. We describe a psychologically-plausible theory to control the diculty of questions. Our contributions include designing a protocol to evaluate the characteristics of the generated questions, in particular, question diculty. We report on our experience on generating questions from ontologies and present promising results of evaluating our diculty-control theory.

View PDFchevron_right
An Experimental Evaluation of Automatically Generated Multiple Choice Questions from Ontologies
ghader kurdi

Lecture Notes in Computer Science, 2017

In order to provide support for the construction of MCQs, there have been recent efforts to generate MCQs with controlled difficulty from OWL ontologies. Preliminary evaluation suggests that automatically generated questions are not field ready yet and highlight the need for further evaluations. In this study, we have presented an extensive evaluation of automatically generated MCQs. We found that even questions that adhere to guidelines are subject to the clustering of distractors. Hence, the clustering of distractors must be realised as this could affect the prediction of difficulty.

View PDFchevron_right
The Development of MCQ generating system based on Ontology Concepts
nor aiza moketar

International Journal of Advanced Trends in Computer Science and Engineering

Education system needs an effective method to evaluate students in a competent way on their major concepts learned from their studies. Generating Multiple Choice Questions (MCQ) process is cumbersome and needs a lot of effort especially among novice instructors. The current MCQ generators do not automatically generate questions but only randomly display questions from the question bank. However, it is not easy to load a massive number of questions in the question banks where it is normally done in stages to have massive question collection in the question bank. Therefore, this paper proposes an automatic question generator specifically for MCQ to enable a massive number of questions generated in just a few seconds. In addition, it is useful for students who have to do self-learning for their tutorial exercises especially during the COVID-19 pandemic crisis where online learning is widely used to replace the traditional face-to-face sessions. The generator has adopted an ontological approach for question generation strategies, and it is implemented using rule-based reasoning. There are six modules proposed and discussed in this paper including the concept extraction from ontology, concept, and question stem mapping, generating appropriate answer options using ontology relation information and reordering of answer options. The functionality and validation test showed that Multiple Choice Question Generator (MCQ-G) can generate MCQ with appropriate answer options. This work will be extended in the future using the difference domain ontology and question types.

View PDFchevron_right
Validation of Course Ontology Elements for Automatic Question Generation
Noor Teo

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2016

Recent research has led to the emergence of ontology-based question generation and aims to benefit instructors by providing support and intelligent assistance for the automatic generation of questions. However, existing ontologies are not designed mainly for this purpose, and the concern is that an ontology will not be competent enough to act as a semantic source for the question generation process. Therefore, the aim of this work is to validate how well the elements represented in course ontology can be used for the purpose of automatic question generation. In this work, we choose to validate Operating System ontologies and identify related question sources from textbooks on this subject as competency questions. Finally, the result shows that the evaluated ontologies can provide knowledge for generating useful assessment questions. Furthermore, the list of categorized question templates and their variations are generated using a strategy based on the validated ontology.

View PDFchevron_right

Related topics

  • Mathematics
  • Computer Science
  • Artificial Intelligence
  • Ontology
  • Assessment
  • Semantics
  • Ontology (Computer Science)
  • Semantic similarity
  • Semantic Web technology - Ontolo...
  • Medicine
  • MCQ Generation
  • Choice Behavior
  • Automatic Item Generation

    • Cited by

    An Ontology-Driven Learning Assessment Using the Script Concordance Test
    Nenad Petrović

    Applied Sciences

    Assessing the level of domain-specific reasoning acquired by students is one of the major challenges in education particularly in medical education. Considering the importance of clinical reasoning in preclinical and clinical practice, it is necessary to evaluate students’ learning achievements accordingly. The traditional way of assessing clinical reasoning includes long-case exams, oral exams, and objective structured clinical examinations. However, the traditional assessment techniques are not enough to answer emerging requirements in the new reality due to limited scalability and difficulty for adoption in online education. In recent decades, the script concordance test (SCT) has emerged as a promising tool for assessment, particularly in medical education. The question is whether the usability of SCT could be raised to a level high enough to match the current education requirements by exploiting opportunities that new technologies provide, particularly semantic knowledge graphs...

    View PDFchevron_right
    A comparative study of methods for a priori prediction of MCQ difficulty
    ghader kurdi

    Semantic Web, 2021

    Successful exams require a balance of easy, medium, and difficult questions. Question difficulty is generally either estimated by an expert or determined after an exam is taken. The latter provides no utility for the generation of new questions and the former is expensive both in terms of time and cost. Additionally, it is not known whether expert prediction is indeed a good proxy for estimating question difficulty. In this paper, we analyse and compare two ontology-based measures for difficulty prediction of multiple choice questions, as well as comparing each measure with expert prediction (by 15 experts) against the exam performance of 12 residents over a corpus of 231 medical case-based questions that are in multiple choice format. We find one ontology-based measure (relation strength indicativeness) to be of comparable performance (accuracy = 47%) to expert prediction (average accuracy = 49%).

    View PDFchevron_right
    Ontology-Based Generation of Medical, Multi-term MCQs
    ghader kurdi

    International Journal of Artificial Intelligence in Education, 2019

    Designing good multiple choice questions (MCQs) for education and assessment is time consuming and error-prone. An abundance of structured and semi-structured data has led to the development of automatic MCQ generation methods. Recently, ontologies have emerged as powerful tools to enable the automatic generation of MCQs. However, current question generation approaches focus on knowledge recall questions. In addition, questions that have so far been generated are, compared to manually created ones, simple and cover only a small subset of the required question complexity space in the education and assessment domain. In this paper, we focus on addressing the limitations of previous approaches by generating questions with complex stems that are suitable for scenarios beyond mere knowledge recall. We present a novel ontology-based approach that exploits classes and existential restrictions to generate case-based questions. Our contribution lies in: (1) the specification of procedure for generating case-based questions which involve (a) assembling complex stems, (b) selecting suitable options, and (c) providing explanations for option correctness/incorrectness, (2) an implementation of the procedure using a medical ontology and (3) and evaluation of our generation technique to test question quality and their suitability in practise. We implement our approach as an application for a medical education scenario on top of a large knowledge base in the medical domain. We generate more than 3 million questions for four physician specialities and evaluate our approach in a user study with 15 medical experts. We find that using a stratified random sample of 435 questions out of which 316 were rated by two experts, 129 (30%) are considered appropriate to be used in exams by both experts and a further 216 (50%) by at least one expert.

    View PDFchevron_right
    Evaluating the quality of the ontology-based auto-generated questions
    mona demaidi

    Smart Learning Environments, 2017

    An ontology is a knowledge representation structure which has been used in Virtual Learning Environments (VLEs) to describe educational courses by capturing the concepts and the relationships between them. Several ontology-based question generators used ontologies to auto-generate questions, which aimed to assess students' at different levels in Bloom's taxonomy. However, the evaluation of the questions was confined to measuring the qualitative satisfaction of domain experts and students. None of the question generators tested the questions on students and analysed the quality of the auto-generated questions by examining the question's difficulty, and the question's ability to discriminate between high ability and low ability students. The lack of quantitative analysis resulted in having no evidence on the quality of questions, and how the quality is affected by the ontology-based generation strategies, and the level of question in Bloom's taxonomy (determined by the question's stem templates). This paper presents an experiment carried out to address the drawbacks mentioned above by achieving two objectives. First, it assesses the auto-generated questions' difficulty, discrimination, and reliability using two statistical methods: Classical Test Theory (CTT) and Item Response Theory (IRT). Second, it studies the effect of the ontology-based generation strategies and the level of the questions in Bloom's taxonomy on the quality of the questions. This will provide guidance for developers and researchers working in the field of ontology-based question generators, and help building a prediction model using machine learning techniques.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved