Validating Ontologies for Question Generation

Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, 2016

Recent research has led to the emergence of ontology-based question generation and aims to benefit instructors by providing support and intelligent assistance for the automatic generation of questions. However, existing ontologies are not designed mainly for this purpose, and the concern is that an ontology will not be competent enough to act as a semantic source for the question generation process. Therefore, the aim of this work is to validate how well the elements represented in course ontology can be used for the purpose of automatic question generation. In this work, we choose to validate Operating System ontologies and identify related question sources from textbooks on this subject as competency questions. Finally, the result shows that the evaluated ontologies can provide knowledge for generating useful assessment questions. Furthermore, the list of categorized question templates and their variations are generated using a strategy based on the validated ontology.

Ontology-based Multiple Choice Question (MCQ) generation has a relatively short history. Many attempts have been carried out to develop methods to generate MCQs from ontologies. However, there is a still a need to understand the applicability of these methods in real educational settings. In this paper, we present an empirical evaluation of ontology-based MCQ generation. We examine the feasibility of applying ontology-based MCQ generation methods by educators with no prior experience in ontology building. The findings of this study show that this is feasible and can result in generating a reasonable number of educationally useful questions with good predictions about their difficulty levels.

Multiple choice questions (MCQs) are considered highly useful (being easy to take or mark) but quite difficult to create and large numbers are needed to form valid exams and associated practice materials. The idea of re-using an existing ontology to generate MCQs almost suggests itself and has been explored in various projects. In this project, we are applying suitable educational theory regarding assessments and related methods for their evaluation to ontology-based MCQ generation. In particular, we investigate whether we can measure the similarity of the concepts in an ontology with sufficient reliability so that this measure can be used to control the difficulty of the MCQs generated. In this report, we provide an overview of the background to this research, and describe the main steps taken and insights gained.

2000

: The goal of this project was to enable knowledge engineers to construct knowledge bases (KBs) faster. To achieve this goal, we investigated two techniques: knowledge reuse and axiom templates. The results were demonstrated by developing a question-answering system for the crisis management challenge problem (CMCP). The solution of the CMCP required addressing problems broader than just knowledge reuse and axiom templates. The technical issues addressed in the process can be broadly classified into two categories: knowledge base content development and knowledge server development. We developed a geo-political KB for the CMCP. Specifically, we represented knowledge about international actions, terrorist groups, terrorist events, military capabilities, escalation and de-escalation of conflict, threats, and so forth. In the process of encoding the domain knowledge, we developed and adapted techniques for representing temporal knowledge and qualitative influences, and identified princ...

2008

ABSTRACT The aim of this paper is to present an innovative approach for generating multiple choice questions in automatic way. Although other approaches have been already reported in the literature, the approach presented in this paper is based on domain specific ontologies and it is independent of lexicons such as WordNet or other linguistic resources.

Ontologies with potential educational value are available in di↵erent domains. However, it is still unclear how such ontologies can be exploited to generate useful instructional content, e.g., assessment questions. In this paper, we present an approach to automatically generate multiple-choice questions from OWL ontologies. We describe a psychologically-plausible theory to control the diculty of questions. Our contributions include designing a protocol to evaluate the characteristics of the generated questions, in particular, question diculty. We report on our experience on generating questions from ontologies and present promising results of evaluating our diculty-control theory.

Natural Language Generation is the task of gener- ating natural language text suitable for human consumption from machine representation of facts which can be pre-structured in some linguistically amenable fashion, or completely unstructured. An ontology is a formal explicit description of concepts in a domain of discourse. An ontology is considered as a formal knowledge repository which can be used as a resource for NLG tasks. A domain ontology will provides the input for content determination and micro-planing of NLG task. A linguistic ontology can be used for lexical realization. The logically structured manner of knowledge organization within an ontology enables to perform reasoning tasks like Con- sistency checking, Concept Satisfiability, Concept Subsumption and Instance Checking. These types of logical inferencing actions will be applied to derive descriptive texts as answers to user queries from ontologies. Thus a simple natural language based Question Answering system can be implement, guided by robust NLG techniques that act upon ontologies. Some of the tools for constructing ontologies ( Protege, Natural OWL, etc.) and their combination with NLG process will also be discussed.

Lecture Notes in Computer Science, 2017

In order to provide support for the construction of MCQs, there have been recent efforts to generate MCQs with controlled difficulty from OWL ontologies. Preliminary evaluation suggests that automatically generated questions are not field ready yet and highlight the need for further evaluations. In this study, we have presented an extensive evaluation of automatically generated MCQs. We found that even questions that adhere to guidelines are subject to the clustering of distractors. Hence, the clustering of distractors must be realised as this could affect the prediction of difficulty.

People have an individual understanding of the surrounding environment. It is the result of perception and past experiences that are unique to every person. Individual understanding can be seen as a compound of mental models that can be externalized. When acquired, these mental models can be formalized and used by machines. This procedure is already systematized in the field of ontology engineering. Harvesting knowledge from people and formalizing it is still a human-intensive task. It addresses the knowledge engineering problem of transition from elicitation to conceptualization, which requires the involvement of experts. There are many knowledge elicitation techniques (e.g., interviews, item sorting, laddering, etc.) used in ontology development in various combinations and sequences. However, the lack of a common formal model to precisely describe knowledge elicitation techniques prevents these techniques to be combined and being used in one system. There are many methodologies proposed for ontology development to be applied by humans. However, there are no methodologies for ontology development, where the machine conducts the ontology acquisition process. This master thesis especially addresses the question of how a machine should interact with a human in order to formalize his thoughts in form of a domain ontology. First, fields and studies relevant for solving the problem of automated ontology elicitation are presented: ontology engineering, knowledge engineering, dialogue management and theories on the nature of question. Second, I describe an implementable ontology elicitation method and its underlying knowledge structures. This method combines ontology development methodologies and knowledge elicitation techniques. To handle the transition from elicitation to conceptualization I propose the QAF triad. The QAF triad is presented from three perspectives: (1) as a conceptual structure that is binding the components of a dialogue-based interview – question, answer, and fact (i.e., as the effect of answering the question); (2) as interaction model between human and machine; (3) as a mechanism for “understanding”, i.e., the semantic mapping from natural to formal language. I also propose methods to construct meaningful and precise questions for the elicitation interview. Finally, a feasibility study of the proposed ontology elicitation method and the related improvement observations are presented. This master thesis constitutes a proof of concept that it is possible to create means for machines to elicit ontologies from lambda-users. The case-study described in the last part of the thesis shows that the proposed interview protocol is suitable to elicit domain ontologies from lambda-users and that it can be implemented into a dialogue management system. There are still open questions left for the future. Exploring those questions will definitely enrich my current model, and which I intend to treat them during subsequent PhD research.

Smart Learning Environments, 2017

An ontology is a knowledge representation structure which has been used in Virtual Learning Environments (VLEs) to describe educational courses by capturing the concepts and the relationships between them. Several ontology-based question generators used ontologies to auto-generate questions, which aimed to assess students' at different levels in Bloom's taxonomy. However, the evaluation of the questions was confined to measuring the qualitative satisfaction of domain experts and students. None of the question generators tested the questions on students and analysed the quality of the auto-generated questions by examining the question's difficulty, and the question's ability to discriminate between high ability and low ability students. The lack of quantitative analysis resulted in having no evidence on the quality of questions, and how the quality is affected by the ontology-based generation strategies, and the level of question in Bloom's taxonomy (determined by the question's stem templates). This paper presents an experiment carried out to address the drawbacks mentioned above by achieving two objectives. First, it assesses the auto-generated questions' difficulty, discrimination, and reliability using two statistical methods: Classical Test Theory (CTT) and Item Response Theory (IRT). Second, it studies the effect of the ontology-based generation strategies and the level of the questions in Bloom's taxonomy on the quality of the questions. This will provide guidance for developers and researchers working in the field of ontology-based question generators, and help building a prediction model using machine learning techniques.