Automatic Item Generation

Background knowledge is typically needed for successful comprehension of topical and domain specific reading passages, such as in the STEM domain. However, there are few automated measures of student knowledge that can be readily deployed and scored in time to make predictions on whether a given student will likely be able to understand a specific content area text. In this paper, we present our effort in developing K-tool, an automated system for generating topical vocabulary tests that measure students' background knowledge related to a specific text. The system automatically detects the topic of a given text and produces topical vocabulary items based on their relationship with the topic. This information is used to automatically generate background knowledge forms that contain words that are highly related to the topic and words that share similar features but do not share high associations to the topic. Prior research indicates that performance on such tasks can help determine whether a student is likely to understand a particular text based on their knowledge state. The described system is intended for use with middle and high school student population of native speakers of English. It is designed to handle single reading passages and is not dependent on any corpus or text collection. In this paper, we describe the system architecture and present an initial evaluation of the system outputs.

Purpose: To compare the structural quality of multiple choice questions (MCQs) generated by a large language model, a type of artificial intelligence (AI), GPT-4, against humanauthored items at both novice and expert level. Methods: We conducted a blinded analysis of 124 MCQs: 40 generated by GPT-4, 39 from human item-writers at Novice level, and 45 from human item-writers at Expert level. A generic prompt for GPT-4 was engineered, which included item-writing guidance, example MCQs, and key learning points. A standardized scoring system was developed including content validity, scope, item anatomy, cognitive skill level, item-writing flaws, feedback comprehensiveness, veracity and adequacy of clinical reasoning, and global impression of fitness for use. A consensus panel objectively evaluated each item, blinded to the author, using the scoring system. Results: Analysis showed that all groups (Novice, Expert, and AI) were able to generate items within scope. Expert items performed better than Novice items in all categories. There was no difference in the global impressions of Expert and AI items, which suggests overall comparability. A statistically significant, albeit small, difference was observed with Expert items performing better than AI items in the specific domains of content validity, feedback veracity and clinical reasoning, and testing at higher order cognitive skill levels. However, both groups met acceptable standards in these domains. AI items had a higher rate than Expert items of being deemed unfit for use requiring major revision, indicating erroneous correct answers, and displaying biased answer positioning. Conclusions: GPT-4 can produce MCQs testing clinically complex concepts for medical assessment. While the structural quality of AI-generated MCQs is comparable to experts overall, human oversight is necessary to ensure content validity and optimize item quality.

The use of automatic item generation (AIG) methods offers potential for assessing clinical reasoning (CR) skills in medical education, a critical skill combining intuitive and analytical thinking. In preclinical education, these skills are commonly evaluated through written exams and case-based multiple-choice questions (MCQs), which are widely used due to the high number of students, ease of standardization, and quick evaluation. This research generated CRfocused questions for medical exams using two primary AIG methods: template-based and nontemplate-based (using AI tools like ChatGPT for a flexible approach). A total of 18 questions were produced on ordering radiologic investigations for abdominal emergencies, alongside facultydeveloped questions used in medical exams for comparison. Experienced radiologists evaluated the questions based on clarity, clinical relevance, and effectiveness in measuring CR skills. Results showed that ChatGPT-generated questions measured CR skills with an 84.52% success rate, faculty-developed questions with 82.14%, and template-based questions with 78.57%, indicating that both AIG methods are effective in CR assessment, with ChatGPT performing slightly better. Both AIG methods received high ratings for clarity and clinical suitability, showing promise in producing effective CR-assessing questions comparable to, and in some cases surpassing, facultydeveloped questions. While template-based AIG is effective, it requires more time and effort, suggesting that both methods may offer time-saving potential in exam preparation for educators.

We aimed to determine the quality of AI-generated (ChatGPT-4 and Claude 3) Script Concordance Test (SCT) items through an expert panel. We generated SCT items on abdominal radiology using a complex prompt in large language model (LLM) chatbots (ChatGPT-4 and Claude 3 (Sonnet) in April 2024) and evaluated the items' quality through an expert panel of 16 radiologists. Expert panel, which was blind to the origin of the items provided without modifications, independently answered each item and assessed them using 12 quality indicators. Data analysis included descriptive statistics, bar charts to compare responses against accepted forms, and a heatmap to show performance in terms of the quality indicators. SCT items generated by chatbots assess clinical reasoning rather than only factual recall (ChatGPT: 92.50%, Claude: 85.00%). The heatmap indicated that the items were generally acceptable, with most responses favorable across quality indicators (ChatGPT: 71.77%, Claude: 64.23%). The comparison of the bar charts with acceptable and unacceptable forms revealed that 73.33% and 53.33% of the questions in the items can be considered acceptable, respectively, for ChatGPT and Claude. The use of LLMs to generate SCT items can be helpful for medical educators by reducing the required time and effort. Although the prompt provides a good starting point, it remains crucial to review and revise AI-generated SCT items before educational use. The prompt and the custom GPT, "Script Concordance Test Generator", available at https://chatgpt.com/g/g-RlzW5xdc1-script-concordance-test-generator, can streamline SCT item development.

Objective: Artificial Intelligence (AI) offers opportunities for radiologists to enhance workflow efficiency, perform faster and repeatable segmentation, and detect lesions more easily. The aim of this study is to investigate the current knowledge and general attitudes of radiology resident physicians towards AI. Additionally, it seeks to assess the current state of AI/ML/DL education in radiology residency, the awareness and use of available educational resources. Methods: A cross-sectional study was conducted using an online survey from October 2023 to February 2024. The survey included demographic data, AI knowledge, attitudes towards AI, and the role of AI in medical education. Survey questions were developed based on literature and reviewed by experts in medical education and radiology. Results: The study included 155 participants (38.7% female) with an average age of 28.81±4.77 years. About 80.6% were aware of AI terms, with a mean knowledge score of 3.02±1.39 on a 7-point Likert scale. Most participants (90.3%) had no programming knowledge. Only 22.6% used AI tools occasionally. The majority (73.4%) believed AI would change radiology's future, though only 10.3% felt radiologists' jobs were at risk. Regarding AI education, 84.5% reported no formal training, and awareness of online resources was low. Conclusion: The study found that while awareness of AI among radiology residents is high, their knowledge and practical use of AI tools are limited. AI education is largely absent from residency programs, and awareness of online educational resources is low. These findings highlight the need for integrating AI training into radiology education and increasing awareness of available resources.

Objectives: The aim of this study is to compare the ability of artificial intelligencebased chatbots, ChatGPT-4o and Claude 3.5, to interpret mammography images. The study focuses on evaluating their accuracy and consistency in BI-RADS classification and breast parenchymal type assessment. It also aims to explore the potential of these technologies to reduce radiologists' workload and identify their limitations in medical image analysis. Methods: A total of 53 mammography images obtained between January and July 2024 were analyzed, focusing on BI-RADS classification and breast parenchymal type assessment. The same anonymized mammography images were provided to both chatbots under identical prompts. Results: The results showed accuracy rates for BI-RADS classification ranging from 18.87% to 26.42% for ChatGPT-4o and 18.7% for Claude 3.5. When BI-RADS categories were grouped into benign group(BI-RADS 1,2) and malignant group(BI-RADS 4,5), the combined accuracy was 57.5% for ChatGPT-4o (initial evaluation) and 55% (second evaluation), compared to 47.5% for Claude 3.5. Breast parenchymal type accuracy rates were 30.19% and 22.64% for ChatGPT-4o, and 26.42% for Claude 3.5. Conclusions: The findings indicate that chatbots demonstrate limited accuracy and reliability in interpreting mammography images. These results highlight the need for further optimization, larger datasets, and advanced training processes to improve their performance in medical image analysis.

Medical education always evolves to incorporate more tools for specific needs in assessing clinical reasoning skills. Among these tools, Script Concordance Test (SCT) has a particular importance due to its focus on assessing decision-making in uncertain clinical situations. However, development of SCT items is effortful. Artificial intelligence tools, such as large language models, offer significant benefits. These models are already used for generating multiple-choice questions, and their use in generating SCTs offers great promise. However, this requires well-designed prompts to generate SCTs. This article proposes a generic prompt for the ChatGPT-4, Claude 3, Llama 3, and ChatGPT-4o large language model chatbots to generate SCTs, which can be tailored to various fields of medicine and different stages of medical education. It can help to streamline the development process of SCTs. Initial findings are promising, and there is a need for generating SCTs using large language models and conducting research to assess the quality of SCTs.

Virtual Learning Environments provide teachers with a web-based platform to create different types of feedback which vary in the level of details given in the feedback content. Types of feedback can range from a simple correct or vice-versa to a detailed explanation about the reason why the correct answer is correct and the incorrect answer is incorrect. However, these environments usually follow the 'one size fits all' approach and provide all students with the same type of feedback regardless of students' individual characteristics and the assessment question's individual characteristics. This approach is likely to negatively affect students' performance and learning gain. Several personalised feedback frameworks have been proposed which adapt the different types of feedback based on the student characteristics and/or the assessment question characteristics. The frameworks have three drawbacks: firstly, creating the different types of feedback is a time consuming process, as the types of feedback are either hard-coded or auto-generated from a restricted set of solutions created by the teacher or a domain expert; secondly, they are domain dependent and cannot be used to auto-generate feedback across different educational domains; thirdly, they have not attempted any integration which takes into consideration both the characteristics of the assessment questions and the student's characteristics. This thesis contributes to research carried out on personalised feedback frameworks by proposing a generic novel system which is called the Ontology-based Personalised Feedback Generator (OntoPeFeGe). OntoPeFeGe has three aims: firstly, it uses any pre-existing domain ontology which is a knowledge representation of the educational domain to auto-generate assessment questions with different characteristics, in particular, questions aimed to assess students at different levels in Bloom's taxonomy 1 ; secondly, it associates each auto-generated question with specialised domain independent types of feedback; thirdly, it provides students with personalised feedback which adapts the types of feedback based on the student and the assessment question characteristics. OntoPeFeGe allowed the integration of student's characteristics, the assessment question's characteristics, and the personalised feedback, for the first time. The 1 Bloom's taxonomy categorises the assessment questions into the following six major levels, which are arranged in a hierarchical order according to the complexity of the cognitive process involved: knowledge, comprehension, application, analysis, synthesis, and evaluation. xii Declaration No portion of the work referred to in this thesis has been submitted in support of an application for another degree or qualification of this or any other university or other institute of learning. xiv Copyright i. The author of this thesis (including any appendices and/or schedules to this thesis) owns certain copyright or related rights in it (the "Copyright") and s/he has given Birmingham City University certain rights to use such Copyright, including for administrative purposes. ii. Copies of this thesis, either in full or in extracts and whether in hard or electronic copy, may be made only in accordance with the Copyright, Designs and Patents Act 1988 (as amended) and regulations issued under it or, where appropriate, in accordance with licensing agreements which the University has from time to time. This page must form part of any such copies made. iii. The ownership of certain Copyright, patents, designs, trade marks and other intellectual property (the "Intellectual Property") and any reproductions of copyright works in the thesis, for example graphs and tables ("Reproductions"), which may be described in this thesis, may not be owned by the author and may be owned by third parties. Such Intellectual Property and Reproductions cannot and must not be made available for use without the prior written permission of the owner(s) of the relevant Intellectual Property and/or Reproductions. xv I am grateful to all the people who helped and inspired me during the preparation of this thesis, especially my supervisor Prof. Mohamed Medhat Gaber, who gave up his valuable time to help me extend the research work, analyse the experimental results, read my thesis draft, and offered comments that had led to many improvements. Thanks for Birmingham City University staff for providing me with help, support and facilities to carry out my research. I would also like to thank Dr. Nick Filer who supervised me at the beginning of my PhD until his plans for retirement, and the University of Manchester staff who facilitated the experimental studies. Words fail me to express my appreciation to my family who provided me with their support, understanding and endless love, through my PhD years. Father Nabil and mother Amneh you are my role model and have always inspired me to become the person I am today. Thank you for believing in me and giving me the chance to accomplish my dreams. Sisters Lana and Hala, brothers Iyad and Murad, thank you for being always there for me and standing by my side. I am lucky to have an amazing family like you. Finally, I would like to thank my dear friends. xvi Educational Domain Corpus Document Name The Data Networking and the Computer Networks courses One textbook Computer networking: a top-down approach (Kurose and Ross, 2013) Twenty-seven lecture slides

The aim of this paper is to present an innovative approach for generating multiple choice questions in automatic way. Although other approaches have been already reported in the literature, the approach presented in this paper is based on domain specific ontologies and it is independent of lexicons such as WordNet or other linguistic resources. The paper also reports on a first prototype implementation of such an approach which creates multiple choice question items using the Semantic Web standard technology OWL (Ontology Web Language). The proposed approach is independent of the domain since questions are generated according to specific ontology-based strategies. In current implementation, simple natural language generation techniques are used in order to provide the items in the questionnaires.

In introductory programming courses, proficiency is typically achieved through substantial practice in the form of relatively small assignments and quizzes. Unfortunately, creating programming assignments and quizzes is both, time consuming and error prone. Furthermore, grading the assignments and providing timely and detailed feedback is paramount to student improvement. We use Automatic Item Generation (AIG) in order to address the problem of creating numerous programming exercises that can be used for assignments or quizzes in introductory programming courses. AIG is based on the use of test-item templates with embedded variables and formulas. The variables and formulas in the template are resolved by a computer program with actual values to generate test-items. Thus, hundreds or even thousands of test-items can be generated with a single test-item template. We discuss a semantic-based AIG approach for automatically generating programming exercises. The approach was incorporated into an existing selfassessment and practice tool for students learning computer programming. The tool has been used in different introductory programming courses to generate a set of practice exercises different for each student, but with the same difficulty and quality.

Multiple Choice Questions (MCQs) are a popular assessment method because they enable automated evaluation, flexible administration and use with huge groups. Despite these benefits, the manual construction of MCQs is challenging, time-consuming and error-prone. This is because each MCQ is comprised of a question called the "stem", a correct option called the "key" along with alternative options called "distractors" whose construction demands expertise from the MCQ developers. In addition, there are different kinds of MCQs such as Wh-type, Fill-in-the-blank, Odd one out, and many more needed to assess understanding at different cognitive levels. Automatic Question Generation (AQG) for developing heterogeneous MCQ stems has generally followed two approaches: semantics-based and machine-learning-based. Questions generated via AQG techniques can be utilized only if they are grammatically correct. Semantics-based techniques have been able to generate a range ...

Multiple Choice Questions (MCQs) are a popular assessment method because they enable automated evaluation, flexible administration and use with huge groups. Despite these benefits, the manual construction of MCQs is challenging, time-consuming and error-prone. This is because each MCQ is comprised of a question called the "stem", a correct option called the "key" along with alternative options called "distractors" whose construction demands expertise from the MCQ developers. In addition, there are different kinds of MCQs such as Wh-type, Fill-in-the-blank, Odd one out, and many more needed to assess understanding at different cognitive levels. Automatic Question Generation (AQG) for developing heterogeneous MCQ stems has generally followed two approaches: semantics-based and machine-learning-based. Questions generated via AQG techniques can be utilized only if they are grammatically correct. Semantics-based techniques have been able to generate a range ...

As education gets increasingly digitized, and intelligent tutoring systems gain commercial prominence, scalable assessment generation mechanisms become a critical requirement for enabling increased learning outcomes. Assessments provide a way to measure learners’ level of understanding and difficulty, and personalize their learning. There have been separate efforts in different areas to solve this by looking at different parts of the problem. This paper is a first effort to bring together techniques from diverse areas such as knowledge representation and reasoning, machine learning, inference on graphs, and pedagogy to generate automated assessments at scale. In this paper, we specifically address the problem of Multiple Choice Question (MCQ) generation for vocabulary learning assessments, specially catered to young learners (YL). We evaluate the efficacy of our approach by asking human annotators to annotate the questions generated by the system based on relevance. We also compare ...

This paper presents a new conceptual framework and corresponding psychometric model designed for the pre-calibration of automatically generated items. This model utilizes a multi-level framework and a combination of crossed fixed and random effects to capture key components of the generative process, and is intended to be broadly applicable across research efforts and contexts. Unique among models proposed within the AIG literature, this model incorporates specific mean and variance parameters to support the direct assessment of the quality of the item generation process. The utility of this framework is demonstrated through an empirical analysis of response data collected from the online administration of automatically generated items intended to assess young students' mathematics fluency. Limitations in the application of the proposed framework are explored through targeted simulation studies, and future directions for research are discussed.

A new method for generating the stem components of Multiple Alternate Choice (MAC) test items is proposed which makes use of a theory about causal coherence relation primitives. A trial of the Causal Relation Explication, Addition and Manipulation (CREAM) ...

An ontology is a knowledge representation structure which has been used in Virtual Learning Environments (VLEs) to describe educational courses by capturing the concepts and the relationships between them. Several ontology-based question generators used ontologies to auto-generate questions, which aimed to assess students' at different levels in Bloom's taxonomy. However, the evaluation of the questions was confined to measuring the qualitative satisfaction of domain experts and students. None of the question generators tested the questions on students and analysed the quality of the auto-generated questions by examining the question's difficulty, and the question's ability to discriminate between high ability and low ability students. The lack of quantitative analysis resulted in having no evidence on the quality of questions, and how the quality is affected by the ontology-based generation strategies, and the level of question in Bloom's taxonomy (determined by the question's stem templates). This paper presents an experiment carried out to address the drawbacks mentioned above by achieving two objectives. First, it assesses the auto-generated questions' difficulty, discrimination, and reliability using two statistical methods: Classical Test Theory (CTT) and Item Response Theory (IRT). Second, it studies the effect of the ontology-based generation strategies and the level of the questions in Bloom's taxonomy on the quality of the questions. This will provide guidance for developers and researchers working in the field of ontology-based question generators, and help building a prediction model using machine learning techniques.

In order to provide support for the construction of MCQs, there have been recent efforts to generate MCQs with controlled difficulty from OWL ontologies. Preliminary evaluation suggests that automatically generated questions are not field ready yet and highlight the need for further evaluations. In this study, we have presented an extensive evaluation of automatically generated MCQs. We found that even questions that adhere to guidelines are subject to the clustering of distractors. Hence, the clustering of distractors must be realised as this could affect the prediction of difficulty.

Designing good multiple choice questions (MCQs) for education and assessment is time consuming and error-prone. An abundance of structured and semi-structured data has led to the development of automatic MCQ generation methods. Recently, ontologies have emerged as powerful tools to enable the automatic generation of MCQs. However, current question generation approaches focus on knowledge recall questions. In addition, questions that have so far been generated are, compared to manually created ones, simple and cover only a small subset of the required question complexity space in the education and assessment domain. In this paper, we focus on addressing the limitations of previous approaches by generating questions with complex stems that are suitable for scenarios beyond mere knowledge recall. We present a novel ontology-based approach that exploits classes and existential restrictions to generate case-based questions. Our contribution lies in: (1) the specification of procedure for generating case-based questions which involve (a) assembling complex stems, (b) selecting suitable options, and (c) providing explanations for option correctness/incorrectness, (2) an implementation of the procedure using a medical ontology and (3) and evaluation of our generation technique to test question quality and their suitability in practise. We implement our approach as an application for a medical education scenario on top of a large knowledge base in the medical domain. We generate more than 3 million questions for four physician specialities and evaluate our approach in a user study with 15 medical experts. We find that using a stratified random sample of 435 questions out of which 316 were rated by two experts, 129 (30%) are considered appropriate to be used in exams by both experts and a further 216 (50%) by at least one expert.

In order to provide support for the construction of MCQs, there have been recent efforts to generate MCQs with controlled difficulty from OWL ontologies. Preliminary evaluation suggests that automatically generated questions are not field ready yet and highlight the need for further evaluations. In this study, we have presented an extensive evaluation of automatically generated MCQs. We found that even questions that adhere to guidelines are subject to the clustering of distractors. Hence, the clustering of distractors must be realised as this could affect the prediction of difficulty.

The experiment described in this paper investigates the effectiveness of a format of Multiple Choice test item that is new to the featured domain. The response data for candidates answering Multiple Alternative Choice (MAC) test items are compared to response data for 4-option Multiple Choice test items covering the same content. The results show how analysis of responses to the test items in the MAC test item format have identified candidate knowledge gaps more precisely than response analysis for the items in the traditional 4-option Multiple Choice format.

With recent advancements in Semantic Web technologies, a new trend in MCQ item generation has emerged through the use of ontologies. Ontologies are knowledge representation structures that formally describe entities in a domain and their relationships, thus enabling automated inference and reasoning. Ontology-based MCQ item generation is still in its infancy, but substantial research efforts are being made in the field. However, the applicability of these models for use in an educational setting has not been thoroughly evaluated. In this paper, we present an experimental evaluation of an ontology-based MCQ item generation system known as OntoQue. The evaluation was conducted using two different domain ontologies. The findings of this study show that ontology-based MCQ generation systems produce satisfactory MCQ items to a certain extent. However, the evaluation also revealed a number of shortcomings with current ontology-based MCQ item generation systems with regard to the education...

Concept inventories (CI) constitute a key thread in Physics Educational Research. As such, understanding the methodology and the technique of developing a good CI is essential for all physics teachers. This research aims to develop a circular-motion concept Inventory (CMCI) that is valid in the Ugandan context. To reach a consensus, we used the Delphi technique to collect the data from eleven experts in the physics discipline. These experts were asked to rank each CI item in the inventory, based on the relevant criteria, for assigning a degree of relevance for adoption on a scale ranging from one to four, one being "not relevant" and four being "highly relevant.” Because the technique does not require experts to meet face-to-face, they remained anonymous to one another. These experts are provided with structured questionnaires of CI items from the Rotational-Kinematics Inventory (RKI) and Rolling and Rotational Motion-Concept (RRMC) inventories in the first round, in ...

Successful exams require a balance of easy, medium, and difficult questions. Question difficulty is generally either estimated by an expert or determined after an exam is taken. The latter provides no utility for the generation of new questions and the former is expensive both in terms of time and cost. Additionally, it is not known whether expert prediction is indeed a good proxy for estimating question difficulty. In this paper, we analyse and compare two ontology-based measures for difficulty prediction of multiple choice questions, as well as comparing each measure with expert prediction (by 15 experts) against the exam performance of 12 residents over a corpus of 231 medical case-based questions that are in multiple choice format. We find one ontology-based measure (relation strength indicativeness) to be of comparable performance (accuracy = 47%) to expert prediction (average accuracy = 49%).

With the growing interest in using ontologies in semanticallyenabled applications, the interest in enhancing the quality of such ontologies has grown as well. Standard reasoning services focus on certain obvious dimensions of quality, e.g., to detect inconsistencies and incoherence. In addition, bespoke tools have been presented to address the completeness dimension of quality, e.g., missing entailments. These tools are usually focused on very restricted subsets of all the possible missing entailments, i.e., only atomic subsumptions. We present a new protocol to detect both existing invalid entailments and missing valid entailments. We also present a case study to evaluate the usefulness of the presented protocol for ontology validation purposes.

Ontology-based Multiple Choice Question (MCQ) generation has a relatively short history. Many attempts have been carried out to develop methods to generate MCQs from ontologies. However, there is a still a need to understand the applicability of these methods in real educational settings. In this paper, we present an empirical evaluation of ontology-based MCQ generation. We examine the feasibility of applying ontology-based MCQ generation methods by educators with no prior experience in ontology building. The findings of this study show that this is feasible and can result in generating a reasonable number of educationally useful questions with good predictions about their difficulty levels.

Different computational models for generating analogies of the form ''A is to B as C is to D'' have been proposed over the past 35 years. However, analogy generation is a challenging problem that requires further research. In this article, we present a new approach for generating analogies in Multiple Choice Question (MCQ) format that can be used for students' assessment. We propose to use existing high-quality ontologies as a source for mining analogies to avoid the classic problem of hand-coding concepts in previous methods. We also describe the characteristics of a good analogy question and report on experiments carried out to evaluate the new approach.

Abstract. In this paper, we propose a new approach to generate analogy questions of the form” A is to B as... is to?” from ontologies. Analogy questions are widely used in multiple-choice tests such as SATs and GREs and are used to assess student's higher cognitive abilities. The design, implementation and evaluation of the new approach are presented in this paper. The results show that mining ontologies for such questions is fruitful.

Assessment plays an important role in learning and Multiple Choice Questions (MCQs) are quite popular in large-scale evaluations. Technology-enabled learning necessitates a smart assessment. Therefore, automatic MCQ generation became increasingly popular in the last two decades. Despite a large amount of research effort, system generated MCQs are not useful in real educational applications. This is because of the inability to produce diverse and human-alike distractors. Distractors are the wrong choices given along with the correct answer (key) to befuddle the examinee. In several domains, the MCQs deal with names or named entities. However, existing literature is not adequate in generating quality named entity distractors. In this paper, we present a method for automatic generation of named entity distractors. The technique uses a combination of statistical and semantic similarities. To compute the statistical similarity, a set of class-specific attributes are defined are their values are extracted from the web. Semantic similarity is computed using a predicate-argument extraction based method. The proposed technique is tested in cricket domain because of the availability of a large number of web resources and MCQs for dataset preparation. An evaluation strategy is proposed along with a set of metrics. A set of human evaluators performed the evaluation and they found that the average closeness value of the distractors as 2.3. This value indicates that 2.3 out of 3 system-generated distractors are as good as human-generated distractors. Two good distractors make an MCQ usable in real assessment. So, the proposed technique is capable of generating high-quality distractors.

Automatic Item Generation (AIG) is the process of designing and producing items for a test, as well as generating different versions of exams that are conceptually and statistically equivalent. Automatic Item Generation tools are developed with the assistance of information systems, which make these tools very efficient. Under this aim, GenerEx, an automatic item generation tool, was developed. GenerEx is used to automatically generate different versions of the Basic Competences Exam (Excoba). Even though AIG represents a great advance for the development of psychological and educational assessment, it is a methodological challenge to obtain evidence of validity of the enormous quantity of possible items and tests generated in an automatic process. This paper has the purpose of describing an approach to analyze the internal structure and the psychometric equivalence of exams generated by GenerEx and, additionally, to describe kinds of results obtained to reach this objective. The approach is based on the process for selecting samples from the generation tool, founded on the assumption that items and exams must be psychometrically equivalent. This work includes three kinds of conceptually different and complementary analysis: the Classical Test Theory, Item Response Theory and Confirmatory Factor Analysis. Results show that GenerEx produces psychometrically similar exams; however there are problems in some learning areas. The methodology was useful for obtaining a description about GenerEx's psychometric functioning Reception Date

We have developed new software named GaMeTix for management of large collections of examination questions written in a variety of MCQ (Multiple Choice Question) formats. This application provides a wide scale of functionality modes like collecting and editing sets of questions, generating electronic versions of examination tests, printing examination paper sheets and exporting sets of questions in a plain text document for hard copy archiving or transfer to specific electronic testing applications. The content of the database is searchable according to several criteria using sets of filters that characterize each question. Collections of MC questions can be divided or merged together according to results of the filtering function. Examination questions can be complemented with pictures or diagrams in .jpg format. GaMeTix is a portable, freeware application that runs on MS Windows operating systems.

Advancements in Semantic Web techniques have led to the emergence of ontology based question generation. Ontologies are used to represent domain knowledge in the form of concepts, instances and their relationships as their elements. Many research strategies for generating questions using ontologies have been proposed but little work has been done on investigating whether an ontology is an appropriate source of data for question generation. Since there is no standard guideline for developing an ontology, the representation of ontology elements might vary in many ways, and this paper aims to investigate how the naming of ontology elements would affect the questions generated. In order to achieve this aim, two research questions will be investigated which are: how many correct questions can be generated from an ontology, and what are the reasons for incorrect questions being generated. Categorized question templates and a set of question strategies for mapping templates with a concept ...

As education gets increasingly digitized, and intelligent tutoring systems gain commercial prominence, scalable assessment generation mechanisms become a critical requirement for enabling increased learning outcomes. Assessments provide a way to measure learners’ level of understanding and difficulty, and personalize their learning. There have been separate efforts in different areas to solve this by looking at different parts of the problem. This paper is a first effort to bring together techniques from diverse areas such as knowledge representation and reasoning, machine learning, inference on graphs, and pedagogy to generate automated assessments at scale. In this paper, we specifically address the problem of Multiple Choice Question (MCQ) generation for vocabulary learning assessments, specially catered to young learners (YL). We evaluate the efficacy of our approach by asking human annotators to annotate the questions generated by the system based on relevance. We also compare ...

In this paper, we present a methodology to generate diagrammatical multiple choice questions from a knowledge base. When considering a knowledge base, the main strategies discussed in the literature are class-based, property-based and terminology-based, and the generated multiple choice questions are typical with all the choices (correct answer and distractors) being of atomic types such as plain text. In this paper, we introduce graph-based strategies, enabling the generation of choices (correct answer and distractors) in the form of complex structures such as diagrams, and discuss different approaches to take into account difficulty levels when generating the questions.

As education gets increasingly digitized, and intelligent tutoring systems gain commercial prominence, scalable assessment generation mechanisms become a critical requirement for enabling increased learning outcomes. Assessments provide a way to measure learners’ level of understanding and difficulty, and personalize their learning. There have been separate efforts in different areas to solve this by looking at different parts of the problem. This paper is a first effort to bring together techniques from diverse areas such as knowledge representation and reasoning, machine learning, inference on graphs, and pedagogy to generate automated assessments at scale. In this paper, we specifically address the problem of Multiple Choice Question (MCQ) generation for vocabulary learning assessments, specially catered to young learners (YL). We evaluate the efficacy of our approach by asking human annotators to annotate the questions generated by the system based on relevance. We also compare ...

The purpose of this paper is to analyse if Multiple-Choice Tests may be considered an interesting alternative for assessing knowledge, particularly in the Mathematics area, as opposed to the traditional methods, such as open questions exams. In this sense we illustrate some opinions of the researchers in this area. Often the perception of the people about the construction of this kind of exams is that they are easy to create. But it is not true! Construct well written tests it’s a hard work and needs writing ability from the teachers. Our proposal is analyse the construction difficulties of multiple choice tests as well some advantages and limitations of this type of tests. We also show the frequent critics and worries, since the beginning of this objective format usage. Finally in this context some examples of Multiple-Choice Items in the Mathematics area are given, and we illustrate as how we can take advantage and improve this kind of tests.

Images are an essential tool for communicating with children, particularly at younger ages when they are still developing their emergent literacy skills. Hence, assessments that use images to assess their conceptual knowledge and visual literacy, are an important component of their learning process. Creating assessments at scale is a challenging task, which has led to several techniques being proposed for automatic generation of textual assessments. However, none of them focuses on generating image-based assessments. To understand the manual process of creating visual assessments, we interviewed primary school teachers. Based on the findings from the preliminary study, we present a novel approach which uses image semantics to generate visual multiple choice questions (VMCQs) for young learners, wherein options are presented in the form of images. We propose a metric to measure the semantic similarity between two images, which we use to identify the four options – one answer and thre...

In this paper, we present a methodology to generate diagrammatical multiple choice questions from a knowledge base. When considering a knowledge base, the main strategies discussed in the literature are class-based, property-based and terminology-based, and the generated multiple choice questions are typical with all the choices (correct answer and distractors) being of atomic types such as plain text. In this paper, we introduce graph-based strategies, enabling the generation of choices (correct answer and distractors) in the form of complex structures such as diagrams, and discuss different approaches to take into account difficulty levels when generating the questions.

The aim of this paper is to present an innovative approach for generating multiple choice questions in automatic way. Although other approaches have been already reported in the literature, the approach presented in this paper is based on domain specific ontologies and it is independent of lexicons such as WordNet or other linguistic resources. The paper also reports on a first prototype implementation of such an approach which creates multiple choice question items using the Semantic Web standard technology OWL ...

Background: Multiple choice question (MCQ) generation from a text is a popular research area. They are widely used for a learner's knowledge assessment in the education sector. Manual generation of Mcq is hard, time-consuming and expensive. We have developed a system using various applications of Natural Language Processing Techniques. In this paper, we present a set of MCQs (Multiple Choice Questions) generated for any given input text along with a set of distractors and the correct answer to the question. In addition, The BERT(Bidirectional Encoder Representations from Transformers) model is used to generate a summary of the input text. The RAKE (Random Automatic Keyword Extraction) technique is used for extracting the keyword from the generated summary. This keyword is used as a parameter to generate questions as well as distractors for the MCQs.Distractors are the options that are generated apart from the correct answer. For the generation of MCQ, the wordnet and conceptnet algorithms are used. For the question generation, a T5 Transformer Model is trained using the encoder-decoder architecture. For the generation of MCQ, the wordnet and conceptnet algorithms are used. The final result has is a combination of questions with four options and a correct answer. Materials and Methods: In this study , The system will start with a prompt to provide an input file as a text document file .A description box will be available which will give information about the working of the software. Once the user enters the texts file , they will immediately get a list of multiple choice questions along with correct answer. The text file will be converted into a summary using the summarizer. The summary will be a single paragraph. The keywords will be extracted from the summary using RAKE. The Raw Automatic Keyword Extractor will generate mcq using the graph technique of natural language processing. Once the keyword is extracted the summary and the keyword will be provided as an input to the question Generation model. The question generation model will provide a set of questions on the basis of keywords and the summary provided. The keyword that is associated with the question will be used for distractor generation.The distractor generator will be done using conceptnet and wordnet. The final result will be a list of mcqs with correct answer under each question. Results: The final result will be a set of Multiple choice questions with a set of distractors and a correct answer to the question. Conclusion: The basic motivation of creating this desktop application were examinations attempted during Covid-19 pandemic which were inefficient. This system tends to overcome many flaws that the previous application sustained. The system summarizes the text and generates MCQs with distractors and right answer with probably higher accuracy.

Introduction: Multiple choice questions (MCQs) are frequently used to assess in depth Knowledge of
students in various subjects. In fact, a rightly frame MCQ with acceptable quality, measured by Difficulty
Index (DIFI), Discrimination Index (DI) and Distractor Efficiency (DE) is the tool for assessment of
student’s performance.
Objective: To evaluate MCQs or items with their DIFI, DI and DE & to make database of valid items for
future examination that increases creditability of Assessment.
Materials and Methods: MCQ examination was conducted as a part of Day to Day Assessment for 1st
year MBBS students in Biochemistry Department at the end of 2nd term. 05 Set of 30 MCQs each had
been randomly distributed among 126 students and each MCQ / Item was analyzed. Data was entered and
analyzed in MS Excel 2007 for statistical analysis. Mean, Standard Deviations were calculated of DIFI,
DE & DI for each items.
Results: Out of 30 items, 11 had “good to excellent” DIF I (31 - 60%) and 1 6 had good to excellent ” DI
(> 0.25). Mean DE was 64.4% considered as nearer to acceptable. Out of all 05 groups & 90 D istractors
average 30-32 were Non Functional Distractors. Negative DI indicates poor preparedness of students and
issues with framing of some of the MCQs.
Conclusion: Study emphasizes, each assessment test of MCQs can be improved by developing improved
items, maintaining database & selection of quality MCQs which truly assess the knowledge of students
according to their abilities to differentiate in correct manner.

Introduction: Multiple Choice Questions (MCQs) is used extensively as a test format in nursing education. However, making MCQs still remains a challenge to educators. To avoid issues about its quality, this should undergo item analysis. Thus, the study evaluated item and test quality using difficulty index (DIF) and discrimination indices (DI), with distractor efficiency (DE); determined the reliability using Kuder-Richardson 20 coefficients (KR 20 ); and identified which valid measure was developed.

The paper presents a novel approach which aims to support the training of decision makers in maritime environmental pollution policies via an intelligent tutoring system that focuses on the automatic generation of self-assessment material. More specifically, the paper reports on the automatic construction of multiple-choice question tests from knowledge bases ie domain ontologies, instances, if-then rules, and multimedia objects. The reported work has been conducted with a prototype environmental pollution knowledge ...

With the growing interest in using ontologies in semantically-enabled applications, the interest in enhancing the quality of such on-tologies has grown as well. Standard reasoning services focus on certain obvious dimensions of quality, e.g., to detect inconsistencies and inco-herence. In addition, bespoke tools have been presented to address the completeness dimension of quality, e.g., missing entailments. These tools are usually focused on very restricted subsets of all the possible missing entailments, i.e., only atomic subsumptions. We present a new protocol to detect both existing invalid entailments and missing valid entailments. We also present a case study to evaluate the usefulness of the presented protocol for ontology validation purposes.

Multiple choice questions (MCQs) are considered highly useful (being easy to take or mark) but quite difficult to create and large numbers are needed to form valid exams and associated practice materials. The idea of re-using an existing ontology to generate MCQs almost suggests itself and has been explored in various projects. In this project, we are applying suitable educational theory regarding assessments and related methods for their evaluation to ontology-based MCQ generation. In particular, we investigate whether we can measure the similarity of the concepts in an ontology with sufficient reliability so that this measure can be used to control the difficulty of the MCQs generated. In this report, we provide an overview of the background to this research, and describe the main steps taken and insights gained.

Ontology-based Multiple Choice Question (MCQ) generation has a relatively short history. Many attempts have been carried out to develop methods to generate MCQs from ontologies. However, there is a still a need to understand the applicability of these methods in real educational settings. In this paper, we present an empirical evaluation of ontology-based MCQ generation. We examine the feasibility of applying ontology-based MCQ generation methods by educators with no prior experience in ontology building. The findings of this study show that this is feasible and can result in generating a reasonable number of educationally useful questions with good predictions about their difficulty levels.

—Several attempts have already been made to automate the generation of assessment questions. These attempts were mainly technical and lacked a theoretical backing. We explore psychological and educational theories to support the development of principled methods to generate questions and control their properties. We present a similarity-based theory to control the difficulty of multiple-choice questions and show its practicality and consistency with the psychological and educational theories.

Ontologies with potential educational value are available in di↵erent domains. However, it is still unclear how such ontologies can be exploited to generate useful instructional content, e.g., assessment questions. In this paper, we present an approach to automatically generate multiple-choice questions from OWL ontologies. We describe a psychologically-plausible theory to control the diculty of questions. Our contributions include designing a protocol to evaluate the characteristics of the generated questions, in particular, question diculty. We report on our experience on generating questions from ontologies and present promising results of evaluating our diculty-control theory.

In this paper, we propose a new approach to generate analogy questions of the form " A is to B as ... is to ? " from ontologies. Analogy questions are widely used in multiple-choice tests such as SATs and GREs and are used to assess student's higher cognitive abilities. The design, implementation and evaluation of the new approach are presented in this paper. The results show that mining ontologies for such questions is fruitful.

Different computational models for generating analogies of the form ''A is to B as C is to D'' have been proposed over the past 35 years. However, analogy generation is a challenging problem that requires further research. In this article, we present a new approach for generating analogies in Multiple Choice Question (MCQ) format that can be used for students' assessment. We propose to use existing high-quality ontologies as a source for mining analogies to avoid the classic problem of hand-coding concepts in previous methods. We also describe the characteristics of a good analogy question and report on experiments carried out to evaluate the new approach.