Optimal correction for guessing in multiple-choice tests

International Journal of Selection and Assessment, 2010

European Journal of Cognitive Psychology, 2007

To maximise multiple choice test scores under formula scoring, students must be able to monitor their knowledge level, reporting correct answers and withholding incorrect ones. Importantly though, neither metacognitive monitoring nor optimal criterion setting has received much attention in the formula-scoring literature. The present research examined the role that both these parameters have on obtained test score. Students wrote three exams under formula-scoring rules, and from these data, we created bias profiles using methodology developed by Higham , which allowed us to estimate the optimal number of questions that students needed to answer to maximise their scores. The results indicated that higher scoring students monitored their knowledge best, but that all students tended to guess too often, lowering their test scores. This conservatism maintained despite feedback from previous tests. The results also showed that monitoring is a stable participant skill that generalises across tests.

Under formula-scoring rules for multiple-choice exams, a penalty is applied to incorrect responses to reduce noise in the observed score. To avoid the penalty individuals are allowed to "pass," and therefore they must be able to strategically regulate the accuracy of their reporting by deciding which and how many questions to answer. To investigate the effect of bias within this framework, introduced bias profiles, which show the score obtained under formula scoring (corrected score) as a function of the omission rate. Bias profiles estimate the optimal number of questions that should be answered to maximize the corrected score (i.e., optimal bias). Our initial research showed that individuals tend to be too conservative when setting reporting criteria, "omitting" too many answers. The present three experiments introduced a feedback manipulation whereby participants were informed of the optimal omission rate after completing a test and asked to alter their reporting decisions accordingly. This feedback and concomitant alteration of reporting decisions led to improved corrected scores on true/false (Experiment 1), 2-alternative tests (Experiments 2), and 4-alternative tests (Experiment 3). Importantly, corrected scores at optimal bias also were higher than at forced-report for both true/false and 2-alternative tests. Furthermore, in Experiment 3, feedback based on one test improved scores on a second test, and participants were more likely to perform optimally on a third test without feedback. These effects suggest that optimal-bias feedback may have long-term effects and generalize to new tests.

2017

Introduction: Research has asserted MCQ items using three response options (one correct answer with two distractors) is comparable to, and possibly preferable over, traditional MCQ item formats consisting of four response options (e.g., one correct answer with three distractors), or five response options (e.g., one correct answer with four distractors). Some medical educators have also adopted the practice of using 3-option responses on MCQ exams as a response to the difficulty experienced in generating additional plausible distractors. To date, however, little work has explored how 3-option responses might impact validity threats stemming from random guessing strategies, and what impact 3-option responses might have on cut-score determinations, particularly in the context of medical education classroom assessments. The purpose of this work is to further explore these critically important considerations that largely have gone ignored in the medical education literature to this point...

Journal of the Finnish Economic Association, 2023

Journal of Advances in Medical Education & Professionalism, 2017

Introduction: Research has asserted MCQ items using three response options (one correct answer with two distractors) is comparable to, and possibly preferable over, traditional MCQ item formats consisting of four response options (e.g., one correct answer with three distractors), or five response options (e.g., one correct answer with four distractors). Some medical educators have also adopted the practice of using 3-option responses on MCQ exams as a response to the difficulty experienced in generating additional plausible distractors. To date, however, little work has explored how 3-option responses might impact validity threats stemming from random guessing strategies, and what impact 3-option responses might have on cut-score determinations, particularly in the context of medical education classroom assessments. The purpose of this work is to further explore these critically important considerations that largely have gone ignored in the medical education literature to this point...

2016

ABSTRACT. The aim ofsthis study was to show that the superiority of corrected-for-guessing scores over nuiber right scores, as true score estimates depeAds on the ability of examinees to recognize situations where they can eliminate one or more alternatives as-incorrect andto omit items.where they would only be guessing randomly. Previous ' investigations'show examinees lack this ability. An instructional strategy for teaching examinees to use pertial information in taking multiple choide tests under formula scoring conditions was tested on 280 fourth, fifth and sixth grade, private and publiC school students from the United-States Virgin Islands. 'Ma * method is outlined and followed by sections dealing with the administration of pre-tests and post- tests ' and the provisiOn of training to selected groups (in the form of an algorithm ffm attacking test Items, and practice sessions with immediate feedback). It 1 8 PAicated it the final.discussion of.research resu...

We present a new model of liberal (also known as free-choice) multiple-choice tests in which a candidate’s test score is represented as a function of two other variables: their inclination to eliminate perceived distracters and the accuracy with which they are able to do this. We propose a number of theoretical relationships between these three variables that appear to support the notion that good students ought to be better judges of their level of knowledge – and hence better able to benefit through the expression of partial knowledge – than weak students, as previous empirical studies have found. However, the results of our own small empirical study were not in line with our expectations. This work has given us a better understanding of the test format, but we now doubt whether it does in fact provide the benefits that have previously been claimed.

Acta psychologica, 2003

This paper assesses framing effects on decision making with internal uncertainty, i.e., partial knowledge, by focusing on examinees' behavior in multiple-choice (MC) tests with different scoring rules. In two experiments participants answered a general-knowledge MC test that consisted of 34 solvable and 6 unsolvable items. Experiment 1 studied two scoring rules involving Positive (only gains) and Negative (only losses) scores. Although answering all items was the dominating strategy for both rules, the results revealed a greater tendency to answer under the Negative scoring rule. These results are in line with the predictions derived from Prospect Theory (PT) [Econometrica 47 (1979) 263]. The second experiment studied two scoring rules, which allowed respondents to exhibit partial knowledge. Under the Inclusion-scoring rule the respondents mark all answers that could be correct, and under the Exclusion-scoring rule they exclude all answers that might be incorrect. As predicted by PT, respondents took more risks under the Inclusion rule than under the Exclusion rule. The results illustrate that the basic process that underlies choice behavior under internal uncertainty and especially the effect of framing is similar to the process of choice under external uncertainty and can be described quite accurately by PT. (Y. Bereby-Meyer). 0001-6918/02/$ -see front matter Ó 2002 Elsevier Science B.V. All rights reserved. PII: S 0 0 0 1 -6 9 1 8 ( 0 2 ) 0 0 0 8 5 -9

Mind & Society, 2005

We offer a case-study in irrationality, showing that even in a high stakes context, intelligent and well trained professionals may adopt dominated practices. In multiple-choice tests one cannot distinguish lucky guesses from answers based on knowledge. Test-makers have dealt with this problem by lowering the incentive to guess, through penalizing errors (called formula scoring), and by eliminating various cues for outperforming random guessing (e.g., a preponderance of correct answers in middle positions), through key balancing. These policies, though widespread and intuitively appealing, are in fact ''irrational'', and are dominated by alternative solutions. Number-right scoring is superior to formula scoring, and key randomization is superior to key balancing. We suggest that these policies have persisted since all stake-holders -test-makers, test-takers and test-coaches -share the same faulty intuitions.