580 California St., Suite 400
San Francisco, CA, 94104
Key research themes
1. How does test-wiseness influence test-taking performance and can it be effectively enhanced through training?
This area investigates test-wiseness as a cognitive ability and a strategic skillset that individuals use to navigate test formats, differentiate correct answers, and optimize their test results beyond raw content knowledge. Understanding the nature of test-wiseness and its teachability is crucial as it impacts test validity and fairness by possibly inflating or deflating scores based on strategic test behavior rather than actual subject mastery.
Key finding: The paper conceptualizes test-wiseness as a specific cognitive skill distinct from guessing or risk-taking and capable of improvement via targeted training. It reports that instructional programs focusing on test-taking... Read more
Key finding: This study empirically identifies common test-taking strategies used by students, distinguishing between effective ('test-wise') strategies—like excluding incorrect options and thoroughly reading instructions—and ineffective... Read more
Key finding: This work demonstrates that brief interventions teaching test-taking strategies (test-wiseness) significantly improve standardized numeracy test performance among primary school students. The findings caution against... Read more
2. How can psychometric and cognitive models capture and explain test-wiseness effects in multiple-choice and adaptive testing contexts?
This theme explores formal models and empirical methods to disentangle actual ability from test-taking strategies (test-wiseness), especially in computer adaptive testing (CAT) and multiple-choice exams. Developing models that quantify guessing, aberrant responses, and strategic elimination can improve the accuracy of ability estimation and test fairness.
Key finding: Using a mixture of binomial models reflecting different levels of incorrect answer elimination, the study quantifies test-wiseness in multiple-choice exams, finding that about 26% of responses correspond to random guessing,... Read more
Key finding: This CAT simulation study shows that aberrant responses such as lucky guesses and careless errors bias ability estimation, but modeling these via a four-parameter logistic (4PL) IRT model that incorporates an inattention... Read more
Key finding: The study establishes that adaptive test features, notably item difficulty sequencing, impact test takers' perceived performance and reactions, mediating motivation, satisfaction, and anxiety. The research suggests that... Read more
3. What approaches can improve the prediction, measurement, and evaluation of test-related behaviors like test-wiseness and their impact on testing outcomes?
This research cluster focuses on operationalizing, predicting, and evaluating test-wiseness and related constructs such as testedness and flakiness through empirical metrics, machine learning, and large-scale empirical studies. It addresses the challenge of measuring test-taking constructs that influence testing outcomes, aiming to enhance test quality, security, and interpretability.
Key finding: By integrating coverage and test suite size into a novel testability metric and leveraging 262 software metrics with machine learning, this study achieves robust prediction (R²=0.68, MSE=0.03) of source code testability. This... Read more
Key finding: Through a systematic review, this paper highlights limited maturity and lack of empirical validation in knowledge on software testing techniques, pointing to a need for stronger empirical studies in real-world settings. The... Read more
Key finding: This research introduces an approach to predict flaky (non-deterministic) tests via natural language processing of test code vocabulary. Achieving an F-measure of 0.95, it evidences that linguistic patterns in test code... Read more
Key finding: By fine-tuning CodeBERT on test case source codes, Flakify predicts flaky tests more accurately (precision and recall improvements of 10 and 18 percentage points, respectively) than prior white-box methods that use production... Read more