Papers by Diego Zapata-Rivera
Conversation-based assessment: current findings and future work
Elsevier eBooks, 2023
Learner Modeling in Conversation-Based Assessment
Lecture Notes in Computer Science, 2022
Using Trialogues to Measure English Language Skills
J. Educ. Technol. Soc., 2015
We explored the use of technology-assisted, trialogue-based tasks to measure the English language... more We explored the use of technology-assisted, trialogue-based tasks to measure the English language proficiency of students learning English as a second or foreign language. A presumed benefit of the system for language assessment is its suitability for use in scenario-based tasks that integrate multiple language skills. This integration allows test developers to simulate real-life language use situations by administering more authentic test tasks to strengthen the link between test performance and its interpretation as an indicator of expected success in performing real-life language use tasks. We conducted a usability study with 20 English language learners (ELLs) in grades 3 to 5, representing ten different first languages. Results suggested that the trialoguebased tasks engaged students in an authentic experience and provided evidence of their English proficiency. Perception data indicated that students perceived the trialogue-based tasks positively and understood how to respond t...
Providing Formative Feedback to Pre-service Teachers as they Practice Facilitation of High-Quality Discussions in Simulated Mathematics and Science Methods Classrooms
Communicating Measurement Error Information to Teachers and Parents
Score Reporting Research and Applications, 2018
Practical Assessment, Research and Evaluation, 2021
Individual-student score reports sometimes include information about precision of scores (i.e., m... more Individual-student score reports sometimes include information about precision of scores (i.e., measurement error). In this study, we specifically investigated if parents understand this information when presented. We conducted an online experimental study where 196 parents of middle school children, from various parts of the country, were randomly assigned to three conditions with different amounts of measurement error information. Parents in all conditions answered a series of comprehension questions about a student's performance on a hypothetical test. Results indicate that when information about error was presented, parents showed a significantly better understanding of score variability. Moreover, when asked about their preference for such information, parents across all three conditions indicated that they would like such information to be included in their child's report. Results from this study highlight the importance of clear communication of technical information to stakeholders, particularly parents, who are a diverse stakeholder group. The correct interpretation and use of test performance results by various stakeholders is an essential part of the argument-based approach to validation . These test performance results in the form of scores and their intended meaning are communicated to stakeholders (including parents) through some form of a score report. Reporting scores meaningfully to stakeholders, including parents, so that they are accurately interpreted and appropriately used, is critical to the validity arguments supporting the assessment (Tannenbaum, 2019).
Computer Assisted Language Learning, 2018
This study investigated a new conversation-based assessment for English language learners. In thi... more This study investigated a new conversation-based assessment for English language learners. In this assessment, students converse with animated agents in natural language conversations to assess various aspects of English language learning. In a between subjects design, 31 students (N ¼ 31) were asked questions by either the newly created system or human interviewers. Results revealed that there was no difference in answers in regards to impacting the constructs under investigation. However, there were other differences including higher word count and potentially more complex language usage when speaking to a human. Furthermore, students liked interacting with the system. These results suggest that the newly created system can assess students' English capabilities comparably to human interviewers and implementing such a formative assessment is feasible.
IEEE Transactions on Learning Technologies, 2018
Students can experience a variety of emotions while completing assessments. Some emotions can get... more Students can experience a variety of emotions while completing assessments. Some emotions can get in the way of students performing their best (e.g., anxiety, frustration), whereas other emotions can facilitate student performance (e.g., engagement). Many new, non-traditional assessments, such as automated conversation-based assessments (CBA), are designed with the intention to create a test-taking experience that maximizes beneficial emotions for students and minimizes detrimental emotions. However, there is a paucity of research on students' actual emotional experiences during these nontraditional assessments. Across two studies, we investigated students' moment-to-moment emotions during two CBAs that differed on construct (science inquiry, mathematical reasoning) and environment (virtual world, chat box). We found a similar set of emotions to frequently occur across the two studies and came to the preliminary conclusion that students have a generally positive experience with CBAs and that boredom, confusion, curiosity, delight, engagement/flow, frustration, happiness/enjoyment, hope, and pride are the prevalent emotions in CBAs. The temporal dynamics of emotions as well as the emotion-performance relationship were also investigated across the two studies. Lastly, we discuss how these findings can inform the development of emotion-sensitive CBAs that can facilitate students being able to perform to the best of their ability.
Exploring the Effectiveness of a Measurement Error Tutorial in Helping Teachers Understand Score Report Results
Educational Assessment, 2016
The goal of this study was to explore the effectiveness of a short web-based tutorial in helping ... more The goal of this study was to explore the effectiveness of a short web-based tutorial in helping teachers to better understand the portrayal of measurement error in test score reports. The short video tutorial included both verbal and graphical representations of measurement error. Results showed a significant difference in comprehension scores between each of two tutorial groups (basic and enhanced) and the control group (no tutorial) but not between the two tutorial groups. Results also provided evidence of teachers' misconceptions about the meaning of measurement error and confidence bands.
Intelligent Systems
International Encyclopedia of Education, 2010
Frontiers in Education, Jul 17, 2023
The field of score reporting continues to evolve because of new challenges, opportunities, and ne... more The field of score reporting continues to evolve because of new challenges, opportunities, and needs of society (e.g., COVID, remote teaching and learning). In this paper, the new challenges and opportunities in score reporting are discussed from the personal perspective of four experts who have previously conducted research in designing score reports in education. Comments are organized around four key questions concerning challenges raised by the Covid pandemic, how research will change, what current research is being conducted, and new directions in the field of score reporting.
ETS Research Report Series
Substantial progress has been made toward applying technology enhanced conversation‐based assessm... more Substantial progress has been made toward applying technology enhanced conversation‐based assessments (CBAs) to measure the English‐language proficiency of English learners (ELs). CBAs are conversation‐based systems that use conversations among computer‐animated agents and a test taker. We expanded the design and capability of prior conversation‐based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 resp...
ETS Research Report Series, 2007
This paper describes research and development efforts related to adaptive technologies, which can... more This paper describes research and development efforts related to adaptive technologies, which can be combined with other technologies and processes to form an adaptive system. The goal of an adaptive system, in the context of this paper, is to create an instructionally sound and flexible environment that supports learning for students with a range of abilities, disabilities, interests, backgrounds, and other characteristics. After defining key terms and establishing a rationale for adaptation, we present a general framework to organize adaptive technologies. We then describe experts' thoughts on what to adapt and how to adapt and conclude with a summary of key challenges and potential futures of adaptive technologies.
Many adult workers need to keep up with advances in technology to remain relevant in the job mark... more Many adult workers need to keep up with advances in technology to remain relevant in the job market. Adults now need 21st century skills including Computational Thinking. It is challenging for adults to find training opportunities that take into account their limited time, educational, and resource constraints. Our approach provides support to adult learners and their tutors to help them reach their goals. This support can take the form of facilitation messages that suggest possible learning activities, hints on study and time management for learners, and instructional suggestions and alerts for tutors based on current information gleaned from the learner and the interaction. We have collected data with adult learners and their tutors, and are designing an automated facilitator system that can provide tutors and learners with feedback according to their needs. CCS Concepts CCS → Applied computing → Education
ETS Research Report Series, 2017
Since its 1947 founding, ETS has conducted and disseminated scientific research to support its pr... more Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.
The Handbook of Cognition and Assessment, 2016
Learner Modeling in the Context of Caring Assessments
Adaptive Instructional Systems, 2020
Despite recent advances in technology, personalized learning to address diverse needs of students... more Despite recent advances in technology, personalized learning to address diverse needs of students remains difficult to achieve at scale. With the availability and affordability of smart devices in the era of the Internet of Things, learners, parents, and educators are more “connected” than ever before. Education stakeholders and technology developers can leverage these advances to collect data about, inform, deliver, and improve education for all learners. In this paper, we review the core components of a Smart Learning framework and describe a personalized mastery-based learning system that leverages the framework to deliver personalized learning at scale. In the context of Smart Learning in the Internet of Things, we propose an Ambient and Pervasive Personalized Learning Ecosystem (APPLE), a learner-centered approach that uses Bloom’s Four Agents of Change in the Internet of Things ecosystem to provide learners a comprehensive and personalized learning experience. This ecosystem u...
Frontiers in Education
Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digit... more Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digital learning environments to provide students, teachers, and administrators with a wealth of information about student’s engagement, experiences, and performance on tasks. LADs have become increasingly popular, particularly in formative learning contexts, and help teachers make data-informed decisions about a student’s developing skills on a topic. LADs afford the possibility for teachers to obtain real-time data on student performance, response processes, and progress on academic learning tasks. However, data presented on LADs are often not based on an evaluation of stakeholder needs, and have been found to not be clearly interpretable and actionable for teachers to readily adapt their pedagogical actions based on these insights. We elaborate on how insights from research focused on interpretation and use of Score Reporting systems and research on open learner models (OLMs) can be used t...
2020 6th International Conference of the Immersive Learning Research Network (iLRN), 2020
This Special Session will examine the opportunities and challenges associated with developing and... more This Special Session will examine the opportunities and challenges associated with developing and implementing authentic assessments in XR to inform teachers, faculty, researchers, and industry trainers interested in advancements in this field. Methods for conducting real-time assessments in XR environments will be demonstrated and include audience participation. A brief introduction regarding the state of assessment in XR will be presented by moderator Dr. Cindy Ziker, followed by four presenters and an interactive discussion on assessment in XR that will include topics posed by the audience. Evelien Ydo will present on performance-based assessment (PBA) and simulation-based assessment (SBA) using virtual reality. Dr. Diego Zapata-Rivera's presentation will focus on conducting adaptive, embedded assessments that leverage the virtual presence of experts, teachers, and observers, and interactions with virtual agents and virtual objects. Dr. Mathew Hillier will present on the "Transforming Assessment Project", that includes the use of unique software tools and augmented reality. Dr. Michael Casale will present on innovations in work-based assessment and industry training.
Lecture Notes in Computer Science, 2018
Pedagogical agents are widely employed in intelligent tutoring systems and game-based learning en... more Pedagogical agents are widely employed in intelligent tutoring systems and game-based learning environments, but research suggests that learning benefits from virtual agents vary as a function of agent features (e.g., the form or register of agent dialogue) and student characteristics (e.g., prior knowledge). Students' responses to agent-based conversations provide useful evidence of students' knowledge and skills for assessment purposes; however, it is unclear whether these agent design features and student characteristics similarly influence students' interactions with agents in assessment (versus learning) contexts. In this paper, we explore relationships between agent features and student characteristics within a conversation-based assessment of science inquiry skills. We examined the effects of virtual agents' knowledge status (low vs. high) and language use (comparative vs. argumentative question framing) on agent perceptions (ratings) in a conversation-based assessment of scientific reasoning (i.e., using data to predict the weather). Preliminary results show that the effects of these features on students' perceptions of agents varied as a function of students' background characteristics, consistent with research on learning from agents. Implications for designing agent-based assessments will be discussed.
Uploads
Papers by Diego Zapata-Rivera