Harold O'Neil

Followers

Following

Co-authors

Public Views

Interests

Uploads

Papers by Harold O'Neil

Assessment of Teamwork Skills Using Computer-Based Teamwork Simulations

Estimate of the Potential Costs and Effectiveness of Scaling Up CRESST Assessment Software

This report examines issues in the scale-up of assessment software from the Center for Research o... more This report examines issues in the scale-up of assessment software from the Center for Research on Evaluation, Standards, and Student Testing (CRESST). "Scale-up" is used in a metaphorical sense, meaning adding new assessment tools to CRESST's assessment software. During the past several years, CRESST has been developing and evaluating a series of software products to measure various types of learning. Collectively, these are called the Integrated Assessment System. The integrated simulation CRESST has developed includes both collaborative and individual concept mapping tasks, a problem-solving search task, an explanation task, and a metacognitive questionnaire. New tools should measure one or more aspects of the CRESST model of learning, be stand-alone or integrate with other tools, have authoring capacity, use automated scoring, and be Internet-deployable. Specifications are listed for eight proposed tools, and each is evaluated for the CRESST requirements. The tools are: (1) Collaborative Concept Mapper, a tool for group use; (2) Flowcharter, a tool to allow students to sequence events; (3) Idea Generator and Evaluator, a tool for group problem solving; (4) Model Simulator, a model building tool; (5) Networked Team Simulator, a team model builder; (6) Multimedia Concept Mapper, a multimedia tool for concept mapping; (7) Outliner, an organizing tool; and (8) Problem Solver, an instrument to measure problem solving processes. (Contains 4 tables and 13 references.) (SLD)

Download

Adding self-explanation prompts to an educational computer game

Computers in Human Behavior, 2014

Proponents envision a role for computer games in improving student learning of academic material,... more Proponents envision a role for computer games in improving student learning of academic material, including mathematics and science. Asking learners to engage in self-explanations during learning has been found to be an effective instructional method. In the present experiment, we examined the effects of adding a self-explanation prompt-asking players to answer one of three questions after completing each level of the game-within a children's math game on addition of fractions. Middle-school participants played either a base version of the game (n = 57) or the base version with a self-explanation instructional feature (n = 57). Participants' learning was measured by a fractions posttest and their learning processes measured via in-game measures of game progress and errors. When we separated the selfexplanation condition into participants who used a focused self-explanation strategy versus those who did not, the focused participants had significantly fewer game level deaths, game level resets, and progressed significantly farther in the game, compared to the control group, than participants not using a focused self-explanation strategy. The major new contribution of this study is that self-explanation can help the process of playing educational games in some situations and hurt in others. In particular, the most effective self-explanation prompts were aimed at helping learners make connections between game terminology and mathematics terminology, whereas the least effective self-explanation prompts asked very simple or very abstract questions.

Download

Reliability and validity of a computer-based knowledge mapping system to measure content understanding

Computers in Human Behavior, May 1, 1999

... The other system is a networked collaborative mapping system designed to measure teamwork ski... more

The Role of Anxiety on Novices' Rifle Marksmanship Performance

Educational Assessment, Sep 1, 2005

ABSTRACT

The use of computer-based collaborative knowledge mapping to measure team processes and team outcomes

Computers in Human Behavior, May 1, 1999

In this study we examined the feasibility and validity of using a computer-based, networked colla... more In this study we examined the feasibility and validity of using a computer-based, networked collaborative knowledge mapping system to measure teamwork skills. Student groups were assessed with our system twice in the same academic year, once in the fall and once in the following spring. Our study focused on the nature of t h e interaction between team members as they jointly constructed a knowledge map. Each student was randomly assigned to a team and communicated (anonymously) with other members by sending pre-defined messages. Teamwork processes were measured by examining message usage. Each message was categorized as belonging to one of six team processes: (a) adaptability, (b) communication, (c) coordination, (d) decision-making, (e) interpersonal, and (f) leadership. Team performance was measured by scoring each team's knowledge map using four expert maps as the criterion. No significant correlation was found between the team processes and team outcomes. This unexpected finding may be due in part to a split-attention effect resulting from the design of t h e user interface. However, student teams were able to successfully construct knowledge maps, suggesting that our general approach to using networked computers to measure group processes remain viable given existing alternatives.

Download

Methodological Approaches to Online Scoring of Essays

In this report, we examine the feasibility of scoring essays using computerbased techniques. We r... more In this report, we examine the feasibility of scoring essays using computerbased techniques. We review two approaches that have been used to classify documents, surface-and word-based analysis techniques. We omit other text analysis techniques, such as content analyses (Krippendorff, 1980; Roberts &

Download

Computer-Based Collaborative Knowledge Mapping To Measure Team Processes and Team Outcomes

The feasibility and validity of using a computer-based networked collaborative knowledge mapping ... more The feasibility and validity of using a computer-based networked collaborative knowledge mapping system to measure teamwork skills was examined. Student groups (10 groups of 3 ninth graders each) were assessed with the system twice in the academic year, once in the fall and once the following spring. The study focused on the nature of the interaction among team members as they jointly constructed a knowledge map. Each student was randomly assigned to a team and communicated (anonymously) with other members by sending predefined messages. Teamwork processes were measured by examining message usage. Each message was categorized as belonging to one of six team processes: (1) adaptability; (2) communication; (3) coordination; (4) decision making; (5) interpersonal; and (6) leadership. Team performance was measured by scoring each team's knowledge map using four expert maps as the criterion. No significant correlation was found between team processes and team outcomes. This unexpected finding may be due in part to a split-attention effect resulting from the design of the user interface. However, student teams were able to construct knowledge maps successfully, suggesting that the general approach to using networked computers to measure group processes remains viable. Two appendixes contain messages grouped by category and message content and the handout describing messages. (Contains 17 tables and 21 references.) (Author/SLD) *

Download

Examining the Validity of Knowledge Mapping as a Measure of Elementary Students' Scientific Understanding. CSE Technical Report

Knowledge mapping is expected to measure deep conceptual understanding and allow students to char... more Knowledge mapping is expected to measure deep conceptual understanding and allow students to characterize relationships among concepts in a domain visually. This research examined the validity of knowledge mapping as an assessment tool in science. The approach to investigating this validity was three-pronged. First, a model was outlined for the creation of knowledge mapping tasks, proposing a standard set of steps and using content area and educational experts to ensure the content validity of the measures. Next, a scoring method was developed to evaluate student performance. This report contains a discussion of the methods reliability and its relation to other possible scoring systems. Finally, the report presents the statistical results based on the participation of 56 fourth and fifth graders, including comparative analyses, the multitrait-multimethod (MTMM) validity analyses involving two traits (students understanding of hearing and of vision), and three different measurement methods (knowledge mapping, essay, and multiple-choice tasks), critical proposition analyses, and analyses of students, propositional elaborations. Results show knowledge maps to be sensitive to students competency level, with mixed MTMM results. The report Project 1.3. Technology in Action

Download

Use of Networked Collaborative Concept Mapping To Measure Team Processes and Team Outcomes

The feasibility of using a computer-based networked collaborative concept mapping system to measu... more The feasibility of using a computer-based networked collaborative concept mapping system to measure teamwork skills was studied. A concept map is a node-link-node representation of content, where the nodes represent concepts and links represent relationships between connected concepts. Teamwork processes were examined for a group concept mapping activity using networked computers and HyperCard software developed for the study. Two pilot studies investigated the feasibility of the approach, and one of these is reported in this study. Thirty 9th graders worked together in the first pilot study to create collaborate maps on the computer systems. Results of this study and a second confirmatory study resulted in development of a study with 23 groups of 3 students each from middle schools and high schools. Usable data were recorded for 15 groups (45 participants). Results suggest that the real-time measurement of teamwork skills is feasible and that students using the networked computers were able to create a concept map jointly. The number of concepts used by groups ranged from 7 to 18, and the number of links ranged from 8 to 26. It is suggested, however, that the task involved may be more like a small group task than a "team" task if the purpose of a team is defined as performing a task an individual cannot perform alone. The developed software architecture has been designed to be

Download

Designing and Implementing an Academic Scorecard

Change: The Magazine of Higher Learning, 1999

... Michael A. Diamond is Executive Vice Provost of USC. Michael R. Moore is Director of the Stra... more

Reliability and Validity of a State Metacognitive Inventory: Potential for Alternative Assessment

The Journal of Educational Research, 1996

An assumed advantage of alternative assessments is that they result in more higher level thinking... more An assumed advantage of alternative assessments is that they result in more higher level thinking or metacognitive skills. We believe that this advantage should be measured directly and explicitly. Unfortunately, few standardized measures of metacognitive skills (planning, monitoring, cognitive strategies, and awareness) exist. In our studies, for 12th graders, alpha reliability estimates and factor analysis indicated that our metacognitive subscales are reliable (alpha above .70) and unidimensional (one factor per subscale). Because the subscales have only 5 items each, they meet brevity standards. Construct validity of our state metacognitive inventory is acceptable. Results indicate that our state metacognitive inventory yields useful information about both the assessment and students. Alternative assessments share several common characteristics. Herman, Aschbacher, and Winters (1992) provide an excellent listing of such characteristics: (a) Ask students to perform, create, produce, or do something;Ê(b) tap higher level thinking and problem-solving skills;Ê(c) use tasks that represent meaningful instructional activities;Ê(d) invoke real-world applications;Ê(e) people, not machines, do the scoring, using human judgment; andÊ(f) require new instructional and assessment roles for teachers (p. 6). Performance assessment is by nature a process that requires extended engagement by students in order to demonstrate their proficiency. They may conduct multistep experiments, write well-documented research papers, organize and supervise group problem solving, or present a description of previously developed work. Although the exact nature of these tasks may differ

Download

Empirical investigations versus anecdotal observations concerning anxiety in computer-assisted instruction

Journal of School Psychology, 1970

Abstract Empirical data are offered as counter evidence to the negative criticisms of Computer-As... more

Ethnicity, Effort, Self-Efficacy, Worry, and Statistics Achievement in Malaysia: A Construct Validation of the State-Trait Motivation Model

Educational Assessment, 2002

The relations between motivational constructs, effort, self-efficacy and worry, and statistics ac... more

Computer-Based Feedback for Computer-Based Collaborative Problem Solving

Computer-Based Diagnostics and Systematic Analysis of Knowledge, 2009

Collaborative problem solving is defined as problem solving activities that involve interactions ... more Collaborative problem solving is defined as problem solving activities that involve interactions among a group of individuals. Collaborative problem solving is considered a necessary skill for success in today’s world and schooling. The purpose of this chapter is to further investigate the role of computer-based feedback in collaborative problem solving, in particular, the effect of narration plus on-screen text versus

A Cognitive Demands Analysis of Innovative Technologies

PsycEXTRA Dataset

Feasibility of Machine Scoring of Concept Maps

PsycEXTRA Dataset

This report documents progress at the Center for Research on Evaluation, Standards, and Student T... more This report documents progress at the Center for Research on Evaluation, Standards, and Student Testing (CRESST) on the feasibility of scoring concept maps using technology. CRESST, in its integrated simulation approach to assessment, has assembled a suite of performance assessment tasks (the integrated simulation) onto which they have mapped the types of learning-expected of students. One element of the integrated simulation is an online concept mapping construction and scoring system. A concept map is a graphical representation of information consisting of nodes and labeled lines. The nodes correspond to concepts in a subject area or domain, and lines indicate a relationship between pairs of concepts. Concept mapping software has been developed to permit students to construct concept mapd and to provide real-time scoring and feedback to students based on an expert's map. An expert criterion map is used to score students' concept maps. CRESST has also studied the viability of using collaborative concept mapping in a networked computer environment as an assessment tool. Ongoing research suggests that both types of concept map use are promising. A feasible solution for schools without extensive computer resources is suggested through computerized scoring of paper-and-pencil administered concept maps. Several technologies for this purpose that CRESST has explored are described. (Contains 2 tables, 3 figures, and 12 references.) (SLD)

Download

Classification of learning outcomes: evidence from the computer games literature

The Curriculum Journal, 2005

Following up on an earlier issue of The Curriculum Journal (Vol. 16, No. 1), this article focuses... more Following up on an earlier issue of The Curriculum Journal (Vol. 16, No. 1), this article focuses on learning outcomes in the context of video games. Learning outcomes are viewed from two theoretical frameworks: Kirkpatrick's levels of evaluation and the CRESST model of learning. These are used to analyse the outcomes claimed in journal articles that report empirical work, indicating the usefulness of the frameworks, and the necessity to consider the role of affective learning. The article ends with some comments on the relationship of instructional design to effective games and learning outcomes.

Download

Measuring problem solving in computer environments: current and future states

Computers in Human Behavior, 2002

The first part of this article will review the status of technology and educational reform; the s... more The first part of this article will review the status of technology and educational reform; the second will treat traditional approaches to problem-solving measurement common today in technological settings; and the third will propose how problem-solving assessment may change and, in particular, the role that authoring strategies may play in measuring problemsolving performance successfully in computer environments. We will suggest four areas that must be considered in which computers would increase the fidelity and validity of measures of complex problem solving: (1) the intentions and skills of assessment designers; (2) the range of performance that counts as problem solving; (3) the ways in which validity evidence can be sought; and (4) the degree to which the measurement produces results that generalize across tasks and contexts.

Download

The Validity of Knowledge Mapping as a Measure of Elementary Students' Scientific Understanding

PsycEXTRA Dataset

Although first popular as an instructional tool in the classroom, knowledge mapping has been used... more Although first popular as an instructional tool in the classroom, knowledge mapping has been used increasingly as an assessment tool. Knowledge mapping is expected to measure deep conceptual understanding and allow students to characterize relationships among concepts in a domain visually. Our research examines the validity of knowledge

Download

Harold O'Neil

Uploads

Papers by Harold O'Neil

Log In