Challenges of assessing collaborative problem solving

2017

There is an increased interest in collaboration and teamwork in the workforce, higher education, and K-12 education. The Assessment and Teaching of the 21st Century Skills (ATC21S) included collaboration among the most important skills necessary for a successful career. While there is widespread agreement in the field of education that collaboration is an important skill, there is less agreement on how to build an assessment to measure it. In 2015 the Organization for Economic Cooperation and Development (OECD) published its draft framework, which included strong rationale for the inclusion of CPS, calling collaborative problem solving “a critical and necessary skill across educational settings and in the workforce.” This plan by the OECD to add CPS to its 2015 PISA test has important implications for U.S. assessments, including the National Assessment of Educational Progress (NAEP).

Interactive Learning Environments, 2021

In this paper, we present an agenda for the research directions we recommend in addressing the issues of realizing and evaluating communication in CPS instruments. We outline our ideas on potential ways to improve (1) generalizability in Human–Human assessment tools and ecological validity in Human–Agent ones; (2) flexible and convenient use of restricted communication options; and (3) an evaluation system of both Human–Human and Human-Agent instruments. Furthermore, in order to demonstrate possible routes for realizing some of our suggestions, we provide examples through an introduction of the features of our own CPS instrument. It is a Human– Human pre-version of a future Human–Agent instrument and a promising diagnostic and research tool in its own right, as well as the first example of transforming the so-called MicroDYN approach so that it is suitable for Human–Human collaboration. We offer new alternatives for communication in addition to pre-defined messages within the test, which are also suitable for automated coding. For example, participants can send or request visual information in addition to verbal messages. As regards evaluation as a hybrid solution, not only are the pre-defined messages proposed as indicators of different CPS skills, but so are a number of behavioural patterns.

Elsevier eBooks, 2023

by the primary investigator through observation of the students during PBL sessions. Duration of study was six months. Results: We found that in our students, development of social dimension skills is facilitated to a greater extent than the development of cognitive dimension skills through the process of PBL. These skills are generally better developed in the leader compared to the scribe and members in a group. They are also more developed in females compared to males. Modification in them is also observed as the year's progress. Conclusion: Although PBLs facilitate development of CPS skills' progression however in our curriculum, PBLs mainly focus on social skills development and have less emphasis on cognitive skill development. Thus, hybrid instructional strategies with components from TBL and mentorship are recommended for better development of CPS skills.

Assessment and Teaching of 21st Century Skills, 2014

In his book "Cognition in the Wild", Hutchins (1995) invites his readers to scan their immediate environment for objects that were not produced through collaborative efforts of several people, and remarks that the only object in his personal environment that passed this test was a small pebble on his desk. In fact, it is remarkable how our daily lives are shaped by collaboration. Whether it is in schools, at the workplace, or in our free time, we are constantly embedded in environments that require us to make use of social skills in order to coordinate with other people. Given the pervasiveness of collaboration in everyday life, it is somewhat surprising that the development of social and collaborative skills is largely regarded as something that will occur naturally and does not require any further facilitation. In fact, groups often fail to make use of their potential (Schulz-Hardt, Brodbeck, Group performance and leadership. In: Hewstone M, Stroebe W, Jonas K (eds) Introduction to social psychology: a European perspective, 4th edn, pp 264-289. Blackwell, Oxford, 2008) and people differ in the extent to which they are capable of collaborating effi ciently with others. Therefore, there is a growing awareness that collaborative skills require dedicated teaching efforts (Schoenfeld, Looking toward the 21st century: challenges of educational theory and practice. Edu Res 28:4-14, 1999). Collaborative problem solving has been identifi ed as a particularly promising task that draws upon various social and cognitive skills, and that can be analysed in classroom environments where skills are both measurable and teachable. This chapter provides a conceptual framework of collaborative problem solving that is informed by fi ndings from fi elds of research as diverse as cognitive science, education, social psychology and psycholinguistics.

International Journal of Emerging Technologies in Learning (iJET)

As been highlighted by many, for instance by PISA, Collaborative Problem Solving (CPS) is a critical and necessary 21st century skill across educational settings. While many initiatives have been launched to investigate the nature of these skills, fewer are the attempts to understand how they should be assessed. However, in 2015, the PISA organization presented a framework for assessing CPS skills. This paper reports on a study investigating the predictive validity of the PISA assessment framework and if and how modes of communication influence the assessment of 24 students’ collaborative problem solving activities when using a computer-based assessment task system. The findings presented demonstrate that the PISA CPS assessment framework have a weak predictive validity, does not count for quality or productivity in communication, and that the mode of communication indeed influence CPS processes and in turn what is possible to assess.

2015

Assessing collaborative problem solving (CPS) is an integrated part of the computer-supported collaborative learning (CSCL). We present some preliminary results from a project developed for assessing the CPS using web-based simulation. In the simulation, two participants collaborate via a chat box to complete a task on volcano science. By comparing the responses from 486 individuals and 278 teams (dyads) recruited from Amazon Mechanical Turk, we found the performance from the teams (dyads) is significantly higher than that from individuals. We also find that the item difficulty in the simulation affects both the processes and outcomes of the collaboration.

Journal of Chemical Education, 2008

and keywords Full text (PDF) Links to cited URLs and JCE articles Color figures Supplement Detailed description of the experimental methods used in this study Discussion of the data modeling using artificial neural nets (ANN), hidden Markov modeling (HMM), and item response theory (IRT)

ETS Research Report Series, 2019

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.

AERA Open, 2017

Collaborative problem solving (CPS) has been described as a critical skill for students to develop, but the nature of communication between individuals in a collaborative situation causes difficulty in measuring students' ability using traditional testing approaches, such as multiple choice, extended answer, peer review, or teacher observation. Further, when students are scored on a group task, teachers often allocate one score to the whole group of students, disregarding individual students' performances, and that score might reflect achievement of the correct solution rather than contribution to the collaborative process. Another issue is that teachers who wish to assess CPS in their classrooms may not have the scope to do so if this requires deviating from teaching the mandated curriculum. Such issues in assessing CPS arise regardless of the teachers' specific content area specialization. This paper considers collaborations in mathematics classrooms in junior high school or middle school (with students in approximately Years [or Grades] 6, 7, and 8) in six different countries. Students in this age range commonly study basic algebra and the concepts of variables, integers, and polynomials. Their teachers need to manage the transition from arithmetic to algebra and accommodate current curriculum reforms emphasizing capabilities such as CPS and critical thinking. These latter capabilities require accurate and meaningful assessment, just as the more familiar content areas do. These issues and complications in measuring CPS while teaching a content domain underlie the overarching research question addressed in this study: How can teachers assess CPS for students individually while they teach mathematics? Problem solving has been assessed for several decades, following ) publication of his four-step process within the domain of mathematics. As this process was embedded within mathematics curricula, problem solving was generally regarded as a mathematics skill. Studies in the latter part of the 20th century and in the early part of the 21st century linked problem solving with decision making, critical thinking, and collaboration (e.g., . The inclusion of these skills extended the exploration of problem solving beyond the field of mathematics into the sciences and other discipline areas, and the skills are now broadly considered to be independent of those disciplines and a subject of learning in their own right. The combination

2018

This paper presents an interactive team collaborative learning and problem-solving (ITCLP) framework for effective teamwork learning and assessment. Modeling the dynamics of a collaborative, networked system involving multimodal data presents many challenges. This framework incorporates an Artificial Intelligence (AI), a Machine Learning (ML) and computational psychometrics (CP) based methodology, system architecture, and algorithms to find patterns of learning, interactions, relationships, and effective teamwork assessment from a collaborative learning environment (CLE). Collaborative learning may take place in peer-to-peer or in large groups, to discuss concepts, or find solutions to real-time problems or working on situational judgement task (SJT). Intelligent Tutoring Systems (ITSs) have been mostly used as a supportive system for the varied needs of individual learners. The ITCLP framework enables development of ITSs for team tutoring and facilitates collaborative problem solvi...