Critical decision method for eliciting knowledge

Ergonomics, 2015

An important part of the application of sociotechnical systems theory (STS) is the development of methods, tools and techniques to assess human factors and ergonomics workplace requirements. We focus in this paper on describing and evaluating current STS methods for workplace safety, as well as outlining a set of six case studies covering the application of these methods to a range of safety contexts. We also describe an evaluation of the methods in terms of ratings of their ability to address a set of theoretical and practical questions (e.g. the degree to which methods capture static/dynamic aspects of tasks and interactions between system levels). The outcomes from the evaluation highlight a set of gaps relating to the coverage and applicability of current methods for STS and safety (e.g. coverage of external influences on system functioning; method usability). The final sections of the paper describe a set of future challenges, as well as some practical suggestions for tackling ...

International Journal of Commerce and Management, 2009

A review of CIOs representing 11 large corporations (by Icelandic standards) in seven industries yielded a set of trends that have emerged as representative of current Icelandic computer-based initiatives. Among these trends: a desire to reduce labor costs is not the primary reason for outsourcing; project management and requirements determination skills are in need; the establishment of IT strategy is typically not driven by the board of directors; while establishing systems that satisfy international standards is sought by most companies, none are eager to pursue certification; outsourcing is generally restricted to the fewest possible time zones; the most commonly used country for outsourcing is the Baltic nations and India; several attempts to utilize Russian companies failed, usually due to a lack of communication skills on the part of the Russians; IT departments utilize outsourcing to secure sophisticated skills for one-time tasks, thereby permitting the company to maintain small IT staffs. Industry seeks business school graduates who can contribute IT strategy in the board room, create effective requirements documents, and possess a vision with respect to the application of technology to the competitive nature of business.

Qualitative Health Research, 2010

Although subject to considerable research from perspectives including general practitioners, patients, and perspective guidelines, chronic low back pain (CLBP) continues to be a common but contentious condition in primary care. We used medical consultation records, critical incident interviews, and a think-aloud problem-solving task to examine how general practitioners applied professional knowledge of the condition, especially in relation to psychosocial care. Using qualitative content analysis and thematic analysis, we identified a pragmatic, goal-focused approach to patients, a schema based on biomedical knowledge and tacit theories of motivation. The doctors' expectations for CLBP included uncertainty over symptoms and doubts over patient credibility, which helped to explain an autonomous rather than collaborative approach to managing back pain patients. The findings are discussed in light of social representations theory, self-determination, and research on the therapeutic relationship.

Educational Administration Quarterly, 2010

Purpose: This study examines learning, and both cognitive and behavioral change among a sample of randomly assigned urban principals, half of whom participated in a sustained, district-based professional development program (DPD). Research Methods: Latent class analyses of daily log data, qualitative typology development, and case studies of change provide a rich portrait of the learning and change process. Findings: Few dramatic transformations of practice. Instead, principals attributed to the DPD a gradual refinement of existing practice through a process that allowed them to "break down" declarative knowledge to better understand its consequences for their work, but also provided knowledge structures, tools, and routines for reintegrating ideas from the program into strategically valuable procedural

Perspectives on Medical Education, 2013

A central venous catheter (CVC) is an important medical tool used in critical care and emergent situations. Integral to proper care in many circumstances, insertion of a CVC introduces the risk of central line-associated blood stream infections and mechanical adverse events; proper training is important for safe CVC insertion. Cognitive task analysis (CTA) methods have been successfully implemented in the medical field to improve the training of postgraduate medical trainees, but can be very time-consuming to complete and require a significant time commitment from many subject matter experts (SMEs). Many medical procedures such as CVC insertion are linear processes with well-documented procedural steps. These linear procedures may not require a traditional CTA to gather the information necessary to create a training curriculum. Accordingly, a novel, streamlined CTA method designed primarily to collect cognitive cues for linear procedures was developed to be used by medical professionals with minimal CTA training. This new CTA methodology required fewer trained personnel, fewer interview sessions, and less time commitment from SMEs than a traditional CTA. Based on this study, a streamlined CTA methodology can be used to efficiently gather cognitive information on linear medical procedures for the creation of resident training curricula and procedural skills assessments.

Journal of Cognitive Engineering and Decision Making, 2010

Over the last 20 years, both naturalistic decision making and fast and frugal heuristics programs have radically broken with mainstream decision science, moving beyond the confines of artificial tasks and safe academic laboratories. We document commonalities of these programs and discuss ways in which a synthesis could contribute to a more relevant, precise, predictive, and effective decision science. We begin by reviewing the common roots and philosophies of the two programs, such as their respect for the capable decision maker and their acknowledgment of the importance of task ecology. We then identify four specific areas of synergetic potential, including ecological rationality and metacognition. Our review culminates in a case study of naturalistic heuristics based on a particular class of fast and frugal heuristics. These fast and frugal trees provide examples of effective, well-specified decision-making algorithms applied in a naturalistic domain: emergency medical diagnosis. By leveraging the strengths of each program, we point out some of the ways in which more sustainable progress can be fostered on issues that matter the most-for example, decisions that save and transform lives.

Risk Management, 2016

2010

Through a number of comprehensive reviews, the Institute of Medicine (IOM) has recommended that healthcare organizations develop safety cultures to align delivery system processes with the workforce requirements to improve patient outcomes. Until health systems can provide safer care environments, patients remain at risk for suboptimal care and adverse outcomes. Health science researchers have begun to explore how safety cultures might act as an essential system feature to improve organizational outcomes. Since safety cultures are established through modification in employee safety perspective and work behavior, human resource (HR) professionals need to contribute to this developing

Journal of Cognitive Engineering and Decision Making, 2013

The objective of this article is to improve our understanding of preferences in experienced-based choice. Positioned within the framework of naturalistic decision making, this article responds to the recent call to complement the examination of experience-based choice with studies of cognition in the "wild. " We document an exploratory field study that uses applied cognitive task analysis (ACTA) to examine financial day traders' preferences. Providing real-world examples, our study illustrates how day traders construct their understanding of gains relative to losses and emphasizes the relevance of prospect theory for understanding the asymmetry of human choice. The fourfold pattern of preferences as studied in the wild is risk seeking for medium-and highprobability gains, risk averse for small-probability gains, risk averse for small-probability losses, and risk averse for medium-and high-probability losses. Our results differ from the fourfold pattern of preferences exhibited by experience-based choice when studied in the laboratory. The implications of this work for prospect theory and the distinction between "experience through learning" and "experience through professional training" are discussed alongside the merits of the ACTA technique for professional expert domain-based knowledge elicitation.

2013

Cognitive Systems Engineering principles and methods guide analysis and understanding of complex work domains. Taking the CSE analytic findings to the next step, design, is often not clearly specified in process form. Some reasons for a weakly specified and documented CSE analysis-to-design translation process are:

Journal of Cognitive Engineering and Decision Making, 2014

The aim of this study was to describe the decision making of professional defensive football players according to the recognition-primed decision (RPD) model and to investigate relationships between their recognition processes and use of salient situational features in a naturalistic setting. Behavioral data were obtained from video recordings of four high-level football players in a top-level game, supplemented by verbal data collected during video-cued recall interviews, with the players watching the videotapes. Seven critical defensive stages in one game were studied. The data were analyzed using content analysis in relation to the RPD model in five steps. Considering the salient features the athletes used, 112 decisions were classified into three types of processes and eight typical decisions for defensive stages. These findings are discussed from three perspectives: the salient features the athletes used to assess the situation, the recognition processes of expert sport players' decision making, and future directions for research and decision-making training.

Ergonomics, 2015

Various countermeasures are used to mitigate signal passed at danger (SPAD) events on railways, yet they continue. While risk factors that destabilise cognitive processes have been identified, less has been published on the relationship between these factors and the informal strategies that drivers themselves adopt to reduce individual SPAD risk. This study aimed to address this gap and used a participative approach to collect and thematically analyse data from 28 drivers across eight rail organisations in Australia and New Zealand. The results showed not all formal countermeasures were considered effective, and identified several informal strategies. These aimed to reduce task disruption, service distortion and maintain connectedness to signals. While some evidenced redundancies in the task and cab, others did not reduce baseline risk. This paper explores the relationship between the established risks and identified strategies towards evaluating the utility of formal and informal mitigations. The research has application to the investigation of collision risk in all transport domains.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2014

Journal of Contingencies and Crisis Management, 2016

2017) Towards the classification of fireground cues: a qualitative analysis of expert reports. Journal of Contingencies and Crisis Management, 25 (4).

Ergonomics, 1992

Cognitive simulations are runnable computer programs that represent models of human cognitive activities. We show how one cognitive simulation built as a model of some of the cognitive processes involved in dynamic fault management can be used in conjunction with small-scale empirical data on human performance to uncover the cognitive demands of a task, to identify where intention errors are likely to occur, and to point to improvements in the person-machine system. The simulation, called Cognitive Environment Simulation or CES, has been exercised on several nuclear power plant accident scenarios. Here we report one case to illustrate how a cognitive simulation tool such as CES can be used to clarify the cognitive demands of a problem-solving situation as part of a cognitive task analysis.

BMC Medical Education

Background: Handoff education is both formal and informal and varies widely across medical school and residency training programs. Despite many efforts to improve clinical handoffs, little evidence has shown meaningful improvement. The objective of this study was to identify residents' perspectives and develop a deeper understanding on the necessary training to conduct safe and effective patient handoffs. Methods: A qualitative study focused on the analysis of cognitive task interviews targeting end-of-shift handoff experiences with 35 residents from three geographically dispersed VA facilities. The interview data were analyzed using an iterative, consensus-based team approach. Researchers discussed and agreed on code definitions and corresponding case examples. Grounded theory was used to analyze the transcripts. Results: Although some residents report receiving formal training in conducting handoffs (e.g., medical school coursework, resident boot camp/workshops, and handoff debriefing), many residents reported that they were only partially prepared for enacting them as interns. Experiential, practice-based learning (i.e., giving handoffs, covering night shift to match common issues to handoff content) was identified as the most suited and beneficial for delivering effective handoff training. Six skills were described as critical to learning effective handoffs: identifying pertinent information, providing anticipatory guidance, applying acquired clinical knowledge, being concise, incorporating delivery strategies, and appreciating the styles/preferences of handoff recipients. Conclusions: Residents identified the immersive performance and the experience of covering night shifts as the most important aspects of learning to execute effective handoffs. Formal education alone can miss the critical role of real-time sense-making throughout the process of handing off from one trainee to another. Interventions targeting senior resident mentoring and night shift could positively influence the cognitive and performance capacity for safe, effective handoffs.

The International Journal of Aviation Psychology, 2003

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2001

Incident reports are intended to help identify problems in safety critical systems and aid in preventing subsequent accidents, thus improving safety. Unfortunately, incident-reporting forms are generally unstructured, providing little guidance to the reporter on how to describe the critical events of an incident. As a result, most reports contain information on what happened, as opposed to why an incident happened, and hence make the identification of intervention and prevention strategies onerous. The purpose of the present study was to help remedy this situation by developing and testing a tool, coined the Critical Event Recall Tool (CERT) for improving the quality of information contained in narrative reports of incidents. Results of this study indicate that CERT improved the analytical content of general aviation pilots' reports of a critical incident that occurred during a simulated crosscountry flight and that pilots who used CERT generally felt that the tool was easy to use and understand.

Journal of Ambient Intelligence and Smart Environments

This article introduces user experience research that has been carried out by evaluating a video-illustrated science fiction prototype with process control workers. Essentially, the prototype 'A remote operator's day in a future control center in 2025' was aimed at discovering opportunities for new interaction methods and ambient intelligence for the factories of the future. The theoretical objective was to carry out experience design research, which was based on explicit ambient user experience goals in the nominated industrial work context. This article describes the complete creative prototyping process, starting from the initial user research that included evaluations of current work practices, technological trend studies and co-design workshops, and concluding with user research that assessed the final design outcome, the science fiction prototype. The main contribution of the article is on the ambient user experience goals, the creation process of the video-illustrated science fiction prototype, and on the reflection of how the experience-driven prototype was evaluated in two research setups: as video sequences embedded in a Web survey, and as interviews carried out with expert process control workers. For the science fiction prototyping process, the contribution demonstrates how the method may employ video-illustration as a means for future-oriented user experience research, and how complementary user-centered methods may be used to validate the results.

Journal of Nursing Education, 2014

Advancing the science of nursing education will require the discipline to conduct research that investigates complex phenomena, such as students' clinical thinking and decision-making skills, using multiple methods. The research methods developed in other disciplines can provide nursing education researchers with new ways to investigate clinical teaching and learning in nursing. The critical decision method (CDM), derived from psychology and human factors engineering, is a technique by which researchers elicit experts' thinking and the cognitive work informing decision making in the context of practice. This article describes how the CDM was adapted to study nursing students' situation awareness, cues for action, and pattern recognition during clinical experiences. The CDM is a promising method for investigators to use to conduct research in nursing education and to inform the design of clinical experiences to promote these critical abilities. [

Military Psychology

This thesis is dedicated to my family and friends who supported me through graduate school (special thanks to William Brock Jr. for your unwavering support). This thesis is also dedicated to all of the members of the US Armed Forces that spend their days protecting the rights and freedoms of the American people. One member, specifically, made this thesis possible. Thank you. v ACKNOWLEDGMENTS I would like to acknowledge my friends, family, and the wonderful people in the ODU Psychology Department that contributed to my academic voyage. I would like to thank my advisor, Dr. Jim Bliss, for his support and guidance during the most difficult journey in my life (thus far). Special thanks to committee members Dr. Chris Brill and Dr. Lau-Baracco for their valuable insight and advice that has greatly improved the quality of the work presented here. Thank you to members of the R.E.A.C.T.S. lab for your unwavering support and all of your help; your friendship will never be forgotten. vi

Simulation & Gaming, 2009

Simulation-based training (SBT) is a methodology for providing systematic and structured learning experiences. The effectinvess of this methodology is dependent on the quality of performance measurement practices in place. Performance measurement during SBT must be diagnosed; that is, the causes of effective and ineffective performance must be determined. This diagonstic measurement drives the systematic decisions concerning corrective feedback and remediation. The purpose of this article is to provide a state of the science review of human performance measurement systems in SBT. To this end, three specific goals are addressed. First, a review of the theoretical foundations being used to drive performance measurement systems in SBT is provided. Second, an overview of the methodologies and approaches to measurement in SBT is provided. Third, a set of best practices for designing performance measurement systems for use in SBT are provided. These best practices are based on the scientific and practice-based literatures.

Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare. International Symposium of Human Factors and Ergonomics in Healthcare, 2018

Critical Decision Method (CDM), a popular cognitive task analysis (CTA) method, is an in-depth retrospective interview that uses a historical non-routine incident to identify experts' decision-making factors in complex socio-technical settings with high consequences for failure. However, it is challenging to use CDM to make comparisons, including those between experts and trainees. We describe an alternative CTA method used to study physicians' decision making for ordering diagnostic imaging. After being primed with 11 simulated patient scenarios, nine attending and 11 resident physicians were asked to map out and present their decision-making process with a bullseye participatory design toolkit. Interviews were analyzed qualitatively, revealing four common decision factors: diagnostic efficacy, patient safety, organizational constraints, and patient comfort. The bullseye maps were used to quantitatively measure priority differences between these decision factors. Attending ...

Proceedings of the Human Factors and Ergonomics Society Annual Meeting

The Decision Ladder (DL) Template is often used to understand workers’ reasoning processes. In high-hazard industries such as oil refining, frontline workers’ decisions are crucial to managing abnormal situations and preventing unwanted events. This paper describes research undertaken–using the DL Template–to analyze workers’ responses to abnormal situations as elicited during Critical Decision Method interviews. The paper firstly describes the evolution DL Template since Rasmussen’s original writings. It then describes challenges encountered when using the traditional DL Template to analyze interview responses. A modified DL Template is proposed that provides a more detailed top portion of the template to allow more explicit analysis of how knowledge-based reasoning might be used when addressing abnormal, safety-critical situations.

Journal of Engineering Education

Background: Capstone design courses represent a critical juncture in students' development at the transition from school to work. However, few studies have systematically explored teaching in this context, leaving a significant gap in our ability to concretely describe faculty practices in ways that support subsequent explorations of the relationships between teaching practices and learning outcomes. Purpose/Hypothesis: The aim of this study was to develop a comprehensive description of the pedagogical practices used by capstone design faculty from a functional perspective and provide researchers with a framework for subsequent work. Design/Method: This study used qualitative methods to analyze interviews with 42 capstone faculty; the participants represent a stratified purposeful sample of respondents to a national survey. Analysis focused on descriptive coding, beginning with a priori codes, to define broad functions, supplemented with emergent coding to identify concrete practices used in the capstone context. Results: The study resulted in a model of capstone design teaching that includes nine functions (challenge, protect, coach, promote employability, provide exposure, provide role models, accept and confirm, counsel, and build rapport) and 28 associated practices. Conclusions: Capstone faculty use a range of practices designed not only to coach students through the engineering design process but also to more broadly prepare students for workplace practice and build their identity as engineering professionals.

Journal of the Association for Information Science and Technology, 2014

To see the final version of this work please visit the publisher's website. Access to the published online version may require a subscription.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting

With an increasing frequency of data breaches suffered by organizations, computer network defense (CND) is becoming an increasingly important concern. With understanding of how cybersecurity professionals engage in the cognitive aspects of their work, human factors researchers and practitioners can improve tools and training. By optimizing the tools and training network defenders rely on to detect and respond to novel network threats, the cybersecurity workforce will be strengthened. While cognitive task analysis (CTA) is well-positioned to represent the cognitive work of CND, we identify challenges practitioners are likely to encounter in the field. Through a review of published CTAs in CND and other domains, we provide guidance for future CTA efforts in CND. Finally, we present an argument for the use of Elicitation by Critiquing (EBC) and demonstrate its efficacy in mitigating the challenges of applying CTA in CND.

Human Factors and Ergonomics in Manufacturing & Service Industries

This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.

Frontiers in Pediatrics

Approximately 10% of newborn infants require resuscitative intervention at birth. Ideally, this care is provided by a team of expert healthcare professionals who possess exceptional cognitive, psychomotor, and communication skills. Human errors and deviations from resuscitation protocol are common and may be attributable to excessive cognitive demand experienced by the resuscitation team. Cognitive Task Analysis (CTA) is a group of methods used to assess knowledge, judgments, goals, and decisionmaking of expert healthcare professionals. These methods may be used during neonatal resuscitation to gain an improved understanding of the approaches used by healthcare professionals. CTA methods have been applied in many medical disciplines including neonatology. CTA has been used to identify information previously confined to the intuition of experts. This information has been used to assess, develop, and improve medical technology, clinical decision support tools (DSTs), communication structure, and training methods. Knowledge attained through CTA might be applied similarly to neonatal resuscitation, which may in turn decrease human errors, and improve patient safety.

2021

Many of the obstacles to effective delivery of rail projects (in terms of cost, time and quality) can be traced back to poor collaboration across complex design teams and supply chains. As in any infrastructure delivery process, it is important to make decisions collaboratively at an early design stage. Advanced systems such as Building Information Modelling (BIM) and Geographic Information Systems (GIS) can facilitate collaboration during the decision-making process and boost work efficiencies. Such potential benefits are not realised because the roles of BIM and GIS in facilitating collaboration are not clearly understood or articulated. This paper aims to identify and articulate collaboration requirements during the design stage of rail projects. To achieve this, a mixed-method approach was employed to examine the issues that hinder collaboration in rail projects. An online questionnaire was designed to assess the state-of-art in BIM and GIS, followed by fifteen follow-up face to...

International Journal of Energy Sector Management, 2014

Purpose – The purpose of this research is to develop a building information modelling (BIM)-enabled sustainable design process model that identifies critical decisions actions in the design process along with the information and level of detail that facilitate an informed and timely decision. Building performance analysis is usually performed after the design and construction documents are produced, resulting in lost opportunities. Design/methodology/approach – A number of research methods have been adopted; these include extensive literature review and 11 in-depth exploratory interviews with industry practitioners (sustainable building design experts, early BIM adopters). Findings – Project delivery methods have a significant effect on the sustainable outcome of buildings. The development of a structured process can assist sustainable design practice among building professionals. Learning from implemented projects, that have utilised BIM processes, facilitates the scope of creating...

International Journal of Emergency Services, 2014

Purpose – Experienced fire ground commanders are known to make decisions in time-pressured and dynamic environments. The purpose of this paper is to report some of the tacit knowledge and skills expert firefighters use in performing complex fire ground tasks. Design/methodology/approach – This study utilized a structured knowledge elicitation tool, known as the critical decision method (CDM), to elicit expert knowledge. Totally, 17 experienced firefighters were interviewed in-depth using a semi-structured CDM interview protocol. The CDM protocol was analysed using the emergent themes analysis approach. Findings – Findings from the CDM protocol reveal both the salient cues sought, which the authors termed critical cue inventory (CCI), and the goals pursued by the fire ground commanders at each decision point. The CCI is categorized into five classes based on the type of information each cue generates to the incident commanders. Practical implications – Since the CDM is a useful tool ...

Drug Safety, 2011

This narrative review includes a summary of research examining prescribing errors, prescription decision making and the role computerized decision support plays in this decision-making process. A reduction in medication prescribing errors, specifically a reduction in the selection of inappropriate medications, is expected to result from the implementation of an effective computerized decision support system. Previous research has investigated the impact of the implementation of electronic systems on medication errors more broadly. This review examines the specific characteristics of decision support systems that may contribute to fewer knowledge-based mistakes in prescribing, and critically appraises the large volume of information available on the decision-making process of selecting medicines for prescription. The results highlight a need for work investigating what decision strategies are used by doctors with different levels of expertise in the prescribing of medications. The nature of the relationship between decision support and decision performance is not well

Cognition, Technology & Work, 2018

This paper reports the findings of a series of interviews with search and rescue volunteers. Participants were asked to recall accounts of particular incidents which involved searching for a missing adult who could be considered 'vulnerable'. The purpose of this study was to discover what types of decisions are made during missing incidents; including a consideration of the factors which affect these decisions and the main focuses of attention throughout the incident. Such an understanding may help to shed light on best practices which could inform decision-making support tools for families of the missing and identify the user-requirements of a future technology designed to help find missing people. Interviews were conducted using the critical decision method (CDM) to elicit specific information about the decisions and challenges faced by search and rescue teams during missing person searches. Critical decision points were identified and sequenced for each incident. Emergent thematic analysis (EMA) was applied to the transcripts to identify themes across various incidents; these themes were explored in detail using a mixed-method approach. This study builds upon the methodological approach of CDM using a two-tiered approach to analysis which seeks to discover the focus of practitioners' attention as they progress through missing person searches. A decision-sequence diagram was created to clearly show the sequence of each decision and trends across all incidents; a table was produced to show the relative importance of each aspect across decisions. Finally, strengths and weaknesses of this approach to incident analysis are discussed.

Cognition, Technology & Work, 2019

More than 20 years ago, Woods proposed a model that accounts for the inherent complexity faced by operators when managing abnormal and emergency situations in highly complex sociotechnical systems. The model was reviewed a decade later and only a few studies have applied it to aviation. This paper proposes adjustments to the original model, based on recent theoretical developments and empirical evidence on the anomaly management activity in aviation. The model was divided into five components; three of which-activity, types of reasoning involved, and resources-were revisited and further developed. The two other components-fault behaviour and unit of analysis-were not updated and only dicussed in the aviation context. As a result, the revisited model descriptively clarifies how the activity of anomaly management emerges from the use of a wide repertoire of strategies, involving a spectrum of types of reasoning and a set of resources for action, which are not limited to those anticipated by designers, such as checklsits and the warning system. An instantiation of the revisited model highligths the implications of false alarms, which trigger a cascade of disturbances that, in turn, requires adaptive strategies based on heuristics and analogies and supported by pilot's experience. The revisited model can support a more accurate analysis of anomalous situations and the redesign of work systems to achieve a better performance.

Cognition, Technology & Work, 2012

It is estimated that 10-15 % of UK hospital inpatients have diabetes. Poor glycemic control is a care quality problem that has been linked to organizational factors such as inadequate training, inadequate protocols, problems with communication and teamwork, and difficulty coordinating mealtimes. Interventions using specialist diabetes teams have been effective in addressing some of these problems and have led to increased staff and patient self-efficacy and reduced length of stay. The aim of this study was to investigate how inpatient diabetes care is delivered and how resilience is created and/or breaks down, and to identify the implications for quality improvement. In-depth interviews (n = 32) with diabetes specialist and non-specialist staff were conducted in an acute medical admissions environment in an 850-bed teaching hospital. The Critical Decision Method, a content-orientated knowledge elicitation technique, was adapted to guide interview schedules, which explored key decisions, gaps and discontinuities in care and strategies for work system improvement. Care is delivered through the coordination of a multilayered team of different professionals. Specialists provide expertise and problem solving through case-based reasoning using problem-solving skills acquired through past experience. Ward staff focus on processes and immediate patient needs and are more reliant on decision protocols. Gaps in care can occur and result in delays in referring to specialists and clinical inertia. Specialists are a key source of resilience in the system and bridge gaps by acting reactively to problems, proactively monitoring and anticipating problems, providing staff education, and patient support and education. Opportunities for supporting clinicians to bridge gaps in care were identified.

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2019

For conversational agents to provide benefit to intelligence analysis they need to be able to recognise and respond to the analysts intentions. Furthermore, they must provide transparency to their algorithms and be able to adapt to new situations and lines of inquiry. We present a preliminary analysis as a first step towards developing conversational agents for intelligence analysis: that of understanding and modeling analyst intentions so they can be recognised by conversational agents. We describe in-depth interviews conducted with experienced intelligence analysts and implications for designing conversational agent intentions.

Theoretical Issues in Ergonomics Science, 2009

Given a specific training problem, how much is known about choosing the best mix of technologies and methods that are both cost effective and that also meet given requirements? Beyond technical specifications of systems and components, a theoretically thorough method is still not used for identifying training requirements and how to best fit technology to meet those requirements. What is needed is a way to 'bound' training systems so that it can be clearly stated what they are and are not best used for. What are their limits? What are their strengths? The method described in this paper is based on a well-developed taxonomy of human performance. It is linked to conventional task analysis techniques to show how to identify what tasks a training system can be used for. An example is used to illustrate the method, but it has been applied to many training simulators. The process is meant to objectively link training requirements to technologies in a repeatable fashion such that training system builders, buyers and users can better understand the limits of their system.

2021

Resilience allows teams to function at their optimal capacity and skill level in times of uncertainty. The SARS-CoV-2 (COVID-19) pandemic created a perfect opportunity to study resilience culture during a time of limited healthcare team experience, protocols, and specific personal protective equipment (PPE) needed. Little is known about healthcare team resilience as a phenomenon; existing definitions and empiric referents do not capture the nature of healthcare team resilience, as the traditional focus has been placed on individual resilience. This qualitative research protocol provides the rationale and methodology to examine this phenomenon and builds a bridge between resilience engineering and individual resilience. The sample is composed of healthcare team members from the US. This research may add to the body of knowledge on resilience culture in healthcare teams during the COVID-19 pandemic. This qualitative research protocol paper outlines the rationale, objective, methods, a...

Journal of Cognitive Engineering and Decision Making, 2019

Resilience engineering (RE) has ushered new approaches to learning about work in complex sociotechnical systems. In terms of improving safety, RE marks a shift from the traditional approach of retrospectively investigating adverse events, toward learning proactively about patterns in everyday work, including how things go well. This study applied the RE framework to the health care domain, by developing and implementing a new knowledge-elicitation protocol to learn about how frontline care providers achieve safe and effective patient care in their everyday work. Eighteen participants, including physicians, nurses, residents, and clinical leaders from a range of specialties, were interviewed using the new protocol. Qualitative analysis of the data revealed multiple themes and patterns which underlie resilient functioning of individuals, teams, and the organization as a whole. Further, a Resilience Mapping Framework (RMF) was developed based on major thematic categories to systematica...

Architectural Engineering and Design Management, 2016

Environmental sustainability considerations are often treated as an add-on to building design, following ad hoc processes for their implementation. As a result, the most common problem to achieve a sustainable building outcome is the absence of the right information at the right time to make critical decisions. For design team members to appreciate the requirements of multidisciplinary collaboration, there is a need for transparency and a shared understanding of the process. This research presents the findings from 25 in-depth interviews with industry practitioners concerning 10 case studies of buildings, which achieved high sustainability certification ratings (e.g. BREEAM, Passivhaus, Part L), to identify best practices in sustainable building design (SBD). The results identify the key players' roles and responsibilities, tasks, deliverables and critical decision points for SBD. These components have been coordinated explicitly in a systematic process that utilises Information Communication Technology (ICT), Building Information Modelling (BIM), and Building Performance Analysis (BPA) software to realise the benefits of combining distributed teams' expertise.

Built Environment Project and Asset Management, 2020

PurposeSustainability considerations are often treated as an add-on to building design, following ad-hoc processes for their implementation. The purpose of this study was to investigate, model and facilitate the early stages of building information modelling (BIM) enabled sustainable building design (SBD) by formalising the ad-hoc working relationships of the best practices in order to standardise the optimal collaboration workflows.Design/methodology/approachFour stages of data collection were conducted, including a total of 32 semi-structured interviews with industry experts from 17 organisations. Fourteen “best practice” case studies were identified, and roles and responsibilities, resources, information exchanges, interdependencies, timing and sequence of events and critical decisions were examined.FindingsThe research classified the critical components of SBD into a framework utilising content and thematic analyses. These components were coordinated explicitly into a systematic...

Journal of Cognitive Engineering and Decision Making, 2017

Anesthesia takes place in a complex, high-stakes environment where humans and technologies interact to provide medical care to patients. Providing adequate decision support for individuals and teams in anesthesia emergencies is important because emergencies are infrequent and complex. Current designs of decision support in airway management are not context specific and lack a consideration of how decisions are made under time pressure. To fill this gap, this study used frameworks from cognitive systems engineering to explore decisions of experienced anesthesia providers. The goal was to identify the decision pathways used in challenging airway management situations. The critical decision method was employed to interview anesthesia providers about a challenging incident they had experienced. Results illustrated that decisions were based on prior experience and made through a process of recognition. The vast majority of decisions were recognition primed, characterized by a direct link between cue familiarity and action generation. A few decisions involved concurrent option comparison, which was still based on situation recognition. Different cues received through teamwork, technologies, and patients contributed to the decisions. As different cognitive pathways may require different design solutions, the findings of this study are being used to help develop decision support tools in anesthesia.

Technical Communication Quarterly, 2019

Journal of Cognitive Engineering and Decision Making, 2015

Ensuring that unmanned aerial systems’ (UAS) control stations include a tight coupling of systems engineering with human factors, cognitive analysis, and design is key to their success. We describe a combined cognitive task analysis (CTA) and design thinking effort to develop interfaces for an operator controlling an autonomous helicopter, a prototype system that the Office of Naval Research is developing. We first conducted CTA interviews with subject-matter experts having expertise in UAS flight operations, helicopter resupply, military ground forces, and marine airspace control. Data informed the development of analysis products, including human–system interface requirements, which drove the creation of design concepts through ideation sessions using design thinking methods. We validated and refined the design concepts with UAS pilots. We provide an overview of our process, illustrated by details of a timeline display development. Significant aspects of our work include close int...

Human Factors: The Journal of the Human Factors and Ergonomics Society, 2004

Recent “naturalistic” theories of decision making emphasize the role of stored prior experiences or cases as a guide to current action. However, there is little empirical evidence on the role that case-based remindings play in real-life decision making. The present study utilized a Web-based survey to collect data about the role of prior cases in pilot decision making about critical flight events. Results showed that more than half of the 1081 pilots who responded could provide details about utilizing a previous case in responding to a critical flight event. These events were most likely to involve weather or equipment failure. The cases were found to be useful in situational assessment rather than option evaluation. The use of cases increased with age and experience. Data obtained from a concurrent conventional survey showed broadly similar results. The implications of these results are that case-based remindings play an important role in expert pilot decision making and that a tra...

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2016

In the U.S. Navy’s proposed Live-Virtual-Constructive (LVC) air combat training system, live F/A-18 aircraft will fly alongside virtual aircraft flown by pilots in simulators and constructive aircraft generated by computers. The Navy is using a human-centered, gradual, iterative, and research-based strategy to transition to LVC training. Part of this strategy entails a multi-year effort to preemptively identify and mitigate potential safety concerns associated with LVC training. Two cycles of event-driven interviews with 31 Navy aircrew and training professionals, followed by extensive review by two naval air combat subject-matter experts and other stakeholders, produced a list of LVC-related safety concerns. Researchers assessed the safety risk level of each concern to prioritize mitigation efforts. The latest cycle of data collection and subject-matter expert review, reported herein, focused on developing mitigations for the safety concerns and identifying characteristics of the c...

Applied Ergonomics, 2021

A new technique for the assessment of Team Situation Awareness (TSA) accuracy based upon post task Critical Decision Method structured interviews was developed and tested using 39 combat-ready F/A-18 pilots. Pilots undertook a number of simulated air combat scenarios, flying in flights of four aircraft against a formation of enemy aircraft. Results showed a strong curvilinear relationship where high TSA accuracy resulted in higher performance in some areas of air combat, measured with friendly losses and kills. There were diminishing returns in performance as TSA accuracy increased. This may explain why previous studies on air combat have found relatively weak relationships between situation awareness and performance where the relationship has been assumed to be linear.

Advances in Informatics and Computing in Civil and Construction Engineering, 2018

It is becoming increasingly clear that there is a gap between the expected and actual performance of buildings. A growing body of evidence suggests that the most common barrier to achieving design intent is the absence of comprehensive information during design and construction stages, leading to poor decision-making which impacts on performance and Whole Life Cost (WLC). Building Information Modelling (BIM) has the potential to facilitate a more comprehensive and accurate design approach from the early stages. A detailed and accurate model can allow designers and clients to understand the wider impacts of design changes, and to track this information through construction stages. However, dependencies between design decisions and WLC have yet to be understood. This paper is based on a project that focuses on the Private Rental Sector (PRS), which is the fastest growing new sector in the UK housing market, also known as Build-to-Rent (BTR). The study adopts a mixed method approach for the development and validation of a structured standardised process for more accurate WLC estimation through BIM. As a result, the main problems in WLC BIM management are identified, and coordinated into a reverse-engineered systematic process that uses the Integrated DEFinition (IDEF) 3 structured diagramming modelling technique, and the Industry Foundation Classes (IFC) as a basis for large dataset management. The research outputs aim to enhance BIM lifecycle management through a smart decision-making approach that is integral to the natural design development process.

Journal of Cognitive Engineering and Decision Making, 2011

Although insight is often invoked as a phenomenon of problem solving and innovation, it has rarely been studied in a naturalistic fashion. The purpose of the study reported here was to learn more about insights as they occur in field settings as opposed to controlled laboratory conditions. The authors collected a set of 120 examples of insight taken from cognitive task analysis interviews, media accounts, and other sources and coded each incident using a set of 14 features. The results generated a descriptive model of insight that is different from the findings that emerge from research with puzzle problems. It posits multiple pathways for gaining insights. One pathway is triggered by detecting a contradiction. A second pathway is triggered by a need to break through an impasse. The third pathway gets triggered by seeing a connection.

Journal of Cognitive Engineering and Decision Making

Operations Control Centres (OCCs) are critical structures used by airlines to oversee the execution of all planned flights, managing punctuality, regularity and customer support. In this study, we investigated the decision-making during flight disruptions inside an OCC from the naturalistic decision-making perspective. We conducted a mini-ethnography case study in a major South American airline, focussing on how functions critical to the flight disruption management cope with variability. Data collection included document analyses, field notes, direct observations and interviews. The functional description of work-as-done revealed how the OCC constantly and actively looks for signs of disruption while monitoring the normal operation and rebalancing resources. The decision-making process is distributed and decentralised across multiple functions, where experts from each function rely on a repertoire of strategies to deploy innovative solutions to dynamic scenarios. Five different mec...

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

In this paper, we present the hypothesis that system transparency is critical for tasks that involve expert sensemaking. Artificial Intelligence (AI) systems can aid criminal intelligence analysts, however, they are typically opaque, obscuring the underlying processes that inform outputs, and this has implications for sensemaking. We report on an initial study with 10 intelligence analysts who performed a realistic investigation exercise using the Pan natural language system [10, 11], in which only half were provided with system transparency. Differences between conditions are analysed and the results demonstrate that transparency improved the ability of analysts to reason about the data and form hypotheses. CCS CONCEPTS • Human-centered computing → Interactive systems and tools; User studies.

International Journal of Ethics Education

Paramedics encounter ethical dilemmas at work, and while previous research has improved ethics education and practice, more can be learned from the lived experience of paramedics facing ethical challenges. This paper explores the lived experience of two paramedics, one with five years’ experience, the other twenty, presented with comparable cases. The participants, who were interviewed in a broader qualitative study exploring practical wisdom, were asked to select a case involving ethical decision-making from their own practice experience. Semi-structured interviews employed the five-step Critical Decision Method as a framework to explore various aspects of the decision-making process. The two cases were identified as exemplars of prominent themes suitable for discussion. The findings highlighted several factors that influence paramedic ethical decision-making, including practical wisdom, communication and teamwork, clinical support, and education. This case comparison identifies a ...

BMC Health Services Research, 2014

Background: The utilisation of good design practices in the development of complex health services is essential to improving quality. Healthcare organisations, however, are often seriously out of step with modern design thinking and practice. As a starting point to encourage the uptake of good design practices, it is important to understand the context of their intended use. This study aims to do that by articulating current health service development practices. Methods: Eleven service development projects carried out in a large mental health service were investigated through in-depth interviews with six operation managers. The critical decision method in conjunction with diagrammatic elicitation was used to capture descriptions of these projects. Stage-gate design models were then formed to visually articulate, classify and characterise different service development practices. Results: Projects were grouped into three categories according to design process patterns: new service introduction and service integration; service improvement; service closure. Three common design stages: problem exploration, idea generation and solution evaluation-were then compared across the design process patterns. Consistent across projects were a top-down, policy-driven approach to exploration, underexploited idea generation and implementation-based evaluation. Conclusions: This study provides insight into where and how good design practices can contribute to the improvement of current service development practices. Specifically, the following suggestions for future service development practices are made: genuine user needs analysis for exploration; divergent thinking and innovative culture for idea generation; and fail-safe evaluation prior to implementation. Better training for managers through partnership working with design experts and researchers could be beneficial.

Proceedings of the Human Factors and Ergonomics Society ... Annual Meeting, 2008

Careful consideration must be given to the implementation of automation into complex systems. Much research in adaptive automation has identified challenges for system implementation. A key focus of this research has surrounded the methods of automation invocation including critical events, measurement, and modeling techniques. However, little consideration has been given to selecting and implementing appropriate techniques for a given system as a guide to designers of adaptive automation. This paper proposes such a methodology. We demonstrate the use of this methodology by describing a case study about a system designed to support effective communication and collaboration between the commander and vehicle operator in an unmanned aerial vehicle (UAV) system.

Behavioral Sciences

In this position paper, we start by identifying the issues inherent to coach development; we then consider the current status of coach development and present our position before concluding with key points and suggesting resolutions for the issues. Our intention is to propose the progression of appropriate practices and approaches for the professional development and preparation of coaches. In coach development, a lack of clarity exists at both organisational and individual levels, particularly around the role of and aims for coach developers. Organisationally, we consider a radical reframing required to progress the profession of coach development. We also suggest that many individuals currently involved in coach development do not possess the requisite knowledge to move the field forward. Our aspirations for coach development include recognising the need for expertise and what it looks like in practice. Coaching and coach development interactions should examine particular coaching...

Journal of Human Kinetics, 2019

Goalkeepers hold a key position for success in team sports competitions. They perform in dynamical contexts and are highly submitted to time pressure. The purpose of this naturalistic case study, therefore, was to explore how a handball expert goalkeeper deals with the uncertainty of the competition settings to make successful decisions. An individual self-confrontation interview was held with a goalkeeper while he watched duels with potential throwers in an official competition. A mixed method was used combining the first-person and third-person point of view. Verbal data were supplemented by observational data (distance measures between the goalkeeper and the potential thrower) in 83 short accounts of decision-making situations. Qualitative analysis resulted in 419 units of salient features, in three types of processes related to the Recognition-Primed Decision model, and in four micro-decisions. Non-parametrical statistical analysis indicated that there was a significant effect o...

Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2001

Cognitive task analysis (CTA) methodologies are used to discover expertise that domain practitioners utilize in order to perform tasks, but are unable to reliably articulate when asked directly. We describe a methodology that walks through a novice's process on a task in order to serve as a probe for elicitation of expert commentary. Six experienced intelligence analysts with three levels of prior knowledge critiqued the process a junior analyst employed to determine the causes and consequences of the Ariane 501 rocket launch failure. Preliminary data analyses reveal interesting insights about the nature of expertise in intelligence analysis. This appears to be a promising alternative methodology for CTA when there are access restrictions, low frequency or unpredictability of target observation events, or difficulty in recruiting experts who wish to be evaluated on their task performance.

Human Factors: The Journal of the Human Factors and Ergonomics Society, 2021

Objective To understand how firefighters’ use of rules (i.e., standard operating procedures [SOPs]) and deliberative decision making (i.e., operational discretion [OD]) interacts with acute stress. Background Current operational guidance for UK firefighters combines the provision of SOPs, for routine incidents, with the use of OD, under prescribed conditions (e.g., when there is a risk to human life). However, our understanding of the use of SOPs and OD is limited. Methods Incident commanders (ICs; n = 43) responded to simulated emergency incidents, which either licensed the use of OD or required use of a SOP. Video footage of IC behavior was used to code their response as involving a SOP or OD, while levels of acute stress were assessed using a blood-based measure and self-report. Results ICs were less likely to use OD selectively in the simulated emergency incident that licensed its use than in the one for which use of an SOP was appropriate; IC command level did not affect this p...

Emergency Medicine Journal

BackgroundBlood transfusion for bleeding trauma patients is a promising pre-hospital intervention with potential to improve outcomes. However, it is not yet clear which patients may benefit from pre-hospital transfusions. The aim of this study was to enhance our understanding of how experienced pre-hospital clinicians make decisions regarding patient blood loss and the need for transfusion, and explore the factors that influence clinical decision-making.MethodsPre-hospital physicians, from two air ambulance sites in the south of England, were interviewed between December 2018 and January 2019. Participants were involved in teaching or publishing on the management of bleeding trauma patients and had at least 5 years of continuous and contemporary practice at consultant level. Interviews were semi-structured and explored how decisions were made and what made decisions difficult. A qualitative description approach was used with inductive thematic analysis to identify themes and subthem...

Handbook of Research on Information Management and the Global Landscape

Frontiers in sports and active living, 2023

There is growing recognition of the value of "in situ" coach development practice across a variety sporting contexts. Unfortunately, however, there remains a limited number of tools available with which to observe coaching practice. In this study, we pilot and test a quasi-systematic tool for observation in the form of the 3Ps. Drawing on a range of representational perspectives, the theoretically neutral labels of "procedure", "planning", and "process" were developed for the purpose of holistic observation. In order to test the tool, a group of experienced coach development practitioners (n = 10) integrated the tool into their practice over a 12-month programme of professional development. Those participants subsequently took part in semi-structured interviews, in which they expressed a strong sense of acceptability, perceiving effectiveness and positive opportunity cost. We propose that the 3Ps tool presents a holistic and practically useful means of observing coaches' professional judgment and decision making. We also suggest future directions for the researcher who seeks to generate evidence in a naturalistic coaching context.

TransNav, the International Journal on Marine Navigation and Safety of Sea Transportation

Icebreakers are a critical shared resource between Finland and Sweden required to keep the Baltic Sea clear for differing waterborne activities throughout the winter season. The safety and efficiency (both ecological and in terms of time) of the winter transport system is highly dependent on the decision-making process followed by the icebreakers. The captains in charge of the icebreakers must decide the priority of assistance among all the merchant vessels that may be in need within a given time window. While captains successfully do this every year, with experience, this decision-making becomes second nature and a transparent picture of how the decisions are made is often missing. It is not always clear what salient features captains pay attention to, and how they use those features in their reasoning process to reach a decision during operations. This paper presents a pilot study that uses cognitive task analysis (CTA) to outline captains' decision-making process for ice breaker assistance allocation. In-depth interviews of three subject matter experts were conducted using a naturalistic icebreaker scenario. Results include identified critical decision points, identified, and prioritized salient features, and characterized icebreaker assistance strategies.