Synthesizing and Reporting Milestones-Based Learner Analytics

Academic medicine : journal of the Association of American Medical Colleges, 2015

Competency-based education, including assessment of specialty-specific milestones, has become the dominant medical education paradigm; however, how to determine baseline competency of entering interns is unclear-as is to whom this responsibility falls. Medical schools should take responsibility for providing residency programs with accurate, competency-based assessments of their graduates. A University of Michigan ad hoc committee developed (spring 2013) a post-Match, milestone-based medical student performance evaluation for seven students matched into emergency medicine (EM) residencies. The committee determined EM milestone levels for each student based on assessments from the EM clerkship, end-of-third-year multistation standardized patient exam, EM boot camp elective, and other medical school data. In this feasibility study, the committee assessed nearly all 23 EM milestones for all seven graduates, shared these performance evaluations with the program director (PD) where each ...

Medical teacher, 2017

Competency-based assessment seeks to align measures of performance directly with desired learning outcomes based upon the needs of patients and the healthcare system. Recognizing that assessment methods profoundly influence student motivation and effort, it is critical to measure all desired aspects of performance throughout an individual's medical training. The Accreditation Council for Graduate Medical Education (ACGME) defined domains of competency for residency; the subsequent Milestones Project seeks to describe each learner's progress toward competence within each domain. Because the various clinical disciplines defined unique competencies and milestones within each domain, it is difficult for undergraduate medical education to adopt existing GME milestones language. This paper outlines the process undertaken by one medical school to design, implement and improve competency milestones for medical students. A team of assessment experts developed milestones for a set of ...

MedEdPORTAL Publications

Introduction: Traditional normative Likert-type evaluations of faculty teaching have several drawbacks, including lack of granular feedback, potential for inflation, and the halo effect. To provide more meaningful data to faculty on their teaching skills and encourage educator self-reflection and skill development, we designed and implemented a milestone-based faculty clinical teaching evaluation tool. Methods: The evaluation tool contains 10 questions that assess clinical teaching skills with descriptive milestone behavior anchors. Nine of these items are based on the Stanford Faculty Development Clinical Teaching Model and annual Accreditation Council for Graduate Medical Education (ACGME) resident survey questions; the tenth was developed to address professionalism at our institution. The tool was developed with input from residency program leaders, residents, and the faculty development committee and piloted with graduate medical education learners before implementation. Results: More than 7,200 faculty evaluations by learners and 550 faculty self-evaluations have been collected. Learners found the form easy to use and preferred it to previous Likert-based evaluations. Over the 2 years that faculty selfevaluations have been collected, their scores have been similar to the learner evaluation scores. The feedback provided faculty with more meaningful data on teaching skills and opportunities for reflection and skill improvement and was used in constructing faculty teaching skills programs at the institutional level. Discussion: This innovation provides an opportunity to give faculty members more meaningful teaching evaluations and feedback. It should be easy for other institutions and programs to implement. It leverages a familiar milestone construct and incorporates important ACGME annual resident survey information.

BMC medical education, 2024

Background This study evaluated the utility of using Accreditation Council for Graduate Medical Education (ACGME) Milestones as a formative assessment tool for the fifth-and sixth-grade medical students' performance in their internal medicine (IM) clerkship and the same students' performance in their post-graduate year (PGY) IM training. Methods Retrospective data were collected from 65 medical students completing the two-year IM clerkship in the academic years 2019 and 2020 and 26 of the above students completing their PGY-1 training at the same university hospital in the academic year 2021. Data included the assessment results of 7 of the ACGME IM Milestones, information on admitted patients assigned to the students, and surveys of the students' satisfaction. The analysis included 390 assessment results during the IM clerkship and 78 assessment results during the PGY-1 training. Clinical teachers commonly rated level 3 to medical students in the IM clerkship, with PC-2 subcompetency receiving the lowest rating among seven subcompetencies. The levels of most subcompetencies showed stationary in the two-year IM clerkship. Significant improvement was observed in all subcompetencies during the PGY-1 training. The medical students in the second-year IM clerkship expressed higher satisfaction with implementing Milestones than in their first-year IM clerkship and perceived Milestones assessments' usefulness as learning feedback. Conclusions Using ACGME Milestones as a formative assessment tool in the IM clerkship yielded promising outcomes. Longitudinal follow-up of subcompetencies facilitated tracking students' development and providing constructive feedback.

Journal of graduate medical education, 2009

The Outcome Project requires high-quality assessment approaches to provide reliable and valid judgments of the attainment of competencies deemed important for physician practice. The Accreditation Council for Graduate Medical Education (ACGME) convened the Advisory Committee on Educational Outcome Assessment in 2007-2008 to identify high-quality assessment methods. The assessments selected by this body would form a core set that could be used by all programs in a specialty to assess resident performance and enable initial steps toward establishing national specialty databases of program performance. The committee identified a small set of methods for provisional use and further evaluation. It also developed frameworks and processes to support the ongoing evaluation of methods and the longer-term enhancement of assessment in graduate medical education. The committee constructed a set of standards, a methodology for applying the standards, and grading rules for their review of assessm...

Journal of Contemporary Medical Education

Introduction Accreditation Council for Graduate Medical Education's (ACGME's) Milestones assessment requirement has placed new demands on Program Directors (PDs), especially those with limited knowledge of assessment and evaluation activities. There is a lack of clarity on how Program Director (PDs)/Associate PDs (APDs) are effectively implementing milestones assessment and evaluation practices in the Graduate Medical Education programs. The purpose of this study was to investigate current assessment practices, needs, and challenges of PDs in implementing milestones assessment within their residency and fellowship programs in a pediatric hospital setting. Methods This study used a collective case study approach to obtain information from PDs, APDs, and Clinical Competency Committee (CCC) Chairs in 19 graduate programs at a pediatric hospital. We used structured meetings with planned agendas and a pre-formatted template to itemize program needs/difficulties/challenges in the milestone assessment. We used cross-case thematic content analysis to identify categories and themes to compare differences and commonalities across programs. Results A total of 38 PDs, APDs, and CCC Chairs from 19 different specialties/subspecialties participated in this study. Thirteen types of assessment and evaluation tools were consistently used across programs. Three categories emerged in relation to those assessment and evaluation types (direct, indirect, and multi-source). Rotation evaluation (84.2%), direct observation (73.2%), and 360-degree assessment (68.4%) were primarily used for measuring patient care among the six core competencies. Programs' needs varied from curriculum and assessment tool development to alignment of milestones items, and to creating a systematic assessment management plan. The most common challenges were difficulties related to logistics and tracking of evaluation in the survey management system (52.6%), challenges with time management (47.3%), and difficulty in determining and interpreting the milestones' numbers and levels (31.5%). Conclusions Milestones assessment and evaluation in medical education can be a challenge, but a priority for many training programs. Our study indicated that milestones assessment and evaluation in medical education are far more complex than we expect. Multiple assessment methods must be utilized to evaluate all essential competencies for accurate measurement of trainees' performance abilities. Our study uncovered several issues PDs faced during the implementation of milestones assessment and needs and challenges.

The Medical Journal of Australia

Medical teacher, 2015

This paper presents perspectives and controversies surrounding the use of milestones to assess competency in outcomes-based medical education. Global perspectives (Canada, Europe, and the United States) and developments supporting their rationales are discussed. In Canada, there is a significant movement away from conceptualizing competency based on time, and a move toward demonstration of specific competencies. The success of this movement may require complex (rather than reductionist) milestones that reflect students' progression through complexity and context and a method to narrate their journey. European countries (United Kingdom, France, and Germany) have stressed the complexity associated with time and milestones for medical students to truly achieve competence. To meet the changing demands of medicine, they view time as actually providing students with knowledge and exposure to achieve various milestones. In the United States, milestones are based on sampling throughout ...

JAMA Network Open, 2023

The evaluation of trainees in graduate medical education training programs using Milestones has been in place since 2013. It is not known whether trainees who have lower ratings during the last year of training go on to have concerns related to interactions with patients in post training practice. OBJECTIVE To investigate the association between resident Milestone ratings and post training patient complaints. DESIGN, SETTING, AND PARTICIPANTS This retrospective cohort study included physicians who completed Accreditation Council for Graduate Medical Education (ACGME)-accredited programs between July 1, 2015, and June 30, 2019, and worked at a site that participated in the national Patient Advocacy Reporting System (PARS) program for at least 1 year. Milestone ratings from ACGME training programs and patient complaint data from PARS were collected. Data analysis was conducted from March 2022 to February 2023. EXPOSURES Lowest professionalism (P) and interpersonal and communication skills (ICS) Milestones ratings 6 months prior to the end of training. MAIN OUTCOMES AND MEASURES PARS year 1 index scores, based on recency and severity of complaints. RESULTS The cohort included 9340 physicians with median (IQR) age of 33 (31-35) years; 4516 (48.4%) were women physicians. Overall, 7001 (75.0%) had a PARS year 1 index score of 0, 2023 (21.7%) had a score of 1 to 20 (moderate), and 316 (3.4%) had a score of 21 or greater (high). Among physicians in the lowest Milestones group, 34 of 716 (4.7%) had high PARS year 1 index scores, while 105 of 3617 (2.9%) with Milestone ratings of 4.0 (proficient), had high PARS year 1 index scores. In a Author affiliations and article information are listed at the end of this article.

Journal of Surgical Education, 2017

OBJECTIVES: With the recent utilization of Accreditation Council for Graduate Medical Education developmental milestones as part of resident evaluation, we sought to explore whether milestone-based ratings were associated with American Board of Surgery In-Training Examination (ABSITE) scores. METHODS: Midyear milestone ratings were obtained from the Accreditation Council for Graduate Medical Education Accreditation Database System for years 2014, 2015, and 2016 for all postgraduate years 1-5 general surgery residents in our program and paired with ABSITE scores (n ¼ 69) from January of the following year. Linear regression was used to assess predictor importance of milestones on both ABSITE percentage correct scores and ABSITE percentile scores. RESULTS: Minimal to small positive correlations were observed between milestones and ABSITE percentile scores (r ¼ 0.09-0.25), while moderately large correlations were observed between milestones and percentage correct scores (r ¼ 0.65-0.76). The Medical Knowledge 1 (MK1) milestone significantly predicted ABSITE percentage correct scores, and explained 60% of the variance (adjusted R 2 ¼ 0.603). MK1 also significantly predicted ABSITE percentile scores, although weaker in magnitude, with MK1 explaining 20% of the variance (adjusted R 2 ¼ 0.197). Postgraduate year level and other milestones were not influential predictors of ABSITE scores. CONCLUSIONS: The midyear MK1 milestone rating was predictive of ABSITE scores and may serve as a useful marker for Clinical Competency Committees to identify residents who could benefit from additional support to prepare for the ABSITE, although given the small exploratory nature of this study, additional research is still needed.