Emergency Medicine Milestone

Jump Simulation is participating in a nationwide study determining the reliability of high fidelity simulation cases that are used to assess Emergency Medicine residents.

Background

The Accreditation Council of Graduate Medical Education (ACGME) developed the Next Accreditation System (NAS) as a means to transition to competency-based assessment of physicians-in-training. In this system, EM residents are now assessed biannually on 23 competencies, generally termed "the Milestones." Each milestone represents a competency within which Emergency Medicine (EM) resident performance will evolve across a continuum from novice to expert level as they progress through their training.

The ACGME and American Board of Emergency Medicine recommended simulation as one of the assessment modalities for the majority of EM competencies. It was proposed that 10 of the milestone competencies could be assessed using high fidelity simulation cases.

This study was developed to determine whether high-fidelity simulation-based assessment tools modeled after the NAS have acceptable reliability and construct validity. The tools were designed to assess the level of resident competencies within different simulation scenarios.

Methodology

Data collection began in 2013 and took place at multiple sites. Sessions at Jump were launched in January and February 2015. 31 emergency medicine residents participated locally. About 80 were involved nationally.

During the course of the study, Emergency Medicine residents from all training levels served individually as the lead physician in the high fidelity simulation scenarios. They were evaluated by Emergency Medicine faculty physicians using the NAS assessment tools.

The assessment tools were composed of "critical actions" that are specific to each scenario, yet map back to NAS milestone behaviors. Each critical action was scored as an observed/not observed item.

The faculty received standardized training on how to use the assessment tools. More than one rater per scenario allowed for calculations of inter-rater reliability.

Results

Data collection for The High Fidelity Simulation for Assessment of the Milestones in Emergency Medicine project is complete - analysis is underway.