Skip to content
Home » Emergency Medicine Resident Selection Process | OAEM – Dove Medical Press

Emergency Medicine Resident Selection Process | OAEM – Dove Medical Press

Javascript is currently disabled in your browser. Several features of this site will not function whilst javascript is disabled.
open access to scientific and medical research
Papers Published
Open access peer-reviewed scientific and medical journals.
Learn more
Dove Medical Press is a member of the OAI.
Learn more
Bulk reprints for the pharmaceutical industry.
Learn more
We offer real benefits to our authors, including fast-track processing of papers.
Learn more
Register your specific details and specific drugs of interest and we will match the information you provide to articles from our extensive database and email PDF copies to you promptly.
Learn more
Back to Journals » Open Access Emergency Medicine » Volume 15
Authors Bray K , Burge K, Patel O, Yadav I, Haynes W, Van Wagoner N, Khoury CA
Received 29 September 2022
Accepted for publication 25 December 2022
Published 12 January 2023 Volume 2023:15 Pages 15—20
DOI https://doi.org/10.2147/OAEM.S389868
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 4
Editor who approved publication: Dr Hans-Christoph Pape
Kevin Bray,1 Kaitlin Burge,1 Om Patel,1 Ishant Yadav,1 William Haynes,1 Nicholas Van Wagoner,2 Charles A Khoury3

1Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, USA; 2Division of Infectious Diseases, University of Alabama at Birmingham, Birmingham, AL, USA; 3Department of Emergency Medicine, University of Alabama at Birmingham, Birmingham, AL, USA

Correspondence: Kevin Bray, Email [email protected]

Background: Beginning January 26th, 2022, the National Board of Medical Examiners transitioned scoring of the United States Medical Licensing Examination (USMLE) Step 1 from a 3-digit score to pass/fail. In the past, the Step 1 score has been weighted heavily by program directors (PDs) as one of the most important metrics when assessing medical student’s competitiveness.
Objective: The objective of this study was to evaluate the perceptions of emergency medicine (EM) PDs on the transition to a pass/fail USMLE Step 1 exam, and to elicit the opinions of EM PDs on the USMLE examinations’ ability to predict resident performance.
Methods: A survey consisting of ranking and multiple-choice questions was sent to EM PDs. The multiple-choice questions were asked to determine EM PDs level of confidence in the ability of Step 1 and Step 2 Clinical Knowledge (CK) to predict a student’s ability to succeed in residency. The ranking questions focused on assessing each program’s current resident selection practices in comparison to expected selection criteria changes following a transition to pass/fail Step 1. R studio and MATLAB were used for statistical analysis, and a P value < 0.05 was considered significant.
Results: The survey was completed by 57 (20.21%) EM PDs. When asked if Step 1 and Step 2 CK are accurate predictors of a resident’s ability to perform clinically within EM, only 10.5% of PDs answered ‘yes’ to Step 1 being predictive, compared to 31.6% for Step 2 CK. Regarding selection criteria, the top quartile of attributes (standardized letters of evaluation [1st], away rotations [2nd], clerkship grades [3rd] and Step 2 CK score [4th]) remained the same following the transition.
Conclusion: Our results indicate that the top quartile of attributes might remain the same, despite most PDs agreeing that Step 2 CK is a better predictor of a resident’s performance.

Keywords: USMLE, NBME, Step 1, Step 2, emergency medicine, residency, program director, medical student

Beginning January 26th, 2022, the National Board of Medical Examiners (NBME) transitioned scoring of Step 1 of the United States Medical Licensing Examination (USMLE) from a 3-digit score to pass/fail.1 Historically in emergency medicine (EM), the Step 1 score has been weighted heavily by program directors (PDs) as one of the most important metrics in assessing a medical student’s competitiveness for EM residency programs, alongside standardized letters of evaluation (SLOE).2 In the United States (US), the selection process for residency programs focuses on locally determined competitiveness to compare applicants such as past academic records, standardized testing scores, resumes, and personal statements. Other countries, such as the United Kingdom and Australia, have adapted a more holistic selection criteria which encompasses a wide range of written and observed formats such as multiple mini-interviews, situational judgement tests, and clinical problem-solving tests to better assess an applicant’s competitiveness.3 The NBME proposed the transition to a pass/fail Step 1 with the hopes of reducing the emphasis on Step 1 performance in residency screening, accelerating research regarding USMLE performance in predicting a resident’s clinical and board examination performance, and minimizing racial demographic differences within medical boards performance.4 While the US has not yet adapted the methods of holistic selection criteria as seen in other countries, the transition to a pass/fail Step 1 is the first step in adapting a more holistic approach to the residency application process.
In the past, Step 1 has been an important milestone in the medical education pathway but has been known to create a significant amount of physical, mental, and financial stress for students during their dedicated preparation time. One study suggested that the pressure to perform well on Step 1 led to 77% of surveyed students feeling “burned out” and 72% of surveyed students reporting a decrease in their overall quality of life following the preparation process.5 Other studies report that, on average, students spend more than 10 hours per day for over 4 weeks preparing for the Step 1 examination.6 Despite this time spent studying, Step 1 scores have not been shown to predict a resident’s performance in EM residency.7
Demographic differences have also been observed to play a part in USMLE performance. In the past, the emphasis on Step 1 scores during the residency selection process has been argued as being an objective, standardized measure. However, Step 1 scores have been shown to disadvantage female and underrepresented minority applicants.8 With a transition to pass/fail Step 1 scoring, the residency selection processes may adopt a more holistic selection process, enhancing the diversity within the physician workforce.9
Many hope that the transition to a pass/fail Step 1 will lead to decreased mental and physical stress on students. Despite the mitigation of student stress, PDs may be concerned at the prospect of having one less piece of objective data to compare prospective applicants. Patel et al analyzed the differences between procedural and nonprocedural specialties and how they ranked attributes before and after the implementation of Step 1 pass/fail. The study concluded that while there were individual differences placed on residency applicant characteristics by procedural and non-procedural specialty PDs, there was little change in rank order of applicant attributes when compared before and after the transition to pass/fail Step 1. Nonprocedural specialties considered clerkship grades and a dean’s letter most important when compared to procedural specialties who considered Step 1 score and letters of recommendation most important prior to transition to a pass/fail Step 1.10 It is unclear how this transition to a pass/fail Step 1 exam will affect how EM, a nonprocedural specialty, and how EM PDs will assess applicants’ competitiveness to residency programs. The purpose of this study was to evaluate the perceptions of EM PDs on the transition to a pass/fail USMLE Step 1 exam. We predicted that other metrics such as USMLE Step 2 Clinical Knowledge (CK) score, extracurricular activities, standardized letters of evaluation (SLOEs), research involvement, and community service would become more important in the EM resident selection process following the transition to a pass/fail USMLE Step 1. Furthermore, we hoped to elicit the opinions of EM PDs on the ability of USMLE examinations to predict resident clinical and board examination performance.
An IRB-approved secure survey consisting of multiple choice and ranking questions was created using Qualtrics software and sent out to the PDs of all 282 EM residency programs accredited by the Accreditation Council for Graduate Medical Education in the United States. The IRB was approved through the University of Alabama at Birmingham Institutional Review Board. The survey was sent to PDs via email. This PD email list was compiled by using the American Medical Association’s Residency and Fellowship Database. The initial email to residency PDs was followed with two reminders, one sent two weeks after the original email was sent and another four weeks after the original email was sent. Each PD consented to participate in this study before completing the survey. The ranking questions focused on assessing each program’s current resident selection practices and then asked them to compare their current practices with how they expected their selection criteria to change for future applicants following the transition to pass/fail Step 1. The ranking questions were comprised of 16 attributes that each PD ranked in importance based on their own resident selection process. PDs were also asked a series of multiple-choice questions to determine their level of confidence in the ability of USMLE Step 1 and Step 2 CK to predict a student’s clinical skills and ability to succeed in residency. Following the data collection period, R studio and MATLAB were used for statistical analysis and a p-value <0.05 was considered significant. Chi squared tests were used to analyze the results of the multiple-choice questions in R Studio using Tidyverse and knitr packages, and two-tailed paired T-tests were used to analyze ranking questions in MATLAB (version 9.8).
The multiple-choice portion of the survey was completed by 57 (20.21%) EM PDs (Table 1). Thirty-four of the responses came from academic programs, 14 were academic-affiliated, and 9 were community-based programs. Of the 57 program directors who completed the multiple-choice portion of the survey, 6 (10.53%) of the responses came from a PD affiliated with top 15 NIH funded program. PDs agreed that both USMLE Step 1 and Step 2 CK adequately predicted a resident’s ability to pass the EM board certification exam at the end of residency (52.6% and 73.7% respectively, p<0.01). When asked if Step 1 and Step 2 CK are accurate predictors of a resident’s ability to perform clinically within EM, only 10.5% of PDs answered “yes” to Step 1 being predictive (p<0.01), compared to 31.6% for Step 2 CK (p=0.33). When asked if a medical student’s class rank will be considered more following the transition to Step 1 Pass/fail, most respondents (52.6%, p<0.01) believe that a medical student’s rank will be considered more. With regards to the transition to pass/fail Step 1, 68.4% (p<0.01) of the respondents reported that students would not be as prepared for residency compared to students who took a graded Step 1.

Table 1 Emergency Medicine Program Directors’ Responses Concerning the Predictive Value of Step 1 and Step 2 on Resident Performance

Table 1 Emergency Medicine Program Directors’ Responses Concerning the Predictive Value of Step 1 and Step 2 on Resident Performance
43 (15.2%) of PDs completed the entirety of the survey section evaluating applicant attributes. PDs ranked 14 of the 16 attributes higher following the transition to a pass/fail Step 1 (Table 2). Although the change was minimal, two attributes (SLOE and Step 2 CK score) remained in the top quartile of attributes compared to previous years. Were ranked lower following the transition to a pass/fail Step 1. However, analysis using a paired T-test comparing PDs perceived rankings pre-transition and post-transition to a pass/fail Step 1 revealed that these two changes were statistically insignificant, with p-values equal to 0.24 and 0.92, respectively. The changes did not affect the overall order of the top quartile of attributes compared to previous years. The first quartile of attributes pre-transition and post-transition are as follows: SLOEs [1st], away rotations [2nd], clerkship grades [3rd] and Step 2 CK score [4th]. Previously ranked the 5th highest attribute, the Step 1 numerical score is expected to be replaced by increased emphasis on extracurricular involvement, leadership experiences, class rank, and volunteering.

Table 2 Perceived Relative Rankings by 43 Emergency Medicine Program Directors of Various Residency Applicant Attributes Before and After Implementation of a Pass/Fail Step 1

Table 2 Perceived Relative Rankings by 43 Emergency Medicine Program Directors of Various Residency Applicant Attributes Before and After Implementation of a Pass/Fail Step 1
Our results indicate that EM PDs may not place more emphasis on the Step 2 CK score following the transition to a pass/fail Step 1 despite most PDs agreeing that Step 2 CK is a better predictor of a resident’s board examination performance. With the removal of a scored Step 1, EM PDs are likely to continue to consider SLOEs, away rotations, clerkship grades, and Step 2 CK score as being the most important attributes when assessing a medical student’s competitiveness to their residency program. According to the AAMC, EM programs around the country received an average of 861.59 residency applications in 2022.11 As programs will no longer have a numerical Step 1 to initially screen applications, this transition may encourage a more holistic review of an applicant’s entire application.
The AAMC defines a holistic review as a flexible and individualized approach to assessing an applicant’s competitiveness through a balanced consideration given to experiences, attributes, and scholarly metrics.12 The transition to a more holistic review in the medical resident selection process will promote a more diverse applicant pool to which programs are able to find applicants that align with the institution’s mission and goals. A more diverse physician workforce ensures all backgrounds, beliefs, and perspectives are represented in the medical field.
Resident performance: Our results suggest that PDs believe the Step 2 CK score to be a better predictor for EM specialty board examination performance compared to Step 1 score. PDs agreed that Step 1 score is not a strong predictor for residency clinical performance (p<0.01), but remained relatively neutral when asked if Step 2 CK was a strong predictor in residency clinical performance (p=0.33). However, when asked if students will be better prepared clinically after Step 1 transitions to pass/fail, 68% of respondents reported that students will not be as prepared and 32% remained neutral, even though the Step 1 exam places greater emphasis on basic science knowledge as compared to clinical knowledge. One possible explanation for this is that PDs may believe that intensive studying for a scored examination may help develop the discipline needed to perform well clinically. Medical students agree that the Step 1 exam falls short on focusing on the clinical knowledge required to successfully treat patients.13 The Step 2 CK exam, on the other hand, focuses more on testing a student’s ability to apply medical knowledge to clinical scenarios they will likely encounter in the future. With less emphasis placed on preparation for the Step 1 exam, medical students responded to a survey that they would be more likely to focus on other factors that will prepare them better for residency, including the Step 2 CK exam and clinical research.14
There were several limitations to this study. The response rate was low for this survey. Given the low response rate and incomplete responses, there is a chance that this sample size might not be representative of all EM PDs. On the Qualtrics survey, PDs initially completed the multiple-choice portion of the survey. However, 14 of the PDs did not complete the second half of the survey consisting of the ranking questions. Therefore, the total number of responses for the multiple-choice questions did not equal the number of responses for the ranking questions. The purpose of this study was to develop an initial understanding of PD perceptions of a pass/fail USMLE Step 1 and to assess which components of the application may become more important following this change. Research in the coming years should be geared towards assessing the actual observed change. Follow-up data can then be compared to the initial data to determine if PDs will place more emphasis on attributes that were not accounted for in this study.
Without a 3-digit score on the USMLE Step 1 examination, our results indicate that SLOEs, away rotations, clerkship grades, and Step 2 CK scores will continue to be in the highest quartile of attributes considered by PDs when assessing a medical student’s competitiveness for residency programs. PDs indicate that they may not place a greater emphasis on Step 2 CK scores following the transition despite most PDs agreeing that the Step 2 CK score is a better predictor of EM board certification exam performance compared to the Step 1 score. More emphasis may be placed on leadership experiences, extracurricular involvement, class rank, and volunteering. Further research is required in the coming years to determine how the transition to a pass/fail USMLE Step 1 will impact the resident selection process.
The authors report no conflicts of interest in this work.
1. USMLE. USMLE step 1 transition to pass/fail only score reporting. Available from: https://www.usmle.org/usmle-step-1-transition-passfail-only-score-reporting. Accessed January 8, 2023.
2. Program NRM. Data release and research committee: results of the 2018 NRMP program director survey. Available from: https://www.nrmp.org/wp-content/uploads/2021/07/NRMP-2018-Program-Director-Survey-for-WWW.pdf. Accessed January 8, 2023.
3. Roberts C, Khanna P, Rigby L, et al. Utility of selection methods for specialist medical training: a BEME (best evidence medical education) systematic review: BEME guide no. 45. Med Teach. 2018;40(1):3–19. doi:10.1080/0142159X.2017.1367375
4. Committee TIP. Summary report and preliminary recommendations from theinvitational conference on USMLE scoring (InCUS); 2019. Available from: https://www.usmle.org/sites/default/files/2021-08/incus_summary_report.pdf. Accessed January 8, 2023.
5. Tackett S, Jeyaraju M, Moore J, et al. Student well-being during dedicated preparation for USMLE step 1 and COMLEX Level 1 exams. BMC Med Educ. 2022;22(1):16. doi:10.1186/s12909-021-03055-2
6. Kumar AD, Shah MK, Maley JH, Evron J, Gyftopoulos A, Miller C. Preparing to take the USMLE step 1: a survey on medical students’ self-reported study habits. Postgrad Med J. 2015;91(1075):257–261. doi:10.1136/postgradmedj-2014-133081
7. Wagner JG, Schneberk T, Zobrist M, et al. What predicts performance? A multicenter study examining the association between resident performance, rank list position, and United States medical licensing examination step 1 scores. J Emerg Med. 2017;52(3):332–340. doi:10.1016/j.jemermed.2016.11.008
8. Edmond MB, Deschenes JL, Eckler M, Wenzel RP. Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews. Acad Med. 2001;76(12):1253–1256. doi:10.1097/00001888-200112000-00021
9. Pershing S, Co JPT, Katznelson L. The new USMLE step 1 paradigm: an opportunity to cultivate diversity of excellence. Acad Med. 2020;95(9):1325–1328. doi:10.1097/ACM.0000000000003512
10. Patel OU, Haynes WK, Burge KG, et al. Results of a national survey of program directors’ perspectives on a pass/fail US medical licensing examination step 1. JAMA Netw Open. 2022;5(6):e2219212. doi:10.1001/jamanetworkopen.2022.19212
11. Service ERA. ERAS statistics preliminary data- residency; 2022. Available from: https://www.aamc.org/data-reports/interactive-data/eras-statistics-data. Accessed January 8, 2023.
12. AAMC. Holistic principles in resident selection: an introduction. Available from: https://www.aamc.org/media/44586/download?attachment. Accessed January 8, 2023.
13. Carmody JB, Green LM, Kiger PG, et al. Medical student attitudes toward USMLE step 1 and health systems science – A multi-institutional survey. Teach Learn Med. 2021;33(2):139–153. doi:10.1080/10401334.2020.1825962
14. Girard AO, Qiu C, Lake IV, Chen J, Lopez CD, Yang R. US medical student perspectives on the impact of a pass/fail USMLE step 1. J Surg Educ. 2022;79(2):397–408. doi:10.1016/j.jsurg.2021.09.010
Creative Commons License © 2023 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution – Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.
Contact Us     Privacy Policy     Associations & Partners     Testimonials     Terms & Conditions     Recommend this site   Top
Contact Us    Privacy Policy
© Copyright 2023    Dove Press Ltd     software development by maffey.com Web Design by Adhesion
The opinions expressed in all articles published here are those of the specific author(s), and do not necessarily reflect the views of Dove Medical Press Ltd or any of its employees.
Dove Medical Press is part of Taylor & Francis Group, the Academic Publishing Division of Informa PLC
Copyright 2017 Informa PLC. All rights reserved. This site is owned and operated by Informa PLC ( “Informa”) whose registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 3099067. UK VAT Group: GB 365 4626 36
In order to provide our website visitors and registered users with a service tailored to their individual preferences we use cookies to analyse visitor traffic and personalise content. You can learn about our use of cookies by reading our Privacy Policy. We also retain data in relation to our visitors and registered users for internal purposes and for sharing information with our business partners. You can learn about what data of yours we retain, how it is processed, who it is shared with and your right to have your data deleted by reading our Privacy Policy.
If you agree to our use of cookies and the contents of our Privacy Policy please click ‘accept’.

source

Leave a Reply

Your email address will not be published. Required fields are marked *