Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools

Jesse Burk-Rafel; Ricardo W. Pulido; Yousef Elfanagely; Joseph C. Kolars

doi:10.1371/journal.pone.0224675

Peer Review History

Original SubmissionJuly 21, 2019
5 Sep 2019 Decision Letter - Andrew Carl Miller, Editor Transfer Alert This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present. PONE-D-19-20613 Institutional differences in USMLE performance: Cross-sectional study of 89 US allopathic medical schools PLOS ONE Dear Dr. Burk-Rafel, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. We would appreciate receiving your revised manuscript by Oct 20 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'. Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. We look forward to receiving your revised manuscript. Kind regards, Andrew Carl Miller Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide. 3. Thank you for stating the following in the Competing Interests section: "I have read the journal's policy and the authors of this manuscript have the following competing interests: Dr. Burk-Rafel reports working as a research consultant for ScholarRx, a digital learning platform that includes USMLE preparation services, during the late stages of writing this manuscript. ScholarRx was not involved in this study in any way. All other authors declare no competing interests." a. Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. b. Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf. Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: http://journals.plos.org/plosone/s/competing-interests 4. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical. Additional Editor Comments: Please ensure that your manuscript adheres to the STROBE guidelines and provide citation: von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. PMID: 17941714 Please indicate the study’s design with a commonly used term in the title or the abstract (eg. cohort, observational, retrospective, etc.) Was a sample size calculation performed? If so, please provide. Is the same data available for osteopathic medical schools? If so, why were they excluded? 42 allopathic schools were excluded from the analysis for not providing complete USMLE schools over the 3-year period. It would be nice to know more about those schools. Are there any commonalities amongst those schools that make them different from the other 89? Were they more likely to be new schools, struggling schools, etc. How do you think that this impacts the results? How does school age impact the results (e.g. first 5 years vs longer established)? Does the percentage of international students affect the results? Do schools with a higher percentage of students with other advanced degrees (masters, PhD, etc) perform better (or worse)? Is there a way to carry this forward to see the specialties that these students choose? Is there any relationship to schools having a higher or lower percentage of students matriculating into highly vs. less competitive specialties for residency? [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Partly ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: I Don't Know Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: No Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: It would be helpful to have a Table in the supporting document list the mean USMLE scores and mean MCAT scores for each institution since those data were the data used for the analysis. All of these data are publicly available why not list them. Reviewer #2: Thank you for the opportunity to review this manuscript. This addresses an important topic in continuing to explore the impact of medical school curricula and other factors that contribute to USMLE interinstitutional variation. This large scale study does have major limitations which deserve greater discussion. Specific comments are listed below. 1. Page 5: describe where data will be available. 2. Methods: Further information is needed to better understand the data limitations of the self-reported data. For example, is anything known about consistency of reporting of the MCAT score? Is this is superscore of the last 3 takes or single best take? Is there a consistent reporting structure between institutions or is this reported completely at the institution’s discretion? The variability in how this data was reported represents a major potential limitation of the study. 3. Undergraduate gpa is a frequently discussed variable for MCAT performance. However, an increasing number of students have graduate degrees. Does percentage of students with graduate degrees and graduate gpa warrant a discussion in this manuscript? 4. Please describe how the MCAT score data was defined. Was this related to a single section of the MCAT or an average of the 3 sections? 5. In the results section, for overall interpretation of the results of this study, it would be important to understand the national USMLE average scores for the time period investigated. Is this sample representative of all medical schools? 6. During the past couple of decades, most medical schools have undergone significant curricular changes related to increased emphasis on active learning and decreasing lecture time. Many medical schools and students feel that the emphasis on USMLE performance competes with a curriculum focused on clinical performance. In future studies, it will be important to explore if certain types of curricula are more or less successful in preparing students for USMLE examinations. Thank you again for the opportunity to review this manuscript. Despite the limitations, I feel that this type of larger study is needed to better understand USMLE performance at the institutional level. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step. Attachments Attachment Submitted filename: PLOS ONE Review.docx https://doi.org/10.1371/journal.pone.0224675.r001
Revision 1
16 Oct 2019 Author Response Response to Reviewers (see attached 'Response to Reviewers' document for formatted version) RE: PONE-D-19-20613 “Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools” In response to the reviewers’ comments, we addressed each point. 1. Please amend either the title on the online submission form (via Edit Submission) or the title in the manuscript so that they are identical. The title of the online submission form and manuscript is “Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools.” We changed the title on the online submission form and manuscript to be identical. 2. Please ensure that your manuscript adheres to the STROBE guidelines and provide citation. We have reviewed the STROBE guidelines and cited the citation into our manuscript. We attest that the manuscript adheres to the STROBE guidelines and we emphasize that our manuscript is purely an exploratory observational study meant to stimulate and generate a hypothesis based on reported data from prior years. The methodology of our study should still be generalizable to future studies as new data is reported. 3. Please indicate the study’s design with a commonly used term in the title or the abstract (eg. cohort, observational, retrospective, etc.) To further indicate the study’s design, we used the term “observational” in the abstract. The title already specifies it as “cross-sectional,” a type of observational study. 4. Was a sample size calculation performed? If so, please provide. A sample size calculation was not performed because we sought to obtain a complete census of all US allopathic medical schools. A statement to this effect was added to the methods. 5. Is the same data available for osteopathic medical schools (DO schools)? If so, why were they excluded? Osteopathic institutions were excluded from this study, as osteopathic students historically took the COMLEX licensing examination rather than the USMLE and very few US osteopathic institutions reported USMLE performance data. It is important to note as well that at the time of data collection only 25 DO institutions had at least a first admitting class year, of these approximately 15 osteopathic schools were formed from 2002-2012. The rapid development and the continuous changes in curricula as typically seen in new institutions were additional reasons for excluding the small number of DO institutions that did report data. Theses reasons for excluding osteopathic institutions are now included in the methods. 6. 42 allopathic schools were excluded from the analysis for not providing complete USMLE schools over the 3-year period. It would be nice to know more about those schools. Are there any commonalities amongst those schools that make them different from the other 89? Were they more likely to be new schools, struggling schools, etc. How do you think that this impacts the results? We have added national average performance data to Table 1 and S1 Table, and included a statement in results: “The average GPA, MCAT scores, and USMLE Step 1 scores for the 89 reporting institutions were slightly higher than national averages for all matriculants/examinees.” We go on in the discussion to discuss: “We found that reporting institutions, as compared to an average of all students nationally, had higher average GPA and MCAT scores, with an associated 1.5-point higher average USMLE Step 1 score. Statistical comparisons of these differences are not advisable given the different units of reporting (institutions vs. individuals); yet the very small differences suggest that the reporting institutions were nationally representative.” Overall, we are pleased that the reporting institutions very nearly matched national averages, suggesting good representativeness. Unfortunately, we do not have information regarding the non-reporting institutions to characterize their reasons for not reporting to US News and World Report. 7. How does school age impact the results (e.g. first 5 years vs longer established)? We do not have this data. We have added this to the discussion as an important limitation and area for further investigation. 8. Does the percentage of international students affect the results? Unfortunately, schools did not report the percent of international students within their classes. This would be an interesting covariate for future study. 9. Do schools with a higher percentage of students with other advanced degrees (masters, PhD, etc) perform better (or worse)? Schools' percent of non-traditional students (who took >1 year to do something between undergraduate and medical school) was not associated with differential performance, when controlling for MCAT and GPA. We do not have data on the percent of advanced degrees when entering medical school, but have added this as a limitation and area for future investigation. 10. Is there a way to carry this forward to see the specialties that these students choose? Is there any relationship to schools having a higher or lower percentage of students matriculating into highly vs. less competitive specialties for residency? Unfortunately, the main residency match data does not include match information at the level of schools and schools did not report distribution of specialties at match in this data set. However, the data set did include the percent of graduates entering primary care specialties (here defined as family medicine, pediatrics, and internal medicine), which have lower USMLE screening thresholds and are thus less “competitive”. Although having more primary care grads had a significant negative correlation with USMLE scores at baseline, it was not significant after controlling for GPA and MCAT. We have included several sentences in the discussion on this topic. 11. It would be helpful to have a Table in the supporting document list the mean USMLE scores and mean MCAT scores for each institution since those data were the data used for the analysis. All of these data are publicly available why not list them. We have created a supporting table which lists the mean GPA, MCAT, and USMLE scores for all 89 institutions. This table is referred to in the results portion of our paper and included in the Supporting Information. 12. Describe where data will be available. Data will be available in the Supporting Information table. 13. Further information is needed to better understand the data limitations of the self-reported data. For example, is anything known about consistency of reporting of the MCAT score? Is this a superscore of the last 3 takes or single best take? Is there a consistent reporting structure between institutions or is this reported completely at the institution’s discretion? The variability in how this data was reported represents a major potential limitation of the study. We do not know how schools reported their MCAT scores. We have added this to the limitations’ discussion of the paper and understand its importance. In a study conducted by Zhao et al, four different approaches were investigated when attempting to predict USMLE Step 1 scores using MCAT scores. It was concluded that averaging MCAT scores for an individual is the best predictor for Step 1 scores, highlighting the importance of how MCAT scores are gathered. However, it is reassuring that the average scores (GPA, MCAT, and USMLE) for all reporting institutions very nearly matched national averages for the study period, suggesting good fidelity of reporting. 14. Undergraduate GPA is a frequently discussed variable for MCAT performance. However, an increasing number of students have graduate degrees. Does the percentage of students with graduate degrees and graduate gpa warrant a discussion in this manuscript? We do not have data concerning the percentage of students with graduate degrees and graduate GPA. We did note that the schools' percent of non-traditional students (who took >1 year to do something between undergraduate and medical school) was not associated with differential performance, when controlling for MCAT and GPA. We do not have data on the percent of advanced degrees when entering medical school, but added this as an important factor for future research. A literature review was performed to further investigate students with graduate degrees and their graduate GPA. No papers were found discussing the relationship between graduate GPA and medical school examinations, highlighting this as an important area for further investigation. 15. Please describe how the MCAT score data was defined. Was this related to a single section of the MCAT or an average of the 3 sections? The MCAT score data was defined as the average institutional scores for each of the 3 sections (biological sciences, physical sciences, verbal reasoning) to create a total score institutional average. We included this description of the MCAT score data to the methods. 16. In the results section, for overall interpretation of the results of this study, it would be important to understand the national USMLE average scores for the time period investigated. Is this sample representative of all medical schools? We have included the national average GPA, MCAT, and USMLE scores for the study period in Table 1 and S1 Table for reference. We have also added reflection in the results and limitations section regarding the representativeness of the sample, highlighting that the sample had very similar scores compared to national averages, suggesting a nationally representative sample. 17. During the past couple of decades, most medical schools have undergone significant curricular changes related to increased emphasis on active learning and decreasing lecture time. Many medical schools and students feel that the emphasis on USMLE performance competes with a curriculum focused on clinical performance. In future studies, it will be important to explore if certain types of curricula are more or less successful in preparing students for USMLE examinations. We are aware that some medical schools have undergone significant curricular changes. More specifically, there has been a trend in increasing clinical experience and decreasing time in the classroom. Studies completed by Daniel et al and Jurich et al have discussed the pros and cons experienced with this fundamental change in curriculum. We have added this information to the discussion portion of our study and recommended future studies explore which types of curricula are more or less successful in preparing students for USMLE examinations. 18. The one suggestion is to clearly state in the conclusion that institutional mean GPA and mean MCAT scores correlate strongly with USMLE performance whereas student body demographic and other institutional variables were weak or insignificant when controlling for MCAT and GPA. We updated the conclusion to clearly state that the institutional mean GPA and mean MCAT scores correlate strongly with USMLE performance whereas student body demographic and other institutional variables were weak or insignificant when controlling for MCAT and GPA. Attachments Attachment Submitted filename: Response to Reviewers.docx https://doi.org/10.1371/journal.pone.0224675.r002
21 Oct 2019 Decision Letter - Andrew Carl Miller, Editor Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools PONE-D-19-20613R1 Dear Dr. Burk-Rafel, We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements. Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication. Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. With kind regards, Andrew Carl Miller Academic Editor PLOS ONE Additional Editor Comments (optional): Thank you for the important and interesting article. Reviewers' comments: https://doi.org/10.1371/journal.pone.0224675.r003
Formally Accepted
23 Oct 2019 Acceptance Letter - Andrew Carl Miller, Editor PONE-D-19-20613R1 Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools Dear Dr. Burk-Rafel: I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. For any other questions or concerns, please email plosone@plos.org. Thank you for submitting your work to PLOS ONE. With kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Andrew Carl Miller Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0224675.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .