Internal medicine fellowship programs have an incentive to select fellows who will ultimately publish. Whether an applicant's publication record predicts long term publishing remains unknown.
Using records of fellowship bound internal medicine residents, we analyzed whether publications at time of fellowship application predict publications more than 3 years (2 years into fellowship) and up to 7 years after fellowship match. We calculate the sensitivity, specificity, positive and negative predictive values and likelihood ratios for every cutoff number of application publications, and plot a receiver operator characteristic curve of this test.
Of 307 fellowship bound residents, 126 (41%) published at least one article 3 to 7 years after matching, and 181 (59%) of residents do not publish in this time period. The area under the receiver operator characteristic curve is 0.59. No cutoff value for application publications possessed adequate test characteristics.
Citation: Prasad V, Rho J, Selvaraj S, Cheung M, Vandross A, Ho N (2014) Can a Resident's Publication Record Predict Fellowship Publications? PLoS ONE 9(3): e90140. https://doi.org/10.1371/journal.pone.0090140
Editor: Kewei Chen, Banner Alzheimer's Institute, United States of America
Received: September 23, 2013; Accepted: January 25, 2014; Published: March 21, 2014
This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Fellows in the subspecialties of internal medicine must demonstrate evidence of research productivity, including publications, in accordance with current Accreditation Council for Graduate Medical Education guidelines  As such, fellowship program directors have incentive to select applicants who are likely to engage in scholarship and publish those results. One criterion used by fellowship programs to predict who will publish during and beyond fellowship is the number of publications a resident has at the time of application. The logic here is straightforward. Applicants, who publish more prior to fellowship, are likely to publish more in the long term. While no data exists for fellowship programs, one group has found that medical students with stronger publication records were ranked more favorably by general surgery residencies .
Although the idea that past publications predict future ones is accepted informally, and widely used to advise fellowship applicants, it has rarely been examined in peer review literature, and there is little published evidence in support of this claim. In part, this may be because fellowship selection procedures are guarded at the institutional level. One investigation of the publication habits of urology residents concluded there was a strong positive correlation between publications during residency and those afterwards (Spearman's rank correlation = 0.5, p<0.001) .
However, simply showing a statistically significant relationship between early publications and future publications does not mean that this fact is meaningful, or that it can be used to better select applicants. The key question is whether correlations offer ability to predict future outcomes. To understand this concept consider that fellowship directors must use all of the information applicants select to make a decision as to who will be interviewed. Some of the variables that program directors consider are: USMLE scores, medical school, hospital of residency, history or research, and the essay . However, another consideration is past research projects and interests, of which publications serves as a measure. Even if an applicant's publications are only used as an adjunct factor to selecting those to interview (as is the case), publications are discrete variables (applicants have either 0, 1, 2, or n many papers). Thus, one would be interested to know whether or not any discrete number serves as a useful cut-point. This is analogous to knowing that certain d-dimer level excludes pulmonary embolism, rather than merely knowing that risk increases with rising numbers. To best assess whether application publications can predict future publication one can construct a receiver-operator-characteristic curve to see whether any cut point offers reliable test characteristics. This point has been made elsewhere, regarding medical education publications  , and medical tests .
Here, we sought to investigate formally whether publications at time of fellowship application predict future publications in a sample of fellowship bound internal medicine residents. We report the sensitivity, specificity, positive and negative predictive values (PPV and NPV) of application publications as predictor of future publications, and present a receiver operator characteristic (ROC) curve for this test.
We assembled a database of fellowship bound internal medicine residents and their publication histories, described fully elsewhere  and summarized here. Using an Internet search engine (Google), we obtained publically available records of internal medicine residency fellowship match results. As all data we acquired was in the public domain, our investigation did not require institutional review board approval.
In contrast with other graduate fields and medical specialties, there is no published ranking for internal medicine residency programs. Programs were searched if they were associated with the top 20 medical schools according to US News and World Report Graduate School Rankings 2013. The following residency programs were searched: Brigham and Women's Hospital, Massachusetts General Hospital, Beth Israel Deaconess Medical Center, Johns Hopkins University Medical Center, Columbia University Medical Center, Yale University Medical Center, The University of Pennsylvania Medical Center, Washington University Medical Center in St. Louis, University of Washington Medical Center, Duke University Medical Center, Cornell Medical Center, University of Michigan Medical Center, Northwestern Memorial Hospital, University of Chicago Medical Center, Mount Sinai Medical Center, University of California San Francisco Medical Center, University of California Los Angeles Medical Center, University of California San Diego Medical Center, University of Texas Southwestern Medical Center, Stanford University Medical Center, University of Pittsburgh Medical Center, and Vanderbilt University Medical Center.
Programs were eligible if they listed the fellowship placement of a resident by name, location, and specialty. Data from centers were excluded if lists of resident placement were not published by year, if published resident placement list sizes varied significantly from year to year, raising the question of whether publish lists were complete, and/or if data for residents who pursued general medicine or hospital medicine was not listed.
Data was obtained for the following programs and the following match years: Columbia (2007–2009), University of Chicago (2006–2009), University of California Los Angeles (2006–2009), University of California San Diego (2005–09), University of Pennsylvania (2008–09). Residents who did not pursue fellowship were excluded from this analysis. Residents were considered fellowship bound if they pursued one of the following specialties: hematology/oncology, pulmonary, critical care, pulmonary/critical care, endocrinology and metabolism, gastroenterology, cardiovascular disease, nephrology, rheumatology, geriatrics, and infectious disease.
Data on the publication record of each resident were obtained by PubMed query using multiple search strategies, including, but not limited to 1) full name 2) last name plus first initial and residency program 3) last name plus first initial and fellowship program, 4) last name plus first initial and field of interest. When a resident was identified, his/her PubMed record was explored both backwards and forwards in time, using additional search strategies, including, but not limited to, unique initials, coauthors, subject matter, and principal investigators. Data was de-identified after publication history was obtained for analysis datasets.
The publications of internal medicine graduates were considered in two-year intervals. The fellowship match (defined as June 15th of the corresponding year) was used to align all records and defined as the zero point in time. One year before and after this date was termed the ‘perimatch’ period, and labeled accordingly on all graphs. Publications during the ‘perimatch’ period were considered among the publications at time of application, though some occurred up to one year after matching. As the Electronic Residency Application Service (ERAS) service allows applicants to report publications under review at a journal, we feel that a one-year inclusion post match approximates an applicant's submitted publication record on the ERAS application.
When graduates only completed a portion of their final two-year interval (for instance 18 out of 24 months), their rate of publication was assumed to be constant for that interval, and adjusted accordingly. We examined the data of 307 residents with follow up at least 3 years after fellowship match or more than 2 years after the start of fellowship, and up to 7 years after fellowship match. The full data set from which this project was derived had a robust rate of data acquisition. Data was obtained for 823 out of 922 names, a recovery rate of 89.3%. We were unable to recover data in cases where a search strategies returned a large number of results (e.g. John Smith, Neha Patel) and results could not be meaningfully filtered by institutional affiliation. Thus, accurate delineation of publications became unreasonable, and the name was omitted.
Our dataset, fully described elsewhere , included 307 residents for whom we had documented follow up of at least 2 years after the start of fellowship, and up to 7 years after fellowship match. Of these 307 fellowship bound internal medicine residents, 126 (41%) ultimately published at least one article between 3 and 7 years after fellowship match (Late Publishers). The remainder, 181 residents (59%) did not publish during this time (Non Publishers). The specialty choices of these residents are varied, and are shown in Figure 1. The number of publications at time of fellowship match for both Late and Non Publishers are shown in Table 1. Of note, the majority of fellows publish between 0 to 3 articles during the study period. Figure 2 shows the number of manuscripts fellows had published at the time of fellowship application, stratified by whether they published at least 1 manuscript (Late Publishers) after the match. Late Publishers had more manuscripts at time of application than non-publishers (2.03 vs 1.19, p = 0.008)
The sensitivity, specificity, positive and negative predictive values and likelihood ratios at various cut-off values for the number of publications at time of match is shown in Table 2. A receiver operator characteristic curve was constructed (Figure 3), with an area under the curve noted to be 0.59. Figure 4 shows a simple scatterplot of publications at the time of fellowship application (x-axis) versus those that occurred thereafter (y-axis). The r-squared coefficient for a linear regression of the data is 0.005.
This study of 307 fellowship bound internal medicine residents is the first to examine whether publication records at the time of fellowship match predict late publishing (between 3 and 7 years after fellowship match). With an area under the curve of 0.59, application publications fail as a predictor of late publications. We provide no evidence to support the widely held conception that those who publish at the time of fellowship match are more likely to publish in the long term. Additionally, our results are aligned with a small study of neurosurgical residents, which shows that publications do not predict academic or private practice careers .
The results presented here appear to contradict previous reports, which noted a strong, and statistically significant relationship between publications during residency and those afterwards  and publications during residency and choice of career (academic vs. private practice) . Notably, the r-squared value we observed: 0.005 suggests that only one-half of one percent of variation in post-match publications is explained by application publications. However, simple comparisons may differ from more rigorous testing. For instance, although we noted that Late Publishers had more articles at time of application than Non Publishers (2.03 vs. 1.19), correlation coefficients and test characteristics, two more rigorous tests, were not met. Other groups have shown unprofessional comments on dean's letters are significantly associated with future board disciplinary action , however, more rigorous analyses demonstrate that there is no way to leverage this information as a useful screening tool, and it is far more likely to wrongfully cast doubt upon students . Similarly, the publication record of an applicant at the time of fellowship match may lead fellowship directors to believe they have an understanding of that person's academic potential; however, our data do not show how this can be operationalized.
There are several limitations to our current work. The publication habits of the residents in the schools we examined may not reflect those of other schools. As with any study of this kind, the set of programs we examined is arbitrary. We chose major residency programs affiliated with top 20 medical schools according to US News and World Report, similar to the methodology used by Yang, et al. in their survey of urology publications . We were unable to study the quality of publications. Perhaps residents who published more first author papers, or in journals of higher impact factor were indeed more likely to publish in the long term than those who did not. Additionally, we did not analyze whether the type of publication offered predictive power, i.e. do basic science publications predict future publication? Although, we could perform such an analysis, it would run the risk of generating false positives simply by virtue of multiple hypothesis testing . We were not able to identify which residents earned other degrees (e.g. PhD), and whether those residents had different patterns of publication. As with any project of this kind, it is possible that any particular article may have been misattributed to a given author. In general, we erred on the side of inclusion when assessing publication histories, perhaps overestimating publication records. At the same time, there are reasons to hypothesize that we underestimated scholarship output. For instance, we did not consider other potentially worthy measures, such publications of book chapters, evidence updates, or non-peer reviewed medical newspapers. Surveys of residents may be best to ascertain these publications, and serve as a springboard for future research.
Finally, this project utilized the judgment of multiple different physicians; it is possible that others may disagree with our classification of articles. However, as articles were only classified based on the methods used, we doubt others would differ significantly. Despite these limitations, internal checks were used to detect misattributed articles, and we believe the results presented here are accurate for the group and time examined.
In addition to our findings, we suggest that our methodology of evaluating an association be adopted in future medical education publications. If groups identify factors that predict future performance, it would be useful to formally provide test characteristics of that factor, and not merely report its statistical significance in aggregate. Both in our findings and methods, our results may be of interest to a large group of fellowship and residency program directors as well as fellowship applicants.
VP had full access to the data, and takes responsibility for the integrity of the data and the accuracy of the data analysis. The views and opinions of Dr. Prasad do not reflect those of the National Institutes of Health.
Conceived and designed the experiments: VP JR. Performed the experiments: VP JR SS MC AV NH. Analyzed the data: VP JR SS MC AV NH. Wrote the paper: VP.
- 1. Accreditation Council for Graduate Medical Education (ACGME) (2007) ACGME Program Requirements for Graduate Medical Education in the Subspecialties of Internal Medicine.
- 2. Stain SC, Hiatt JR, Ata A, Ashley SW, Roggin KK, et al. (2013) Characteristics of highly ranked applicants to general surgery residency programs. JAMA Surgery 148: 413–417.
- 3. Yang G, Zaid UB, Erickson BA, Blaschko SD, Carroll PR, et al. (2011) Urology resident publication output and its relationship to future academic achievement. J Urol 185: 642–646.
- 4. Kanna B, Gu Y, Akhuetie J, Dimitrov V (2009) Predicting performance using background characteristics of international medical graduates in an inner-city university-affiliated Internal Medicine residency training program. BMC Med Educ 9: 42.
- 5. Prasad V (2011) Are we treating professionalism professionally? Medical school behavior as predictors of future outcomes. Teach Learn Med 23: 337–341.
- 6. Prasad V (2011) Caution needed when predicting problem residents. Acad Med 86: 2; author reply 2.
- 7. Ioannidis JA, Tzoulaki I (2010) What makes a good predictor?: The evidence applied to coronary artery calcium score. JAMA 303: 1646–1647.
- 8. Prasad V, Rho J, Selvaraj S, Toomey C, Vandross A, et al. (2012) Publication trends among internal medicine residents and graduates. Am J Med 125: 939–944.
- 9. Lawton MT, Narvid J, Quinones-Hinojosa A (2007) Predictors of neurosurgical career choice among residents and residency applicants. Neurosurgery 60: 934–939; discussion 934–939.
- 10. Crowley RW, Asthagiri AR, Starke RM, Zusman EE, Chiocca EA, et al. (2012) In-training Factors Predictive of Choosing and Sustaining a Productive Academic Career Path in Neurological Surgery. Neurosurgery 70: 1024–1032 1010.1227/NEU.1020b1013e3182367143.
- 11. Papadakis MA, Teherani A, Banach MA, Knettler TR, Rattner SL, et al. (2005) Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med 353: 2673–2682.
- 12. Ioannidis JA (2012) THe importance of potential studies that have not existed and registration of observational data sets. JAMA 308: 575–576.