Figures
Abstract
Purpose
The impact of resident rotation on patient outcomes in the intensive care unit (ICU) has been poorly studied. The aim of this study was to address this question using a large ICU database.
Methods
We retrospectively analyzed the French CUB-REA database. French residents rotate every six months. Two periods were compared: the first (POST) and fifth (PRE) months of the rotation. The primary endpoint was ICU mortality. The secondary endpoints were the length of ICU stay (LOS), the number of organ supports, and the duration of mechanical ventilation (DMV). The impact of resident rotation was explored using multivariate regression, classification tree and random forest models.
Results
262,772 patients were included between 1996 and 2010 in the database. The patient characteristics were similar between the PRE (n = 44,431) and POST (n = 49,979) periods. Multivariate analysis did not reveal any impact of resident rotation on ICU mortality (OR = 1.01, 95% CI = 0.94; 1.07, p = 0.91). Based on the classification trees, the SAPS II and the number of organ failures were the strongest predictors of ICU mortality. In the less severe patients (SAPS II<24), the POST period was associated with increased mortality (OR = 1.65, 95%CI = 1.17–2.33, p = 0.004). After adjustment, no significant association was observed between the rotation period and the LOS, the number of organ supports, or the DMV.
Citation: Chousterman BG, Pirracchio R, Guidet B, Aegerter P, Mentec H, CUB-REA network (2016) Impact of Resident Rotations on Critically Ill Patient Outcomes: Results of a French Multicenter Observational Study. PLoS ONE 11(9): e0162552. https://doi.org/10.1371/journal.pone.0162552
Editor: Sachin Yende, University of Pittsburgh, UNITED STATES
Received: January 21, 2016; Accepted: August 24, 2016; Published: September 14, 2016
Copyright: © 2016 Chousterman et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The CUB-REA steering committee policy is to prohibit the public diffusion of the database. The database is made of clinical data that are still collected according to ethical considerations and national regulation. Data can be accessed only by approved individuals after evaluation of their project by the committee. However, Prof P. Aegerter (philippe.aegerter@aphp.fr), in charge of the database, can be contacted to request the data for additional information or control by any interested reader.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
It is commonly believed that the quality of health care may decrease when residents are rotating [1]. However, the actual impact of resident rotation on patient outcomes remains under debate. This effect, previously described in the United Kingdom[2] and the USA[3], is often referred to as the the “July effect”. A recent systematic review confirms that year-end resident rotations may be associated with increased mortality and decreased efficiency of care [4]. However, because of the high heterogeneity of these data, the magnitude of this effect has not been precisely estimated.
The importance of the July effect might differ between specialties. We hypothesized that the impact of resident turnovers on intensive care might be larger. Only two studies have specifically focused on patients in intensive care unit (ICU) [5, 6]; these studies consistently reported no association between resident rotation and patient outcome. Nevertheless, these two studies may have been underpowered. Moreover, the July effect might be even more pronounced when new residents are recruited twice a year, as in France [7, 8], rather than once at the end of the year, as in the UK or the USA. The aim of the present study was to estimate the effect of bi-annual resident rotation on ICU outcomes in a large French cohort of critically ill patients.
Methods
To assess the impact of resident turnover on ICU outcomes, we used data included in the Collège des Utilisateurs de Base de Données en Réanimation (CUB-REA database) [9, 10] between 1996 and 2010.
CUB-REA
The CUB-REA network is a group of up to 38 ICUs in teaching hospitals in Paris (France) and its suburbs. Its steering committee is composed of nine medical doctors and a database administrator (P.A.). At all participating ICUs, data from all consecutive patients are prospectively collected using a standardized web-based case report form. Each center is responsible for the completeness of the data. External quality controls are performed on a regular basis. The CUB-REA network and database were approved by the Commision Nationale de l’Informatique et des Libertés (French Watchdog Privacy Authority) (agreement #564407). Patient records/information were anonymized and de-identified prior to analysis
Data Collected
In France, resident rotation occurs twice a year [7, 8] simultaneously throughout the country (at the beginning of November and at the beginning of May). Resident rotations encompass both resident rotations (residents advanced in their curriculum joining a department) and the arrival of new residents (first or second semesters in France). We recorded the exact rotation dates during the 1996–2010 period and then defined two periods: the period corresponding to the first month following the rotation was referred to as the POST period; the period corresponding to the fifth month of the rotation was referred to as the PRE period. The PRE period was not defined as the month immediately before the rotation to avoid any crossover between periods.
For each patient, demographic characteristics such as age, gender, comorbidities, admission diagnoses for ICU admission, admission modalities (direct or secondary admission), the modified Charlson comorbidity index [11, 12], and the Simplified Acute Physiology Score II (SAPS II) [13] score were collected. The intensity of ICU care was evaluated based on the number and type of organ supports provided during the ICU stay. The following therapies/monitoring devices were considered: mechanical ventilation (either invasive or non-invasive), catecholamine infusion, renal replacement therapy, and intracranial pressure monitoring. For all relevant variables, the percentage of missing data was less than 1%.
Study Goals and Endpoints
The primary outcome was ICU mortality. The secondary outcomes were the length of stay in the ICU (LOS), the duration of mechanical ventilation (DMV), and the number of organ supports. The primary goal of this study was to assess the impact of the rotation period (POST versus PRE) on the specified endpoints.
Statistical Analysis
Continuous variables are reported as means and standard deviations (SDs) or as medians and 25th-75th percentiles as appropriate. The categorical variables are presented as counts and percentages (%). Comparisons were performed using Wilcoxon and Fisher’s exact tests as appropriate. The impact of resident rotation was initially estimated using mixed-effect multivariate logistic regression models adjusted for age, reason for ICU admission, number of days in the hospital before ICU admission, season, SAPS II [13], modified Charlson comorbidity index [11, 12], the number of organ failures, the occurrence of shock/ARDS, the interaction between the period and the season (fixed effects), and the center (random effect). Adjusted Cox proportional hazard models were used to fit the LOS or DMV values. The number of organ supports was analyzed using a generalized linear model.
To account for potential complex interactions between the variables, we used a non-parametric classification tree-based approach for variable importance measure. The association between ICU mortality and the potential predictors was explored using recursive partitioning (tree package for R [14]) with pruning to prevent over-fitting [15]. We used a random forest model [16] (RandomForest package for R [15, 17]) to quantitatively estimate variable importance based on a permutation accuracy importance measure,[15, 16]. To select the predictors of interest, we used a conservative decision rule, as suggested by Strobl et al. [15]. In addition, we quantitatively estimated the contribution of each potential predictor to the prediction of ICU mortality using the z-scores together with their p-values [15].
Sensitivity analyses were conducted by assessing the impact of resident rotations on different strata of severity as defined by the SAPS II. Thresholds were based on the 25th and 75th percentiles of the SAPS II distribution in our cohort.
All analyses were performed using R 2.15.1 statistical software (The R Foundation for Statistical Computing, Vienna, Austria) on a Mac OS X platform.
Results
Patient Characteristics
Thirty semesters, corresponding to 30 resident rotations, were analyzed. During this period (1996–2010), 262,772 patients were admitted to the 38 ICUs: 44,431 patients during the PRE period (the fifth month of the resident rotation) and 42,979 patients during the POST period (the first month of the resident rotation). The patient characteristics are summarized in Table 1, and the 38 participating ICUs are described in S1 Table. The median patient age was 59 [43–73. The median SAPS II at ICU admission was 36 [23–52]. The patient characteristics were similar between the PRE and POST periods (Table 1).
ICU Mortality
The average ICU mortality rate was 18.4% (95%CI = 18.2; 18.5). In the mild (SAPS II < 24), moderate (SAPS II between 24 and 52), and high severity (SAPS II > 52) groups, the average ICU mortality rate was 2.4% (95%CI = 2.3; 2.5), 9.3% (95%CI = 9.2; 9.5), and 53.5% (95%CI = 53.1; 53.9), respectively. Within rotations, the ICU admission and mortality rates were found to fluctuate over time (Fig 1).
Upper panel: winter semester; lower panel: summer semester.
No significant difference in ICU mortality was observed between the PRE and the POST periods (PRE, 17.9%; POST, 18.3%; p = 0.14; Table 1). Kaplan Meier curves for 28-day mortality are provided in Fig 2. After adjusting for age, reason for ICU admission, number of days in the hospital before ICU admission, season, SAPS II, Charlson comorbidity index, number of organ failures, occurrence of shock/ARDS, center, and the interaction between the season and the rotation period, the POST period was not found to be associated with increased ICU mortality (OR = 1.01, 95%CI = 0.94; 1.07, p = 0.91).
Solid line: PRE period, dashed line: POST period.
We supplemented our analysis with recursive partitioning-based classification trees and a random forest model. The optimal tree obtained after pruning is presented in Fig 3. As illustrated in this figure, two predictors, the SAPS II and the number of organ failures, overtook all other potential explanatory variables. As demonstrated by a permutation-based variance importance measure from the random forest model (Fig 4), the value of the rotation period for predicting ICU mortality was found to be negligible. Moreover, the z-score corresponding to the rotation period, the SAPS II and the number of organ failures was -2.62 (p = 0.11), 235.70 (p = 0.009) and 111.44 (p < 0.001), respectively.
Each node in the tree represents the splitting variable, as well as the splitting threshold for continuous variables.
The variables are ranked from the most important (top) to the least important (bottom). Variable importance is represented on the x-axis as the z-score. SAPS II: Simplified Acute Physiology Score II; ARDS: acute respiratory distress syndrome; Charlson: modified Charlson comorbidity index; ICU: intensive care unit.
The impact of resident turnover varied across the severity strata (Fig 5). In the mild severity group (SAPS II < 24, n = 22,971), the POST period was associated with an increased ICU mortality (OR = 1.65, 95%CI = 1.17; 2.33, p = 0.004). However, the POST period was not associated with ICU mortality in the moderate (defined as a SAPS II between 24 and 52; n = 43,196, OR = 0.94, 95%CI = 0.84; 1.04, p = 0.25) or high severity (defined as a SAPS II score > 52) group (n = 21,243, OR = 1.01, 95%CI = 0.92; 1.10, p = 0.84).
OR: odds ratio.
Secondary Outcomes
The median LOS was 3 [2–7] days for both the PRE and POST periods (p = 0.78). This result did not change after adjusting for potential confounders (HR = 1.01, 95%CI = 0.99; 1.02, p = 0.39).
The DMV was similar between the PRE and the POST periods (PRE period: mean = 3.4 days [min = 0; max 240] vs. POST period: mean = 3.5 [min = 0; max = 272]; HR = 1.01, 95%CI = 0.99; 1.02, p = 0.153). This result remained consistent after adjusting for potential confounders (HR = 0.99, 95%CI = 0.98; 1.01, p = 0.61).
The daily average number of organ supports per patient was slightly lower during the POST period than during the PRE period (PRE period: mean = 0.68 [min = 0; max = 11] vs. PRE period: mean = 0.71 [min = 0; max = 14]; risk difference (RD) = -0.15, 95%CI = [-0.28; -0.01], p = 0.040). The association between the number of organ supports and the rotation period was not significant after adjusting for potential confounders (RD = -0.08, 95%CI = [-0.21]; 0.04, p = 0.20).
Discussion
In this large retrospective multicenter study in France, we found that biannual residents rotation exerts no significant impact on overall ICU mortality during the first month of the rotation. However, resident rotation was associated with an increase in ICU mortality in the less severe patients. The LOS, the DMV, and the number of organ supports were not significantly altered according to the rotation period.
The “July effect” has been extensively studied in various settings [3, 5, 6, 18–33], although the negative impact of new residents arrival remains under debate [2–4, 34–36]. Based on large cohort studies, some authors have reported associations between resident rotations, mortality rate, and adverse events [18, 23, 27, 32, 34, 37]. A recent systematic review [4] drew the conclusion that mortality increases and efficiency decreases in hospitals as a result of year-end changeovers. However, other results are conflicting [18–20, 28, 38–41], and several sources of heterogeneity might explain such discrepancies, especially the hospital type (teaching versus no teaching) or the intensity of care. The July effect is likely to be more pronounced in high-risk units such as ICUs. Accordingly, Jena et al. [27] recently reported increased mortality rates in patients hospitalized in July for acute myocardial infarction. However, Barry et al. [5] studied ICU patients admitted to a major teaching hospital and did not observe any association between resident rotation and patient outcomes. We hypothesized that assessing the impact of resident rotations would require a much larger database.
Based on standard multivariate regression analysis, we did not observe any significant adverse effect associated with new residents recruitment. However, this estimation may be biased because of the inability of the multivariate main term logistic regression model to capture complex interactions between predictors. Therefore, we also used a non-parametric modeling approach. Based on classification trees and a random forest model, we consistently found that the contribution of the resident turnover to ICU mortality was negligible. This result is in line with those concerning our secondary endpoints. We were unable to detect any associations between the residency period, the LOS, the DMV, or the number of organ supports, all of which are typically closely associated with ICU outcome. However, despite the absence of an effect on the overall population, resident rotation was associated with increased ICU mortality in the subgroup of patients with a SAPS II score < 24. There are several potential explanations for this result. First, when caring for the most severe patients, residents are likely to be more closely supervised by a senior physician but likely have greater autonomy when treating less severely ill patients. In addition, due to the high probability of death in severe ICU patients medical errors may be more likely to jeopardize the prognosis of a less severely ill patient. However, our results do not agree with those reported by Jena et al. [27] and Shuhaiber et al. [32], both of whom observed a more pronounced effect or resident rotation in high-risk patients. Nonetheless, the latter two studies focused on cardiac catheterization and cardiac surgery respectivelywhere the autonomy given to residents is typically more limited than that given to ICU residents.
Our study contains some limitations. First, it is a retrospective study. Therefore, some specific confounding factors may be missing and biasing the results. However, as illustrated in the tree analysis, ICU outcome is essentially driven by patient severity, which is well captured by the severity scores. French ICUs are described as closed units, i.e., units in which most medical decisions are made by attending intensivists rather than external consultants. Moreover, senior attending intensivists are present in the ICU every day and typically operate during night shifts. This particular organization of ICUs may limit the generalizability of our results to different health care systems. We chose to focus on the periods surrounding resident rotation dates. However, the rotation of other members of the medical staff may interfere with our results. For example, French fellows typically rotate after two years. Thus, when the new residents and the new fellows arrived simultaneously, the observed impact may have increased. However, we were unable to address this specific question in our study. Certain characteristics of the residents (previous experience in intensive care, etc.) are likely to interfere with the results. However, such information was not available. The data collection period was very long. We cannot avoid the confounders that ICU care and resident supervision have changed during this period. However, including time as a covariate in the models did not affect our results. Finally, the primary outcome was ICU mortality. Therefore, we cannot rule out a possible effect on subsequent outcomes such as readmission rate, 28-day mortality, and long-term health-related quality of life.
Conclusion
In conclusion, we found that in France, bi-annual resident rotation exerts no impact on overall ICU mortality, the LOS, the DMV, or the intensity of ICU care. However, resident rotation may affect the prognoses of the less severe ICU patients. Thus, staff physicians should invest additional effort into supervising ICU care for such patients, especially at the beginning of each rotation.
Supporting Information
S1 Table. Description of the ICUs included in the study
https://doi.org/10.1371/journal.pone.0162552.s001
(DOCX)
Acknowledgments
Members of the CUB-REA network:
P. Trouiller (Hôpital Antoine Béclère, Clamart), J-P. Bedos (Hôpital André Mignot, Versailles), A. Vieillard-Baron (Hôpital Ambroise Paré, Boulogne), Y. Cohen (Hôpital Avicenne, Bobigny), J. Labrousse (Hôpital Boucicaut, Paris), M. Safar (Hôpital Broussais, Paris), C. Richard (Hôpital du Kremlin-Bicêtre, Kremlin- Bicêtre), M. Wolff (Hôpital Bichat, Paris), G. Choukroun (Centre Hospitalier Sud-Francilien), J.-P. Mira (Hôpital Cochin, Paris), B. Verdière (Hôpital Delafontaine, Saint-Denis), P. Kalfon (Hôpital des Diaconesses, Paris), J.-Y. Fagon (Hôpital Européen Georges Pompidou, Paris), P. Ho (Hôpital de Gonesse, Gonesse), C. Brun-Buisson (Hôpital Henri Mondor, Créteil), A. Rabbat (Hôpital Hôtel-Dieu, Paris), F. Blot (Institut Gustave Roussy, Villejuif), C. Baillard (Hôpital Jean Verdier, Bondy), D. Dreyfus (Hôpital Louis Mourier, Colombes), F. Baud (Hôpital Lariboisière, Paris), C. Bornstain (Hôpital Intercommunal Le Raincy-Montfermeil), L. Pallot (Hôpital André Grégoire, Montreuil), H. Bismuth (Hôpital Paul Brousse, Villejuif), T. Similowski (Hôpital Pitié-Salpétrière, Paris), J. Chastre (Hôpital Pitié-Salpétrière, Paris), F. Bolgert (Hôpital Pitié -Salpétrière, Paris), J.-L. Ricome (Centre Hospitalier Intercommunal Poissy-Saint-Germain-en-Laye), H. Outin (Centre Hospitalier Intercommunal Poissy-Saint-Germain-en-Laye), F. Santoli (Hôpital Robert Ballanger, Aulnay sous Bois), D. Annane (Hôpital Raymond Poincaré, Garches), B. Guidet (Hôpital Saint-Antoine, Paris), B. Misset (Hôpital Saint-Joseph, Paris), B. Schlemmer (Hôpital Saint-Louis, Paris), L. Jacob (Hôpital Saint-Louis, Paris), M. Fartoukh (Hôpital Tenon, Paris), F. Bonnet (Hôpital Tenon, Paris), and H. Mentec (Hôpital Victor Dupouy, Argenteuil).
All questions regarding the CUB-REA network should be addressed to: B. Guidet—Service de Réanimation Médicale, Hôpital Saint-Antoine, AP-HP, Paris, France—(bertrand.guidet@aphp.fr).
Author Contributions
- Conceptualization: BC RP HM.
- Formal analysis: BC RP HM.
- Investigation: BC RP.
- Methodology: BC RP BG PA.
- Project administration: BC.
- Resources: BG PA HM.
- Supervision: HM.
- Visualization: BC RP.
- Writing – original draft: BC RP HM.
- Writing – review & editing: BC RP BG PA HM.
References
- 1.
Huckman, RS, Song H, Barro JR. "Cohort Turnover and Productivity: The July Phenomenon in Teaching Hospitals." Harvard Business School Working Paper, No. 14–113, May 2014.
- 2. Aylin P, Majeed FA. The killing season—fact or fiction? Bmj 1994, 309(6970):1690.
- 3. Buchwald D, Komaroff AL, Cook EF, Epstein AM. Indirect costs for medical education. Is there a July phenomenon? Archives of internal medicine 1989, 149(4):765–768. pmid:2495778
- 4. Young JQ, Ranji SR, Wachter RM, Lee CM, Niehaus B, Auerbach AD. "July effect": impact of the academic year-end changeover on patient outcomes: a systematic review. Annals of internal medicine 2011, 155(5):309–315. pmid:21747093
- 5. Barry WA, Rosenthal GE. Is there a July phenomenon? The effect of July admission on intensive care mortality and length of stay in teaching hospitals. Journal of general internal medicine 2003, 18(8):639–645. pmid:12911646
- 6. Finkielman JD, Morales l J, Peters SG, Keegan MT, Ensminger SA, Lymp JF, et al. Mortality rate and length of stay of patients admitted to the intensive care unit in July. Critical care medicine 2004, 32(5):1161–1165. pmid:15190967
- 7.
Décret n°88–321 du 7 avril 1988 fixant l'organisation du troisième cycle des études médicales. Journal officiel de la République Française. 1988. http://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000704452.
- 8.
Décret n°2004–67 du 16 janvier 2004 relatif à l'organisation du troisième cycle des études médicales. Journal officiel de la République Française. 2004. http://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000000781658.
- 9. Aegerter P, Auvert B, Buonamico G, Sznajder M, Beauchet A, Guidet B, et al. [Organization and quality control of a clinical database on intensive care medicine in central and suburban Paris]. Revue d'epidemiologie et de sante publique 1998, 46(3):226–237. pmid:9690289
- 10. Galbois A, Aegerter P, Martel-Samb P, Housset C, Thabut D, Offenstadt G, et al. Improved prognosis of septic shock in patients with cirrhosis: a multicenter study*. Critical care medicine 2014, 42(7):1666–1675. pmid:24732239
- 11. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Medical care 2005, 43(11):1130–1139. pmid:16224307
- 12. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of chronic diseases 1987, 40(5):373–383. pmid:3558716
- 13. Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. Jama 1993, 270(24):2957–2963. pmid:8254858
- 14.
Ripley B. Classification and Regression Trees. R package version 1.0–35 2014.
- 15. Strobl C, Malley J, Tutz G. An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological methods 2009, 14(4):323–348. pmid:19968396
- 16.
Breiman L, Cutler A. 2009. Random forests—Classification manual. URL: http://www.math.usu.edu/%04adele/forests/.
- 17. Liaw A. and Wiener M. Classification and Regression by randomForest. R News 2(3), 18–22. 2002.
- 18. Anderson KL, Koval KJ, Spratt KF. Hip fracture outcome: is there a "July effect"? American journal of orthopedics 2009, 38(12):606–611. pmid:20145785
- 19. Averbukh Y, Southern W. A "reverse july effect": association between timing of admission, medical team workload, and 30-day readmission rate. Journal of graduate medical education 2014, 6(1):65–70. pmid:24701313
- 20. Bakaeen FG, Huh J, LeMaire SA, Coselli JS, Sansgiry S, Atluri PV, et al. The July effect: impact of the beginning of the academic cycle on cardiac surgical outcomes in a cohort of 70,616 patients. The Annals of thoracic surgery 2009, 88(1):70–75. pmid:19559195
- 21. Bohl DD, Fu MC, Gruskay JA, Basques BA, Golinvaux NS, Grauer JN. "July effect" in elective spine surgery: analysis of the American College of Surgeons National Surgical Quality Improvement Program database. Spine 2014, 39(7):603–611. pmid:24384663
- 22. Dhaliwal AS, Chu D, Deswal A, Bozkurt B, Coselli JS, Lemaire SA, et al. The July effect and cardiac surgery: the effect of the beginning of the academic cycle on outcomes. American journal of surgery 2008, 196(5):720–725. pmid:18789415
- 23. Englesbe MJ, Pelletier SJ, Magee JC, Gauger P, Schifftner T, Henderson WG, et al. Seasonal variation in surgical outcomes as measured by the American College of Surgeons-National Surgical Quality Improvement Program (ACS-NSQIP). Annals of surgery 2007, 246(3):456–462; discussion 463–455. pmid:17717449
- 24. Garcia S, Canoniero M, Young L. The effect of July admission in the process of care of patients with acute cardiovascular conditions. Southern medical journal 2009, 102(6):602–607. pmid:19434039
- 25. Harring TR, Nguyen NT, Liu H, Goss JA, O'Mahony CA. Liver transplant fellowship and resident training is not a part of the "July effect". The Journal of surgical research 2013, 182(1):1–5. pmid:23478081
- 26. Hennessey PT, Francis HW, Gourin CG. Is there a "July effect" for head and neck cancer surgery? The Laryngoscope 2013, 123(8):1889–1895. pmid:23737378
- 27. Jena AB, Sun EC, Romley JA. Mortality among high-risk patients with acute myocardial infarction admitted to U.S. teaching-intensive hospitals in July: a retrospective observational study. Circulation 2013, 128(25):2754–2763. pmid:24152859
- 28. Karipineni F, Panchal H, Khanmoradi K, Parsikhia A, Ortiz J. The "July effect" does not have clinical relevance in liver transplantation. Journal of surgical education 2013, 70(5):669–679. pmid:24016380
- 29. McDonald RJ, Cloft HJ, Kallmes DF. Impact of admission month and hospital teaching status on outcomes in subarachnoid hemorrhage: evidence against the July effect. Journal of neurosurgery 2012, 116(1):157–163. pmid:21942725
- 30. Nicolas K, Raroque S, Rowland DY, Chaiban JT. Is there a "july effect" for inpatient glycemic control? Endocrine practice: official journal of the American College of Endocrinology and the American Association of Clinical Endocrinologists 2014, 20(9):919–924.
- 31. Ravi P, Trinh VQ, Sun M, Sammon J, Sukumar S, Gervais MK, et al. Is there any evidence of a "July effect" in patients undergoing major cancer surgery? Canadian journal of surgery Journal canadien de chirurgie 2014, 57(2):82–88. pmid:24666444
- 32. Shuhaiber JH, Goldsmith K, Nashef SA. Impact of cardiothoracic resident turnover on mortality after cardiac surgery: a dynamic human factor. The Annals of thoracic surgery 2008, 86(1):123–130; discussion 130–121. pmid:18573410
- 33. Yaghoubian A, de Virgilio C, Chiu V, Lee SL. "July effect" and appendicitis. Journal of surgical education 2010, 67(3):157–160. pmid:20630426
- 34. Jen MH, Bottle A, Majeed A, Bell D, Aylin P. Early in-hospital mortality following trainee doctors' first day at work. PloS one 2009, 4(9):e7103. pmid:19774078
- 35. Warraich HJ. ‘July effect’ may be the opposite of what you think. The Boston Globe 2014.
- 36. Wright M, Mankey CG, Miller BW. Improving upon the 'July effect': a collaborative, interdisciplinary orientation for internal medicine interns. Medical education online 2013, 18:23249. pmid:24369763
- 37. Phillips DP, Barker GE. A July spike in fatal medication errors: a possible effect of new medical residents. Journal of general internal medicine 2010, 25(8):774–779. pmid:20512532
- 38. Bohl DD, Fu MC, Golinvaux NS, Basques BA, Gruskay JA, Grauer JN. The "July effect" in primary total hip and knee arthroplasty: analysis of 21,434 cases from the ACS-NSQIP database. The Journal of arthroplasty 2014, 29(7):1332–1338. pmid:24631125
- 39. DiBiase LM, Weber DJ, Sickbert-Bennett EE, Denniston C, Rutala WA. July effect: impact of the academic year-end changeover on the incidence of healthcare-associated infections. Infection control and hospital epidemiology: the official journal of the Society of Hospital Epidemiologists of America 2014, 35(3):321–322.
- 40. Glick SS. Impact of the "July effect" on patient outcomes. Annals of internal medicine 2012, 156(2):167; author reply 168.
- 41. Riguzzi C, Hern HG, Vahidnia F, Herring A, Alter H. The July effect: is emergency department length of stay greater at the beginning of the hospital academic year? The western journal of emergency medicine 2014, 15(1):88–93. pmid:24578770