The aims of this study were to describe the key features of acute NHS Trusts with different levels of research activity and to investigate associations between research activity and clinical outcomes.
National Institute for Health Research (NIHR) Comprehensive Clinical Research Network (CCRN) funding and number of patients recruited to NIHR Clinical Research Network (CRN) portfolio studies for each NHS Trusts were used as markers of research activity. Patient-level data for adult non-elective admissions were extracted from the English Hospital Episode Statistics (2005-10). Risk-adjusted mortality associations between Trust structures, research activity and, clinical outcomes were investigated.
Low mortality Trusts received greater levels of funding and recruited more patients adjusted for size of Trust (n = 35, 2,349 £/bed [95% CI 1,855–2,843], 5.9 patients/bed [2.7–9.0]) than Trusts with expected (n = 63, 1,110 £/bed, [864–1,357] p<0.0001, 2.6 patients/bed [1.7–3.5] p<0.0169) or, high (n = 42, 930 £/bed [683–1,177] p = 0.0001, 1.8 patients/bed [1.4–2.1] p<0.0005) mortality rates. The most research active Trusts were those with more doctors, nurses, critical care beds, operating theatres and, made greater use of radiology. Multifactorial analysis demonstrated better survival in the top funding and patient recruitment tertiles (lowest vs. highest (odds ratio & 95% CI: funding 1.050 [1.033–1.068] p<0.0001, recruitment 1.069 [1.052–1.086] p<0.0001), middle vs. highest (funding 1.040 [1.024–1.055] p<0.0001, recruitment 1.085 [1.070–1.100] p<0.0001).
Research active Trusts appear to have key differences in composition than less research active Trusts. Research active Trusts had lower risk-adjusted mortality for acute admissions, which persisted after adjustment for staffing and other structural factors.
Citation: Ozdemir BA, Karthikesalingam A, Sinha S, Poloniecki JD, Hinchliffe RJ, Thompson MM, et al. (2015) Research Activity and the Association with Mortality. PLoS ONE 10(2): e0118253. doi:10.1371/journal.pone.0118253
Academic Editor: James D. Chalmers, University of Dundee, UNITED KINGDOM
Received: August 29, 2014; Accepted: January 4, 2015; Published: February 26, 2015
Copyright: © 2015 Ozdemir et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: The data is available from the Department of Health. Performance Data and statistics. http://webarchive.nationalarchives.gov.uk/20130107105354/http://www.dh.gov.uk/en/Publicationsandstatistics/Statistics/Performancedataandstatistics/index.htm. (accessed 01 July 2013). Health and Social Care Information Centre. Find data. http://www.hscic.gov.uk/searchcatalogue (accessed 01 July 2013). NHS England. Statistical Work Areas. http://www.england.nhs.uk/statistics/statistical-work-areas/ (accessed 01 July 2013). There is sufficient detail in the manuscript for readers to locate the datasets used in the manuscript on the websites but also at http://www.england.nhs.uk/statistics/statistical-work-areas/.
Funding: Peter Holt is a Clinician Scientist funded with a personal award by the National Institute for Health Research (NIHR) (NIHR-CS-011-008), but has no direct financial involvement with the Comprehensive Clinical Research Network (CCRN). Jonathan Gower is employed as the assistant director of the CCRN but had no influence over the analysis or results. Alan Karthikesalingam is a Clinical Lecturer supported by the Circulation Foundation Surgeon Scientist Award. All other authors have had no support from any organisation for the submitted work; no other relationships or, activities that could appear to have influenced the submitted work. The NIHR had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The views expressed in this publication are those of the authors and not necessarily those of the NIHR, NHS, or Department of Health.
Competing interests: The authors have declared that no competing interests exist.
It is widely assumed that patients cared for in research active institutions have better outcomes but evidence for this has been limited. The UK National Institute for Health Research (NIHR) was established in April 2006 to carry forward the vision and goals outlined in the 2005 ‘Best Research for Best Health’ government white paper. The overall mission of the NIHR is to create a health research system in which the NHS supports individuals working in world-class facilities, conducting leading-edge research, focused on the needs of patients and, the public.
One important element in the NIHR’s health research system is the support for, and development of, a national Clinical Research Network (CRN). This provides the infrastructure needed to support both patients and healthcare professionals to participate in research activity. Six NIHR CRNs exist for specific conditions, such as the NIHR Cancer Research Network and the NIHR Primary Care Research Network. The Comprehensive Clinical Research Network (CCRN) provides support for research activity that falls outside the scope of the other networks. Given its wide remit, the CCRN is responsible for distributing most of the NIHR Clinical Research Network funding, equalling 85% of £285m per annum, underpinning research activity across a wide range of disciplines. This translates into both direct research time for clinicians and allied health professionals (nurses, pharmacists, radiographers, laboratory staff etc.) and to research governance support. The level to which the NIHR CCRN supports individual Trusts depends, in the main, on their overall research activity.
Whilst the output of NIHR-funded studies is considerable, there are limited data to suggest a link between research activity and patient outcomes.[2–4] Furthermore, the assessment of any such association is complex.[5,6] In healthcare systems outside the NHS, there is some evidence that the formation of research networks has associations with improved health outcomes. For example, in the USA, an analysis of practice-based research networks (PBRNs) in primary care showed that improved clinical outcomes were apparent for participating practices, and suggested that this was due to a number of factors, each underpinned by infrastructure support from the PBRN. Other studies of primary care and oncology networks have supported this, demonstrating that research networks have “positive and long lasting effects”.[8,9]
Until recently the availability of data on research activity in NHS Trusts was limited. Since 2010 accurate data has been available of NIHR funding and patient recruitment to studies. The aims of this study were to utilise this data to address two questions. First, to investigate whether differences in structural and process factors could be identified between Trusts with varying levels of research activity. Second, to determine whether any association exists between research activity and mortality outcomes from English NHS Trusts for acute admissions.
Three indicators of Trust research activity were utilised, NIHR CCRN funding, number of patients recruited to NIHR CRN portfolio studies and Trust teaching status. To be eligible for inclusion in the portfolio studies must be peer-reviewed, awarded funds through a national competitive process, of discernable value to the NHS and, of high quality. Funding data was restricted to the Comprehensive Clinical Research Network. Patient recruitment to portfolio studies included all Clinical Research Networks (comprehensive, primary care, cancer, diabetes, medicines for children, stroke, dementia and neurodegenerative diseases and mental health networks). Research funding data were provided directly by the NIHR CCRN team from the financial accounts covering the English financial year’s 2010/11. The data comprised the total annual funding given to individual NHS Trusts. Number of patients recruited in 2010/11 to portfolio studies is made publically available by the NIHR annually via The Guardian newspaper website. English NHS Trusts were classified as teaching hospitals if they had a direct and specific link with a member of the Medical School Council in England.
Data on Trust medical staffing numbers, nurse staffing numbers, critical care bed provision, the use of radiological investigations and, operating theatre number were obtained from the Health and Social Care Information Centre (HSCIC).[15–17] Trust operational expense data (as defined by International Financial Reporting Standards) was obtained from trust annual reports. This data was adjusted by the Market Forces Factor which accounts for the particular difficulties of delivering care in the catchment area covered by that Trust.
Patient-level clinical, demographic, and outcomes data were obtained from the Hospital Episode Statistics (HES) 1st April 2005–31st March 2010. The HES is an administrative data warehouse that records the details of every patient admission in England and holds patient-level data on demographics, co-morbidities and, social deprivation indices. A cohort of patients admitted non-electively was taken from a previously published study, and encompassed the breadth of emergency admissions in adults to providers with an acute service. The full detail of case selection and risk models are contained in the prior publication, but, in summary, patient selection was based on primary diagnostic (ICD-10), or procedural code (OPCS-4) (S1 Table). The primary outcome measure was in-hospital death.
As the focus of this study was to examine the relationships between research activity and outcome in acute NHS Trusts, those without emergency departments and specialist institutions, such as rheumatological or women’s hospitals, were excluded. The study was restricted to Trusts in England. Analysis was performed at NHS Trust level (i.e. potentially including more than one physical site) and the term “Trust” is used synonymously with “hospital”.
Analyses were undertaken with SAS version 9.2 (SAS Institute, USA) and R (R Core Team 2013). Trusts were categorised into low mortality, expected mortality and high mortality categories of risk-adjusted inpatient death. Risk adjustment is described previously and used multi-level models incorporating patient demographics, comorbidity and social deprivation scores (based on the patients geographic area of residence) and, a random effects Trust model to account for clustering of outcomes.[19–23] Statistically significant divergence was defined as Trusts outlying the 95% confidence interval of the Poisson distribution and Trusts were subsequently classified as having mortality rates that were high, as expected or, low. Mean CCRN funding and 95% confidence intervals were calculated for each of these three strata. Means were compared with one way ANOVA and Tukey’s range test.[24,25]
Trusts were categorised into tertiles based on their level of NIHR CCRN research funding and separately based on number of patients in NIHR CRN portfolio studies. Associations between these tertiles and mortality was tested in a risk adjusted binary logistic regression model. Odds ratios and 95% confidence intervals were reported and p<0.05 was considered significant.
We used two approaches to determine if the association identified could be explained by the structural or process characteristics of Trusts studied:
- The unifactorial association of tertile of medical staffing, nurse staffing, critical care beds, radiology utilisation, operating theatres, operational expenses and, teaching status in a risk adjusted binary logistic regression model was identified. Variables with p<0.05 were entered into a multifactorial risk adjusted binary logistic regression model that included tertile of funding and tertile of patient recruitment.
- K means clustering was undertaken to categorise Trusts according to the availability or utilisation of medical staffing, nurse staffing, critical care beds, radiology utilisation and operating theatres. After re-scaling each variable partitioning around k-medoids was used to determine that English NHS Trusts were clustered into two major groups. The structural details of the two clusters are detailed in Table 1, but can be summarised as trusts characterised by a greater number of doctors, nurses, critical care beds, operating theatres and, radiodiagnostic utilisation (higher resource cluster, n = 23) and those with tendency to less of the same variables (standard resource cluster, n = 117). The unifactorial association of tertile of research funding and tertile of patients recruited was re-analysed within each of these two clusters.
The HES data contained in this study was obtained with the necessary pre- requisite permissions from the NHS Information Centre (NHS IC). The local ethics board (Wandsworth Research Ethics Committee) has confirmed that formal ethical approval is not required for HES extracts as the data source is pseudo-anonymised and publicly available (upon request). The data was stored on the St George’s University of London secure central server. Although the data are pseudo-anonymised, they fall under the regulation of the Data Protection Act and thus permission has been obtained by the Department of Outcomes Research from the local Caldicott Guardian to store and analyse the data. We did not have direct access to patient notes/records. Informed consent is not obtained from patients included in the study as the data available is pseudo-anonymised and we therefore cannot identify individual patients, though we can track patients through care spells.
156 trusts were identified of which 140 were suitable for inclusion in the analysis. There was wide variation in the overall NIHR CCRN funding (median £1,297,801, IQR £1,039,015), number of study patients (median 1070, IQR 2300), scaled NIHR CCRN funding (median £1,002/bed, IQR £1,281/bed) and, scaled study patients median (1.8, IQR 2.2) allocated to individual NHS Trusts (Table 2). Trusts in the top tertile of CCRN funding had or utilised greater levels of resource relative to Trust size than Trusts attracting lower levels of CCRN funding (Table 3). Trusts in the top tertile of patient recruitment had greater number of medical and nursing staff as well as critical care beds than trusts with lower levels of recruitment (Table 4).
Acute admission analysis
The dataset comprised 2,349,160 adult acute admissions to English NHS trusts in the five-year period. Of these, 1,852,827 were emergency medical admissions and 496,333 emergency surgical admissions. The crude inpatient mortality rate was 15.3% (further demographic details in S2 Table). The total number of patients included is slightly smaller than in our previous study using the same patient definitions, as, due to mergers/closures, 2010/11 research funding data was restricted to fewer trusts. Low mortality outlying Trusts (n = 35, 2600 £/bed, CI 2045–3154) had higher research network funding than Trusts with expected (n = 63,1278 £/bed, CI 959–1597, p<0.0001) or high outlying mortality (n = 42, 987 £/bed, CI 778–1196, p<0.0001) (Fig. 1). Low mortality outlying Trusts (n = 35, 5.9 /bed, CI 2.7–9.0) had more patients recruited than Trusts with expected (n = 63, 2.6 /bed, CI 1.7–3.5, p<0.0169) or high outlying mortality (n = 42, 1.8 /bed, CI 1.4–2.1, p<0.0005).
For each group, the mean and 95% CI funding are shown. The low mortality Trusts had significantly higher levels of CCRN funding than the as expected (p<0.0001) or high (p = 0.0001) mortality Trusts.
When Trusts were arranged in tertiles of research funding, patients treated in Trusts in the lowest (1.156 [1.144–1.168] p<0.0001) and middle (1.135 [1.123–1.147] p<0.0001) tertiles had higher odds of death than those in the top tertile (Table 5). Trusts in the lower tertiles of medical staffing, nurse staffing, critical care beds, radiodiagnostic utilisation, operating theatres, operational expenditure and, teaching hospitals also had higher odds ratios for inpatient death (Table 5).
In multivariate analysis, incorporating research funding, medical staffing, nurse staffing, critical care beds, radiodiagnostic utilisation, operating theatres, operational expenditure and, teaching hospital status, the association between research funding and mortality (lowest versus highest tertile (1.050 [1.033–1.068] p<0.0001), middle versus highest tertile (1.040 [1.024–1.055] p<0.0001)) as well as patient recruitment and mortality persisted (lowest versus highest tertile (1.069 [1.052–1.086] p<0.0001), middle versus highest tertile (1.085 [1.070–1.100] p<0.0001)) (Table 5). The goodness of fit for the multifactorial model was good with a c-statistic (area under the receiver operating characteristic curve) of 0.81. When the analysis was restricted to the standard resource cluster, Trusts in the low (funding 1.147 [1.134–1.160] p<0.0001, recruitment 1.075 [1.063–1.086]) and middle tertiles (funding 1.122 [1.110–1.135] p<0.0001, recruitment 1.072 [1.060–1.083]) of research funding (Fig. 2) and, patients recruited had higher risk-adjusted odds ratios for inpatient death than the highest tertiles. In the high resource cluster, Trusts in the low tertile of research funding (1.223 [1.189–1.258] p<0.0001) (Fig. 3), patient recruitment (1.130 [1.118–1.141]) and, middle tertile of patient recruitment (1.115 [1.104–1.126]) had higher risk-adjusted odds ratios for inpatient death than the highest tertiles.
The analysis is restricted to standard resource trusts. For each group, the mean and 95% CI are shown. Trusts in the lowest and middle funding tertile had significantly higher mortality relative to the highest funded trusts.
This study characterises differences in the structures and processes of care between trusts with differential levels of research activity. The most research active Trusts tend, relative to their size to be better staffed in terms of doctors and nurses, have more critical care beds and operating theatres and, have a greater utilisation of advanced radiodiagnostics. Trusts with the best emergency mortality outcomes were those that were most research active. The risk adjusted odds of death was lower in research active Trusts but also Trusts with more of other resources. The association of research activity with outcome could not be explained by staffing, critical care and operating theatre provision or, radiodiagnostic utilisation as the association persisted in the multifactorial models and analysis within clusters of resource.
In multi-level modelling the association of both research funding and number of patients recruited to studies to outcome was independent to teaching hospital status. These findings were consistent with previous work demonstrating no direct relationship between teaching status and healthcare outcomes, but that the number of citations attributable to a trust was correlated with Hospital Standardised Mortality Ratios (HSMR).[27,28]
Whilst there is a plausible link between research engagement and healthcare performance, little empirical evidence currently exists to support this. In particular it has been suggested that research engagement could improve the processes and outcomes of care. These results support this possibility, with research activity being associated with outcomes for acute admissions, rather than being limited only to research participants.
Jarman et al. has previously identified medical staffing to be the best predictor of outcomes for emergency admissions in 1999 but did not include a measure of research activity beyond teaching status. This study demonstrated statistically significant associations independent of teaching status, staffing and, a number of other hospital structures and process.
In terms of understanding these associations, a recent systematic review suggested that engagement with research by individuals and healthcare organisations increased the likelihood of a positive impact on healthcare performance. The number of studies was small, but possible mechanisms for better outcomes included beneficial changes in institutional structure and human capital, specific processes of care related to conducting trials, improvements in organisational mechanisms, and greater levels of collaboration between organisations, teams and, individuals. The review concluded that there was “cumulative evidence that organisations in which the research function is fully integrated into the organisational structure can out-perform other organisations that pay less formal heed to research and its outputs,” which is supported by these findings.
A benefit extending beyond direct research participants has been noted previously, and was reinforced by these pan-provider results. One possible explanation is that some attributes of the setting in which care is delivered, such as equipment and personnel, which are brought in to perform research may remain in place after the research is completed. Furthermore it has been suggested that patients in research active hospitals may have better outcomes than patients in poorly research-active hospitals because greater research participation leads to accumulated knowledge, develops infrastructure and, brings in resources that can be used to improve clinical care.[31,32] This broader impact observed in hospitals with greater network support is supported by three previous studies.[33–35] The suggestion was that hospitals within research networks implement research findings more easily and more quickly, and that clinicians were more likely to adopt evidence-based practice and, follow up-to-date clinical guidelines.
Finally, an American oncological study suggested that greater network involvement was of greater importance than teaching hospital status in the adoption of new and novel treatments, which is commensurate with these results. The reason suggested for this were that research networks act at the interface of research and quality improvement, with networks evolving into learning communities, providing grounds for generalisable solutions to clinical problems.[36,37]
Strengths and limitations
In terms of determining the reproducibility of these results, a number of different datasets were employed in the analyses, drawn from a number of sources and encompassing different information. The NIHR CCRN funding and recruitment data were provided by the NIHR from their own financial accounts and are accurate for individual trusts. Trust level structure and process data were taken from external datasets, used by the Department of Health for regulatory purposes.
The definition of teaching hospital that we utilised is a traditional one. It should be noted that the majority of district general Trusts in England are involved in teaching both doctors in training and, particularly in recent years increasing numbers of medical students.
We modelled risk adjusted mortality around adult acute admissions over a five-year period, using established methods.[19,38] Risk-standardisation used hierarchical models to account for clustering of deaths, and included widely validated factors.[39–41] The ascertainment of the mortality outcomes from the ONS Registry data adds strength to these results. A further strength of the analysis is the case selection of a group of commonly encountered pathologies ensuring that the link between mortality and quality is plausible as the conditions are amenable to salvage. For example the colorectal laparotomy group only included patients undergoing surgery therefore effectively excluding moribund patients or those with extensive disseminated disease judged to be unfit or inappropriate for surgery. Previous work has identified that mortality metrics calculated using a restricted group of pathologies reduces over-dispersion of data and therefore is superior for comparative statistics. Nevertheless highly specialised or tertiary referral centres often have a high percentage of research active clinical academics and also regional referral intakes with sicker or more complex patients. It is possible therefore that the most physiologically unwell patients (for example in myocardial infarction or pancreatitis) may be cared for in research active units. This would if anything however weaken the association of research activity with better mortality outcomes. It may however explain the unexpected finding in the multifactorial analysis that trusts in the lowest tertile of nurse staffing and critical care provision had more favourable emergency mortality outcomes. Medical staffing and nurse staffing are highly correlated in the English NHS (Spearman rank correlation coefficient of 0.81). Further secondary modelling with an interaction term between medical and nurse staffing demonstrates an interaction but the association with research activity remain (results not shown). It may be that the impact of staffing is complex and needs further separate analysis but this was not pursued, as it is not within the aims of this paper.
We characterised the categories of research funding in terms of a number of hospital structures and processes. We considered the possibility that these were confounders of the association of funding or recruited patient with outcome in our statistical analysis, but the association persisted. The possibility remains however of endogenity due to omitted co-factors. Future research will need to assess further the interaction of infrastructure with research activity. The aim of our study however was not to prove causality between funding or recruited patient number with outcome. Both were used as surrogates for research activity. Although we controlled for a number of important factors it seems likely that other unmeasured confounders (resources) could influence outcome. Inverse causality is also a possibility wherein better-resourced hospitals with better outcomes attract research funding and therefore research active. It is of note however that in our sensitivity analysis research funding and recruitment were associated with outcome even in the higher resource cluster of trusts. There may be organisational, administrative or managerial factors which impact on a Trusts ability to gain research funding, recruit patients to studies as well as patient care. This could therefore confound the relationship between research activity and outcome. Accurate funding and patient recruitment data prior to 2010/2011 was not available. Future research should study longitudinal trends in research activity, potential confounders and outcome. Such an analysis would better help to understand the mechanisms (whether causal or confounded) behind the association of research activity with outcome.
This work was limited by the use of only a single measure of research funding, NIHR CCRN core funding, and a focus on mortality as an outcome. Determining the association of the largest national funding stream on a hard, validated outcome measure would appear an intuitive starting position in a detailed examination of measuring the success of national research infrastructure. Future research should include measures of morbidity and patient reported outcome measures. More granular information with details of multiple funding streams would provide a clearer picture of how and where the greatest gains in outcome have been made. It was not possible to include a measure of research funding by industry, which can account for a significant proportion of activity in some trusts, as such data is not readily available in the United Kingdom. It is therefore plausible that the associations observed are secondary to alternative research funding streams. This would not however detract from the observation that research activity associates with clinical outcomes. Future research should investigate whether the type of studies to which Trusts recruit (interventional versus observational) is associated with differences in outcome. Accurate data for the type of study was not available for 2010/2011 but is now being collected and being made publically available. In addition, non-mortality outcomes should be investigated where data exist on a national scale.
This study provides evidence that research activity in acute English NHS Trusts is associated with lower mortality outcomes for emergency admissions. The reasons for this is not fully established. These results should act as a catalyst to understand the relationships in more detail with scope for further quantitative analyses.
S1 Appendix. Strobe Statement.
S1 Table. OPCS-4 codes used to define emergency medical conditions and emergency surgical procedures.
*NEC = not elsewhere classified, **EC = elsewhere classified.
S2 Table. Demographic data and crude inpatient mortality outcomes for acute admissions.
Disclaimer: Peter Holt is a Clinician Scientist funded with a personal award by the National Institute for Health Research (NIHR) (NIHR-CS-011-008), but has no direct financial involvement with the Comprehensive Clinical Research Network (CCRN). Jonathan Gower is employed as the assistant director of the CCRN but had no influence over the analysis or results. All other authors had no other relationships or, activities that could appear to have influenced the submitted work. The NIHR had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The views expressed in this publication are those of the authors and not necessarily those of the NIHR, NHS, or Department of Health.
Conceived and designed the experiments: BAO AK SS JDP AB JDG MMT RJH PJH. Performed the experiments: BAO SS JDP PJH. Analyzed the data: BAO AK AB SS JDP MMT RJH PJH. Contributed reagents/materials/analysis tools: BAO SS JDP AB JDG. Wrote the paper: BAO AK SS JDP JDG MMT RJH PJH AB.
- 1. Research and Development Directorate. Best Research for Best Health. A new national health research strategy. Department of Health. 2006.
- 2. Wright D, Milne R, Price A, Tose N. Assessing the international use of health technology assessments: exploring the merits of different methods when applied to the National Institute of Health Research Health Technology Assessment (NIHR HTA) programme. Int J Technol Assess Health Care. 2013;29: 192–197. doi: 10.1017/S026646231300010X. pmid:23552058
- 3. Hanney S, Buxton M, Green C, Coulson D, Raferty J. An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess. 2007;11: iii-iv, ix-xi, 1–180. doi: 10.3310/hta11530
- 4. Hanney S, Boaz A, Jones T, Soper B. Engagement in research: an innovative three-stage review of the benefits for health-care performance. Health Serv and Deliv Res. 2013;1(8). doi: 10.3310/hsdr01080
- 5. Druss BG, Marcus SC. Tracking publication outcomes of National Institutes of Health grants. Am J Med. 2005;118: 658–663. pmid:15922698 doi: 10.1016/j.amjmed.2005.02.015
- 6. Ward V, House A, Hamer S. Developing a framework for transferring knowledge into action: a thematic analysis of the literature. J Health Serv Res Policy. 2009;14: 156–164. doi: 10.1258/jhsrp.2009.008120. pmid:19541874
- 7. Rhyne R, Sussman AL, Fernald D, Weller N, Daniels E, Williams RL, et al. Reports of persistent change in the clinical encounter following research participation: a report from the Primary Care Multiethnic Network (PRIME Net). J Am Board Fam Med. 2011;24: 496–502. doi: 10.3122/jabfm.2011.05.100295. pmid:21900432
- 8. Siegel RM, Bien J, Lichtenstein P, Davis J, Khoury JC, Knight JE, et al. A safety-net antibiotic prescription for otitis media: the effects of a PBRN study on patients and practitioners. Clin Pediatr (Phila). 2006;45: 518–524. pmid:16893856 doi: 10.1177/0009922806290567
- 9. Warnecke RB, Johnson TP, Kaluzny AD, Ford LG. The community clinical oncology program: its effect on clinical practice. Jt Comm J Qual Improv. 1995;21: 336–339. pmid:7581736
- 10. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet. 2007;370: 1453–1457. pmid:18064739 doi: 10.1016/s0140-6736(07)61602-x
- 11. National Institute for Health Research website. Eligibility Criteria for NIHR CRN support. Available: http://www.crn.nihr.ac.uk/wp-content/uploads/About%20the%20CRN/Eligibility%20Criteria%20for%20NIHRCRN%20support.pdf.Accessed 17 July 2014.
- 12. National Institute for Health Research Comprehensive Clinical Research Network. NIHR CCRN Funding Data. In: Department of Health, ed. Leeds, 2012.
- 13. The Guardian website. Clinical Research Zone. Trust Research Activity League Tables. Available: http://www.theguardian.com/healthcare-network-nihr-clinical-research-zone/table/trust-research-activity-league-tables?guni=Article:in%20body%20link. Accessed 01 July 2013.
- 14. The Guardian website. Healthcare Professionals Network. England’s NHS teaching hospitals: a capital affair. Available: http://www.theguardian.com/healthcare-network/2011/jun/29/england-nhs-teaching-hospitals-london-capital. Accessed 01 July 2013.
- 15. Department of Health website. Performance Data and statistics. Available: http://webarchive.nationalarchives.gov.uk/20130107105354/http://www.dh.gov.uk/en/Publicationsandstatistics/Statistics/Performancedataandstatistics/index.htm. Accessed 01 July 2013.
- 16. Health and Social Care Information Centre website. Find data. Available: http://www.hscic.gov.uk/searchcatalogue. Accessed 01 July 2013.
- 17. NHS England website. Statistical Work Areas. Available: http://www.england.nhs.uk/statistics/statistical-work-areas/. Accessed 01 July 2013.
- 18. data.gov.uk website. NHS Payment by Results 2010–2011. Available: http://data.gov.uk/dataset/payment-by-results-2010-11-national-tariff-information. Accessed 20 November 2014.
- 19. Holt PJ, Sinha S, Ozdemir BA, Karthikesalingam A, Poloniecki JD, Thompson MM. Variations and inter-relationship in outcome from emergency admissions in England: a retrospective analysis of Hospital Episode Statistics from 2005–2010. BMC Health Serv Res. 2014;14: 270. doi: 10.1186/1472-6963-14-270. pmid:24947670
- 20. Holt PJ, Poloniecki JD, Loftus IM, Michaels JA, Thompson MM. Epidemiological study of the relationship between volume and outcome after abdominal aortic aneurysm surgery in the UK from 2000 to 2005. Br J Surg. 2007;94: 441–448. pmid:17385180 doi: 10.1002/bjs.5725
- 21. Moxey PW, Hofman D, Hinchliffe RJ, Jones K, Thomspon MM, Holt PJ. Epidemiological study of lower limb amputation in England between 2003 and 2008. Br J Surg. 2010;97: 1348–1353. doi: 10.1002/bjs.7092. pmid:20632310
- 22. Moxey PW, Hofman D, Hinchliffe RJ, Poloniecki J, Loftus IM, Thompson MM, et al. Volume-outcome relationships in lower extremity arterial bypass surgery. Ann Surg. 2012;256: 1102–1107. doi: 10.1097/SLA.0b013e31825f01d1. pmid:22824857
- 23. Sinha S, Hofman D, Stoker DL, Friend PJ, Poloniecki JD, Thompson MM, et al. Epidemiological study of provision of cholecystectomy in England from 2000 to 2009: retrospective analysis of Hospital Episode Statistics. Surg Endosc. 2013;27: 162–175. doi: 10.1007/s00464-012-2415-0. pmid:22806509
- 24. Everitt BS, Hothorn T. A Handbook of Statistical Analyses Using R. 2nd ed. Boca Raton: Chapman and Hall/CRC; 2009. Available: http://cran.r-project.org/web/packages/HSAUR2/vignettes/Ch_analysis_of_variance.pdf. Accessed 05 August 2013.
- 25. R Documentation website. Compute Tukey Honest Significant Differences. Available: https://stat.ethz.ch/R-manual/R-patched/library/stats/html/TukeyHSD.html. Accessed 05 August 2013.
- 26. Hosmer DW, Lemeshow S. Applied Logistic Regression. 2nd ed. New York, NY: John Wiley & Sons, Inc; 2000.
- 27. Bennett WO, Bird JH, Burrows SA, Counter PR, Reddy VM. Does academic output correlate with better mortality rates in NHS trusts in England? Public Health. 2012;126 Suppl 1: S40–S43. doi: 10.1016/j.puhe.2012.05.021. pmid:22795835
- 28. Papanikolaou PN, Christidi GD, Ioannidis JP. Patient outcomes with teaching versus nonteaching healthcare: a systematic review. PLoS Med. 2006;3: e341. pmid:16968119 doi: 10.1371/journal.pmed.0030341
- 29. Jarman B, Gault S, Alves B, Hidler A, Dolan S, Cook A, et al. Explaining Differences in English Hospital Death Rates Using Routinely Collected Data. BMJ. 1999;318: 1515–1520. pmid:10356004 doi: 10.1136/bmj.318.7197.1515
- 30. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360: 491–499. doi: 10.1056/NEJMsa0810119. pmid:19144931
- 31. Rochon J, du Bois A. Clinical research in epithelial ovarian cancer and patients' outcome. Ann Oncol. 2011;22 Suppl 7: vii16–vii19. doi: 10.1093/annonc/mdr421. pmid:22039139
- 32. Christmas C, Durso SC, Kravet SJ, Wright SM. Advantages and challenges of working as a clinician in an academic department of medicine: academic clinicians' perspectives. J Grad Med Educ. 2010;2: 478–484. doi: 10.4300/JGME-D-10-00100.1. pmid:21976102
- 33. Abraham AJ, Knudsen HK, Rothrauff TC, Roman PM. The adoption of alcohol pharmacotherapies in the Clinical Trials Network: the influence of research network participation. J Subst Abuse Treat. 2010;38: 275–283. doi: 10.1016/j.jsat.2010.01.003. pmid:20117908
- 34. Carpenter WR, Reeder-Hayes K, Bainbridge J, Meyer AM, Amos KD, Weiner BJ, et al. The role of organizational affiliations and research networks in the diffusion of breast cancer treatment innovation. Med Care. 2011;49: 172–179. doi: 10.1097/MLR.0b013e3182028ff2. pmid:21206296
- 35. Laliberte L, Fennell ML, Papandonatos G. The relationship of membership in research networks to compliance with treatment guidelines for early-stage breast cancer. Med Care. 2005;43: 471–479. pmid:15838412 doi: 10.1097/01.mlr.0000160416.66188.f5
- 36. Mold JW, Peterson KA. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann Fam Med. 2005;3 Suppl 1: S12–S20. pmid:15928213 doi: 10.1370/afm.303
- 37. Lindbloom EJ, Ewigman BG, Hickner JM. Practice-based research networks: the laboratories of primary care research. Med Care. 2004;42: III45–III49. pmid:15026664 doi: 10.1097/01.mlr.0000119397.65643.d4
- 38. Sinha S, Karthikesalingam A, Poloniecki JD, Thompson MM, Holt PJ. Inter-Relationship of Procedural Mortality Rates in Vascular Surgery in England Retrospective Analysis of Hospital Episode Statistics From 2005 to 2010. Circ Cardiovasc Qual Outcomes. 2014;7: 131–141. doi: 10.1161/CIRCOUTCOMES.113.000579. pmid:24399331
- 39. Armitage JN, van der Meulen JH, Royal College of Surgeons Co-morbidity Consensus Group. Identifying co-morbidity in surgical patients using administrative data with the Royal College of Surgeons Charlson Score. Br J Surg. 2010;97: 772–781. doi: 10.1002/bjs.6930. pmid:20306528
- 40. Aylin P, Bottle A, Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ. 2007;334: 1044. pmid:17452389 doi: 10.1136/bmj.39168.496366.55
- 41. DeLong E. Hierarchical modeling: its time has come. Am Heart J. 2003;145: 16–18. pmid:12514649 doi: 10.1067/mhj.2003.27
- 42. Bottle A, Jarman B, Aylin P. Hospital standardized mortality ratios: Sensitivity analyses on the impact of coding. Health Serv Res. 2011;46: 1741–1761. doi: 10.1111/j.1475-6773.2011.01295.x. pmid:21790587