Correction
23 Aug 2018: Hodgson LE, Roderick PJ, Venn RM, Yao GL, Dimitrov BD, et al. (2018) Correction: The ICE-AKI study: Impact analysis of a Clinical prediction rule and Electronic AKI alert in general medical patients. PLOS ONE 13(8): e0203183. https://doi.org/10.1371/journal.pone.0203183 View correction
Figures
Abstract
Background
Acute kidney injury (AKI) is assoicated with high mortality and measures to improve risk stratification and early identification have been urgently called for. This study investigated whether an electronic clinical prediction rule (CPR) combined with an AKI e-alert could reduce hospital-acquired AKI (HA-AKI) and improve associated outcomes.
Methods and findings
A controlled before-and-after study included 30,295 acute medical admissions to two adult non-specialist hospital sites in the South of England (two ten-month time periods, 2014–16); all included patients stayed at least one night and had at least two serum creatinine tests. In the second period at the intervention site a CPR flagged those at risk of AKI and an alert was generated for those with AKI; both alerts incorporated care bundles. Patients were followed-up until death or hospital discharge. Primary outcome was change in incident HA-AKI. Secondary outcomes in those developing HA-AKI included: in-hospital mortality, AKI progression and escalation of care. On difference-in-differences analysis incidence of HA-AKI reduced (odds ratio [OR] 0.990, 95% CI 0.981–1.000, P = 0.049). In-hospital mortality in HA-AKI cases reduced on difference-in-differences analysis (OR 0.924, 95% CI 0.858–0.996, P = 0.038) and unadjusted analysis (27.46% pre vs 21.67% post, OR 0.731, 95% CI 0.560–0.954, P = 0.021). Mortality in those flagged by the CPR significantly reduced (14% pre vs 11% post intervention, P = 0.008). Outcomes for community-acquired AKI (CA-AKI) cases did not change. A number of process measures significantly improved at the intervention site. Limitations include lack of randomization, and generalizability will require future investigation.
Conclusions
In acute medical admissions a multi-modal intervention, including an electronically integrated CPR alongside an e-alert for those developing HA-AKI improved in-hospital outcomes. CA-AKI outcomes were not affected. The study provides a template for investigations utilising electronically generated prediction modelling. Further studies should assess generalisability and cost effectiveness.
Citation: Hodgson LE, Roderick PJ, Venn RM, Yao GL, Dimitrov BD, Forni LG (2018) The ICE-AKI study: Impact analysis of a Clinical prediction rule and Electronic AKI alert in general medical patients. PLoS ONE 13(8): e0200584. https://doi.org/10.1371/journal.pone.0200584
Editor: Emmanuel A. Burdmann, University of Sao Paulo Medical School, BRAZIL
Received: March 5, 2018; Accepted: June 28, 2018; Published: August 8, 2018
Copyright: © 2018 Hodgson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Grants from the Small Business Research Initiative (National Institute for Health Research Devices for Dignity HTC partnered with the Department of Health) and the British Renal Society / British Kidney Patient Association, 14-013 (LEH) helped fund integration and implementation of the technology into Patientrack from the Hospital Trust’s electronic servers. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Acute kidney injury (AKI) is a clinical syndrome, defined through an acute increase in serum creatinine (SCr) and/or reduction in urine volume. However, there is a continuum of injury before sufficient loss of excretory kidney function can be detected with a change in SCr [1]. Up to 20% of adults admitted acutely to hospital meet AKI KDIGO (Kidney Disease: Improving Global Outcomes) criteria at, or during their stay, with high associated mortality [2]. This may reflect the limited ability of conventional markers to detect renal injury in a timely fashion, as well as heterogeneity in aetiology, which in turn provides some explanation as to why clinical trials of therapeutic interventions have proved disappointing [3].
There has been an international call to address deficits in recognition and management of patients with AKI, including early risk assessment at which point interventions could be most beneficial [1, 4, 5]. Implementation of clinical prediction rules (CPRs) is one strategy to improve risk assessment [6, 7], which relate multiple predictors, such as medical history and diagnostic tests, to risk of an outcome [8]. Unfortunately few CPRs have undergone impact analyses to investigate whether implementation is beneficial, with no such studies in the field of hospital-acquired AKI (HA-AKI). A second strategy, investigated predominantly by before-after observational studies, is the use of AKI e-alerts, with process measure improvements, such as reduced use of nephrotoxic medications, inconsistently reported [7]. Two studies from the UK reported reduced in-patient mortality and AKI progression in patients who had a linked care bundle completed [9, 10], and a large US before-after study suggested a significant reduction in mortality with the introduction of an AKI alert [11]. However, the only randomized controlled trial (RCT) reported to date utilising an e-alert showed no benefit [12]. An NHS patient safety alert in 2014 mandated English hospitals deploy an e-alert upon AKI development [13], however, uncertainty around the relative benefits of alerts and CPRs remains, necessitating ongoing rigorous evaluation [7, 14].
This impact analysis study employed one of the few available externally validated AKI CPRs–the Acute Kidney Injury Prediction Score (APS)—derived in a UK general medical population, which combines data on age, medical history and physiological parameters [15, 16]. The aims of the study included:
- In emergency medical admissions, without identifiable AKI at admission, could introduction of an electronically generated AKI CPR reduce the incidence of HA-AKI,
- In concert with the CPR, does an e-alert for new HA-AKI reduce associated complications including in-hospital mortality, escalation to intensive care and maximal AKI stage,
- Does identification of a group at high risk of developing HA-AKI improve outcomes,
- What is the effect of e-alerting community-acquired AKI (CA-AKI) at admission to hospital?
Methods
Ethical approval was given by NHS Research Ethics Committee London—City Road and Hampstead (REC reference 13/LO/0960) and the study registered with clinicaltrials.org (NCT03047382) (S1 Protocol and S1 TREND Checklist available as supplementary files) [17]. A prospective controlled before-after impact analysis of the APS CPR and AKI e-alert was performed on two adult general acute medical units. Part of the same NHS Trust, the two sites were split into intervention and control and the study ran over two ten-month periods (2014–2016). There was a bed down period for the intervention of two months whilst the technology was safely introduced (Table A in S1 File for variables and weightings of the APS CPR). The Trust is a non-specialist hospital organisation of 870 beds, with the two sites 20 miles apart with similar catchment areas on the South Coast of England, though the control site has a higher rural population and less socio-economic deprivation [18]. Combined annual emergency department attendance is over 135,000 with 40–60 acute medical admissions per 24-hour period. The APS CPR was derived at the intervention site [15]. There was no cross-site contamination of clinical staff during the study: Consultant physicians and nursing staff do not work on both sites, nor do junior staff rotate between the hospitals (see Table 1 for demographic and clinical characteristics of the emergency admissions).
At admission, all in-patients have physiological observations measured and entered via handheld systems into the clinical data software system (Patientrack Sydney, NSW, Australia). Previous ICD-10 coded history (heart failure, liver disease and diabetes mellitus), were retrieved and chronic kidney disease (CKD) was defined as an eGFR <60 mls/min/1.73m2 prior to admission (in those with an available baseline SCr); baseline SCr was defined using NHS England’s National algorithm [19]. KDIGO criteria for AKI was employed using only SCr as few of this patient group have routine urinary catheters placed (SCr increase of ≥1.5 from baseline value or absolute value ≥354μmol/l for CA-AKI; increase ≥1.5 from admission value or ≥26.5μmol/l within a rolling 48 hours during the first seven days of admission for HA-AKI). If no baseline was available, CA-AKI was assumed absent unless SCr was ≥354μmol/l. Patients were included if ≥18 years of age, stayed at least one night in the medical unit and had at least one SCr repeated.
Exclusion criteria:
- Patients moved directly from the A&E department to the ICU (as neither area uses the Patientrack data system),
- Aged <18 years,
- Non-medical (General Surgery, Trauma & Orthopaedics, Obstetrics and Gynaecology) admissions and,
- Discharged without spending a night in hospital.
Patients were followed-up until hospital discharge or in-hospital death (Fig 1 summarises the flow of participants in the study periods). As all patients admitted had an APS calculated automatically by the Patientrack system before any outcome had occurred, there were no missing demographic or physiological data at admission.
CA-AKI–Community-acquired AKI, HA-AKI–Hospital-acquired AKI, SCr–serum Creatinine.
Intervention
The intervention is summarised in Fig 2. At admission those with an APS ≥5 points were flagged ‘at risk of AKI’ (coloured AMBER) in multiple electronic locations (Fig 3) and a care bundle generated (Tables B and C In S1 File). Those with an APS <5 points were classed GREEN, with no alert. This cut-off was chosen for its relatively high specificity and positive predictive value in derivation and validation, aiming to minimise false-positives that may limit uptake [15, 16]. Patients with CA-AKI were flagged RED, as were all who subsequently developed HA-AKI, with care bundles generated. Response was multidisciplinary: ward physicians, nurses, health care assistants and pharmacists, together with the critical care outreach nurse team reviewing patients remotely or at the bedside. None of the researchers involved in data analysis were involved in management of the patients and had access only to the fully anonymised individual level data and were blinded to other patient data, as well as components of the calculated APS score.
Summary of intervention. At the control site no alerts were generated. RED boxes = AKI (community or hospital-acquired), AMBER = APS ≥5 points–cut-off for flagging patient at risk of AKI, GREEN box–all other patients (APS <5 points). HA-AKI–Hospital-acquired AKI, PAS–Patient administration system, SCr–serum creatinine. Patientrack AKI ALERT* ^AKI Prediction Score (APS)–clinical prediction rule.
Top left (A): Patient tile indicating presence and stage of AKI, baseline SCr and care bundle task status; top right (B): electronic observation chart with AKI status (present or at risk) in top right of screen with link to SCr results; bottom left (C): graph of SCr; bottom right (D): AKI care bundle. SCr–serum creatinine. Note that details for illustration only and are not of a real patient.
Barriers to integration and acceptance of CPRs and clinical decision support systems (CDSS) and expert guidance were considered prior to incorporation into the clinical workflow as part of a multi-faceted AKI guideline implementation [20, 21]. The controlled before-after design has been reported in previous impact analysis studies [22, 23]. The mandated NHS England patient safety alert limited the timeframe over which the study, a natural experiment, could be performed before the e-alert was introduced at the second (control) site.
Outcomes
The primary outcome to assess the APS CPR was change in incidence of HA-AKI (KDIGO change in SCr) within seven days of hospital admission following intervention when adjusting for baseline differences between intervention and control sites. Other outcomes in those who developed HA-AKI included mortality in-hospital and seven days following admission, maximal increase in AKI KDIGO Stage and in SCr from admission, and ICU escalation. These outcomes were also assessed in patients highlighted by the CPR at high AKI risk (AMBER) and for patients with CA-AKI. Process measures were collected to attempt to explain the effects of the intervention. These included drug prescribing, completion of the suggested care bundles and a notes review of a randomly selected collection of HA-AKI cases.
Statistical analysis
To avoid potential bias due to existing background differences between hospitals and temporal changes, we adopted the difference-in-differences approach by including time variable (before-after) and its interaction with the hospitals [24–29]. All models were adjusted by age and co-morbidities (ICD-10 diagnoses of heart failure, CKD, vascular disease, hypertension, diabetes mellitus and liver disease). The estimates of the intervention with a time period interaction term with associated 95% confidence intervals were reported. Difference-in-differences analysis makes two main assumptions [26]. First, the parallel trends assumption holds that intervention and control groups have equivalent trends for the outcome(s) prior to the start of the intervention. This implies that, without treatment, outcome(s) would be expected to change at the same rate. Secondly, the common shocks assumption infers that any event(s) occurring during or after the intervention will equally affect the intervention and control groups. To test whether the groups’ case mix did not change differentially over time, we examined for differences between patient characteristics before and after implementation in both groups (Table 1). Generalised estimating equations (GEE) with logistic link function were chosen to estimate the risks of binary outcomes between the intervention and control hospitals. GEE takes into account the clustering effects of multiple hospital admissions for the same individuals [30]. Generalised linear mixed models were employed to analyses continuous variables (change in SCr and hospital length of stay) including individual patients as a random effect [27]. Categorical outcomes were evaluated with the Cochran-Mantel-Haenszel χ2 tests. T-tests or Mann-Whitney U tests were used to compare continuous variables. Data were extracted fully anonymised on Microsoft® Excel® and all analyses were conducted using SAS (V9.4®).
Results
Over the 20-month study period 30,295 admissions were analysed (Fig 1) with a mean age of 74.5 (standard deviation ±17.0), length of stay 11 days (±13.1) and in-hospital mortality 8.3%. AKI was present on admission (CA-AKI) in 8.3% (n = 2,502) and 7.3% (n = 2,040) developed HA-AKI. Table 1 shows the sites were relatively well matched at admission pre and post intervention, though the intervention site had higher lengths of stay, age, admission tachypnoea and some coded co-morbidities (liver disease, hypertension and vascular disease). Across all medical admissions between periods, at both sites there was no overall change in mortality, whilst length of stay reduced (Table D in S1 File). More patients were escalated to ICU at the intervention site in the second period (P = 0.013) though on difference-in-differences analysis there was no significant change in mortality or escalation to ICU.
Incidence of HA-AKI
Difference-in-differences analysis suggested a reduction in HA-AKI at the intervention site (OR 0.990, 95% CI 0.981–1.000, P = 0.049). Unadjusted incidence of HA-AKI did not change significantly at the intervention (8.28% pre, 7.73% post intervention, OR 0.928, 95% CI 0.824–1.045, P = 0.223) or control sites (6.55% pre, 6.67% post intervention, OR 1.019, 95% CI 0.888–1.170, P = 0.805) (Table 2). In patients with only a single admission during the period of study, HA-AKI significantly reduced at the intervention site (9.4% pre vs 7.1% post intervention, OR 0.739, 95% CI 0.618–0.884, P = 0.001), with no change at the control site (6.7% pre, 6.2% post intervention, OR 0.910, 95% CI 0.735–1.127, P = 0.412).
HA-AKI outcomes
In cases of new HA-AKI in-patient mortality, using difference-in-differences analysis, significantly reduced at the intervention site (OR 0.924, 95% CI 0.858–0.996, P = 0.038) (Fig 4). Unadjusted in-patient mortality decreased at the intervention site (27.46% pre vs 21.67% post, OR 0.731, 95% CI 0.560–0.954, P = 0.021) with no change at control site (22.92% pre vs 24.72% post, OR 1.104, 95% CI 0.807–1.511, P = 0.576) (Table 3). A similar reduction in mortality by day seven was found. ICU escalation, progression to stage 3 AKI and maximal increase in SCr reduced at the intervention site, whilst increasing at the control site, without reaching statistical significance on difference-in-differences analysis. Length of stay did not change at either site.
HA-AKI–hospital-acquired AKI, OR–odds ratio (95% CI).
At the intervention site, of those flagged as high risk (AMBER alert) 14% went on to develop HA-AKI vs 5% with no flag (GREEN), P<0.001. Following the intervention, in-patient mortality in this high-risk amber group significantly decreased (14% pre vs 11% post, OR 0.78, 0.66–0.94, P = 0.008) with no change at control site (Table 4 and Table E in S1 File). There were no significant changes in outcomes in those with CA-AKI at either site including mortality and AKI progression (Table 4 and Table E in S1 File). Process measure improvements at the intervention site vs control included an increased stopping of potentially nephrotoxic medications (P<0.001) and AKI documentation (P = 0.033) (Tables F and G in S1 File). Electronic bundle compliance (submission of a complete bundle) was 26% in those with AKI and 15% in all those flagged at risk.
Discussion
This is the first impact analysis study combining an AKI CPR and an AKI e-alert in hospitalised patients. The intervention was associated with a decrease in the incidence of new AKI but, importantly a significant improval in survival in those individuals who developed HA-AKI was also found. This may reflect a synergistic benefit of highlighting those at risk and de novo AKI, implying best practice would be institution of a care bundle in both groups, in tandem with a multimodal service improvement approach. Other outcome measures including escalation of care and progression of AKI were suggested. There was no change in outcome following introduction of an e-alert for patients with established AKI at admission to hospital, indicating the measures introduced did not affect the underlying disease process.
The primary aim of the study was to investigate whether an AKI CPR could prevent HA-AKI. On a difference-in-differences analysis a reduction was achieved. The treatment effect may have been limited due to a lack of specificity in the CPR—in derivation and validation the APS had moderate discrimination (AUROC of 0.65–0.72) and to avoid alert fatigue, only patients with a score ≥5 were alerted to [15, 16]. Furthermore, as AKI was only diagnosed through change in SCr, known to lag behind insult [31], it is possible a proportion classed as HA-AKI reflected community-acquired injury. Further important findings are that both in those highlighted as at high risk of developing AKI (AMBER) and in patients who subsequently developed HA-AKI, improved mortality outcomes were found, which may reflect systematic early recognition, with prompt care initiated even without documented evidence of a completed care bundle. Indeed without these alerts many of these patients would not have triggered an urgent response: for example median national early warning score (NEWS) was only 2 in the AMBER group, yet mortality ranged 11–14%. Based on the NEWS alone, such patients would not have been flagged high risk. This study provides evidence that application of best practice can translate into improved outcomes even in a heterogenous group at high risk of complications.
Comparison with other studies
To date, the only RCT of an AKI e-alert showed no outcome benefit, though the intervention in that study was limited to an electronic link to practice guidelines [12]. In two UK studies, in patients who had developed AKI, a mortality benefit was reported in the 22–26% of those with a completed care bundle [9, 10]. A recent large before-after study in the US suggested a significant mortality benefit for the use of an AKI flag [11]. A before-after study where an alert was delivered the day after AKI recognition, found improved appropriate renal function monitoring, increased nephrology consultations and found less severe AKI after the intervention. However the groups had significant differences and there was no contemporaneous control.[32] One cluster RCT of a CDSS targeting patients with impaired renal found a significant increase in appropriate adjusting of prescribing which supports our finding of improved prescribing.[33] Though rarely reported, other CPR impact analysis studies have described successful integration into a CDSS [34, 35].
Intervention uptake
The majority of studies reporting on the effects of alerts incorporated as part of CDSSs are medication related [36], however, such recommendations are often ignored [37–39]. Relatively little evidence explains why some succeed, with a paucity of evidence to demonstrate convincing improvement in patient outcomes [21, 40–42]. However, two systematic reviews [21, 43], suggested three independent factors influence success:
- automatic provision of decision support as part of clinician workflow at the time and location of decision-making,
- provision of recommendations rather than just assessments and,
- computer based decision support.
These factors and others were considered during study design, which provides a proof of concept for future investigations of complex healthcare interventions. Alerts could be quickly acknowledged, with a single alert at admission for those at risk, with a further alert on patients who met AKI criteria within the nursing clinical workflow [21, 44]. A cut-off on the APS CPR was chosen to minimise false positives to avoid blunting and eventual elimination of responses [45, 46]. Optimal visual field positioning and the use of appropriate colours (red and amber) were chosen as they are associated with an increase in hazard perception and prioritisation [47–50]. Despite these measures, compliance with electronic submission of care bundles was low ranging 15–26%, similar to previous AKI studies [9, 51, 52]. This could have been improved by making the alert interruptive however, this may have increased frustration and alert fatigue [9, 53, 54]. Physicians were expected to submit the completed bundle however it was not part of their workflow. However, crucially the intervention was multimodal: highly visible flagging in multiple electronic areas accessed by all members of the clinical team, alongside education, that may have changed group behavior not captured by a formal bundle submission. Whilst establishing conclusive proof of behavioural change is challenging, we found significant improvement in process measures, such as stopping potential nephrotoxic medication.
Study strengths and weaknesses
This natural experiment introducing an electronic alert intervention at one site within a similar geographical area to a control site allows for unrelated change in disease occurrence and practice, with adjustment for confounders and has been described in previous impact studies [22, 23]. HA-AKI associated mortality at the two sites (20–26%) was similar to the other UK study showing a mortality benefit [9]. The process measure evaluation identified increased documentation, coding of AKI and stopping of nephrotoxic medications at the intervention site, which adds to the main data analysis suggesting both improved recognition and clinician behaviours following intervention (Tables F and G in S1 File).
The study took place as part of an information technology service improvement intervention that did not allow for a RCT design, with selection bias a concern. Between sites in both periods overall mortality was higher at the intervention site, as was the rate of HA-AKI, which may reflect differences in co-morbidities and socioeconomic factors. However, the before-after controlled design using difference-in-difference analyses is accepted as a way of detecting the effects of an intervention, and to control for confounders and secular trends [55, 56]. Outcomes did not improve in the control hospital suggesting other external factors did not significantly influence outcomes seen at the intervention site. However, given the difficulty of eliminating bias, a single study is unlikely to be definitive and replication and synthesis of evidence across studies is needed to support inferences about effectiveness [57]. Future studies of such organizational interventions could include a cluster RCT design to address for potential limitations of the presented study. A lack of power from two sites over ten-month periods could explain why outcomes, including ICU escalation and AKI severity, though improving at the intervention site, did not reach statistical significance–though for example in the second period significantly fewer patients with HA-AKI were escalated than at the control site. Generalisability may be limited by the study population on the South-Coast of England, being older (mean age 74) than the English NHS average admitted under general medicine (65 years), though the National average geriatric admission age is 78 [58]. Both sites are non-specialist general hospitals and the analysis used medical patients–future studies could examine generalisability to different hospital populations. However, our population probably reflects the majority of acute admissions in the UK given that the mean age of patients with CA-AKI in the RISK study being 75 years (unpublished, personal communication with study authors). The study did not assess whether impact was sustained or whether the intervention could be successfully introduced at the control site–the recommended fourth phase of impact analysis and implementation for example through a time-series design [20]. Calculation of the APS CPR past history relied on previous coded events attending hospital and could therefore have been incomplete. The predictive accuracy of the CPR could be improved by planned linkage with primary care as well as provide information of GP prescribed drugs, known to be associated with an increased risk of AKI.
Future directions
The concept of a continuously learning healthcare system was articulated by the Institute of Medicine in the US as a way of showing how evidence informs practice and vice versa in an iterative process of innovation and collaboration between research, information technology and clinical practice (summarized Fig 5) [59–61]. This study provides new insights in this emerging field, and future studies could build on knowledge for example by utilising more comprehensive records and dynamic data on physiological and blood parameters in the acute setting. This could enable the electronic record to display risk stratification at multiple time points from primary care to hospital admission with subsequent updating during an in-patient stay. Trends could provide powerful data overcoming limitations of current prediction models derived at a single time point. This study suggested a benefit from combining a CPR with an e-alert for HA-AKI. However, NHS England has required AKI alerting for patients presenting to hospital and this study does not provide evidence to support this to be sufficient to improve outcomes.
Conclusions
In an impact analysis performed on general medical admissions, introduction of an AKI CPR was associated with a reduction in HA-AKI. In the context of a multi-faceted intervention, including an electronically generated care bundle for patients at risk of AKI using a CPR and e-alert for patients with HA-AKI, a significant reduction in mortality was demonstrated. An e-alert had no affect on outcomes for patients admitted with established AKI from the community. Further studies in the field are warranted to confirm or refute these findings and assess for generalisability to other hospital populations, healthcare systems, and for sustainability.
Acknowledgments
Borislav Dimitrov passed away before the submission of the final version of this manuscript. Professor Lui Forni accepts responsibility for the integrity and validity of the data collected and analyzed. Thanks to Mr Tim Short, Mr Mark Dennis and staff at Patientrack for invaluable help with IT integration and data retrieval.
References
- 1.
KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney Int 2012;Suppl 2(1):1–136. Epub 2012/03/01. https://doi.org/10.1038/kisup.2012.6 pmid:25018915; PubMed Central PMCID: PMCPMC4089619.
- 2. Susantitaphong P, Cruz DN, Cerda J, Abulfaraj M, Alqahtani F, Koulouridis I, et al. World incidence of AKI: a meta-analysis. Clin J Am Soc Nephrol. 2013;8(9):1482–93. Epub 2013/06/08. pmid:23744003; PubMed Central PMCID: PMCPMC3805065.
- 3. Joannidis M, Druml W, Forni LG, Groeneveld ABJ, Honore PM, Hoste E, et al. Prevention of acute kidney injury and protection of renal function in the intensive care unit: update 2017: Expert opinion of the Working Group on Prevention, AKI section, European Society of Intensive Care Medicine. Intensive Care Med. 2017;43(6):730–49. Epub 2017/06/04. pmid:28577069; PubMed Central PMCID: PMCPMC5487598.
- 4. Harel Z, Wald R, Bargman JM, Mamdani M, Etchells E, Garg AX, et al. Nephrologist follow-up improves all-cause mortality of severe acute kidney injury survivors. Kidney Int. 2013;83(5):901–8. Epub 2013/01/18. pmid:23325077.
- 5. Mehta RL, Cerda J, Burdmann EA, Tonelli M, Garcia-Garcia G, Jha V, et al. International Society of Nephrology's 0by25 initiative for acute kidney injury (zero preventable deaths by 2025): a human rights case for nephrology. Lancet. 2015;385(9987):2616–43. Epub 2015/03/18. pmid:25777661.
- 6. James MT, Hobson CE, Darmon M, Mohan S, Hudson D, Goldstein SL, et al. Applications for detection of acute kidney injury using electronic medical records and clinical information systems: workgroup statements from the 15(th) ADQI Consensus Conference. Can J Kidney Health Dis. 2016;3:9. Epub 2016/03/01. pmid:26925245; PubMed Central PMCID: PMCPMC4768328.
- 7. Hoste EA, Kashani K, Gibney N, Wilson FP, Ronco C, Goldstein SL, et al. Impact of electronic-alerting of acute kidney injury: workgroup statements from the 15(th) ADQI Consensus Conference. Can J Kidney Health Dis. 2016;3:10. Epub 2016/03/01. pmid:26925246; PubMed Central PMCID: PMCPMC4768416.
- 8. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:b604. Epub 2009/04/02. pmid:19336487.
- 9. Kolhe NV, Staples D, Reilly T, Merrison D, McIntyre CW, Fluck RJ, et al. Impact of Compliance with a Care Bundle on Acute Kidney Injury Outcomes: A Prospective Observational Study. PLoS One. 2015;10(7):e0132279. Epub 2015/07/15. pmid:26161979; PubMed Central PMCID: PMCPMC4498890.
- 10. Kolhe NV, Reilly T, Leung J, Fluck RJ, Swinscoe KE, Selby NM, et al. A simple care bundle for use in acute kidney injury: a propensity score-matched cohort study. Nephrol Dial Transplant. 2016;31(11):1846–54. Epub 2016/11/04. pmid:27190331.
- 11. Al-Jaghbeer M, Dealmeida D, Bilderback A, Ambrosino R, Kellum JA. Clinical Decision Support for In-Hospital AKI. J Am Soc Nephrol. 2017. Epub 2017/11/04. 29097621. pmid:29097621
- 12. Wilson FP, Shashaty M, Testani J, Aqeel I, Borovskiy Y, Ellenberg SS, et al. Automated, electronic alerts for acute kidney injury: a single-blind, parallel-group, randomised controlled trial. Lancet. 2015. Epub 2015/03/03. pmid:25726515.
- 13.
Stage 3 ‘Directive’ Patient Safety Alert: ‘Standardising the early identification of Acute Kidney Injury’. [Internet]. https://www.england.nhs.uk/2014/06/09/psa-aki/; 2014; June [cited 25/11/2015]. Available from: https://www.england.nhs.uk/2014/06/09/psa-aki/
- 14. Lachance P, Villeneuve P-M, Wilson FP, Selby NM, Featherstone R, Rewa O, et al. Impact of e-alert for detection of acute kidney injury on processes of care and outcomes: protocol for a systematic review and meta-analysis. BMJ Open. 2016;6(5):e011152. PMC4861089. pmid:27150187
- 15. Forni LG, Dawes T, Sinclair H, Cheek E, Bewick V, Dennis M, et al. Identifying the patient at risk of acute kidney injury: a predictive scoring system for the development of acute kidney injury in acute medical patients. Nephron Clin Pract. 2013;123(3–4):143–50. Epub 2013/07/28. pmid:23887252.
- 16. Hodgson LE, Dimitrov BD, Roderick PJ, Venn R, Forni LG. Predicting AKI in emergency admissions: an external validation study of the acute kidney injury prediction score (APS). BMJ Open. 2017;7(3):e013511. Epub 2017/03/10. pmid:28274964; PubMed Central PMCID: PMCPMC5353262.
- 17. Des Jarlais DC, Lyles C, Crepaz N, the TG. Improving the Reporting Quality of Nonrandomized Evaluations of Behavioral and Public Health Interventions: The TREND Statement. American Journal of Public Health. 2004;94(3):361–6. PMC1448256. pmid:14998794
- 18.
Statistics OfN. 2011 Census General Report for England and Wales https://www.ons.gov.uk/census/2011census/2011censusdata: ONS Statistics OfN; 2011.
- 19. Sawhney S, Fluck N, Marks A, Prescott G, Simpson W, Tomlinson L, et al. Acute kidney injury-how does automated detection perform? Nephrol Dial Transplant. 2015;30(11):1853–61. Epub 2015/05/01. pmid:25925702; PubMed Central PMCID: PMCPMC4617372.
- 20. Wallace E, Smith SM, Perera-Salazar R, Vaucher P, McCowan C, Collins G, et al. Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs). BMC Med Inform Decis Mak. 2011;11:62. Epub 2011/10/18. pmid:21999201; PubMed Central PMCID: PMCPMC3216240.
- 21. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. Epub 2005/03/16. pmid:15767266; PubMed Central PMCID: PMCPMC555881.
- 22. Stiell IG, McKnight RD, Greenberg GH, McDowell I, Nair RC, Wells GA, et al. Implementation of the Ottawa ankle rules. JAMA. 1994;271(11):827–32. Epub 1994/03/16. pmid:8114236.
- 23. Stiell IG, Wells GA, Hoag RH, Sivilotti ML, Cacciotti TF, Verbeek PR, et al. Implementation of the Ottawa Knee Rule for the use of radiography in acute knee injuries. JAMA. 1997;278(23):2075–9. Epub 1997/12/24. pmid:9403421.
- 24.
Card D, Krueger AB. Minimum wages and employment: A case study of the fast food industry in New Jersey and Pennsylvania. National Bureau of Economic Research, 1993.
- 25.
Ashenfelter OC, Card D. Using the longitudinal structure of earnings to estimate the effect of training programs. National Bureau of Economic Research Cambridge, Mass., USA; 1984.
- 26. Dimick JB, Ryan AM. Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA. 2014;312(22):2401–2. Epub 2014/12/10. pmid:25490331.
- 27. Morris S, Hunter RM, Ramsay AIG, Boaden R, McKevitt C, Perry C, et al. Impact of centralising acute stroke services in English metropolitan areas on mortality and length of hospital stay: difference-in-differences analysis. BMJ: British Medical Journal. 2014;349. pmid:25098169
- 28. Patel MS, Volpp KG, Small DS, Hill AS, Even-Shoshan O, Rosenbaum L, et al. Association of the 2011 ACGME resident duty hour reforms with mortality and readmissions among hospitalized Medicare patients. JAMA. 2014;312(22):2364–73. Epub 2014/12/10. pmid:25490327.
- 29. Rajaram R, Chung JW, Jones AT, Cohen ME, Dahlke AR, Ko CY, et al. Association of the 2011 ACGME resident duty hour reform with general surgery patient outcomes and with resident examination performance. JAMA. 2014;312(22):2374–84. Epub 2014/12/10. pmid:25490328.
- 30. Liang K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986:13–22.
- 31. Uchino S. Creatinine. Curr Opin Crit Care. 2010;16(6):562–7. Epub 2010/08/26. pmid:20736825.
- 32. Park S, Baek SH, Ahn S, Lee KH, Hwang H, Ryu J, et al. Impact of Electronic Acute Kidney Injury (AKI) Alerts With Automated Nephrologist Consultation on Detection and Severity of AKI: A Quality Improvement Study. Am J Kidney Dis. 2018;71(1):9–19. Epub 2017/07/30. pmid:28754457.
- 33. Awdishu L, Coates CR, Lyddane A, Tran K, Daniels CE, Lee J, et al. The impact of real-time alerting on appropriate prescribing in kidney disease: a cluster randomized controlled trial. J Am Med Inform Assoc. 2016;23(3):609–16. Epub 2015/11/29. pmid:26615182.
- 34. Boutis K, Grootendorst P, Willan A, Plint AC, Babyn P, Brison RJ, et al. Effect of the Low Risk Ankle Rule on the frequency of radiography in children with ankle injuries. CMAJ. 2013;185(15):E731–8. Epub 2013/08/14. pmid:23939215; PubMed Central PMCID: PMCPMC3796622.
- 35. McGinn TG, McCullagh L, Kannry J, Knaus M, Sofianou A, Wisnivesky JP, et al. Efficacy of an evidence-based clinical decision support in primary care practices: a randomized clinical trial. JAMA Intern Med. 2013;173(17):1584–91. Epub 2013/07/31. pmid:23896675.
- 36. Phansalkar S, Edworthy J, Hellier E, Seger DL, Schedlbauer A, Avery AJ, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. Journal of the American Medical Informatics Association: JAMIA. 2010;17(5):493–501. PMC2995688. pmid:20819851
- 37. Nightingale PG, Adu D, Richards NT, Peters M. Implementation of rules based computerised bedside prescribing and administration: intervention study. BMJ: British Medical Journal. 2000;320(7237):750–3. PMC27317. pmid:10720357
- 38. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of Drug Safety Alerts in Computerized Physician Order Entry. Journal of the American Medical Informatics Association: JAMIA. 2006;13(2):138–47. PMC1447540. pmid:16357358
- 39. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163(21):2625–31. Epub 2003/11/26. pmid:14638563.
- 40. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339–46. Epub 1998/10/30. pmid:9794315.
- 41. Kanouse DE, Kallich JD, Kahan JP. Dissemination of effectiveness and outcomes research. Health Policy. 1995;34(3):167–92. Epub 1995/11/04. pmid:10153899.
- 42. Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinical decision-support systems on practitioner performance and patient outcomes: a synthesis of high-quality systematic review findings. J Am Med Inform Assoc. 2011;18(3):327–34. Epub 2011/03/23. pmid:21422100; PubMed Central PMCID: PMCPMC3078663.
- 43. Moxey A, Robertson J, Newby D, Hains I, Williamson M, Pearson S-A. Computerized clinical decision support for prescribing: provision does not guarantee uptake. Journal of the American Medical Informatics Association: JAMIA. 2010;17(1):25–33. PMC2995634. pmid:20064798
- 44. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523–30. Epub 2003/08/20. pmid:12925543; PubMed Central PMCID: PMCPMC264429.
- 45. Ash JS, Berg M, Coiera E. Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-related Errors. Journal of the American Medical Informatics Association: JAMIA. 2004;11(2):104–12. PMC353015. pmid:14633936
- 46. Bliss JP, Dunn MC. Behavioural implications of alarm mistrust as a function of task workload. Ergonomics. 2000;43(9):1283–300. Epub 2000/10/03. pmid:11014752.
- 47. Wickens CD, Carswell CM. The Proximity Compatibility Principle: Its Psychological Foundation and Relevance to Display Design. Human Factors: The Journal of the Human Factors and Ergonomics Society. 1995;37(3):473–94.
- 48. Sanders AF. Some aspects of the selective process in the functional visual field. Ergonomics. 1970;13(1):101–17. Epub 1970/01/01. pmid:5416864.
- 49. Poulton EC. Searching for newspaper headlines printed in capitals or lower-case letters. J Appl Psychol. 1967;51(5):417–25. Epub 1967/10/01. pmid:6074392.
- 50. Braun CC, Silver NC. Interaction of signal word and colour on warning labels: differences in perceived hazard and behavioural compliance. Ergonomics. 1995;38(11):2207–20. Epub 1995/11/01. pmid:7498185.
- 51. Kanagasundaram NS, Bevan MT, Sims AJ, Heed A, Price DA, Sheerin NS. Computerized clinical decision support for the early recognition and management of acute kidney injury: a qualitative evaluation of end-user experience. Clinical Kidney Journal. 2016;9(1):57–62. PMC4720208. pmid:26798462
- 52. McCoy AB, Waitman LR, Gadd CS, Danciu I, Smith JP, Lewis JB, et al. A computerized provider order entry intervention for medication safety during acute kidney injury: a quality improvement report. Am J Kidney Dis. 2010;56(5):832–41. Epub 2010/08/17. pmid:20709437; PubMed Central PMCID: PMCPMC2963668.
- 53. Isaac T, Weissman JS, Davis RB, Massagli M, Cyrulik A, Sands DZ, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med. 2009;169(3):305–11. Epub 2009/02/11. pmid:19204222.
- 54. Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. Epub 2013/02/16. pmid:23412440.
- 55. Benning A, Dixon-Woods M, Nwulu U, Ghaleb M, Dawson J, Barber N, et al. Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ. 2011;342. pmid:21292720
- 56. Brown C, Hofer T, Johal A, Thomson R, Nicholl J, Franklin BD, et al. An epistemology of patient safety research: a framework for study design and interpretation. Part 2. Study design. Quality and Safety in Health Care. 2008;17(3):163–9. pmid:18519621
- 57. Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new MRC guidance. Journal of epidemiology and community health. 2012;66(12):1182–6. PMC3796763. pmid:22577181
- 58.
Hospital Episode Statistics, Admitted Patient Care, England—2012–13, http://content.digital.nhs.uk/hesdata. 2013.
- 59. Friedman C, Rubin J, Brown J, Buntin M, Corn M, Etheredge L, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43–50. Epub 2014/10/25. pmid:25342177; PubMed Central PMCID: PMCPMC4433378.
- 60.
Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Grossmann C, Powers B, McGinnis JM, editors. Washington DC: National Academy of Sciences.; 2011.
- 61. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207–10. Epub 2012/08/08. pmid:22868839.