https://orcid.org/0000-0001-5873-8030
Figures
Abstract
Background
Measuring sepsis incidence and associated mortality at scale using administrative data is hampered by variation in diagnostic coding. This study aimed first to compare how well bedside severity scores predict 30-day mortality in hospitalised patients with infection, then to assess the ability of combinations of administrative data items to identify patients with sepsis.
Methods
This retrospective case note review examined 958 adult hospital admissions between October 2015 and March 2016. Admissions with blood culture sampling were matched 1:1 to admissions without a blood culture. Case note review data were linked to discharge coding and mortality. For patients with infection the performance characteristics of Sequential Organ Failure Assessment (SOFA), National Early Warning System (NEWS), quick SOFA (qSOFA), and Systemic Inflammatory Response Syndrome (SIRS) were calculated for predicting 30-day mortality. Next, the performance characteristics of administrative data (blood cultures and discharge codes) for identifying patients with sepsis, defined as SOFA ≥2 because of infection, were calculated.
Results
Infection was documented in 630 (65.8%) admissions and 347 (55.1%) patients with infection had sepsis. NEWS (Area Under the Receiver Operating Characteristic, AUROC 0.78 95%CI 0.72–0.83) and SOFA (AUROC 0.77, 95%CI 0.72–0.83), performed similarly well for prediction of 30-day mortality. Having an infection and/or sepsis International Classification of Diseases, Tenth Revision (ICD-10) code (AUROC 0.68, 95%CI 0.64–0.71) performed as well in identifying patients with sepsis as having at least one of: an infection code; sepsis code, or; blood culture (AUROC 0.68, 95%CI 0.65–0.71), Sepsis codes (AUROC 0.53, 95%CI 0.49–0.57) and positive blood cultures (AUROC 0.52, 95%CI 0.49–0.56) performed least well.
Citation: Bateson M, Marwick CA, Staines HJ, Patton A, Stewart E, Rooney KD (2023) Performance of bedside tools for predicting infection-related mortality and administrative data for sepsis surveillance: An observational cohort study. PLoS ONE 18(3): e0280228. https://doi.org/10.1371/journal.pone.0280228
Editor: Aleksandar R. Zivkovic, Heidelberg University Hospital, GERMANY
Received: June 4, 2021; Accepted: December 23, 2022; Published: March 2, 2023
Copyright: © 2023 Bateson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data included as a supporting file.
Funding: The study was supported by Sepsis Research (FEAT) (charity no. SCO44017, grant ref. 007). Sepsis Research (FEAT) had no influence on study design, data collection, analysis, interpretation of data or writing of the manuscript. The author receiving the award was MB. https://sepsisresearch.org.uk/.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: K.R. was the National Clinical Lead for Sepsis and the Deteriorating Patient workstream of the Scottish Patient Safety Programme between May 2012 and October 2017. As such, he was instrumental in the National Sepsis strategy and the recommendation and adoption of NEWS across Scotland.
Introduction
Sepsis, redefined in 2016 as “life-threatening organ dysfunction caused by a dysregulated host response to infection” [1], is globally estimated to cause between five and 11 million deaths per year [2,3]. Survivors of sepsis are at increased risk of hospital readmission and long-term functional sequelae [4,5]. Sepsis related morbidity, mortality and consequent financial burden [6,7] has ignited an international focus on improving the prevention, recognition, and management of sepsis [8]. Evaluating the impact of changes intended to improve practice requires accurate surveillance of sepsis incidence and outcomes which is stable over time.
Sepsis identification and surveillance have been complicated by changes in definitions over time and the number of clinical tools used to screen for sepsis. In 2016, the Sepsis-3 [1] consensus guidelines set out diagnostic criteria for sepsis as organ dysfunction, defined as an acute increase in total Sequential [Sepsis-Related] Organ Failure Assessment (SOFA) [9] score of at least two, as a consequence of infection. Using the Sepsis-3 definition in emergency departments and general wards is challenging since SOFA is not routinely used outside of critical care. Furthermore, SOFA is calculated only once per 24 hours which limits its ability to detect sepsis early, crucial to the improvement of patient outcomes [10].
In recognition of the need for a bedside tool to identify patients with infection at risk of poor outcomes the Sepsis-3 guidelines recommended the use of qSOFA (quick SOFA) outside critical care [1]. Recently however, due to evidence that qSOFA lacks sensitivity the Surviving Sepsis Campaign guidelines have cautioned against its use as a standalone sepsis screening tool [11]. Other severity scores commonly in use in the United Kingdom, and internationally, include the National Early Warning Score (NEWS & NEWS2) [12,13] and Systemic Inflammatory Response Syndrome (SIRS) criteria [14]. However, the scores differ in their ability to identify patients with infection at high risk of death [15,16]. There is a need to establish how these scores compare to SOFA, as the Sepsis-3 [1] gold standard, to inform the development of a clinically relevant measure for sepsis surveillance.
The development of sepsis surveillance measures using clinical rather than administrative data is desirable because of concerns around inconsistency in the application of administrative codes to patients with sepsis [17]. While electronic health records have the potential to enable sepsis surveillance based on clinical criteria, national use of electronic health records is not yet widespread even in high income countries making purely clinically based measures impractical for many settings. Meantime, there is a need for an objective and reliable method of identifying the population with sepsis at scale to monitor outcomes. The most commonly used administrative data for tracking epidemiological trends are the International Classification of Diseases, Tenth Revision (ICD-10) codes [18]. The use of ICD-10 codes to track sepsis is controversial due to changes in definition, trends in coding practice and the range of potentially appropriate codes [19]. Following the implementation of National Health Service (NHS) Digital Coding Guidance NHS England reported a doubling in the use of sepsis codes (A40, streptococcal sepsis; A41, other sepsis) between 2016/17 [20] and 2017/2018 [21]. Sudden changes in coding practice are multifactorial but introduce problems in using ICD-10 codes as an isolated method for sepsis surveillance, risking misinterpretation of such changes as changes in sepsis incidence [22,23].
Including clinical data such as blood cultures in a surveillance measure for sepsis may offer some mitigation for the caveats of coding, but it has potential limitations. There will be a proportion of patients who do not have blood cultures obtained despite meeting clinical criteria to do so, leading to data which are missing not at random. Despite this there remains a need for a clinically relevant method to identify patients with sepsis which can be applied at scale, ideally from routinely collected administrative data [22–24].
Our aims were:
- To calculate the performance characteristics of clinical severity scores (SOFA, qSOFA, NEWS, SIRS) in predicting the 30-day mortality of people with infection.
- To calculate the performance characteristics of administrative data (ICD-10 codes and blood culture sampling) in identifying people with sepsis, defined as SOFA ≥2 as a consequence of infection.
Methods
The data were generated as part of routine NHS care. The Caldicott Guardian for NHS Greater Glasgow & Clyde, who is responsible for protecting confidentiality of people’s health and care information and making sure it is used appropriately, gave permission for access to the data. The data were anonymised for analysis at the earliest opportunity. The West of Scotland Research Ethics Committee (ref: 17/WS/0044) approved the study without the need for informed consent.
Patient population
This was a single-centre retrospective case note review of adult patients admitted to a large district general hospital in Greater Glasgow & Clyde, in the west of Scotland, United Kingdom. We identified all adult (≥16 years) admissions between 1st October 2015 and 31st March 2016 from the hospital administration system and linked them to blood culture data from the hospital microbiology system. Patients admitted to maternity units were excluded due to differences in physiology in pregnancy which necessitate the use of maternity specific severity scores. Patients who were discharged or died within the first 24 hours were also excluded.
Admissions with a blood culture were matched 1:1 to admissions without a blood culture who were in the same ward or clinical area at time of blood culture, (+/-24hrs for general wards and +/-96hrs for critical care), and who had similar length of hospital stay (+/-24hrs for general wards and +/-48hrs for critical care). Patients could be admitted more than once during the data collection period. Admissions which were more than 48hrs from previous discharge were included as the variable of interest was the coding applied in the study.
It was estimated from unpublished local data that 40–60% of patients with a blood culture taken during a hospital admission had sepsis. The sampling frame deliberately over-sampled patients likely to have infection as the aim of the case note review was to evaluate candidate sepsis detection and outcome tools rather than to estimate the incidence of sepsis. A target sample size of 1000 patients was estimated to provide sufficient data and is larger than most studies involving manual chart review. From matched admissions, all eligible critical care (intensive care, high dependency or coronary care) admissions were included, and a random sample of ward patients was selected to complete the cohort, using a computer-generated list of random numbers.
Case note data
Case note review and data extraction were completed by a registered nurse (MB) using scanned paper clinical records. Demographic data included age, gender, and socioeconomic status. Postcode was used to allocate Scottish Index of Multiple Deprivation (SIMD), a Scottish Government measure of socioeconomic deprivation split into quintiles, with 1 being least affluent and 5 most affluent [25].
Each entire admission was manually reviewed for evidence of infection and categorised as the patient having no infection, infection without sepsis, or sepsis. Infection was defined as a documented source of infection and administration of either oral or intravenous antibiotics. The onset of infection was defined as the date and time of first blood culture taken or first antibiotic dose, whichever was earliest. As we were interested in identifying acute infection and sepsis, pre-existing infections with oral antibiotics prescribed in the community, prior to admission and without any escalation in hospital were excluded from the ‘infection’ group. A Consultant Physician in Infectious Diseases (CM) made the final decision on queries relating to the presence or absence of infection. Any documentation of suspected or confirmed sepsis by the clinical team was recorded during the case note review.
Physiological data to allow calculation of all the included severity scores were collected at the point of highest acuity, defined as the highest NEWS score in the 48hrs before or after onset of infection. Sepsis was defined as per Sepsis-3 [1]: infection plus a SOFA score of ≥ 2 for patients in general wards, or infection plus a change in SOFA score (measured once per 24hrs) of ≥2 for patients in critical care, in the 2 days before or after onset of infection.
Administrative data
Data on blood culture sampling and results for the cohort were extracted by microbiology staff. Results were reviewed and categorised as negative, positive, or contaminated by a Consultant Physician in Infectious Diseases (CM) (S1 Table).
ICD-10 discharge codes assigned for each admission were extracted from Trakcare®, the hospital administration system. A list of infection and sepsis codes (S2 Table) was constructed, informed by the literature [24,26], by MB and CM.
The microbiology, ICD-10 and case note data were linked using a pseudo-anonymised patient CHI using a key, which was kept separately to the clinical data in a password-protected file on a secure server.
Score calculation
Each patient with at least one episode of infection (as defined above) had the following scores calculated: SOFA, qSOFA, NEWS and SIRS (Table 1). NEWS and SIRS scores were calculated as per published tables [12,14]. As SOFA was designed for critical care it was anticipated that there would be a significant number of missing values for the respiratory component which requires an arterial blood gas to calculate the PaO2/FiO2 ratio. There are several available alternatives for ward patients [27,28] which use combinations of inspired oxygen and oxygen saturations. We used Valik et al’s [29] method which allocates a SOFA score of 1 for oxygen saturations of 91–94% and 2 for oxygen saturations of less than 91% as a replacement for the missing data because it performed better than either the original SOFA scoring [9] or Gadrey et al [27]. For both SOFA and qSOFA, AVPU (Alert, Verbal or Painful stimulus required to provoke a response, Unconscious) scoring was applied to calculate the neurological components when GCS was unavailable, as per the method published by Zadravecz et al. [30]. Other missing data were treated as normal values in score calculation.
Calculation of NEWS, SIRS, qSOFA was undertaken using the physiological observations closest to time of onset of infection and also at the point of the highest NEWS in the 48hrs pre/post onset of infection. The highest NEWS score was identified and used in the analysis. SOFA was calculated using the worst blood results and worst physiological observations for each 24hr period in the 48hrs pre and post onset of infection. The highest SOFA score was used in the analysis.
Baseline SOFA score for patients in general wards was assumed to be zero, as per Sepsis-31 guidance so a SOFA score of ≥2 related to infection was considered to be indicative of sepsis, whilst critical care patients required a change in SOFA score of ≥2, consequent to infection. A cut off score of ≥2 was applied to qSOFA [1] and SIRS [14]. Performance of NEWS was assessed using both the medium (NEWS ≥5) and high (NEWS ≥7) clinical risk thresholds [12].
Statistical analysis
Descriptive statistics were used to summarise the cohort and compare groups. Pearson Chi-Square was calculated for categorical variables. Mann-Whitney U was applied to assess differences between continuous variables in two groups. Kruskall-Wallis was used to assess differences among more than two groups, on a single continuous variable. A p value of less than 0.05 was considered significant. No adjustment was made for multiplicity.
Performance characteristics (sensitivity, specificity, negative and positive predictive values, and receiver operating characteristic curves) of the severity scores were assessed for their ability to predict 30-day mortality from the onset of infection using the clinical cut-off scores for each tool. The area under the receiver operating characteristic curve (AUROC) and 95% confidence intervals were calculated for each severity score across all possible values of each score. Differences between AUROCs were assessed using DeLong’s test [31].
The performance characteristics and AUROC of administrative data items, singly and in combination, to identify people with sepsis were calculated, both for sepsis according to the Sepsis-3 SOFA definition, and to other severity score(s) performing best at predicting 30-day mortality after infection onset, i.e. that identified high risk infection patients.
Analyses were undertaken in SPSS version 25.0 (Armonk, NY, USA), and SAS version 9.4 (Cary, NC, USA).
Results
During the study period there were 11207 adult admissions with a hospital length of stay of ≥24hrs (Fig 1). Of these, 9381 admissions had no blood culture and 1826 had at least one blood culture sample taken. Matching of admissions with and without a blood culture obtained 1796 matched ward admissions (898 matched pairs) and 70 matched critical care admissions (35 matched pairs).
All 70 critical care admissions were included and a simple random sample of 930 ward admissions (465 matched pairs) was obtained. Ultimately, 21 admissions were excluded due to inaccessible or missing clinical notes and matched admissions were also excluded, resulting in a final sample of 958 admissions.
Study population
The final cohort (n = 958) had a median age of 68 (IQR 52–79) and 54% were female. The majority were emergency admissions (94.3%) and the cohort included both community-acquired infection and infection acquired during the hospital stay. Of the total cohort, 630 (65.7%) were treated for infection and of those, 347 (55.1%) had sepsis, defined using SOFA (Table 2). People with sepsis were older (p = <0.001), had a longer hospital stay (p<0.001) and were more likely to die within 30 days (p<0.001) compared to those without sepsis (Table 2). Only 32.6% (n = 113) of patients with sepsis, had suspected or confirmed sepsis documented contemporaneously in their medical notes.
In addition to the 62 admissions matched by critical care area at time of blood culture, 30 patients matched in general wards were admitted to critical care at some point during their stay. Of the 92 critical care patients, 59 (64.1%) were admitted to critical care within 48hrs pre or post onset of infection.
Performance of SOFA, qSOFA, NEWS and SIRS in predicting mortality
Of the 630 patients with infection 55.1% had a SOFA score of ≥2 (or change of ≥2 for critical care patients), meeting the Sepsis-31 criteria for sepsis (Table 3). 14.0% patients with infection had a qSOFA score of ≥2. Initial missing data for the respiratory component of SOFA (86.7%, n = 546) and the GCS component of qSOFA and SOFA (74.1%, n = 467) were replaced as described in Methods [29,30]. All other variables had little missing data (at least 93.5% completeness) with missing values assumed to be normal for the purpose of severity score calculation.
The performance characteristics of each severity score in predicting 30-day mortality from onset of infection were calculated using the usual clinical cut-off scores (Table 3). A sensitivity analysis for in-hospital mortality is available as supplementary information (S5 Table). For predicting 30-day mortality qSOFA had the lowest sensitivity (42.3%%, 95%CI 31.2–54.0), but the highest specificity (90.0%, 95%CI 87.2–92.4%). SIRS had the highest sensitivity (89.7%, 95%CI 80.8–95.5%) but lowest specificity (39.3%, 95%CI 35.2–43.5%).
NEWS (AUROC 0.78, 95%CI 0.72–0.83) and SOFA (AUROC 0.77, 95%CI 0.72–0.83) performed similarly well for prediction of 30-day mortality from onset of infection when plotted across all values (Fig 2). qSOFA performed less well (AUROC 0.74, 95%CI 0.68–0.79) but the difference from SOFA was not statistically significant (DeLong’s test p = 0.33). SIRS had the poorest AUROC (0.68, 95%CI 0.63–0.74, DeLong’s test p = 0.03) (Fig 2).
The performance characteristics of each tool were also calculated separately for ward patients only to investigate the performance of SOFA in this cohort. For ward patients (n = 571) who were not admitted to critical care during the 48hrs pre or post onset of infection NEWS predicted 30-day mortality (AUROC 0.80, 95%CI 0.73–0.86) better across all values than any other tool (SOFA, 0.73, 95%CI 0.66–0.80; SIRS, 0.70, 95% CI 0.64–0.77; qSOFA, 0.72, 95%CI 0.65–0.80), although the difference for SOFA was not quite statistically significant (DeLong’s p = 0.07), in predicting 30-day mortality but performed significantly better than SIRS (p = 0.003) or qSOFA (p = 0.007).
Performance characteristics of routine data to identify people with sepsis
Only 27 (7.8%) sepsis admissions had a sepsis ICD-10 code (A40, A41 or R57.2) (Table 4). Coding for infection was more frequent with 226 (65.1%) of sepsis admissions being allocated at least one infection code. More than 80% of infection and sepsis codes allocated were from three ICD-10 chapters: diseases of the respiratory system (J00-J99), certain infectious and parasitic diseases (A00-B99) and diseases of the genitourinary system (N00-N99).
Having a blood culture taken (AUROC 0.63, 95%CI 0.59–0.66) and infection codes (AUROC 0.66, 95%CI 0.63–0.70) had higher AUROCs to identify people with sepsis, defined by Sepsis-3 SOFA critiera, than either a positive blood culture result (AUROC 0.52, 95%CI 0.49–0.56), or a sepsis ICD-10 code (A40, A41, R57.2) (AUROC 0.53, 95%CI 0.49–0.57) (Table 4). In combination, having an infection code and/or a sepsis code (AUROC 0.68, 95%CI 0.64–0.71) performed the same as having at least one of: an infection code; a sepsis code, or; a blood culture (AUROC 0.68, 95%CI 0.65–0.71).
The performance of administrative data in identifying people with infection and NEWS ≥7 (as an alternative sepsis identification method) was similar to its ability in identifying people meeting the Sepsis-3 SOFA criteria (S6 Table).
Discussion
To the best of our knowledge this is the first study which investigates the performance of qSOFA, SIRS, NEWS and SOFA in predicting 30-day mortality of patients with infection in a general hospital cohort. For sepsis outcome surveillance using routinely collected administrative data, ICD-10 sepsis codes and positive blood cultures were insufficiently sensitive but blood culture sampling and infection codes both performed better.
Screening patients for infection across the whole hospital stay rather than only at admission was a strength of this study and provided a complete picture of sepsis incidence within the cohort. The use of clinical data to identify people with sepsis also avoided the bias of selecting patients on the basis of their ICD-10 codes and enabled assessment of coding patterns. We were also able to detect mortality following discharge.
This study was limited by its retrospective, single centre design. Although excluding patients who died or were discharged within 24 hours of admission could have introduced selection bias, they were unlikely to have sufficient physiological data for the purposes of this study, which may have skewed the performance characteristics of the severity scores and administrative data. The need to replace AVPU and the respiratory component of SOFA due to the lack of recorded GCS, used in qSOFA and SOFA, and arterial blood gas values, used in SOFA, was unsurprising given that general ward patients often do not require arterial blood gas sampling and would not routinely have a GCS recorded unless there was concern about their neurological status. Other studies have also noted GCS as often not recorded [32].
Whilst assuming a baseline SOFA score of zero in ward patients is recommended in the Sepsis-3 guidelines [1], it is acknowledged that the lack of a true pre-admission SOFA score has potential to bias the results [27].
Our findings are consistent with some of the broader literature demonstrating superior performance of NEWS over qSOFA and SIRS in predicting mortality, [15,33–35] although other studies have reported SOFA as being superior to NEWS in predicting mortality [16,36]. SOFA was, however, designed and largely studied for use within critical care settings and its prognostic accuracy has been less thoroughly evaluated within the general ward population. Our findings add to the body of knowledge on SOFA performance outside of critical care.
The suboptimal performance of qSOFA in predicting mortality, particularly in terms of sensitivity, is similar to that reported elsewhere [37–39]. Our study also found that, compared to NEWS, qSOFA did not perform well in identifying people who met the Sepsis-3 SOFA criteria (infection plus SOFA ≥2 or change of 2 for critical care admissions). These findings add to the evidence challenging the use of qSOFA over NEWS [11,15]. Similar to other studies, SIRS had a high sensitivity but poor specificity and as such is a poor predictor of those patients with infection at an increased risk of a poor outcome [15,34].
In line with other studies [29,40], our research reported significant under coding for sepsis and consequent poor performance of the ICD-10 sepsis codes in identifying patients with sepsis. The growth in sepsis coding observed in the literature has contributed to reporting of significant increases in documented sepsis incidence [26,41,42]. Our finding of lack of sensitivity in sepsis codes and lack of specificity in infection codes, supports concerns about the quality of coding, its impact on epidemiological trends and potential for misinformed influence on policy and practice [17,42].
There is a growing body of literature which demonstrates the potential of machine learning in predicting sepsis in individual patients [43,44] and therefore also theoretical future capacity for sepsis surveillance at scale. This however requires fully automated electronic health records which are not yet standard in the UK or other countries in the Global North. Consequently, there remains a need for an objective, stable and clinically relevant proxy measure for sepsis surveillance. Positive blood cultures predictably performed poorly as a surrogate marker for sepsis in our cohort and, as reported elsewhere, lacks the sensitivity required for sepsis surveillance [45]. Blood culture sampling alone, however, performed significantly better than ICD-10 sepsis codes in identifying patients with sepsis, demonstrating potential as a component of routine sepsis surveillance. As an administrative data variable, blood culture sampling is unaffected by trends in diagnostic coding but is susceptible to changes and local variation in sampling practice. Further study is required to assess the effect that standardisation of blood culture sampling in sepsis has on its sensitivity and specificity as a measure.
Although no one combination of administrative data variables had an AUROC of >0.7, generally considered to be the cut-off for acceptable discrimination [46], this study did identify blood cultures as data routinely collected in Scotland with better predictive accuracy for sepsis than the sepsis explicit ICD-10 codes currently used. Blood culture data could offer an objective adjunct to coding with potential for application at a population level in contexts where it is electronically available at scale. The development of a stable measure to identify people with sepsis from administrative data would make a significant contribution to both sepsis surveillance and, more broadly, to the decisions policy makers make in relation to resource allocation in a fiscally pressured system. It would also address the concerns raised about potential for instability in coding and over-diagnosis [26,42]. Further investigation and validation of a combined measure is required in a larger, multicentre cohort.
Conclusions
Our results support the use of SOFA and NEWS ≥7 to identify patients with infection at increased risk of death at 30 days, from onset of infection. The use of sepsis explicit ICD-10 codes to identify people with sepsis performed poorly in this cohort. Blood cultures as clinical, but routinely stored data, demonstrated potential as one component of a surrogate marker for sepsis, along with infection ICD-10 codes. This study provides important insights into potential use of administrative data for sepsis surveillance at scale.
Supporting information
S1 Table. Organisms grown from blood cultures sampled from study cohort.
https://doi.org/10.1371/journal.pone.0280228.s001
(DOCX)
S2 Table. ICD-10 infection and sepsis codes reported in this cohort.
https://doi.org/10.1371/journal.pone.0280228.s002
(DOCX)
S3 Table. Sequential Organ Failure Assessment (SOFA) scoring system.
Vincent JL, Moreno R, Takala J. Willatts S, De Mendonҫa A, Bruining H, et al. Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Intensive Care Medicine. 1996; 22(7):707–710.
https://doi.org/10.1371/journal.pone.0280228.s003
(DOCX)
S4 Table. National Early Warning Score (NEWS).
Royal College of Physicians. National Early Warning Score (NEWS). Standardising the assessment of acute-illness severity in the NHS. 2012. London: Royal College of Physicians.
https://doi.org/10.1371/journal.pone.0280228.s004
(DOCX)
S5 Table. Performance characteristics of severity scores to predict in-hospital mortality.
https://doi.org/10.1371/journal.pone.0280228.s005
(DOCX)
S6 Table. Performance characteristics of administrative data to identify patients with infection & NEWS ≥7.
*PPV: Positive Predictive Value; NPV: Negative Predictive Value.
https://doi.org/10.1371/journal.pone.0280228.s006
(DOCX)
Acknowledgments
We would like to thank Sonnda Catto for assistance in developing the data collection instrument.
References
- 1. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016; 315(8):801–810. pmid:26903338
- 2. Fleischmann C, Scerag A, Adhikari NK, Hartog CS, Tsaganos T, Schlattmann P, et al. Assessment of Global Incidence and Mortality of Hospital-treated sepsis. Current estimates and limitations. Am J Respir Crit Care Med. 2016; 193(3):259–272. pmid:26414292
- 3. Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet. 2020; 395:200–211. pmid:31954465
- 4. Mayr FB, Talisa VB, Balakumar V, Chang CH, Fine M, Yende S. Proportion and cost of unplanned 30-day readmissions after sepsis compared with other medical conditions. JAMA. 2017; 317(5):530–531. pmid:28114505
- 5. Yende S, Austin S, Rhodes A, Finfer S, Opal S, Thompson T, et al. Long-Term Quality of Life among survivors of severe sepsis: analyses of two international trials. Crit Care Med. 2016; 44(8):1461–1467. pmid:26992066
- 6. Torio CM, Moore BJ. National Inpatient Hospital Costs: The most expensive conditions by Payer, Statistical Brief #204. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. 2013.
- 7.
York Health Economics Consortium. The Cost of Sepsis Care in the UK. Final Report. 2017. University of York.
- 8. World Health Organisation. Improving the prevention, diagnosis and clinical management of sepsis. (WHA70.7). 2017.
- 9. Vincent JL, Moreno R, Takala J. Willatts S, De Mendonҫa A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Intensive Care Medicine. 1996;22(7):707–710. pmid:8844239
- 10. Machado FR, Nsutebu E., AdDulaziz S, Daniels R, Finfer S, Kissoon N, et al. Sepsis 3 from the perspective of clinicians and quality improvement initiatives. Journal of Critical Care. 2017; 40:315–317. pmid:28478045
- 11. Evans L, Rhodes A, Alhazzani W, Antonelli M, Coopersmith C, French C, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock. Crit Care Med. 2021; 49(11):e1063–1143.
- 12.
Royal College of Physicians. National Early Warning Score (NEWS). Standardising the assessment of acute-illness severity in the NHS. 2012. London: Royal College of Physicians.
- 13.
Royal College of Physicians. National Early Warning Score (NEWS) 2. Standardising the assessment of acute-illness severity in the NHS. 2017. London: Royal College of Physicians.
- 14. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive Care Med. 2003; 29(4):530–538.
- 15. Goulden R, Hoyle MC, Monis J, Railton D, Riley V, Martin P, et al. qSOFA, SIRS and NEWS for predicting in hospital mortality and ICU admission in emergency admissions treated as sepsis. Emergency Medical Journal 2018; 0:1–5. pmid:29467173
- 16. Kovach C, Fletcher GS, Rudd KE, Grant RM, Carlbom DJ. Comparative prognostic accuracy of sepsis scores for hospital mortality in adults with suspected infection in non-ICU and ICU at an academic public hospital. PLoS ONE. 2019; 14(9):e0222563. pmid:31525224
- 17. Rhee C, Dantes RB, Epstein L, Klompas M. Using objective clinical data to track progress on preventing and treating sepsis: CDC’s new ‘Adult Sepsis Event’ surveillance strategy. BMJ Quality and Safety. 2019; 28:305–309. pmid:30254095
- 18.
World Health Organisation. International Statistical Classification of Diseases and Related Health Problems. 10th Revision. 2010. World Health Organisation.
- 19. Gohil SK, Cao C, Phelan M, Tjoa T, Rhee C, Platt R, et al. Impact of Policies on the Rise in Sepsis Incidence 2000–2010. Clinical Infectious Diseases. 2016; 62(6):695–703. pmid:26787173
- 20. NHS Digital. Hospital Admitted Patient Care Activity. 2017 https://digital.nhs.uk/data-and-information/publications/statistical/hospital-admitted-patient-care-activity/2016-17.
- 21. NHS Digital. Hospital Admitted Patient Care Activity. 2018. https://digital.nhs.uk/data-and-information/publications/statistical/hospital-admitted-patient-care-activity/2017-18.
- 22. Rhee C, Gohil S, Klompas M. Regulatory Mandates for Sepsis Care–Reasons for Caution. New England Journal of Medicine. 2014; 370(18):1673–1676. pmid:24738642
- 23.
Academy of Medical Royal Colleges. Statement on the initial antimicrobial treatment of sepsis. 2022. Academy of Medical Royal Colleges.
- 24. Inada-Kim M, Page B, Maqsood I, Vincent C. Defining and measuring suspicion of sepsis: an analysis of routine data. BMJ Open. 2017: 7:3014885. pmid:28601825
- 25. Scottish Government. Introducing the Scottish Index of Multiple Deprivation. 2016. https://www.gov.scot/publications/scottish-index-multiple-deprivation-2016/.
- 26. Jolley RJ, Quan H, Jetté N, Sawka KJ, Diep L, Goliath J, et al. Validation and optimisation of an ICD-10-coded case definition for sepsis using administrative health data. BMJ Open. 2015; 5(12):e009487. pmid:26700284
- 27. Gadrey S, Clay R, Zimmet A, Lawson A, Oliver S, Samuel F, et al. The Relationship between acuity of Organ Failure and Predictive Validity of Sepsis-3 Criteria. Critical Care Explorations. 2020; 2(10):e0199. pmid:33063019
- 28. Pandharipande PP, Shintani AK, Hagerman HE, St Jacques PJ, Rice TW, Sanders NW, et al. Derivation and validation of SpO2/Fio2 ratio to impute for Pao2/Fio2 ratio in the respiratory component of the Sequential Organ Failure Assessment Score. Crit Care Med. 2009; 37(4):1317–1321. pmid:19242333
- 29. Valik JK, Mellhammar L, Sundén-Cullberg J, Ward L, Unge C, Dalianis H, et al. Peripheral Oxygen Saturation Facilitates Assessment of Respiratory Dysfunction in the Sequential Organ Failure Assessment Score with Implications for the Sepsis-3 Criteria. Crit Care Med. 2021; 50(3):e272–e283.
- 30. Zadravecz FJ, Tien L, Robertson-Dick BJ, Yuen TC, Twu NM, Churpek MM, et al. Comparison of mental-status scales for predicting mortality on the general wards. Journal of Hospital Medicine. 10(10):658–663. pmid:26374471
- 31. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988; 44:837–845. pmid:3203132
- 32. Moskowitz A, Patel PV, Grossestreuer AV, Chase M, Shapiro NI, Berg K, et al. Quick Sequential Organ Failure Assessment and Systemic Inflammatory Response Syndrome Criteria as Predictors of Critical Care Intervention among Patients with Suspected Infection. Critical Care Medicine. 2017; 45(11):1813–1819. pmid:28759474
- 33. Brink A, Alsma J, Verdonschot RJCG, Rood PPM, Zietse R, Lingsma HF, et al. Predicting mortality in patients in suspected sepsis at the Emergency Department; A retrospective cohort study comparing qSOFA, SIRS and National Early Warning Score. PLoS ONE. 2019; 14(1):e0211133. pmid:30682104
- 34. Churpek MM, Snyder A, Sokol S, Pettit NN, Edelson DP. Investigating the impact of different suspicion of infection criteria on the accuracy of quick sepsis-related organ failure assessment, systemic inflammatory response syndrome and early warning scores. Critical Care Medicine. 2017; 45:1805–1812. pmid:28737573
- 35. Redfern OC, Smith GB, Prytherch DR, Meredith P, Inada-Kim , Schmidt PE. A comparison of the quick sequential (sepsis-related) organ failure assessment score and the national early warning Score in Non-ICU patients with/without infection. Critical Care Medicine. 2018; 46(12):1923–1933. pmid:30130262
- 36. Szakmany T, Pugh R, Kopczynska M, Lundin RM, Sharif B, Morgan P, et al. Defining sepsis on the wards: results on a multi-centre point-prevalence study comparing two sepsis definitions. Anaesthesia. 2018; 73:195–204. pmid:29150856
- 37. Fernando SM, Tran A, Taljaard M, Cheng W, Rochwerg B, Seely AJE, et al. Prognostic Accuracy of the Quick Sequential Organ Failure Assessment for Mortality in Patients with Suspected Infection. Annals of Internal Medicine. 2018; 168:266–275. pmid:29404582
- 38. Maitra S, Som A, Bhattacharjee S. Accuracy of quick Sequential Organ Failure Assessment (qSOFA) score and systemic inflammatory response syndrome (SIRS) criteria for predicting mortality in hospitalized patients with suspected infection: a meta-analysis of observational studies. Clinical Microbiology and Infection. 2018; 24(11):1123–112. pmid:29605565
- 39. Silcock DJ, Corfield AR, Staines H, Rooney K. Superior performance of National Early Warning Score compared with quick Sepsis-related Organ Failure Assessment Score in predicting adverse outcomes: a retrospective observational study of patients in the prehospital setting. European Journal of Emergency Medicine. 2018. pmid:30585862
- 40. Fleischmann-Struzek C, Thomas-Rüddel DO, Schettler A, Schwarzkopf D, Stacke A, Seymour CW, et al. Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data. PLoS ONE. 2018; 13(7):e0198847. pmid:30059504
- 41. Fleischmann-Struzek C, Mikolajetz A, Schwarzkopf D, Cohen J, Hartog CS, Pletz M, et al. Challenges in assessing the burden of sepsis and understanding the inequalities of sepsis outcomes between National Health Systems: secular trends in sepsis and infection incidence and mortality in Germany. Intensive Care Medicine. 44:1826–1835. pmid:30284637
- 42. Singer M, Inada-Kim M, Shankar-Hari M. Sepsis hysteria: excess hype and unrealistic expectations. The Lancet. 2019; 394(10208): 1513–1514. pmid:31657730
- 43. van Wyk F, Khojandi A, Mohammed A, Begoli E, Davis RL, Kamaleswaran R. A minimal set of physiomarkers in continuous high frequency data streams predict adult sepsis onset earlier. Int. J. Med. Inform. 2019; 122: 55–62. pmid:30623784
- 44. Mohammed A, van Wyk F, Chinthala LK, Khojandi A, Davis RL, Coopersmith CM, et al. Temporal Differential Expression of Physiomarkers Predicts Sepsis in Critically Ill Adults. Shock. 2021; 56(1):58–64. pmid:32991797
- 45. Marwick CA, Guthrie B, Pringle JEC, McLeod SR, Evans JMM, Davey PG. Identifying which septic patients have increased mortality risk using severity scores: a cohort study. BMC Anesthesiology. 2014; 14:1. pmid:24383430
- 46.
Hosmer DW Jr, Lemeshow S, Sturdivant RX. Applied Logistical Regression. 3rd ed. Hoboken NJ: John Wiley & Sons, Inc.; 2013.