The authors have declared that no competing interests exist.
There has been renewed interest in lactate as a risk biomarker in sepsis and septic shock. However, the ability of the odds ratio (OR) and change in the area under the receiver operator characteristic curve (AUC-ROC) to assess biomarker added-value has been questioned.
A sepsis cohort was identified from the ICU database of an Australian tertiary referral hospital using APACHE III diagnostic codes. Demographic information, APACHE III scores, 24-hour post-admission patient lactate levels, and hospital mortality were accessed.
Hospital mortality was modelled using a base predictive logistic regression model and sequential addition of admission lactate, lactate clearance ([lactateadmission—lactatefinal]/lactateadmission), and area under the lactate-time curve (LTC). Added-value was assessed using lactate index OR; AUC-ROC difference (base-model versus lactate index addition); net (mortality) reclassification index (NRI; range -2 to +2); and net benefit (NB), the number of true positives per patient adjusted for the number of false positives. The data set comprised 717 patients with mean(SD) age and APACHE III score 61.1(16.5) years and 68.3(28.2) respectively; 59.2% were male. Admission lactate was 2.3(2.5) mmol/l; with lactate of ≥ 4 mmol/L (37% hospital mortality) in 17% and patients with lactate < 4 mmol/L having 18% hospital mortality. The admission base-model had an AUC-ROC = 0.81 with admission lactate OR = 1.127 (95%CI: 1.038, 1.224), AUC-ROC difference of 0.0032 (-0.0037, 0.01615; P = 0.61), and NRI 0.240(0.030, 0.464). The over-time model had an AUC-ROC = 0.86 with (i) clearance OR = 0.771, 95%CI: 0.578, 1.030; P = 0.08; AUC-ROC difference 0.001 (-0.003, 0.014; P = 0.78), and NRI 0.109(-0.193, 0.425) and (ii) LTC OR = 0.997, 95%CI: 0.989, 1.005, P = 0.49; AUC-ROC difference 0.004 (-0.002, 0.004; P = 0.34), and NRI 0.111(-0.222, 0.403). NB was not incremented by any lactate index.
Lactate added-value assessment is dependent upon the performance of the underlying predictive model and should incorporate risk reclassification and net benefit measures.
The recent interest in the role of lactate as a biomarker of risk in the critically ill and in sepsis and septic shock in particular [
The statistical methods used in the assessment of lactate as a biomarker in sepsis [
With the above caveats in mind, we undertook analysis of the added value of lactate as a risk [
St Vincent's Hospital Melbourne in Victoria is a 400-bed university affiliated tertiary referral hospital. The single intensive care unit of 20 beds admits approximately 1700 patients each year and they include those undergoing cardiac surgery and neurosurgery. Patient observations are prospectively entered within a clinical information system (IntelliSpace Critical Care and Anesthesia, Philips) which also imports the results of routine biochemical and haematology tests. In addition, detailed patient information is entered within a second database that provides information to the Australian and New Zealand Intensive Care Society (ANZICS) adult patient database [
Continuous variables were reported as mean(SD) and statistical significance was ascribed at P ≤ 0.05; analysis was conducted using Stata™ V14.2 (2016, College Station, TX) and R statistical software (V 3.3.1).
The overall modelling process is shown in
Model | Functionals | Development |
---|---|---|
Initial logistic | Non-linear covariate effects (fractional polnomials) | Information criteria: AIC, BIC |
Interactions | Discrimination: AUC-ROC | |
Collinearity check | Calibration: Hosmer-Lemeshow test | |
Polynomial calibration plots | ||
In-sample and out-of-sample predictive bias | ||
Overfitting | ||
Lactate added | ||
Lactate form | Initial lactate | |
Fractional lactate clearance: (admission-final)/admission | ||
Area under lactate-time curve | ||
Sensitivity analysis: | ||
Lactate change: admission-final | ||
Lactate ratio: final/admission | ||
Log lactate ratio: log(final/admission) | ||
Added value | ||
AUC-ROC difference | ||
Net reclassification index (NRI) | Bootstrapped confidence intervals | |
Integrated discrimination improvement index (IDI) | Bootstrapped confidence intervals | |
Net benefit |
The modelling process was considered in two stages: a base logistic model for hospital outcome was developed with particular attention paid to the functional form of continuous variables (using fractional polynomials [
Calibration plots (observed binary responses versus predicted probabilities) were undertaken using 'givitiR' [
Categorical variables were parameterised as indicator variables including calendar years; the latter were included in all models.
the primary analysis followed the literature examples and addressed initial lactate (mmol/L), fractional lactate clearance ([lactateadmission—lactatefinal]/lactateadmissionl) [
The added value [ AUC-ROC difference (model with and without the marker) using bootstrap 95% intervals (n = 1000). The NRI (theoretical range -2 to +2) computed by assessing the change (movement “up” or “down” within categories) in the classification of the risk / probability of patients with respect to the end point (hospital mortality) by the addition of the new marker in question; that is, NRI = The IDI, a complement to the AUC-ROC, is defined as: IDI = (ISnew − ISold) − (IPnew − IPold), where IS is the integral of sensitivity over all possible cut-off values and IP is the corresponding integral of “1-specificity” [ Bootstrap 95% CI (n = 1000) of both NRI and IDI for event, non-event and overall are reported as opposed to P-values [ Net benefit, the number of true positives per patient adjusted for the number of false positives, that is:
Net benefit is typically used to assess the value of a diagnostic test over a range of "probability thresholds" (relative value of treatment if disease is present to value of avoiding unnecessary treatment). However, net benefit has been demonstrated to be a proper measure of model performance [
The data set (collected over 7 years) comprised 717 patients with mean(SD) age and APACHE III score 61.1(16.5) years and 68.3(28.2.2) respectively; 59.2% were male and 27% were ventilated in the first 24 hours. ICU and hospital length of stay (days) were 4.3(6.4) median 2, and 23.7(28.5) median 15, respectively. ICU and hospital mortality outcome were 12.3% and 21% respectively. The Charlson Comorbidity Index ranged from 0 to 15, median 1 and interquartile range 3. On admission lactate was 2.3(2.5) mmol/l; 17% of patients had a lactate of ≥ 4 mmol/l, with 37% hospital mortality and patients with a lactate < 4 mmol/l had a hospital mortality of 18%.
The performance of univariate predictors of hospital outcome was compared between initial lactate (OR 1.185, 95%CI: 1.049, 1.270), lactate clearance (OR 0.640. 95%CI: 0.501, 0.817), area under the lactate-time curve (OR 1.013, 95%CI: 1.008, 1.018) and APACHE III score (OR 1.053, 95%CI: 1.044, 1.063); only the latter demonstrated non-linear effect form and was parameterised as a third-degree fractional polynomial. As seen in
Although the logistic AUC-ROC differed between each of the predictors, a different perspective results when comparing the net benefit curves, as seen in
The best fitting model (n = 681 evaluable patients) incorporated age and initial lactate (linear effects), index of comorbidity (as a 0.5, 3 fractional polynomial) and categorical variables indicating coma, cirrhosis and a heart rate ≥ 150 beats per minute. Model parameter estimates, diagnostics and risk reclassification measures are seen in
Odds Ratio | P | Lower 95%CI | Upper 95% CI | |
Age (years) | 1.032 | 0.000 | 1.015 | 1.048 |
CCI: FP 0.5 | 0.199 | 0.023 | 0.049 | 0.800 |
CCI: FP 3 | 40.045 | 0.000 | 13.043 | 122.948 |
Cirrhosis | 3.669 | 0.004 | 1.497 | 8.990 |
Coma | 80.901 | 0.000 | 14.846 | 440.857 |
MPM_O_HR | 5.680 | 0.002 | 1.854 | 17.403 |
Lactate | 1.127 | 0.005 | 1.038 | 1.224 |
Year 2 | 1.044 | 0.917 | 0.465 | 2.343 |
Year 3 | 0.627 | 0.219 | 0.298 | 1.320 |
Year 4 | 0.913 | 0.803 | 0.447 | 1.867 |
Year 5 | 0.400 | 0.013 | 0.194 | 0.824 |
Year 6 | 0.449 | 0.036 | 0.213 | 0.949 |
Year 7 | 0.387 | 0.058 | 0.145 | 1.034 |
Hosmer-Lemeshow statistic | 0.580 | |||
AUC-ROC | 0.785(0.727, 0.819) | |||
Condition number | 11.8 | |||
In-sample bias | 0.97% | |||
Over-fitting | 7.8% | |||
Out-of-sample-bias | 8.7% | |||
Estimate | P | Lower 95%CI | Upper 95% CI | |
Event | -0.241 | -0.375 | -0.071 | |
Non-event | 0.481 | 0.370 | 0.577 | |
Overall | 0.240 | 0.030 | 0.464 | |
Event | 0.007 | -0.0001 | 0.030 | |
Non-event | 0.002 | -0.0001 | 0.008 | |
Overall | 0.010 | -0.0001 | 0.038 | |
AUC-ROC difference | 0.003 | 0.061 | -0.004 | 0.016 |
CCI, Charlson comorbidity index. FP, fractional polynomial. HR, heart rate. H-L, Hosmer-Lemeshow
AUC-ROC, area under the receiver operator characteristic curve. diff, difference (model with and without lactate). NRI(>0), category free net reclassification index. IDI, integrated discrimination improvement index.
As a sensitivity analysis with respect to the added value of a biomarker in a “poorly” performing model [
Odds Ratio | P | Lower 95%CI | Upper 95% CI | |
Age (years) | 1.026 | 0.001 | 1.011 | 1.042 |
CCI: FP 0.5 | 0.232 | 0.032 | 0.061 | 0.882 |
CCI: FP 3 | 23.086 | 0.000 | 8.356 | 63.784 |
Cirrhosis | 3.259 | 0.008 | 1.371 | 7.750 |
Lactate | 1.201 | 0.000 | 1.120 | 1.289 |
Year 2 | 1.058 | 0.885 | 0.494 | 2.266 |
Year 3 | 0.639 | 0.214 | 0.315 | 1.296 |
Year 4 | 0.920 | 0.809 | 0.467 | 1.813 |
Year 5 | 0.433 | 0.016 | 0.220 | 0.854 |
Year 6 | 0.407 | 0.016 | 0.196 | 0.848 |
Year 7 | 0.482 | 0.114 | 0.195 | 1.191 |
Hosmer-Lemeshow statistic | 0.770 | |||
AUC-ROC | 0.740(0.675, 0.777) | |||
Estimate | P | Lower 95%CI | Upper 95% CI | |
Event | -0.186 | -0.316 | -0.026 | |
Non-event | 0.604 | 0.467 | 0.657 | |
Overall | 0.418 | 0.185 | 0.587 | |
Event | 0.031 | 0.009 | 0.066 | |
Non-event | 0.008 | 0.002 | 0.018 | |
Overall | 0.040 | 0.0120 | 0.084 | |
AUC-ROC difference | 0.033 | 0.031 | 0.008 | 0.065 |
CCI, Charlson comorbidity index. FP, fractional polynomial. HR, heart rate. H-L, Hosmer-Lemeshow
AUC-ROC, area under the receiver operator characteristic curve. diff, difference (model with and without lactate). NRI(>0), category free net reclassification index. IDI, integrated discrimination improvement index.
Both the scatter plot of fractional clearance against initial lactate and Kaiser’s R (= 0.322) favoured fractional clearance over lactate change. However, the minimum slope of the reduced major axis (= 1.112) of log lactateinitial-log lactatefinal suggested efficacy for the log lactate ratio which was also considered.
The best fitting model (n = 662 evaluable patients) incorporated age and clearance (linear effect), index of comorbidity (as a 0.5, 3 fractional polynomial), APACHE III score (third-degree fractional polynomial) and categorical variables indicating coma and cirrhosis (the variable denoting heart rate ≥ 150 beats per minute was non-significant at P = 0.123 and was removed from the model with no change of information criteria). Model parameter estimates, diagnostics and risk reclassification measures are seen in
Odds Ratio | P | Lower 95%CI | Upper 95% CI | |
Age (years) | 1.017 | 0.057 | 1.000 | 1.035 |
CCI: FP 0.5 | 0.237 | 0.047 | 0.057 | 0.983 |
CCI: FP 3 | 28.496 | 0.000 | 8.336 | 97.411 |
APACHE III score: FP 3 | 7.182 | 0.000 | 4.181 | 12.337 |
Coma | 11.646 | 0.021 | 1.458 | 93.036 |
Cirrhosis | 2.955 | 0.029 | 1.118 | 7.809 |
Clearance | 0.771 | 0.078 | 0.578 | 1.030 |
Year 2 | 1.433 | 0.417 | 0.602 | 3.412 |
Year 3 | 0.829 | 0.658 | 0.361 | 1.905 |
Year 4 | 1.053 | 0.898 | 0.478 | 2.321 |
Year 5 | 0.472 | 0.067 | 0.211 | 1.055 |
Year 6 | 0.623 | 0.277 | 0.266 | 1.462 |
Year 7 | 0.377 | 0.098 | 0.119 | 1.197 |
Hosmer-Lemeshow statistic | 0.470 | |||
AUC-ROC | 0.838(0.784, 0.865) | |||
Condition number | 11.8 | |||
In-sample bias | 1.45% | |||
Over-fitting | 5.6% | |||
Out-of-sample-bias | 6.9% | |||
Estimate | P | Lower 95%CI | Upper 95% CI | |
Event | -0.135 | -0.310 | 0.217 | |
Non-event | 0.244 | -0.112 | 0.416 | |
Overall | 0.109 | -0.193 | 0.425 | |
Event | 0.002 | -0.003 | 0.016 | |
Non-event | 0.001 | -0.001 | 0.004 | |
Overall | 0.002 | -0.0030 | 0.020 | |
AUC-ROC difference | 0.001 | 0.78 | -0.003 | 0.014 |
CCI, Charlson comorbidity index. FP, fractional polynomial. H-L, Hosmer-Lemeshow
AUC-ROC, area under the receiver operator characteristic curve. diff, difference (model with and without clearance). NRI(>0), category free net reclassification index. IDI, integrated discrimination improvement index.
A second sensitivity analysis was performed, restricting the lactate time span (admission to last) to ≥ 6 hours; the clearance estimate was OR 0.777, 95%CI: 0.583, 1.037. The decision curve analysis graph of net benefit (24-hour model versus 24 hour model plus clearance) was unchanged (
The same base model as above for lactate clearance analysis was used. Area under the lactate-time curve (n = 603 evaluable patients) was non-significant at OR 0.997, 95%CI: 0.989, 1.005, P = 0.49. Model parameter estimates, diagnostics and risk reclassification measures are seen in
Odds Ratio | P | Lower 95%CI | Upper 95% CI | |
Age (years) | 1.010 | 0.278 | 0.992 | 1.029 |
CCI: FP 0.5 | 0.284 | 0.068 | 0.073 | 1.097 |
CCI: FP 3 | 33.894 | 0.000 | 9.423 | 121.918 |
APACHE III score: FP 3 | 7.337 | 0.000 | 3.991 | 13.488 |
Coma | 12.620 | 0.031 | 1.258 | 126.555 |
Cirrhosis | 2.308 | 0.107 | 0.834 | 6.382 |
AUC-lactate | 0.997 | 0.494 | 0.989 | 1.005 |
Year 2 | 1.484 | 0.397 | 0.595 | 3.703 |
Year 3 | 0.820 | 0.661 | 0.339 | 1.986 |
Year 4 | 1.018 | 0.966 | 0.440 | 2.360 |
Year 5 | 0.481 | 0.087 | 0.207 | 1.114 |
Year 6 | 0.641 | 0.333 | 0.261 | 1.576 |
Year 7 | 0.404 | 0.124 | 0.127 | 1.282 |
Hosmer-Lemeshow statistic | 0.180 | |||
AUC-ROC | 0.823(0.766, 0.840) | |||
Condition number | 6.7 | |||
In-sample bias | 1.03% | |||
Over-fitting | 4.9% | |||
Out-of-sample-bias | 6.03% | |||
Estimate | P | Lower 95%CI | Upper 95% CI | |
Event | 0.328 | -0.349 | 0.468 | |
Non-event | -0.217 | -0.284 | 0.352 | |
Overall | 0.111 | -0.222 | 0.403 | |
Event | -0.0001 | -0.004 | 0.008 | |
Non-event | -0.0001 | -0.001 | 0.002 | |
Overall | 0.002 | -0.0010 | 0.010 | |
AUC-ROC difference | -0.002 | 0.340 | -0.004 | 0.005 |
CCI, Charlson comorbidity index. APIII, APACHE III. FP, fractional polynomial. AUC-lactate, area under the lactate-time curve. H-L, Hosmer-Lemeshow AUC-ROC, area under the receiver operator characteristic curve. diff, difference (model with and without area under the lactate-time curve). NRI(>0), category free net reclassification index. IDI, integrated discrimination improvement index.
Log lactate ratio, when added to the base model above was non significant (OR 1.349, 95%CI: 0.892, 2.040; P = 0.156) and the net benefit curves were again almost coincident (graph not shown).
In agreement with prior reports [
Analysis using a single biological measurement will be subject to random measurement error and the (regression) coefficient estimate will be biased to the null (regression dilution bias). Repeated measurement, as in the area of the lactate-time curve, would be, prima facie, the preferred measurement variable [
Previous multivariate analyses have used a variety of modelling approaches to ascertain the added value of lactate; ranging from a focus on an ensemble of specific lactate indices with or without other predictive variables [
Of more import, with deletion of two covariates, a poorly performing model (in terms of the scalar value of the AUC-ROC) produced a statistical (P = 0.03) difference in the AUC-ROC with addition of lactate and a substantive increase in the NRI (0.240 to 0.418), with the major re-classification occurring in the non-event category, but no discernible difference in net benefit. Neither of the over-time multivariate models, starting with a base model AUC-ROC of 0.86, produced significance in lactate indices, differences in AUC-ROC or net benefit, although the level of net benefit from threshold probabilities 0.4–1 was greater than 0.05 compared with the admission model. These observations are consistent with studies showing that the ability of a biomarker to add value to an existing model will depend upon the existing performance (value increments will be easier in poorly performing models [
Reports on the added value of lactate in sepsis have used AUC-ROC differences almost exclusively; but the inherent problem with this strategy is the clinical interpretability of (small) difference in AUC-ROC and what level of difference is meaningful [
Decision curve analysis and the concept of net benefit have not been previously applied to the study of lactate indices as septic risk markers. As net benefit incorporates both true positives and false positives, it can be used to compare models across a range of probability thresholds and is informative as to clinical value [
We were unable to demonstrate increments of net benefit for lactate as a sepsis risk biomarker, in either univariate or multivariate settings; a finding that is akin to the conclusion of the meta-analysis of Zhang and Xu [
The two randomised controlled trials which have addressed the issue of lactate-guided therapy have also not resolved the question. Jones and co-workers [
The current study proceeded from a modest sample size and did not formally address the utility of lactate with admission values ≥ 4 mmol/l, as in the Rivers trial [
We conclude that the ability to demonstrate lactate as a sepsis risk biomarker depends upon the performance of the underlying base model and any such demonstration must embrace other assessments of added value such as risk reclassification and net benefit. Current lactate markers, in particular, initial lactate and lactate clearance, may be subject to regression dilution and regression to the mean.
(DOCX)
(DOCX)
(DOCX)
(DOCX)