PERSEVERE is a risk model for estimating mortality probability in pediatric septic shock, using five biomarkers measured within 24 hours of clinical presentation.
Here, we derive and test a temporal version of PERSEVERE (tPERSEVERE) that considers biomarker values at the first and third day following presentation to estimate the probability of a “complicated course”, defined as persistence of ≥2 organ failures at seven days after meeting criteria for septic shock, or death within 28 days.
Biomarkers were measured in the derivation cohort (n = 225) using serum samples obtained during days 1 and 3 of septic shock. Classification and Regression Tree (CART) analysis was used to derive a model to estimate the risk of a complicated course. The derived model was validated in the test cohort (n = 74), and subsequently updated using the combined derivation and test cohorts.
A complicated course occurred in 23% of the derivation cohort subjects. The derived model had a sensitivity for a complicated course of 90% (95% CI 78–96), specificity was 70% (62–77), positive predictive value was 47% (37–58), and negative predictive value was 96% (91–99). The area under the receiver operating characteristic curve was 0.85 (0.79–0.90). Similar test characteristics were observed in the test cohort. The updated model had a sensitivity of 91% (81–96), a specificity of 70% (64–76), a positive predictive value of 47% (39–56), and a negative predictive value of 96% (92–99).
Citation: Wong HR, Weiss SL, Giuliano JS Jr, Wainwright MS, Cvijanovich NZ, Thomas NJ, et al. (2014) The Temporal Version of the Pediatric Sepsis Biomarker Risk Model. PLoS ONE 9(3): e92121. https://doi.org/10.1371/journal.pone.0092121
Editor: Lyle L. Moldawer, University of Florida College of Medicine, United States of America
Received: December 11, 2013; Accepted: February 18, 2014; Published: March 13, 2014
Copyright: © 2014 Wong et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Supported by National Institutes of Health Grants RC1HL100474, RO1GM064619, RO1GM099773, and an Innovation Award from the Center for Technology Commercialization at the Cincinnati Children’s Hospital Research Foundation. Supported in part by an Institutional Clinical and Translational Science Award, NIH/NCRR 8UL1 TR000077. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: I have read the journal’s policy and have the following conflicts: Dr. Wong and the Cincinnati Children’s Hospital Research Foundation have submitted a provisional patent application for the biomarker model. PCT/US2013/025233 (published 8/15/13 as WO 2013/119871), entitled “Biomarkers of Septic Shock”, filed 2/7/13 and claiming priority to U.S. Provisional application 61/595,996, filed 2/7/12 Dr. Lindsell is named as a co-inventor in the above patent application. The remaining authors have no competing interests to report. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
We previously derived, updated, and validated the pediatric sepsis biomarker risk model (PERSEVERE; PEdiatRic SEpsis biomarkEr Risk modEl) , . PERSEVERE is based on a decision tree approach. Classification and regression tree (CART) methodology was used to estimate 28-day mortality probability for pediatric septic shock based on a panel of five biomarkers and age, with an area under the receiver operating characteristic curve of 0.88. The biomarkers that were used to derive PERSEVERE were selected objectively based on extensive genome-wide expression studies designed for the discovery of candidate stratification biomarkers –. Furthermore, the biomarkers were measured from serum samples obtained during the first 24 hours of presentation to the pediatric intensive care unit (PICU) with septic shock, which is a clinically relevant time period for assigning mortality risk in this heterogeneous population.
While the ability of PERSEVERE to assign a reliable mortality probability during the initial stages of septic shock has inherent utility at multiple levels, it fails to consider temporal changes in biomarker levels, and how these temporal changes may further inform the estimation of risk for poor outcome. This is important because the natural history of septic shock is intrinsically dynamic and subject to change in response to therapy –. Consequently, the risk for poor outcome also changes over time and it is biologically plausible that temporal changes in the PERSEVERE biomarkers may reflect this change.
In the current study we have derived a temporal version of PERSEVERE (tPERSEVERE), which incorporates biomarker measurements at two time points during the initial three days of illness to estimate the probability of a poor outcome termed “complicated course”. We subsequently test the prognostic accuracy of tPERSEVERE in an independent test cohort.
The Institutional Review Boards (IRB) of each participating institution approved secondary use of biological specimens and clinical data: Cincinnati Children’s Hospital Medical Center, The Children’s Hospital of Philadelphia, Yale University School of Medicine, Ann & Robert H. Lurie Children’s Hospital of Chicago, Children’s Hospital and Research Center Oakland, Penn State Hershey Children’s Hospital, Children’s Mercy Hospital, Children’s Hospital of Orange County, Akron Children’s Hospital, Nationwide Children’s Hospital, Children’s National Medical Center, Morgan Stanley Children’s Hospital, Columbia University Medical Center, Miami Children’s Hospital, Texas Children’s Hospital, CS Mott Children’s Hospital at the University of Michigan, St. Christopher’s Hospital for Children, and Children’s Hospital of Wisconsin. Written consent was obtained from the parents or legal guardians of all subjects enrolled, unless stated otherwise.
Derivation Cohort Study Subjects
Seventeen institutions contributed biological specimens and clinical data to a central repository, with approval from the Institutional Review Boards of each participating institution. Data collection methods were previously described in detail , . Briefly, children ≤10 years of age admitted to the PICU and meeting pediatric-specific criteria for septic shock were eligible for enrollment. After informed consent from parents or legal guardians, serum samples were obtained within 24 hours of initial presentation to the PICU with septic shock; these are referred to as “day 1” samples. Forty-eight hours after obtaining day 1 samples, a second serum sample was obtained if possible; these are referred to as “day 3” samples. Of the 355 subjects in the original PERSEVERE derivation and validation cohorts, there were 225 with biomarker data available for both day 1 and day 3. The current analysis included these 225 subjects, all of whom were enrolled between May 2002 and August 2010.
The 130 subjects excluded because day 3 biomarker data were not available were enrolled between May 2005 and December 2011. The mortality rate among the excluded subjects (19.2%) was significantly higher than that of the included subjects. Among the excluded subjects who died, 56% of the deaths occurred before day 3. The excluded and included subjects were otherwise not significantly different with respect to age, pediatric risk of mortality (PRISM) score, and incidence of a complicated course.
Test Cohort Study Subjects
The test cohort subjects were pooled from four sources, with approval from the respective Institutional Review Boards. Thirty-three subjects were included from an ongoing genomics study in pediatric septic shock being conducted at 17 participating institutions –. The enrollment criteria are identical to those for the derivation cohort. The current analysis included subjects enrolled between September 2011 and May 2013, which represents subjects enrolled after the original derivation and validation of PERSEVERE.
Eleven subjects were included from among those enrolled in a prospective quality improvement program at one institution. This institution uses PERSEVERE to benchmark outcomes for all patients admitted to the PICU with septic shock. Enrollment procedures are identical to those described above, except that there is no age restriction and the Institutional Review Board of Cincinnati Children’s Hospital Medical Center granted permission for waiver of informed consent. Serum samples were collected from residual blood samples in the clinical laboratory. Subjects from this source were enrolled between September 2012 and May 2013.
Nineteen subjects (age range: 8 days to 18 years) were participants in a prospective, observational study at Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, Illinois, evaluating nitric oxide metabolism and mitochondrial function in children with septic shock . Of the 30 subjects with septic shock enrolled in that study, 19 had serum samples available for analysis. The current analysis included subjects enrolled between May 2009 and June 2010.
Eleven subjects (age range: 2 to 20 years old) were participants in a prospective, observational study at Yale-New Haven Children’s Hospital, New Haven, Connecticut, evaluating angiopoietin levels in children with septic shock . Of the 17 subjects with septic shock enrolled in that study, 11 had serum samples available for analysis. The current analysis included subjects enrolled between September 2009 and December 2011.
For all studies, annotated clinical and laboratory data were collected daily while the participant was in the PICU. Illness severity was assessed at the time of presentation using the PRISM score . The number of organ failures during the initial 7 days of PICU admission was recorded using pediatric-specific criteria . All-cause mortality was tracked for 28 days after meeting criteria for septic shock. Our composite endpoint termed “complicated course”, is defined as persistence of two or more organ failures at seven days after meeting criteria for septic shock, or death within 28 days of presentation, as previously described –.
PERSEVERE includes C-C chemokine ligand 3 (CCL3), interleukin 8 (IL8), heat shock protein 70 kDa 1B (HSPA1B), granzyme B (GZMB), and matrix metallopeptidase 8 (MMP8). Serum concentrations of these biomarkers were measured using a multi-plex magnetic bead platform (MILLIPLEX MAP) designed for this project by the EMD Millipore Corporation (Billerica, MA). Biomarker concentrations were measured in a Luminex 100/200 System (Luminex Corporation, Austin, TX), according the manufacturers’ specifications. Assay performance data were previously published .
Initially, data are described using medians, interquartile ranges, frequencies, and percentages. Comparisons between groups used the Mann-Whitney U-test, Chi-square, or Fisher’s Exact tests as appropriate. Descriptive statistics and comparisons used SigmaStat Software (Systat Software, Inc., San Jose, CA).
CART analysis was used to derive tPERSEVERE (Salford Predictive Modeler v7.0, Salford Systems, San Diego, CA) , , , . The primary outcome variable for the modeling procedures is complicated course. The absolute day 1 and day 3 biomarker values, the percentage change in biomarker values from day 1 to day 3, and age were considered in the modeling procedures. Weighting of cases and the addition of cost for misclassification were not used in the modeling procedures. The code used to generate the model is available from the authors. Performance of the derived model is reported using diagnostic test statistics with 95% confidence intervals computed using the score method as implemented by the VassarStats Website for Statistical Computation .
Derivation of tPERSEVERE
Table 1 shows the demographic and clinical characteristics of the derivation cohort (n = 225). The 52 (23%) subjects with a complicated course had a higher median PRISM score and were less likely to have a causative organism isolated compared to the 173 subjects with a non-complicated course. No other differences were observed.
Figure 1 shows the derived model. Maximum accuracy was achieved with five biomarker variables: absolute day 1 IL8 and CCL3 values; and absolute day 3 IL8, CCL3, and HSPA1B values. None of the other biomarker variables or age contributed to predictive accuracy. There were four low probability terminal nodes for a complicated course (0.0 to 7.9% probability; terminal nodes TN1, TN2, TN4, and TN6) and four high probability terminal nodes (35 to 58% probability; TN3, TN5, TN7, and TN8). Among the 126 subjects classified as low probability, 121 (96%) had a non-complicated course and five (4%) had a complicated course. Among the 99 subjects classified as high probability, 47 (47%) had a complicated course. Table 2 shows the diagnostic test characteristics of the derived decision tree.
The classification tree consists of 7 biomarker-based decision rules and 14 daughter nodes. The classification tree includes day 1 and day 3 data for interleukin-8 (IL8) and C-C chemokine ligand 3 (CCL3), and day 3 data heat shock protein 70 kDa 1B (HSPA1B). Each node provides the biomarker serum concentration-based decision rule, and the number of subjects with and without complicated course (CC), with the respective rates. For consistency, the serum concentrations of all biomarkers are provided in pg/ml. Terminal nodes (TN) 1, 2, 4, and 6 are considered low risk nodes, whereas terminal nodes 3, 5, 7, and 8 are considered high-risk terminal nodes. To calculate the diagnostic test characteristics, all subjects in the low risk terminal nodes (n = 126) were classified as predicted to not have a complicated course, whereas all subjects in the high risk terminal nodes (n = 99) were classified as predicted to have a complicated course.
The independent test cohort consisted of 74 subjects with septic shock, of whom 16 (22%) had a complicated course. Table 1 shows the demographics and clinical characteristics of the test cohort. Compared to the derivation cohort, the test cohort subjects had a higher median age, and a higher proportion had no race reported, no causative organism isolated, and malignancy. Within the test cohort, the subjects with a complicated course had a lower proportion of males, compared to the subjects with a non-complicated course. No other differences were observed.
The test cohort subjects were classified according to the derived model. Among the 50 subjects classified as low probability for a complicated course, 47 (94%) had a non-complicated course and three (6%) had a complicated course. Among the 24 subjects classified as high probability, 13 (54%) had a complicated course. Table 2 shows the diagnostic test characteristics of tPERSEVERE in the test cohort.
tPERSEVERE was updated using all 299 subjects in the combined derivation and test cohorts. We again considered all potential biomarker variables and age in the updating process. Figure 2 shows the updated version of tPERSEVERE. Maximum accuracy was achieved with the same biomarker variables as the originally derived decision tree, except that day 3 CCL3 data no longer added to the predictive accuracy. In addition, a day 1 CCL3-based decision rule replaced the day 1 IL8-based, first-level decision rule in the originally derived decision tree. The updated version of tPERSEVERE contains four low probability terminal nodes for a complicated course (0.0 to 6.1% probability; TN1, TN2, TN4, and TN6), and four high probability terminal nodes (35.3 to 57.9% probability; TN3, TN5, TN7, and TN8). Among the 168 subjects classified as low probability, 162 (96%) had a non-complicated course and six (4%) had a complicated course. Among the 131 subjects classified as high probability, 62 (47%) had a complicated course. Table 2 shows the diagnostic test characteristics of the updated version of tPERSEVERE.
The classification tree consists of 7 biomarker-based decision rules and 14 daughter nodes. The classification tree includes day 1 and 3 interleukin-8 (IL8 data), day 1 C-C chemokine ligand 3 (CCL3) data, and day 3 heat shock protein 70 kDa 1B (HSPA1B) data. Each node provides the biomarker serum concentration-based decision rule, and the number of subjects with and without a complicated course (CC), with the respective rates. For consistency, the serum concentrations of all stratification biomarkers are provided in pg/ml. Terminal nodes (TN) 1, 2, 4, and 6 are considered low risk nodes for a complicated course, whereas terminal nodes 3, 5, 7, and 8 are considered high-risk terminal nodes for a complicated course. To calculate the diagnostic test characteristics, all subjects in the low risk terminal nodes (n = 168) were classified as predicted to not have a complicated course, whereas all subjects in the high risk terminal nodes (n = 131) were classified as predicted to have a complicated course.
The temporal version of PERSEVERE (tPERSEVERE) reasonably estimates the risk of a complicated course in a heterogeneous cohort of children with septic shock. The study subjects were drawn from multiple centers and pooled from four distinct databanks, thus adding substantial variability with regard to pathology and therapeutic interventions. Despite the concern that such heterogeneity might diminish the accuracy of predictions, tPERSEVERE performed reliably. This suggests that tPERSEVERE will be generalizable upon further testing.
We note that the positive predictive value of tPERSEVERE is substantially lower than the negative predictive value. The positive and negative predictive values of a diagnostic test are influenced by the prevalence of the outcome of interest . In this study, the prevalence of a complicated course was about 23%, so one would expect that the positive predictive value would be lower than the negative predictive value. Further, if one assumes that therapeutic interventions are beneficial and can ameliorate the risk of a poor outcome, then some of the false positives (which lower the positive predictive value and specificity) likely represent patients in whom the predicted poor outcome was prevented by therapeutic interventions.
The high sensitivity allows one to reliably identify patients at risk for a poor outcome, while the high negative predictive value allows one to identify those who are low risk. Indeed, a dichotomous interpretation of the model is that it can be used to divide a heterogeneous cohort of children with septic shock into two groups that differ by a factor of ten in the probability of a poor outcome. An alternative interpretation of the model is to view each terminal node individually, which allows for the assignment of a range of probabilities for a complicated course.
In the current study, we focused our modeling procedures on a composite outcome variable, complicated course, whereas in our previous studies we focused on 28-day mortality , . There are two primary reasons for this change in focus. First, while 28-day mortality is an important outcome variable, mortality alone does not fully capture all septic shock-associated morbidity. Organ failure has been associated with poor functional outcomes in septic shock survivors , and so the composite variable used in this study has been recently proposed as a clinically relevant study endpoint , . Second, the incidence of mortality in the study cohorts was too low for reliable modeling. We note that the five false negative subjects in the derivation cohort and the three false negatives in the test cohort all survived. This suggests that tPERSEVERE has high reliability for predicting mortality, even though it was derived to estimate the risk of a complicated course.
We previously compared the performance of PERSEVERE to that of PRISM, and found that PERSEVERE outperformed PRISM , . We have not compared tPERSEVERE to PRISM because the latter is not intended to be used as a temporal scoring system. We propose that tPERSEVERE could be used as an adjunct to traditional physiological parameters for monitoring therapeutic interventions in children with septic shock. Assuming that the risk of a complicated course is modified by therapy, tPERSEVERE provides an objective readout of therapeutic effectiveness by comparison to the baseline risk predicted by PERSEVERE. A changing risk, reflected by changing biomarkers, might even serve as a surrogate outcome variable in Phase 1 or 2 interventional clinical trials.
We note that in the initially derived tPERSEVERE, 49% of the derivation cohort subjects and 47% of the test cohort subjects occupy terminal nodes 1 and 8, which are dependent only on day 1 data. However, in the updated model, there is only one terminal node that is dependent exclusively on day 1 data (TN1), and only 28% of the subjects occupy this node. The remaining terminal nodes are informed by both day 1 and day 3 biomarker data. This suggests that as we accrue more data pertaining to changing biomarkers, we may continue to improve our ability to both predict risk and monitor therapeutic effect.
We note the limitations of our study. The test cohort, while representative of the general pediatric septic shock population, is relatively small. There is potential for selection bias in the derivation cohort because not all of the original PERSEVERE subjects were available for deriving tPERSEVERE. tPERSEVERE requires prospective testing; we are currently in the process of enrolling this prospective test cohort. Finally, tPERSEVERE, in the absence of PERSEVERE, does not capture early septic shock mortality (i.e. <3 days).
In conclusion, we have derived, tested, and updated a temporal version of PERSEVERE. Pending further prospective validation, we propose that tPERSEVERE has potential to serve as an adjunct to physiological assessments for monitoring how risk for poor outcomes changes during the period of early intervention in children with septic shock, or to serve as a surrogate outcome variable in clinical trials.
The authors thank the following research coordinators for their effort and dedication in enrolling study participants: Tasha Capozzi, Mary Ann De Liberto, Mercedes Galera-Perez, Kristin Greathouse, Lauren Hoadley, Katherine Luther, Stephanie Osborne, Amber Hughes-Schalk, Tonia Polanski, Julie Simon, Debra Spear, Lisa Steele, Naresh B. Talathoti, Tiffany Vertican, Monica Weber, Andrew A. Wiles, Trisha Williams, and Erin Zielinski
Conceived and designed the experiments: HRW CJL. Performed the experiments: SB EB KH PL. Analyzed the data: HRW CJL. Contributed reagents/materials/analysis tools: SLW JSG MSW NZC NJT GLA NA MTB MH RJF AS KM PAC TPS JN MQ AC JCF RG. Wrote the paper: HRW CJL. Read and edited the manuscript: HRW SLW JSG MSW NZC NJT GLA NA MTB MH RJF AS KM PAC TPS JN MQ AC JCF RG SB EB KH PL CJL.
- 1. Wong HR, Salisbury S, Xiao Q, Cvijanovich NZ, Hall M, et al. (2012) The pediatric sepsis biomarker risk model. Crit Care 16: R174.
- 2. Wong HR, Weiss SL, Giuliano JS, Jr., Wainwright MS, Cvijanovich NZ, et al.. (2014) Testing the prognostic accuracy of the updated pediatric sepsis biomarker risk model. PLoS One (in press, DOI 10.1371/journal.pone.0086242).
- 3. Kaplan JM, Wong HR (2011) Biomarker discovery and development in pediatric critical care medicine. Pediatr Crit Care Med 12: 165–173.
- 4. Wong HR, Lindsell CJ, Pettila V, Meyer NJ, Thair SA, et al.. (2013) A Multibiomarker-Based Outcome Risk Stratification Model for Adult Septic Shock. Crit Care Med.
- 5. Hanna W, Wong HR (2013) Pediatric sepsis: challenges and adjunctive therapies. Crit Care Clin 29: 203–222.
- 6. Wong HR (2013) Genome-wide expression profiling in pediatric septic shock. Pediatr Res 73: 564–569.
- 7. Wynn J, Cornell TT, Wong HR, Shanley TP, Wheeler DS (2010) The host response to sepsis and developmental impact. Pediatrics 125: 1031–1041.
- 8. Wong HR, Shanley TP, Sakthivel B, Cvijanovich N, Lin R, et al. (2007) Genome level expression profiles in pediatric septic shock indicate a role for altered zinc homeostasis in poor outcome. Physiol Genomics 30: 146–155.
- 9. Cvijanovich N, Shanley TP, Lin R, Allen GL, Thomas NJ, et al. (2008) Validating the genomic signature of pediatric septic shock. Physiol Genomics 34: 127–134.
- 10. Shanley TP, Cvijanovich N, Lin R, Allen GL, Thomas NJ, et al. (2007) Genome-level longitudinal expression of signaling pathways and gene networks in pediatric septic shock. Mol Med 13: 495–508.
- 11. Wong HR, Cvijanovich N, Allen GL, Lin R, Anas N, et al. (2009) Genomic expression profiling across the pediatric systemic inflammatory response syndrome, sepsis, and septic shock spectrum. Crit Care Med 37: 1558–1566.
- 12. Wong HR, Cvijanovich N, Lin R, Allen GL, Thomas NJ, et al. (2009) Identification of pediatric septic shock subclasses based on genome-wide expression profiling. BMC Med 7: 34.
- 13. Wynn JL, Cvijanovich NZ, Allen GL, Thomas NJ, Freishtat RJ, et al. (2011) The influence of developmental age on the early transcriptomic response of children with septic shock. Mol Med 17: 1146–1156.
- 14. Basu RK, Standage SW, Cvijanovich NZ, Allen GL, Thomas NJ, et al. (2011) Identification of candidate serum biomarkers for severe septic shock-associated kidney injury via microarray. Crit Care 15: R273.
- 15. Wong HR, Cvijanovich NZ, Allen GL, Thomas NJ, Freishtat RJ, et al. (2011) Validation of a gene expression-based subclassification strategy for pediatric septic shock. Crit Care Med 39: 2511–2517.
- 16. Wong HR, Freishtat RJ, Monaco M, Odoms K, Shanley TP (2010) Leukocyte subset-derived genomewide expression profiles in pediatric septic shock. Pediatr Crit Care Med 11: 349–355.
- 17. Wong HR, Cvijanovich N, Wheeler DS, Bigham MT, Monaco M, et al. (2008) Interleukin-8 as a stratification tool for interventional trials involving pediatric septic shock. Am J Respir Crit Care Med 178: 276–282.
- 18. Weiss SL, Haymond S, Ralay Ranaivo H, Wang D, De Jesus VR, et al. (2012) Evaluation of asymmetric dimethylarginine, arginine, and carnitine metabolism in pediatric sepsis. Pediatr Crit Care Med 13: e210–218.
- 19. Giuliano JS, Jr., Tran K, Li FY, Northrup V, Tala JA, et al.. (2013) The temporal kinetics of circulating angiopoietin levels in children with sepsis. Pediatr Crit Care Med.
- 20. Pollack MM, Patel KM, Ruttimann UE (1997) The Pediatric Risk of Mortality III–Acute Physiology Score (PRISM III-APS): a method of assessing physiologic instability for pediatric intensive care unit patients. J Pediatr 131: 575–581.
- 21. Goldstein B, Giroir B, Randolph A (2005) International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med 6: 2–8.
- 22. Mickiewicz B, Vogel HJ, Wong HR, Winston BW (2013) Metabolomics as a novel approach for early diagnosis of pediatric septic shock and its mortality. Am J Respir Crit Care Med 187: 967–976.
- 23. Abulebda A, Cvijanovich N, Thomas NJ, Allen GL, Anas N, et al. (2013) Post-intensive care unit admission fluid balance and pediatric septic shock outcomes: A risk-stratified analysis. Crit Care Med 42: 397–403.
- 24. Xiao W, Mindrinos MN, Seok J, Cuschieri J, Cuenca AG, et al. (2011) A genomic storm in critically injured humans. J Exp Med 208: 2581–2590.
- 25. Che D, Liu Q, Rasheed K, Tao X (2011) Decision tree and ensemble learning algorithms with their applications in bioinformatics. Adv Exp Med Biol 696: 191–199.
- 26. Muller R, Mockel M (2008) Logistic regression and CART in the analysis of multimarker studies. Clin Chim Acta 394: 1–6.
- 27. VassarStats Website for Statistical Computation. http://faculty.vassar.edu/lowry/VassarStats.html. Accessed September 1, 2013.
- 28. Typpo KV, Petersen NJ, Hallman DM, Markovitz BP, Mariscalco MM (2009) Day 1 multiple organ dysfunction syndrome is associated with poor functional outcome and mortality in the pediatric intensive care unit. Pediatr Crit Care Med 10: 562–570.