Figures
Abstract
Background
We aimed to develop an adaptable prognosis prediction model that could be applied at any time point during the treatment course for patients with cancer receiving chemotherapy, by applying time-series real-world big data.
Methods
Between April 2004 and September 2014, 4,997 patients with cancer who had received systemic chemotherapy were registered in a prospective cohort database at the Kyoto University Hospital. Of these, 2,693 patients with a death record were eligible for inclusion and divided into training (n = 1,341) and test (n = 1,352) cohorts. In total, 3,471,521 laboratory data at 115,738 time points, representing 40 laboratory items [e.g., white blood cell counts and albumin (Alb) levels] that were monitored for 1 year before the death event were applied for constructing prognosis prediction models. All possible prediction models comprising three different items from 40 laboratory items (40C3 = 9,880) were generated in the training cohort, and the model selection was performed in the test cohort. The fitness of the selected models was externally validated in the validation cohort from three independent settings.
Results
A prognosis prediction model utilizing Alb, lactate dehydrogenase, and neutrophils was selected based on a strong ability to predict death events within 1–6 months and a set of six prediction models corresponding to 1,2, 3, 4, 5, and 6 months was developed. The area under the curve (AUC) ranged from 0.852 for the 1 month model to 0.713 for the 6 month model. External validation supported the performance of these models.
Citation: Uneno Y, Taneishi K, Kanai M, Okamoto K, Yamamoto Y, Yoshioka A, et al. (2017) Development and validation of a set of six adaptable prognosis prediction (SAP) models based on time-series real-world big data analysis for patients with cancer receiving chemotherapy: A multicenter case crossover study. PLoS ONE 12(8): e0183291. https://doi.org/10.1371/journal.pone.0183291
Editor: Wen-Lung Ma, China Medical University, TAIWAN
Received: August 4, 2016; Accepted: August 2, 2017; Published: August 24, 2017
Copyright: © 2017 Uneno et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The study was supported by research grants from Chugai Pharmaceutical Co., Ltd, Mitsui Knowledge Industry Co., Ltd, and Taiho Pharmaceutical Co., Ltd. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Manabu Muto received research grant from Chugai Pharmaceutical Co., Ltd, Mitsui Knowledge Industry Co., Ltd, and Taiho Pharmaceutical Co., Ltd. This does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials.
Introduction
Prognosis prediction is one of the most clinically relevant issues for both physicians and patients with cancer. Accurate prognosis prediction can help physicians select the optimal treatment (e.g., discontinuation of aggressive interventions at the end-of-life). It is well-known that patients’ understanding of their prognosis is poor, [1, 2] and overestimation of their life expectancy could positively affect the preference for more aggressive treatments. [3,4]
Many attempts have been made to address this issue, and several prognosis prediction models have been proposed: the palliative prognostic index (PPI), palliative prognostic (PaP) score, delirium-palliative prognostic (D-PaP) score, and prognosis in palliative care study (PiPS) models. [5–8] However, these models were developed using the variables from a single time point (e.g., admission date or baseline assessment date), which limits the use of these models under baseline conditions. Because patient’s condition during the treatment course can change from the baseline, the development of an adaptable prognosis prediction model that could be applied at any time point after the initiation of chemotherapy, is eagerly warranted in daily clinical practice.
To develop adaptable prognosis prediction models using laboratory variables, we used every serial laboratory variable monitored within 1 year before a death event as time-inclusive data.
Patients and methods
Study population and data collection
Between April 2004 and September 2014, a total of 4,997 patients with cancer who had received systemic chemotherapy were registered in the prospective cohort database (CyberOncology®, Cyber Laboratory Inc., Tokyo, Japan) at the Kyoto University Hospital. [9] (Fig 1A) Among them, 2,693 patients with death records at the time of this analysis (March 2015) were considered eligible for this study (Table 1). These patients were randomly divided into training (n = 1,341) and test cohorts (n = 1,352). One patient from the training cohort and two from the test cohort were excluded because of insufficient data for this analysis. (Fig 1A) In total, 7,606,544 laboratory data at 159,316 time points covering 1,147 items [e.g., white blood cell counts and albumin (Alb) levels] that had been monitored within 1 year before death events were retrieved from CyberOncology® and electronic medical records. To improve the feasibility of the models in daily clinical practice, we considered that it was preferable to be comprised of routinely monitored laboratory items. Therefore, we excluded items monitored at less than 50% of all time points (e.g., tumor markers) and urine-based data. Finally, 3,471,521 laboratory data at 115,738 time points covering 40 laboratory items were utilized for constructing prognosis prediction models. (Fig 1A) For external validation, we enrolled 75 patients corresponding to 1,581 laboratory data at 527 time points from the Mitsubishi Kyoto Hospital between February 2014 and August 2014, 37 patients corresponding to 582 laboratory data at 194 time points from the Kyoto Min-iren Chuo Hospital between April 2014 and May 2015 and 255 patients corresponding to 765 laboratory data at 255 time points from the Japan prognostic assessment tools validation (J-ProVal) study between September 2012 and April 2014. [10] (S1 Table) The J-ProVal study was a multicenter prospective cohort study involving a total of 58 palliative care facilities in Japan. We included the patients who were receiving chemotherapy from the J-ProVal study cohort. This study was approved by the institutional review boards of the Kyoto University Hospital (E2200) and other participating hospitals.
(A) Flow of the study. (B) Comparison of time-series real-world big data analysis with conventional methods. Upper: Time-series real-world big data analysis included every time point monitored within 1 year before the death event in the analysis as an explanatory variable. Each laboratory variable was used as time-inculsive data, classified event data and control data bounded by n months before the date death. Lower: The conventional method involved single time point (such as admission date or baseline assessment date) as an explanatory variable. AUC, area under the curve; ROC, receiver operating characteristic curve; Black arrow, explanatory variable.
Statistical analysis
Development of adaptable prognosis prediction models in the training cohort.
First, we linked 1,730,535 laboratory variables to 57,581 time points and used these time-inclusive data. We aimed to develop a set of adaptable prognosis prediction models that could predict death events within n months (n = 1–6). Therefore, when constructing such model, laboratory variables monitored within n months before the death event were classified as event data, whereas other data monitored between n + 1 and 12 months before the event were classified as control data. [11] (Fig 1B) Because the positive predictive value (PPV) and negative predictive value (NPV) are affected by imbalances in the sample size between the control and event data, they were adjusted using the oversampling method to make PPV and NPV interpretable. [12, 13]
We considered that three explanatory variables were feasible for the developed model to be generalizable and easy-to-use in daily clinical practice. Therefore, all possible combinations of three different items (e.g., C-reactive protein, calcium, and total protein) from 40 laboratory items (40C3 = 9,880) were generated; the same number of prognosis prediction models comprising three different items were tested using regularized logistic regression analysis, and the coefficients of explanatory variables were calculated. [12] Because our current prognostic models handled the discrimination of binary survival outcomes within n months (n = 1–6), we considered that logistic regression was more feasible than the Cox regression model.
Model selection in the test cohort.
The area under the curve (AUC) and cutoff value were estimated via receiver operating characteristic (ROC) curve analysis in the test cohort comprising 1,740,986 laboratory data at 58,157 time points. Model selection was performed on the basis of the AUC values derived from prognosis models predicting death events within n months (n = 1–6).
External validation of selected models.
External validation was performed using 2,163 laboratory data at 721 time points from the two independent community hospitals and 765 laboratory data at 255 time points from J-ProVal study cohort. Because the participants in the J-ProVal study who survived for more than 180 days were censored, the validation of the six month model was skipped for the J-ProVal study cohort.
All statistical analyses were performed using GNU R software (version 3.2.0; R Project for Statistical Computing, Vienna, Austria) and Python (version 3.5.1; Python Software Foundation).
Results
Development of prognosis prediction models
A total of 59,280 prediction models comprising three laboratory items for death events within n months (n = 1–6) were generated using the training cohort (9,880 models per each prediction month). The performance of these models was evaluated on basis of AUC using the test cohort. Among the top 10 prediction models based on AUC values, the combination of Alb, lactate dehydrogenase (LDH), and neutrophil (Neu) levels yielded the best and second best prediction performance at four (2, 3, 4, and 5 months) and two (1 and 6 months) of the six tested time points, respectively, (S2 and S3 Tables) and was therefore selected as our prognosis prediction model. A time-series heat map of the three items (Alb, LDH, and Neu) revealed gradual increases in LDH and Neu levels and a decrease in Alb as patients approached the death event. (Fig 2) AUC, sensitivity, specificity, PPV, NPV, and accuracy of prediction model corresponding to each prediction period (1, 2, 3, 4, 5, and 6 months) are summarized in Table 2, and the regression equations are summarized in Table 3.
Right: Mean Alb, Neu, and LDH levels with 95% confidence intervals.
Predictive performance among different tumor types
Next, we tested the fitness of these prediction models across 10 different tumor types. With all tumor types, our models yielded a high level of predictive performance, with AUC values consistently > 0.750 for 1 and 2 months before death events (Fig 3).
External validation of new prognosis prediction models
External validation was performed using data from two independent community hospitals and the J-ProVal study cohort. The outcomes from two independent community hospitals and the J-ProVal study cohort further supported the good performance of these models as AUC values of > 0.730 and > 0.700 were achieved, respectively (Fig 4) (S4 Table).
Comparison of area under the receiver operating characteristic curve (AUC) values for the prediction of death events among the (A) test cohort at the Kyoto University Hospital and (B) validation cohort (blue: Kyoto Mitsubishi Hospital, green: Kyoto Min-iren Chuo Hospital, red: J-ProVal study).
Discussion
In this study, using time-series real-world big data, we successfully developed a set of six adaptable prognosis prediction models for patients with cancer receiving chemotherapy. Our current models allow physicians to select the optimal model according to the clinical condition of the patient and apply them repeatedly at any time point. In other words, physicians can re-estimate the prognosis of patients at any time point after the initiation of chemotherapy. We expect that our adaptable models will provide valuable information to both physicians and patients during decision-making for optimal treatment (e.g., indication of palliative chemotherapy or discontinuation of aggressive interventions or referral timing for palliative care service), which is necessary in daily clinical practice.
Previous prognosis prediction models were developed using the variables from a single time point (e.g., admission date or baseline assessment date). In contrast, our current models are based on more robust data than previous models, utilizing every laboratory variables monitored within 1 year before death events. (43 different time points per patient, Fig 1B) To the best of our knowledge, no previous studies have applied a similar method to the construction of prognosis prediction models. We have named this method as “time-series real-world big data analysis.”
With regard to statistical analysis, we selected logistic regression rather than Cox regression, which is commonly used to develop the prognostic models. Cox regression models are commonly used to estimate the survival, while logistic regression models are to predict a binary outcome. Because our current prognostic models are only for discriminating the binary survival outcomes within a specific time frame, we considered logistic regression is more feasible. In addition, we compared the prediction performance between time-dependent Cox regression models and logistic regression models in the current internal validation cohorts and found that there were no significant differences in prediction performance between logistic regression models and time-dependent Cox regression models. (S5 Table)
Our models have another unique feature. Unlike previously published models, which were designed to predict survival durations from specific time points, our models were designed to predict death events within 1–6 months and used different sets of coefficients according to the intended prediction period. For example, if we intend to predict death events within the following 6 months, we would use the model with a coefficient set at 6 months: (p = 1/(1 + exp(0.334 × Alb − 0.001 × LDH − 0.012 × neutrophil + 0.031)) (Table 3) Similarly, predictions of death events within 3 months would require a corresponding change in the coefficient set: (p = 1/(1 + exp(0.482 × Alb − 0.001 × LDH − 0.017 × neutrophil + 0.039)) (Table 3)
In this study, a combination of Alb, LDH, and Neu was selected from 40 explanatory variables according to the prediction performance estimated by AUC. A decrease in Alb levels reflects poor nutrition and/or hypercatabolism in patients with cancer, [14–16] whereas elevated LDH levels are closely associated with an increased tumor volume [14, 16, 17]. An elevated neutrophil count or neutrophil-to-lymphocyte ratio has also been reported as an independent prognostic factor in many studies [18, 19]. Therefore, the use of these three factors in our prediction models represents a rational choice.
Moreover, we tested the 2-item and 4-item prognostic models to compare their performance. Models using more items show better performance; however, the difference in AUC between the 3- and 4-item prognostic models was smaller than that between 2- and 3-item prognostic models. (S6 Table) Based on these results, we selected 3-item prognostic models in this study.
Our current prediction models exhibited similar levels of predictive performance across different tumor types according to AUC estimations, suggesting that the models are generalizable to all patients with cancer receiving chemotherapy. The ability to reproduce a high AUC in three independent external validation cohorts further provides strong support for the generalizability of these models.
Undoubtedly, accurate prognostic predictions facilitate the provision of optimal treatments to patients; however, physicians must handle such information carefully because some patients might not want to know about their prognoses. In this regard, the results obtained from any prognosis prediction models should be used properly in daily clinical practice in accordance with individual patients’ requests.
A potential limitation of our study is the lack of robust evidence that shows the clinical utility of prognosis prediction models including our current models. To clarify these clinical questions, we plan to assess the clinical utility of our models in a large prospective study.
In summary, by applying “time-series real-world big data analysis,” we have developed robust and adaptable prognosis prediction models for patients with cancer receiving chemotherapy. We expect that this “time-series real-world big data analysis” will be a promising tool for the future construction of prediction models for other purposes.
Supporting information
S1 Table. Patient characteristics of the validation cohorts at the Kyoto Mitsubishi, Kyoto Min-iren Chuo hospitals and the J-ProVal study.
J-ProVal study, Japan prognostic assessment tools validation study.
https://doi.org/10.1371/journal.pone.0183291.s001
(XLSX)
S2 Table. Top 10 prediction models for the occurrence of death events within 1–3 months.
ALB, albumin; ALP, alkaline phosphatise; Ca, calcium; Cl, chloride; CRP, C-reactive protein; K, potassium; LDH, lactate dehydrogenase; MCH, mean corpuscular hemoglobin; MCHC, mean corpuscular hemoglobin concentration; MCV, mean corpuscular volume; Na, sodium; PPV, positive predictive value; NPV, negative predictive value.
https://doi.org/10.1371/journal.pone.0183291.s002
(XLSX)
S3 Table. Top 10 prediction models for the occurrence of death events within 4–6 months.
ALB, albumin; ALP, alkaline phosphatise; Ca, calcium; Cl, chloride; CRP, C-reactive protein; K, potassium; LDH, lactate dehydrogenase; MCH, mean corpuscular hemoglobin; MCHC, mean corpuscular hemoglobin concentration; MCV, mean corpuscular volume; Na, sodium; PPV, positive predictive value; NPV, negative predictive value.
https://doi.org/10.1371/journal.pone.0183291.s003
(XLSX)
S4 Table. External validation of albumin, lactate dehydrogenase, and neutrophil models for prediction of death events within 1–6 months.
PPV, positive predictive value; NPV, negative predictive value.
https://doi.org/10.1371/journal.pone.0183291.s004
(XLSX)
S5 Table. Comparison of the prediction performance between time-dependent Cox regression models and logistic regression models.
AUC, area under the curve; LR, logistic regression.
https://doi.org/10.1371/journal.pone.0183291.s005
(XLSX)
S6 Table. Comparison of prediction performance among 2-item, 3-item and 4-item prognostic models in the current internal and external validation cohorts.
To save time, we used representative prognostic items to develop the 4-item prognostic models and selected albumin (Alb), neutrophil (Neu), lactate dehydrogenase (LDH), and blood urine nitrogen (BUN) prognostic models on the basis of area under the receiver operating characteristic curve (AUC) values.
https://doi.org/10.1371/journal.pone.0183291.s006
(XLSX)
Acknowledgments
This work was supported by the FOCUS Establishing Supercomputing Center of Excellence. We thank Dr. Akiko Tamon, Dr. Masatoshi Hamanaka, and Dr. J.B. Brown for their help in the conception and design of the study and data collection. We also appreciate the Japan ProVal Study Group (Prognostic scores Validation Study Group), the participating study sites and site investigators, Satoshi Inoue, M.D. (Seirei Hospice, Seirei Mikatahara General Hospital), Masayuki Ikenaga, M.D. (Hospice Children’s Hospice Hospital, Yodogawa Christian Hospital), Yoshihisa Matsumoto, M.D., Ph.D. (Department of Palliative Medicine, National Cancer Center Hospital East), Mika Baba, M.D. (Saito Yukoukai Hospital Department of palliative care), Ryuichi Sekine, M.D. (Department of Pain and Palliative Care, Kameda Medical Center), Takashi Yamaguchi, M.D., Ph.D. (Department of Palliative Medicine, Kobe University Graduate School of Medicine), Takeshi Hirohashi, M.D. (Department of Palliative Care, Mitui Memorial Hospital), Tsukasa Tajima, M.D. (Department of Palliative Medicine, Tohoku University Hospital), Ryohei Tatara, M.D. (Osaka City General Hospital, Department of Palliative Medicine), Hiroaki Watanabe, M.D. (Komaki City Hospital), Hiroyuki Otani, M.D. (Department of Palliative Care Team, and Palliative and Supportive Care, National Kyushu Cancer Center), Chizuko Takigawa, M.D. (Department of Palliative Care, KKR Sapporo Medical Center), Yoshinobu Matsuda, M.D. (Department of Psychosomatic Medicine, National Hospital), Hiroka Nagaoka, M.D. (University of Tsukuba, Department of Medical Social Service Center for Palliative), Masanori Mori,M.D. (Seirei Hamamatsu General Hospital), Yo Tei, M.D. (SeireiHospice, Seirei Mikatahara General Hospital), Shuji Hiramoto, M.D. (Department of Oncology, Mitsubishi Kyoto Hospital), Akihiko Suga, M.D. (Department of Palliative Medicine Shizuoka Saiseikai General Hospital), Takayuki Hisanaga, M.D. (Tsukuba Medical Center Foundation), Tatsuhiko Ishihara, M.D. (Okayama Saiseikai General Hospital, Palliative Care Department), Tomoyuki Iwashita, M.D. (Matsue City Hospital), Keisuke Kaneishi, M.D., Ph.D. (Department of Palliative Care Unit, JCHO Tokyo Shinjuku Medical Center), Shohei Kawagoe, M.D. (Aozora clinic), Toshiyuki Kuriyama,M.D., Ph.D. (Department of Palliative Medicine, Wakayama Medical University Hospital Oncology Center,), Takashi Maeda, M.D. (Department of Palliative Care, Tokyo Metropolitan Cancer and Infectious Disease Center Komagome Hospital), Ichiro Mori, M.D. (Gratia Hospital Hospice), Nobuhisa Nakajima, M.D., Ph.D. (Department of Palliative Medicine, Graduate School of Medicine, Tohoku University), Tomohiro Nishi, M.D. (Kawasaki Comprehensive Care Center, Kawasaki Municipal Ida Hospital), Hiroki Sakurai, M.D. (Department of Palliative Care, St. Luke’s International Hospital, Tokyo), Satofumi Shimoyama,M.D., Ph.D. (Department of Palliative care, Aichi Cancer Center Hospital), Takuya Shinjo, M.D. (Shinjo Clinic, Kobe), Hiroto Shirayama, M.D. (Iryouhoujinn Takumikai Osaka Kita Homecare Clinic), Takeshi Yamada, M.D., Ph.D. (Department of Gastrointestinal and Hepato-Biliary-Pancreatic Surgery, Nippon Medical School), Shigeki Ono, M.D. (Division of Palliative Medicine, Shizuoka Cancer Center Hospital), Taketoshi Ozawa, M.D., Ph.D. (Megumi Zaitaku Clinic), Ryo Yamamoto, M.D. (Saku Central Hospital Advanced Care Center, Department of Palliative Medicine), Naoki Yamamoto,M.D., Ph.D. (Shinsei Hospital, Department of Primary Care Service), Hideki Shishido, M.D. (Shishido Internal Medicine Clinic), Mie Shimizu, M.D. (Saiseikai Matsusaka General Hospital), Masanori Kawahara, M.D., Ph.D. (Soshukai Okabe Clinic), Shigeru Aoki, M.D. (Sakanoue Family Clinic), Akira Demizu, M.D. (Demizu Clinic), Masahiro Goshima, M.D., Ph.D. (Homecare-Clinic Kobe), Keiji Goto, M.D. (Himawari Zaitaku Clinic), Yasuaki Gyoda,M.D., Ph.D. (Kanamecyo Home Care clinic), Jun Hamano, M.D. (Division of Clinical Medicine, Faculty of Medicine, University of Tsukuba), Kotaro Hashimoto, M.D. (Fukushima Home Palliative Care Clinic), Sen Otomo, M.D. (Shonan International Village Clinic), Masako Sekimoto, M.D. (Sekimoto Clinic), Takemi Shibata,M.D. (Kanwakeakurini kku Eniwa), Yuka Sugimoto, M.D. (Sugimoto Homecare Clinic), Mikako Matsunaga, M.D. (Senri Pain Clinic), Yukihiko Takeda, M.D. (Hidamari Clinic), Takeshi Sasara,M.D. (Yuuaikai Nanbu Hospital), and Jun Nagayama, M.D. (Peace Clinic Nakai).
References
- 1. Eidinger RN, Schapira DV. Cancer patients' insight into their treatment, prognosis, and unconventional therapies. Cancer. 1984;53(12):2736–40. pmid:6326993.
- 2. Weeks JC, Catalano PJ, Cronin A, Finkelman MD, Mack JW, Keating NL, et al. Patients' expectations about effects of chemotherapy for advanced cancer. New Engl J Med. 2012;367(17):1616–25. pmid:23094723; PubMed Central PMCID: PMCPMC3613151.
- 3. Enzinger AC, Zhang B, Schrag D, Prigerson HG. Outcomes of Prognostic Disclosure: Associations With Prognostic Understanding, Distress, and Relationship With Physician Among Patients With Advanced Cancer. J Clin Oncol. 2015;33(32):3809–16. pmid:26438121; PubMed Central PMCID: PMCPMC4737862.
- 4. Weeks JC, Cook EF, O'Day SJ, Peterson LM, Wenger N, Reding D, et al. Relationship between cancer patients' predictions of prognosis and their treatment preferences. JAMA. 1998;279(21):1709–14. pmid:9624023.
- 5. Morita T, Tsunoda J, Inoue S, Chihara S. The Palliative Prognostic Index: a scoring system for survival prediction of terminally ill cancer patients. Suppor Care Cancer. 1999;7(3):128–33. pmid:10335930.
- 6. Pirovano M, Maltoni M, Nanni O, Marinari M, Indelli M, Zaninetta G, et al. A new palliative prognostic score: a first step for the staging of terminally ill cancer patients. Italian Multicenter and Study Group on Palliative Care. J Pain Symptom Manage. 1999;17(4):231–9. pmid:10203875.
- 7. Scarpi E, Maltoni M, Miceli R, Mariani L, Caraceni A, Amadori D, et al. Survival prediction for terminally ill cancer patients: revision of the palliative prognostic score with incorporation of delirium. Oncologist. 2011;16(12):1793–9. pmid:22042788; PubMed Central PMCID: PMCPMC3248779.
- 8. Gwilliam B, Keeley V, Todd C, Gittins M, Roberts C, Kelly L, et al. Development of prognosis in palliative care study (PiPS) predictor models to improve prognostication in advanced cancer: prospective cohort study. BMJ. 2011;343:d4920. pmid:21868477; PubMed Central PMCID: PMCPMC3162041.
- 9. Matsumoto S, Nishimura T, Kanai M, Ishiguro H, Yasuda K, Yoshikawa A, et al. Development of a novel information technology (IT) system using the electronic medical record (EMR) in daily clinical practice. J Clin Oncol. 2007;25:18s (suppl;abstr 17066).
- 10. Baba M, Maeda I, Morita T, Inoue S, Ikenaga M, Matsumoto Y, et al. Survival prediction for advanced cancer patients in the real world: A comparison of the Palliative Prognostic Score, Delirium-Palliative Prognostic Score, Palliative Prognostic Index and modified Prognosis in Palliative Care Study predictor model. Eur J Cancer. 2015;51(12):1618–29. pmid:26074396
- 11. Redelmeier DA, Tibshirani RJ. Interpretation and bias in case-crossover studies. J Clin Epidemiol. 1997;50(11):1281–7. pmid:9393384.
- 12. Fan R-E, Chang K-W, Hsieh C-J, Wang X-R, Lin C-J. LIBLINEAR: A Library for Large Linear Classification. J Mach Learn Res. 2008;9:1871–1874.
- 13. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. J Artif Intell Res 2002;16:321–57.
- 14. Barbot AC, Mussault P, Ingrand P, Tourani JM. Assessing 2-month clinical prognosis in hospitalized patients with advanced solid tumors. J Clin Oncol. 2008;26(15):2538–43. pmid:18487571.
- 15. Vigano A, Bruera E, Jhangri GS, Newman SC, Fields AL, Suarez-Almazor ME. Clinical survival predictors in patients with advanced cancer. Arch Intern Med. 2000;160(6):861–8. pmid:10737287.
- 16. Penel N, Negrier S, Ray-Coquard I, Ferte C, Devos P, Hollebecque A, et al. Development and validation of a bedside score to predict early death in cancer of unknown primary patients. PloS One. 2009;4(8):e6483. pmid:19649260; PubMed Central PMCID: PMCPMC2715134.
- 17. Suh SY, Ahn HY. Lactate dehydrogenase as a prognostic factor for survival time of terminally ill cancer patients: a preliminary study. Eur J Cancer. 2007;43(6):1051–9. pmid:17349786.
- 18. Mantovani A, Allavena P, Sica A, Balkwill F. Cancer-related inflammation. Nature. 2008;454(7203):436–44. pmid:18650914.
- 19. Templeton AJ, McNamara MG, Seruga B, Vera-Badillo FE, Aneja P, Ocana A, et al. Prognostic role of neutrophil-to-lymphocyte ratio in solid tumors: a systematic review and meta-analysis. J Nal Cancer Inst. 2014;106(6):dju124. pmid:24875653.