https://orcid.org/0000-0002-6421-9867
Figures
Abstract
Context
Multimorbidity is highly prevalent among older adults and associated with a high mortality. Prediction of mortality in multimorbid people would be clinically useful but there is no mortality risk index designed for this population. Our objective was therefore to develop and internally validate a 1-year mortality prognostic index for older multimorbid adults.
Methods
We analysed data of the OPERAM cohort study in Bern, Switzerland, including 822 adults aged 70 years or more with multimorbidity (3 or more chronic medical conditions) and polypharmacy (use of 5 drugs or more for >30 days). Time to all-cause mortality was assessed up to 1 year of follow-up. We performed a parametric Weibull regression model with backward stepwise selection to identify mortality risk predictors. The model was internally validated and optimism corrected using bootstrapping techniques. We derived a point-based risk score from the regression coefficients. Calibration and discrimination were assessed by the calibration slope and C statistic.
Results
805 participants were included in the analysis. During 1-year of follow-up, 158 participants (20%) had died. Age, Charlson-Comorbidity-Index, number of drugs, body mass index, number of hospitalizations, Barthel-Index (functional impairment), and nursing home residency were predictors of 1-year mortality in a multivariable model. Using these variables, the 1-year probability of dying could be predicted with an optimism-corrected C statistic of 0.70. The optimism-corrected calibration slope was 0.93. Based on the derived point-based risk score to predict mortality risk, 7% of the patients classified at low-risk of mortality, 19% at moderate-risk, and 37% at high-risk died after one year of follow-up. A simpler mortality score, without the Charlson-Comorbidity-Index and Barthel-Index, showed reduced discriminative power (optimism-corrected C statistic: 0.59) compared to the full score.
Conclusion
We developed and internally validated a mortality risk index including for the first-time specific predictors for multimorbid adults. This new 1-year mortality prediction point-based score allowed to classify multimorbid older patients into three categories of increasing risk of mortality. Further validation of the score among various populations of multimorbid patients is needed before its implementation into practice.
Citation: Gastens V, Chiolero A, Anker D, Schneider C, Feller M, Bauer DC, et al. (2022) Development and validation of a new prognostic index for mortality risk in multimorbid adults. PLoS ONE 17(8): e0271923. https://doi.org/10.1371/journal.pone.0271923
Editor: Robert Daniel Blank, Medical College of Wisconsin, UNITED STATES
Received: February 3, 2022; Accepted: July 10, 2022; Published: August 5, 2022
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: This study involves human research participant data containing sensitive patient information. In the EU Horizon 2020 grant agreement for the OPERAM study, it had been specified that the data will be made available upon request if the use has been approved by an ethical committee. Therefore, restrictions to make the underlying data directly publicly available are both due to legal and ethical reasons, as health data are sensitive data. Data for this study will be made available for scientific purposes upon request for researchers whose proposed use of the data has been approved by the OPERAM publication committee. After approval and signing of a data transfer agreement ensuring adherence to privacy and data handling, data and documentation will be made available through a secure file exchange platform. Partially deidentified participant data, a data dictionary and annotated case report forms will be made available. For data access, external researchers can fill in the contact form on https://www.operam-cohort.biham.ch/ or operam-2020.eu.
Funding: This work was supported by Swiss National Science Foundation grants (320030_188549/01 to AC; 325130_204361 to CDG; www.snf.ch). This work is part of the project “OPERAM: OPtimising thERapy to prevent Avoidable hospital admissions in the Multimorbid elderly” supported by the European Union's Horizon 2020 research and innovation programme under the grant agreement No 6342388 (ec.europa.eu/programmes/horizon2020), and by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 15.0137 (www.sbfi.admin.ch/sbfi/en/home.html). The opinions expressed and arguments employed herein are those of the authors and do not necessarily reflect the official views of the EC and the Swiss government. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Multimorbidity is highly prevalent especially in older adults [1] and, due to the population ageing, the number of people with multimorbidity is growing rapidly [2, 3]. Multimorbidity is associated with a high mortality rate [2] and previous research suggests that all-cause mortality risk for individuals with multimorbidity is nearly 2–3 times higher compared with individuals without multimorbidity [4–6].
Many guidelines recommend tailoring preventive care of multimorbid people according to life expectancy, and hence on the mortality risk [7]. Indeed, patients with high short-term mortality might not have the time to benefit from a preventive care intervention. Because multimorbid patients have a relatively short life expectancy, they are at higher risk of not having the time to benefit. It is therefore necessary to have a valid index for mortality prediction in multimorbid patients.
In a systematic review, Yourman et al. have identified 16 mortality prognostic indices for older adults [8]. While several of these prognostic indices were fairly accurate to predict mortality, the authors concluded that none could be recommended for a widespread use. One major limitation is that none of these prognostic indices has been tested prospectively in various samples. Key is that their transportability in other populations is unknown and clinicians cannot use these indices with confidence across different groups of patients [8]. Further, none of these indices has been developed specifically in multimorbid older adults. Current mortality indices do not consider predictors specific to multimorbid older adults such as the severity of comorbidities and functional impairment. Therefore, there is no tool recommended to accurately predict mortality in older multimorbid adults.
Our objective was therefore to develop and internally validate a 1-year mortality prognostic index for older multimorbid adults.
Methods and analysis
Source of data and study design
We used data from 822 participants of an ongoing cohort study in Bern, Switzerland. Participants were originally enrolled in the clinical trial OPtimising thERapy to prevent Avoidable hospital admissions in Multimorbid older people (OPERAM [9, 10]). Participants were assigned to receive either standard care or a medication review by a Systematic Tool to Reduce Inappropriate Prescribing (STRIP) with observation of the primary outcome of drug-related hospital admission (DRA) over 1 year.
For the current study, we used data collected at baseline (December 2016-October 2018) and up to 1 year after baseline (until October 2019). The local Ethics Committee in Bern, Switzerland, approved the study protocol with the project number 2018–00784. Study nurses collected baseline data by a personal interview with the participant and from medical files. The follow-up was conducted via phone calls. Phone interviews were held with participants or relatives, otherwise with a proxy or with the general practitioner, when the participants were not reachable or not able to answer.
We developed and validated the mortality prognostic index following the Prognosis Research Strategy (PROGRESS) framework [11], and reported it following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [12, 13]. We further followed the recommendations of Moons et al. [14, 15] for risk prediction models. This study is part of a research project whose protocol has been published previously [10].
Participants
Participants (N = 822) were enrolled at the time of a hospitalization in the Inselspital, University Hospital, Bern, Switzerland. Inclusion criteria were age of 70 years or older, multimorbidity (3 or more chronic medical conditions), and polypharmacy (use of 5 drugs or more for >30 days).
Written informed consent by patients themselves or, in the case of cognitive impairment by a legal representative, had already been obtained before enrolment. Patients planned for direct admission to palliative care (<24 hours after admission), or patients undergoing a structured drug review other than the trial intervention, or who had passed a structured drug review within the last 2 months were excluded. Patients for whom it was not possible to obtain an informed consent were excluded.
Outcome
The outcome was time to all-cause mortality over one year of follow-up. Information on death and relative date was collected by study nurses through follow-up calls or primary care physician contact.
Candidate predictors identification
Candidate 1-year mortality predictors were derived from previous research efforts in this field [8, 16], ease and reliability of measurement in clinical setting, and background knowledge on potential associations with mortality. We also considered factors included in the OPERAM dataset that may not be identified from the literature but are specific to multimorbid patients. All candidate predictors were based on baseline characteristics. We included demographic variables (age, sex), clinical characteristics (Charlson-Comorbidity-Index, number of drugs, body mass index, weight loss during the last year), smoking, functional status variables (Barthel-Index, falls, nursing home residence), and hospitalization. The variables about falls and hospitalization reflected the number of events during the last 12 months before index hospitalization. The Charlson-Comorbidity-Index was originally developed to predict mortality, with higher scores indicating higher mortality risk. It was calculated using ICD-10 codes by the adaptation of Quan et al. and implemented in the comorbidity library for the R environment [17, 18]. The Barthel-Index measures performance in activities of daily living (ADL) on an ordinal scale from 0 to 100 with higher scores indicating more independence. Continuous variables were categorised for ease of use at the point of care [19]. We based the categorisation decisions on the analysis of the frequency distribution of the variables and on clinical rationale.
Statistical analysis
We have calculated the required sample size for conducting our multivariable prediction model utilizing the criteria proposed by Riley et al. [20] and implemented in the pmsampsize library for the R environment [21]. The minimum sample size required with 12 candidate predictor parameters, an expected outcome event rate of 0.15 per year, and an anticipated Cox-Snell R2 of 0.126 (C statistic of [16] 0.82, [22]) is 799 with 10 events per predictor parameter. Our sample size of 822 is therefore adequate for this project.
The relationship between candidate predictors and outcome was analysed using a parametric Weibull regression model. We first performed a univariable analysis between each potential candidate predictors and the outcome. We then used multiple imputation (number of multiple imputations, m = 10) for missing values under a missing at random assumption in order to reduce bias and avoid excluding participants from the analysis [14]. We performed stepwise backwards variable selection based on the Akaike Information Criterion (AIC) in each imputed dataset. For automatic selection procedures, backward elimination is recommended in TRIPOD [13, 23]. We started with the full model, sequentially dropping variables to maximally reduce AIC. The final model was formed of predictors for which AIC cannot be minimized further and which appeared in all models of the imputed datasets. The final model was fitted in each imputed dataset and results pooled according to Rubin’s rules. We used the mice library for multiple imputation and pooling, and the MASS library with the stepAIC function for stepwise model selection via AIC in the R environment [24, 25].
We investigated the predictive accuracy of the final model by testing calibration and discrimination. The apparent performance and discrimination of the model was assessed with C statistic [14]. We evaluated potential overfitting and optimism by internal validation with bootstrapping techniques [14, 15]. We performed 500 bootstrap cycles. In each bootstrap sample, we derived a mortality prediction model and the relative risk score, as done in the original sample. We calculated optimism as difference in performance measure (C statistic) between the original sample and the respective bootstrap sample. This was repeated for all bootstrap samples to estimate the average optimism. We evaluated the calibration slope and intercept (calibration-in-the-large). We assessed graphical discrimination with a Kaplan-Meier plot of the risk groups.
Point-based risk score.
From the final model, we derived a point-based risk score by assigning points to each risk factor. Each β coefficient was divided by the lowest β coefficient and rounded to the nearest integer. We calculated the total risk score for each participant by summing the points for each risk factor [16].
Sensitivity analyses.
We performed a flexible parametric model to assess the robustness of the results obtained with the Weibull regression model. We performed a univariable analysis of the specific items in the Charlson-Comorbidity- Index and Barthel-Index. Further, to develop a score easier to use in practice, we performed the described methods above to develop and test a simplified model without the Barthel-Index and Charlson-Comorbidity-Index, because the assessment of both these indices can be difficult at the point-of-care.
Results
Among 822 participants, 805 were included in the analytical sample. We excluded 17 participants because they left the study, and most data were missing. Baseline characteristics of the participants are reported in Table 1. The mean (min to max) age of participants was 79.7 (70 to 99) years and 42% were women. During a mean (min to max) follow-up of 12.2 months (11.0 to 17.1), 158 participants (20%) had died. The proportion of missing data ranged from 0% to 10% in the predictor variables (S1 Table). The univariable analysis showed that age, Charlson-Comorbidity-Index, number of drugs, BMI, number of hospitalizations, and Barthel-Index were associated with 1-year mortality (S1 Table).
The final risk prediction model included the predictors age, BMI, Charlson-Comorbidity-Index, hospitalizations, drugs, Barthel-Index, and nursing home residence (Table 2). We generated 10 imputed datasets by multiple imputation for missing data. The final predictor variables were retained in all imputed datasets after stepwise selection. Table 3 showed apparent and internal validation performance statistics of our risk prediction model. After adjustment for optimism with bootstrapping, our final model was able to discriminate participants with and without death within one year with a C statistic of 0.70. The optimism-corrected calibration slope was 0.93.
Sensitivity analyses
We found similar results by applying the flexible parametric model compared to the main analysis (S3 Table). Univariable analysis of the Charlson-Comorbidity-Index and the Barthel-Index showed the potential of including specific items instead of the entire indices in further projects (S4 Table). A simplified risk score without the Barthel-Index and Charlson-Comorbidity-Index was developed. The risk score points assigned to each of the final predictors in this simpler risk score are listed in Table 4. The simplified model had a weaker discrimination performance with a C statistic of 0.59 (Table 5).
Point-based risk score
The risk score points assigned to each of the final predictors are listed in Table 2. A risk score was calculated for each participant by adding the points for each predictor present. For example, a 81-year-old (1 points) woman with a CC-Index of 3 (4 points), taking 5 drugs (0 points), with a BMI of 28 (2 points), 2 hospitalizations in the last 12 months (1 point), a Barthel-Index of 80 (3 points), and living in a nursing home residence (3 points) would have a total risk score of 14 points.
The mean 1-year mortality risk score in our sample was 7.7 (standard deviation 3.9); it ranged from 2 to 21. Based on the derived point-based risk score to predict mortality risk, 7% (95% CI: 6.3–7.7) of patients in the low-risk category (0 to 5 points) died, 19% (11.4–24.6) with moderate-risk (6 to 10 points), and 37% (25.2–44.8) with high-risk of 1-year mortality (>10 points) (S5A Table). Fig 1A showed the Kaplan-Meier plot of the three risk groups and good graphical discrimination.
A. Kaplan-Meier curves in three risk groups to visually assess separation (low risk: 0–5 points (green), moderate risk: 6–10 points (orange), high risk: 11–21 (red)). B. Kaplan-Meier curves of the simplified score in two risk groups to visually assess separation (low risk: 0–7 points (orange), high risk: 8–14 (red)).
Based on the simpler risk score to predict mortality risk, 13% (95% CI: 10.9–15.1) of patients in the low-risk category (0 to 7 points) died, and 25% (20.1–29.9) with high-risk of 1-year mortality (>7 points) (S5B Table). Fig 1B showed the Kaplan-Meier plot of two risk groups of this simplified risk score.
Discussion
We have developed and internally validated a new risk prediction model to estimate the 1-year mortality risk in older multimorbid adults. The final model included the seven predictors age, BMI, Charlson-Comorbidity-Index, number of hospitalizations, number of drugs, Barthel-Index, and nursing home residence. Using this score, we could classify patients into categories of increasing risk of 1-year mortality with a substantial risk difference. A simpler score was able to categorize 1-year mortality risk with a weaker discriminative power.
We used high-quality data from a large prospective cohort study of multimorbid older adults. In contrast to existing mortality risk indices, our index focuses specifically on multimorbid older patients and accounted for the severity of comorbidity (Charlson-Comorbidity-Index) and functional impairment (Barthel-Index) [16]. The interpretation of performance measures such as the C statistic depends on the clinical area. Other prognostic indices for 1-year mortality in older adults show similar discrimination performance, e.g. C statistic of 0.68 or 0.79 [26, 27]. We have applied a particularly robust methodological framework by following the research guidelines in this field, namely the Prognosis Research Strategy (PROGRESS) framework, and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. Notably, we ensured adequate sample size, used multiple imputation for missing data, and applied bootstrapping techniques for internal validation [13, 23].
One major limitation is the lack of external validation but we will explore opportunities to test the score in a different dataset of older multimorbid adults. However, we did our best to assess internal validation by using the bootstrap method for internal validation that makes use of the entire sample. This optimism-corrected validation analysis indicates that both calibration slope and intercept deviate from the null value of one (0.93; 95% CI: 0.92, 0.94) and zero (0.61; 95% CI: 0.56, 0.66) (Table 3), respectively. The calibration slope evaluates the spread of the estimated risks and has a target value of 1. A slope < 1 suggests that estimated risks are too extreme, a slope > 1 suggests that risk estimates are too moderate [28]. The calibration intercept, which is an assessment of calibration-in-the-large, has a target value of 0; negative values suggest overestimation, whereas positive values suggest underestimation [28]. Therefore, our results indicate suboptimal prediction accuracy as the risk estimates were considered too extreme (for the calibration slope) and the model underestimating the predicted risk (for the calibration intercept). As the study participants were included at the time of hospitalization, they may not be representative of all patients with multimorbidity, and this could reduce the transportability of our findings to other populations. Another limitation is that indices such as the Barthel-Index and the Charlson-Comorbidity-Index used as predictor may reduce the ease of use of the risk score at the point-of-care. We have therefore developed a simpler score without such variables. This simpler score showed reduced discriminative power compared to the full score. This could highlight the importance of taking the severity of comorbidities and functional impairment into account to improve risk prediction in older multimorbid people. Additional studies will be needed to test the usefulness of both scores in practice. Finally, prognostic information longer than 1-year mortality risk is needed. We will expand this model to 3-year mortality risk once the 3-year follow-up data collection is completed [10]. For this next research project, we might consider nomogram analysis as a graphically intuitive representation [29].
Our results will be useful for both clinical and research activities by helping health care providers to tailor preventive care according to the estimated mortality risk. Eventually, our study can help preventing under- and overuse of preventive care in the growing older population.
Supporting information
S1 Table. Missing data in the predictors in the 805 participants.
https://doi.org/10.1371/journal.pone.0271923.s001
(DOCX)
S2 Table. Univariable analysis of candidate predictors with a Weibull model from information obtained at baseline.
https://doi.org/10.1371/journal.pone.0271923.s002
(DOCX)
S3 Table. Multivariable analysis of candidate predictors with a flexible parametric survival model.
https://doi.org/10.1371/journal.pone.0271923.s003
(DOCX)
S4 Table. Univariable analysis of the specific items in the Charlson Comorbidity Index and Barthel-Index.
https://doi.org/10.1371/journal.pone.0271923.s004
(DOCX)
S5 Table. A. Stratification of 1-year mortality risk by points of the final score. B. Stratification of 1-year mortality risk by points of the simplified score.
https://doi.org/10.1371/journal.pone.0271923.s005
(DOCX)
S1 Fig.
A. Calibration plot of the final prediction model for 1-year mortality. B. Calibration plot of the simplified model for 1-year mortality.
https://doi.org/10.1371/journal.pone.0271923.s006
(TIF)
References
- 1. Violan C., Foguet-Boreu Q., Flores-Mateo G., Salisbury C., Blom J., Freitag M., et al. Valderas, "Prevalence, determinants and patterns of multimorbidity in primary care: a systematic review of observational studies," PloS one, vol. 9, p. e102149, 2014. pmid:25048354
- 2. Menotti A., Mulder I., Nissinen A., Giampaoli S., Feskens E. J. M. and Kromhout D., "Prevalence of morbidity and multimorbidity in elderly male populations and their impact on 10-year all-cause mortality: The FINE study (Finland, Italy, Netherlands, Elderly)," Journal of clinical epidemiology, vol. 54, p. 680–686, 2001. pmid:11438408
- 3. Cassell A., Edwards D., Harshfield A., Rhodes K., Brimicombe J., Payne R. et al., "The epidemiology of multimorbidity in primary care: a retrospective cohort study," Br J Gen Pract, vol. 68, p. e245–e251, 2018. pmid:29530918
- 4. Jani B. D., Hanlon P., Nicholl B. I., McQueenie R., Gallacher K. I., Lee D. et al., "Relationship between multimorbidity, demographic factors and mortality: findings from the UK Biobank cohort," BMC medicine, vol. 17, p. 1–13, 2019.
- 5. Ryan B. L., Allen B., Zwarenstein M., Stewart M., Glazier R. H., Fortin M., et al., "Multimorbidity and mortality in Ontario, Canada: A population-based retrospective cohort study," Journal of comorbidity, vol. 10, p. 2235042X20950598, 2020. pmid:32923405
- 6. Nunes B. P., Flores T. R., Mielke G. I., Thumé E. and Facchini L. A., "Multimorbidity and mortality in older adults: a systematic review and meta-analysis," Archives of gerontology and geriatrics, vol. 67, p. 130–138, 2016. pmid:27500661
- 7. Grossman D. C., Curry S. J., Owens D. K., Bibbins-Domingo K., Caughey A. B., Davidson K. W., et al., "Screening for prostate cancer: US Preventive Services Task Force recommendation statement," Jama, vol. 319, p. 1901–1913, 2018. pmid:29801017
- 8. Yourman L. C., Lee S. J., Schonberg M. A., Widera E. W. and Smith A. K., "Prognostic Indices for Older Adults: A Systematic Review," JAMA, vol. 307, pp. 182–192, 1 2012. pmid:22235089
- 9. Adam L., Moutzouri E., Baumgartner C., Loewe A. L., Feller M., K. M\textquoterightRabet-Bensalah, et al., "Rationale and design of OPtimising thERapy to prevent Avoidable hospital admissions in Multimorbid older people (OPERAM): a cluster randomised controlled trial," BMJ Open, vol. 9, 2019. pmid:31164366
- 10. Gastens V., Del Giovane C., Anker D., Feller M., Syrogiannouli L., Schwab N., et al., "Development and validation of a life expectancy estimator for multimorbid older adults: a cohort study protocol," BMJ open, vol. 11, p. e048168, 2021. pmid:34433596
- 11. Steyerberg E. W., Moons K. G. M., van der Windt D. A., Hayden J. A., Perel P., Schroter S., et al., "Prognosis Research Strategy (PROGRESS) 3: prognostic model research," PLoS medicine, vol. 10, 2013. pmid:23393430
- 12. Collins G. S., Reitsma J. B., Altman D. G. and Moons K. G. M., "Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement," BMJ, vol. 350, 2015.
- 13. Moons K. G. M., Altman D. G., Reitsma J. B., Ioannidis J. P. A., Macaskill P., Steyerberg E. W., et al., "Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration," Annals of internal medicine, vol. 162, p. W1–W73, 2015. pmid:25560730
- 14. Moons K. G. M., Kengne A. P., Woodward M., Royston P., Vergouwe Y., Altman D. G. et al., "Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker," Heart, vol. 98, p. 683–690, 2012. pmid:22397945
- 15. Moons K. G. M., Kengne A. P., Grobbee D. E., Royston P., Vergouwe Y., Altman D. G. et al., "Risk prediction models: II. External validation, model updating, and impact assessment," Heart, vol. 98, p. 691–698, 2012. pmid:22397946
- 16. Lee S. J., Lindquist K., Segal M. R. and Covinsky K. E., "Development and Validation of a Prognostic Index for 4-Year Mortality in Older Adults," JAMA, vol. 295, pp. 801–808, 2 2006. pmid:16478903
- 17. Gasparini A., Salmasian H. and Williman J., "comorbidity: Computing Comorbidity Scores," 2021.
- 18. Quan H., Sundararajan V., Halfon P., Fong A., Burnand B., Luthi J.-C., et al., "Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data," Medical care, p. 1130–1139, 2005. pmid:16224307
- 19. Turner E. L., Dobson J. E. and Pocock S. J., "Categorisation of continuous risk factors in epidemiological publications: a survey of current practice," Epidemiologic Perspectives & Innovations, vol. 7, p. 1–10, 2010. pmid:20950423
- 20. Riley R. D., Ensor J., Snell K. I. E., Harrell F. E., Martin G. P., Reitsma J. B., et al., "Calculating the sample size required for developing a clinical prediction model," BMJ, vol. 368, 2020. pmid:32188600
- 21. Ensor J., Martin E. C. and Riley R. D., "pmsampsize: Calculates the Minimum Sample Size Required for Developing a Multivariable Prediction Model," 2019.
- 22. Riley R. D., Snell K. I. E., Ensor J., Burke D. L., Harrell F. E. Jr, Moons K. G. M. et al., "Minimum sample size for developing a multivariable prediction model: PART II—binary and time-to-event outcomes," Statistics in Medicine, vol. 38, pp. 1276–1296, 2019. pmid:30357870
- 23.
Royston P. and Sauerbrei W., Multivariable model-building: a pragmatic approach to regression anaylsis based on fractional polynomials for modelling continuous variables, vol. 777, John Wiley & Sons, 2008.
- 24. van Buuren S., Groothuis-Oudshoorn K., Vink G., Schouten R., Robitzsch A., Rockenschaub P., et al., "mice: Multivariate Imputation by Chained Equations," 2021.
- 25. Ripley B., Venables B., Bates D. M., Hornik K., Gebhardt A. and Firth D., "MASS: Support Functions and Datasets for Venables and Ripley’s MASS," 2021.
- 26. Levine S. K., Sachs G. A., Jin L. and Meltzer D., "A prognostic model for 1-year mortality in older adults after hospital discharge," The American journal of medicine, vol. 120, p. 455–460, 2007. pmid:17466658
- 27. Walter L. C., Brand R. J., Counsell S. R., Palmer R. M., Landefeld C. S., Fortinsky R. H. et al., "Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization," Jama, vol. 285, p. 2987–2994, 2001. pmid:11410097
- 28. Van Calster B., McLernon D. J., Van Smeden M., Wynants L. and Steyerberg E. W., "Calibration: the Achilles heel of predictive analytics," BMC medicine, vol. 17, p. 1–7, 2019.
- 29. Balachandran V. P., Gonen M., Smith J. J. and DeMatteo R. P., "Nomograms in oncology: more than meets the eye," The lancet oncology, vol. 16, p. e173–e180, 2015. pmid:25846097