Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A prognostic model for development of significant liver fibrosis in HIV-hepatitis C co-infection

  • Nasheed Moqueet,

    Affiliation Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, QC, Canada

  • Cynthia Kanagaratham,

    Affiliation Department of Medicine and Department of Human Genetics, McGill University, Montreal, QC, Canada

  • M. John Gill,

    Affiliation Southern Alberta HIV Clinic, Calgary, Alberta, Canada

  • Mark Hull,

    Affiliation BC Centre for Excellence in HIV/AIDS, St. Paul's Hospital, Vancouver, BC, Canada

  • Sharon Walmsley,

    Affiliation Toronto General Research Institute, University Health Network, University of Toronto, Toronto, ON, Canada

  • Danuta Radzioch,

    Affiliation Department of Medicine and Department of Human Genetics, McGill University, Montreal, QC, Canada

  • Sahar Saeed,

    Affiliation Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, QC, Canada

  • Robert W. Platt,

    Affiliation Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, QC, Canada

  • Marina B. Klein ,

    marina.klein@mcgill.ca

    Affiliations Department of Epidemiology, Biostatistics, and Occupational Health, McGill University, Montreal, QC, Canada, Division of Infectious Diseases and Chronic Viral Illness Service, McGill University Health Centre, Montreal, QC, Canada

  • for the Canadian Co-infection Cohort Study (CTN 222)

    Membership of the Canadian Co-infection Cohort Study (CTN 222) is provided in the Acknowledgments

A prognostic model for development of significant liver fibrosis in HIV-hepatitis C co-infection

  • Nasheed Moqueet, 
  • Cynthia Kanagaratham, 
  • M. John Gill, 
  • Mark Hull, 
  • Sharon Walmsley, 
  • Danuta Radzioch, 
  • Sahar Saeed, 
  • Robert W. Platt, 
  • Marina B. Klein, 
  • for the Canadian Co-infection Cohort Study (CTN 222)
PLOS
x

Abstract

Background

Liver fibrosis progresses rapidly in HIV-Hepatitis C virus (HCV) co-infected individuals partially due to heightened inflammation. Immune markers targeting stages of fibrogenesis could aid in prognosis of fibrosis.

Methods

A case-cohort study was nested in the prospective Canadian Co-infection Cohort (n = 1119). HCV RNA positive individuals without fibrosis, end-stage liver disease or chronic Hepatitis B at baseline (n = 679) were eligible. A random subcohort (n = 236) was selected from those eligible. Pro-fibrogenic markers and Interferon Lambda (IFNL) rs8099917 genotype were measured from first available sample in all fibrosis cases (APRI ≥ 1.5 during follow-up) and the subcohort. We used Cox proportional hazards and compared Model 1 (selected clinical predictors only) to Model 2 (Model 1 plus selected markers) for predicting 3-year risk of liver fibrosis using weighted Harrell’s C and Net Reclassification Improvement indices.

Results

113 individuals developed significant liver fibrosis over 1300 person-years (8.63 per 100 person-years 95% CI: 7.08, 10.60). Model 1 (age, sex, current alcohol use, HIV RNA, baseline APRI, HCV genotype) was nested in model 2, which also included IFNL genotype and IL-8, sICAM-1, RANTES, hsCRP, and sCD14. The C indexes (95% CI) for model 1 vs. model 2 were 0.720 (0.649, 0.791) and 0.756 (0.688, 0.825), respectively. Model 2 classified risk more appropriately (overall net reclassification improvement, p<0.05).

Conclusions

Including IFNL genotype and inflammatory markers IL-8, sICAM-1, RANTES, hs-CRP, and sCD14 enabled better prediction of the 3-year risk of significant liver fibrosis over clinical predictors alone. Whether this modest improvement in prediction justifies their additional cost requires further cost-benefit analyses.

Introduction

Liver disease has become one of leading non-AIDS causes of death among HIV-infected individuals in the developed world, mainly due to co-infection with hepatitis C (HCV). While HIV treatment improves outcomes in HCV co-infection [1, 2], compared to HCV mono-infected individuals, liver fibrosis progression remains accelerated, leading to cirrhosis, hepatocellular carcinoma or end-stage liver disease (ESLD) [3, 4]. Reasons for this acceleration include biological factors and possibly HIV therapy-related toxicity. HIV itself suppresses the immune response to HCV [3, 5], triggering a cycle where inflammatory and fibrogenic cells continually stimulate each other, distorting the hepatic architecture, eventually leading to fibrosis. Liver fibrosis progression is thus caused by heightened inflammation rather than direct HCV replication.

In Canada, the majority of the HIV-HCV co-infected population is made up of current or former injection drug users for whom treatment access and adherence may be challenging. While international clinical guidelines recognize that co-infected individuals should be prioritized for HCV treatment [6, 7], the high cost of treatment (between $50,000 and $120,000 for a course of the new direct-acting antivirals or DAAs) in Canada [8, 9] has meant that reimbursement by public and private payers has been restricted to those with advanced fibrosis (METAVIR stage F2 and higher), based on liver biopsy and increasingly by transient elastography or use of noninvasive indices such as aspartate aminotransferase (AST) to platelet ratio index (APRI) and Fibrosis-4 (FIB-4) [10]. Given this restriction, earlier indicators of fibrosis could be helpful in identifying persons at higher risk for liver disease progression in order to target effective intervention and treatment strategies in a cost-effective manner.

The principal risk factors associated with fibrosis progression include alcohol intake (>50 g/day), infection with HCV genotype 3, male sex, excess weight, liver steatosis, presence of metabolic syndrome and/or type II diabetes, host genetic factors such as single nucleotide polymorphisms (SNPs) in the Interferon Lambda (IFNL) gene, hepatitis B co-infection, immunodeficiency related to HIV or the use of immunosuppressant drugs, and HIV therapy interruption [1115].

We hypothesized that the addition of genetic markers associated with fibrosis progression and a panel of immune markers (summarized in S1 Table) representative of the underlying inflammatory mechanisms would improve prediction of fibrosis risk beyond traditional clinical risk factors alone and would thus be of value in optimal use of new and expensive HCV therapies.

Methods

Source population

The Canadian Co-infection Cohort is an open prospective cohort of HIV-HCV co-infected patients recruited from 18 centers across Canada[16]. It has been approved by the following: the community advisory committee of the CIHR-Canadian HIV Trials Network, the Biomedical B Research Ethics Board of the McGill University Health Centre (BMB-06-006t and BMC-07-004), the UBC-Providence Health Care Research Ethics Board (H08-00474), the Institutional Review Board Services, Regina Qu’Appelle Health Region Research Ethics Board (REB-14-70), the Conjoint Health Research Ethics Board of the University of Calgary (20931), the Nova Scotia Health Research Ethics Board (CDHA-RS2007-118), the Windsor Regional Hospital Research Ethics Board (07-122-17), the Veritas Independent Review Board, the Hamilton Integrated Research Ethics Board (06–397), the Comité d'éthique de la recherche du CHUM (2003–1582, SL 03.008-BSP), the Comité d'éthique de la recherche du CHU de Québec-Université Laval (C11-12-153), the Sunnybrook Health Sciences Centre Research Ethics Board (252–2008), the Research Ethics Board of Health Sciences North (605), the University Health Network Research Ethics Board (06-0629-BE), the Ottawa Health Science Network Research Ethics Board (2007229-01H) and the Biomedical Research Ethics Board (12–178). Patient records/information was anonymized and de-identified prior to analysis. All patients gave written informed consent before undergoing an initial evaluation and were followed at study visits every six months.

The CCC represents approximately 23% of the co-infected population under care. At every study visit, socio-demographic and behavioural information was collected using validated questionnaires, along with plasma, serum and peripheral blood mononuclear cells (PBMC). For our study, we included data from 1119 patients collected up until July 2012.

To be included in the CCC, patients must be over 16 years or older, give informed consent, be HIV infected (confirmed via ELISA with Western blot), and have HCV infection or evidence of HCV exposure (HCV-antibody positive by ELISA with recombinant immunoblot assay II (RIBA II) or enzyme immunoassay (EIA) or if serologically false negative, HCV–RNA-positive).

Study population

For our study, HCV RNA-negative participants or those who had significant fibrosis (APRI≥1.5), end-stage liver disease or chronic Hepatitis B at study entry were excluded, as were individuals on HCV treatment. HCV RNA was measured using qualitative tests (COBAS AMPLICOR HCV Test, version 2.0, Roche Diagnostics, Hoffmann-La Roche Ltd, Laval, Canada, lower limit of detection <50 IU ml-1) and was available at most visits. Presence of Hepatitis B surface antigen was used to determine Hepatitis B chronicity.

As immune and genetic markers were not measured during regular CCC visits, we used a case cohort study design as an economical way to gather this information. From an eligible study sample of n = 679 (Fig 1), a random subsample or “subcohort” was selected from the population at entry, to provide comparison observations for each event of significant liver fibrosis, occurring during study follow-up. Because the subcohort was a representation of the full cohort, it also contained a few incident cases.

thumbnail
Fig 1. Source and study population for developing a prognostic model for significant liver fibrosis.

Abbreviations: ESLD, end-stage liver disease; HCV, Hepatitis C virus.

https://doi.org/10.1371/journal.pone.0176282.g001

Outcome measure

Progression to significant liver fibrosis (METAVIR stage F2-F4) was defined by an APRI≥1.5. APRI is calculated as follows: [(AST/upper limit of normal)/platelet count (109 /L)] x 100. It consists of routinely available and non-invasive measures that were available for almost every visit. An APRI cutoff of 1.5 or higher has been validated against liver biopsies in our study population for detection of significant liver fibrosis (METAVIR stages F2-F4) with a sensitivity of 52%, specificity and positive predictive value (PPV) of over 99% and an AUC of 0.85 ± 0.06.[17] Despite the lower sensitivity, APRI cutoffs of 1.5 and 2 have also been shown to be associated with cirrhosis, other adverse liver and clinical outcomes, and death in our study cohort [18] as well as in others [19, 20].

Clinical predictors

All values were time-fixed at first available visit and distribution was evaluated in the subcohort and compared to the full eligible study sample to mimic how a predictive score would be applied in practice. Predictors considered included age; sex; ethnicity; alcohol use; body mass index (BMI); HIV viral load; CD4 count; and baseline APRI. Variables included in fibrotic staging indexes from other studies such as total bilirubin [21, 22] or gamma-glutamyltransferase (GGT) [2123] were also considered.

Markers of interest

Of interest to us was the genetic marker at IFNL SNP rs8099917, which has been linked with elevated histological inflammatory activity [14, 2428] as well as Natural Killer (NK) cell activation, resulting in cell death of infected liver cells and a pro-inflammatory environment [29, 30]. We chose the following immune markers based on their specific roles in liver fibrosis development (S1 Table): the cytokines transforming growth factor beta 1 (TGF-β1) and tumor necrosis factor alpha (TNF-α); the chemokines interleukin-8 (IL-8); monocyte chemotactic protein-1 (MCP-1 or CCL2); macrophage inflammatory protein 1 (MIP1α or CCL3; MIP1β or CCL4); Regulated upon Activation, Normal T cell Expressed and Secreted protein (RANTES or CCL5); CXCL9; and CXCL11; endothelial activation markers soluble Intercellular Adhesion Molecule 1 (sICAM-1) and soluble Vascular Cell Adhesion Molecule 1 (sVCAM-1); high-sensitivity C-reactive protein (hsCRP); and soluble CD14 (sCD14), a marker of microbial translocation [31, 32].

Immune markers were measured in patients with available samples from visit 1 or 2 from all the cases and the subcohort (n = 171 from subcohort and 46 non-subcohort cases, Fig 1). Frozen, never thawed plasma or serum samples were used. After thawing on ice, viral activity was inactivated (with a 0.5% sample concentration of Triton X-100 and 30 minute incubation at room temperature). Samples were aliquoted, refrozen and thawed on ice before running with a commercial assay from Millipore on a MAGPIX instrument (Millipore Corporation, Billerica MA) according to the manufacturer’s instructions. Samples were diluted according to kit recommendations: 3-plex (RANTES, sICAM1, sVCAM1; TGF- β1, 2, 3), 6-plex (IL6, IL8, MCP1, MIP1α, MIP1β, and TNFα) and 2-plex (CXCL9 and CXCL11). Standards were prepared in the same background as samples.

Commercially available ELISA kits were used to measure plasma and serum levels of soluble CD14 (sCD14, R&D Systems, Minneapolis, MN, USA) with dilutions of 1:300. High-sensitivity C-reactive protein (hsCRP) was tested based on manufacturer’s instructions, using immunoassay kits from Synchron LX 20 PRO (Beckman Coulter, Ontario, Canada). Hyaluronic acid, another direct marker of fibrogenesis [33] that has been included in other fibrotic indexes [22, 34] was also measured with a 1:30 dilution using the Hyaluronan Quantikine ELISA kit (R&D Systems, Minneapolis, MN, USA).

To measure IFNL genotypes, never thawed plasma and serum samples were processed using a real-time PCR assay developed by the Bay Area Genetic Lab (BAGL, Ontario, Canada), as previously described [35].

Statistical analysis

Survival analysis.

The subcohort (n = 236) was generated with a random sampling fraction of 0.45 and included 54 cases. Cases that were not in the subcohort (n = 59) entered via delayed entry. The time axis was follow-up time in study. Cox proportional hazards was used for analysis, with robust variance and Barlow weights to account for the case cohort design [36].

Descriptive analysis was conducted in the subcohort using box plots, histograms, correlation matrices with Spearman’s correlation coefficient, scatter plots and Q-Q plots. Values of the markers that were near the limits of detection were assigned the lowest detectable value.

Predictor selection and functional form.

Predictors were selected based on their availability to physicians, strength of correlation with each other [37], magnitude of associations in univariable analyses, ability to improve model fit as indicated by the Akaike Information Criterion (AIC) and ability to maximize discrimination as measured by Harrell’s C [38]. For immune markers, variables were also included in models if they captured a different stage of the underlying etiology of fibrosis development or were linked to fibrogenesis in the literature.

Univariable and multivariable analyses were conducted on untransformed variables, as well as after log-transformation or using median or quartile distributions of the immune markers. Continuous variables were centered at their mean values. Log transformation was used for all included continuous variables such as baseline APRI and immune markers IL-8, sICAM-1, RANTES, hsCRP, and sCD14 but not for age, which was modeled as a restricted cubic spline with 3 knots at the 10th, 50th and 90th percentiles corresponding to ages 33, 44, and 54. HIV viral load was dichotomized (undetectable or not at ≤ 50 copies/ml), as were alcohol use (currently drinking or not); HCV genotype (3 vs. non-3, i.e. types 1, 2, and 4); and host IFNL genotype rs8099917 (TT vs. non-TT).

Proportional hazards were assessed using the using Stata command–stphtest, detail- which uses scaled Schoenfeld residuals to check if proportional hazards holds globally and for included predictors.

Predictive accuracy.

Discrimination, calibration and changes in reclassification were compared between Model 1 (selected clinical predictors only) and Model 2 (clinical predictors from Model 1 plus selected genetic and immune markers) for predicting 3-year risk of significant liver fibrosis.

Discrimination was measured with a weighted [39] Harrell’s C or concordance index using Stata command–somersd- with robust jackknife estimator for standard errors [38]. A C-index value of 1 indicates perfect discrimination, while 0.5 means no better than random guessing. Calibration was assessed statistically (Hosmer-Lemeshow statistic and the Gronnesby and Borgan (GB) test) [40], and graphically with the Stata command–stcoxgrp using imputed data [41].

Change in reclassification was measured by the net reclassification improvement (NRI) summary index [42]. We calculated both the category-based and the continuous NRIs. For the category-based NRI, we used 3 clinically relevant risk categories: low risk, < = 10%; medium risk, >10–25%; and high risk, >25%. The categories were determined based on estimates of mortality from liver disease in those with chronic HCV infection from published reports [43] as well as opinions of knowledgeable hepatologists and clinicians. For the continuous NRI, no categories were needed and any upward or downward movement in risk was considered, regardless of magnitude.

The models were internally validated using bootstrapping. All analyses were conducted using Stata 12.

Multiple imputation.

Missingness for plasma samples and other variables was assumed to be at random. Multiple Imputation by Chained Equations (MICE) was used on the full cohort to account for all missing data, using all the predictors in the final models, all the immune markers, as well as variables that were possibly related to the reasons for missingness.

We compared predictive accuracy after using unweighted Cox proportional hazards regression on the imputed full cohort data [44].

Results

The subcohort selected was representative of both the CCC and the eligible subpopulation from which it was derived (Tables 1 and 2). A notable difference was in the median APRI score at baseline, which was much higher in the CCC overall, likely due to the 276 prevalent cases of significant liver fibrosis and end-stage liver disease (ESLD) that were excluded. The majority of the study participants were white males with a median age of 44 years; half had been infected with HCV for 18 years and almost half reported drinking alcohol (Table 1). Most were receiving HIV antiretroviral therapy and had well-controlled HIV with good CD4 recovery (>350 cells/μl) and undetectable HIV viral load.

thumbnail
Table 1. Baseline characteristics of the source, study and analytic populations.

https://doi.org/10.1371/journal.pone.0176282.t001

thumbnail
Table 2. Multivariable results of Cox models analyzing the association of significant liver fibrosis [HR (95% CI)] before and after multiple imputation.

https://doi.org/10.1371/journal.pone.0176282.t002

One hundred and thirteen individuals developed significant liver fibrosis over 1300 years of risk for an event rate of 8.63 per 100 person-years (95% CI: 7.08, 10.60 per 100 py). Significant liver fibrosis cases were much more likely to be female, currently drinking alcohol, infected with HCV genotype 3 and carriers of the rs8099917 TT genotype (Table 2). Surprisingly, they were also more likely to have undetectable levels of HIV RNA at first visit. As expected, even at baseline, the median APRI value was higher among cases than those in the subcohort.

Immune marker values were available for 74% of the individuals in the analytic sample. The median values of the following immune markers were higher in cases than the subcohort: IL-8, MIP1α, MIP1β, TNFα, CXCL9, sICAM-1 and sVCAM-1 (S2 Table, S1 Fig). The remaining markers (RANTES, TGF-β1, MCP-1, CXCL11, sCD14 and hs-CRP) showed the opposite, that is, were lower in cases than the subcohort. The genetic marker at IFNL rs8099917 was available in 92% of the individuals included in the analysis. The pro-inflammatory TT genotype was more common in cases than the subcohort (Table 2).

The final clinical predictors in Model 1 were sex, current alcohol use (yes or no), HIV viral load (undetectable or not at ≤ 50 copies), natural log-transformed baseline APRI, HCV genotype 3 and age (Table 2). Other factors like CD4 count or ethnicity did not improve predictive ability and therefore were not included. Model 1 was nested in Model 2, which also included the genetic marker IFNL rs8099917 and the following 5 log-transformed immune markers: IL-8, sICAM-1, RANTES, hsCRP, and sCD14 (Table 2). While other combinations of markers and predictors were tested, these markers were selected because they target different stages of liver fibrosis development, are known risk factors for fibrosis, improved model fit or discrimination or displayed the strongest associations with the outcome in univariable analyses (S3 Table). Even in multivariable analysis, most of the selected markers had a stronger association with the outcome than any of the clinical predictors with the exception of baseline APRI (Table 3). The selected markers were also strongly correlated with the other markers that were not included.

thumbnail
Table 3. Predictive accuracy for 3-year risk of significant liver fibrosis using model 1 (clinical predictors only) and model 2 (Model 1+ Selected Markers).

https://doi.org/10.1371/journal.pone.0176282.t003

The bootstrapped Harrell’s C indexes differed between models 1 and 2, regardless of multiple imputation (Table 2). The higher values for model 2 versus model 1 indicated that adding the six markers improved the discrimination capacity beyond that of traditional clinical factors. Using only clinical risk factors indicated that there was a 73% probability that predicted risk is higher for cases than non-cases. That probability rose to 76% using selected markers. Other clinical factors such as CD4 count or the remaining immune markers did not substantially increase the C-index or improve model fit as measured by the AIC. Variables included in indexes from other studies such as hyaluronic acid, total bilirubin, or GGT also did not increase the C-index, improve calibration, or improve model fit.

Both models 1 and 2 were well-calibrated as measured by the Hosmer-Lemeshow test and the GB test (p>0.05 in both tests, Table 2 and S2 Fig). This indicated that there were no significant deviations by observed risk from subgroups of predicted risk estimates, regardless of number or location of cutpoints. Risk stratification also appeared to improve with the inclusion of the genetic and immune markers in model 2. The category-based NRI indicated that the classification improved more in those not developing fibrosis (i.e. the non-events) than in those with the outcome: 5.1% were correctly reclassified (i.e. moved to a higher risk category) in those with events, while almost 21% of the non-events were correctly recategorized (i.e. correctly moved to a lower risk category) (Table 4). The continuous NRI, on the other hand, demonstrated improvement in both, with a greater movement among cases than non-cases (Table 5).

thumbnail
Table 4. Net reclassification improvement (NRI) index for 3-year risk of significant liver fibrosis using model 1 (clinical predictors only) and model 2 (Model 1+ Selected Markers).

https://doi.org/10.1371/journal.pone.0176282.t004

thumbnail
Table 5. Comparison of category-based and continuous net reclassification improvement (NRI) indexes for 3-year risk of significant liver fibrosis using model 1 (clinical predictors only) and model 2 (Model 1+ Selected Markers).

https://doi.org/10.1371/journal.pone.0176282.t005

Discussion

Our results demonstrate that specific immune markers improved ability to predict 3-year risk of significant liver fibrosis over traditional clinical factors alone in a cohort representative of HIV-HCV co-infected Canadians. While improvement in discrimination was modest, adding immune markers improved risk classification. Since this is a population at high risk for accelerated liver fibrosis, there is a great need for better clinical prognosis and risk assessment; such a prognostic tool may facilitate prioritizing expensive HCV treatment by improving identification of those for whom it may be safely delayed.

We tested clinical predictors linked with fibrosis that were most likely to be available to clinicians. The immune markers we screened, however, are not routinely collected at a clinical setting and are not cheap to measure (for the 6 additional markers included in Model 2, costs were approximately $110 for reagents alone). Nevertheless, the blood draw required for immune marker measures is less invasive than a biopsy and does not require specialist appointments or equipment and is far less expensive than current HCV treatment. Many of the markers were correlated with hepatic levels[45] and with each other, so we selected those most representative of different stages of fibrosis development, those with the strongest links to fibrosis, and those most likely to improve discrimination.

Other than baseline APRI, many of the selected markers displayed stronger relationships with fibrosis than any clinical predictor studied. In univariable and multivariable analysis, both IL-8 and sICAM-1 acted profibrogenically, being linked with a higher rate of significant fibrosis. These markers also appeared to behave in a dose-dependent manner when examined at the median and quartile level. RANTES and hs-CRP, on the other hand, appeared protective in univariable analysis (S3). We summarized other studies that could account for these trends from an etiological perspective (S1 Table). However, as our purpose in modeling them was solely for prediction, we cannot infer any causality from these associations.

Other factors, such as hyaluronic acid [22, 34], total bilirubin [21, 22], or GGT [2123] have been included in other fibrotic indexes but did not improve model fit or prognostic ability in our study. In studies of the Hepascore and the SHASTA index [22, 34], these variables were collected not for predicting risk but for diagnostic and staging purposes in individuals who had already developed liver disease. As such, they were collected at the same time point as the liver biopsy sample, which was used to determine the outcome (fibrosis or cirrhosis). In our study, however, the markers were measured in disease-free individuals up to three years before significant fibrosis set in, thus somewhat incorporating the stochastic nature of prognosis[46].

Using these chosen six markers improved our ability to discriminate between those who develop significant liver fibrosis and those who do not, as indicated by the higher C-indexes in model 2 versus model 1 and the wider separation of the Kaplan-Meier curves from the risk score tertiles (Fig 2), though the improvement was minor and not statistically significant. Nevertheless, discrimination by both models was similar to other comparable indexes such as the prognostic score for prediction of ESLD in HIV-HCV co-infected individuals on cART (C-statistic = 0.73), which can be considered good and of some clinical utility [47]. Other prognostic indexes which predicted risk of mortality in cirrhotics (Child-Pugh score, MELD score) or in HIV-infected individuals on cART (VACS Index) [48] had C-statistics around 0.80 or higher, which are considered very good or excellent [49, 50].

thumbnail
Fig 2. Tertiles of risk score against Kaplan-Meier estimates of risk of significant liver fibrosis.

Blue = score tertile 1 (lowest risk); Red = score tertile 2; Green = score tertile 3 (highest risk).

  • Model 1 included the following clinical predictors: sex, current alcohol use, HIV viral load, baseline APRI, HCV genotype 3 and age
  • Model 2 included Model 1 predictors and the following: genetic marker at IFNL rs8099917 and 5 immune markers IL-8, sICAM-1, RANTES, hsCRP, and sCD14

  1. The wider separation of the Kaplan-Meier curves in Model 2 provides a visual representation of the improvement in discrimination with the addition of the markers compared to Model 1. The risk score is constructed from the linear predictors of the Cox model. The linear predictor is a weighted sum of the variables in the final model, where the weights are the regression coefficients. (See also S4 Table).

https://doi.org/10.1371/journal.pone.0176282.g002

While improvement in discrimination was modest, inclusion of immune markers improved the net reclassification improvement index, as indicated by both category-based and continuous NRI estimates. Results from the category-based NRI indicate that measuring the markers in model 2 correctly reduced risk estimates in those who did not develop liver fibrosis. While this does not enable identification of higher-risk individuals for treatment [51], it could help identify patients for whom treatment might be safely deferred (up to 21% of individuals who eventually did not develop fibrosis over 3 years). The continuous NRI, on the other hand, indicated that model 2 improved ability to identify higher-risk individuals for treatment. However, these changes in predicted risk might not be clinically significant, as values of continuous NRIs are often higher than their category-based counterparts [51]. When viewed together with the Harrell’s C-index, the NRI seemed to support improved discrimination and risk classification with inclusion of genetic and inflammation markers. These results, however, must be interpreted with caution. Category-based NRIs are sensitive to the number of risk categories as well as the chosen cutpoints [42], while continuous NRIs can have large values for even weak markers [51]. A cost-benefit analysis that takes into account the expense of these markers as well as the high cost of HCV treatment might be useful as a next step.

The strength of our study includes a large source population that is broadly representative of Canadian co-infected patients, thus making our results directly generalizable and clinically relevant. This is important as marker and disease prevalence or marker correlations with other known risk factors can all affect the estimates of the discriminatory capacity of a marker [52]. Inclusion of specific markers in model 2 appeared to provide higher discrimination than the clinical predictors in model 1 whether multiple imputation was used or not. Finally, the list of immune markers we tested target different stages of the underlying mechanism of fibrosis progression, enhancing our ability to capture the outcome at various stages of development.

The limitations of our study include having marker measures at only visit 1 or 2, so we are unable to assess the predictive value of markers at other time points or measure the predictive value of changes of marker levels. However, as we were interested in prognosis, using the first available sample mirrors what would occur in clinical practice when a patient is first evaluated. We also lacked the power to assess interaction or effect measure modification of markers in different subgroups. Furthermore, missing samples in the subcohort and cases also reduced power, which we addressed with multiple imputation. Weakened associations after imputation can be caused by misspecification of the imputation model or if variables capturing the missing mechanism are excluded [53]. However, our results did not change with different imputation models. Since our primary goal was to assess the prognostic ability of specific markers, using the full sample size gave us sufficient power to assess calibration and discrimination properly, despite some potential misspecification. HCV duration, while informative, is hard to estimate in a real-life clinical scenario and is only approximate in our study. We chose to use age instead to capture some of the time element that HCV duration would have provided. Most of the injection drug users in our study first started injecting around the same age, approximately 18 years before cohort entry. Our study modeled the markers specifically for prognostic assessment precluding causal inferences. Finally, external validation in an independent dataset should be performed before applying this model clinically. Data provided in S2 and S4 Tables will be especially relevant to future external validation studies.

In conclusion, we found that in an HIV-HCV co-infected population, incorporating a genetic marker from IFNL rs8099917 and the immune markers IL-8, sICAM-1, RANTES, hs-CRP, and sCD14 allowed us to better predict the 3-year risk of significant liver fibrosis over traditional clinical risk factors alone. While the improvement in discrimination was small, the model with the markers also classified risk and fit better than the one without the markers. To assess whether this improvement justifies the additional cost of measuring these markers in the face of highly expensive HCV treatment requires further cost-benefit analyses.

Supporting information

S1 Table. Summary of immune markers of interest.

https://doi.org/10.1371/journal.pone.0176282.s001

(DOC)

S2 Table. Distribution of immune markers in random subcohort and cases outside subcohort.

https://doi.org/10.1371/journal.pone.0176282.s002

(DOC)

S3 Table. Univariable results of the association of significant liver fibrosis with log-transformed immune markers [HR (95% CI)].

https://doi.org/10.1371/journal.pone.0176282.s003

(DOC)

S4 Table. Estimated regression coefficients and standard errors for calculating risk score [estimated beta (SE)] in final models 1 and 2.

https://doi.org/10.1371/journal.pone.0176282.s004

(DOC)

S1 Fig. Medians of log-transformed immune markers: Subcohort vs. cases outside subcohort.

Abbreviations: TGF-β1, transforming growth factor beta 1; sICAM-1, soluble intercellular adhesion molecule 1; sVCAM-1, soluble vascular cell adhesion molecule 1; RANTES, Regulated upon Activation, Normal T cell Expressed and Secreted protein; sCD14, soluble CD14; TNF-α, tumor necrosis factor alpha; MIP1β, macrophage inflammatory protein 1 beta; MCP-1, monocyte chemotactic protein-1; CXCL11, chemokine (C-X-C motif) ligand 11; CXCL9, chemokine (C-X-C motif) ligand 9; hsCRP high-sensitivity C-reactive protein; MIP1α, macrophage inflammatory protein 1 alpha; IL-8, interleukin-8.

https://doi.org/10.1371/journal.pone.0176282.s005

(TIF)

S2 Fig. Calibration with predicted survival curves and Kaplan-Meier estimates in model 1 (clinical predictors only) and Model 2 (Model 1 plus IFNL rs8099917 and 5 Selected Immune Markers).

  1. a). Left panel: Equal-sized Quintiles of 3-year Risk in Model 1 (top) vs. Model 2 (bottom)

Smooth lines represent predicted survival probabilities, and vertical capped lines denote Kaplan–Meier estimates with 95% confidence intervals. Five prognosis groups are plotted: the “Good” group (darkest lines) and the “Poor” group (palest lines) at the highest and lowest risk categories, respectively, with the other 3 in between.

  1. b). Right panel: 3 Unequal Risk Groups (Cut at the 25th and the 75th Percentiles of the Failure Times)

Smooth lines represent predicted survival probabilities, and vertical capped lines denote Kaplan–Meier estimates with 95% confidence intervals. Three prognosis groups are plotted: the “Good” group (darkest lines), the “Intermediate” group (medium-dark lines), and the “Poor” group (paler lines). Using unequal sized risk groups allows identification of individuals with the most extreme prognosis [41].

  • Model 1 included the following clinical predictors: sex, current alcohol use, HIV viral load, baseline APRI, HCV genotype 3 and age
  • Model 2 included Model 1 predictors and the following: genetic marker at IFNL rs8099917 and 5 immune markers IL-8, sICAM-1, RANTES, hsCRP, and sCD14.

Abbreviations: IL-8, interleukin-8; sICAM-1, soluble intercellular adhesion molecule 1; RANTES, Regulated upon Activation, Normal T cell Expressed and Secreted protein; sCD14, soluble CD14; hsCRP high-sensitivity C-reactive protein.

https://doi.org/10.1371/journal.pone.0176282.s006

(TIF)

Acknowledgments

Special thanks to Rhyan Pineda for sample preparation and for running the hyaluronic acid assay.

The Canadian Co-infection cohort investigators (CTN222) are: Drs. Lisa Barrett, QEII Health Science Center for Clinical Research, Halifax, NS; Jeff Cohen, Windsor Regional Hospital Metropolitan Campus, Windsor, ON; Brian Conway, Vancouver Infectious Diseases Research and Care Centre, Vancouver, BC; Curtis Cooper, The Ottawa Hospital Research Institute, Ottawa ON; Pierre Côté, Clinique du Quartier Latin, Montréal, QC; Joseph Cox, MUHC IDTC-Montréal General Hospital, Montréal, QC; M. John Gill, Southern Alberta HIV Clinic, Calgary, AB; Shariq Haider, McMaster University, Hamilton, ON; Mark Hull, BC Centre for Excellence in HIV/AIDS, Vancouver, BC; Marina Klein, McGill University Health Centre, Division of Infectious Diseases and Chronic Viral Illness Service, Montreal, QC; Julio Montaner, St. Paul’s Hospital, Vancouver, BC; Erica Moodie, McGill University, Montreal, QC; Neora Pick, Oak Tree Clinic, Children's and Women's Health Centre of British Columbia, University of British Columbia, Vancouver, BC; Anita Rachlis, Sunnybrook & Women’s College Health Sciences Centre, Toronto, ON; Danielle Rouleau, Centre Hospitalier de l’Université de Montréal, Montréal, QC; Roger Sandre, HAVEN Program, Sudbury, ON; Mark Tyndall, Department of Medicine, Infectious Diseases Division, University of Ottawa, Ottawa ON; Steve Sanche, SHARE University of Saskatchewan, Saskatoon, SK; Marie-Louise Vachon, Centre Hospitalier Universitaire de Québec, Québec, QC; Sharon Walmsley, University Health Network, Toronto, ON; Alex Wong, Regina Qu'Appelle Health Region, Regina General Hospital, Regina, SK.; and David Wong, University Health Network, Toronto, ON.

CTN222 lead and corresponding author: Marina B. Klein, marina.klein@mcgill.ca

We thank all study coordinators and nurses for their assistance with study coordination, participant recruitment and care.

Author Contributions

  1. Conceptualization: NM MBK RWP.
  2. Data curation: CK DR NM.
  3. Formal analysis: NM RWP MBK CK.
  4. Funding acquisition: MBK.
  5. Investigation: NM CK.
  6. Methodology: NM RWP MBK.
  7. Project administration: NM SS.
  8. Resources: CK DR MBK MJG MH SW CTN222.
  9. Supervision: MBK RWP DR.
  10. Validation: CK DR.
  11. Visualization: NM.
  12. Writing – original draft: NM MBK.
  13. Writing – review & editing: NM CK DR MJG MH SW SS RWP MBK.

References

  1. 1. Price JC, Thio CL. Liver Disease in the HIV-Infected Individual. Clinical gastroenterology and hepatology: the official clinical practice journal of the American Gastroenterological Association. 2010;8(12):1002–12.
  2. 2. Lo Re V 3rd, Kallan MJ, Tate JP, Localio AR, Lim JK, Goetz MB, et al. Hepatic decompensation in antiretroviral-treated patients co-infected with HIV and hepatitis C virus compared with hepatitis C virus-monoinfected patients: a cohort study. Annals of internal medicine. 2014;160(6):369–79. PubMed Central PMCID: PMCPMC4254786. pmid:24723077
  3. 3. Roe B, Hall WW. Cellular and molecular interactions in coinfection with hepatitis C virus and human immunodeficiency virus. Expert reviews in molecular medicine. 2008;10:e30. Epub 2008/10/22. pmid:18928579
  4. 4. Benhamou Y, Bochet M, Di Martino V, Charlotte F, Azria F, Coutellier A, et al. Liver fibrosis progression in human immunodeficiency virus and hepatitis C virus coinfected patients. The Multivirc Group. Hepatology. 1999;30(4):1054–8. Epub 1999/09/25. pmid:10498659
  5. 5. Graham CS, Baden LR, Yu E, Mrus JM, Carnie J, Heeren T, et al. Influence of human immunodeficiency virus infection on the course of hepatitis C virus infection: a meta-analysis. Clinical infectious diseases: an official publication of the Infectious Diseases Society of America. 2001;33(4):562–9. Epub 2001/07/20.
  6. 6. American Association for the Study of Liver Diseases and the Infectious Diseases Society of America. HCV Guidance: Recommendations for Testing, Managing, and Treating Hepatitis C. 2015.
  7. 7. European Association for the Study of the LiverAsociacion Latinoamericana para el Estudio del H. EASL-ALEH Clinical Practice Guidelines: Non-invasive tests for evaluation of liver disease severity and prognosis. Journal of hepatology. 2015;63(1):237–64. pmid:25911335
  8. 8. Peltekian KM. With all the new treatment regimens, complete elimination of hepatitis C virus in Canada is a possibility! But when will Canadians have access these drugs? Canadian journal of gastroenterology & hepatology. 2014;28(8):417–8. PubMed Central PMCID: PMC4210230.
  9. 9. Hull M, Shafran S, Tseng A, Giguere P, Klein MB, Cooper C. CIHR Canadian HIV Trials Network Co-Infection and Concurrent Diseases Core: Updated Canadian guidelines for the treatment of hepatitis C infection in HIV-hepatitis C coinfected adults. The Canadian journal of infectious diseases & medical microbiology = Journal canadien des maladies infectieuses et de la microbiologie medicale / AMMI Canada. 2014;25(6):311–20. PubMed Central PMCID: PMC4277159.
  10. 10. British Columbia PharmaCare Information for Prescribers: Determining fibrosis stage for the treatment of chronic hepatitis C. British Columbia PharmaCare, Division MBaPS; October 2014.
  11. 11. Romero-Gomez M, Eslam M, Ruiz A, Maraver M. Genes and hepatitis C: susceptibility, fibrosis progression and response to treatment. Liver international: official journal of the International Association for the Study of the Liver. 2011;31(4):443–60. Epub 2011/03/09.
  12. 12. Ghany MG, Kleiner DE, Alter H, Doo E, Khokar F, Promrat K, et al. Progression of fibrosis in chronic hepatitis C. Gastroenterology. 2003;124(1):97–104. Epub 2003/01/04. pmid:12512034
  13. 13. Probst A, Dang T, Bochud M, Egger M, Negro F, Bochud PY. Role of hepatitis C virus genotype 3 in liver fibrosis progression—a systematic review and meta-analysis. Journal of viral hepatitis. 2011;18(11):745–59. Epub 2011/10/14. pmid:21992794
  14. 14. Eslam M, Hashem AM, Leung R, Romero-Gomez M, Berg T, Dore GJ, et al. Interferon-lambda rs12979860 genotype and liver fibrosis in viral and non-viral chronic liver disease. Nature communications. 2015;6:6422. pmid:25740255
  15. 15. Moqueet N, Cooper C, Gill J, Hull M, Platt RW, Klein MB, et al. Responder Interferon lambda Genotypes Are Associated With Higher Risk of Liver Fibrosis in HIV-Hepatitis C Virus Coinfection. The Journal of infectious diseases. 2016;214(1):80–6. PubMed Central PMCID: PMCPMC4907413. pmid:26984148
  16. 16. Klein MB, Saeed S, Yang H, Cohen J, Conway B, Cooper C, et al. Cohort profile: the Canadian HIV-hepatitis C co-infection cohort study. International journal of epidemiology. 2010;39(5):1162–9. Epub 2009/09/30. pmid:19786463
  17. 17. Al-Mohri H, Cooper C, Murphy T, Klein MB. Validation of a simple model for predicting liver fibrosis in HIV/hepatitis C virus-coinfected patients. HIV medicine. 2005;6(6):375–8. Epub 2005/11/05. pmid:16268817
  18. 18. Al-Mohri H, Murphy T, Lu Y, Lalonde RG, Klein MB. Evaluating liver fibrosis progression and the impact of antiretroviral therapy in HIV and hepatitis C coinfection using a noninvasive marker. J Acquir Immune Defic Syndr. 2007;44(4):463–9. Epub 2007/01/11. pmid:17211282
  19. 19. Macias J, Giron-Gonzalez JA, Gonzalez-Serrano M, Merino D, Cano P, Mira JA, et al. Prediction of liver fibrosis in human immunodeficiency virus/hepatitis C virus coinfected patients by simple non-invasive indexes. Gut. 2006;55(3):409–14. Epub 2005/08/25. PubMed Central PMCID: PMC1856097. pmid:16118349
  20. 20. Macias J, Mira J, Gilabert I, Neukam K, Roldan C, Viloria M, et al. Combined use of aspartate aminotransferase, platelet count and matrix metalloproteinase 2 measurements to predict liver fibrosis in HIV/hepatitis C virus-coinfected patients. HIV medicine. 2011;12(1):14–21. Epub 2010/05/26. pmid:20497249
  21. 21. Imbert-Bismut F, Ratziu V, Pieroni L, Charlotte F, Benhamou Y, Poynard T. Biochemical markers of liver fibrosis in patients with hepatitis C virus infection: a prospective study. Lancet. 2001;357(9262):1069–75. Epub 2001/04/12. pmid:11297957
  22. 22. Adams LA, Bulsara M, Rossi E, DeBoer B, Speers D, George J, et al. Hepascore: an accurate validated predictor of liver fibrosis in chronic hepatitis C infection. Clinical chemistry. 2005;51(10):1867–73. pmid:16055434
  23. 23. Forns X, Ampurdanès S, Llovet JM, Aponte J, Quintó L, Martı , et al. Identification of chronic hepatitis C patients without hepatic fibrosis by a simple predictive model. Hepatology. 2002;36(4):986–92.
  24. 24. Abe H, Ochi H, Maekawa T, Hayes CN, Tsuge M, Miki D, et al. Common variation of IL28 affects gamma-GTP levels and inflammation of the liver in chronically infected hepatitis C virus patients. Journal of hepatology. 2010;53(3):439–43. Epub 2010/06/26. pmid:20576307
  25. 25. Barreiro P, Pineda JA, Rallon N, Naggie S, Martin-Carbonero L, Neukam K, et al. Influence of interleukin-28B single-nucleotide polymorphisms on progression to liver cirrhosis in human immunodeficiency virus-hepatitis C virus-coinfected patients receiving antiretroviral therapy. The Journal of infectious diseases. 2011;203(11):1629–36. Epub 2011/05/20. pmid:21592993
  26. 26. Rembeck K, Westin J, Lindh M, Hellstrand K, Norkrans G, Laging M, et al. Association between interleukin-28B-related genetic variants and liver histopathology differs between hepatitis C virus genotypes. Hepatology. 2012;56(1):394. pmid:22270967
  27. 27. Bochud P, Bibert S, Kutalik Z, Goossens N, Müllhaupt B, Gerlach T, et al. IL28B polymorphisms linked to poor response to treatment are associated with low necroinflammatory activity and slow fibrosis progression in HCV genotype non1-infected patients. Journal of hepatology. 2011;54(54):S539.
  28. 28. D'Ambrosio R, Aghemo A, De Francesco R, Rumi MG, Galmozzi E, De Nicola S, et al. The association of IL28B genotype with the histological features of chronic hepatitis C is HCV genotype dependent. International journal of molecular sciences. 2014;15(5):7213–24. PubMed Central PMCID: PMC4057668. pmid:24776764
  29. 29. Naggie S, Osinusi A, Katsounas A, Lempicki R, Herrmann E, Thompson AJ, et al. Dysregulation of innate immunity in hepatitis C virus genotype 1 IL28B-unfavorable genotype patients: impaired viral kinetics and therapeutic response. Hepatology. 2012;56(2):444–54. PubMed Central PMCID: PMC3361636. pmid:22331604
  30. 30. Suppiah V, Gaudieri S, Armstrong NJ, O'Connor KS, Berg T, Weltman M, et al. IL28B,HLA-C, and KIR Variants Additively Predict Response to Therapy in Chronic Hepatitis C Virus Infection in a European Cohort: A Cross-Sectional Study. PLoS medicine. 2011;8(9):e1001092. pmid:21931540
  31. 31. Marchetti G, Tincati C, Silvestri G. Microbial translocation in the pathogenesis of HIV infection and AIDS. Clinical microbiology reviews. 2013;26(1):2–18. PubMed Central PMCID: PMC3553668. pmid:23297256
  32. 32. Caradonna L, Mastronardi ML, Magrone T, Cozzolongo R, Cuppone R, Manghisi OG, et al. Biological and clinical significance of endotoxemia in the course of hepatitis C virus infection. Current pharmaceutical design. 2002;8(11):995–1005. pmid:11945146
  33. 33. Patel K, Lajoie A, Heaton S, Pianko S, Behling CA, Bylund D, et al. Clinical use of hyaluronic acid as a predictor of fibrosis change in hepatitis C. Journal of gastroenterology and hepatology. 2003;18(3):253–7. Epub 2003/02/27. pmid:12603524
  34. 34. Kelleher TB, Mehta SH, Bhaskar R, Sulkowski M, Astemborski J, Thomas DL, et al. Prediction of hepatic fibrosis in HIV/HCV co-infected patients using serum fibrosis markers: the SHASTA index. Journal of hepatology. 2005;43(1):78–84. Epub 2005/05/17. pmid:15894397
  35. 35. Moqueet N, Infante-Rivard C, Platt RW, Young J, Cooper C, Hull M, et al. Favourable IFNL3 genotypes are associated with spontaneous clearance and are differentially distributed in Aboriginals in Canadian HIV-hepatitis C co-infected individuals. International journal of molecular sciences. 2015;16(3):6496–512. PubMed Central PMCID: PMC4394544. pmid:25803108
  36. 36. Barlow WE, Ichikawa L, Rosner D, Izumi S. Analysis of case-cohort designs. Journal of clinical epidemiology. 1999;52(12):1165–72. Epub 1999/12/02. pmid:10580779
  37. 37. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:b604. pmid:19336487
  38. 38. Newson RB. Comparing the predictive powers of survival models using Harrell's C or Somers' D. Stata Journal. 2010;10(3):339–58.
  39. 39. Sanderson J, Thompson SG, White IR, Aspelund T, Pennells L. Derivation and assessment of risk prediction models using case-cohort data. BMC medical research methodology. 2013;13:113. PubMed Central PMCID: PMC3848813. pmid:24034146
  40. 40. Gronnesby JK, Borgan O. A method for checking regression models in survival analysis based on the risk score. Lifetime data analysis. 1996;2(4):315–28. pmid:9384628
  41. 41. Royston P. Tools for checking calibration of a Cox model in external validation: Prediction of population-averaged survival curves based on risk groups. Stata Journal. 2015;15(1):275–91.
  42. 42. Pencina MJ, D'Agostino RB Sr., D'Agostino RB Jr., Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in medicine. 2008;27(2):157–72; discussion 207–12. Epub 2007/06/15. pmid:17569110
  43. 43. Vergniol J, Boursier J, Coutzac C, Bertrais S, Foucher J, Angel C, et al. Evolution of noninvasive tests of liver fibrosis is associated with prognosis in patients with chronic hepatitis C. Hepatology. 2014;60(1):65–76. pmid:24519328
  44. 44. Marti H, Chavance M. Multiple imputation analysis of case-cohort studies. Statistics in medicine. 2011;30(13):1595–607. pmid:21351290
  45. 45. Wald O, Weiss ID, Galun E, Peled A. Chemokines in hepatitis C virus infection: pathogenesis, prognosis and therapeutics. Cytokine. 2007;39(1):50–62. pmid:17629707
  46. 46. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clinical chemistry. 2008;54(1):17–23. Epub 2007/11/21. pmid:18024533
  47. 47. Lo Re V, Kallan MJ, Tate JP, Lim JK, Goetz MB, Klein MB, et al. Predicting Risk of End-Stage Liver Disease in Antiretroviral-Treated Human Immunodeficiency Virus/Hepatitis C Virus-Coinfected Patients. Open Forum Infectious Diseases. 2015;2(3):ofv109. pmid:26284259
  48. 48. Tate JP, Justice AC, Hughes MD, Bonnet F, Reiss P, Mocroft A, et al. An internationally generalizable risk index for mortality after one year of antiretroviral therapy. AIDS (London, England). 2013;27(4):563–72.
  49. 49. Kamath PS, Kim WR, Advanced Liver Disease Study G. The model for end-stage liver disease (MELD). Hepatology. 2007;45(3):797–805. pmid:17326206
  50. 50. Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA: the journal of the American Medical Association. 2012;307(2):182–92. PubMed Central PMCID: PMCPMC3792853. pmid:22235089
  51. 51. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Annals of internal medicine. 2014;160(2):122–31. pmid:24592497
  52. 52. Greenland P, O'Malley PG. When is a new prediction marker useful? A consideration of lipoprotein-associated phospholipase A2 and C-reactive protein for stroke risk. Archives of internal medicine. 2005;165(21):2454–6. Epub 2005/11/30. pmid:16314539
  53. 53. White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Statistics in medicine. 2011;30(4):377–99. pmid:21225900