https://orcid.org/0000-0001-5998-3839
Figures
Abstract
Background
Clinical risk score tools require validation in diverse settings and populations before they are widely implemented. We aimed to externally validate an HIV risk assessment tool for predicting HIV acquisition among pregnant and postpartum women. In the context of prevention of mother-to-child transmission programs, risk score tools could be used to prioritize retesting efforts and delivery of pre-exposure prophylaxis (PrEP) to pregnant and postpartum women most at risk for HIV acquisition while minimizing unnecessary perinatal exposure.
Methods
Data from women enrolled in a cross-sectional study of programmatic HIV retesting and/or receiving maternal and child health care services at five facilities in Western Kenya were used to validate the predictive ability of a simplified risk score previously developed for pregnant/postpartum women. Incident HIV infections were defined as new HIV diagnoses following confirmed negative or unknown status during pregnancy. Predictive performance was assessed using the area under the receiver operating characteristic curve (AUC) and Brier score.
Results
Among 1266 women with 35 incident HIV infections, we found an AUC for predicting HIV acquisition of 0.60 (95% CI, 0.51, 0.69), with a Brier score of 0.27. A risk score >6 was associated with a 2.9-fold increase in the odds of HIV acquisition (95% CI, 1.48, 5.70; p = 0.002) vs scores ≤6. Women with risk scores >6 were 27% (346/1266) of the population but accounted for 52% of HIV acquisitions. Syphilis, age at sexual debut, and unknown partner HIV status were significantly associated with increased risk of HIV in this cohort.
Citation: Chhun N, Rothschild CW, Penumetsa M, Matemo D, Kithao P, Richardson BA, et al. (2024) Evaluating the performance of a risk assessment score tool to predict HIV acquisition among pregnant and postpartum women in Kenya. PLoS ONE 19(7): e0306992. https://doi.org/10.1371/journal.pone.0306992
Editor: Matt A. Price, International AIDS Vaccine Initiative, UNITED STATES OF AMERICA
Received: June 19, 2023; Accepted: June 25, 2024; Published: July 10, 2024
Copyright: © 2024 Chhun et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data underlying the results presented in the study are available from https://github.com/nchhun/ExternalValidationRiskScore.
Funding: The research described in this publication was supported by the National Institute of Allergy and Infectious Diseases (R03AI140922 [AD], K01AI116298 [AD], and the University of Washington/Fred Hutch Center for AIDS Research (P30 AI027757)) and the Eunice Kennedy Shriver National Institute of Child Health & Human Development (F31HD097841 [CR], F31HD105513 [NC]) of the National Institutes of Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health (https://www.nih.gov/) The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Clinical risk scores have been utilized in a variety of settings and populations to identify individuals who are at highest risk for HIV acquisition [1–5]. A prior risk score developed specifically for pregnant and postpartum women was moderately predictive of new HIV acquisition (area under the curve [AUC] of 0.76 (95% Confidence Interval [CI], 0.67, 0.85)) in Kenya [6]. However, this tool has not been externally validated. The tool was developed prior to the introduction of pre-exposure prophylaxis (PrEP) for pregnant and postpartum women. In resource-limited settings, a risk score could be used to prioritize HIV retesting efforts in prevention of mother-to-child transmission (PMTCT) programs, and delivery of PrEP to pregnant and breastfeeding women most at risk for HIV acquisition while minimizing unnecessary perinatal exposure.
Strategies that identify and prevent incident maternal HIV acquisition during pregnancy and the postpartum period are needed for elimination of mother-to-child transmission (EMTCT) [7–9]. While maternal HIV incidence is high in sub-Saharan Africa, recent trends suggest incidence is declining from 3.8 per 100 person-years (PY) between 1980–2012 [10] to 2.1 per 100 PY after 2014 [11]. One strategy for detecting incident maternal HIV, as recommended by the World Health Organization (WHO), is maternal HIV retesting in the third trimester, with catch-up testing at labor, delivery, and early postpartum [12]. Some countries may also consider additional time points later in the postpartum period to retest women [12]. This approach to retesting promotes detection of new maternal HIV infections among women who may not have tested during antenatal care or who initially tested negative and acquired HIV later in their pregnancy or during the postpartum period.
Current WHO guidelines recommend PrEP as an HIV prevention strategy for pregnant and breastfeeding women in high incidence settings [12,13]. Increased engagement with the healthcare system as a result of retesting efforts offers healthcare providers an opportunity to screen and counsel women at high risk for HIV acquisition about initiating PrEP. Prioritizing PrEP provision to women at highest need may reduce HIV transmission from their sexual partners and reduce perinatal transmission. Use of HIV risk assessment tools in antenatal and postnatal care settings may have potential to prevent and identify incident maternal HIV acquisition, minimizing additional resources necessary under a universal screening approach. Targeting resources to women at highest risk while limiting potential perinatal exposure is an approach that optimizes PrEP use during pregnancy and postpartum [14–16]. As such, efforts to evaluate how a risk score tailored for peripartum women performs in the context of declining incidence and roll-out of PrEP in pregnancy may be helpful in assessing utility of this approach.
We externally validated an existing HIV risk score designed for pregnant and postpartum women in a new population of pregnant and postpartum women from the same region in Kenya [17]. We also examined whether there were additional predictors of HIV acquisition that have emerged since development of the initial risk score.
Materials and methods
Study population
Data were from a cross-sectional study on maternal HIV retesting and a retrospective review of programmatic HIV retesting documented in maternal child health booklets; both studies aimed to identify incident HIV infections among Kenyan women receiving maternal and child health care services as previously described [18]. In the cross-sectional study, participants were enrolled between January 2017 and July 2019 from two health facilities, Ahero sub-District and Bondo District Hospitals in western Kenya. Women were enrolled during the third trimester of pregnancy (≥28 weeks gestation), at delivery, or at 6 weeks, 6 months, or 9 months postpartum. Women were eligible for study participation if they were ≥14 years; tested HIV-negative ≥3 months prior (if enrolled during pregnancy, labor and delivery, or 6 weeks postpartum), had no documentation of an initial test, or had an unknown HIV status (if enrolled during the labor and delivery or the postpartum period); and were willing to provide written informed consent. At enrollment, all women were administered a survey on sociodemographic, partner characteristics, sexual risks and behaviors, reproductive, clinical and HIV history information. Syphilis serology results were abstracted from maternal child health (MCH) booklets. After survey administration, HIV testing and counseling was conducted per Government of Kenya guidelines [19]; women newly diagnosed with HIV were referred for follow-up care and treatment at MCH clinics [18].
Women in the retrospective chart review of programmatic HIV retesting study had data abstracted when they were enrolled at 6 weeks or 9 months postpartum from January to July 2019, also from Ahero sub-District and Bondo District, in addition to three other health facilities, Rachuonyo and Siaya District Hospitals in Western Kenya, and Riruta Health Centre in Nairobi. Women identified with incident HIV were administered the same sociodemographic and behavioral survey as participants enrolled in the cross-sectional study, as previously described [18]. Similarly, syphilis serology was also abstracted from MCH booklets.
Incident HIV infections were defined as those with a documented prior negative HIV infection during or after pregnancy or unknown status during pregnancy. In addition, active case finding for pregnant and postpartum women with prior known incident maternal infection (negative rapid HIV test followed by later positive test before 1 year postpartum) reported by healthcare providers at the study sites was conducted; HIV retesting information was abstracted from programmatic MCH data at the current visit or retrospectively for women enrolled from both studies. In summary, the study population included women with HIV who were identified from the cross-sectional study, case finding, and from the programmatic HIV retesting data abstraction study, and HIV-negative women from the cross-sectional study. HIV-negative women who were enrolled in the programmatic HIV retesting data abstraction study were excluded because HIV risk factors were not collected.
Study procedures were approved by the Kenyatta National Hospital/University of Nairobi Ethics and Research Committee and the University of Washington Institutional Review Board. All authors had access to information that could identify individual participants during data collection; however, data were de-identified for analysis and accessed from May 2020 to May 2023. All participants provided written informed consent.
Statistical analysis
We aimed to externally validate the previously developed risk score [6]. The analytic sample was comprised of women who had complete data on all risk factors included in the simplified version of the risk score described by Pintye et al which focused on indicators routinely assessed in MCH settings; these include total number of lifetime sexual partners, male partner with unknown HIV status, and reactive syphilis serology test in pregnancy via rapid plasma reagin (RPR). The risk score was derived by assigning the following point values each indicator: one per sexual partner, six for a partner with unknown status, five for a reactive RPR, and zero if for none of these indicators. Since laboratory test results for bacterial vaginosis and candidiasis were not conducted as part of the cross-sectional study, nor routinely performed and documented in medical records, it was not possible to externally validate the full risk score.
In the external validation of the simplified risk score, individual risk score values were calculated for each woman. A score cut-point of 6 was used to define high risk corresponding to the previously developed score. Predictive ability of the simplified risk score, and the multivariable model from which the score was derived [6], was evaluated through estimation of area under the receiver operating characteristic curve (AUC-ROC). Further validation of the predictive ability of the risk score was done using 10-fold cross validation, a method which randomly partitions the data into 10 random subsets to calculate an average AUC [20]. The average AUC from the 10-fold cross validation was compared to the AUC from the complete dataset to ascertain robustness of the risk score tool using AUC values <0.7 to reflect poor performance [21]. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were estimated with varying risk score cut-points; with an optimal cut-point defined by Youden’s J statistic. Additionally, the number needed to screen (NNS) at each cut-point to detect one individual with HIV was calculated using the inverse of the predictive summary index (PSI), 1/PSI. The PSI is determined from the formula PPV + NPV– 1 [22]. The Brier score was also calculated to assess overall model performance, using a threshold of <0.25, which indicates the risk score is informative for prediction [20].
After external validation of the risk score, we examined whether additional predictors of HIV acquisition emerged since development of the initial risk score. We assessed this by constructing logistic regression models to characterize the relationship between each independent variable included in the initial risk score developed by Pintye et al., as well as additional correlates not assessed during initial risk score development (familiarity with PrEP as an HIV prevention intervention, and ever use of PrEP) [6]. Variables associated with HIV acquisition at p<0.10 in the univariate models were included in a multivariable model. If potential predictors were collinear, the variable with greater clinical relevance based on the literature was chosen for inclusion. The analysis was restricted to women with RPR information. We also conducted an exploratory analysis where if collinearity was detected, variables with the least missing data were chosen for inclusion in the multivariable model (S1 Table). Covariates included in the multivariable model were then assessed in a stepwise logistic regression model based on the minimum Akaike information criterion (AIC) [23] to identify the combination of factors that best predicted HIV acquisition.
A sensitivity analysis was conducted excluding women enrolled in other studies on maternal PrEP use as they may have higher risks of HIV acquisition and their inclusion may bias the risk score towards higher predictive ability of HIV acquisition. All analyses were conducted in Stata version 16.0 (StataCorp, College Station, TX), with cross validation analysis using the cvauroc package, and model selection using stepwise logistic regression with estat ic command post estimation.
Results
In total, 2770 women were identified in the cross-sectional study (2761 HIV-negative, 9 incident HIV infections) and an additional 36 incident infections (27 identified through active case finding, 9 through the programmatic HIV retesting data abstraction study). We included 1266 (45%) of these women in this analysis (1231 HIV-negative, 35 incident infections); women were excluded from the analysis if they had missing data on syphilis RPR (n = 1424), lifetime number of sexual partners (n = 107), and/or missing partner HIV status (n = 9).
Among all women, the overall median age was 23 years (interquartile range [IQR]: 20, 27), the majority (62%) of whom were between 21–30 years (Table 1). Most (72%) were currently in a relationship, for a median duration of 4 years, and with an older male partner (median 5 years older, IQR: 3, 7). One-third had a male partner with unknown HIV status. The median number of lifetime sexual partners was 2 (IQR: 1, 3). The majority (66%) reported having ever heard of PrEP, and among those who had heard of PrEP, only 8% had ever used PrEP. None of the women had an unknown HIV status at the time of enrollment. Among the 35 women with incident HIV infections included in the risk score validation, 15 (43%) were detected during pregnancy (9 during the third trimester, 6 at delivery) and 20 (57%) in the postpartum period (9 at 6 weeks, 2 at 6 months, and 9 at 9 months).
In the cohort, the median simplified risk score was 2 (IQR: 2, 7), with 27% (347/1266) defined as high-risk among all women, and 51% of women with incident HIV (18/35) identified as high risk (Table 2). A risk score >6 was associated with a 2.9-fold increase in the odds of HIV acquisition (95% CI, 1.48, 5.70; p = 0.002) and corresponded to an AUC for predicting HIV acquisition of 0.60 (95% CI, 0.51, 0.69; Table 3 and Fig 1), with a Brier score of 0.27. For each additional point increase in the risk score there was a trend for increasing odds of HIV acquisition (OR = 1.06, 95% CI, 0.99, 1.13; p = 0.09). Results were similar in a sensitivity analysis excluding 93 women (n = 1 with incident maternal HIV) co-enrolled in PrEP studies from the analysis.
Notes: Risk score and multivariable logistic model both include number of sexual partners, knowledge of partners’ HIV status, and syphilis status.
Using the external validation sample, the risk score cut-point used was 6, with 51.4% sensitivity and 72.3% specificity, 5.0% PPV, and 98.1% NPV. The number needed to screen to identify one individual with HIV was 32. In the cross-validation of the continuous risk score, the average AUC was 0.62 (95% CI, 0.44, 0.65; Fig 2).
Notes: 95% confidence intervals (CI) calculated using the bootstrap bias-corrected approach. Cross-validation conducted using Stata package cvauroc. Each dashed line represented one of the 10-fold samples; cvAUC = cross-validated area-under-the-receiver-operating-characteristic-curve.
In univariate logistic regression models, risk of HIV acquisition was associated with syphilis infection, a history of STIs, having a polygamous relationship, age at sexual debut less than 17 years, having a male partner of unknown HIV status, and having an older partner (Table 4). Differences in HIV risk by PrEP knowledge or ever use of PrEP were not detected. Having syphilis, an age of sexual debut less than 17 years, and having a male partner of unknown HIV status (compared to those with a negative or positive status) remained significantly associated with HIV acquisition in the multivariable model. In the stepwise model selection, syphilis infection (OR = 7.48, 95% CI: 1.48, 37.83), having an age at sexual debut less than 17 years (OR = 2.87, 95% CI: 1.27, 6.49), and having a male partner of unknown HIV status (OR = 2.53, 95% CI: 1.17, 5.49) were associated with maternal HIV acquisition.
Discussion
Overall, the simplified risk score performed moderately based on the Brier score at predicting risk of HIV acquisition during pregnancy and postpartum period in our external validation cohort, with an AUC lower than previously reported by Pintye et al (0.60, 95% CI: 0.51, 0.69 vs. 0.76, 95% CI: 0.67, 0.85, respectively); indicating poor performance. We found the optimal cut-point for the risk score in the external validation sample was 6.5, the same cut-point identified by Pintye and colleagues; a risk score >6 was associated with a nearly 3-fold increased odds of HIV acquisition, identifying more than a quarter of women as high risk. Half of all women with incident HIV infection would have been correctly classified as high risk. Although risk scores have utility in identifying individuals at risk for HIV acquisition, performance reproducibility of risk score may vary, with the same score demonstrating both poor and moderate predictive ability [1,24,25], highlighting the impact of different settings and populations on risk score performance.
Prioritizing strategies for early detection, such as a simple HIV risk score that can be administered by a healthcare worker in antenatal and postnatal care settings using indicators routinely assessed has the potential to avert incident maternal and pediatric HIV acquisition. While the risk score performance was lower in our external validation than in the initial internal validation, maternal characteristics (having a partner of unknown HIV status and syphilis [17]) included in the simplified risk score are robust and remained predictive of HIV acquisition. Utilizing dual HIV/syphilis testing capable of simultaneously testing for both infections into antenatal care has been shown to be cost effective, and is now recommended by WHO as a first line HIV test [26]. Use of dual HIV/syphilis rapid diagnostic testing offer opportunities to increase coverage of syphilis testing and treatment within MCH settings, and has potential to identify women to prioritize for HIV retesting later during pregnancy or postpartum. In addition, this strategy aligns with WHO guidance and goals for EMTCT of both HIV and syphilis [27].
In the analysis of this cohort, we found that early age at sexual debut was an additional cofactor for HIV acquisition. Prior studies in other populations and settings have found similar results [28,29]. This cofactor could easily be added to the variables routinely collected in the context of MCH clinical settings to identify women at highest risk for HIV and prioritize delivery of prevention interventions, such as PrEP.
Our analysis had several strengths. Few HIV risk assessment tools have been externally validated, and our analysis permitted assessment of utility of implementation and whether performance was altered based on changing epidemiology and prevention interventions. We evaluated additional potential correlates for HIV acquisition, including familiarity with PrEP as an HIV prevention intervention, and ever use of PrEP. While the PrEP variables were not significant in the analysis, complementary efforts to identify high risk women to target with HIV prevention interventions, such as PrEP, may bolster testing and treatment approaches for EMTCT.
Our study has limitations. We only included women in our analysis with complete data on all variables assessed in the risk score; while this may bias our results, multiple imputation may also introduce bias in assessment of risk score performance [30]. Additionally, inclusion of women with incident infection from active case finding may have increased the frequency of HIV in our study, which would bias PPV, NPV, and NNS. The simplified HIV risk assessment tool was originally developed and internally validated using data from a longitudinal study of pregnant and postpartum women; although we externally validated this risk score tool in a population of pregnant and postpartum women from the same region in Kenya, we used data from a cross-sectional study, which would fail to detect changes in risk factors over time. Given the cross-sectional nature of this study, risk factors were assessed concurrently with, rather than prior to, diagnosis of incident infection which may have limited the predictive estimate. Furthermore, the sample size, and number of women with incident infections, for the analysis to explore potential additional risk factors for HIV acquisition was small and led to imprecise estimates of risk.
In summary, using indicators routinely assessed in antenatal care, the external validation of the simplified risk score did not perform as well as it had initially at predicting risk of HIV acquisition. However, in settings with limited resources and in the absence of other tools to ascertain who to prioritize for retesting or PrEP services, this tool may still be valuable. Further evaluation of the simplified risk score in various resource limited settings is needed to determine utility for prioritizing retesting efforts and prevention strategies to avert HIV acquisition among pregnant, postpartum, and breastfeeding women who would most benefit.
Supporting information
S1 Checklist. STROBE statement—checklist of items that should be included in reports of observational studies for manuscript (PONE-D-23-16018).
https://doi.org/10.1371/journal.pone.0306992.s001
(DOCX)
S1 Table. Correlates of HIV acquisition for women receiving maternal and child health care during pregnancy and postpartum in western Kenya.
Notes: * among partnered women only in unadjusted analyses; polygamous relationship excluded to allow unmarried women to be included in the multivariable model; a Unadjusted analysis; b Covariates based on factors associated with HIV acquisition (P<0.10), n = 2774; Due to collinearity, variables marital status, partner age difference, and syphilis were excluded from the multivariable analysis; c Covariates for stepwise multivariable model based on Akaike information criterion score, n = 2774.
https://doi.org/10.1371/journal.pone.0306992.s002
(DOCX)
Acknowledgments
We thank our study participants, our study team, and the Global Center for Integrated Health of Women, Adolescents, and Children (Global WACh).
References
- 1. Balkus JE, Brown E, Palanee T, Nair G, Gafoor Z, Zhang J, et al. An Empiric HIV Risk Scoring Tool to Predict HIV-1 Acquisition in African Women. J Acquir Immune Defic Syndr. 2016;72(3):333–43. pmid:26918545
- 2. Kahle EM, Hughes JP, Lingappa JR, John-Stewart G, Celum C, Nakku-Joloba E, et al. An empiric risk scoring tool for identifying high-risk heterosexual HIV-1-serodiscordant couples for targeted HIV-1 prevention. J Acquir Immune Defic Syndr. 2013;62(3):339–47. pmid:23187945
- 3. Menza TW, Hughes JP, Celum CL, Golden MR. Prediction of HIV acquisition among men who have sex with men. Sex Transm Dis. 2009;36(9):547–55. pmid:19707108
- 4. Smith DK, Pals SL, Herbst JH, Shinde S, Carey JW. Development of a clinical screening index predictive of incident HIV infection among men who have sex with men in the United States. J Acquir Immune Defic Syndr. 2012;60(4):421–7. pmid:22487585
- 5. Hoenigl M, Weibel N, Mehta SR, Anderson CM, Jenks J, Green N, et al. Development and validation of the San Diego Early Test Score to predict acute and early HIV infection risk in men who have sex with men. Clin Infect Dis. 2015;61(3):468–75. pmid:25904374
- 6. Pintye J, Drake AL, Kinuthia J, Unger JA, Matemo D, Heffron RA, et al. A Risk Assessment Tool for Identifying Pregnant and Postpartum Women Who May Benefit From Preexposure Prophylaxis. Clin Infect Dis. 2017;64(6):751–8. pmid:28034882
- 7. Egbe TO, Tazinya RM, Halle-Ekane GE, Egbe EN, Achidi EA. Estimating HIV Incidence during Pregnancy and Knowledge of Prevention of Mother-to-Child Transmission with an Ad Hoc Analysis of Potential Cofactors. J Pregnancy. 2016;2016:7397695. pmid:27127653
- 8. Thomson KA, Hughes J, Baeten JM, John-Stewart G, Celum C, Cohen CR, et al. Increased Risk of HIV Acquisition Among Women Throughout Pregnancy and During the Postpartum Period: A Prospective Per-Coital-Act Analysis Among Women With HIV-Infected Partners. J Infect Dis. 2018;218(1):16–25. pmid:29514254
- 9. Machekano R, Tiam A, Kassaye S, Tukei V, Gill M, Mohai F, et al. HIV incidence among pregnant and postpartum women in a high prevalence setting. PLoS One. 2018;13(12):e0209782. pmid:30592749
- 10. Drake AL, Wagner A, Richardson B, John-Stewart G. Incident HIV during pregnancy and postpartum and risk of mother-to-child HIV transmission: a systematic review and meta-analysis. PLoS Med. 2014;11(2):e1001608. pmid:24586123
- 11. Graybill LA, Kasaro M, Freeborn K, Walker JS, Poole C, Powers KA, et al. Incident HIV among pregnant and breast-feeding women in sub-Saharan Africa: a systematic review and meta-analysis. AIDS. 2020;34(5):761–76. pmid:32167990
- 12.
World Health Organization (WHO). Consolidated guidelines on HIV testing services, 2019. Geneva: World Health Organization; 2020.
- 13.
World Health Organization (WHO). Preventing HIV during pregnancy and breastfeeding in the context of PrEP Technical brief. Geneva: World Health Organization; July 2017.
- 14. Pintye J, Davey DLJ, Wagner AD, John-Stewart G, Baggaley R, Bekker LG, et al. Defining gaps in pre-exposure prophylaxis delivery for pregnant and post-partum women in high-burden settings using an implementation science framework. Lancet HIV. 2020;7(8):e582–e92. pmid:32763221
- 15. Dettinger JC, Kinuthia J, Pintye J, Abuna F, Begnel E, Mugwanya K, et al. Perinatal outcomes following maternal pre-exposure prophylaxis (PrEP) use during pregnancy: results from a large PrEP implementation program in Kenya. J Int AIDS Soc. 2019;22(9):e25378. pmid:31498563
- 16. Dettinger JC, Kinuthia J, Pintye J, Mwongeli N, Gomez L, Richardson BA, et al. PrEP Implementation for Mothers in Antenatal Care (PrIMA): study protocol of a cluster randomised trial. BMJ Open. 2019;9(3):e025122. pmid:30850409
- 17. Kinuthia J, Drake AL, Matemo D, Richardson BA, Zeh C, Osborn L, et al. HIV acquisition during pregnancy and postpartum is associated with genital infections and partnership characteristics. AIDS. 2015;29(15):2025–33. pmid:26352880
- 18. Penumetsa M, Neary J, Farid S, Kithao P, Richardson BA, Matemo D, et al. Implementation of HIV Retesting During Pregnancy and Postpartum in Kenya: A Cross-Sectional Study. Glob Health Sci Pract. 2022;10(1). pmid:35294386
- 19.
National AIDS and STI Control Programme, Ministry of Health, Kenya. Guidelines for HIV Testing Services in Kenya. Nairobi: NASCOP; 2015.
- 20. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38. pmid:20010215
- 21.
Hosmer David W. Jr., Lemeshow Stanley, and Sturdivant Rodney X. 2013. Applied Logistic Regression. 3rd ed. Hoboken, NJ: John Wiley & Sons.
- 22. Larner AJ. Number Needed to Diagnose, Predict, or Misdiagnose: Useful Metrics for Non-Canonical Signs of Cognitive Status? Dement Geriatr Cogn Dis Extra. 2018;8(3):321–7. pmid:30386368
- 23. Vrieze SI. Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychol Methods. 2012;17(2):228–43. pmid:22309957
- 24. Rosenberg NE, Kudowa E, Price JT, Pettifor A, Bekker LG, Hosseinipour MC, et al. Identifying Adolescent Girls and Young Women at High Risk for HIV Acquisition: A Risk Assessment Tool From the Girl Power-Malawi Study. Sex Transm Dis. 2020;47(11):760–6. pmid:33045165
- 25. Giovenco D, Pettifor A, MacPhail C, Kahn K, Wagner R, Piwowar-Manning E, et al. Assessing risk for HIV infection among adolescent girls in South Africa: an evaluation of the VOICE risk score (HPTN 068). J Int AIDS Soc. 2019;22(7):e25359. pmid:31353814
- 26. Rodriguez PJ, Roberts DA, Meisner J, Sharma M, Owiredu MN, Gomez B, et al. Cost-effectiveness of dual maternal HIV and syphilis testing strategies in high and low HIV prevalence countries: a modelling study. Lancet Glob Health. 2021;9(1):e61–e71. pmid:33227254
- 27. World Health Organization (WHO). Dual HIV/Syphilis rapid diagnostic tests can be used as the first test in antenatal care. November 2019. Policy Brief WHO/CDS/HIV/19.38.
- 28. Wand H, Ramjee G. The relationship between age of coital debut and HIV seroprevalence among women in Durban, South Africa: a cohort study. BMJ Open. 2012;2(1):e000285. pmid:22223838
- 29. Pettifor AE, van der Straten A, Dunbar MS, Shiboski SC, Padian NS. Early age of first sex: a risk factor for HIV infection among women in Zimbabwe. AIDS. 2004;18(10):1435–42. pmid:15199320
- 30. Hughes RA, Heron J, Sterne JAC, Tilling K. Accounting for missing data in statistical analyses: multiple imputation is not always the answer. Int J Epidemiol. 2019;48(4):1294–304. pmid:30879056