https://orcid.org/0000-0003-1233-5724
Figures
Abstract
Introduction
Loss to follow-up (LTFU) among adolescents and young adults living with HIV (AYALWH) is a barrier to optimal health and HIV services. We developed and validated a clinical prediction tool to identify AYALWH at risk of LTFU.
Methods
We used electronic medical records (EMR) of AYALWH ages 10 to 24 in HIV care at 6 facilities in Kenya and surveys from a subset of participants. Early LTFU was defined as >30 days late for a scheduled visit in the last 6 months, which accounts for clients with multi-month refills. We developed a tool combining surveys with EMR (‘survey-plus-EMR tool’), and an ‘EMR-alone’ tool to predict high, medium, and low risk of LTFU. The survey-plus-EMR tool included candidate sociodemographics, partnership status, mental health, peer support, any unmet clinic needs, WHO stage, and time in care variables for tool development, while the EMR-alone included clinical and time in care variables only. Tools were developed in a 50% random sample of the data and internally validated using 10-fold cross-validation of the full sample. Tool performance was evaluated using Hazard Ratios (HR), 95% Confidence Intervals (CI), and area under the curve (AUC) ≥ 0.7 for good performance and ≥0.60 for modest performance.
Results
Data from 865 AYALWH were included in the survey-plus-EMR tool and early LTFU was (19.2%, 166/865). The survey-plus-EMR tool ranged from 0 to 4, including PHQ-9 ≥5, lack of peer support group attendance, and any unmet clinical need. High (3 or 4) and medium (2) prediction scores were associated with greater risk of LTFU (high, 29.0%, HR 2.16, 95%CI: 1.25–3.73; medium, 21.4%, HR 1.52, 95%CI: 0.93–2.49, global p-value = 0.02) in the validation dataset. The 10-fold cross validation AUC was 0.66 (95%CI: 0.63–0.72). Data from 2,696 AYALWH were included in the EMR-alone tool and early LTFU was 28.6% (770/2,696). In the validation dataset, high (score = 2, LTFU = 38.5%, HR 2.40, 95%CI: 1.17–4.96) and medium scores (1, 29.6%, HR 1.65, 95%CI: 1.00–2.72) predicted significantly higher LTFU than low-risk scores (0, 22.0%, global p-value = 0.03). Ten-fold cross-validation AUC was 0.61 (95%CI: 0.59–0.64).
Citation: Wilson K, Agot K, Dyer J, Badia J, Kibugi J, Bosire R, et al. (2023) Development and validation of a prediction tool to support engagement in HIV care among young people ages 10–24 years in Kenya. PLoS ONE 18(6): e0286240. https://doi.org/10.1371/journal.pone.0286240
Editor: Brian C. Zanoni, Emory University School of Medicine, UNITED STATES
Received: June 15, 2022; Accepted: May 11, 2023; Published: June 30, 2023
Copyright: © 2023 Wilson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Our study is part of the Prevention and Treatment through a Comprehensive Care Continuum for HIV-affected Adolescents in Resource Constrained Settings (PATC3H), an NICHD-funded consortium of investigators conducting ongoing studies. The data for this prediction tool came from the formative phase of our study. Our randomized trial is ongoing. PATC3H has a data sharing policy that requires we upload final de-identified datasets within one year of acceptance of the primary [trial] manuscript or trial closure. Data are uploaded to a public-facing Data and Specimen Hub (DASH) https://dash.nichd.nih.gov. We request to comply with the PATC3H data sharing policy for this study and upload the cleaned and de-identified dataset to DASH by 2024. We provide our data availability statement in this cover letter, as requested: We will make a de-identified risk score dataset available through the NICHD Data and Specimen Hub https://dash.nichd.nih.gov according to the data sharing policy of the Prevention and Treatment through a Comprehensive Care Continuum for HIV-affected Adolescents in Resource Constrained Settings (PATC3H) Consortium.
Funding: This research was sponsored by the National Institute of Child Health & Human Development (UG3 HD096906) and Prevention and Treatment through a Comprehensive Care Continuum for HIV-affected Adolescents in Resource Constrained Settings (PATC3H) (https://www.patc3h.org/SitePages/Overview.aspx) Additional support was provided by the University of Washington (UW) Global Center for Integrated Health of Women, Adolescents, and Children (Global WACh); the Center for Global Health Nursing; and Center for AIDS Research (CFAR; P30 AI027757; PI: J. Baeten). The Center for AIDS Research is supported by the following NIH Institutes and Centers: NIAID, NCI, NIMH, NIDA, NICHD, NHLBI, NIA, NIGMS, NIDDK. (https://depts.washington.edu/cfar/?q=home) KW, KA, JD, PK, II, JB, RB, SS, GJS, JN, JK, KBS, were supported by both awards. NC was the NICHD Program Officer for this study and was funded separately. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Adolescents and young adults living with HIV (AYALWH) in sub-Saharan Africa continue to experience lower retention in care and poorer outcomes than adults, despite the availability of antiretroviral therapy (ART) to improve individual health and reduce transmission risk [1]. Once treatment has been initiated, loss-to-follow-up (LTFU) among AYALWH ages 10 to 24 ranges from 20–30% depending on the population and outcome definition [2–4]. In Kenya, a country with a high burden of HIV, adolescents and young adults comprise nearly 50% of new HIV infections [5] and LTFU has been reported between 15–50% [6, 7]. LTFU can increase the risk of viral non-suppression through interruption of ART [8, 9]. An analysis of the population-based Kenya AIDS Indicator Survey data (2012) of factors associated with community viral load prevalence in Kenya found that younger age (15–29 years versus 30–64) and being out of HIV care (LTFU or never enrolled) were associated with detectable viral load (≥500 copies/ml) [10]. Recently, UNAIDS announced a renewed commitment to improve retention in care among AYALWH as a key strategy to end AIDS by 2030 [11].
Several factors can increase risk of LTFU from care among AYALWH [12]. These include HIV-related stigma [13], lack of ‘youth friendly’ providers or spaces [14, 15], lack of support transitioning to adult care [15], and depression [16]. Potential interventions to reduce risk of LTFU among AYALWH [17, 18] include family-based economic support [19–21], ‘youth-friendly’ services [14] and peer-support groups [22]. However, these interventions require significant resource investments for already strained health systems, and there is a need to develop approaches to prioritize which interventions to provide AYALWH when resources are scarce [23].
Many countries in sub-Saharan Africa have adopted differentiated service delivery models, a form of client-centered care intended to improve system efficiencies, quality, and outcomes [23]. Clients identified as clinically stable can shift to less frequent visits and multi-month ART refills, while unstable clients continue with standard care. In Kenya, AYALWH ages 20 and older are eligible for differentiated care, including multi-month ART refills and longer time between visits [24]. The emphasis on differentiated care prioritizes individuals doing well in care, however, there remains a need for guidelines to systematically improve care for those who are at risk for LTFU or who are unstable in care.
Clinical prediction tools are an effective, data-driven strategy to improve care and treatment decisions for a range of health conditions [25–31] including HIV [28, 31–33]. They are developed using data from similar client populations and adapted to work in a routine care setting [29, 34]. With expanded use of digital systems, there are new opportunities to use prediction tools to support HIV services using data from electronic health records systems [35, 36]. Clinical prediction tools have been developed to identify women who may be eligible for Pre-Exposure Prophylaxis (PrEP) services [27, 37, 38], adults in need of HIV testing [33], and adults at risk of viral failure who would benefit from adherence support [31, 39]. To date, there are no prediction tools designed to identify AYALWH at risk of LTFU.
A clinical prediction tool to identify AYALWH at risk of LTFU could support clinicians to better allocate intensified care to at-risk AYALWH before they are lost and to identify stable AYALWH for differentiated services. To address this gap, we developed and validated a clinical prediction tool to identify AYALWH at risk of LTFU using surveys and routine data.
Materials and methods
Setting and population
We conducted a prospective cohort study among AYALWH enrolled in care at six facilities in Kisumu and Homa Bay counties in Kenya. Selection criteria for the facilities were having an active electronic medical records (EMR) system, at least 100 AYALWH enrolled in care, and permission from facility managers. Data sources for this clinical prediction tool included AYALWH EMR and surveys. EMR data included all AYALWH ages 10 to 24 years enrolled in HIV care from October 1, 2018, until administrative censoring on February 29, 2020, due to COVID-19. Starting in April 2019, all eligible AYALWH ages 10 to 24 enrolled in care at the time were invited to participate in a cohort study that included behavioral surveys at enrollment and every 6 months. Study staff obtained written informed consent from AYALWH ages 18 to 24 or caregiver consent and adolescent assent for adolescents ages 10 to 17 years. Study staff administered face-to-face surveys during routine clinic visits in the participants’ preferred language (Kiswahili, Dholuo, or English).
Outcomes and predictors
The primary outcome of the prediction tool was loss to follow-up, defined as >30 days late for a scheduled visit in the EMR during a 6-month period. We chose this definition to measure early risk of LTFU among AYALWH when interventions to support clinic attendance may be most effective [39]. Counting from last scheduled visit rather than last actual visit accounted for AYALWH who were on multi-month dispensing (MMD) schedules. For example, a client on a 3-month refill regimen would be LTFU if they were more than 30 days late a quarterly visit (4 months out of care). AYALWH were excluded if their first visit in the EMR occurred within 30 days of February 29, 2020, or if they had an enrollment survey that could not be linked to their EMR.
Candidate predictors of LTFU were selected from AYALWH surveys and EMR data based on plausibly and prior studies among AYALWH [6, 13, 14]. We used an adapted ecological framework [40] to guide our thinking, and present example predictors in Fig 1.
Variables from surveys included social (pays own expenses), service (any caregiver accompaniment, any unmet clinical needs, peer support group attendance, use of any differentiated ART refill models, seen at youth-friendly vs. adult clinic), interpersonal (exposure to any physical, sexual, or emotional violence), and individual characteristics not captured in EMR (e.g. alcohol use, other drug use, did not complete secondary school). The variable unmet clinical needs was computed as indicating at least one need was not met after reading aloud a list of standard services that should be offered at the clinic visit, including contraception and pregnancy information, screening for STIs, depression, gender-based violence (GBV), and substance use, nutrition assessment, and referral to peer support groups. Asking clients whether their clinical needs were met at the visit is often used to evaluate quality of care [41]. Resilience was defined using a 2-item Connor-Davidson Resilience Scale [42]. Depressive symptoms were measured using Patient Health Questionnaire-9 (PHQ-9), social support by the Multidimensional Scale of Perceived Social Support (MSPSS) [43], alcohol use by the Alcohol Use Disorders Identification Test-Consumption (AUDIT-C) [44], and exposure to any physical, sexual, or emotional violence in the last six months using an adapted version of the World Health Organization (WHO) Violence Against Women survey [45]. Assessment of depressive symptoms using PHQ-9 score, substance use, exposure to recent violence, and participation in peer support group attendance are recommended for AYALWH in Kenya, although these data are not yet routinely incorporated in EMR systems [33, 36]. Candidate predictors available in EMR were age at first visit since October 1, 2018, marital status (single, married, divorced/widowed), WHO stage (recoded as a binary variable: 3 or 4, ‘advanced disease’ versus 1 or 2, ‘asymptomatic/early stage infection’) [46], transfer status (transferred in vs. never transferred), and time on ART. We created variables for newly enrolled in care (less than 6 months versus 6 months or more) and time on ART (less than 3 years versus 3 or more) from existing data. We used enrollment values of all predictors except for time on ART and WHO status, which were time-varying in the EMR. We computed time on ART as the total days between date of ART initiation until the outcome occurred or administrative censoring on February 29, 2020, converted to years. We used the first non-missing value for WHO stage after entry into the analysis cohort from each client record.
We developed two versions of the tool to predict high, medium, and low risk of LTFU. The ‘survey-plus-EMR’ tool included data among the subset of AYALWH who were enrolled in the cohort study and had surveys linked to EMR data (April 2019 to February 2020). We also developed an EMR-alone tool that included a larger sample of AYALWH in care (October 2018 to February 2020), some of whom were not enrolled in the cohort study therefore lacked survey data. We used methods for prediction rules [27, 28] and from risk scores used to predict HIV infection among women in Kenya [28] and South Africa [37] and virologic failure among adults living with HIV in Haiti [31]. Continuous variables and categorical variables with three or more levels were transformed into binary values. These included age group (adolescents as 10–19 and young adults as 20–24 years), marital status (single vs. ever married), PHQ-9 (0–4 vs 5 or higher), and AUDIT-C (0–3 vs 4 or higher).
Univariable Cox regression models were developed to estimate hazard ratios (HR) and 95% Confidence Intervals (95% CI) and two-sided α = 0.05. All regression models accounted for variability between facilities (e.g. size, volume, location) by clustering on facility. To evaluate potential effect modification by age group, all models were re-run with an interaction term between the predictor and binary age. We proposed developing age-group stratified prediction tools if the p-values of most interaction terms were <0.1. Since this was not the case, age group was included as a pre-specified predictor in the model. All predictors with significance level of 0.1 or less were included in multivariable regression models. We did not adjust for multiple comparisons to minimize loss of power. In this complete-case analysis, only AYALWH with non-missing data for all predictors were included. Overall, 70 individuals were dropped from the multivariable analyses for the survey-plus-EMR tool due to missing one or more responses, 41 from the training dataset and 29 from the validation dataset. We used Akaike Information Criteria (AIC) backwards elimination to determine the most predictive variables to include in the final model. If key variables were necessary to make the predictive model robust, these were retained in the model. Because WHO stage had >10% missing data, that variable was excluded from multivariable analysis, and all cases with missing WHO would have been dropped. We decided not to perform multiple imputation for variables with missing data to preserve the dataset as it would be used in routine care, where imputation is not conducted. Age at ART initiation was excluded due to collinearity with age group. A score was assigned to each predictor from the final stepwise regression model by dividing the coefficient for each predictor by the smallest coefficient among all predictors in the model and rounding to the nearest integer [46].
Tool performance was assessed through model testing according to standard steps for prediction tool development [46, 47]. Specifically, we calculated area under the receiver operating curve (AUC, and range 0–1.0) which is identical to the c-statistic for continuous outcomes [29] using optimal cut points of a binary version of the score to assess ability to differentiate between individuals at high and low risk of LTFU. We used the standard cut off AUC ≥0.7 for ‘good’ performance, and AUC of 0.60–0.69 for modest performance [29, 48]. We used a Brier score (range 0–1.0) to estimate the accuracy to predict LTFU. We then created a three-level score and assigned numeric values to the low, medium, and high-risk categories based on the score distribution in tertials. Each tool was developed in a random sample of 50% of the data (‘training dataset’) and externally validated in the remaining 50% of the data (‘validation dataset’) that was not used to generate the initial score. We validated the score using unadjusted Cox regression models in the validation cohort, accounting for clustering by facility. We used unadjusted analyses to develop a tool that providers could understand and explain to clients. We used the Global Wald test of significance for the risk score, against the null hypothesis that none of the risk levels in the score were different than zero. We also conducted 10-fold cross-validation procedures in the full dataset to evaluate the generalizability of each tool, estimated by AUC. We compared the ‘survey-plus-EMR’ tool with the EMR-alone tool descriptively to determine whether the EMR tool was able to provide comparable clinical prediction. All analyses were conducted in Stata 16.0 (College Station, Texas).
Sample size and power
We estimated the minimal detectable hazard ratios of LTFU assuming sample size of 1,350 AYALWH enrolled on the cohort with EMR data and a LTFU proportion of 25% for an effective analytic sample of ~1,000. We estimated the minimum detectable difference in LTFU proportion at 80% power for a range of predictor levels (10%-50%), at α = 0.05, for the training and validation cohorts (n = 500 each). Under these assumptions, we estimated 80% power to detect RR ≥1.66 for common exposures and ≥2.12 for rare exposures in the training or validation samples, respectively.
This study was approved by the University of Washington Institutional Review Board and Maseno University Ethics Review Committee. We received permissions from County and health facility leadership prior to accessing EMR and viral load data.
Results
Survey-plus-EMR prediction tool
In the six facilities, 973 AYALWH had enrolled in the cohort study at the time of the analysis, and 108 did not have a linked EMR, resulting in 865/973 (88.9%) eligible individuals. Overall, 6-month LTFU was 19.2% and similar between the training and validation cohorts (88/433, 20.3% vs. 78/432, 18.1%, p = 0.40) (Table 1).
In the training dataset of 432 AYALWH, we evaluated social, service, interpersonal (partner violence), psychosocial, and individual-level variables from surveys as candidate predictors of LTFU in univariable analysis (Table 2). Variables from EMR were age group, sex, marital status, newly enrolled, transfer status, time on ART, and WHO stage. Of the variables tested, PHQ-9 score, any unmet service need, no participation in a peer support group, enrollment in adult care, and newly enrolled were independently significantly associated with LTFU and evaluated in multivariable analyses (N = 795).
Variables most predictive of LTFU in the final model using AIC selection were PHQ-9 score, no participation in a peer support group, and at least one unmet service need (lowest AIC = 837.29, partial log likelihood = —415.61, k = 3 parameters) (Table 3). Including age group in the model did not improve model performance. Total prediction scores ranged from 0 to 4. Using a 0.5 cut point for the binary form of the score, the AUC was 0.58 (95% CI: 0.51–0.64, standard error = 0.04), with sensitivity of 0.53 and specificity of 0.62, and Brier score of 0.40 (Table 3 and Fig 2A). The three-level prediction tool classified 15.9% of AYALWH as high risk, 25.6% medium risk, and 59.1% as low risk of LTFU, with medium- and high-risk scores associated with significantly greater risk of actual LTFU (medium risk, HR 1.52, 95%CI: 0.93–2.49, high risk, HR 2.16, 95%CI: 1.25–3.73; Global Wald Chi-square = 7.72 and p-value = 0.02 comparing medium and high to low-risk scores, degrees of freedom = 2).
a. Survey plus EMR (N = 391); b. EMR alone (N = 1,349).
In the validation cohort (n = 404), medium (HR 1.58, 95%CI: 1.07–2.33) and high-risk scores (HR 1.46, 95%CI: 0.76–2.81) predicted greater risk of LTFU compared to low-risk scores and had a trend for an association (Global Wald Chi-square = 5.29 and p-value = 0.07, degrees of freedom = 1). There was limited generalizability using 10-fold cross validation in the full sample (AUC = 0.66, 95%CI: 0.63–0.72, standard error = 0.025, Chi-square = 11.72, p-value<0.001).
EMR-alone prediction tool
A total of 2,696 records from six facilities were included in this analysis. Most AYALWH were ages 10–14 (28.2%) or 20–24 years (46.6%), female (70.8%), had been in HIV care at least 6 months (78.9%), and WHO stage 1 or 2 (85.1%). The median time on ART was 3.1 years (interquartile range [IQR] 0.1–7.6). Overall, 6-month LTFU in the EMR cohort was 28.6% (n = 770), with no significant difference between training and validation cohorts (27.4% vs. 29.7%, p = 0.20) (Table 1). The training dataset included 1,349 records. In the univariable analyses, we evaluated age group, marital status, sex, age at ART initiation (continuous), newly enrolled, transfer status, time on ART, and WHO stage as predictors of LTFU (Table 2). WHO status and newly enrolled were associated with significantly greater likelihood of LTFU in univariable analyses. Age group was retained because it improved model performance.
The final predictive model using AIC included age group and newly enrolled status (lowest AIC = 5518.17, partial log likelihood = -2592.085, k = 2 parameters). Individual scores ranged from 0 (low risk) to 2 (high risk) (Table 2). The prediction tool performance using an optimal cut point of 0.5 was poor and similar to the survey-plus-EMR tool. The AUC was 0.58 (95%CI: 0.55–0.61, standard error = 0.16), and sensitivity and specificity were 0.62 and 0.51, respectively (Table 3 and Fig 2B). As a three-level score, there was a stepwise trend of higher risk of LTFU with higher scores. When the prediction tool was evaluated in the validation cohort (n = 1,347), medium (HR 1.65, 95%CI: 1.00–2.72) and high-risk scores (HR 2.40, 95%CI: 1.17–4.96) predicted significantly higher LTFU compared to low-risk scores (Global Wald Chi-square = 6.84 and p-value = 0.03). In 10-fold cross validation, the prediction tool had an AUC of 0.61 (95% CI:0.59–0.63, standard error = 0.0120, Chi-square = 14.04, p-value<0.001).
Discussion
We developed two clinical prediction tools, one using EMR data combined with survey data and one using only EMR data, to predict LTFU among AYALWH and found similarly poor predictive performance. Overall, both tools showed poor performance by standard measures. The survey-plus-EMR tool, which included PHQ-9 score, peer support group enrollment, and any unmet care needs had slightly better performance (AUC) than the EMR-alone tool in the training dataset. The EMR-alone tool, which only included age and time in care, predicted LTFU in the validation cohort. The tools had similar generalizability when cross-validated in the full samples. While both tools demonstrated moderate predictive ability, results may guide future clinical prediction tools and decision-support interventions with AYALWH.
We chose the outcome of early LTFU to support early intervention before AYALWH are lost. The survey-plus-EMR analysis evaluated several additional variables that are likely associated with adolescent LTFU [48] to determine whether their inclusion could improve tool performance. These included depressive symptoms, alcohol and other drug use, social support, peer support group attendance, and exposure to violence. The final full model included PHQ-9 score, peer support group attendance, and any unmet clinical need at the time of clinic visit. The EMR-alone tool included two variables from EMR, age group and time in care. Including age-group improved performance of the EMR-alone tool although not the survey-plus-EMR tool. One reason could be that the EMR-alone tool had different (and fewer) predictors than the survey-plus-EMR tool.
We found that older age group (20–24 years) and longer time in care (six months or more) were predictors of LTFU, and older age has been shown to be associated with LTFU in prior studies [2]. Young adults may have higher risk of LTFU compared to adolescents because of less caregiver support to adhere to clinic visits or life changes, including marriage or seasonal travel. Longer time in care may have been a marker of AYALWH who had recently switched to multi-month ART refills. This change to their visit routine may have resulted in LTFU. Both versions of the tool were parsimonious, including fewer than five variables that are relatively easy to obtain and simple scores ranging from 0 to 2 or 4.
Compared to other clinical prediction tools developed for HIV prevention and care in sub-Saharan Africa, our tool included fewer variables and demonstrated lower performance by AUC. The ability of our tool to discriminate between AYALWH who were lost and not lost in both the training and validation cohorts ranged from 0.61 to 0.66, which is lower than the commonly used threshold of 0.7 [29]. Other tools developed to predict risk of HIV infection [33], HIV incidence among women [27, 28] and female adolescents [38, 49] and risk of viral non-suppression among adults [49] report performance by AUC ranging from 0.69 [27, 33, 37] to >0.90 [38]. One study among AYALWH enrolled in care in South Africa developed and validated a clinical prediction tool to determine readiness to transition to adult care, using the outcome of viral suppression [50]. This tool showed good performance across measures (e.g. AUC of 0.84) and included six predictors (ART regimen line, gender, HIV status disclosure, HIV Adolescent Readiness to Transition Scale score [51], age at ART initiation, and prior alcohol use). Direct comparisons of prediction tools are challenging because prediction models are highly sensitive to the variables included, variable section methods, and distribution of characteristics in the underlying population [34, 52]. In addition, our study outcome of LTFU measures clinic attendance and is not a direct proxy for ART adherence or viral non-suppression.
Other HIV-related prediction tools with performance of <0.7 by AUC have been evaluated in different research cohorts [37, 38] with intended use in routine care [37, 53]. For example, Puttkammer et al. [31] developed a prediction tool of ART failure among adults in HIV care in Haiti using the national EMR data, which had an AUC of 0.61. The authors then evaluated the feasibility of an intervention to improve viral suppression among adults with HIV in Haiti that combined an EMR-based alert system using the prediction tool to identify clients at risk of treatment failure and brief adherence counseling [53]. The intervention was feasible and acceptable, and associated with significantly improved adherence and a non-significant increased trend in viral suppression compared to baseline. However, providers reported challenges using the EMR-based tool including imperfect understanding of the alert criteria and mismatch between what the alert showed and how the client felt at the visit, which could raise doubts about the tool. Using prediction tools with sub-optimal performance (AUC <0.7) risk incorrectly identifying clients as needing a specific intervention or missing clients who do. However, as the study by Puttkammer et al., highlights, prediction tools with modest performance can offer useful information about challenges of implementing prediction tools into practice as well as other HIV service and client factors that need to be addressed for decision-support interventions in HIV care. Given the modest performance of both tools in our study, we do not recommend adapting these tools for routine care given the potential of mis-identifying AYALWH at risk of LTFU. Instead, the predictors evaluated in survey-plus-EMR and EMR-alone tools could be used as the basis for development of new prediction tools in similar populations and inform potential targets of future decision-support interventions with AYALWH.
Our study had strengths and limitations. To our knowledge, this is the first study to develop and validate a clinical prediction tool to identify AYALWH at risk of LTFU. We assessed psychosocial and service-related variables that have not been tested in prior prediction models for LTFU, and we considered both survey-plus-EMR and EMR-alone versions. The EMR-alone tool was limited to variables in that dataset, which were primarily clinical and demographic variables. LTFU using EMR data is subject to misclassification due to variable data quality and completeness [12]. We tried to minimize the possibility by using a definition of LTFU that aligns with the definition that facilities use, which accounted for more staggered visit schedules for clients in differentiated care. However, we could not verify whether some AYALWH we classified as LTFU had transferred out or were receiving care at another facility temporarily since EMRs are not synchronized across facilities. The survey-plus-EMR tool may have been underpowered to evaluate predictors with prevalence less than 10% in the sample, including harmful alcohol use and drug use, which may also be associated with LTFU in adolescent populations [50]. We also had to exclude outcomes after the initiation of COVID-19 response measures in Kenya, which reduced both sample size and follow-up time. The prevalence of LTFU in the survey-plus-EMR tool cohort was substantially lower than in the EMR-alone cohort, which likely reflected selection bias of AYALWH who enrolled in the survey when they came for clinic visits. In addition, there was a higher proportion of females and AYALWH ages 20–24 years in the EMR-alone compared to the survey-plus-EMR tool. The prevalence of LTFU in the survey-plus-EMR tool cohort was substantially lower than in the EMR-alone cohort, which likely reflected selection bias of AYALWH who enrolled in the survey. In addition, there was a higher proportion of females and AYALWH ages 20–24 years in the EMR-alone compared to the survey-plus-EMR tool. It is possible that the number and type of client-level predictors and associations with LTFU may differ in more urban settings. Differences in samples limited direct comparisons between tools. However, by using both data sources, we could evaluate a larger cohort (EMR-alone) that was generalizable to the clinic population and a sample that included behavioral survey data that are not captured in the EMR. New probabilistic methods, specifically Machine Learning, are being applied to clinical prediction models in HIV research [52]. While the data and technical requirements to use Machine Learning were outside the scope of this study, future studies could evaluate the feasibility of using this approach for prediction modeling in routine clinic settings [54, 55]. Finally, decisions about whether and how to use a prediction tool in routine practice depend on multiple considerations, including stakeholder engagement in intervention development, available services, and data systems [35] and qualitative and/or quantitative assessments of clinical usefulness [56–59].
Conclusions
In summary, our clinical prediction tools using surveys-plus-EMR or EMR-alone showed modest predictive ability to identify AYALWH at risk of loss to follow-up and would have limited use to improve clinical decision making in this population. Accurately predicting LTFU among AYALWH remains challenging, especially as national guidelines change about visit timing for AYALWH change. Regardless, these results offer insights for future provider decision-support interventions to reduce LTFU among AYALWH clients, specifically the importance of including standard mental health screening and peer support services with this population.
Acknowledgments
We thank young people who generously provided their time to participate in this research, study staff, facility manager and staff, and our community advisory board. We acknowledge the support of Impact Research & Development Organization (IRDO), University of Washington Center for Global Health Nursing and the Global Center for Integrated Health of Women, Adolescents, and Children (Global WACh), and the Kisumu and Homa Bay County Directors of Health.
References
- 1. UNAIDS. SEIZING THE MOMENT: GLOBAL AIDS UPDATE 2020 2020.
- 2. Kariminia A, Law M, Davies MA, Vinikoor M, Wools-Kaloustian K, Leroy V, et al. Mortality and losses to follow-up among adolescents living with HIV in the IeDEA global cohort collaboration. J Int AIDS Soc 2018;21:e25215. pmid:30548817
- 3. Zanoni BC, Archary M, Buchan S, Katz IT, Haberer JE. Systematic review and meta-analysis of the adolescent HIV continuum of care in South Africa: the Cresting Wave. BMJ Glob Health 2016;1:e000004. pmid:28588949
- 4. Mugisha V, Teasdale CA, Wang C, Lahuerta M, Nuwagaba-Biribonwoha H, Tayebwa E, et al. Determinants of mortality and loss to follow-up among adults enrolled in HIV care services in Rwanda. PLoS One 2014;9:e85774. pmid:24454931
- 5. National AIDS Control Council. Kenya HIV County Profiles. Nairobi: 2016.
- 6. Wilson KS, Mugo C, Moraa H, Onyango A, Nduati M, Inwani I, et al. Health provider training is associated with improved engagement in HIV care among adolescents and young adults in Kenya. AIDS 2019;33:1501–10. pmid:30932957
- 7. Ojwang’ V, Penner J, Blat C, Agot K, Bukusi E, Cohen C. Loss to follow-up among youth accessing outpatient HIV care and treatment services in Kisumu, Kenya. AIDS Care 2016;28. https://doi.org/10.1080/09540121.2015.1110234.
- 8. Crawford TN, Thornton A. Retention in Continuous Care and Sustained Viral Suppression. Journal of the International Association of Providers of AIDS Care (JIAPAC) 2017;16:42–7. pmid:27852944
- 9. Maina EK, Mureithi H, Adan AA, Muriuki J, Lwembe RM, Bukusi EA. Incidences and factors associated with viral suppression or rebound among HIV patients on combination antiretroviral therapy from three counties in Kenya. Int J Infect Dis 2020;97:151–8. pmid:32497804
- 10. Cherutich P, Kim AA, Kellogg TA, Sherr K, Waruru A, de Cock KM, et al. Detectable HIV Viral Load in Kenya: Data from a Population-Based Survey. PLoS One 2016;11:e0154318. pmid:27192052
- 11. UNAIDS. Global AIDS Strategy 2021–2026. Geneva, Switzerland: 2021.
- 12. Frijters EM, Hermans LE, Wensing AMJ, Devillé WLJM, Tempelman HA, de Wit JBF. Risk factors for loss to follow-up from antiretroviral therapy programmes in low-income and middle-income countries. AIDS 2020;34:1261–88. https://doi.org/10.1097/QAD.0000000000002523.
- 13. Pantelic M, Casale M, Cluver L, Toska E, Moshabela M. Multiple forms of discrimination and internalized stigma compromise retention in HIV care among adolescents: findings from a South African cohort. J Int AIDS Soc 2020;23:e25488. pmid:32438498
- 14. Zanoni BC, Sibaya T, Cairns C, Lammert S, Haberer JE. Higher retention and viral suppression with adolescent-focused HIV clinic in South Africa. PLoS One 2017;12:e0190260. pmid:29287088
- 15. Barr EA, Raybin JL, Dunlevy H, Abuogi L, Jones J. Transition From Pediatric and Adolescent HIV Care to Adult HIV Care and the Patient-Provider Relationship: A Qualitative Metasynthesis. J Assoc Nurses AIDS Care 2021. https://doi.org/10.1097/JNC.0000000000000239.
- 16. Tucker JD, Tso LS, Hall B, Ma Q, Beanland R, Best J, et al. Enhancing Public Health HIV Interventions: A Qualitative Meta-Synthesis and Systematic Review of Studies to Improve Linkage to Care, Adherence, and Retention. EBioMedicine 2017;17:163–71. pmid:28161401
- 17. Murray KR, Dulli LS, Ridgeway K, Dal Santo L, de Mora D, Olsen P, et al. Improving retention in HIV care among adolescents and adults in low- and middle-income countries: A systematic review of the literature. PLoS One 2017;12:e0184879. pmid:28961253
- 18. Casale M, Carlqvist A, Cluver L. Recent Interventions to Improve Retention in HIV Care and Adherence to Antiretroviral Treatment Among Adolescents and Youth: A Systematic Review. AIDS Patient Care STDS 2019;33:237–52. pmid:31166783
- 19. Bermudez LG, Ssewamala FM, Neilands TB, Lu L, Jennings L, Nakigozi G, et al. Does Economic Strengthening Improve Viral Suppression Among Adolescents Living with HIV? Results From a Cluster Randomized Trial in Uganda. AIDS Behav 2018;22:3763–72. pmid:29846836
- 20. Tozan Y, Sun S, Capasso A, Shu-Huah Wang J, Neilands TB, Bahar OS, et al. Evaluation of a savings-led family-based economic empowerment intervention for AIDS-affected adolescents in Uganda: A four-year follow-up on efficacy and cost-effectiveness. PLoS One 2019;14:e0226809. pmid:31891601
- 21. Ssewamala FM, Dvalishvili D, Mellins CA, Geng EH, Makumbi F, Neilands TB, et al. The long-term effects of a family based economic empowerment intervention (Suubi+Adherence) on suppression of HIV viral loads among adolescents living with HIV in southern Uganda: Findings from 5-year cluster randomized trial. PLoS One 2020;15:e0228370. pmid:32040523
- 22. MacKenzie RK, Lettow M, Gondwe C, Nyirongo J, Singano V, Banda V, et al. Greater retention in care among adolescents on antiretroviral treatment accessing “Teen Club” an adolescent‐centred differentiated care model compared with standard of care: a nested case–control study at a tertiary referral hospital in Malawi. J Int AIDS Soc 2017;20. https://doi.org/10.1002/jia2.25028.
- 23. Grimsrud A, Barnabas R v, Ehrenkranz P, Ford N. Evidence for scale up: the differentiated care research agenda. J Int AIDS Soc 2017;20:22024. pmid:28770588
- 24. NASCOP. Differentiated Care: Operational Guide. Nairobi, Kenya: 2017.
- 25. Oh B, Hwangbo S, Jung T, Min K, Lee C, Apio C, et al. Prediction models for clinical severity of COVID-19 patients using multi-center clinical data in Korea. J Med Internet Res 2021. https://doi.org/10.2196/25852.
- 26. Shin MY, Moon HS, Kwon IS, Park JH, Kim JS, Kang SH, et al. Development and Validation of a Risk Scoring Model for Early Prediction of Severe Colon Ischemia. Dig Dis Sci 2020. pmid:33242157
- 27. Balkus JE, Brown E, Palanee T, Nair G, Gafoor Z, Zhang J, et al. An Empiric HIV Risk Scoring Tool to Predict HIV-1 Acquisition in African Women. J Acquir Immune Defic Syndr 2016;72:333–43. pmid:26918545
- 28. Pintye J, Drake AL, Kinuthia J, Unger JA, Matemo D, Heffron RA, et al. A Risk Assessment Tool for Identifying Pregnant and Postpartum Women Who May Benefit From Preexposure Prophylaxis. Clin Infect Dis 2017;64:751–8. pmid:28034882
- 29. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010;21:128–38. pmid:20010215
- 30. Wand H, Guy R, Donovan B, McNulty A. Developing and validating a risk scoring tool for chlamydia infection among sexual health clinic attendees in Australia: a simple algorithm to identify those at high risk of chlamydia infection. BMJ Open 2011;1:e000005. pmid:22021721
- 31. Puttkammer N, Zeliadt S, Balan JG, Baseman J, Destiné R, Domerçant JW, et al. Development of an electronic medical record based alert for risk of HIV treatment failure in a low-resource setting. PLoS One 2014;9:e112261. pmid:25390044
- 32. Heffron R, Cohen CR, Ngure K, Bukusi E, Were E, Kiarie J, et al. A Scoring Tool to Identify East African HIV-1 Serodiscordant Partnerships with a High Likelihood of Pregnancy. PLoS One 2015;10:e0145515. pmid:26720412
- 33. Muttai H, Guyah B, Musingila P, Achia T, Miruka F, Wanjohi S, et al. Development and Validation of a Sociodemographic and Behavioral Characteristics-Based Risk-Score Algorithm for Targeting HIV Testing Among Adults in Kenya. AIDS Behav 2021;25:297–310. pmid:32651762
- 34. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users’ guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 2000;284:79–84. pmid:10872017
- 35. Muthee V, Bochner AF, Osterman A, Liku N, Akhwale W, Kwach J, et al. The impact of routine data quality assessments on electronic medical record data quality in Kenya. PLoS One 2018;13:e0195362. pmid:29668691
- 36. Kang’a S, Puttkammer N, Wanyee S, Kimanga D, Madrano J, Muthee V, et al. A national standards-based assessment on functionality of electronic medical records systems used in Kenyan public-Sector health facilities. Int J Med Inform 2017;97:68–75. pmid:27919397
- 37. Balkus JE, Brown ER, Palanee-Phillips T, Matovu Kiweewa F, Mgodi N, Naidoo L, et al. Performance of a Validated Risk Score to Predict HIV-1 Acquisition Among African Women Participating in a Trial of the Dapivirine Vaginal Ring. J Acquir Immune Defic Syndr 2018;77:e8–10. pmid:28961677
- 38. Rosenberg NE, Kudowa E, Price JT, Pettifor A, Bekker LG, Hosseinipour MC, et al. Identifying Adolescent Girls and Young Women at High Risk for HIV Acquisition: A Risk Assessment Tool From the Girl Power-Malawi Study. Sex Transm Dis 2020;47:760–6. pmid:33045165
- 39. van Wyk B, Kriel E, Mukumbang F. Retention in care for adolescents who were newly initiated on antiretroviral therapy in the Cape Metropole in South Africa. South Afr J HIV Med 2020;21:1077. pmid:32832112
- 40. McLeroy KR, Bibeau D, Steckler A, Glanz K. An Ecological Perspective on Health Promotion Programs. Health Educ Q 1988;15. pmid:3068205
- 41. Schmidt S, Thyen U, Herrmann-Garitz C, Bomba F, Muehlan H. The Youth Health Care measure-satisfaction, utilization, and needs (YHC-SUN)-development of a self-report version of the Child Health Care (CHC-SUN) proxy-measure. BMC Health Serv Res 2016;16. https://doi.org/10.1186/s12913-016-1419-1.
- 42. Vaishnavi S, Connor K, Davidson JRT. An abbreviated version of the Connor-Davidson Resilience Scale (CD-RISC), the CD-RISC2: psychometric properties and applications in psychopharmacological trials. Psychiatry Res 2007;152:293–7. pmid:17459488
- 43. Zimet GD, Powell SS, Farley GK, Werkman S, Berkoff KA. Psychometric Characteristics of the Multidimensional Scale of Perceived Social Support. J Pers Assess 1990;55. pmid:2280326
- 44. Bush K. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med 1998;158. pmid:9738608
- 45. Garcia-Moreno C, Jansen HA, Ellsberg M, Heise L, Watts CH. Prevalence of intimate partner violence: findings from the WHO multi-country study on women’s health and domestic violence. The Lancet 2006;368. https://doi.org/10.1016/S0140-6736(06)69523-8.
- 46.
World_Health_Organization. Interim WHO clinical staging of HVI/AIDS and HIV/AIDS case definitions for surveillance: African Region. Geneva, Switzerland: World Health Organization; 2005.
- 47. Moons KG, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 2012;98:683–90. pmid:22397945
- 48. Mburu M, Guzé MA, Ong’wen P, Okoko N, Moghadassi M, Cohen CR, et al. Evaluating the effectiveness of the HIV adolescent package of care (APOC) training on viral load suppression in Kenya. Public Health 2019;173:146–9. pmid:31310874
- 49. Mbengue MAS, Chasela C, Onoya D, Mboup S, Fox MP, Evans D. Clinical predictor score to identify patients at risk of poor viral load suppression at six months on antiretroviral therapy: results from a prospective cohort study in Johannesburg, South Africa. Clin Epidemiol 2019;11:359–73. pmid:31191029
- 50. Zanoni BC, Musinguzi N, Archary M, Sibaya T, Haberer JE. Development of a transition readiness score for adolescents living with perinatally-acquired HIV and transitioning to adult care. AIDS Behav 2022;26:3131–8. pmid:35362907
- 51. Zanoni BC, Archary M, Sibaya T, Musinguzi N, Kelley ME, McManus S, et al. Development and validation of the HIV adolescent readiness for transition scale (HARTS) in South Africa. J Int AIDS Soc 2021;24:e25767. pmid:34235876
- 52. Marcus JL, Sewell WC, Balzer LB, Krakower DS. Artificial Intelligence and Machine Learning for HIV Prevention: Emerging Approaches to Ending the Epidemic. Curr HIV/AIDS Rep. 2020;17(3):171–9. pmid:32347446
- 53. Puttkammer N, Simoni JM, Sandifer T, Chéry JM, Dervis W, Balan JG, et al. An EMR-Based Alert with Brief Provider-Led ART Adherence Counseling: Promising Results of the InfoPlus Adherence Pilot Study Among Haitian Adults with HIV Initiating ART. AIDS Behav 2020;24:3320–36. pmid:32715409
- 54. Stockman J, Friedman J, Sundberg J, Harris E, Bailey L. Predictive Analytics Using Machine Learning to Identify ART Clients at Health System Level at Greatest Risk of Treatment Interruption in Mozambique and Nigeria. J Acquir Immune Defic Syndr. 2022;90(2):154–60. pmid:35262514
- 55. Maskew M, Sharpey-Schafer K, De Voux L, Crompton T, Bor J, Rennick M, et al. Applying machine learning and predictive modeling to retention and viral suppression in South African HIV treatment cohorts. Sci Rep. 2022;12(1):12715. pmid:35882962
- 56. Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart 2012;98:691–8. pmid:22397946
- 57. Fox MP, Rosen S. Retention of Adult Patients on Antiretroviral Therapy in Low- and Middle-Income Countries: Systematic Review and Meta-analysis 2008–2013. J Acquir Immune Defic Syndr 2015;69:98–108. pmid:25942461
- 58. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014;35:1925–31. pmid:24898551
- 59. Richardson S, Khan S, McCullagh L, Kline M, Mann D, McGinn T. Healthcare provider perceptions of clinical prediction rules. BMJ Open 2015;5:e008461. pmid:26338684