Correcting Mortality for Loss to Follow-Up: A Nomogram Applied to Antiretroviral Treatment Programmes in Sub-Saharan Africa

Matthias Egger and colleagues present a nomogram and a web-based calculator to correct estimates of program-level mortality for loss to follow-up, for use in antiretroviral treatment programs.


Introduction
The World Health Organization (WHO) estimates that about 4 million people were receiving antiretroviral therapy (ART) in lowand middle-income countries by the end of 2008, with coverage reaching 42% of the estimated 9.5 million in need of ART [1]. Sub-Saharan Africa represented 70% of the estimated treatment need and 73% of the total number of people receiving treatment in low-and middle-income countries at the end of 2008 [1].
The provision of ART in resource-limited settings follows a public health approach, which is characterized by a limited number of regimens and the standardization of clinical and laboratory monitoring [2]. This approach has been shown to result in similar or superior adherence to therapy and similar virological response when compared to industrialized countries [3][4][5]. Loss of patients to follow-up and care is, however, an important problem in resource-limited settings: A systematic review of published retention rates in ART clinics in sub-Saharan Africa showed that the proportion of patients retained 2 years after starting therapy was approximately 60% [6]. Similarly, in a collaborative analysis of patients starting ART in 15 treatment programmes in Africa, Asia, and South America we found that 21% of patients were lost to follow-up 6 months after starting ART [7]. A systematic review and meta-analysis of studies tracing patients lost to follow-up found that these patients experience high mortality [8] compared to patients remaining in care [9].
The successful treatment of individual patients and the monitoring and evaluation of ART programmes both depend on regular and complete patient follow-up. Programmes with high rates of loss to follow-up and poor ascertainment of deaths in patients lost will underestimate mortality of all patients starting ART. For example, standard Kaplan-Meier survival analyses in which follow-up time in patients lost to follow-up is censored at the last visit will be biased because mortality of these patients is assumed to be identical to comparable patients remaining in care. Analyses restricted to patients remaining in care will also underestimate mortality among all patients who started ART. Biased estimates of programme-level mortality hamper the evaluation of single programmes and the comparison between different programmes and settings.
Nomograms are widely used in medicine [10]. They are graphs that allow the approximate graphical computation of a function; placing a line across its several scales immediately solves the formula [11]. We propose a nomogram and a web-based calculator to correct estimates of programme-level mortality for loss to follow-up. We illustrate its use in a case study from Kenya. We show how mortality among patients lost to follow-up can be predicted on the basis of studies that traced patients to ascertain their vital status. Finally, we apply these methods to 11 ART programmes in sub-Saharan Africa and compare uncorrected and corrected estimates of mortality at 1 year to assess the typical bias that is introduced when loss to follow-up is ignored.

Deriving the Nomogram
Mortality of all patients starting ART in a treatment programme over a defined time period is a weighted average of mortality among patients remaining in care and patients lost to follow-up. It depends on the percentage of patients lost, mortality among patients lost to follow-up, and mortality among patients not lost to follow-up. The mortality observed among patients remaining in care can be multiplied by a correction factor C to obtain an estimate of programme-level mortality that takes deaths among patients lost to follow-up into account. This correction factor can be obtained from a nomogram. The algebraic derivation of this nomogram is as follows: Let Dividing both sides of this equation by M NL , we obtain the correction factor which can be used to obtain M C for a given M NL from M C = M NL C. The nomogram (Figure 1) plots the ratio of the mortality among patients lost and not lost to follow-up (M L /M NL ) on lines defined by the proportion of patients lost to follow-up (r) which are used to read off correction factor C. The broken lines refer to the casestudy described in Box 1. Further details on calculations are provided in Text S1. Figures S1 and S2 provide clean versions of the nomogram. These are also available from http://www.iedeasa.org.

Two Ways of Estimating Mortality among Patients Lost to Follow-Up
The parameters r and M NL can be observed directly, but mortality among patients lost to follow-up must be estimated. This can be done in dedicated studies tracing patients lost to follow-up, for example by visiting the homes of patients lost to follow-up or by linking treatment programme with death registry data to ascertain the vital status of patients lost to follow-up [12][13][14][15]. This method will be henceforth be called ''tracing method.'' In many situations, however, data from dedicated tracing studies are not available. In this case, mortality among patients lost to follow-up may be predicted on the basis of published data from similar settings. We used Brinkhof and colleagues' systematic review and meta-regression analysis of studies tracing patients lost to follow-up [8]. The meta-regression analysis was based on a total of 15 studies in patients on ART from sub-Saharan Africa. There was an inverse relation between mortality among those lost to follow-up and the rate of loss to follow-up in the programme: The higher the rate of loss to follow-up the lower the mortality among those lost. Figure 2 shows the predicted mortality among patients lost to follow-up according to the percentage of patients lost in the programme, with 95% confidence intervals (CIs). The equation is as follows: where a = 0.57287 and b = 24.04409. Mortality among patients lost to follow-up can be calculated using the formula above or read off Figure 2, based on the proportion of patients lost to follow-up in the programme. This method will henceforth be called ''meta method.''

Allowing for Uncertainty
We used Monte Carlo simulations with 100,000 iterations to calculate 95% CIs for the corrected programme-level mortality (M C ). These simulations allow for uncertainty in the estimation of (i) mortality among patients remaining in care (M NL ); (ii) in the proportion of patients lost to follow-up (r); and (iii) of mortality among patients lost to follow-up (M L ). Further details are provided in Text S1.

Application to ART Programmes in Sub-Saharan Africa
The International epidemiological Databases to Evaluate AIDS (IeDEA, see http://www.iedea-hiv.org) is a collaborative network of HIV/AIDS treatment programmes in seven regions of the world, including North America, Asia and the Pacific, the Caribbean and Latin America, and four regions in sub-Saharan Africa (Central Africa, East Africa, West Africa, and Southern Africa). IeDEA and its predecessor, the ART in Lower Income Countries (ART-LINC) collaboration, have been described in detail elsewhere [16][17][18][19][20]. These collaborative networks were established to address clinical and operational research questions that require large patient numbers or many treatment programmes, for example to compare outcomes of ART between different settings, delivery modes, and types of monitoring.
We included 11 ART programmes from sub-Saharan Africa. No direct evidence on the mortality of patients lost to follow-up from tracing studies was available for these programmes, and mortality was therefore predicted using the meta method. All patients aged 16 years or older with complete data on sex and date of birth, and who were treatment naïve at start of combination ART were included in the analysis. Combination ART was defined as a minimum of three antiretroviral drugs from two drug classes. Advanced stage of disease was defined as WHO stages III or IV or Centers for Disease Control and Prevention (CDC) clinical stage C. Measurements of laboratory values closest to the starting date of ART (within 6 months before up to one week after the date of starting ART) were taken as the baseline levels. The data collected at participating sites were cleaned, merged, and analysed centrally. At all sites, institutional review boards had approved the collection and transfer of data.
For each programme we determined the proportion of patients lost to follow-up at 1 year (r). A patient was considered lost to follow-up if the last visit was more than 9 months before the closure date for that site, with the closure date defined as the most recent visit date recorded in the database. This allowed for time during which a patient could have returned for a visit. Only patients who potentially had 9 months of follow-up were included in the calculations of r. We used Kaplan-Meier methods to estimate mortality (with 95% CI) at 1 year for patients remaining in care (M NL ).

Web Calculator
The calculations of the nomogram, the prediction of mortality among patients lost to follow-up based on the meta-regression analysis described above, and the calculation of 95% CIs have been implemented on a dedicated website at http://www.iedea-sa. org.

Case Study from Western Kenya Using the Tracing Method
Box 1 and Figure 1 illustrate the correction of naïve estimates of programme-level mortality in six simple steps. The data are from the Academic Model Providing Access to Healthcare (AMPATH), a large ART programme in western Kenya [21,22]. Patients who miss scheduled appointments are actively traced by outreach teams, and data from the outreach programme were used to estimate mortality among patients lost to follow-up. The data are based on a previous detailed analysis of two AMPATH sites [23]. The uncorrected estimate (M U ) of mortality at these two sites for the period between 1 January 2005 and 31 January 2007 was 1.7% (95% CI 1.3%-2.0%), the estimate for mortality among patients not lost to follow-up was 2.2% (95% CI 1.8%-16.6%), and the corrected estimate using the nomogram was 9.5% (Box 1). Entering the data into the web calculator at http:// www.iedea-sa.org/ gives a more precise corrected estimate of programme mortality of 9.4%, with 95% CI 8.1%-10.9%. Of note, the results from the nomogram are similar to the corrected estimates that were obtained for these AMPATH sites by Yiannoutsos et al. using more complex statistical methods [23,24].

Correction of Mortality in ART Programmes in Sub-Saharan Africa Using the Meta Method
A total of 24,257 patients from 11 ART programmes in 10 countries (Botswana, Côte d'Ivoire, Kenya, Malawi, Rwanda, Senegal, South Africa, Uganda, Zambia, Zimbabwe) were included: 1,363 deaths were observed during the first year of ART. Table 1 shows the characteristics of the 11 programmes. Four programmes were located in South Africa and one programme had sites in six different countries. The number of patients treated at each site ranged from 369 to 4,705 patients. Eight sites were public (government) clinics offering ART free of charge, two were run by a nongovernmental organisation offering free ART, one was a research site offering free ART, and one site was a private clinic operating on a fee-for-service basis. Median

Box 1. Example from Western Kenya Using Tracing Method
The Academic Model Providing Access to Healthcare (AMPATH) is a large ART programme in western Kenya [21,22]. AMPATH is based on a partnership between the Moi University School of Medicine in Eldoret and Indiana University School of Medicine in Indianapolis, Indiana, United States of America. AMPATH provides HIV care and treatment to over 70,000 adults and children living with HIV/AIDS in 18 clinics throughout western Kenya. Patients are managed according to National Kenyan protocols, which are consistent with WHO guidelines. AMPATH undertakes active outreach to patients who miss scheduled appointments (called lost to follow-up in this article). A locator card is completed for all patients enrolling in the programme, which includes contact information and a map to the patient's residence. The results of the outreach programme have been described for the Moi Teaching and Referral Hospital in the city of Eldoret and the rural health centre in Mosoriot [23]. Not all patients lost to follow-up could be located and mortality among those who could not be found might have been even higher than in those who were successfully traced. Overall mortality among patients lost to follow-up may thus have been higher than 20%. The effect of assuming a higher mortality among patients lost to follow-up can be examined in a sensitivity analysis: repeating the steps assuming that 25% or 30% of patients lost to follow-up had died results in estimates of corrected programme-level mortality of 11.4% and 13.5%. Clearly, deaths in patients lost to follow-up are an important issue when estimating programme-level mortality in the AMPATH programme.
age across all programmes was 35 years (interquartile range 30-41 years) and 16,018 patients (66.0%) were women. The median CD4 cell count at baseline ranged from 83 to 156 cells/mL across programmes. Table 2 lists the uncorrected estimates (M U ) of programme-level mortality at 1 year (which do not consider mortality among patients lost to follow-up), the proportion of patients lost to follow-up at 1 year (r), the estimated 1-year mortality among patients remaining in care (M NL ), the predicted mortality among patients lost to follow-up (M L , obtained from the meta-regression model of Brinkhof et al. [8]), correction factor C and the corrected estimates of programmelevel mortality at 1 year (M C ), which take mortality among patients lost to follow-up into account, with 95% CIs calculated as described above. Figure 3 shows the nomogram populated with the data from the 12 antiretroviral treatment programmes.
The uncorrected estimates of programme-level mortality at 1 year ranged from 1.3% to 11.0%, mortality at 1 year among patients retained in care from 1.4% to 12.0%; loss to follow-up at 1 year from 2.8% to 28.7%; and correction factor C from 1.19 to 8.04. The corrected estimates of programme-level mortality ranged from 4.4% to 18.8%. Figure 4 shows a scatter plot of the uncorrected estimates of programme-level mortality (M U ) against the corrected estimates (M C ): the absolute difference between uncorrected and corrected mortality at 1 year was relatively small (1.7%-3.6%) in seven programmes, but more substantial in the remaining four programmes. The largest difference in mortality was 9.8%, in a programme with 28.7% of patients lost to follow-up at 1 year. The uncorrected estimate of programme-level mortality for this programme was 1.4% and the corrected estimate 11.2%.

Discussion
Regular and complete patient follow-up is essential both for the care of individual patients and the monitoring and evaluation of outcomes of ART programmes. Individual treatment decisions can then be made in the light of clinical and laboratory results and the rate of complications and mortality can be accurately estimated at the programme level. Loss to follow-up is, however, an important problem in ART programmes in resource-limited settings [6,7,18,25], and poor ascertainment of deaths in patients lost to follow-up may mean that programmelevel mortality, i.e. mortality of all patients starting ART, is underestimated [8,23]. Previous analyses of treatment pro-grammes have generally censored follow-up time at the last visit to the clinic, and patients lost to follow-up therefore contributed follow-up time but no deaths [26][27][28] (here referred to as M U ).  We developed a simple nomogram that allows programme managers to read off a correction factor for a range of plausible mortality rates among patients lost to follow-up. This factor can then be used to assess to what extent the mortality observed among patients retained in care underestimates mortality at the programme level. We applied the nomogram method to ART programmes in sub-Saharan Africa to estimate programme-level mortality at 1 year after starting ART and found that the bias was modest in many programmes, because loss to follow-up was relatively low. It is clear from the shape of the nomogram that if the proportion of patients lost is low (say, below 10%), the correction factor will not be greatly affected by different assumptions for mortality among patients lost to follow-up. In programmes with fewer than 10% of patients lost, the mortality observed among patients retained in care will thus generally provide a reasonable estimate, which will underestimate programme-level mortality by only a few percentage points. Conversely, if a large proportion of patients are lost, as in the case study from western Kenya (Box 1), the bias will be substantial even if the ratio of mortality between patients lost and not lost is relatively low, and the amount of bias will increase steeply with higher ratios.
Mortality among patients lost to follow-up is high: Brinkhof et al. recently reviewed studies that traced patients who became lost to ART programmes in resource-limited settings [8]. The vital status of about two-thirds of patients could be ascertained, and among these many had died: in ART programmes from sub-Saharan Africa the combined mortality from meta-analysis was 46% (95% CI 39%-54%) [8]. Patients were often lost in the first few months of ART and died soon thereafter [8].These findings are in accordance with previous analyses from the ART-LINC collaboration [27] and other treatment programmes, for example the Médecins Sans Frontières programmes in Malawi [28] and South Africa [29], or the ART programme in Mbarara, Uganda [25]. Of note, the percentage of patients lost to follow-up in these programmes was associated with estimated mortality rates in the patients lost: the estimated mortality declined from around 60% to 20% as the percentage of patients lost to the programme increased from 5% to 50%. As discussed in detail elsewhere [8], those lost to follow-up in programmes with high rates of loss to follow-up might include many low-risk patients who self-transferred to another programme, for example because of a more convenient location of the new clinic, to avoid stigma or due to work-related reasons. The negative association between the proportion lost and mortality among patients lost to follow-up will attenuate the effect of a high rate of loss to follow-up on the correction factor C.
In the absence of direct evidence, the regression analysis of Brinkhof et al. [8] can be used to predict mortality among patients lost to follow-up for a given programme. The model will provide a sensible range of estimates of mortality, which can then be used in analyses to adjust overall mortality. We used this approach when applying the nomogram to the 11 ART programmes from sub-Saharan Africa. Statistical uncertainty is, however, substantial, as documented by the wide confidence intervals around the predicted mortality among patients lost to follow-up (the grey area in Figure 2), and this translated into wide confidence intervals around the corrected estimates of programme-level mortality (see last column of Table 2).
In addition to the statistical uncertainty, the applicability of the results from the regression analysis must also be considered: The studies examined loss to follow-up and mortality in the first months after starting ART, and at present the regression model should therefore not be used to estimate mortality among patients lost to follow-up later on, for example in the second or third year of ART.
The different determinants of loss to follow-up, including ''silent transfer'' to another programme (without notifying the programme where ART was initiated), financial constraints, and improving or deteriorating health will change with time since starting ART, and mortality among patients lost to programmes will change accordingly. In the present study we focused on loss to follow-up in the first year of ART, and the use of data from the review and meta-regression analysis was therefore appropriate. We stress that patterns of loss to follow-up and associated mortality may also change with calendar years, for example with increasing CD4 cell counts at the start of ART [17].
To obtain accurate estimates of mortality among all patients starting ART, programmes should therefore make an effort to trace patients lost to follow-up and ascertain their vital status. The results from these efforts should be made available to the scientific community so that the regression model can be updated and improved, for the benefit of programmes in settings where tracing of patients lost is not possible. Also note that results of studies tracing patients lost to follow-up can be used to directly correct estimates of mortality in the programme, for example by using double-sampling designs or weighted Kaplan-Meier methods [23,30]. These methods make strong assumptions, however, and some require expert statisticians.
The nomogram method has the important advantage of being simple and adapted to the field. Among the data used in the nomogram the estimated or assumed mortality among patients lost to follow-up (M L ) will always be associated with the greatest degree of statistical uncertainty and the greatest risk of bias. The estimates of mortality among patients remaining in care (M NL ) might, however, also be biased: some patients will not meet criteria for loss to follow-up and their follow-up will be censored at the last visit. Mortality rates might be higher in these patients than in otherwise identical patients whose followup time was not censored. In other words, we cannot exclude the possibility that some degree of ''informative censoring'' might introduce some bias in our estimate of mortality among patients retained in care. Finally, the nomogram can provide only a sensitivity analysis without formal estimates of uncertainty. We overcame this by creating a web calculator with a user-friendly interface, which calculates 95% CI taking into account the statistical uncertainty in the input parameters of the nomogram.
ART programmes should strive to prevent loss to follow-up: interventions that prevent loss to follow-up in resource-limited settings can improve survival and are cost-effective by international criteria [31]. For example, outreach teams that routinely trace patients, combined with other measures, substantially reduce loss to follow-up [18]. Financial constraints are a common reason for not returning to the clinic [8], and mortality in programmes that charge user fees has been shown to be higher than in those offering free treatment [27]. Decentralization of services, task shifting to lay care providers, longer drug refill periods for stable patients, and provision of transport vouchers for those in need are some of the strategies that could address this issue. Strengthening of referral systems and regular exchange of information between clinics, together with patient education, could increase the recording of transfers and ensure continuity of care. Finally, when assessing outcomes, programmes should routinely report both mortality among patients retained in care at a given point in time and the proportion of patients lost to follow-up at that time. A simple nomogram can then be used to estimate mortality among all patients who started ART. Figure S1 Nomogram for obtaining correction factors to adjust programme-level mortality estimates, based on the observed mortality among patients not lost to follow-up (LTFU), the observed proportion of patients lost and an estimate of mortality among patients lost. Horizontal axis shows ratios from 1 to 15. Found at: doi:10.1371/journal.pmed.1000390.s001 (0.13 MB TIF) Figure S2 Nomogram for obtaining correction factors to adjust programme-level mortality estimates, based on the observed mortality among patients not lost to follow-up (LTFU), the observed proportion of patients lost and an estimate of mortality among patients lost. Horizontal axis shows ratios from 1 to 50.

Editors' Summary
Background. AIDS has killed more than 25 million people since 1981 and about 33 million people (30 million of them in low-and middle-income countries) are now infected with HIV, which causes AIDS. HIV destroys immune system cells, leaving infected individuals susceptible to other infections. Early in the AIDS epidemic, most HIV-infected people died within 10 years of infection. Then, in 1996, highly active antiretroviral therapy (ART) became available. For people living in affluent, developed countries, HIV/AIDS became a chronic condition, but for people living in low-and middleincome countries, ART was prohibitively expensive and HIV/ AIDS remained a fatal illness. In 2003, this situation was declared a global health emergency and governments, international agencies, and funding bodies began to implement plans to increase ART coverage in developing countries. By the end of 2009, 5.25 million of the 14.6 million people in low-and middle-income countries who needed ART (36%) were receiving it.
Why Was This Study Done? ART program managers in developing countries need to monitor the effectiveness of their programs to ensure that their limited resources are used wisely. In particular, they need accurate records of the death (mortality) rates in their programs. However, in resource-limited countries, many patients drop out of ART programs. In sub-Saharan Africa, for example, only about 60% of patients are retained in ART programs 2 years after starting therapy. In many programs, it is not known how many of the patients lost to follow-up subsequently die, but it is known that mortality is higher among these patients than among those who remain in care. Thus, in programs with high dropout rates and poor ascertainment of death in patients lost to follow-up, estimates of the mortality of all patients starting ART are underestimates. In this study, the researchers develop a simple nomogram (a graphical method for finding the value of a third variable from the values of two other variables) to correct estimates of program-level mortality for loss to follow-up.
What Did the Researchers Do and Find? The researchers' nomogram uses the percentage of patients lost to follow and the estimated ratio of mortality between patients lost and not lost to follow-up to provide a correction factor that converts mortality among patients remaining in care to mortality among all the patients in a program. The researchers first applied their nomogram to the Academic Model Providing Access to Healthcare (AMPATH), a large ART program in Kenya. They used data collected by outreach teams to estimate mortality among the 40.5% of patients lost to follow-up at two AMPATH sites between 1 January 2005 and 31 January 2007. The uncorrected estimate of mortality over this period was 2.8%, whereas the corrected estimate obtained using the nomogram was 9.4%. The researchers then applied their nomogram to 11 other African ART programs. This time, the researchers used a statistical model to provide estimates of mortality among patients lost to follow-up. Mortality among patients retained in care was 1.4% to 12.0% at 1 year; loss to follow-up ranged from 2.8% to 28.7%. The nomogram provided a correction value for mortality among all patients in the ART program of 1.2 to 8.0, which resulted in absolute differences between uncorrected and corrected mortality of 1.6% to 9.8%. The largest absolute difference was in the program with the largest percentage of patients lost to follow-up.
What Do These Findings Mean? These findings indicate that, in ART programs where a large percentage of patients are lost to follow-up, program-level mortality estimates based on the mortality among patients retained in the program can be substantial underestimates. This bias needs to be taken into account when comparing the effectiveness of different programs, so the researchers recommend that all programs routinely report mortality among patients retained in care and the proportion of patients lost to follow-up. The nomogram developed by the researchers can then be used to estimate mortality among all patients who started ART using a range of plausible mortality rates among patients lost to follow-up. To help program managers make use of the nomogram, the researchers provide a user-friendly web calculator based on the nomogram on the International epidemiologic Databases to Evaluate AIDS (IeDEA) Southern Africa website. Aids (IeDEA) Southern Africa website provides access to a calculator for correcting overall program-specific mortality for loss to follow-up