Systematic Review of the Performance of Rapid Rifampicin Resistance Testing for Drug-Resistant Tuberculosis

Introduction Rapid tests for rifampicin resistance may be useful for identifying isolates at high risk of drug resistance, including multidrug-resistant TB (MDR-TB). However, choice of diagnostic test and prevalence of rifampicin resistance may both impact a diagnostic strategy for identifying drug resistant-TB. We performed a systematic review to evaluate the performance of WHO-endorsed rapid tests for rifampicin resistance detection. Methods We searched MEDLINE, Embase and the Cochrane Library through January 1, 2012. For each rapid test, we determined pooled sensitivity and specificity estimates using a hierarchical random effects model. Predictive values of the tests were determined at different prevalence rates of rifampicin resistance and MDR-TB. Results We identified 60 publications involving six different tests (INNO-LiPA Rif. TB assay, Genotype MTBDR assay, Genotype MTBDRplus assay, Colorimetric Redox Indicator (CRI) assay, Nitrate Reductase Assay (NRA) and MODS tests): for all tests, negative predictive values were high when rifampicin resistance prevalence was ≤ 30%. However, positive predictive values were considerably reduced for the INNO-LiPA Rif. TB assay, the MTBDRplus assay and MODS when rifampicin resistance prevalence was < 5%. Limitations In many studies, it was unclear whether patient selection or index test performance could have introduced bias. In addition, we were unable to evaluate critical concentration thresholds for the colorimetric tests. Discussion Rapid tests for rifampicin resistance alone cannot accurately predict rifampicin resistance or MDR-TB in areas with a low prevalence of rifampicin resistance. However, in areas with a high prevalence of rifampicin resistance and MDR-TB, these tests may be a valuable component of an MDR-TB management strategy.


Introduction
Mycobacterium tuberculosis (TB) remains a major causes of death and disability globally [1]. Multidrug-resistant TB (MDR-TB -defined as resistance to at least isoniazid and rifampicin), threatens to undermine TB control efforts [2]. Management of MDR-TB requires effective case detection, treatment, prevention, surveillance and monitoring [3]. However, the diagnosis of MDR-TB has remained a challenge in most countries, and less than 5% of all TB patients were tested for MDR-TB in 2010 [1].
Expanding capacity to diagnose drug-resistant TB and decreasing time to diagnosis remain priorities for global TB control. Rapid tests for rifampicin resistance may quickly identify TB patients who are likely to have MDR-TB, and the use of these tests has been recommended in high risk MDR-TB settings [4]. However, there may be significant variability in the performance of different diagnostics [5]. In addition, differences in the population prevalence of rifampicin resistance and MDR-TB may have a considerable impact on the predictive value of a strategy for rapid diagnosis of rifampicin resistance and MDR-TB.
A prior systematic review evaluated 151 published studies to determine the predictive value of rapid rifampicin resistance testing for MDR-TB [6]. This review searched only one electronic database (Medline) through 6 September 2008 for articles in English and did not use the hierarchical methods for meta-analysis recommended by the Cochrane Diagnostic Test Accuracy Working Group [7]. Several other systematic reviews have also evaluated non-commercial phenotypic and genotypic methods for rifampicin resistance but these reviews have not generated predictive values for rifampicin resistance testing as a surrogate for MDR-TB [5,[8][9][10][11][12][13].
We performed a systematic review and meta-analysis to estimate the diagnostic accuracy (sensitivity and specificity) of WHO-endorsed rapid phenotypic and genotypic tests for rifampicin resistance. We reasoned that if rapid tests for rifampicin resistance detection alone could accurately predict rifampicin resistance and the conditional probability of MDR-TB is high, the use of such tests would provide a simple, and effective method for MDR-TB case identification.

Methods
We followed guidelines for systematic reviews of diagnostic test accuracy recommended by the Cochrane Collaboration Working Group [7] and for the preferred reporting items for systematic reviews and meta-analysis (PRISMA) [14]. Because this was a systematic review of published data, no approval through the University of Washington Institutional Review Board was necessary.

Types of studies
We included studies in populations that compared the results of the index test for rifampicin resistance with the reference standard (see definitions below). We included cross-sectional studies and cohort studies of diagnostic accuracy. We excluded studies that employed a case-control design and studies that were reported in correspondence and conference abstracts.

Participants
We included adults and children suspected or confirmed as having TB disease, from all settings and countries. We included all direct specimens and culture isolates from humans suspected or confirmed as having M. TB.

Index tests
We included studies that evaluated the following index tests for rifampicin resistance I. Genotypic tests.
• We excluded studies that performed direct evaluation of sputum samples by the CRI methods, as the WHO does not recommend this strategy [15]. We excluded the Xpert MTB/RIF test (Cepheid, Sunnyvale, CA, USA), as a separate systematic review of this test was performed in parallel to our review [16].

Reference Standards
We included studies that used one of the following reference standards for rifampicin resistance testing I. Solid Culture.
• Löwenstein-Jensen agar using the proportion, absolute concentration, or resistance ratio method • Middlebrook 7H10 or 7H11 agar medium using the proportion, absolute concentration, or resistance ratio method II. Liquid Culture.

Outcomes
Studies that reported data from which we could extract true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) were included. Sensitivity is the percentage of rifampicin resistant results correctly identified as rifampicin resistant when compared with the reference standard. Specificity is the percentage of rifampicin susceptible results correctly identified as rifampicin susceptible when compared with the reference standard. Positive predictive value (PPV) is the proportion of positive results that are true positives at a given prevalence of rifampicin resistance; negative predictive value (NPV) is the proportion of negative results that are true negatives at a given prevalence of rifampicin resistance.

Search methods
We searched Medline, Embase and the Cochrane Library on 1 January 2012, using the search strategy described in the supplemental materials. We attempted to identify all studies published in English, French, or Spanish. If a prior index test systematic review had been published, we updated the systematic reviews to include studies published since prior reviews were completed. Appendix S1 in file S2 shows a representative search strategy.

Selection of studies
Two researchers independently screened the titles and abstracts identified by electronic literature searching. Any citation identified by either of the review authors during this screen (screen 1) was selected for full-text review. Full-text papers (screen 2) were then reviewed for study eligibility using predefined inclusion and exclusion criteria. In screen 2, any discrepancies were resolved by discussion between the review authors or if they were unable to resolve, by decision of a third review author (DH). A list of excluded studies and their reasons for exclusion was maintained.

Data extraction and management
Two researchers extracted data from each study using a piloted data extraction form. Based on the pilot, the extraction form was finalized. Two review authors then independently extracted data on the following characteristics: author, year of study, study design, patient/specimen selection, reference standard, index test, country of testing, method (indirect or direct), and the number of TP, TN, FP, and FN.

Assessment of methodological quality
Two researchers independently assessed study quality with the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) version 2 [17]. As recommended, items for each domain in the QUADAS-2 list were scored as low, high, or unclear concern of risk of bias or applicability. Appendix S3 in file S2 describes the criteria that needed to be met for each study to be rated as low, high, or unclear for each of the QUADAS-2 items.

Statistical analysis and data synthesis
Data were entered into excel (2007, Microsoft Co, Redmond, WA). Analyses were performed using STATA/IC (version 11.0) [18]. Test sensitivity and specificity along with the 95% CI were calculated using exact methods and forest plots were generated using RevMan (version 5.1) [19]. We determined pooled sensitivity and specificity estimates using hierarchical random effects models [20] and performed these metaanalyses with the user-written command metandi in STATA/IC (version 11.0). In cases where fewer than four studies were identified, sensitivity and specificity estimates were recorded for individual studies but a meta-analysis was not performed. Predictive values were estimated under three hypothetical scenarios of rifampicin resistance prevalence (3%, 15%, and 30%) and determined mathematically as described in Table 1. Separate analyses were performed for culture isolates and clinical specimens. We also generated continuous estimates of index test predictive value across a continuum of rifampicin resistance values [21].

Assessment of reporting bias
Data included in this review did not allow for formal assessment of publication bias using methods such as funnel plots or regression tests because such techniques have not been found to be helpful for diagnostic studies [7,22].

GRADE
The quality of evidence was assessed using the GRADE approach [23]. The GRADE approach defines the quality of a body of evidence as the extent to which one can be confident that an estimate of effect or association is close to the quantity of specific interest. Quality of a body of evidence involves consideration of within-study risk of bias (methodological quality), directness of evidence, heterogeneity, precision of effect estimates and risk of publication bias. The quality rating across studies has four levels: high, moderate, low or very low. The GRADE Profiler software (version 3.6) was used for performing the GRADE analyses [24].

Search results
Our electronic search resulted in 597 unique citations. Fifty additional citations were identified from reviewing bibliographies of previous systematic reviews [5,[8][9][10][11][12][13]. Finally, 167 articles were retrieved for full-text review and 60 papers were determined to meet eligibility criteria and included in the systematic review. Because some papers included more than one index test, there were 62 datasets (hereafter referred to as studies) involving 9821 participants ( Figure 1).

Study characteristics
Studies were conducted in a variety of settings and with diverse populations. Of the total 62 studies, 19

Methodological quality of included studies
Studies were assessed with QUADAS 2, all studies had a perfect score for flow and timing. For the domain concerning patient selection, approximately 8% of studies were considered to be at high risk of bias and 42% of studies at unknown risk of bias because these studies lacked consecutive or random selection of patients/samples, used a case-control study design, or did not report this information. For the domain concerning the index test, only 27% of studies were considered to be at low risk of bias because the study reported blinding of the index test result and the reference standard result. The domain 'applicability of patient selection' addresses concerns that the included patients/samples and setting do not match the review question. Of the included studies, 90% enrolled patients suspected of having TB or drug resistant TB and had low concern of bias in the applicability domain ( Figure S1 in File S1).

Genotypic tests for rifampicin resistance
Four studies, including 947 participants, were included in the analysis of the INNO-LiPA Rif. TB assay. Pooled sensitivity for rifampicin resistance was 94.1% (95% CI 86.5, 97.6) and pooled specificity was 98.8% (95% CI 93.8, 99.8) (table 3). Twelve studies, including 3337 participants, were included in the analysis of the MTBDRplus assay. Figure S2, figure S3 and figure S4 in File S1 show forest plots of sensitivity and   Figure 2 and figure 3 show positive and negative predictive values for each test along a continuum of rifampicin resistance prevalence rates in a population.  Figure S5, figure S6 and figure S7 in File S1 show forest plots of sensitivity and specificity for included studies. Table 4 shows performance of MODS and NRA when used only on direct or indirect samples. Data on indeterminate test results were documented in the minority of studies. However, for NRA, 5 studies reported indeterminate results in 6% of tested samples. No information on the number of indeterminate test results was documented for the MODS assay or the CRI assay.
Each test was then evaluated for predictive value assuming a rifampicin resistance prevalence of 3%, 15% and 30% in a hypothetical cohort of 1000 TB cases suspected of having MDR-TB (table 5 & table S1 in File S1). Similar to genotypic tests, all tests showed high NPVs (greater than 95%) across the range of prevalence rates. The PPV of the MODS assay was less than 90% at a prevalence of 3% (79.8%; 95% CI 35.5, 91.5); the PPV was greater than 90% across all prevalence rates for NRA and the CRI assay. Figure 2 and figure 3 show positive and negative predictive values for each test along a continuum of rifampicin resistance prevalence rates in a population.

Discussion
In this systematic review and meta-analysis we evaluated the performance of WHO-endorsed rapid diagnostic tests for rifampicin resistance in comparison to conventional DST by solid or liquid culture for the detection of rifampicin resistant TB. Although we found that most of these tests perform well as a rapid marker of rifampicin resistance when prevalence is high, variability in population prevalence of rifampicin resistance and MDR-TB limited the ability of all of these tests to predict rifampicin resistance in populations with low rates of drug resistance.
These data are in agreement with estimates of sensitivity and specificity found in other systematic reviews on rapid tests for rifampicin resistance [5,6,[8][9][10][11]. Because we aimed to include studies that employed ideal design for evaluation of test accuracy, we excluded case-control studies. This meant there were insufficient studies that met our inclusion criteria to evaluate performance of the MTBDR assay. In addition, the Xpert MTB/RIF test was excluded from our review, because other systematic reviews were currently reviewing performance of this test. However, other estimates of the Xpert MTB/RIF test suggest similar sensitivity and specificity, and most likely similar performance in WHO populations [16,25,26].
In light of the rapid uptake of the Xpert MTB/RIF assay in many areas, and the relatively low rates of rifampicin resistance and MDR-TB in many of the high burden TB countries, it is important to devise a strategy for improved identification of MDR-TB in low prevalence areas. In a recent review of the Xpert MTB/Rif assay, high rates of false positive rifampicin resistance test results were found in a hypothetical cohort of 1000 individuals suspected of having MDR-TB when population prevalence rates of MDR-TB were 2% [16]. One strategy for improving diagnosis of rifampicin resistance in such a cohort might be to perform a second confirmatory test for rifampicin resistance if the Xpert MTB/Rif assay indicates rifampicin resistance. Using data from a recent review, we can estimate that the Xpert MTB/Rif assay has a sensitivity of 94% and a specificity of 98% [16]. In our review, we found the MTBDRplus assay to have the lowest pooled specificity (98%). The specificity of a positive Xpert MTB/Rif assay result for rifampicin resistance AND a positive result for rifampicin resistance by MTBDRplus assay could be calculated using the following equation:    Even if the prevalence of resistance was 2%, the positive predictive value of combined positive resistance testing in this setting would be 98.1% and the false positive rate would be very low (less than 2% of all positive results). Although clinical studies need to be performed to evaluate such a strategy, combined testing may be an effective way to quickly and accurately diagnose drug resistant TB.
In areas where prevalence rates of rifampicin resistance exceed 30%, these tests continued to show very good negative predictive values for rifampicin resistance and, presumably, for MDR-TB. However, we found positive predictive values to be low at prevalence rates of 15% and 3% for rifampicin resistance. This would result in the high occurrence of false positive tests at current WHO estimates of rifampicin resistance among all new TB cases. When considering test performance in a hypothetical cohort of TB cases, we found that only NRA and CRI had a positive predictive value greater than 90% when prevalence of resistance was 3%. This is in agreement with other published data evaluating the predictive value of index tests for rifampicin resistance [6].
We compared studies that evaluated only direct or indirect samples, and found similar test sensitivity and specificity for the MTBDRplus assay and NRA. We had insufficient data to perform this sub-group analysis for the INNO-LiPA Rif. TB assay or the MODS assay. In addition, although the NRA and CRI methods were observed to perform best in the included studies, GRADE evaluation of the included studies for these tests found very serious limitations in blinding and study design, which may have had a significant effect on sensitivity and specificity estimates.
If one considers how these tests would perform in identifying MDR-TB in TB populations, there are a number of important issues to account for. First, we only evaluated test performance for rifampicin resistance and presumed a strong correlation between rifampicin resistance and MDR-TB. In reality, the conditional probability that a TB case has MDR-TB when rifampicin resistance is present will vary by setting. Others have reported that this conditional probability may be low in some settings, although updated estimates are necessary [27]. In our review, the conditional probability of MDR-TB was 93.6%. Second, when considering global estimates of rifampicin resistance (6%) and MDR-TB (5.1%) our tests PPV suggest there will be a high likelihood of false positive test rests in many low prevalence settings (with false positives accounting for 16-28% of all positive test results at a prevalence of resistance at 5.1%). Among the 22 high burden TB countries which account for 80% of the world's burden of TB, the WHO estimates only Afghanistan, China, and the Russian Federation have MDR-TB prevalence rates above this level. This suggests that these tests will perform best as a surrogate test for MDR-TB in targeted populations with higher rates of drug resistance, and studies which further identify patients at high-risk for drug resistance are needed to provide more accurate results.
Overall, the decision of how to use best use rapid tests for rifampicin resistance in an MDR-TB treatment program remains unclear. When considering a positive rapid rifampicin resistance test result in a TB patient, results from conventional drug sensitivity testing (DST) via solid or liquid culture will likely play a large role in identifying the ideal treatment regimen. While awaiting DST results in a patient with a high risk of MDR-TB or a high risk of treatment failure, use of an MDR-TB regimen with isoniazid may be appropriate if resources permit. However, further studies are needed which describe the comparative costs and outcome benefits of such a strategy. In the interim, our review adds further support to the current WHO recommendation that all patient's who have rapid drug resistance testing also undergo full phenotypic DST.
This review has several strengths, including a broad search strategy and inclusion of papers in multiple languages. We used rigorous statistical methods in our pooled estimates of sensitivity and specificity. In addition, we limited study design to include only cross sectional studies, providing a better estimate of the diagnostic accuracy of each index test when compared to the reference standard.
This review also has several limitations. With regards to the MTBDRplus assay, many manuscripts did not specify whether version 1 or version 2 of the assay was used, and this may have affected specificity and sensitivity estimates. Also, only the minority of studies recorded information on indeterminate test results. High rates of indeterminate results could significantly affect test utility even if test performance is excellent. In many studies we were unclear whether patient selection or index test performance could have introduced bias. Ideally diagnostic tests should enroll a consecutive or random sample of eligible patients and blind the operator of the index test to reference standard results. We also did not evaluate critical concentration thresholds for the colorimetric test studies and these results should be interpreted with caution. We were unable to identify many studies that performed a direct comparison of index tests for rifampicin resistance. For this reason, the GRADE quality of the studies was low or very low in most cases, with only MDRTBplus and MODS being graded as moderate quality of evidence (table S2 and table S3 in File S1).

Conclusions
This systematic review updates previous estimates on the performance of rapid tests for identifying rifampicin resistance. It describes limitations in test PPV for rifampicin resistance and MDR-TB in the WHO high burden TB countries. Further studies are needed to better describe and compare test performance as a predictor of rifampicin resistance and MDR-TB and describe the costs and benefits of a rapid strategy for identifying and treating drug resistance.

Supporting Information
Checklist S1. PRISMA checklist. (DOC) File S1. Figure S1, Assessment of study quality. Figure S2, Forest plot: Sensitivity and Specificity of the INNO LiPA. Figure  S3, Forest plot: Sensitivity and Specificity of the MTBDR assay. Figure S4, Forest plot: Sensitivity and Specificity of the MTBDRplus assay. Figure S5, Forest plot: Sensitivity and Specificity of the MODS assay. Figure S6, Forest plot: Sensitivity and Specificity of the Nitrate Reductase assay. Figure S7, Forest plot: Sensitivity and Specificity of the CRI assay. Table S1, Predictive values of rifampicin resistance in a hypothetic cohort of 1000 participants stratified by test method.