Introduction

The use of rapid tests (RTs) for HIV in Brazil dates from 2001 onwards [1], but the first recommendation of the use of RTs for the diagnosis of HIV infection without the need for additional testing was Resolution No. 34 of the Brazilian Ministry of Health of Brazil (BMH), published in 2005 [2]. In 2013, Resolution No. 29 of BMH was published, and a manual entitled “Technical Manual for the Diagnosis of HIV Infection” was released to Brazilian national health services, which currently regulates the diagnosis of HIV infection in Brazil [3–6]. This manual provides a detailed description of the laboratory approaches to characterize HIV infection in flowcharts: a) advocating the use of only RTs for the diagnosis of HIV infection; b) RTs using oral fluid as an alternative diagnostic tool; c) screening immunoassay, either third or fourth generation followed by an HIV viral load in positive samples; or d) screening immunoassay, either third or fourth generation followed by Western Blot/immunoblot (WB/IB) as an alternative.

The use of RTs should preferably be used in situations where there is no laboratory infrastructure or hard-to-reach regions, including Testing and Counseling Centers, Mobile Testing Units, Psychosocial Care Centers, specific and vulnerable population segments, Emergency Care Services, and hospitals. In cases of occupational biological accidents, RTs are recommended for use with pregnant women who have not been tested during prenatal care or whose gestational age does not warrant testing results before delivery, parturient and postpartum women who have not been tested before birth or when the test result is not known at the time of delivery, spontaneous abortion, and people in situations of sexual violence, for prophylaxis purposes. In Brazil, according to current recommendations, one positive test must be followed by another different RT. These tests must be sequential, and it is recommended that, in positive cases, the presence of a virus should be confirmed by the HIV viral load quantification test as soon as possible [3–5].

RTs give same-day results (up to 30 minutes) in a variety of situations and locations [7], and their key features include low cost, quick results, and low degree of complexity of operation and reading [8–10]. Most of these tests are small and portable kits [11]. According to Agustí et al. [12] and Louie et al. [13], RTs have optimized the medical community’s role in identifying and informing infected individuals, especially in health centers, emergency rooms, doctors' offices, and clinics in general. However, some authors report a low RT sensitivity when compared to the gold standard—a screening test followed by a confirmatory test [14].

There are 41 HIV RT kits that have been registered and approved by the Brazilian Health Surveillance Agency (ANVISA) [15], but few studies comparing their analytical performance and feasibility for routine protocols using only RTs for the diagnosis of the infection are available. Data provided by the BMH report forty scientific references; however, only one nationwide study in Brazil compared the effectiveness of RTs in relation to the gold standard [16]. Therefore, the use of RTs in Brazil, not only for screening, but also for the diagnosis of HIV infection, is still a controversial theme in healthcare routines, and thus warrants further investigation.

This study sought to evaluate eight different RTs used to detect anti-HIV antibodies available in Brazil, regarding the analytical performance (sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and accuracy), in human blood serum samples with previously defined serological pattern; to evaluate the quality of the reading pattern of the results (bands); and to establish the agreement between a combination of two kits to support current Brazilian recommendations.

Materials and methods

The Research Ethics Committee (COEP) of Universidade Federal de Minas Gerais (UFMG) approved this research (CAAE 47246115.6.0000.5149) and the need for participant consent was waived.

This is an observational, analytical, and concordance study, which evaluated the analytical performance of eight different commercial RT kits (Table 1) used to detect HIV infection in previously defined serological human serum samples, either positive or negative. The sample calculation test was based on the prevalence of HIV infection in the Brazilian population, considering a sampling error of 0.05% and a significance level of 0.05. Two hundred samples from individuals over 18 months of age were used according to the sample calculation test, and defined as:

1. HIV-reactive samples (n = 100)–samples with results above the cut-off value (1.00) in the screening test (electrochemiluminescence, ECLIA-HIV Combi–HIV-1 antigen and total anti-HIV-1 and anti-HIV-2 antibodies—Roche Diagnostics^®, Mannheim, Germany) and a confirmatory test (Western Blot, New Lav Blot I, BioRad^®, Marnes la Coquette, France), showing at least two of the following bands: p24, gp41, and gp120/gp160;
2. non-reactive HIV samples (n = 100)–samples whose results were below the cut-off value in the screening test (<1.0) and no WB bands.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. List of commercial HIV rapid tests used in this study.

https://doi.org/10.1371/journal.pone.0237438.t001

The RTs followed manufacturer instructions. A result was considered valid when the control band appeared in each strip. The reading of the strips was performed by two independent researchers. The samples were initially classified as reactive or nonreactive, according to the presence or absence of a band in the test area, respectively. To compare the quality of the reading pattern of the bands, the following results were classified according to their color intensity: nonreactive (no band seen), “strong” reactive (band equal or stronger to control band), “weak” reactive (weaker than control band, but still easy to see), and “very weak” reactive (a faint band, hard to see).

The following analytical parameters were then established for each kit, with 95% confidence intervals (CI): sensitivity, specificity, positive (PPV) and negative (NPV) predictive values, positive (RL+) and negative (RL-) likelihood ratios, and accuracy. A true positive (TP) result was achieved when it showed reactivity in the screening, WB and RT; true negatives (TN) showed nonreactive results in the screening, WB and RT; false positives (FP) appeared as nonreactive results in screening and WB tests, but reactive in RTs; and false negatives (FN) appeared as nonreactive results in screening, WB tests, and RTs.

Descriptive statistical analysis of categorical variables was performed, and the Kappa test was used to evaluate the agreement between the kits. Good correlation was considered when the kappa index was ≥ 0.85. The Wilcoxon test was applied to compare the values of sensitivity and specificity reported by the manufacturers. The statistical analysis was performed by the IBM SPSS Statistics for Windows, version 19.0 [17], and a 95% CI (p <0.05) was considered significant.

Results

Of the 200 serum samples analyzed in this study, 115 (57.5%) were male and 85 (42.5%) female. Most of the samples came from patients of the Southeast region of Brazil (50.2%), where the study took place. Distribution according to gender and age demonstrates a higher prevalence of positive results in men aged 26 to 30 years and women aged 31 to 35 years. The average age of the patients was 34.28 (32.75–35.81) years.

Among the eight tested kits, five (Alere Determine HIV 1&2, DPP Rapid Test HIV 1/2, DS Rapid Test HIV, Interkit HIV 1&2, and Bioeasy HIV) achieved 100% sensitivity, while one showed 99% sensitivity (HIV 1/2/O Tri-Line), which was similar to that reported by the manufacturers. However, two kits showed performances far below those reported in prior studies, showing only 92% sensitivity each other (Imuno-rápido HIV 1&2 and Imunocrom HIV 1/2). Many samples showed very weak bands, which were very hard to see, engendering test repetitions to reach a consensus among researchers. In fact, 38 samples showed “weak” reactive bands and 35 showed “very weak” reactive bands, using the Imunocrom HIV 1/2 kit and on Immuno-rápido HIV 1&2 kits. The results were considered as reactive according to the color scale template included within the kit by the manufacturer. According to this, any band color intensity should be considered reactive. This template shows six bands of different color intensity, the first two of which were very difficult to view (similar to the results which were classified as “very weak” reactive by the present work).

These results were considered positive in order to determine the kit’s sensitivity, given that a “band” was seen, although often very faintly (which is different from the true negative result, i.e, no band at all). This will be discussed below.

Regarding the specificity range, there was a variation from 94% to 100%; some of the kits were less specific than those values reported by the manufacturers. The PPV ranged from 90.7% to 100%, while the NPV ranged from 95.5% to 100% (Table 2). Good accuracy was observed in six kits (> 97%).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Analytical performance of eight rapid tests for detection of HIV infection (95% CI).

https://doi.org/10.1371/journal.pone.0237438.t002

Samples initially considered discordant from the gold standard results were repeated. When compared to the initial result, no changes were observed after repeating the tests, demonstrating the good reproducibility of the evaluated kits.

Table 3 shows data concerning the agreement (Kappa index) between kits when paired in a combination of two, as two different kits are recommended for a confirmation of an initially reactive sample in Brazil.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Agreement study—Kappa index (95% CI)—of the eight HIV rapid tests.

https://doi.org/10.1371/journal.pone.0237438.t003

Regarding the quality of the reading pattern of the bands, a variation was observed in the visualization pattern of the bands in a similarly known reactive blood sample when different kits were tested (e.g. Fig 1). For these samples, results were classified as “strong”, “weak”, “very weak” reactive and/or non-reactive (S1 Table) and compared with the color intensity guide template of Imuno-rápido HIV 1&2 provided by the manufacturer.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Variation in the results pattern of the same blood sample known as reactive.

Different color intensity of the bands is observed in the eight tested devices. 1- Alere Determine HIV 1&2 ("strong" reactive); 2- DPP Rapid Test HIV 1/2 ("weak" reactive); 3- DS Rapid Test HIV ("weak" reactive); 4- Immunocrom HIV 1/2 (nonreactive); 5- Immuno-rápido HIV 1&2 ("very weak" reactive), difficult interpretation due to the very weak intensity of the band; 6- Interkit HIV 1&2 ("weak" reactive); 7- HIV 1/2/O Tri-line ("weak" reactive); 8- Bioeasy HIV Test ("strong" reactive).

https://doi.org/10.1371/journal.pone.0237438.g001

False negative and false positive results are shown in Table 4. The number of true positive results verifies the sensitivity rates.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. True positive and negative, false positive and negative of eight Rapid Test brands to detect HIV infection.

https://doi.org/10.1371/journal.pone.0237438.t004

Discussion

The best performing kits based on accuracy analysis were DS Rapid Test HIV and Interkit HIV 1&2. The kits that presented the lowest performance and effectiveness were Imunocrom HIV 1/2 and Immuno-rápido HIV 1&2.

Imunocrom HIV 1/2 showed a large amount of “weak” and “very weak” reactive results (38% and 35%, respectively), as well as Imuno-rápido HIV 1&2 (63% and 17%, respectively). The reading pattern of the bands cannot be mistaken or doubtful to the technical staff because rapid HIV tests provide only qualitative results. Therefore, unnecessary repetitions would be demanded, rising costs in the laboratory. Even with a sensitivity of greater than 90%, we consider its performance inadequate for use in routines and identified the need to use a template with a color scale. Therefore, we suggest that manufacturers conduct a careful revision of these kits, and we would emphasize that clinical laboratories should always proceed with the full verification of the validation processes.

For some authors, the differences found in the sensitivity and specificity when comparing the kits may be due to variations in the ability of the test to detect early seroconversion, due to HIV viral genetic diversity or to the use of different antigens/epitopes or even to the size of the population sample studied [13, 18, 19]. Some studies concluded that RTs were not able to detect antibodies from patients infected with certain HIV subtypes [14]. Machado et al. [20] points out that the use of RTs in a specific geographic area should be validated to ensure that the test is adequately sensitive to the circulating HIV types, as positive and negative predictive values may be affected by the relative distribution of the various subtypes in a given region [18, 20].

For Delaney et al. [21], HIV RTs show lower sensitivity than do some conventional assays, especially during the early phase of infection. False negative results were also observed in individuals with advanced disease and in some patients on antiretroviral therapy. By contrast, in the present study, all samples had been previously evaluated using the gold standard, a screening test algorithm (ECLIA), followed by a confirmatory test (WB). Therefore, both sensitivity and specificity could be compared to this algorithm.

In Brazil, according to Resolution 29 of BMH [3–6], if only RTs are used in the diagnosis, more precision can be observed when a reactive result is repeatedly confirmed by another RT from another manufacturer. Thus, in the paired evaluation of RTs, six combinations of paired kits achieved Kappa > = 0.85. However, it is prudent to conduct further studies that prove the diagnostic efficiency of these RTs and other RTs available on the market.

Considering the sensitivity criteria for the diagnostic use of RTs proposed by the BMH [3–6] (sensitivity ≥ 99.5%), five out of eight kits were approved in this study. Concerning the specificity (specificity ≥ 99.0% according to the MS), only two met the requirements for routine care. Unfortunately, the technical manual fails to mention the studies that gave origin to this information and how these sensitivity/specificity values have been obtained. Likewise, not all RT use instructions report information in which reference tests were used, nor the description of the tested population.

Ferreira Junior et al. [16], in 2005, evaluated seven different RTs commercially available in Brazil and found that of these, only four were considered acceptable for diagnosis. In the present study, even though there were differences in sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and accuracy, of the eight evaluated RTs, six showed accuracies > 97%. Therefore, it is important for physicians to be aware of such discrepancies and that any conflicting results with epidemiology and/or clinics should be interpreted with caution. In cases of doubt, conventional testing is always recommended (a screening immunoassay, followed by a confirmatory test on those initially reactive samples).

Kosack et al. [22], in 2017, evaluated eight HIV RTs using specimens collected from Médecins Sans Frontières sites, between 2011 and 2015, in five African countries. They found a substantial number of false reactive test results evidencing a poor positive predictive values and concluded that only one test met the recommended thresholds for RTs of ≥ 99% sensitivity and ≥ 98% specificity. However, according Johnson et al. [23], commenting Kosack et al. article, a single reactive test result is never sufficient to make an HIV-positive diagnosis and the countries and programs need to ensure that they are following World Health Organization (WHO) recommended HIV testing strategies and use a validated testing algorithm. In Kosack et al. [24] response letter, individual test performances in the target population must be known and the false reactivity of individual samples with multiple tests considered.

Kravitz Del Solar et al. [25] recommends that all countries use WHO pre-qualified RTs within the recommended strategies and algorithms for HIV testing. They also support validation of HIV testing algorithms using in-country specimens to determine optimal performance, and the reverification testing of all people diagnosed with HIV prior to starting treatment as an essential quality assurance measure. In our study, the analytical performance of eight different commercial RT kits used to detect HIV infection were evaluated in previously defined serological human serum samples, either positive or negative, according to the guidelines form BMH and WHO recommendations.

There is no evidence in the literature of other studies that evaluated the same RTs tested in our study. Therefore, our results were important to alert the population which RTs had the best diagnostic accuracy when used in sequence, according to the standards of sensitivity and specificity required by BMH.

In conclusion, performance differences were found among the eights tested kits. Six achieved good agreement for use in routines whenever a result is initially reactive; however, the best combination regarding accuracy was the Bioeasy HIV Test and the Interkit HIV 1&2. It is worth noting that every trial, whether rapid or conventional, requires internal validation, as described in good clinical laboratory practices.

Supporting information

Acknowledgments

We would like to thank the Institute Hermes Pardini for the availability of biological samples and the Department of Complementary Propaedeutics of the UFMG School of Medicine for the use of their laboratory and the materials necessary to perform this study.