Introduction

In June of 2004, the OraQuick ADVANCE® Rapid HIV-1/2 Antibody Test (OraSure Technologies, Inc., Bethehem, PA) was approved as a CLIA-waived rapid HIV test for use with oral mucosal transudate (oral fluid) specimens in addition to whole blood.[1] In February of 2005, this test first became available for point-of-care HIV antibody testing in publicly-funded HIV testing sites in San Francisco. Because California regulations required phlebotomy certification before performing a fingerstick or blood draw (California Code of Regulations 17§1034; California Business and Professions Code §1240-1246.5), most community-based testing sites in San Francisco were prevented from performing rapid HIV tests using whole blood specimens, and therefore the advent of an oral fluid option led to a swift, significant increase in rapid testing in public HIV test sites (Facente, presented at HIV Diagnostics: New Developments and Challenges; Orlando, FL, 2005).

United States regulations currently require that all reactive rapid HIV tests be considered “preliminary positive” and confirmed in a high-complexity laboratory via Western blot (WB) or indirect immunofluoresence assay (IFA).[2] In cases where the WB or IFA is negative or indeterminate, the results are considered “discordant” and follow up testing is indicated. This follow up testing is needed to determine whether the original reactive result was a false positive or if the patient has HIV infection but is in early seroconversion, causing the tests used for confirmation – some with lower sensitivity – to fail to detect antibodies.[3] From 2003 through early 2006 in San Francisco, all patients with discordant results were asked to submit blood specimens after one month for follow up testing according to the above protocol. This was done regularly with all patients who were successfully contacted one month after the initial result disclosure. In February of 2006, with the agreement of the California Department of Public Health, Office of AIDS and Centers for Disease Control and Prevention, San Francisco began using a new algorithm that included the use of Nucleic Acid Amplification Testing (NAAT) to resolve most discordant tests from the first visit (Dowling and Facente, presented at XVI International AIDS Conference; Toronto, Canada, 2006). From this point forward, follow up testing was not regularly done if an oral fluid rapid HIV test was reactive but blood drawn that same day yielded non-reactive results for another OraQuick rapid test, an EIA, an IFA and/or WB, and NAAT.

According to the manufacturer, the OraQuick ADVANCE® has a specificity of 99.8% (95% Confidence Interval (CI) 99.60%–99.89%) when used with oral fluid.[4] However, clusters of excess false positive results have been well documented[5]–[8], including in San Francisco. To date, these false positive clusters have remained unexplained, with issues such as test kit lot defect, storage or testing area temperature, and operator error being generally ruled out as causes.[9]

In April – July 2008, we noticed a cluster of false positive test results in San Francisco which occurred with kits close to their expiration date, and decided to further investigate the association between kit expiration and false positive results. We hypothesized that there would be a greater probability of false positive results as the kit approached its expiration date.

Materials and Methods

Results

Results were analyzed for 20,904 tests performed between January 2005 and December 2007, with either an initial HIV-negative result (n = 20,828), or a preliminary positive result that proved to be false positive after confirmation by IFA or WB (n = 76). Test results by months remaining until expiration are shown in Table 1. Of 1,108 test kits used with ≤1 month until expiration, 13 (1.17%) were false positive, corresponding to a sample specificity of 98.83% (exact binomial 95% CI 98.00%–99.37%).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Test specificity by time to expiration.

https://doi.org/10.1371/journal.pone.0008217.t001

In the unadjusted logistic regression, the odds of a false positive result were higher for the 1,108 test kits used ≤1 month before expiration, compared to 1,251 kits with ≥6 months until expiration (Odds Ratio (OR) 14.8, 95% CI 1.94–113.6, p = 0.009). False positive rates were also elevated for 2,206 kits with two (OR 7.98, 95% CI 1.05–60.8, p = 0.045) and 4,968 kits with three (OR 6.07, 95% CI 0.82–44.9, p = 0.08) months until expiration. Although the rates for months two and three were similar (OR 1.32, 95% CI 0.68–2.55, p = 0.42), the linear trend across the six months was highly statistically significant (p = 0.0008). Results were similar in the adjusted model, shown in Table 2. The contrast between months one and six (OR 11.5, 95% CI 1.49–89.4, p = 0.019) and the trend across months (p = 0.0057) both remained strong. Six of 129 lots (4.7%), eight of 210 technicians (3.8%) and two of 22 sites (9.1%) had false positive rates of 0.5% to 1.0%, while three lots (2.3%), 13 technicians (6.2%), and two sites (9.1%) had rates greater than 1.0%. These categorizations of lot and technician but not site were important independent predictors in this model, which fit well (Hosmer-Lemeshow p-value = 0.77). Results were similar using other specifications of the adjusted model, and in sensitivity analyses, we found no evidence for effects of calendar month (p = 0.51 for heterogeneity) or calendar year (p = 0.22), after adjusting for lot, technician, and site. In the last month before expiration, adjusted specificity estimated using regression standardization was 99.18% (95% bootstrap confidence interval 98.60–99.57%) (Figure 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Regression standardized test specificity by month to expiration, adjusting for site, lot, and technician.

The blue squares in the figure represent the mean adjusted specificity of the tests, and the black ovals represent the limits of the 95% bootstrap confidence interval.

https://doi.org/10.1371/journal.pone.0008217.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Multivariable logistic model for false positive test results.

https://doi.org/10.1371/journal.pone.0008217.t002

Discussion

We found that the rate of false positive results was significantly higher for oral rapid HIV test kits used with ≤1 month remaining until expiration as compared to kits used with ≥6 months remaining until expiration, after adjustment for the effects of technician and site. Before adjustment, both the point estimate (98.83%) and upper bound of an exact 95% confidence interval (99.37%) for the specificity of tests used with ≤1 month until expiration were below the lower bound for specificity claimed by the manufacturer (99.60%). After regression standardization, the point estimate (99.18%) and upper 95% bootstrap confidence limit (99.57%) both remained below the manufacturer's lower bound. Although residual confounding cannot be ruled out, this suggests that site, lot, and technician effects do not explain our findings, and reflect the rigorous quality assurance system that San Francisco has implemented since the beginning of rapid HIV testing.[15] However, no matter how rigorous a quality assurance system is, some errors will inevitably occur, especially in a non-clinical setting. For example, a well-intentioned technician may swipe the patient's gums with the test kit paddle twice rather than once in an effort to ensure adequate sample collection, or a site's lighting may be improvised in a mobile van, making it more difficult for some technicians to clearly see an extremely faint test line. Ideally, a CLIA-waived test kit will be sufficiently robust to mitigate these factors. It is clear from our results that as these OraQuick ADVANCE® test kits age, the overall testing specificity decreases when used with oral fluid, and that tests used within one month of expiration leave little margin for errors from these other sources.

We also found an independent association of test technician on specificity in our adjusted analysis. However, several of the technicians with high false positivity rates had performed very few tests overall, making it difficult to reliably identify those with higher expected long-term rates. Furthermore, test technicians in publicly-funded test sites in the State of California attend a ½ day training, pass a multi-stage proficiency test, and must successfully complete competency assessment testing at least once per year following initial certification. Despite this, they do not have extensive laboratory training, and it is unrealistic to expect that they will always conduct the test perfectly and with no variation. Again, any CLIA-waived, point-of-care test should be sufficiently robust to maintain accuracy given inevitable variations in technician performance.

Since the first cluster of false positive results in San Francisco in late 2005, significant attention has been paid here to a variety of other factors that could possibly be the cause of such a cluster, particularly factors by site and technician. Test storage practices by site, temperature regulation and monitoring at each site, conditions during transport of test kits (i.e. with mobile testing), eyesight of test readers, lighting conditions at each site, and the opportunity for disruption of test kits during processing were all factors that were reviewed at length at all sites in San Francisco. No patterns related to false positive results were detected and no dramatic variations between sites were observed, leading us to believe these factors were not significant to this analysis.

Previous investigations have not successfully determined a cause for the intermittent clusters of false positive results seen around the country since the introduction of the OraQuick ADVANCE® when used with oral fluid.[5]–[9] Our results, however, offer one potential explanation. They also fit with patterns of test kit use relative to the expiration date, as a result of shipping schedules and quantity of test kits in stock for a particular jurisdiction.[16]

Rapid HIV testing is a critical public health intervention. Not only does it improve access to serological testing for millions of Americans at risk for HIV, but it does so while providing same-day screening results to all patients, with only people testing preliminary positive needing to wait the 7- to 14-days for IFA or WB results. In the United States, the Centers for Disease Control and Prevention as well as numerous state and city governments have spent millions of dollars purchasing the OraQuick ADVANCE® as a cornerstone of this intervention.[17]–[19] This test, used with oral fluid, is currently the most widely used point-of-care rapid HIV test in the country (Wesolowski, Burstein, Zhu, and Ethridge. Presented at HIV Diagnostics: New Developments and Challenges; Orlando, FL, 2005). For these reasons, it is imperative to resolve issues of clustering false positive results with oral fluid in order to continue its use to assist in diagnosing new HIV infections.

Anecdotal experiences from various jurisdictions indicate a marked reduction in false positive results using the most recent version of the OraQuick ADVANCE®, released in February 2009. However, additional analyses should be conducted with data sets in jurisdictions using these new tests, to confirm that test remains robust throughout the current 12-month shelf life claim when the test is run in typical field conditions, and that policy changes and procedural adaptations – such as discontinuing use when kits are within a month of expiration – are not necessary in order to ensure adequate specificity. Finally, CDC and Association of Public Health Laboratories' approval of a rapid multi-test algorithm for point-of-care resolution of false positive results in the United States could also address these issues and eliminate the need, in most cases, for additional testing and prolonged uncertainty regarding an individual's HIV status.

Acknowledgments

We thank all of the HIV Counseling, Testing, and Linkage Coordinators, HIV test counselors, and OraQuick test technicians in San Francisco who make the publicly-funded rapid HIV testing possible. We particularly acknowledge Noah Carraher, Stella Cheung, Alice Heimsoth, Maria Homar, Emalie Huriaux, Matthew Jennings, Thomas Knoble, Marise Rodriguez and Estrella Sarabia, members of the Counseling, Testing, and Linkage Unit of the HIV Prevention Section of the San Francisco Department of Public Health whose technical assistance and data support were integral to the quality of information that led to this analysis. We also thank Bernard Branson (Associate Director for Laboratory Diagnostics, Centers for Disease Control and Prevention) and Michelle Roland (Chief, California Department of Public Health, Center for Infectious Diseases, Office of AIDS) for their thoughtful comments on this analysis and manuscript.