New rapid point-of-care (POC) tests are being developed that would offer the opportunity to increase screening and treatment of several infections, including syphilis. This study evaluated three of these new rapid POC tests at a site in Southern California.
Participants were recruited from a testing center in Long Beach, California. A whole blood specimen was used to evaluate the performance of the Dual Path Platform (DPP) Syphilis Screen & Confirm, DPP HIV-Syphilis, and DPP HIV-HCV-Syphilis rapid tests. The gold-standard comparisons were Treponema pallidum passive particle agglutination (TPPA), rapid plasma reagin (RPR), HCV enzyme immunoassay (EIA), and HIV-1/2 EIA.
A total of 948 whole blood specimens were analyzed in this study. The sensitivity of the HIV tests ranged from 95.7–100% and the specificity was 99.7–100%. The sensitivity and specificity of the HCV test were 91.8% and 99.3%, respectively. The treponemal-test sensitivity when compared to TPPA ranged from 44.0–52.7% and specificity was 98.7–99.6%. The non-treponemal test sensitivity and specificity when compared to RPR was 47.8% and 98.9%, respectively. The sensitivity of the Screen & Confirm test improved to 90.0% when cases who were both treponemal and nontreponemal positive were compared to TPPA+/RPR ≥1∶8.
The HIV and HCV on the multi-infection tests showed good performance, but the treponemal and nontreponemal tests had low sensitivity. These results could be due to a low prevalence of active syphilis in the sample population because the sensitivity improved when the gold standard was limited to those more likely to be active cases. Further evaluation of the new syphilis POC tests is required before implementation into testing programs.
Citation: Hess KL, Fisher DG, Reynolds GL (2014) Sensitivity and Specificity of Point-of-Care Rapid Combination Syphilis-HIV-HCV Tests. PLoS ONE 9(11): e112190. https://doi.org/10.1371/journal.pone.0112190
Editor: David W. Dowdy, Johns Hopkins Bloomberg School of Public Health, United States of America
Received: May 14, 2014; Accepted: August 26, 2014; Published: November 6, 2014
Copyright: © 2014 Hess et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. Data have been deposited to Dryad (doi:10.5061/dryad.nh7f4).
Funding: Support was provided by grant R01DA030234 from the National Institute on Drug Abuse awarded to DGF; grant P20MD003942 from the National Institute of Minority Health and Health Disparities funded part of DGF and GLR time; grant ID10-CSULB-008 from the California HIV Research Program awarded to GLR ChemBio staff provided training on the use of their test materials but the funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The POC rapid tests were provided at a reduced cost by Chembio Diagnostics Inc. Chembio also provided the training on their tests. However, they had no involvement in the development of the study design, data collection, data analysis and interpretation, or writing of the manuscript. There are no other relevant declarations relating to employment, consultancy, patents, products in development, marketed products, etc. This does not alter the authors’ adherence to all PLOS ONE policies on sharing data and materials, as detailed online in the guide for authors.
Worldwide, syphilis remains a large problem with an estimated 11 million new cases in adults in 2005 . The majority of these cases occur in developing countries. Since 2000 there has been an increase in the number of syphilis cases in the United States, predominately among men who have sex with men (MSM) . The standard method for syphilis diagnosis requires two screening tests, a non-treponemal test such as the rapid plasma reagin (RPR) or the Venereal Diseases Research Laboratory (VDRL) test, and a treponemal test such as T. pallidum passive particle agglutination (TPPA) or fluorescent treponemal antibody absorption (FTA-ABS) . Traditionally, a treponemal test is only run after a positive non-treponemal test is obtained. Recently, an alternative algorithm has been adopted by some labs and hospitals where a treponemal test is completed first in the sequence . However, the equipment and personnel needed to conduct the standard tests for syphilis can be prohibitively expensive in low-resource settings. In addition, the delay in receiving the test results can lead to missed treatment opportunities because of a failure to return for test results –.
One solution to these issues with traditional testing is point-of-care (POC) rapid tests . In recent years treponemal POC rapid tests have been developed and have been shown to have good sensitivity and specificity , . However, these tests only detect the presence of treponemal antibodies, so they cannot distinguish between active cases and historical, treated cases. Recently, a dual rapid test has been developed that tests for treponemal antibodies and reagin (DPP Syphilis Screen and Confirm). A study on archived samples and a prospective study in China have both shown promising results , . In addition, syphilis POC rapid tests have also been paired with tests for other infections. These tests detect the treponemal antibody as well as HIV infection in one test  and HIV and hepatitis C virus (HCV) in another. Given the high rate of co-infection and similar populations at risk , it would be advantageous to have a test that could simultaneously detect the presence of multiple infections. This study evaluated the performance of three rapid POC tests in an at-risk population seeking HIV and sexually transmitted infection (STI) testing at a testing center in Long Beach, CA.
The protocol was approved by the California State University Long Beach (CSULB), Institutional Review Board (IRB). All participants provided written informed consent using a form approved by the CSULB IRB that included permission to re-contact the participant to notify them of future studies they may be interested in.
Data for this study were collected from May 26, 2011 to June 30, 2013. Participants were recruited from clients at the Center for Behavioral Research and Services (CBRS) in Long Beach, California. CBRS provides free HIV and STD testing to the community as well as conducts research. The pre-study HIV prevalence was 2.6%, the pre-study prevalence of HCV was 48.1%, the pre-study prevalence of RPR was 3.0%, and the pre-study prevalence of TP-PA was 8.1%. Clients who came in for HIV and other STI testing were screened for eligibility. In addition, existing clients who were eligible and whose last visit was more than three months prior were sent letters inviting them to come in for the study. Eligible clients were 15 years of age and older, had not participated previously, and reported being in a behavioral risk group. Behavioral risk groups were defined as: (1) injection drug users (IDU) with verified track marks (e.g., visible signs of injection) , (2) women who reported at least two male partners in the last two years or engaging in anal intercourse, sex trading, or sex with a man who has sex with men (MSM), an IDU, or an HIV positive man, (3) MSM and men who have sex with men and women (MSMW), and (4) transgender individuals. These definitions of the risk groups were based on guidelines from the Los Angeles County Department of Public Health. Clients were not excluded based on prior infection history. When an eligible client agreed to participate, they gave written informed consent, and a California state licensed phlebotomist drew a venous blood sample, by standard laboratory practices, for the POC tests, as well as the gold standard confirmatory tests. Every test was performed by the phlebotomist and was completed on the whole blood specimen of each participant. However, there was variation in sample size by test because not all experimental test kits were available at all times. During the study session, the participant also completed several questionnaires to gather demographic data as well as their impressions of the testing procedure, personality measures, and risk-behavior information. Two weeks after the initial visit, the participant returned for the gold-standard results. Participants were given the results of the POC tests, but it was stressed that these were experimental and that they should return in two weeks to obtain the gold-standard results. Participants were referred to care upon receiving a positive result from the gold standard test. Clinical diagnosis could not be obtained, therefore it was not possible to correlate it with the results of the laboratory testing.
This paper contains the results for the POC tests that included detection of syphilis infection. These were the Dual Path Platform (DPP) Syphilis Screen & Confirm Assay, the DPP HIV-Syphilis Assay, and the DPP HIV-HCV-Syphilis Assay. All three were manufactured by Chembio Diagnostic Systems, Inc. (Medford, NY). The test kits were stored in a temperature-controlled setting, with the temperature being both monitored and recorded. The test description has been described , . The test procedures were based on the manufacturer’s venipuncture whole blood specimen instructions and the phlebotomists were trained in person on-site in Long Beach by Chembio Staff. For the Syphilis Screen & Confirm Assay, 10 µl of blood was added directly to well 1 followed by two drops of Running Buffer. After five minutes, five drops of Running Buffer were added to Well 2. The results were read 10 to 15 minutes after the addition of buffer to Well 2, but not more than 20 minutes from the addition of buffer to Well 1. There were digital timers in the phlebotomy laboratory that were set to the exact time requirements for the tests. If no control line appeared, the test was discarded. If a control line appeared, then the assay was assessed for the appearance of lines which indicated reactivity to treponemal and non-treponemal antibodies. The procedure for the HIV-Syphilis and HIV-HCV-Syphilis assays were similar to the Syphilis Screen & Confirm Assay, except that, for these two tests, the 10 µl of venous blood was added to the sample buffer bottle. The bottle was then gently shaken and two drops were added to Well 1. After five minutes, Running Buffer (with no blood) was added to Well 2. The results were read using the same method as the Syphilis Screen & Confirm Assay.
The sensitivity and specificity of the three test kits were determined by comparing the results to the gold-standard test results. The comparison test for the treponemal antibody test was TPPA (Fujirebio Inc., Tokyo, Japan) and the comparison for the non-treponemal test was RPR (ASI Arlington Scientific, Inc., Springville, UT). The gold standards for HCV and HIV were HCV enzyme immunoassay (EIA) 2.0 (Abbott Laboratories, Abbott Park, IL) and GS HIV-1/HIV-2 PLUS O EIA (Bio-Rad Laboratories, Redmond, WA), respectively. Analyses were performed with the SAS software package version 9.3 (Cary, NC). Sensitivity and specificity were obtained using PROC FREQ and exact confidence intervals were reported. All p-values reported were based on chi-square tests.
Over the course of the study, 2083 people were screened for eligibility. Of these, 859 were deemed not eligible. The majority of people were not eligible because they were not at increased risk for HIV, such as men with only female partners who do not inject drugs and women with only one male partner. Of those who were eligible, 142 were not offered enrollment in the study, which was typically due to time constraints. Of the people who were offered enrollment, 31 declined. The reasons for declining to participate included lack of time to complete the study visit and not wanting to receive test results the same day. The final sample size for this analysis was 948 after removing the 103 participants who were not able to give a blood sample or were missing RPR and TPPA test results. The sample was 54% men, 44% women, and 1.4% transgender. A third of the sample (35%) was non-Hispanic Black, 26% were Hispanic, and 26% were non-Hispanic White (Table 1). The average age was 39.9 years (SD = 12.7) with a range of 15–77 years. About one-fifth (22%) reported injecting drugs, 39% were women at sexual risk, and 37% were MSM/MSMW. The prevalence of HIV in the sample was 7% (n = 66), 18% (n = 166) tested positive for HCV, and 2% (n = 23) tested positive for TPPA and RPR. An additional 87 people tested positive for only TPPA and 4 tested positive only for RPR.
DPP Syphilis Screen & Confirm Assay
The sensitivity of the non-treponemal test when compared to RPR positivity (titer 1∶1 or higher) was 47.8% (95% CI: 26.8–69.4%) and the specificity was 98.9% (95% CI: 97.9–99.5%; Table 2). The sensitivity improves to 57.9% when the gold standard comparison is TPPA and RPR positivity (i.e. remove the biologic false positives) and improves further to 90.0% when the comparison is TPPA positive and an RPR titer of 1∶8 or greater (i.e. the cases more likely to be active syphilis). The sensitivity of the test among those with an RPR titer of 1∶4 or less is significantly less than the sensitivity among those with a titer of 1∶8 or greater (22.2% vs. 90.0%, p = .003). The sensitivity is low for the treponemal test when compared to TPPA positivity, 52.7% (95% CI: 42.1–63.1%). The specificity of this test was 98.7% (95% CI: 97.5–99.4%; Table 2). When participants who were nontreponemal/treponemal rapid test positive were compared to TPPA+RPR ≥1∶8, the sensitivity was 90.0% (95% CI: 55.5–99.8%) and the specificity was 99.6% (95% CI: 98.8–99.9%).
DPP HIV-HCV-Syphilis Assay
The sensitivities of the HIV-HCV-Syphilis tests were 100% (95% CI: 93.8–100%), 91.8% (95% CI: 86.3–95.6%), and 44.0% (95% CI: 34.8–54.3%; Table 3), respectively. The specificity of the HIV test was 99.9% (95% CI: 99.3–100%). The specificity of the HCV test was 99.3% (95% CI: 98.4–99.8%), and the specificity of the syphilis test was 99.4% (95% CI: 98.5–99.8%).
DPP HIV-Syphilis Assay
The original order of the tests in this assay was HIV first, syphilis second. The company switched the order in hopes of increasing the sensitivity of the syphilis test. Therefore, two different sets of sensitivities and specificities are presented. In the original test, the sensitivity and specificity of the syphilis test was 46.4% (95% CI: 27.5–66.1%) and 99.6% (95% CI: 97.7–100%; Table 3), respectively. After the test order was reversed, the sensitivity and specificity did not change, 47.4% (95% CI: 36.0–59.1%) and 99.5% (95% CI: 98.5–99.9%), respectively. The sensitivity and specificity of the HIV test was originally 100% (95% CI: 81.5–100%) and 100% (95% CI: 98.5–100%), respectively. After the change in configuration, the sensitivity was 95.7% (95% CI: 85.2–99.5%) and the specificity was 99.7% (95% CI: 98.8–100%).
We examined concordance of the treponemal result between the three POC tests. Among participants who had data for all three tests and had a positive result on at least one of the three (n = 55), 40 had a positive result on all three tests (73%). The concordance between the two POC tests that included HIV was 100%.
The multi-infection test kits had good sensitivity and specificity for HIV and HCV although the treponemal test had a poor sensitivity. A single rapid test that can simultaneously detect multiple infections would be very useful in the field, given the overlapping risk groups of these infections , . The good sensitivity of the HIV and HCV on the HIV-HCV-Syphilis rapid test suggests that combining the tests onto one device is possible. However, the treponemal test in this case does not appear to be performing well. In addition, these multi-infection tests still have the drawback of only detecting the treponemal antibody making it difficult to distinguish between an active case and a treated case of syphilis. When combining the tests onto one device it may be advantageous to include both the treponemal and nontreponemal tests to avoid over diagnosis and treatment.
We found two other studies of the ChemBio Screen and Confirm syphilis test, the dual treponemal and non-treponemal test, a study by Yin et al. from China  and a study by Causer et al. from Australia . Both of these studies found a better sensitivity than the current study. Our non-treponemal sensitivity value of 47.8 was significantly lower than the sensitivity in the Yin et al. study of 87.6, and our treponemal sensitivity of 52.7 was significantly lower than the Yin et al. value of 96.2 and the Causer et al. value of 89.8. One reason for the discrepancy in results could be due to error in test administration or result interpretation, but given the high sensitivities of the HIV and HCV rapid tests in the multi-infection test kits, and the fact that the phlebotomists were personally trained by company staff, the likelihood of this is low. A more plausible explanation is the low prevalence of high-titer syphilis in the sample population. In the current study the proportion of TPPA+ cases that were high titer was only 11%, which is significantly lower than the 50% prevalence in the other two samples. Indeed, when the gold-standard comparison included only cases with a high titer, the sensitivity of both the treponemal and nontreponemal lines improved greatly. The Yin et al. sample also showed a reduction in sensitivity of the non-treponemal test among lower titer cases. This suggests that the nontreponemal rapid test is not as sensitive to the lower titer samples. A little harder to explain is the low sensitivity of the treponemal results, because TPPA usually remains detectable for an individual’s lifetime after infection, but previous work has shown a decrease in TPPA concentration after treatment . Potentially the TPPA concentration in treated cases has dipped below the detection level of the rapid POC test, but is still detectable on the gold-standard test.
The results of all three studies suggest that this POC test has adequate sensitivity with a case mix that has more high titers, and may not have adequate sensitivity with a case mix that has lower titers. Because the goal of the rapid POC test would be to detect active syphilis that requires treatment, the lower sensitivities on low titer and RPR negative samples may not be detrimental. However, we were not able to definitively determine which cases in our sample were treated and which were active, or the stage of infection among active cases because we were not able to obtain clinical diagnosis information. Therefore, some of the lower titer cases may have still been active cases requiring treatment.
This study included people at higher risk for HIV and STI infection, but there was still a low prevalence of syphilis infection. The sensitivity and specificity estimates would have been more precise if the sample had included more positive cases, but it is also important to determine the accuracy of the test in low prevalence settings if the rapid test is to be used across all types of populations. POC tests are an important new tool in the detection and treatment of infectious diseases , . Despite the fact that we found low sensitivity for the treponemal and nontreponemal POC tests, the results are still promising because the POC test was able to detect a majority of the potentially active cases, which are the more important to diagnosis and treat. However, more studies should be conducted to better characterize these new POC tests.
The authors would like to thank the interviewers and phlebotomists at the Center for Behavioral Research and Services for recruiting the participants, collecting the data and specimens, and conducting the tests. The authors would also like to thank Carol Itatani, Ph.D., for her assistance in project quality control. The POC rapid tests were provided at a reduced cost by Chembio Diagnostics Inc. Chembio also provided the training on their tests. However, they had no involvement in the development of the study design, data collection, data analysis and interpretation, or writing of the manuscript.
Conceived and designed the experiments: DGF GLR. Performed the experiments: KLH DGF GLR. Analyzed the data: KLH DGF GLR. Contributed reagents/materials/analysis tools: KLH DGF GLR. Contributed to the writing of the manuscript: KLH DGF GLR.
- 1. World Health Organization (2011) Prevalence and incidence of selected sexually transmitted infections, Chlamydia trachomatis, Neisseria gonorrhoeae, syphilis, and Trichomonas vaginalis: Methods and results used by WHO to generate 2005 estimates. Geneva: World Health Organization.
- 2. Fenton KA, Breban R, Vardavas R, Okano JT, Martin T, et al. (2008) Infectious syphilis in high-income settings in the 21st century. The Lancet infectious diseases 8: 244–253.
- 3. Mattei PL, Beachkofsky TM, Gilson RT, Wisco OJ (2012) Syphilis: A reemerging infection. American family physician 86: 433–440.
- 4. Binnicker MJ (2012) Which algorithm should be used to screen for syphilis? Current opinion in infectious diseases 25: 79–85.
- 5. Mishra S, Naik B, Venugopal B, Kudur P, Washington R, et al. (2010) Syphilis screening among female sex workers in Bangalore, India: Comparison of point-of-care testing and traditional serological approaches. Sexually transmitted infections 86: 193–198.
- 6. Reynolds GL, Fisher DG, Napper LE, Marsh KA, Willey C, et al. (2008) Results from a multiple morbidities testing program offering rapid HIV testing bundled with hepatitis and sexually transmitted infection testing. Public Health Reports 123: 63–69.
- 7. Marsh KA, Reynolds GL, Rogala BE, Fisher DG, Napper LE (2010) Who chooses a rapid test for HIV in Los Angeles County, California. Evaluation and the Health Professions 33: 177–196.
- 8. Jafari Y, Peeling RW, Shivkumar S, Claessens C, Joseph L, et al. (2013) Are Treponea pallidum specific rapid and point-of-care tests for syphilis accurate enough for screening in resource limited setting? Evidence from a meta-analysis. PloS one 8: e54695.
- 9. Herring AJ, Ballard RC, Pope V, Adegbola RA, Changalucha J, et al. (2006) A multi-centre evaluation of nine rapid, point-of-care syphilis tests using archived sera. Sexually transmitted infections 82 Suppl 5v7–12.
- 10. Mabey D, Peeling RW, Ballard R, Benzaken AS, Galban E, et al. (2006) Prospective, multi-centre clinic-based evaluation of four rapid diagnostic tests for syphilis. Sexually transmitted infections 82 Suppl 5v13–16.
- 11. Yin YP, Chen XS, Wei WH, Gong KL, Cao WL, et al. (2013) A dual point-of-care test shows good performance in simultaneously detecting nontreponemal and treponemal antibodies in patients with syphilis: A multisite evaluation study in China. Clinical infectious diseases : an official publication of the Infectious Diseases Society of America 56: 659–665.
- 12. Castro AR, Esfandiari J, Kumar S, Ashton M, Kikkert SE, et al. (2010) Novel point-of-care test for simultaneous detection of nontreponemal and treponemal antibodies in patients with syphilis. Journal of clinical microbiology 48: 4615–4619.
- 13. Esfandiari J, Gunasekera D, Sayegh F, Addiss K, Sikar-Gang A, et al. (2012) Performance evaluation of the DPP HIV-SYPHILIS assay: A novel, point-of-care rapid HIV 1/2 and syphilis Treponema pallidum antibody combination test. HIV Diagnostics Conference. Atlanta, GA.
- 14. Buchacz K, Klausner JD, Kerndt PR, Shouse RL, Onorato I, et al. (2008) HIV incidence among men diagnosed with early syphilis in Atlanta, San Francisco, and Los Angeles, 2004 to 2005. Journal of acquired immune deficiency syndromes 47: 234–240.
- 15. Cagle HH, Fisher DG, Senter TP, Thurmond RD, Kaster AR (2002) Classifying skin lesions of injection drug users: A method for corroborating disease risk. DHHS Pub No (SMA) 02-XXX. Rockville, MD: Center for Substance Abuse Treatment, Substance Abuse and Mental Health Services Administration.
- 16. Lopez-Zetina J, Ford W, Weber M, Barna S, Woerhle T, et al. (2000) Predictors of syphilis seroreactivity and prevalence of HIV among street recruited injection drug users in Los Angeles County, 1994–6. Sexually transmitted infections 76: 462–469.
- 17. Causer LM, Kaldor JM, Fairley CK, Donovan B, Karapanagiotidis T, et al. (2014) A laboratory-based evaluation of four rapid point-of-care tests for syphilis. PloS one 9: e91504.
- 18. Castro RR, Prieto ES, Santo I, Azevedo J, Exposto FL (2001) Evaluation of the passive particle agglutination test in the serodiagnosis and follow-up of syphilis. American journal of clinical pathology 116: 581–585.
- 19. Tucker JD, Bien CH, Peeling RW (2013) Point-of-care testing for sexually transmitted infections: recent advances and implications for disease control. Current opinion in infectious diseases 26: 73–79.
- 20. Mabey DC, Sollis KA, Kelly HA, Benzaken AS, Bitarakwate E, et al. (2012) Point-of-care tests to strengthen health systems and save newborn lives: the case of syphilis. PLoS medicine 9: e1001233.