In South Africa, various point-of-care hemoglobin meters are used. However, the regulatory framework for approval, implementation and oversight of use of point-of-care hemoglobin meters is suboptimal. We assessed the diagnostic accuracy of the HemoCue Hb 301, STAT-Site MHgb and URIT-12 point-of-care hemoglobin meters, compared to a central laboratory based reference assay, in a central laboratory and a community based clinic in Durban, South Africa. Differences in performance of the point-of-care assays, compared to the reference assay, were more pronounced in the community based clinic. Results were reasonable for the HemoCue Hb 301, but poor for the STAT-Site MHgb and the URIT-12. Poor test performance of point-of-care hemoglobin meters, and inadequate evaluations and oversight in South Africa, leads to suboptimal clinical care and clinical research, and increased costs. There is a need for proper evaluation and quality assurance of point-of-care tests, the results of which should be made widely available to key stakeholders.
Citation: Jaggernath M, Naicker R, Madurai S, Brockman MA, Ndung’u T, Gelderblom HC (2016) Diagnostic Accuracy of the HemoCue Hb 301, STAT-Site MHgb and URIT-12 Point-of-Care Hemoglobin Meters in a Central Laboratory and a Community Based Clinic in Durban, South Africa. PLoS ONE 11(4): e0152184. https://doi.org/10.1371/journal.pone.0152184
Editor: Pal Bela Szecsi, Gentofte University Hospital, DENMARK
Received: May 24, 2015; Accepted: March 10, 2016; Published: April 5, 2016
Copyright: © 2016 Jaggernath et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data is available from FigShare. The DOI is https://dx.doi.org/10.6084/m9.figshare.3119449.v1.
Funding: This study was funded by the Canada-sub-Saharan HIV/AIDS Network (CANSSA) through funding provided by the Global Health Research Initiative (GHRI), itself a collaborative research funding partnership of the CIHR, the Canadian International Development Agency (CIDA), and the International Development Research Centre (IDRC). Open access publication of this article has been made possible through support from the Victor Daitz Information Gateway, an initiative of the Victor Daitz Foundation and the University of KwaZulu-Natal. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Anemia is common in sub-Saharan Africa , and a frequent reason for exclusion from clinical studies . During screening procedures for enrollment in clinical studies, hemoglobin levels are usually assessed using portable point-of-care hemoglobin meters with blood obtained by finger prick. Portable point-of-care assays can give results within minutes, allowing rapid clinical decision-making, during the encounter with the patient . But point-of-care test results must be accurate. Inaccurate point-of-care hemoglobin meters that misdiagnose anemia may lead to (i) false exclusion from, and inclusion in, clinical studies, and (ii) compromise clinical care.
In South Africa there is suboptimal government regulation on the approval and use of point-of-care assays. During screening procedures for a clinical study we observed discrepant results between a point-of-care hemoglobin meter and a laboratory-based testing platform in a certified central laboratory. We therefore performed a prospective study to assess the diagnostic accuracy of three point-of-care hemoglobin meters in (i) a certified central laboratory and (ii) in a community based clinic, compared to a central laboratory reference assay.
Materials and Methods
We determined the diagnostic accuracy of three point-of-care hemoglobin meters in a 2-phase study. In phase 1, the point-of-care tests were performed in a central laboratory (Global Clinical and Viral Laboratory in Amanzimtoti), compared to a laboratory reference standard. In phase 2, the point-of-care tests were performed in a community based clinic (iThembalabantu clinic in the Umlazi township) and the reference test was performed in the central laboratory, using a sample obtained at the same time as the point-of-care testing. The study was approved by the institutional review board of the University of KwaZulu-Natal. For phase 1 we used anonymized routine blood samples. For phase 2, written informed consent was obtained from each patient in English or isiZulu (the predominant local language).
Patients were eligible for inclusion in phase 2 of the study if they met the following criteria: Age 18 years or older, present for clinical care or HIV screening at one of our research clinics or during outreach activities in the greater Durban area, willing to provide written informed consent, and willing to undergo study procedure.
During phase 1 of the study, we used anonymized EDTA whole blood samples submitted to the central laboratory for routine hematology testing. During phase 2 of the study, we used blood obtained by finger prick for the three point-of-care tests performed in the community based clinic, and we drew EDTA whole blood samples for the reference hemoglobin test in the central laboratory. The distance between the community based clinic and the central laboratory was 15km. Samples were transported from the community based clinic to the central laboratory once or twice a day.
We evaluated three point-of-care hemoglobin meters: URIT-12 (URIT Medical Electronic Group, Guilin, Guangxi, China ), STAT-Site MHgb (Stanbio Laboratory, Boerne, TX, USA ), and HemoCue Hb301 (HemoCue AB, Ängelholm, Sweden ). We compared the results from the point-of-care hemoglobin meters to those obtained on an automated hematology analyzer (Sysmex XS-1000i, Sysmex, Kobe, Hyogo, Japan ) as a reference hemoglobin meter in a South African National Accreditation System (SANAS) certified laboratory. The Sysmex XS-1000i was used according to the manufacturer’s recommendations. Calibration was performed annually. QC was performed daily with high, medium and low controls. The total allowable error for hemoglobin was 4.19%. External quality assurance of the laboratory was performed every 4 months. All point-of-care hemoglobin meters were used according to the manufacturers’ recommendations. During phase 1, one technician conducted all tests. During phase 2, one nurse conducted all point-of-care tests in the community based clinic, and one technician subsequently conducted all reference laboratory tests in the central laboratory; the technician was blinded to the point-of-care test results. All tests were performed as single measurements. The point-of-care tests were conducted in random order. Anemia was defined according to World Health Organization criteria as hemoglobin < 12 g/dl for women, and < 13 g/dl for men .
We used GraphPad Prism and Microsoft Excel for graphical representation and statistical analysis. We assessed the accuracy, as measured by the mean difference (bias) and 95% limits of agreement, of the HemoCue 301, URIT-12 and STAT-Site MHgb point-of-care hemoglobin meters compared to the laboratory hemoglobin test as a reference using the Bland-Altman method . We assessed reproducibility (inter-assay variation) during phase 1 by repeat testing a subset of samples with all assays.
Results and Discussion
Results phase 1, central laboratory
During phase 1 of the study, we tested samples from 60 patients in a central laboratory. We performed phase 1 of the study from August 19 to 27, 2013. We performed all four tests on individual samples on the same day.
As depicted in Figs 1 and 2, compared to the Sysmex XS-1000i laboratory reference, results were most accurate for the HemoCue (bias –0.54, limits of agreement –1.28 to 0.20 g/dL, CV <1%), less accurate for the STAT-Site MHgb (bias 1.84, limits of agreement –3.86 to 7.53 g/dL, CV = 30.4%) and least accurate for the URIT-12 (bias 2.64, limits of agreement 0.59 to 4.69 g/dL, CV <5%). The sensitivity and specificity of the point-of-care assays to detect or exclude anemia in the central laboratory is summarized in S1 Table, and S1 Fig.
In phase 1 of the study the STAT-Site had a high failure rate. It seems that this was related to uneven migration of the sample through the sample strip, resulting in the sample migration time exceeding the maximum time specified by the manufacturer.
In phase 1 of the study the STAT-Site had a high failure rate. It seems that this was related to uneven migration of the sample through the sample strip, resulting in the sample migration time exceeding the maximum time specified by the manufacturer.
Results phase 2, community based clinic
During phase 2 of the study, we tested samples from 100 patients (63 women and 37 men) in a community based clinic using three point-of-care devices with blood obtained by finger prick, and compared them with the laboratory reference using a venous EDTA blood sample drawn at the time of point-of-care testing. The median duration between venous blood draw and sample receipt at the central laboratory was 3 hours and 21 minutes (range 38 minutes–8 hours and 25 minutes). The median duration between venous blood draw and test result at the central laboratory was 5 hours and 9 minutes (range 1 hour and 15 minutes–9 hours and 37 minutes). We performed phase 2 of the study from February 10 to 17, 2014. The median age of the 63 women and 37 men was 32 years (range 18–60) and 27 years (range 19–54), respectively. The median Hemoglobin of the 63 women and 37 men was 11.8 g/dL (range 7.6–14.8) and 14.1 g/dL (range 10.8–16.3), respectively.
As depicted in Figs 1 and 2, compared to the Sysmex XS-1000i laboratory reference, results were most accurate for the HemoCue (bias 0.16, limits of agreement –2.16 to 2.47 g/dL), less accurate for the STAT-Site MHgb (bias 0.51, limits of agreement –1.98 to 2.99 g/dL), and least accurate for the URIT-12 (bias 2.65, limits of agreement –0.09 to 5.41 g/dL).
The sensitivity and specificity of the point-of-care assays to detect or exclude anemia in a community based clinical setting is depicted in Table 1 and Fig 3. Of note, the sensitivity of the URIT-12 to detect anemia in the community based clinical setting was 100%, but this number must be seen in context of the considerable bias in the results: out of 100 patients, 96 were diagnosed as anemic by the URIT-12. Of those 96 patients diagnosed by URIT-12 as anemic, only 40 (positive predictive value 42%) were truly anemic according to the laboratory reference and the specificity was 7%, with only 4 of 60 non-anemic patients correctly diagnosed as non-anemic by the URIT-12. The clinical consequence of this structural misclassification by the URIT-12 is overdiagnosis of anemia, and overtreatment. The additional consequence in research settings is that many potential study participants will needlessly be excluded from clinical studies, based on the false assumption that they are anemic.
Dots in the blue quadrant indicate that the point-of-care assay missed anemia, i.e., misclassified a finger prick sample as non-anemic where the venous blood sample was classified as anemic according to the reference assay in a central laboratory. Dots in the red quadrant indicate that the point-of-care assay misclassified as anemic a sample that was non-anemic according to the reference assay.
We found that the diagnostic accuracy of 3 point-of-care hemoglobin meters is different in central laboratory and community based clinic settings, with a trend towards greater variation of results in the community based clinic, for which there are 2 possible explanations: First, in the community based clinic setting we compared blood obtained by finger prick for the point-of-care meter with venous blood for the reference laboratory meter. Differences between finger stick and venous hemoglobin measurements have been observed in several studies, with finger stick measurements both underestimating and overestimating hemoglobin levels [10, 11]. Second, in phase 2 of the study the point-of-care test was performed by a nurse in a community based clinic, while the central laboratory test was performed by a laboratory technician.
The benefit of our 2 phase approach is that in phase 1 in a central laboratory we could already see the structural bias of the URIT-12, and a 10% invalid results rate for the STAT-Site. Based on the phase 1 results we could have rejected both the STAT-Site and the URIT-12. In phase 2 the structural bias of the URIT-12 was confirmed, but the performance of the Stat-Site was slightly better than during phase 1. This 2 phase approach shows that good performance of a point-of-care test in a central laboratory does not guarantee good performance in a community based clinic setting. Evaluation of a point-of-care test in only a community based clinical setting will make it difficult to determine causes of poor performance. Therefore we recommend assessment of the diagnostic accuracy of point-of-care tests in both central laboratory and community based (clinical) settings.
Approval and registration of diagnostics is usually based on central laboratory studies alone. Our study confirms that independent central laboratory and community based evaluations of diagnostic accuracy are important. Appropriate analyses of the data may reveal problems that may not be apparent in the initial registration studies [12, 13]. Several community based evaluations of point-of-care tests [14–18] have revealed problems that do not typically appear in a controlled laboratory environment. Government regulations on the use of diagnostics in South Africa exist but are not very detailed. There seems to be widespread use of point-of-care tests that have not been properly evaluated. Ideally, a diagnostic test should (i) receive approval from regulatory authorities such as the FDA, or CE-marking, and (ii) be evaluated in both central laboratory and community based settings by at least two independent groups, according to STARD guidelines [19–22] (http://www.equator-network.org/reporting-guidelines/stard/), with the results published in the peer reviewed literature.
Another challenge in many African settings is that for many values the clinical laboratory reference intervals derived from Western countries may not be appropriate. Karita et al established reference intervals for routine haematology and biochemistry in healthy adults in Eastern and Southern Africa . Compared to Western reference intervals, the average hemoglobin was lower in several populations. Many otherwise healthy adults would be defined as anaemic, and excluded from clinical studies using inclusion and exclusion criteria based on laboratory reference intervals from other populations .
Taken together, the combination of poor regulation, inaccurate tests, and inappropriate reference intervals creates two problems for clinical medicine, and three problems for clinical studies in sub-Saharan Africa. The two problems for clinical medicine are that on the one hand false anemia will lead to costly and unnecessary overtreatment, and on the other hand true anemia will be missed possibly resulting in greater morbidity and associated costs. The three problems for clinical research are that (i) it is difficult to recruit per se due to inappropriate reference intervals, (ii) inaccurate assays will decrease recruitment (false exclusion) and increase premature termination (false inclusion), and (iii) increase the efforts, duration and costs of clinical studies.
How can we benefit from the portability and speed of point-of-care tests, but also accurately determine whether a patient is truly anemic? A practical solution might be an algorithm where point-of-care results within a chosen interval around the cut-off for anemia must be followed by a laboratory reference test. The width of the interval would depend on the accuracy of the point-of-care assay. The patients with hemoglobin values below the interval would be classified as anemic with a high degree of certainty with the point-of-care assay.
Diagnostic accuracy studies are relatively simple, and many laboratories in sub-Saharan Africa perform in-house and/or community based evaluations of new assays, but few publish the results [24, 25]. Diagnostic accuracy studies are good introductions to laboratory and clinical research. Generic protocols can be developed and used, with minor modifications, for different tests. This is a tremendous opportunity to build research capacity, and at the same time address the practical clinical need to assess the diagnostic accuracy of new tests.
In this study, we have shown that the HemoCue Hb301 is a reasonably accurate point-of-care hemoglobin meter, but the STAT-Site MHgb and URIT-12 are not. When the assays are performed in a community based setting, none of them can match the performance of a laboratory based reference. The widespread use of an inaccurate point-of-care hemoglobin meter such as the STAT-Site MHgb and especially the URIT-12 in the public health sector in South Africa is a concern, as it may result in both overtreatment and undertreatment, with ensuing clinical consequences and costs. There is a need for policy guidelines on the validation, approval and quality control of point-of-care assays to (i) improve patient care and (ii) ensure optimal use of scarce resources. The proposed studies can (i) benefit clinical medicine, (ii) benefit clinical research, and (iii) help build research capacity.
S1 Fig. Correlation of the 3 point-of-care assays with values within the dynamic range of the reference hemoglobin meter in 100 samples in a central laboratory in phase 1 of the study, where all tests were conducted on the same sample.
Dots in the blue quadrant indicate that the point-of-care assay missed anemia, i.e., misclassified a sample as non-anemic that was anemic according to the reference. Dots in the red quadrant indicate that the point-of-care assay misclassified as anemic a sample that was non-anemic according to the reference.
S1 Table. Phase 1 (central laboratory) sensitivity, specificity, and predictive values for the 3 point-of-care hemoglobin meters to detect or exclude anemia.
Sensitivity = % of patients with anemia, according to the reference laboratory test, that were identified as anemic by the point-of-care test; specificity = % of patients without anemia, according to the reference laboratory test, that were identified as non-anemic by the point-of-care test; PPV, positive predictive value = % of patients that were identified as anemic by the point-of-care test that were confirmed as anemic by the reference laboratory test; NPV, negative predictive value = % of patients that were identified as non-anemic by the point-of-care test that were confirmed as non-anemic by the reference laboratory test.
We thank the study participants, the HIV Pathogenesis Programme staff, in particular Ayanda Ngubane and Monica Nyawo, the Global Laboratories staff, Natasha Samsunder from CAPRISA, and our colleagues from the AIDS Healthcare Foundation iThembalabantu Clinic.
This study was funded by the Canada-sub-Saharan HIV/AIDS Network (CANSSA) through funding provided by the Global Health Research Initiative (GHRI), itself a collaborative research funding partnership of the CIHR, the Canadian International Development Agency (CIDA), and the International Development Research Centre (IDRC). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
TN holds the South African DST/NRF Research Chair in Systems Biology of HIV/AIDS, the Victor Daitz Chair in HIV/TB Research and an International Early Career Scientist Award from the Howard Hughes Medical Institute. Preliminary results of these studies were disclosed at the Southern African HIV Clinicians Society Conference, Cape Town, South Africa, 24–27 Sept 2014.
Conceived and designed the experiments: MJ SM HCG. Performed the experiments: MJ RN. Analyzed the data: MJ HCG. Contributed reagents/materials/analysis tools: MJ RN SM MAB TN HCG. Wrote the paper: MJ RN SM MAB TN HCG.
- 1. van Hensbroek MB, Jonker F, Bates I. Severe acquired anaemia in Africa: new concepts. British journal of haematology. 2011;154(6):690–5. pmid:21707575.
- 2. Omosa-Manyonyi GS, Jaoko W, Anzala O, Ogutu H, Wakasiaka S, Malogo R, et al. Reasons for ineligibility in phase 1 and 2A HIV vaccine clinical trials at Kenya AIDS vaccine initiative (KAVI), Kenya. PloS one. 2011;6(1):e14580. pmid:21283743; PubMed Central PMCID: PMC3024980.
- 3. Drain PK, Hyle EP, Noubary F, Freedberg KA, Wilson D, Bishai WR, et al. Diagnostic point-of-care tests in resource-limited settings. The Lancet infectious diseases. 2014;14(3):239–49. pmid:24332389; PubMed Central PMCID: PMC4016042.
- 4. Qin X, Liu YP, Lin FQ, Yi Z, Xu MH, Li RL, et al. [Comparison between Coulter LH-750 and URIT-12 for measuring hemoglobin concentration]. Nan Fang Yi Ke Da Xue Xue Bao. 2008;28(12):2196–8. Epub 2008/12/31. pmid:19114355.
- 5. Gomez-Simon A, Navarro-Nunez L, Perez-Ceballos E, Lozano ML, Candela MJ, Cascales A, et al. Evaluation of four rapid methods for hemoglobin screening of whole blood donors in mobile collection settings. Transfusion and apheresis science: official journal of the World Apheresis Association: official journal of the European Society for Haemapheresis. 2007;36(3):235–42. pmid:17556020.
- 6. Teruya SL, Gil HR, Teresi JA, Kong J, Eimicke J, Helmke S, et al. Facilitating clinical trials of anemia in older adults: a point-of-care system to measure hemoglobin in the home and its agreement with a hospital core laboratory. J Am Geriatr Soc. 2009;57(12):2362–4. Epub 2010/02/20. pmid:20169635.
- 7. Ghys T, Malfait R, J VANdB. Performance evaluation of the Sysmex XS-1000i automated haematology analyser. International journal of laboratory hematology. 2009;31(5):560–6. pmid:18616754.
- 8. Nutritional anaemias. Report of a WHO scientific group. World Health Organization technical report series. 1968;405:5–37. pmid:4975372
- 9. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–10. pmid:2868172.
- 10. Cable RG, Steele WR, Melmed RS, Johnson B, Mast AE, Carey PM, et al. The difference between fingerstick and venous hemoglobin and hematocrit varies by sex and iron stores. Transfusion. 2012;52(5):1031–40. pmid:22014071; PubMed Central PMCID: PMCPMC3623687.
- 11. Patel AJ, Wesley R, Leitman SF, Bryant BJ. Capillary versus venous haemoglobin determination in the assessment of healthy blood donors. Vox Sang. 2013;104(4):317–23. pmid:23294266; PubMed Central PMCID: PMCPMC3633672.
- 12. Gelderblom HC, Beld MG. Hepatitis C virus RNA quantification by the COBAS AmpliPrep/COBAS TaqMan System: averages do not tell the whole story. J Clin Virol. 2007;39(4):326–7. pmid:17600765.
- 13. Gelderblom HC, Menting S, Beld MG. Clinical performance of the new rRoche COBAS TaqMan HCV Test and High Pure System for extraction, detection and quantitation of HCV RNA in plasma and serum. Antiviral therapy. 2006;11(1):95–103. pmid:16518965.
- 14. Pavie J, Rachline A, Loze B, Niedbalski L, Delaugerre C, Laforgerie E, et al. Sensitivity of five rapid HIV tests on oral fluid or finger-stick whole blood: a real-time comparison in a healthcare setting. PloS one. 2010;5(7):e11581. Epub 2010/07/27. pmid:20657834; PubMed Central PMCID: PMC2906506.
- 15. Piwowar-Manning EM, Tustin NB, Sikateyo P, Kamwendo D, Chipungu C, Maharaj R, et al. Validation of rapid HIV antibody tests in 5 African countries. J Int Assoc Physicians AIDS Care (Chic Ill). 2010;9(3):170–2. Epub 2010/06/10. doi: 9/3/170 [pii] pmid:20530471.
- 16. Kilembe W, Keeling M, Karita E, Lakhi S, Chetty P, Price MA, et al. Failure of a novel, rapid antigen and antibody combination test to detect antigen-positive HIV infection in African adults with early HIV infection. PloS one. 2012;7(6):e37154. pmid:22715363; PubMed Central PMCID: PMC3371037.
- 17. Wolpaw BJ, Mathews C, Chopra M, Hardie D, de Azevedo V, Jennings K, et al. The failure of routine rapid HIV testing: a case study of improving low sensitivity in the field. BMC health services research. 2010;10:73. pmid:20307310; PubMed Central PMCID: PMC2851712.
- 18. Hurly DS, Buhrer-Skinner M, Badman SG, Bulu S, Tabrizi SN, Tarivonda L, et al. Field evaluation of the CRT and ACON chlamydia point-of-care tests in a tropical, low-resource setting. Sexually transmitted infections. 2014;90(3):179–84. pmid:24337733.
- 19. Banoo S, Bell D, Bossuyt P, Herring A, Mabey D, Poole F, et al. Evaluation of diagnostic tests for infectious diseases: general principles. Nat Rev Microbiol. 2006;4(9 Suppl):S21–31. Epub 2006/10/13. pmid:17034069.
- 20. Fontela PS, Pant Pai N, Schiller I, Dendukuri N, Ramsay A, Pai M. Quality and reporting of diagnostic accuracy studies in TB, HIV and malaria: evaluation using QUADAS and STARD standards. PloS one. 2009;4(11):e7753. Epub 2009/11/17. pmid:19915664; PubMed Central PMCID: PMC2771907.
- 21. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Standards for Reporting of Diagnostic Accuracy. Clinical chemistry. 2003;49(1):1–6. pmid:12507953.
- 22. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, et al. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clinical chemistry. 2003;49(1):7–18. pmid:12507954.
- 23. Karita E, Ketter N, Price MA, Kayitenkore K, Kaleebu P, Nanvubya A, et al. CLSI-derived hematology and biochemistry reference intervals for healthy adults in eastern and southern Africa. PloS one. 2009;4(2):e4401. Epub 2009/02/07. pmid:19197365; PubMed Central PMCID: PMC2632744.
- 24. Jani IV, Sitoe NE, Chongo PL, Alfai ER, Quevedo JI, Tobaiwa O, et al. Accurate CD4 T-cell enumeration and antiretroviral drug toxicity monitoring in primary healthcare clinics using point-of-care testing. AIDS (London, England). 2011;25(6):807–12. pmid:21378535.
- 25. Muenchhoff M, Madurai S, Hempenstall AJ, Adland E, Carlqvist A, Moonsamy A, et al. Evaluation of the NucliSens EasyQ v2.0 assay in comparison with the Roche Amplicor v1.5 and the Roche CAP/CTM HIV-1 Test v2.0 in quantification of C-clade HIV-1 in plasma. PloS one. 2014;9(8):e103983. pmid:25157919; PubMed Central PMCID: PMC4144839.