Evaluation of PIMA™® Point of Care Technology for CD4 T Cell Enumeration in Kenya

CD4+ T cell enumeration is used to determine eligibility for antiretroviral therapy (ART) and to monitor the immune status of HIV-positive patients; however, many patients do not have access to this essential diagnostic test. Introducing point of care (POC) testing may improve access. We have evaluated Alere’s PIMA™, one such POC device, against conventional CD4+ testing platforms to determine its performance and validity for use in Kenya. In our hands, Alere PIMA™ had a coefficient of variability of 10.3% and of repeatability of 175.6 cells/µl. It differed from both the BD FACSCalibur™ (r2 = 0.762, mean bias −64.8 cells/µl), and the BD FACSCount™ (r2 = 0.874, mean bias 7.8 cells/µl). When compared to the FACSCalibur™ at a cutoff of 350 cells/µl, it had a sensitivity of 89.6% and a specificity of 86.7% in those aged 5 years and over (Kw = 0.7566). With the BD FACSCount™, it had a sensitivity of 79.4% and a specificity of 83.4% in those aged 5 years and over (Kw = 0.7790). The device also differed from PARTEC Cyflow™ (r2 = 0.781, mean bias −24.2 cells/µl) and GUAVA™ (r2 = 0.658, mean bias −0.3 cells/µl) platforms, which are used in some facilities in Kenya. We conclude that with refinement, Alere PIMA™ technology has potential benefits for HIV-positive patients. This study highlights the difficulty in selecting the most appropriate reference technology for technical evaluations.

Introduction CD4+ T cell quantification by flow cytometry is considered accurate, precise and reproducible [1,2]. A CD4+ T cell count is positively linked to long-term survival rates and indicates the level of immunosuppression [3][4][5]. During testing, patients older than five years of age are given an absolute CD4+ T cell count, which is determined as CD4+ cells/mL of blood. For patients five years old and younger, it is necessary to measure the percentage of CD4+ T cells among all lymphocytes [6].
Despite the fact that CD4+ T cell enumeration is essential in the initiation of antiretroviral therapy and the monitoring of care and treatment in Kenya, many HIV-positive patients still do not have reliable access to these services [7,8]. CD4 testing is often only available at centralized laboratories with significant infrastructure investments and highly skilled laboratory technicians [7]. Laboratory networking has improved access somewhat, but sample transportation networks are still so poor that many patients are unable to access adequate and necessary CD4 testing.
Currently, CD4 testing using flow cytometry technology is available in many central and regional laboratories in Kenya using BD FACSCount TM or BD FACSCalibur TM (Becton Dickinson, Franklin Lakes, NJ, USA), PARTEC Cyflow TM (Partec GmbH, Munster, Germany), or GUAVA TM (EMD Millipore Corporation, Billerica, MA, USA) platforms. Unfortunately, this combination of technologies is insufficient to provide CD4 testing to all patients who need them. Long turn-around times for tests sent to central laboratories delay clinical decisions and put a considerable burden on patients. Conventional CD4 testing requires samples be transported in complicated and inefficient sample transportation networks and over long, rough roads. These transportation networks are limited and expensive, and are often compounded by the difficulties of short sample stability.
Fortunately, high quality Point-Of-Care (POC) diagnostic technologies are promising solutions to critical CD4 testing needs in areas without existing laboratory capacity or easy access to conventional CD4 testing laboratories [9][10][11][12][13]. For example, HIV rapid diagnostic tests have increased patient access to HIV diagnosis in the last decade [14]. POC diagnostic technologies offer several advantages in that they are generally small, robust, relatively low cost, require little infrastructure, and require minimum technical skills. Additionally, replacement devices can be sent to a site immediately to sidestep POC device downtime. CD4 POC technologies can ease human resource capacity shortages and testing backlogs at central laboratories by diverting samples that are customarily referred. Most importantly, CD4 POC testing can allow doctors to make treatment decisions faster and achieve significant improvements in patient health outcomes.
POC diagnostic technologies for HIV testing, viral load determination and CD4 enumeration are in various stages of development. PIMA TM (Alere Inc., Waltham, MA) is one such portable bench top fixed volume cytometer for CD4+ T cell enumeration. It uses cartridges, which, according to the manufacturer, have a long shelf life and are stable in conditions of high heat and humidity. It can be operated using external power or an onboard rechargeable battery, which has an eight-hour life. Although this technology is being used in some countries and has generated results comparable to already existing technologies [15][16][17][18][19], we wanted to evaluate the Alere PIMA TM CD4 POC technology in Kenya.

Study Setting
This study was done in nine health facilities offering CD4+ T cell enumeration: Kenya Medical Research Institute-Centre for Biotechnology Research and Development Nairobi (KEMRI-CBRD), Matibabu Foundation's Ukwala, University of Nairobi-Department of Immunology, St. Luke's Hospital Kaloleni, Mariakani District Hospital, Rongo District Hospital, Andersen Medical Centre, Kitale District Hospital and Alupe District Hospital. The sites provided from a range of settings: rural versus peri-urban or urban, high volume versus low volume sites, different day temperatures (ranging from 18uC to 38uC) and different humidity levels (ranging 25%-75%). The sites used different testing platforms (BD FACSCount TM , PARTEC Cyflow TM , GUAVA TM and BD FACSCalibur TM ) for routine CD4+ T enumeration.

Study Participants
All patients attending the selected facilities for HIV treatment and care were eligible for this study. Only patients who provided written informed consent were enrolled in the study. In the case of children below 18 years of age, a parent or legal guardian made the decision regarding participation and gave written informed consent. A total of 1,549 patients were recruited from the nine study sites. Data on gender was available for 1,482 patients: 984 (66.3%) were female, while 498 (33.7%) were male. The median age was 36 years (range 1-75 years old) with an approximately normal distribution; 68 patients were aged 18 years or less. At least 49.7% of the patients were on antiretroviral therapy. Qualified and trained laboratory technicians conducted all tests.

Study Design
In this methods comparison study, venous and capillary blood specimens were collected consecutively from all eligible patients presenting at the health clinics included in the study who agreed to participate in the study through informed consent. Demographic data, CD4+ T cell count, date of clinic visit and antiretroviral (ARV) use were all recorded in a structured questionnaire and entered into an Access database. The data from all testing sites was similarly uploaded remotely to a central server. This study was reviewed and approved by the Kenya Medical Research Institute Ethical Review Committee (Protocol No. SSC1880). For all patients under the age of 18 years, a parent or guardian gave written informed consent. Children between the ages of 13 and 17 years also gave assent. Patients were only provided with CD4+ T cell results obtained from the conventional CD4 testing platforms for further clinical management.

Laboratory Procedures
CD4 testing using each of the available devices was done according to manufacturers' instructions. Whole blood was used for the BD FACSCount TM , the BD FACSCalibur TM , the PARTEC Cyflow TM , the Alere PIMA TM device and the GUAVA TM platforms. Capillary blood was also used for the Alere PIMA TM device. Both internal quality assurance (IQA) and external quality assurance (EQA) schemes were implemented for the platforms according to manufacturers '  Additionally, all technologists are trained annually in good laboratory practice, immunophenotyping for flow cytometry, reverse pipetting, biosafety and good phlebotomy practice. All devices used in this study passed EQA prior to study commencement and all staff conducting testing were trained to perform routine IQA. Control cartridges (both high and low) were run on the PIMA device every morning before tests. PIMA devices reporting errors were not used for this work.

Data Analysis
The results of the evaluation were analyzed using standard statistical methods. The absolute CD4+ T Cell counts derived from Alere PIMA TM device were compared with those derived from existing technologies by calculating the coefficient of determination (r 2 ) and conducting regression analysis using STATA v. 12 for Mac OSX. To determine interchangeability between the device and exisiting platforms, Bland-Altman analysis and Lin's concordance correlation coefficient (rhoC) were used. For the former analysis, the bias was defined as the mean difference between two methods. The limits of agreement (LoA) between the methods compared were calculated as the mean 61.96 Standard Deviations (SD) of the differences between the results obtained. Confidence intervals for bias and for limits of agreement were calculated using formulae previously described by Bland and Altman [20]. The x axis on each Bland-Altman plot was the average value of the two methods while the y axis was the difference between the two methods. For Lin's concordance correlation, the coefficient of determination was derived by squaring the coefficient of determination r and was used to quantify the percentage variation of the dependent variable that could be attributed to the variations in the independent variable of the correlation equation. Both the coefficient of variation and the coefficient of repeatability were also calculated for the Alere PIMA TM device. The coefficient of variation was used to define the instrument precision and was calculated as the (standard deviation6100)/mean of a set of repeated measurements on one sample using one instrument. The coefficient of repeatability, a measure of test-retest precision was defined as the variation in duplicate measurements for several samples, taken by a single technician using the same instrument under the same conditions. It was calculated as 1.96 times the standard deviation of the differences between the two measurements. To determine the effect of the platform under evaluation on eligibility of patients for antiretroviral therapy, Cohen's weighted kappa statistic (Kw) was used.

Results
Since we evaluated Alere PIMA TM in the context of existing technologies. The coefficients of repeatability and the mean bias between Alere PIMA TM and existing technologies are all summarized in Table 1.
We began by evaluating the correlation of the platform in most common usage (BD FACSCount TM ) with the gold standard (BD FACSCalibur TM ) using blood samples from 312 patients collected at KEMRI CBRD. The coefficient of determination (r 2 ) was 0.885. The mean bias between the two platforms was 276.5 cells/ ml (95% LoA: 2316.0, 163.0). Figs. 1a. & 1b. provide concordance and limits of agreement plots for this relationship.
We then compared Alere PIMA TM against the gold standard BD FACSCalibur TM . A total of 396 patients' whole blood specimens (339 from KEMRI CBRD and 57 from the University of Nairobi Immunology Laboratories) were tested on the BD FACSCalibur TM and the Alere PIMA TM platforms. The coefficient of determination (r 2 ) in this case was 0.762. The mean bias between the two platforms was 264.8 cells/ml (95% LoA 2332. 5 According to manufacturer instructions, Alere PIMA TM will mainly use capillary blood samples. We were interested in finding out how capillary blood samples on Alere PIMA TM compare with whole blood tests on the same platform and other platforms. Of the 822 patients enrolled for the comparison between Alere PIMA TM and BD FACSCount TM , 521 provided an additional capillary blood specimen for the Alere PIMA TM . Of these, Alupe Sub District Hospital provided 206, Kitale District Hospital provided 170, Mariakani District Hospital provided 39 and Rongo District Hospital provided 106 samples. An analysis of this data revealed a concordance coefficient of determination (r 2 ) of 0.738. In this approach the mean difference was 8.6 cells/ml (95% LoA 2235.4, 252.7). Figs. 2e. & 2f. are a graphical representation of these findings.
When whole blood was compared with capillary blood from the same patient on the same Alere PIMA TM machine, the results were as follows: The mean bias was 27.7 cells/ml (95% LoA 2236.2, 220.7). The coefficient of determination (r 2 ) was 0.821 (Figs. 3a. & 3b.).
The precision of the Alere PIMA TM platform was determined by running the same sample 16 times using cartridges from the same batch. The mean CD4 count from that sample was 275.9 cells/ml, with a standard deviation of 28.5 cells/ml. The coefficient of variation was therefore 10.3%. To understand the reproducibility of results using the Alere PIMA TM platform, All 211 patients enrolled in the study from Alupe Sub District Hospital provided blood in EDTA tubes each of which was tested twice, on the same Alere PIMA TM device. The mean difference between the tests was 6.9 cells/ml, with a coefficient of repeatability of 175.6 cells/ml.
In comparison, blood collected from 197 patients at Alupe Sub District Hospital was tested using the BD FACSCount TM platform to determine the coefficient of repeatability. The mean difference on the device was 2.2 cells/ml, and the coefficient of repeatability was 66.0 cells/ml.
In Kenya, the Ministry of Health has expressed a desire to identify one point of care platform that can be used interchange-  ably with all the existing technologies. To that end, we compared the Alere PIMA TM to additional CD4 testing technologies to determine interchangeability with various other platforms. In a comparison using capillary blood for Alere PIMA TM and whole blood for PARTEC Cyflow TM comparison, the coefficient of determination (r 2 ) was 0.852 while the mean bias between these two platforms was 210.0 cells/ml (95% LoA 2261.4, 241.4, n = 162). When whole blood specimens (n = 407; 166 patient samples from Andersen Medical Centre and 241 patient samples from University of Nairobi Immunology Department) were used for both platforms the coefficient of determination (r 2 ) was 0.781, while the mean bias between these two CD4 enumeration technologies was 224.2 (95% CI 2277.6-229.3) cells/ml.
Alere PIMA TM was assessed against the GUAVA TM platform with the former using capillary blood samples and the latter whole blood samples in 176 patients (165 from ACK St Luke's Hospital Kaloleni and 11 from Andersen Medical Centre). In this exercise, the coefficient of determination was determined to be 0.681. The mean bias between the machines was 23.9 (95% LoA 2329.6, 281.9). When whole blood was used in this comparison for both platforms, the coefficient of determination (r 2 ) was 0.658. The mean difference was 20.3 (95% LoA 2315.0, 315.6, n = 191) cells/ml. Finally, we assessed whether the sample type might affect the CD4 test result on the Alere PIMA TM platform. 840 samples of each type were compared. r 2 was 0.821. The mean bias between whole blood and capillary blood samples when tested with the PIMA device was 7.7 cells/ml (95% LoA 2220.7, 236.1).
In Kenya, HIV+ patients are eligible for ART initiation when their CD4+ T cell count falls to or below 350 cells/ml. We, therefore, used this threshold to determine the sensitivity and specificity of PIMA TM using either the FACSCalibur TM (which is the gold standard) or the BD FACSCount TM (which is the most commonly available platform).
When compared to the FACSCalibur TM , Alere PIMA TM had a sensitivity of 89.6% and a specificity of 86.7% in those aged 5 years and over (n = 389, Kw = 0.7566 ). On the other hand, when compared with the BD FACSCount TM , Alere PIMA TM had a sensitivity of 79.4% and a specificity of 83.4% in those aged 5 years and over (n = 813, Kw = 0.7790; Table 2).
When we lowered the threshold to 200 cells/ml, the sensitivity of Alere PIMA TM was 86.7% while the specificity was 94.12% when compared with FACSCalibur TM (n = 389 people aged 5 or older, Kw = 0.7619). Against the FACSCount, the sensitivity and specificity were 83.0% and 98.2% respectively (n = 813, Kw = 0.8422, Table 3).
To contextualize these findings, we compared patient eligibility for ART using BD FACSCount TM with BD FACSCalibur as the gold standard at a threshold of 350 cells/ml. The BD FACS-Count TM had a sensitivity of 93.8% and a specificity of 82.4% in those aged 5 years or older (n = 305, Kw = 0.7528). When this threshold was dropped to 200 cells/ml, the sensitivity and specificity became 94.4% and 93.2% respectively (n = 305, Kw = 0.8040).
Finally, we compared the ART eligibility classification rates between the Alere Pima TM and two additional CD4 testing technologies at a threshold of 350 cells/cells/ml (Table 3). With GUAVA TM , Kw was 0.7630 (n = 189) while for PARTEC Cyflow TM , it was 0.7875 (n = 400).

Discussion
By the end of 2011, Kenya had approximately 500,000 adults and 40,000 children on ART and, therefore, in need of routine immunological and virological monitoring. HIV testing and ART are available at thousands of health care facilities nationwide, but CD4+ T cell enumeration for ART initiation eligibility and immunological monitoring is, unfortunately, available in only about 200 out of the more than 8000 health care facilities. Laboratory sample referral networking is well established in a few areas, but for a majority of patients, access to CD4 testing is a challenge. In fact, although health care facilities are able to provide ART and monitor clinical outcomes and the side effects of medication, they are often unable to routinely initiate patients on ART due to the inability to access CD4+ T cell enumeration services.
Many of the existing technologies are expensive and not readily accessible. If a new point of care technology can provide reliable and accurate results, and is interchangeable with existing platforms, then it can be used not only to reduce costs and improve access but also to standardize the testing service. The Kenya Ministry of Health believes that it can benefit from economies of scale in a standardized test environment.
It should be pointed out that, by their very nature, point of care tests may not provide the same sensitivity, specificity and accuracy as the ''gold standard'' reference tests. The trade off is that they can provide access to a CD4 T Cell enumeration service where none exists. To set the pace for this evaluation, we sought to determine whether the commonest CD4 enumeration platform (BD FACSCount TM ) in Kenya is interchangeable with the gold standard (BD FACSCalibur TM ). The coefficient of determination suggested that the agreement between the two platforms was less than perfect. In fact, with a mean bias of 76.5 cells/ml, these platforms are not interchangeable. Differences between the two platforms have been reported in the literature, and may be partially explained by the fact that the BD FACSCalibur TM allows for some subjectivity when laboratory technicians manually gate the CD4+ T cell population.
These differences are clinically important regardless of whether the threshold for ART initiation is set at 350 cells/ml or at 200 cells/ml. If the BD FACSCalibur TM is to be considered the most accurate CD4+ T cell enumeration platform available in Kenya, then our data suggests significant misclassification of patients by the BD FACSCount TM . Many patients who are currently on ART due to their CD4 result using the BD FACSCount TM would not have been initiated on ART when tested with the BD FACSCalibur TM . We recommend that patients remain faithful to one platform during their care. When evaluated for precision, Alere PIMA TM gave a coefficient of variability of 10.3%. This level of precision has been reported before [21]. In our opinion, this precision is less than desirable, and much lower than that reported for either the BD FACSCalibur TM or the BD FACSCount TM [22].
There was minimal bias using either capillary blood or whole blood samples on Alere PIMA TM . This is a positive finding, suggesting that this platform can be used in a variety of settings. Additional, the reproducibility of results using whole blood on the same Alere PIMA TM platform was very high, another encouraging finding.
Previous studies have reported that the Alere PIMA TM device is comparable and interchangeable with the existing CD4 enumeration platforms [17,18,21,[23][24][25][26]. In our hands, we found no such thing. None of the platforms were interchangeable with the Alere PIMA TM platform. For instance, the sensitivity and specificity of Alere PIMA TM was 83.1% and 92.2% respectively when BD FACSCalibur TM was used as the gold standard and 79.4% and 83.4% respectively when BD FACSCount TM was considered as the gold standard. Even when compared with GUAVA TM , Cohen's kappa was 0.7630 while for PARTEC Cyflow TM , it was 0.7875.
In our hands, Alere PIMA TM was not as reliable as previously published literature suggests [18,21,26]. For example, the Alere PIMA TM and BD FACSCount TM coefficients of repeatability were very different, at 179.2 cells/ml and 67.4 cells/ml respectively. It appears that tests on the BD FACSCount TM are more repeatable. Alere PIMA TM technology will need to undergo further refinement to improve repeatability.
From our findings, PIMA tended to misclassify patients regardless of the platform used for comparison. This misclassification was generally in favor of undertreatment, and it ranged from 10.4% (17/389) when compared with the BD FACSCalibur TM to 15.3% (59/385) when compared with the BD FACSCount TM . This would suggest that if Alere PIMA TM technology is put to widespread use, patients who genuinely need treatment may not qualify for ART, probably only getting ART when already very sick. This is a concern that can be resolved by further improvement of the technology.
For this evaluation laboratory technicians were trained for up to 2 hours on the use of the Alere PIMA TM device. Even with minimal training, the laboratory technicians delivered quality results. It is conceivable that if and when the Alere PIMA TM is deployed in lower levels of health care facilities, the operators would likely be nurses, nurse aides or less technically skilled health care workers. Evaluating this device in the field with such operators would yield additional useful performance information.
The Alere PIMA TM has characteristics of particular interest for use in small health centers located in resource-limited settings. For example, it uses a rechargeable battery rather than require constant electricity. With a turnaround time of 20 minutes per test, the Alere PIMA TM can run up to 20 tests per day on venous or capillary blood. Furthermore, the Alere PIMA TM can store up to 1,000 tests and prints results on a thermal paper roll for recording in patient charts. Using this POC device, personnel can conduct CD4 testing with minimal technical training at remote health facilities. Finally, the Alere PIMA TM reagents have a long shelf life and do not require refrigeration.

Conclusion
We conclude that with additional refinement, the potential benefits of the Alere PIMA TM technology for HIV-positive patients cannot be exaggerated. It can expand access to CD4 testing, particularly in rural settings whose needs are currently unmet by existing laboratory testing networks. Additionally, this study highlights the difficulty in selecting the most appropriate 'gold standard' or reference technology for technical evaluations.