Diagnostic accuracy of the Panbio COVID-19 antigen rapid test device for SARS-CoV-2 detection in Kenya, 2021: A field evaluation

Background Accurate and timely diagnosis is essential in limiting the spread of SARS-CoV-2 infection. The reference standard, rRT-PCR, requires specialized laboratories, costly reagents, and a long turnaround time. Antigen RDTs provide a feasible alternative to rRT-PCR since they are quick, relatively inexpensive, and do not require a laboratory. The WHO requires that Ag RDTs have a sensitivity ≥80% and specificity ≥97%. Methods This evaluation was conducted at 11 health facilities in Kenya between March and July 2021. We enrolled persons of any age with respiratory symptoms and asymptomatic contacts of confirmed COVID-19 cases. We collected demographic and clinical information and two nasopharyngeal specimens from each participant for Ag RDT testing and rRT-PCR. We calculated the diagnostic performance of the Panbio™ Ag RDT against the US Centers for Disease Control and Prevention’s (CDC) rRT-PCR test. Results We evaluated the Ag RDT in 2,245 individuals where 551 (24.5%, 95% CI: 22.8–26.3%) tested positive by rRT-PCR. Overall sensitivity of the Ag RDT was 46.6% (95% CI: 42.4–50.9%), specificity 98.5% (95% CI: 97.8–99.0%), PPV 90.8% (95% CI: 86.8–93.9%) and NPV 85.0% (95% CI: 83.4–86.6%). Among symptomatic individuals, sensitivity was 60.6% (95% CI: 54.3–66.7%) and specificity was 98.1% (95% CI: 96.7–99.0%). Among asymptomatic individuals, sensitivity was 34.7% (95% CI 29.3–40.4%) and specificity was 98.7% (95% CI: 97.8–99.3%). In persons with onset of symptoms <5 days (594/876, 67.8%), sensitivity was 67.1% (95% CI: 59.2–74.3%), and 53.3% (95% CI: 40.0–66.3%) among those with onset of symptoms >7 days (157/876, 17.9%). The highest sensitivity was 87.0% (95% CI: 80.9–91.8%) in symptomatic individuals with cycle threshold (Ct) values ≤30. Conclusion The overall sensitivity and NPV of the Panbio™ Ag RDT were much lower than expected. The specificity of the Ag RDT was high and satisfactory; therefore, a positive result may not require confirmation by rRT-PCR. The kit may be useful as a rapid screening tool only for symptomatic patients in high-risk settings with limited access to rRT-PCR. A negative result should be interpreted based on clinical and epidemiological information and may require retesting by rRT-PCR.


Introduction
The COVID-19 pandemic has had major negative socioeconomic effects [1]. Prompt diagnosis of SARS-CoV-2 infection is critical in controlling the pandemic [2]. Real-time reverse transcription-polymerase chain reaction (rRT-PCR) remains the reference standard for the diagnosis of SARS-CoV-2 infection [3]. However, rRT-PCR has limitations, including the need for specialized laboratories, equipment, and staff, longer turnaround time (TAT) for clinical decision making and prevention measures, high cost, and erratic supply of reagents to conduct SARS-CoV-2 rRT-PCR due to increased demand [3][4][5][6]. These limitations have increased the demand for more rapid, cheaper, and easy to perform testing methods [7].
Prompt identification of infected individuals is key in initiation of proper management of cases and the implementation of interventions to limit transmission [2]. Antigen rapid diagnostic tests (Ag RDTs) increase patients' access to diagnosis of SARS-CoV-2 infection as they can be used as point of care tests thereby providing results more prompty. The World Health Organization (WHO) recommends Ag RDTs with a minimum sensitivity and specificity �80% and �97%, respectively where rRT-PCR is unavailable or is associated with long TAT [8].
The performance of Ag RDTs depends on the presence of symptoms, duration since onset, viral load and the disease prevalence [9]. At the time of study conception, Panbio™ Ag RDT was one of two rapid tests approved for SARS-CoV-2 testing in our setup and to the best of our knowledge had not been evaluated. We evaluated the performance of the Panbio™ Ag RDT compared to the reference standard rRT-PCR. The Panbio™ Ag RDT is authorized in Kenya and received an emergency use license (EUL) from WHO [10]. We also assessed the operational characteristics of the Panbio™ Ag RDT in order to identify any user challenges the in the field setting.

Methods
We conducted a prospective cross-sectional evaluation of the Panbio™ Ag RDT in 11 sites in Kenya between March and July 2021. The sites were located in Nairobi County (Nairobi remand Prison, Langata Women's Prison, and Tabitha level 2 Clinic), Kiambu County (Kihara Sub-county Hospital), Nakuru County (Nakuru Prison), Nyeri County (Nyeri level 5 Hospital and Nyeri Prison), Kisumu County (Kisumu Prison-Kodiaga), Mombasa County (Coast General Teaching and Referral Hospital) and Siaya County (Siaya County Referral Hospital and St. Elizabeth Mission Hospital). The protocol was reviewed and approved by the Kenya Medical Research Institute's (KEMRI) scientific ethical review unit (SERU). The US Centers for Disease Control and Prevention (CDC) institutional review board provided non-research determination approval. Further administrative approvals were obtained from the Ministry of Health Kenya, the National Commission for Science, Technology, and Innovation (NACOSTI) Kenya, the Kenya Prisons Service, and the county governments of Nakuru, Nyeri, Mombasa, Kiambu, and Siaya. Written informed consent was obtained for all enrolled participants. Written parental informed consent and assent were obtained for participants aged <18 years. Individuals who declined to participate, including prisoners were offered routine COVID-19 services as per national guidelines.

Clinical case definition for enrollment
We enrolled persons of any age who presented to our study sites with respiratory symptoms (symptomatic participants) that met any of three case definitions, including acute respiratory infection (ARI,) defined as cough or difficulty in breathing or sore throat or coryza, with the onset of symptoms <2 weeks; severe acute respiratory infection (SARI) defined as fever or temperature �38˚C and cough, with onset of symptoms <10 days and requiring hospitalization; or influenza-like illness (ILI) defined as temperature �38˚C and cough, with onset of symptoms <10 days. Body temperature measurement was done using infrared thermometers placed 3 to 5 centimeters from the temple. We also enrolled asymptomatic close contacts of individuals with confirmed SARS-CoV-2 infection exposed between two days before and 14 days after symptoms onset or confirmation of infection. Individuals exposed to persons with confirmed SARS-CoV-2 infection in the following setups were considered close contacts in the following scenarios: working in close proximity or sharing the same environment; face-toface contact within 1 meter for more than 15 minutes; traveling together in any kind of conveyance; living in the same household; healthcare-associated exposure, including providing direct care for COVID-19 patients, visiting patients or staying in the same close environment.

Data and specimen collection
Individuals who met the case definition and consented were enrolled and assigned unique identifiers. We used pretested questionnaires on Android 1 smartphones running Comm-Care 1 Application to collect demographic and clinical information. Data was stored in a secured cloud server. Clinical information collected included the range and duration of respiratory symptoms, presence of comorbidities, and reported history of exposure to confirmed COVID-19. Two nasopharyngeal swabs were collected by a surveillance officer (nurse, clinical officer, or laboratory technologist) trained to collect respiratory specimens from each of the nostrils a few seconds apart. The first specimen was collected using the swab provided in the Panbio™ Ag RDT kit. The second specimen was collected using a polyester-tipped aluminum shafted swab and immediately placed in standard viral transport media prepared in the testing laboratory. The transport media was stored at +2 to +8˚C before specimen collection and during shipment to the CDC-supported KEMRI laboratories in Nairobi or Kisumu for rRT-PCR testing. Aliquots of the specimens were then archived at -80˚C.

Rapid SARS-CoV-2 antigen testing
Rapid testing was conducted using the Panbio™ Ag RDT (Abbott Rapid Diagnostics, US. Ref. 41FK10) per manufacturer instructions [11] by surveillance officers. The Panbio™ Ag RDT is a lateral flow immunoassay that detects the nucleocapsid (N) protein in nasal and nasopharyngeal specimens. Results were read and documented independently by two surveillance officers within 15 to 20 minutes. Positive, negative, and invalid results were interpreted according to the manufacturer's recommendations. Each reader documented the test results in the electronic questionnaire independently. In the event the two readers documented different results, the final result was indicated as indeterminate.

Assessing the operational characteristics of the Panbio™ Ag RDT kit
We assessed operational characteristics of the Panbio™ Ag RDT in the field setting including clarity of kit instructions, technical complexity or ease of use, and the ease of interpretation of results via a standardized questionnaire administered to users on site.

rRT-PCR testing in the laboratory
According to the manufacturer's instructions, total nucleic acid material was extracted from 200μL of the nasopharyngeal specimen using either the Ambion MagMax Total RNA isolation kit (Thermo Fisher Scientific, Greenville, North Carolina, USA) performed on a semi-automated KingFisher flex machine (Thermo Fisher Scientific) or Standard M Spin-X Viral RNA Extraction Kit (SD biosensor) according to the manufacturer's instructions. We then eluted 60uL of total RNA from the extracted sample. The SARS-CoV-2 rRT-PCR was performed using the TaqPath 1-step Multiplex rRT-PCR master mix (Thermo Fisher Scientific) or using the GoTaq 1 Probe 1-Step RT-qPCR System (Promega A6120). Both systems are compatible with the rRT-PCR CDC IDT kit. Each of the 15μL reaction volumes contained 5μL of master mix, 8.5μL of nuclease-free water, and 1.5μL of CDC-IDT primer/probe mix for each of the N1, N2, and RNP3 genes separately. The eluate was then amplified using QuantStudio™ 5 Real-Time PCR System, 0.1ml block (Thermo Fisher Scientific), and analyses were done using QuantStudio 3 and 5 Real-Time PCR System Software version 1.5.1 (Thermo Fisher Scientific). SARS-CoV-2 assay targets, N1 and N2 (N1 and N2) were run simultaneously with the human ribonuclease P gene (RNP) control to monitor the quality of the nucleic acid extraction, specimen quality, and presence of reaction inhibitors for assay performance. The positive, negative, and human specimen controls were included in all assays.
When all controls exhibited expected performance, a positive result for SARS-CoV-2 was considered, if all assay amplification curves crossed the threshold line within Ct <40. If SARS-CoV-2 (N1 and N2) assay results were negative, then the test result was reported as negative. If only one target (N1 and N2) were positive, then it was designated as inconclusive, and retesting was conducted. If upon repeating the test, the results remain inconclusive, then the final result was reported as inconclusive. Results were relayed back to the specific sites for patient management.

Sample size calculation
To evaluate the diagnostic performance of the Panbio™ Ag RDT device, we assumed a prevalence of 10% among symptomatic individuals and 5% in asymptomatic contacts of laboratoryconfirmed persons based on current SARS CoV 2 local surveillance data. We assumed a sensitivity of 90% for the Panbio™ Ag RDT with 7.5% margin of error, and after adjusting by 10% for other concerns we anticipated enrolling 770 symptomatic patients and 1,540 close contacts of confirmed cases.

Statistical analysis
Statistical analysis was conducted using Stata version 15.1 (College Station, Texas). Measures of central tendency and dispersion were calculated for continuous variables such as age, days since onset of symptoms, temperature, and cycle threshold (Ct) values. We calculated frequencies and proportions for categorical variables such as age group, occupation, Ct value cut-offs and the operational characteristics of the Ag RDT. The 95% confidence intervals for categorical variables were calculated using the normal approximation method (Wald's test). We compared proportions between the symptomatic and asymptomatic groups by calculating their Z scores. P-values <0.05 were considered statistically significant.
We calculated the sensitivity, specificity, and positive and negative predictive values for the Panbio™ Ag RDT against rRT-PCR as the reference standard. Stratified analysis of performance was done by age, day's post-onset of symptoms, and Ct values. The measure of agreement between Ag RDT and rRT-PCR results was evaluated using Cohen's Kappa statistic. We interpreted the strength of agreement as slight agreement (κ, 0-0.2), fair (κ, 0.21-0.4), moderate (κ, 0.41-0.6), substantial (κ, 0.61-0.8), and almost perfect (κ, 0.81-1.0) as described by Landis et al. [12].

Enrolled participant characteristics
We enrolled 2,279 participants, where 38.9% had at least one respiratory symptom. The median age was 31.0 years (range <1 month-103 years), and over half (60.9%) were male ( Table 1). The median duration post-onset of symptoms among symptomatic participants was three days, with the majority (67.8%) presenting within 5 days.

Laboratory findings
Panbio™ Ag RDT test results.. Among the 2,277 tested using the Panbio™ Ag RDT device, 12.4% were positive (Fig 1). There were no invalid or indeterminate Ag RDT results. The positivity was significantly higher among symptomatic individuals (18.7%) compared with the asymptomatic (8.4%,) individuals.
rRT PCR test results. We tested 2,277 specimens by rRT-PCR, 24.2% tested positive and 1.4% were inconclusive (Fig 1). The overall median Ct value of the 551 rRT-PCR positive results was 30.

Evaluation of the user's experience of the Panbio™ Ag RDT
A total of 17 surveillance officers comprising 10 (59%) clinicians and 7 (41%) laboratory staff completed the survey. All 17(100%) surveillance officers found the test procedure easy to perform, and the manufacturers' instructions were found to be clear and easy to follow. Four (23.5%) of the surveillance officers noted the difficulty in dispensing a sample onto the test device from the extraction tube when nasopharyngeal specimens were collected from individuals with thick mucus.

PLOS ONE
Diagnostic accuracy of the Panbio™ COVID-19 antigen rapid test device in Kenya, 2021

Discussion
We present findings of a field evaluation of the Panbio™ Ag RDT conducted among symptomatic patients with onset of symptoms up to 14 days and asymptomatic individuals who had contact with individuals with confirmed SARS-CoV-2 infection within two days before detection of the index case and up to 14 days after exposure. Our evaluation was conducted in the third COVID-19 wave in Kenya, however, the majority of our participants were enrolled towards the end of the wave [13,14] where the median national prevalence was 9.6%(IQR 7.9-13.1%) [15]. The proportion of symptomatic individuals enrolled (54%) was higher than observed in the Kenyan population [13]. The overall sensitivity was low (46.6%), detecting slightly less than half of the rRT-PCR confirmed cases, therefore, missing more than half and classifying them as not having SARS-CoV-2 infection when they were indeed infected. In our evaluation, sensitivity varied with the presence or absence of symptoms, days since onset of onset, and viral load. As expected, sensitivity was higher among symptomatic individuals compared to asymptomatic ones (60.6% vs. 34.7%), and those with higher viral loads compared to those with lower ones (Ct value �30 vs. >30). Sensitivity increased with more recent onset of infection, being higher in those with onset of symptoms less than 7 days compared to those with onset >7 days (61.6% vs. 47.1%). By duration since onset of infection, sensitivity was highest among those with onset between 3 and 5 days. Symptomatic individuals had significantly higher viral loads (Fig 2), and with the highest sensitivity (87.0%) observed where Ct values �30, exceeding the WHO recommendation.
The low overall sensitivity observed may be due to the high proportion (61.1%) of asymptomatic participants, who had significantly higher Ct values and therefore more likely to test negative by Ag RDT. Antigen RDTs perform better in individuals with higher viral loads [7].

PLOS ONE
Diagnostic accuracy of the Panbio™ COVID-19 antigen rapid test device in Kenya, 2021 Sensitivity was lower than indicated in the manufacturer's product insert (61.6% vs. 91.1%) [11] and the WHO minimum recommendation (80%) [8]. This is likely due to the selection of specimens with high viral loads by the manufacturer's study for approval as hypothesized by Albert et al., 2021 in an evaluation where SARS-CoV-2 cultures were conducted [4].

PLOS ONE
Diagnostic accuracy of the Panbio™ COVID-19 antigen rapid test device in Kenya, 2021 observed could have been due to the relatively younger particpants (median age 31 years) enrolled in our evaluation compared to other studies with higher median age (39-51.5 years) [18,20]. A recent study demonstrated increasing viral load with increasing age [21] and resultant higher sensitivities. In addition, while we did not ascertain the variants in the specimens tested, the alpha variant and delta variants predominated the third wave in Kenya [22,23]. While some literature indicates that variants of concern (17,18) do not reduce the sensitivity and specificity of Ag RDTs, a recent study reported that sensitivity of the Panbio™ Ag RDT declined significantly in individuals infected with the alpha variant (53.0%) compared to those with non-alpha variants (89.0%) even after adjusting for viral load (p <0.002) [24] as was also demonstrated by Wertenauer et. al in the COVAG study [25]. Mutations occurring in the N protein of the Alpha variant may not be detected by the Panbio™ Ag RDT [26]. Laboratory studies have demonstrated a tenfold reduction in detection of the alpha variant by the Panbio™ Ag RDT [27]. Early laboratory data also demonstrated reduced sensitivity of other Ag RDTs in laboratory studies [28][29][30], while others have demonstrated no impact on performance [31].
Further clinical evaluations may be required to determine the impact of viral mutations on the performance on Ag RDTs. The sensitivity observed among symptomatic individuals was similar to that observed in symptomatic Mexican adults (58.1%, 95% CI: 54.9-61.3%) [32]. However, it was higher than reported in a multicenter evaluation conducted among symptomatic children (60.6% vs. 45.4%) with the onset of symptoms within five days [20].
Sensitivity was higher among symptomatic individuals compared to asymptomatic ones. Similar findings were observed in other evaluations with higher sensitivity among symptomatic individuals compared to asymptomatic in a study in Mexico (58.1% vs. 26.3%) [32], in Spain (80.4% vs. 56.6%) [17], among Greek children (95.2% vs. 22.2%) [33], and among Swiss children (73% vs. 43%) [34]. However, a German study found similar sensitivities between the two groups (86.8% vs. 85.7%) [35]. The symptomatic individuals we enrolled in our evaluation had significantly lower Ct values than asymptomatic participants (Fig 2). Our evaluation further observed that irrespective of symptoms sensitivity was higher among those with Ct value �30 as reported by other authors [17,[35][36][37]. We also observed significantly higher viral loads among symptomatic individuals compared to asymptomatic individuals (p <0.001) as has reported in recent literature [38,39]. In contrast, some studies found no difference in viral load between the two groups [40,41] while others found higher viral load among asymptomatic individuals [42,43]. Higher sensitivity has been observed among individuals presenting in the early symptomatic phase of their infection [19,32,[35][36][37]42,44,45].
While rRT-PCR testing for SARS-CoV-2 is the reference standard for diagnosis, its limited access in our setting and provision of results in a clinically relevant timely manner to inform case management limits its clinical utility. Therefore, given reasonable sensitivity, individuals with a negative Ag RDT test in an individual meeting the clinical and epidemiological criteria may require a confirmatory test by rRT-PCR. The variation in sensitivity by presence or absence of symptoms, symptom duration, and viral load indicates that the kit may only be useful among individuals with high viral loads, especially those with symptoms.
The PPV (90.8%) observed at the prevalence of SARS-CoV-2 infection of 24.5% by rRT-PCR was similar to the projected PPV (89.6%.) assuming the minimum WHO recommended sensitivity (80%) and specificity (97.0%). The NPV (85.0%) was lower compared to that computed NPV (93.7%) assuming the minimum recommendations. Among individuals who tested negative by Ag RDT, the likelihood of being wrongly classified as uninfected was 15%. In 2021, the median positivity rate in Kenya was 7.4% (IQR, 3.2-13.1%) [46]. At the observed sensitivity and specificity and the national positivity rate, PPV would range between 50.2% and 82.8%, and the NPV 92.4%-98.2%).
The overall specificity remained high (98.5%) and was above the WHO recommendation [47] for both symptomatic and asymptomatic individuals and across the other subgroups (Table 2) as has been reported in other evaluations [32,35,36,44,[48][49][50][51]. The Ag RDT had high specificity with less than (1.5%) of the tests (n = 26) giving a false-positive result. This small proportion represents individuals who would be falsely classified as infected and unnecessarily be managed as COVID-19 cases or self-isolated and their contacts traced and quarantined.
However, we observed over half (53%) of the Ag RDT results gave false-negative results with a higher proportion (65%) among the asymptomatic individuals. This observation emphasizes the recommendation by WHO to re-test symptomatic patients with rRT-PCR when they receive a negative Ag RDT result, especially in settings where SARS-CoV-2 prevalence is � 5%. Further, as a screening tool for testing contacts of confirmed cases, the 65% false negatives among asymptomatic contacts represent the proportion that would falsely be classified as virus-free, not self-isolate, and potentially infect others.
The Panbio™ Ag RDT was found to be easy to use by the majority of the users with a quick TAT for clinical decision-making and implementation of preventive measures to contain transmission as observed in the literature [35,36]. The main challenge we observed in conducting Ag RDT testing using the Panbio™ Ag RDT kits was dispensing the specimen onto the test device in samples collected from individuals with thick mucus. This was, however, solved by ensuring that individuals blew their noses before specimen collection.
Our field evaluation had several strengths. The evaluation was conducted in the real-world setting across multiple sites under the point-of-care conditions which had a large sample size comprising two groups (symptomatic and asymptomatic individuals) thereby representing individuals at any point of the COVID-19 disease spectrum.

Limitation
The number of persons enrolled in the two groups, symptomatic and asymptomatic, may not reflect the true prevalence in the general population as we evaluated the Ag RDT during a period of high COVID-19 prevalence. While rRT-PCR was used as the reference standard, it is not a gold standard as it may not truly differentiate replicating virus and residual RNA from recent infection among those with high Ct values. Residual RNA may not be viable and therefore not infectious [52].

Conclusion
The overall sensitivity and PPV of the Panbio™ Ag RDT were much lower than expected. Sensitivity was acceptable in symptomatic individuals with Ct value �30. The specificity of the Ag RDT was high and satisfactory; therefore, a positive result may not require confirmation by rRT-PCR. The kit may be useful as a rapid screening tool for only symptomatic patients in high-risk settings with limited access to rRT-PCR. A negative result should be interpreted based on clinical and epidemiological information and may require retesting by rRT-PCR.