The Abbott PanBio WHO emergency use listed, rapid, antigen-detecting point-of-care diagnostic test for SARS-CoV-2—Evaluation of the accuracy and ease-of-use

Objectives Diagnostics are essential for controlling the pandemic. Identifying a reliable and fast diagnostic device is needed for effective testing. We assessed performance and ease-of-use of the Abbott PanBio antigen-detecting rapid diagnostic test (Ag-RDT). Methods This prospective, multi-centre diagnostic accuracy study enrolled at two sites in Germany. Following routine testing with reverse-transcriptase polymerase chain reaction (RT-PCR), a second study-exclusive swab was performed for Ag-RDT testing. Routine swabs were nasopharyngeal (NP) or combined NP/oropharyngeal (OP) whereas the study-exclusive swabs were NP. To evaluate performance, sensitivity and specificity were assessed overall and in predefined sub-analyses accordingly to cycle-threshold values, days after symptom onset, disease severity and study site. Additionally, an ease-of-use assessment (EoU) and System Usability Scale (SUS) were performed. Results 1108 participants were enrolled between Sept 28 and Oct 30, 2020. Of these, 106 (9.6%) were PCR-positive. The Abbott PanBio detected 92/106 PCR-positive participants with a sensitivity of 86.8% (95% CI: 79.0% - 92.0%) and a specificity of 99.9% (95% CI: 99.4%-100%). The sub-analyses indicated that sensitivity was 95.8% in Ct-values <25 and within the first seven days from symptom onset. The test was characterized as easy to use (SUS: 86/100) and considered suitable for point-of-care settings. Conclusion The Abbott PanBio Ag-RDT performs well for SARS-CoV-2 testing in this large manufacturer independent study, confirming its WHO recommendation for Emergency Use in settings with limited resources.


Introduction
Diagnostics are a cornerstone of pandemic control. In March 2020, the World Health Organisation (WHO) emphasized the importance of access to testing for an effective control of SARS-CoV-2 infections [1]. While reverse-transcriptase polymerase chain reaction (RT-PCR) remains the gold standard among diagnostic tests for SARS-CoV-2, access may be limited due to shortages of instruments, supplies and experienced operators, particularly in resource-limited settings. Antigen-detecting tests (Ag-RDTs) offer an alternative to RT-PCR and have been recommended by the WHO for appropriate settings where nucleic acid amplification technology (NAAT) testing is limited or where prolonged turnaround times slow down clinical testing if they achieve at least �80% sensitivity and �97% specificity compared to a NAAT reference assay [2].
The Foundation of New Innovative Diagnostics (FIND) has identified >80 antigen tests in the pipeline for SARS-CoV-2 [3]. To this date, the WHO has recommended two Ag-RDTs on the Emergency Use Listing based on data from manufacturers as well as one independentlyconducted accuracy study [4]. These two Ag-RDTs are SD Biosensor STANDARD Q (recommended on Sept 22, 2020) followed by the Abbott PanBio Ag-RDT (Oct 2,2020) [5,6]. This manufacturer-independent study follows a WHO-approved protocol to complement the data provided to the WHO by the manufacturer.
Several other studies evaluating the Abbott PanBio Ag-RDT have been published to date. Of three large studies only one single-centre study prospectively enrolled participants [7]. The other two selected stored samples with the representativeness of the selection being unclear in one study [8,9]. One study with a limited sample size (N = 255) concluded that sensitivity was the highest within the first week after symptom onset [10] supported by two additional studies reporting sensitivity to be the highest in participants presenting with a low Ct-value and a respectively high viral load, often observed at the onset of symptoms [11,12]. The sensitivity was determined to be in the range of 60%-92% in these six studies along with a very high specificity of 98.9% and above [7][8][9][10][11][12].
The prospective multi-centre clinical accuracy study reported here, represents the largest manufacturer-independent dataset for the point-of-care (POC) performance of the Abbott PanBio Ag-RDT of date and also represents to our knowledge the only comprehensive easeof-use assessment (EoU) of the Abbott PanBio Ag-RDT.

Ethic statement
The study protocol was approved in March 2020 by the ethical review committee at the Heidelberg University Hospital for the two study sites Heidelberg and Berlin in Germany (Registration number S-180/2020).

Clinical diagnostic accuracy study
The standards for reporting diagnostic accuracy studies (STARD) were followed for this study.
The test uses the lateral flow assay principle in a cassette format design for the detection of viral antigens. The test kits include proprietary swabs for sample collection. As indicated in the instruction for use (IFU), five drops of the extracted specimen in the provided buffer solution are applied to the test device. Colloidal gold conjugated antibodies on the membrane strip react with viral antigens generating a colour change in the device window, which can be interpreted with the naked eye. The results should be interpreted within 15 to 20 minutes of incubation and are considered invalid if interpreted after this timeframe. The manufacturer's IFU were followed during sampling and testing procedures.
Study design and participants. The enrolment of participants was conducted at two sites in Germany; in Heidelberg at a drive-in testing site; in Berlin at a clinical ambulatory testing facility. Inclusion criteria for the participation were age �18 years and classification as being at risk for a SARS-CoV-2 infection by the local health department based on contact with a confirmed SARS-CoV-2 case or symptoms suggestive for infection. Individuals with a prior positive RT-PCR test for SARS-CoV-2 or those who could not give written informed consent due to limited command of English or German were excluded. The study protocol is available upon request.
Study procedures. Individuals meeting the inclusion criteria were invited to participate in the study. After providing written informed consent, participants first underwent the routine swab for RT-PCR testing directly followed by the study-exclusive swab for Ag-RDT testing, performed by the trained study team. Routine sampling for RT-PCR testing was performed with a nasopharyngeal (NP) swab in Heidelberg and a combination of a NP and oropharyngeal (OP) swab in Berlin as per institutional procedure. The study-exclusive sampling for the Ag-RDT PanBio was an NP swab. If NP swabbing was contraindicated for clinical reasons an OP swab was performed. The study-exclusive swab was taken in the same nostril as the routine swab. Laboratory personnel working in both the Ag-RDT testing team and the RT-PCR laboratory were blinded to the results of the other test at all times.
Antigen-detecting testing. Ag-RDT testing was performed in immediate proximity to the sampling in a separate room/container. The Ag-RDT test was started directly after sample collection. The test was conducted as indicated in the IFU, interpreting the test with the naked eye after 15 minutes by two readers blinded to the results of the other. In the case of discrepant results both readers re-interpreted the results and agreed on a final result. Invalid test results were repeated once with the remaining buffer solution in the test tubes.
RT-PCR testing. The collected swabs (Heidelberg, IMPROSWAB, Improve; Berlin, eSwab, Copan) for routine RT-PCR testing were kept in provided Amies solution. The RT-PCR assays performed were Allplex SARS-CoV-2 Assay from Seegene (Seoul, South Korea) in Heidelberg and the Roche cobas SARS CoV-2 assay (Pleasanton, CA United States) on the cobas 1 6800 or 8800 system or the SARS CoV-2 assay from TibMolbiol (Berlin, Germany) in Berlin. Ct-values varied in a range of 2-3 between the three technologies. For the Ct-value presentation and the viral load calculations the E-Gene was used as reference Ctvalue. A conversion of the Ct-values into viral-load was performed using quantified specific in vitro-transcribed RNA [14].
Additional data collection. All participants were asked to provide additional information about their comorbidities, symptoms, symptom duration and severity of disease (questionnaire available in the, Section (B) in S1 File).

Data management
All data were collected and managed using Research Electronic Data Capture (REDCap) tools hosted at Heidelberg University [15].

System usability scale and ease-of-use assessment
A standardized System Usability Scale (SUS) questionnaire and an ease-of-use assessment (EoU) were designed to understand the usability and feasibility of the test [16]. The questionnaire and the EoU survey can be found in the Section (C) and (D) in S1 File. Laboratory personnel from both study sites were invited to complete the questionnaire. An over-all SUS score above 68 is interpreted as above average and anything below the score of 68 is below average [16]. A heat-map was generated to analyse aspects related to the ease-of-use of the test, categorising each as satisfactory, average or dissatisfactory (Section (E) in S1 File and Fig 1). The matrix used for this analysis is also found in the Section (F) in S1 File.

Statistical analysis
The sensitivity and specificity of the Ag-RDT with 95% confidence intervals (95% CIs) were assessed as per Altman compared to RT-PCR as reference standard (sample size calculations are provided in the S1 File; a statistical analysis plan is available upon request) [17]. Sub-analyses were performed by sampling strategy, symptoms, duration of symptoms, Ct-values and study sites. The significance threshold was set at a two-sided alpha value of 0.05. Participants with an invalid PCR result were excluded from the analysis. All analyses and plots were performed using the R version 4.0.3.

Clinical diagnostic accuracy
During the enrolment period, from Sept 28, 2020 to Oct 30, 2020, a total of 1261 eligible participants meeting the inclusion criteria were screened for this study. From these 1261 participants, 1119 agreed to undergo a second swab for study purposes only (Fig 1). 10 participants had to be excluded from the study, initially agreeing on participation but denying a second sample collection after the routine swab was performed. After the data cleaning and the exclusion of one invalid PCR test result (N = 1), a total of 1108 participants were included in the analysis. The site in Heidelberg enrolled 858 participants between the Sept 28 and Oct 30, 2020, and the site in Berlin enrolled 250 participants between Oct 19-30, 2020.
The clinical and demographic characteristics of the enrolled participants are summarized in Table 1. The mean age of all participants was 39.4 years (Standard Deviation (SD) 14.1) with Berlin presenting a younger study population compared to Heidelberg. 50.7% of participants were female and 33.4% had comorbidities. 712 participants (64.7%) reported having symptoms on the testing day, with an average symptom duration in days of 4.01 days (SD 3.1). The populations enrolled in Berlin and Heidelberg were significantly different in that participants in Berlin typically enrolled with symptoms (96.8%) while in Heidelberg almost half of the participants were tested based on high-risk contacts without symptoms and compared to the other half of participants reporting symptoms on the testing day (54.8%). Also, participants in Heidelberg were more likely to present earlier in their course of disease (mean 3.5 versus 4.98 days in Berlin) and were more likely to have comorbidities (36.6% in Heidelberg versus 21.2% in Berlin). In total 106 (9.6%) participants were diagnosed with a SARS-CoV-2 infection by RT-PCR testing during the enrolment period with 23.6% in Berlin and 5.5% in Heidelberg. The mean viral load was 7.4 for both sites with only a slight difference in the SD ( Table 1).

PLOS ONE
Performance of the Abbott PanBio antigen-detecting test for SARS-CoV-2 samples with a Ct-value <30 (see Table 2 and Fig 2). The sensitivity decreased as viral load decreased (Fig 2). A detailed table summarizing the symptoms for Ag-RDT positive and negative participants, the viral load equivalents to the Ct-values and the RT-PCR reference standard is provided in the Section (G) and (H) in S1 File. When assessing test performance by duration of symptoms, we found PanBio performed well in the first 7 days after symptom onset (sensitivity 90.8% (95% CI: 82.2%-95.5%)), with declining sensitivity thereafter (>7 days of symptoms, sensitivity 61.5% (95% CI: 35.5%-82.3%)). This decrease in sensitivity with prolonged symptom duration is also shown in Fig 3, presenting the performance of PanBio with increasing days since symptom onset in relation to the calculated viral load for symptomatic and asymptomatic participants and the Ag-RDT performance.
Out of the total 106 positive RT-PCR cases, 14 participants were without symptoms but with recent high-risk contacts. Within this small participant group the sensitivity of the Ag-RDT was 85.7% (60.1%-96.0%), which compares to the sensitivity of symptomatic participants at 86.8% (78.2%-92.3%); Table 2). Mean Ct-value in asymptomatic was 22.1 (SD 4.4) versus 23.1 (SD 5.0) in symptomatic participants.
The interrater reliability with kappa of 0.99 suggests that the tests results are clearly interpretable. PanBio scored 86 out of 100 points in the SUS showing a test which is easy to use. Problems were encountered when applying the exact amount of the required five drops to the test device (Section (E) in S1 File).

Discussion
This prospective multi-centre clinical diagnostic accuracy study shows that the PanBio Ag-RDT from Abbott has a good sensitivity of 86.8% and a very high specificity of 99.9% for symptomatic participants and participants with confirmed recent high-risk contacts compared to the reference standard RT-PCR. The Ag-RDT is easily performed in a point-of-care setting.
The differences observed in sensitivity between the two enrolment sites are probably explained by the different stages of the pandemic control. Berlin had a substantially higher prevalence with primarily symptomatic patients being tested and testing occurring later in the disease. Although the mean viral load was the same, the distribution of the viral load was not the same and more participants presented with low viral loads in Berlin, later in the course of infection. There were 15 patients presenting with viral loads <6 log10 copies/mL versus only 8 in Heidelberg. The samples with low viral loads, were responsible for 8 out of the 11 discordant results (false-negative) in Berlin, and 2 out of 3 in Heidelberg. Viral load dynamics play an important role in disease transmission, often rising before symptom onset with an observed peak at the time of symptom onset or at day 3-5 of the disease and a rapid decrease within the following days [18]. With a performance of 93.5% for CT value <30 and 90.8% within the first 7 days of symptoms, the test is likely to detect the vast majority of people with transmission potential SARS-CoV2 infections, supporting recent published literature [10,19,20]. These findings show that the PanBio Ag-RDT has a great potential in a public health setting, identifying the transmission relevant infections within the first 7 days of symptoms. Within the limitation of what can be concluded due to the small sample size of participants without symptoms at the time of testing and the fact that these participants were asymptomatic high-risk contacts presenting early in the disease, the performance of the test was as good as in symptomatic patients with a sensitivity respectively of 85.7% versus 86.8%. This suggests the PanBio to be an option for screening independent of symptoms, especially when the time of exposure is known and is supported by recent data suggesting that viral load in adults does not differ between asymptomatic and symptomatic infections [21].
Considering the test's ease-of-use and the rapid turn-around time between 15 and 20 minutes, along with its high specificity, it could be considered for several use-cases: (1) screening in advance of events at high-risk of transmission (e.g. aggregated settings where contact cannot be avoided); or (2) necessary encounters with persons at high-risk for severe disease of SARS-CoV-2 (e.g. visitor in nursing homes) in addition to (3) the use in symptomatic patients within the first week of illness when RT-PCR is not available or together with RT-PCR, when a rapid decision is necessary. Furthermore, given that supervised self-sampling from the anterior nose is a reliable alternative to professional nasopharyngeal sampling, scale-up of testing appears possible without requiring large numbers of trained health-care workers [22,23].
Overall, our study has several strengths. The population enrolled for testing was representative of the pandemic observed in adults in Germany with a broad spectrum of clinical presentations (from asymptomatic with high-risk contacts to severely ill). Due to the wide-spread testing available and the good test and trace capabilities, the population tested is expected to be a representative spectrum of disease. Also, the tests were performed at POC thus mimicking the real-world challenges of POC testing. Lastly, the comprehensive ease-of-use assessment highlighted important points for operationalization of the test.
However, the study also has several limitations. First, it was conducted only in one country, thus making it less representative of the pandemic at large. Second, the reference standard testing was performed on an NP swab in Heidelberg versus an NP/OP swab in Berlin. However, a recent systematic review does not suggest those sampling methods to yield different results [24]. Thirdly, participants without symptoms at the time of testing could potentially have developed symptoms after being tested, yet this information was not recorded during this study. Furthermore, the time of exposure to high-risk contacts for participants without symptoms was not recorded, giving the possibility that these participants presented very early in the course of disease, maybe resulting in false-negatives [25]. And lastly, we performed different PCR methods as a reference standard recognizing the limitations of this method However, we aimed to provide comparability between methods by calibrating the methods and in addition reporting on viral load. Furthermore, we acknowledge the limitation of the PCR method as a reference standard, as it is not always a meaningful test when considering viable virus and risk of transmission [26]. Thus, using the PCR reference standard, we might have underestimated the performance of the Ag-RDT when it comes to detection of viable virus.
In summary, the favourable ease-of-use results and the limited infrastructure required for the Ag-RDT testing procedure, its high specificity in addition to the high sensitivity of the test in persons with high viral load, can empower control of population transmission if implemented in well-designed testing programs [27][28][29]. Policy makers should move from considering only test sensitivity to more holistic testing strategies, incorporating Ag-RDTs in addition to and in combination with RT-PCR to optimize the reach and depth of testing.