Field evaluation of the performance of seven Antigen Rapid diagnostic tests for the diagnosis of SARs-CoV-2 virus infection in Uganda

Objective The objective of this study was to evaluate the performance of seven antigen rapid diagnostic tests (Ag RDTs) in a clinical setting to identify those that could be recommended for use in the diagnosis of SARS-CoV-2 infection in Uganda. Methods This was a cross-sectional prospective study. Nasopharyngeal swabs were collected consecutively from COVID-19 PCR positive and COVID-19 PCR negative participants at isolation centers and points of entry, and tested with the SARS-CoV-2 Ag RDTs. Test sensitivity and specificity were generated by comparing results against qRT-PCR results (Berlin Protocol) at a cycle threshold (Ct) cut-off of ≤39. Sensitivity was also calculated at Ct cut-offs ≤29 and ≤33. Results None of the Ag RDTs had a sensitivity of ≥80% at Ct cut-off values ≤33 and ≤39. Two kits, Panbio™ COVID-19 Ag and VivaDiag™ SARS-CoV-2 Ag had a sensitivity of ≥80% at a Ct cut-off value of ≤29. Four kits: BIOCREDIT COVID -19 Ag, COVID-19 Ag Respi-Strip, MEDsan® SARS-CoV-2 Antigen Rapid Test and Panbio™ COVID-19 Ag Rapid Test had a specificity of ≥97%. Conclusions This evaluation identified one Ag RDT, Panbio™ COVID-19 Ag with a performance at high viral load (Ct value ≤29) reaching that recommended by WHO. This kit was recommended for screening of patients with COVID -19-like symptoms presenting at health facilities.


Introduction
Health, that received patients from all over the country and at points of entry into the country. The study enrolled travellers requiring testing at points-of-entry (POE) into the country and patients admitted at isolation centres.
At isolation centers we used results from previous PCR tests and records of symptoms in the files to identify participants for recruitment. While at points of entry, participants with and without COVID-19 symptoms were enrolled. Patients that had difficulty in breathing and were receiving oxygen therapy or were on ventilators were excluded from the evaluation.
Enrolment of participants was convenient and consecutive from the selected sites. Data were collected from each of the participants and entered in an Excel sheet. Data collected included socio-demographic (age, sex, health unit of isolation or point-of-entry), presence or absence of symptoms, date of admission and date of first symptom(s).

Sample collection and testing
The evaluation of each Ag RDT was performed at different time periods using different sets of samples drawn from participants enrolled at that time. This was because RDTs were received at UVRI from suppliers at different times and in different quantities.
Two nasopharyngeal samples were collected: one from each nostril. One sample was tested on the Ag RDT under evaluation and the second sample was used for qRT-PCR test. The sequence of collection of the nasopharyngeal samples for the Ag RDT and qRT-PCR test was alternated. Samples for qRT-PCR and COVID-19 Ag Respi-Strip were collected in a tube containing virus transport media (Hanks balanced salt solution, Fetal bovine serum, gentamycin sulfate and amphotericin B). The samples for the other six Ag RDT (BIOCREDIT COVID -19 Ag, PCL COVID 19 Ag Rapid FIA, MEDsan 1 SARS-CoV-2 Antigen Rapid test, Panbio™ COVID-19 Ag Rapid Test, Novegent COVID-19 Antigen Rapid Test Kit (colloidal gold), and VivaDiag™ SARS-CoV-2 Ag Rapid Test kit) under evaluation were placed in the buffer provided by the kit manufacturer.
Antigen RDT testing was done within one hour of specimen collection without any cooling, following the manufacturer's instructions. The results of the six Ag RDTs: BIOCREDIT COVID -19 Ag, COVID-19 Ag Respi-Strip, MEDsan 1 SARS-CoV-2 Antigen Rapid test, Pan-bio™ COVID-19 Ag Rapid Test, Novegent COVID-19 Antigen Rapid Test Kit (colloidal gold), and VivaDiag™ SARS-CoV-2 Ag Rapid Test kit were visually read by trained UVRI laboratory staff. While, results from the PCL COVID19 Ag Rapid FIA were not visible and thus only read using the PCLOK EZ instrument supplied with the test kit. Interpretation of Antigen RDT results were as per the respective manufacturer's guidelines. Those reading/interpreting the RDT results sometimes had access to the clinical information of the participant.
Samples collected for qRT-PCR were stored at 2˚-8˚C and transported within one week of collection to UVRI. At UVRI, samples were stored at -80˚C for 1 to 28 days (median of 4-7 days) for the six RDTs except for RESPI Strip which ranged 1-37 days (median of 21 days) before qRT-PCR testing. The personnel carrying out the qRT-PCR testing were blinded to the Antigen RDT results. qRT-PCR testing was carried out at the EPI laboratory, one of the UVRI laboratories that tests for SARS-CoV-2, as described below: RNA extraction. Viral RNA was extracted from the samples using the QIAGEN QIAamp Viral RNA extraction kit (QIAGEN, Hilden, Germany) following the manufacturer's guidelines.
Real-time reverse-transcription PCR. qRT-PCR testing was carried out using the Charite-Berlin Protocol [9]. This assay was selected as the Standard reference for the kit evaluation because the protocol has good sensitivity with a limit of detection (LoD) of 3.9 RNA copies per reaction for E-gene assay and 3.6 RNA copies per reaction for RdRp gene assay using invitro transcribed RNA identical to 2019 novel coronavirus sequences and specificity of 100% [9].
Screening for SARS-CoV viruses was done using the SuperScript™ III Platinum™ One-Step qRT-PCR kit (Invitrogen, Carlsbad, CA) and LightMix 1 SarbecoV E-gene primer/probe mix (TIB MOLBIOL, Berlin, Germany) following the manufacturer's guidelines. All samples that had Ct values <45 in the above assay were subjected to a SARS-CoV-2 confirmatory qRT-PCR using the SuperScript III real-time RT-PCR kit (Invitrogen, Carlsbad, CA) and LightMix 1 Modular SARS-CoV-2 (COVID-19) RdRp primer/probe mix (TIB MOLBIOL, Berlin, Germany) following the manufacturer's guidelines. A sample was considered positive if it had a Ct value �39 on the confirmatory qRT-PCR. A sample was considered negative if the Ct value was undetermined on the screening qRT-PCR or if it was positive on the screening qRT-PCR but undetermined or >39 on the confirmatory qRT-PCR.
The thermal cycling conditions consisted of a reverse transcription reaction at 50˚C for 10 minutes, an activation step at 95˚C for 10 minutes and 45 cycles at 95˚C/15 seconds and 60˚C /1 minute. The platform used for the PCR was Applied Biosystems:7500 Real-Time PCR System (Marsiling, Singapore).

Data management and analysis
Ct values were recorded against each sample's identifier and data entered in an MS Excel sheet which was transformed into a STATA 1 v15 (StataCorp LP, 4905 Lakeway Drive, College Station, TX, USA) file for analysis. Inconsistency checks were performed using STATA and data cleaning done in collaboration with the field and laboratory staff. Samples with missing Antigen RDT and qRT-PCR results were excluded from the analysis.
Univariate analysis was performed to generate descriptive summaries for demographic and sample source characteristics. Frequencies, means, confidence intervals and medians were generated as summary statistics.

Performance evaluation
Sensitivity. Sensitivity was calculated as the number of specimens determined as positive by the Ag RDT under evaluation divided by the number of specimens determined as positive by PCR and expressed as a percentage with confidence intervals.
Specificity. Specificity was calculated as the number of specimens determined as negative by the Ag RDT under evaluation divided by the number of specimens determined as negative by PCR and expressed as a percentage with confidence intervals.
Accuracy. The accuracy was calculated as the proportion of results determined by the Ag RDT under evaluation that agreed with the PCR results and expressed as a percentage with confidence intervals.
False positive rate. The false positive rate was calculated as False Positive/(False Positive + True Negative) and expressed as a percentage.
False negative rate. The false negative rate was calculated as False Negative/(False Negative + True Positive) and was expressed as a percentage.
Sensitivity, specificity, accuracy, false positive rate and false negative rate calculations were performed using the proportion command in STATA 15 and confidence intervals produced with the Wilson score method [10].
Sensitivity of the Ag RDTs was also determined for different Ct cut-off values of �29, �33 and �39.
Sensitivity and specificity for all the Ag RDTs were also determined according to whether symptom onset was within 7 days of the specimen collection or after 7 days.
A sub-analysis (for sensitivity) was also performed for each Ag RDT considering different Ct cut off values following discussions with relevant stakeholders at the Ministry of Health and basing on literature [11][12][13]; a strong positive was defined as Ct value �29, a moderate positive was defined as Ct value 30-37 and a low positive was defined as Ct value 38-39.

Ethical considerations
The study protocol [14] was approved by Uganda Virus Research Institute's Research Ethics Committee (No. GC/127/20) and the Uganda National Council for Science and Technology (No HS637ES). Informed written consent was obtained from each of the participants before enrolment into the study.

Participant demographics
A total of 1,533 participants were included in the evaluation of the seven Ag kits as detailed in Table 1
There were 97 participants with onset of symptoms ranging between 0-7 days prior to the evaluation, 42 with onset of symptoms more than 7 days prior to the evaluation and 108 participants who were asymptomatic before the Ag RDT test (Table 1).
Among the participants with symptom onset to date of Ag RDT test between 0-7 days, the sensitivity of the test was 39.3% (95% CI: 27.7%-52.4%). The sensitivity of the test was 11.1% (3.3%-31.1%) for participants with more than 7 days from symptom onset to date of Ag RDT test: and 21.3% (95% CI: 11.6%-35.8%) for asymptomatic participants.
The BIOCREDIT COVID -19 Ag test result was negative for a large proportion of strong, moderate, and low positive specimens by qRT-PCR: 40%, 90% and 100% respectively. This was observed especially for Ct cut-off values between 30-38 (Table 3).

PLOS ONE
Field evaluation of SARs-CoV-2 Antigen Rapid diagnostic tests in Uganda Sixty-seven (34.5%) participants out of the 194 were positive on PCR with 6 (9.0%) of these positives being females. The median age was 33 years (IQR 28-41 years).
The Respi-Strip determined the largest proportion of strong positives by qRT-PCR as negative (53.6%) and all moderate and low positives by qRT-PCR as negative (100%) ( Table 3).
There were 32 participants with onset of symptoms to date of Ag RDT test between 0-7 days. Of these, 19 were PCR-positive. Another 15 participants reported symptom onset of more than 7 days prior to the Ag RDT test with 9 (60%) being qRT-PCR positive (Table 1).
The PCL test determined a large proportion of samples, whether strong or moderate or low positive by qRT-PCR as negative: 22.2%, 79.3% and 75.0% respectively (Table 3). Of the 243 samples, 123 (50.6%) were qRT-PCR positive: 37 (30.1%) of these positives were females. The median age of the participants was 37 years (IQR 28-49 years). A total of 75(30%) participants had onset of symptoms 0-7 days before the antigen test (Table 1).

MEDsan 1 SARS-CoV-2 Antigen Rapid test
There were 25 qRT-PCR positive participants with the period between symptom onset to the date when the antigen test was done ranging between 0-7 days. For twenty-three (23) participants the period was greater than 7 days.  Table 2). The Ag RDT had a sensitivity of 12.0% (95% CI: 3.96%-33.3%) for participants with 0-7 days between symptom onset and the antigen test and 21.7% (6.7%-44.8%) for participants with more than 7 days between symptom onset and the antigen test.
It was observed that the Panbio™ COVID-19 Ag Rapid test is more likely to determine a sample as positive when a sample has abundant target nucleic acid with Ct � 29 (85.7%). However, when a sample is determined as moderate positive, 69% were determined as negative by Panbio™ COVID-19 Ag Rapid test (Table 3).
There were 137 participants with a period of 0-7 days between symptom onset and the date of the antigen test and 56 of these were qRT-PCR positive. Forty-two (42) qRT-PCR positive participants had symptom onset more than 7 days prior to the test ( Table 1).
The majority of the participants were males (57.6%) and the median age was 47 years (IQR 34-58 years) ( Table 1).
Taking into consideration participants with 0-7 days between symptom onset and the date of the antigen test, the Antigen RDT sensitivity was 65.0% (95% CI: 38.6%-71.4%). For those participants where the period between symptom onset and the date of the antigen test was greater than 7 days the RDT sensitivity was 42.9% (28.4%-58.7%).
Novegent COVID-19 Antigen Rapid test (Colloidal) is more likely to determine a sample as positive (66.7%) if the qRT-PCR result is a strong positive. However, for those samples that were moderately or weakly qRT-PCR positive, this antigen test was more likely to determine them as negative, 59.7% and 83.3% respectively (Table 3).
There were 235(89.3%) participants with a period ranging between 0-7 days from symptom onset to the date when the antigen test was done (Table 1). Of these 235, thirty-eight (38) were positive by qRT-PCR. Twenty-one participants reported symptom onset more than 7 days prior to the antigen test with 5 (23.8%) determined as qRT-PCR-positive.
The VivaDiag™ determined a large proportion of qRT-PCR strong positive cases as positive (80%) though the number of cases in this category was only five. However, VivaDiag™ determined a very large proportion of the qRT-PCR moderately-positive cases (Ct values 30-38) as negative (84.4%) ( Table 3).

Discussion
WHO recommends the use of antigen RDTs for SARS-CoV-2 diagnosis because they provide results within 15-30 minutes compared to qRT-PCR tests where results may not be available for more than 24 hours. RDTs cost far less than PCR tests, and do not require specialized laboratories and highly trained staff [2]. The use of accurate COVID -19 antigen RDTs will enable faster identification and management of COVID-19 patients leading to better control of the pandemic. A large number of antigen RDTs have been developed with varying sensitivities and specificities reported by the manufacturer's IFUs. However, WHO has recommended field evaluations of antigen RDTs before they can be adopted in a national setting [2]. WHO recommends the use of antigen RDTs with a sensitivity of �80% and a specificity of � 97% [2].
In this study, none of the seven antigen RDTs evaluated reached the sensitivity reported by the manufacturers, even after considering samples strongly positive by qRT-PCR (Ct values �29). These RDTs however had good specificity, with only three showing a specificity slightly lower than that reported in the IFUs. Poor performance in real-world settings compared to that found in the manufacturer's IFUs has also been reported by others [4,7,8,[15][16][17][18], emphasizing the need for performance evaluations of antigen RDTs in-country before adoption for use. In our study, we noted that the sensitivity of all seven Ag RDTs improved at lower qRT-PCR Ct values and this has also been observed in other studies [6][7][8][18][19][20]. Lower qRT-PCR Ct values in most cases correlate with higher viral loads and hence with increased transmissibility of the virus, usually in the early phases of the infection [21].
There were two antigen RDTs that reached the WHO recommended sensitivity performance of �80% at a Ct value of �29. These were the Panbio™ COVID-19 Ag Rapid test and the VivaDiag™ SARS-CoV-2 Ag Rapid test with sensitivities of 85.7% and 80% respectively. However, for the VivaDiag, the sample size was very small with only 10 samples at this Ct value. Of the two kits, Panbio™ COVID-19 Ag Rapid test and VivaDiag™ SARS-CoV-2 Ag Rapid test, only the PanBio™ COVID-19 Ag Rapid Test had the WHO recommended specificity performance of � 97%.
There was no correlation of antigen RDT performance with the presence or absence of symptoms. High viral loads usually appear in the pre-symptomatic and early symptomatic phases of the illness (within the first 5-7 days of symptom onset), however asymptomatic individuals can also have high viral loads in the early days of infection [22].
The Panbio™ COVID-19 Ag Rapid Test met the sensitivity and specificity performance levels recommended by WHO at qRT-PCR Ct values of �29. This antigen RDT together with the Standard Q (by SD Biosensor) antigen RDT we previously evaluated [3], have been recommended in Uganda for COVID-19 diagnostic intervention in a phased approach as more experience and confidence is gained in their use through continuous field evaluation and the generation of additional data. These antigen RDTs have been recommended for rapid screening of the following populations; symptomatic alerts and symptomatic contacts of confirmed cases, patients with COVID -19-like symptoms presenting at health facilities.
The Uganda MOH recommends that, any antigen RDT positive case from the above mentioned populations will be considered "COVID-19 positive" and managed accordingly and will not require additional qRT-PCR confirmation except in special cases such as genomic sequencing or routine quality control monitoring. Furthermore, any antigen RDT negative case with highly suggestive symptoms will be considered "a suspect COVID-19 case" until confirmed to be uninfected by qRT-PCR and should be managed with enhanced infection prevention control (IPC). Laboratories at health units are expected to obtain/collect an additional specimen from those symptomatic clients who are antigen RDT negative to be sent for PCR testing.

Study limitations
Since samples were collected from different nostrils for the antigen test (BIOCREDIT COVID -19 Ag, MEDsan 1 SARS-CoV-2 Antigen Rapid test, PCL and Panbio™ COVID-19 Ag Rapid test) and qRT-PCR tests, the two swabs can be considered as two different samples which may lead to variation in the virus content of the collected samples.
The Panbio™ COVID-19 Ag Rapid Test IFU recommends using the test within 5 days of symptom onset. However, in the evaluation of this antigen RDT, specimens from some asymptomatic cases were included. In asymptomatic cases it is difficult to determine the stage of the infection, whether early or late, thus affecting the overall interpretation of the results i.e. whether the antigen test result is negative because the client was in the early stage of disease and thus had low levels of antigen or was due to the poor performance of the test kit.
Participants in the study were recruited at different time periods thus there was varying prevalence of SARS-CoV-2 which could have affected the specificity of the tests. There was also significant difference in the population mix of the participants for the different tests (male/female, mean age, number of PCR positive and PCR negative participants) which may affect the comparability of the tests. Most participants were tested with only one of the evaluated RDT, implying different populations were used for evaluation of each RDT, thus making it difficult to compare the different RDT performance.

Conclusions
This field evaluation of seven SARS CoV-2 RDTs: COVID-19 Ag Respi-Strip, BIOCREDIT COVID -19 Ag, MEDsan 1 SARS-CoV-2 Antigen Rapid test, PCL COVID19 Ag Rapid FIA, Panbio™ COVID-19 Ag Rapid test, Novegent COVID-19 Antigen Rapid test kit (colloidal gold), and VivaDiag™ COVID-19 Ag showed poorer performance than that reported by the manufacturers. Only the Panbio™ COVID-19 Ag Rapid Test reached the WHO recommended performance of a sensitivity of � 80% and a specificity � 97% at qRT-PCR Ct values � 29. This RDT can be recommended for COVID-19 infection diagnosis among symptomatic patients in health facilities in order not to delay patient management while waiting for qRT-PCR results.
Supporting information S1 Fig. Flowchart summarizing test results