Evaluating the field performance of multiple SARS-Cov-2 antigen rapid tests using nasopharyngeal swab samples

The SARS-CoV-2 pandemic, which started in December 2019, has been posing significant challenges to the health care system worldwide. As the pandemic spreads with rapidly increasing number of positive cases, early diagnosis of infected patients is crucial to successfully limit the spread of the virus. Although the real-time reverse-transcription polymerase chain reaction (RT-qPCR) is the recommended laboratory method to diagnose COVID-19 infection, many factors such as availability of laboratory equipment, reagents and trained personnel affect the use of time-consuming molecular techniques. To facilitate on-the-spot diagnosis of COVID-19, SARS-CoV-2 rapid antigen tests were developed by several different manufacturers. The evaluation of such rapid tests is particularly important due to the recent unanimous agreement by the European Commission Member States on a recommendation setting out a framework for the use of antigen rapid tests that contains a list of the mutually recognized assays and the basis of independent validation protocols. To evaluate the on-field performance of ten commercially available SARS-CoV-2 antigen rapid tests (CLINITEST Rapid COVID-19 Antigen Test, GenBody COVID-19 Antigen Test, GENEDIA W COVID-19 Ag Test, Healgen Coronavirus Antigen Rapid Test, Humasis COVID-19 Ag Test, VivaDiag SARS-CoV-2 Ag Rapid Test, Helix i-SARS-CoV-2 Ag Rapid Test, Roche SARS-CoV-2 Rapid Antigen Test, Abbot COVID-19 Ag Rapid Test and Vazyme SARS-CoV-2 Antigen Detection Kit) and compare with RT-qPCR as a reference method, the Hungarian National Public Health Center provided 1,597 antigen rapid tests to the National Ambulance Service, COVID-testing trucks and two hospitals treating COVID-19 patients. Sensitivity, specificity and accuracy were determined by performing the rapid test directly from nasopharyngeal swab samples of symptomatic individuals. For strongly positive samples (Ct < 25) sensitivities ranged between 66.7% and 100%, while for positive samples (Ct < 30) they gave a maximum sensitivity of 87.5%. The specificity of the tests was ranging between 79% to 100%. The results presented here are of high importance to the European Commission and also help governmental decision-making regarding the application of the proper rapid tests for screening different at-risk populations. Nonetheless, SARS-Cov-2 rapid tests play an important role in early and on-the-spot diagnosis of potentially infected individuals.


Introduction
At the end of 2019, a novel respiratory disease with unknown etiology appeared in Wuhan, China. A few weeks later in January 2020, China announced that it is caused by a novel coronavirus named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [1]. Due to the rapid spatiotemporal spreading, it was declared as a pandemic by the World Health Organization (WHO) in March 2020.
In Hungary, the first diagnosed cases occurred also in March 2020, and by this date over 800,000 people got infected (in a country with population of 9.7 million) [2]. From the beginning, over 5.8 million quantitative reverse transcription polymerase chain reaction (RT-qPCR) tests were performed in national and private laboratories countrywide until early June 2021 [3].
Although RT-qPCR is the recommended laboratory method with the highest sensitivity and specificity to diagnose COVID-19 infection, molecular tests require availability of proper laboratory equipment, reagents, trained personnel and strictly regulated shipment of clinical samples [4][5][6]. Furthermore, to prevent the fourth pandemic wave, we need to increase the number of tests even more than before-besides the indisputably important vaccination campaign. To maintain and facilitate an adequate testing capacity, on-field diagnostic methods e.g., antigen-based rapid test devices (Ag-RTD) are indispensable tools beside the gold standard RT-qPCR. In fact, Ag-RTDs will never substitute the RT-qPCR in sensitivity but using them as a tool for point-of-care testing proved to be remarkably helpful for hospitals, ambulance services and mass testing. As Ag-RTDs are easy to perform and utilizing them allows to process a large number of different types of samples within a relatively short time, with the adequate sensitivity they are ideal for testing symptomatic patients and for on-field screening (e.g., in schools or university settings, also employees in offices or factories). According to the interim guidance of the Centers for Disease Control and Prevention (CDC), this is a wellfounded strategy, as screening makes asymptomatic infections easier to identify and interrupts viral transmission routes, especially in the case of high community risks. Negative Ag-RTD results should be strengthened by RT-qPCR, especially in the case of symptomatic cases; however, SARS-CoV-2 positivity within the recommended time interval means proven infection among asymptomatic carriers as well, considering false positivity is likely impossible using a properly inspected and certified (In Vitro Diagnostic-IVD) device [7].
The aim of our study was to evaluate the on-field performance of ten different, commercially available SARS-CoV-2 antigen-based rapid test devices optimized for nasopharyngeal samples among symptomatic patients and compare them to the recommended method RT-qPCR.

Study setup and clinical samples
Between 1 October 2020 and 28 February 2021, the National Public Health Center (NPHC) provided ten different SARS-CoV-2 antigen rapid tests to the National Ambulance Service, COVID testing trucks (operated by NPHC) and two hospitals treating COVID-19 patients (Országos Korányi Pulmonológiai Intézet Budapest and Szent János Kórház és Észak-budai Egyesített Kórházak, Budapest) (ethical approval number OGYÉI/58141/2020). During the strictly regulated routine sampling procedure, nasopharyngeal swab samples were collected in parallel for on-the-spot antigen rapid testing from the left nostril and left part of the throat and taken into test-specific sample buffers. For viral nucleic acid amplification (NAAT) tests samples were taken into viral transport medium (VTM) from the right part of the same respiratory area by health care professionals. A total number of 1,572 rapid testing and RT-qPCR test sampling was performed in parallel. All volunteers were informed and signed a consent form to participate. Results of the antigen tests (readout within the recommended time intervals) and the clinical specimens in VTM were transferred to the NPHC COVID-19 laboratory where the RT-qPCR was performed. Results were evaluated at the NPHC.

Antigen rapid testing
In this study, ten different commercially available SARS-CoV-2 antigen rapid tests were included (Table 1).

Nucleic acid extraction and RT-qPCR
Nucleic acid extraction and RT-qPCR were performed in the NPHC COVID-19 laboratory. Viral RNA was isolated from 200 μL of the clinical specimens using the Roche MagNA Pure 96 Instrument (Roche Life Sciences, Basel). To detect viral nucleic acid RT-qPCR was conducted on the LightCycler 480 Instrument II platform using the PerkinElmer1 SARS-CoV-2 Realtime RT-PCR Assay (PerkinElmer Inc., Waltham, Massachusetts, United States) to detect SARS-CoV-2 ORF1ab and N genes and human Rnase P gene as internal control (IC). RT-qPCR was performed according to the manufacturer's instructions. Samples positive for both SARS-CoV-2 specific genes (with Ct values < 40.0) and the internal control were considered positive. Samples that gave negative results for both ORF1ab and N genes but positive for human IC qualified as negative. RT-qPCR was concluded as inconclusive if positive or inconclusive only for one SARS-CoV-2 gene.

Statistical analysis
Statistical analysis was performed using R statistical software [8]. Overall sensitivity, specificity, accuracy, positive and negative predictive values of Ag-RTDs were estimated with 95% exact binomial confidence interval (upper limit is truncated to 1, if >1) using EpiR package (version 2.0. 19). Sensitivity was also calculated in the case of RT-qPCR positive samples with Ct values � 25 and � 30 with 95% confidence interval.
The effect of Ct value on Ag-RTD result was estimated by a logistic regression model (using logit link). The final model was the following. The antigen test result as the outcome variable was explained by the Ct value. The effect was adjusted for the type of antigen test, age, and sex. There was no interaction term in the final model. The model fit was checked by simulation model diagnostic plots using DHARMa package (version 0.3.3.0).

Sample collection and testing the study groups
An overall number of 1,572 rapid testing with 10 different Ag-RTDs and RT-qPCR sampling was performed parallel, by the National Ambulance Service, COVID testing trucks (operated by NPHC) and two COVID-19 hospitals in Budapest. The outcome of the Ag-RTD was unclear in the case of 4 cases, and RT-qPCR gave inconclusive result with 13 samples. Those samplings were discarded from the statistical analysis resulting a total number of 1,557 enrolled cases. Mean age in the study group was 41.0 years ( Table 2). An overall number of

Performance of the Ag-RTDs
To be able to compare the different Ag-RTDs, it is important to first determine the Ct value distributions measured by RT-qPCR, as Ct value is a semi-quantitative value to calculate viral load in the samples. The mean Ct value ( Table 2) was 27.1 (SD 6.13, CV% 22.7) with the minimum (highest viral load) of 13.0 and the maximum (lowest viral load) of 38.1 (Fig 1). When comparing the RT-qPCR positive samples in the case of the different Ag-RTDs, the mean Ct values were between 24.4 (SD 5.96, CV% 25.6%) and 28.3 (SD 6.30, CV% 22.3). The ratio of true positive (RT-qPCR positive) samples was between 28.9-48.5% (average 36.6%).

PLOS ONE
Field performance of SARS-Cov-2 antigen rapid tests RTD test. The probability of true positive Ag-RTD test was estimated based on a logistic regression model as shown in Fig 4. Based on this probability the sensitivity of each Ag-RTD was also calculated in the case of RT-qPCR positive samples with Ct values � 25 and � 30 with 95% confidence interval. Results are shown in Fig 5A and 5B.

Discussion
In most regions globally a decline in reported positive and fatal cases is observed mainly due to vaccination, but some countries such as India, Colombia, Malaysia and Indonesia are still facing a severe third wave of the SARS-CoV-2 pandemic. The emergence of new, highly virulent, and potentially vaccine escape SARS-CoV-2 variants indicates an acute risk for the fourth wave as previously experienced. The B.1.351 South African variant was first reported in Hungary in February and the P.1 Brazil variant was imported at the beginning of June 2021 [9][10][11][12]. The emergence of the most recent B1.617 Indian variant has been reported by several countries, in Hungary the first two cases occurred at the end of May [3,13]. With potentially rapidly increasing number in positive and fatal cases, early diagnosis of infected individuals is essential to control the rapid spread of the virus. Although RT-qPCR test is the recommended and most reliable diagnostic tool, antigen-based immunochromatographic rapid tests can

PLOS ONE
Field performance of SARS-Cov-2 antigen rapid tests provide a much faster and cost-effective method for on-the-spot diagnosis of COVID-19. During the last year over 160 different commercial tests became available from different manufacturers [14]. However, there are concerns about the overall sensitivity rates due to possible high number of false negative results associated with lower viral loads. Performance of several SARS-CoV-2 rapid antigen tests has been evaluated by many countries, leading to similar conclusions. On 21 January 2021, the European Commission Member States unanimously agreed on a Council recommendation setting a framework for the use of Ag-RTDs and the mutual recognition of COVID-19 test results across the EU [15]. The Council recommendation called on Member States to also agree on a common list of COVID-19 rapid antigen tests, whicht are considered appropriate for use and are in line with countries' testing strategies, such as carrying a CE marking, meet the minimum performance requirements of � 90% sensitivity and � 97% specificity and have been validated by at least one Member State. Of the 10 evaluated Ag-RTDs in this study, 9 tests are on the European Commission's common list of COVID-19 rapid antigen tests, while the Helix i-SARS-CoV-2 Ag Rapid Test is not recognized. Although, several studies engaged in evaluating antigen based RTD performance, none of them has tested as many types of devices with a relatively large number of samples using as our laboratory.
Of the Ag-RTDs that were not available to us, Mak et al. determined the performance of the BIOCREDIT COVID-19 rapid test using a ten-fold serial dilution of a SARS-CoV-2 virus isolate and respiratory samples from confirmed COVID-19 patients taken directly into viral transport media (VTM) or phosphate-buffered saline (PBS). Based on the result of 160 respiratory samples, the rapid test detected between 28.6-81.8% of the RT-qPCR positive, very high viral load (Ct < 18.49) samples, but only 0-21.1% of the samples with Ct values between 18.69 and 31.12. Limit of detection was Ct = 28.47 in the case of the virus isolate [16]. The COVID-19 Ag Respi-Strip (Coris Biocencept, Gembloux, Belgium) device was evaluated by Scohy et al. using 148 nasopharyngeal samples of hospitalized patients and compared to the RT-qPCR method. The overall performance was similar to the BIOCREDIT rapid test with a sensitivity of 30.2% (95% CI: 21.7-39.3%) and a specificity of 100% [17]. Laboratory evaluation of the STANDARD Q COVID-19 Ag Test (SD Biosensor, Gyeonggi-do, Republic of Korea) was performed by Nalumansi et al. using 262 nasopharyngeal swab samples from suspected COVID-19 patients. Based on their results sensitivity and specificity were 70.0% (95% CI: 60-79%) and 92% (95% CI: 87-96%), respectively; accuracy was 84% (95% CI: 79-88%). In the case of the samples with RT-qPCR Ct values � 29, sensitivity increased up to 92% [18]. In our study, we evaluated the GenBody COVID-19 Antigen Test and the Roche SARS-CoV-2 Rapid Antigen Test as well, which are from two different distributors but the same manufacturer as the STANDARD Q COVID-19 Ag Test (SD Biosensor, Gyeonggi-do, Republic of Korea) and found very similar results as Nalumansi et al. Based on the result of 98 samples, the overall sensitivity and specificity of the GenBody Ag-RTD test were found 62% (95% CI: 46-77%) and 90% (95% CI: 79-96%), respectively with accuracy of 78.57% (95% CI: 79-96%). In the case of samples with Ct < 30, sensitivity increased up to 81%. In the case of the Roche Ag-RTD test the overall sensitivity and specificity was 64% (95% CI: 52-75%) and 100% (95% CI: 97-100%) while accuracy was 86.94% (95% CI: 81-91%) based on 199 swab samples. In the case of samples with Ct � 30, sensitivity increased up to 82.5%. Diagnostic performance of Abbott's Panbio COVID-19 Ag Rapid Test (Abbott Rapid Diagnostics) was also evaluated by Landaas et al. in Norway. Their sampling strategy was identical to ours: one nasopharyngeal swab sample taken for RT-qPCR and another for rapid testing at the testing station. A total of 3,991 samples were examined in their study, collected from a single COVID-19 testing station in Oslo and municipality physicians from Norwegian regions with recent COVID-outbreaks. According to their results, the overall sensitivity and specificity of the test were 74.4% (95% CI: 49-79%) and 99.9% (95% CI: 99.7-99.9%), respectively. Among symptomatic patients, sensitivity increased up to 78.9% (95% CI: 73-84%), while in asymptomatic cases it decreased to 55.3% (95% CI: 41-69%). An increase in sensitivity was also observed among samples with Ct � 30 (83.8%; 95% CI: 78-88%) [20]. In our study (based on a total sample number of 543 samples) the overall sensitivity and specificity of the Abbott's Panbio COVID-19 Ag Rapid Test was 51% (95% CI: 44-59%) and 100% (95% CI: 98-100%), respectively. In case of samples with Ct � 30 sensitivity increased up to 76% (95% CI: 67-84%). Torres et al. determined the performance of the CLINITEST Rapid COVID-19 Antigen Test (Siemens Healthineers, Erlangen, Germany) based on 270 samples collected from patients with suspected COVID-19 infection and their contacts, using a sampling strategy similar to ours; one nasal swab obtained for RT-qPCR and another for rapid testing [21]. According to their data, in the case of symptomatic patients the sensitivity and specificity were 80.2% (95% CI: 70.9-87.1%) and 100% (95% CI: 95.8-100%), respectively, while in asymptomatic patients, sensitivity decreased to 60.0% (95% CI: 40.7-76.6%). When evaluating only strong positive samples (Ct � 25), sensitivity increased up to 95.5% (95% CI: 89-98.2%). In our study the CLINITEST Rapid COVID-19 Antigen Test was also evaluated, however, we obtained different results from Torres et al. The overall sensitivity and specificity were 48% (95% CI: 33-63%) and 98% (95% CI: 90-100%), respectively. Sensitivity in the case of samples below Ct value 25 was 79% (95% CI: 58-93%). When comparing sensitivity, we can see that 95% CI ranges overlap in the case of most evaluated Ag-RTDs (overall sensitivity 56% (95% CI: 52-60%), indicating no significant difference (Fig 2). Regarding specificity, 95% CI ranges overlap in the case of the Humasis Ag-RTD and the GenBody Ag-RTD, but not with the other eight Ag-RTDs indicating significant difference; however, the results are not corrected for multiple comparisons. We also calculated positive and negative predictive values for each rapid test, however, comparing them may be misleading as PPV and NPV are dependent from the prevalence that is different in case of the Ag-RTDs.
It is important to note that our study has no bias of freezing and thawing, as two separate samples were obtained on field from different parts of the nostrils and pharynx of every enrolled patient. The first was used for molecular testing and was taken immediately into the NPHC laboratory to perform RT-qPCR, and another sample for rapid testing was carried out on the spot by trained health care professionals. However, as errors can occur during the sampling procedures, sample amounts taken from one individual for the two different tests may vary. Another limitation of the present study is that the Abbott's Panbio COVID-19 Ag Rapid Test was available in a much higher number (N = 546) compared to the others (N = 87-199). Second, clinical symptoms were available only for half of the enrolled volunteers; thus, there was no statistical relevance in comparing the symptoms with the rapid test results.
In conclusion, it is easy and convenient to use antigen-based RTDs as a tool for point-ofcare testing among symptomatic patients; however, on-field applicability for screening of asymptomatic individuals in schools, offices or factories remains concern. In the case of nonclinical screening, by designating the appropriate test, the potentially highest sensitivity matching with highest specificity should be considered. These Ag-RTDs for screening symptomatic individuals are easier to identify, and take less time to interrupt viral transmission routes, especially in the cases of high community risk scenarios. Furthermore, such independent evaluation studies provide help to the stakeholders at the European Health Security Committee, local authorities and institutions to make the best probable decision.