Assessing the performance of a serological point-of-care test in measuring detectable antibodies against SARS-CoV-2

This study investigated the performance of a rapid point-of-care antibody test, the BioMedomics COVID-19 IgM/IgG Rapid Test, in comparison with a high-quality, validated, laboratory-based platform, the Roche Elecsys Anti-SARS-CoV-2 assay. Serological testing was conducted on 709 individuals. Concordance metrics were estimated. Logistic regression was used to assess associations with seropositivity. SARS-CoV-2 seroprevalence was 63.5% (450/709; 95% CI 59.8%-67.0%) using the BioMedomics assay and 71.9% (510/709; 95% CI 68.5%-75.2%) using the Elecsys assay. There were 60 discordant results between the two assays, all of which were seropositive in the Elecsys assay, but seronegative in the BioMedomics assay. Overall, positive, and negative percent agreements between the two assays were 91.5% (95% CI 89.2%-93.5%), 88.2% (95% CI 85.1%-90.9%), and 100% (95% CI 98.2%-100%), respectively, with a Cohen’s kappa of 0.81 (95% CI 0.78–0.84). Excluding specimens with lower (Elecsys) antibody titers, the agreement improved with overall, positive, and negative percent concordance of 94.4% (95% CI 92.3%-96.1%), 91.8% (95% CI 88.8%-94.3%), and 100% (95% CI 98.2%-100%), respectively, and a Cohen’s kappa of 0.88 (95% CI 0.85–0.90). Logistic regression confirmed better agreement with higher antibody titers. The BioMedomics COVID-19 IgM/IgG Rapid Test demonstrated good performance in measuring detectable antibodies against SARS-CoV-2, supporting the utility of such rapid point-of-care serological testing to guide the public health responses and vaccine prioritization.


Introduction
Coronavirus disease 2019 , caused by the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), continues to present a global challenge, leading to health, social, and economic burdens [1]. Qatar experienced a large first SARS-CoV-2 epidemic wave in 2020, with a high rate of laboratory-confirmed infections at >60,000 infections per million population [2][3][4]. The wave predominantly affected the craft and manual workers who constitute just over half of Qatar's total population [2]. Seroprevalence in this part of the population was measured at about 60% following this wave [5,6].
Following this epidemic wave, Qatar's public health authorities expanded serological testing for SARS-CoV-2 antibodies, for both healthcare and research purposes [6][7][8]. Moreover, antibody status was deliberated as one of the criteria for COVID-19 vaccine prioritization [9], and for a waiver of the quarantine requirement for international travelers [10].
To achieve more efficient, cost-effective, and widescale serological testing, the objective of this study was to compare the performance of a rapid point-of-care antibody test, the BioMedomics COVID-19 IgM/IgG Rapid Test [11], to a high-quality, validated, laboratory-based and automated assay, the Roche Elecsys Anti SARS-CoV-2 platform [12,13], one of the most extensively used and investigated commercial platforms, having a specificity �99.8% [14,15] and a sensitivity �89% [12,14]. The relevance of this study is grounded on the utility of knowing antibody status as it can facilitate management of international travel [10], and importantly can optimize vaccination strategies, such as by delaying vaccination for those with prior infection [9], or by offering only one dose to those with a prior infection [16][17][18].

Materials and methods
The study sample included 709 residual blood serum specimens that were collected and then tested for SARS-CoV-2 antibodies between October 10-21, 2020, from individuals receiving routine or other clinical care at Hamad Medical Corporation (HMC), the main provider of healthcare in Qatar, and the nationally designated provider for all COVID-19 healthcare needs. Qatar has a universal and modern healthcare system that is heavily subsidized and accessible to nationals and expatriate residents [8]. HMC provides the core of public healthcare services in Qatar, and has about 85% of the hospital bed capacity in the country. The 709 specimens used in this study were chosen from the residual blood serum specimens collected from outpatient and inpatient attendees at HMC [8].
The BioMedomics COVID-19 IgM/IgG Rapid Test (hereafter "BioMedomics") is a lateral flow immunoassay that contains a colloidal, gold-labeled, recombinant coronavirus antigen and a quality control antibody colloidal gold marker, two detection lines (IgG and IgM lines), and one quality control line (C) fixed on a nitrocellulose membrane [11]. The antigen used in this assay is SARS-CoV-2 MK201027 antigen that is found in the receptor binding domain of the spike protein [19]. Qualitative anti-SARS-CoV-2 results were generated by reading the detection line(s) [11].
Results of the serological testing were subsequently linked to the national centralized SARS-CoV-2 real-time reverse-transcription polymerase chain reaction (RT-PCR) testing and hospitalization database that includes records for all RT-PCR testing and COVID-19 hospitalizations in Qatar since the start of the epidemic [2]. The database also includes the severity classification of hospitalized cases based on individual chart reviews completed by trained medical personnel using the World Health Organization (WHO) criteria [20].
All RT-PCR tests were conducted at the HMC Central Laboratory or Sidra Medicine Laboratory, following standardized protocols.
Cross-tabulations of the serological testing results were conducted using the Elecsys assay as the reference standard. Concordance metrics were estimated and included the positive, negative, and overall percent agreements, noting that this study was not designed to assess sensitivity and specificity of the BioMedomics assay. In addition, Cohen's kappa statistic [27] was estimated to measure the level of agreement, beyond chance, between the two diagnostic approaches. Ranging between 0 and 1, a kappa statistic <0.40 indicates poor agreement, a value between 0.40 and 0.75 denotes fair/good agreement, and a value >0.75 signifies excellent agreement [27]. Statistical significance was set at 5% and a 95% confidence interval (CI) was estimated for each metric.
Univariable and multivariable logistic regressions were conducted to assess associations between seropositivity using the BioMedomics assay and each of the following covariates: RT-PCR cycle threshold (Ct) value, optical density value of the Elecsys assay result, and severity of infection. Covariates with p-values �0.2 in the univariable regression analysis were included in the multivariable model. Covariates with p-values �0.05 in the multivariable analysis were regarded as covariates with strong evidence for an association with the outcome. Analyses were performed using Microsoft Excel and IBM-SPSS version 26.0.
The research methods were approved on June 3, 2020 by the ethics review boards at HMC (HMC IRB number MRC-05-133) and Weill Cornell Medicine-Qatar (WCM-Q IRB number 21-00001), with waiver of informed consent.

PLOS ONE
Validation of a rapid SARS-CoV-2 serological test Among those with a prior RT-PCR-positive result and seropositive in the Elecsys assay, 92.3% (241/261; 95% CI 88.4%-95.3%) were seropositive in the BioMedomics assay. Two individuals were seronegative in both antibody assays, but had a prior RT-PCR-confirmed SARS-CoV-2 diagnosis (S1 Table). The first individual was diagnosed on May 21, 2020 (Ct value was 35.7), 153 days prior to the blood collection date. With the high Ct value, one cannot exclude the possibility of an RT-PCR false positive result. The second individual was diagnosed on October 13, 2020 (Ct value was 15.6), only four days prior to the blood collection date, and thus the blood was probably drawn too early to detect development of antibodies.
Data on COVID-19 disease severity per the WHO classification (derived from the national COVID-19 hospitalization database [2,29]) were available for 47 persons: 7 individuals were classified as mild, 19 were classified as moderate, 14 were classified as severe, and 7 were classified as critical (S1 Table). For all other individuals no severity classification was conducted, due to absence of serious symptoms to require hospitalization and severity assessment, and thus the infection can be assumed to be asymptomatic or mild. No COVID-19 deaths were reported among study participants.
Including only specimens with higher antibody titers in the comparison (i.e., excluding specimens with low Elecsys optical density values <10), the Cohen's kappa statistic was estimated at 0.88 (95% CI 0.85-0.90) indicating "excellent" agreement [27] between the two assays, and at higher value compared to the result for the full sample (Table 2). Including only specimens with low antibody titers in the comparison (i.e. excluding specimens with Elecsys optical density values �10), the Cohen's kappa statistic was estimated at 0.76 (95% CI 0.70-

PLOS ONE
Validation of a rapid SARS-CoV-2 serological test 0.82) also indicating "excellent" agreement [27] between the two assays, but still an inferior agreement compared to the result for the subsample of higher antibody titers, and for the full sample ( Table 2). The cutoff at Elecsys optical density value of 10 to distinguish low from higher antibody titers was informed by the distribution of the optical density values (Fig 1). Specimens with Elecsys optical density value <10 constituted 15.9% (81/510) of all seropositive specimens by the Elecsys assay.
The multivariable logistic regression identified significant associations between seropositivity using the BioMedomics assay and both the Elecsys optical density value and the RT-PCR Ct value or its absence (that is no prior RT-PCR-confirmed infection), but no association with the severity of the infection ( Table 3). The adjusted odds ratio (aOR) of seropositivity was 6.14 (95% CI 3.31-11.4, p<0.001) for those with higher antibody titers (Elecsys optical density values �10) compared to those with low antibody titers (Elecsys optical density values <10), reflecting much higher agreement between the two measures for specimens with higher antibody titers.
The aOR of seropositivity was 0.24 (95% CI 0.09-0.65, p = 0.005) for those with a Ct value �30 compared to those with a Ct value <30, possibly because of some RT-PCR false positivity measures for those with higher Ct values. The aOR of seropositivity was 0.27 (95% CI 0.13-0.56, p<0.001) for those with no Ct value (that is no prior RT-PCR-confirmed infection), reflecting the lower likelihood of having been exposed to the infection if they were never diagnosed with the infection using RT-PCR.
A sensitivity analysis for the multivariable logistic regression, using the Ct value cut-off of 35 instead of 30, confirmed the same results and suggested higher occurrence of RT-PCR false positive results when the Ct value is �35 than when the Ct value is �30 (S3 Table).

Discussion
The above results document good performance of the BioMedomics rapid test in measuring detectable antibodies against SARS-CoV-2. The performance of this point-of-care test was also

PLOS ONE
Validation of a rapid SARS-CoV-2 serological test as expected for a rapid test: it was optimal when antibody titers were high, but less optimal when they were low. However, this should not be a hindrance for its use in facilitating optimal vaccination strategies, such as delaying vaccination for those with a record of prior infection [9], or offering only one vaccine dose to those who are seropositive [16][17][18]. Given the good performance of the BioMedomics assay, this assay can be used for a rapid assessment of prior exposure whether in clinical settings or for self-testing, as also seen in other published studies for this assay [19,25]. These findings demonstrate the utility of this assay to assess past SARS-CoV-2 infection and seropositivity as a marker of immunity against infection, particularly considering the growing evidence indicating that natural infection elicits strong protection against reinfection that lasts for at least several months, even against variants of concern [10, 21,22,[30][31][32]. These findings also support the concept of using rapid antibody testing for more efficient, cost-effective, and widescale serological testing to guide vaccine prioritization, or possible issuance of immunity passports, given supporting evidence for protection against infection (and not only disease) after prior infection or vaccination [10,[33][34][35].
This study has several limitations. We investigated the performance of serological testing on stored serum specimens and not on whole blood, and thus the real-world performance of this rapid assay in a point-of-care setting might be lower. The performance of the rapid assay was compared to a laboratory-based assay that uses a different antigen target, the Roche Elecsys Anti SARS-CoV-2 platform (and to RT-PCR testing), but not to a gold standard neutralization test, as such test was not available to the study investigators. Having said so, the Elecsys assay is one of the most extensively used and investigated commercial platforms, having a specificity �99.8% [13][14][15] and a sensitivity �89% [12,14]. Therefore, it is not likely that this limitation could have affected the findings.
False negative results could have been due to waning of antibody titers over time, especially with a long delay between the primary infection and the antibody test. Although Cohen's kappa is a conventional measure of agreement between assays, its performance can vary with the variation in infection prevalence in the population, despite equal test performance in terms of sensitivity and specificity. Despite the high kappa values qualifying the agreement between the two assays as "excellent", the BioMedomics assay missed detection of SARS-CoV-2 antibodies in >10% of seropositive specimens by Elecsys. We did not investigate the effects of subjective elements on the results' interpretation, such as by different individuals implementing, reading, and interpreting each test separately on the same blood specimen. Nonetheless, there does not appear to be strong reasons to suspect interference of subjective elements on the results' interpretation. Effect of disease severity on antibody titers was not investigated as the number of severe cases was low in the study samplethe epidemic in Qatar had relatively low severity with the young and working-age demographic structure of the population [2,4,36]. Test performance by time since infection was not investigated as record of prior infection was available only for a subset of seropositive individuals. Test performance was investigated for natural-infection antibodies, but not for vaccine-induced antibodies. Such lines of further investigation might be addressed in future studies.
In conclusion, the BioMedomics rapid point-of-care test demonstrated good performance in measuring detectable antibodies against SARS-CoV-2, with better performance for specimens with higher antibody titers, demonstrating the utility of such assays in mass expansion of serological testing to guide public health responses and vaccine prioritization.

S2 Table. Concordance metrics between two SARS-CoV-2 antibody assays: The BioMedomics COVID-19 IgM/IgG Rapid Test and the Roche Elecsys Anti-SARS-CoV-2 including only specimens taken �14 days after the prior RT-PCR-confirmed infection. (DOCX)
S3 Table. Results of univariable and multivariable logistic regression, assessing the association between seropositivity using the BioMedomics COVID-19 IgM/IgG Rapid Test and the following covariates: RT-PCR Ct value (cut-off at 35), Elecsys optical density value, and severity of infection. (DOCX) generating databases for COVID-19 infection, as well as Dr. Asmaa Al-Thani, Chairperson of the Qatar Genome Programme Committee and Board Vice Chairperson of QBB, for her leadership of this effort. We also acknowledge the dedicated efforts of the Clinical Coding Team and the COVID-19 Mortality Review Team, both at Hamad Medical Corporation, and the Surveillance Team at the Ministry of Public Health. We thank Dr. Katharine J. Looker, medical writer from the University of Bristol, for reviewing the manuscript.