Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

The importance of effect sizes when comparing cycle threshold values of SARS-CoV-2 variants

  • Celine Brinkmann,

    Roles Conceptualization, Data curation, Investigation, Methodology, Writing – original draft

    Affiliations Bioscientia Laboratory MVZ Nordrhein, Moers, Germany, Rhine-Waal University of Applied Sciences, Kleve, Germany

    ⨯
  • Peter Gohl,

    Roles Investigation, Methodology

    Affiliation Bioscientia Laboratory MVZ Ingelheim, Ingelheim, Germany

    ⨯
  • Dietrich Mack,

    Roles Investigation, Methodology

    Affiliation Bioscientia Laboratory MVZ Ingelheim, Ingelheim, Germany

    ⨯
  • Johannes Pfeifer,

    Roles Data curation, Investigation, Writing – original draft

    Affiliation Rhine-Waal University of Applied Sciences, Kamp-Lintfort, Germany

    ⨯
  • Mònica Palmada Fenés,

    Roles Supervision, Writing – review & editing

    Affiliation Rhine-Waal University of Applied Sciences, Kleve, Germany

    ⨯
  • Oliver Harzer,

    Roles Investigation, Methodology

    Affiliations Bioscientia Laboratory MVZ Ingelheim, Ingelheim, Germany, Danube Private University (DPU), Krems an der Donau, Austria

    ⨯
  • Bernhard Zöllner

    Roles Conceptualization, Data curation, Investigation, Methodology, Supervision, Writing – original draft

    bernhard.zoellner@bioscientia.de

    Affiliation Bioscientia Laboratory MVZ Nordrhein, Moers, Germany

    ⨯

Abstract

Purpose

We aimed to elaborate whether cycle threshold (Ct) values differ significantly between wild type SARS-CoV-2 (wtV) and certain viral variants and how strong or weak a potential significant effect might be.

Methods

In a retrospective study, we investigated 1873 SARS-CoV-2 positive samples for the occurrence of viral marker mutations. Age, gender, clinical setting, days after onset of symptoms, and Ct values were recorded. Statistical analysis was carried out with special consideration of effect sizes.

Results

During the study period wtV was detected in 1013 samples (54%), while 845 (45%) patients carried the Alpha variant of concern (VOC), and 15 (1%) the Beta VOC. For further analysis, only wtV and the Alpha VOC were included. In a multi-factor ANOVA and post-hoc test with Bonferroni-correction for the age groups we found significant main-effects for Ct values of the viral variant (wtV mean 26.4 (SD 4.27); Alpha VOC mean 25.0 (SD 3.84); F (1,1850) = 55.841; p < .001) and the clinical setting (outpatients: mean 25.7 (SD 4.1); inpatients: mean 27.0 (SD 4.2); F (1,1850) = 8.520, p = .004). However, since the effect sizes were very small (eta squared for the Alpha VOC = .029 and the clinical setting = .004), there was only a slight trend towards higher viral loads of the Alpha VOC compared to wtV.

Conclusions

In order to compare different variants of SARS-CoV-2 the calculation of effect sizes seems to be necessary. A combination of p-values as estimates of the existance of an effect and effect sizes as estimates of the magnitude of a potential effect may allow a better insight into transmission mechanisms of SARS-CoV-2.

Citation: Brinkmann C, Gohl P, Mack D, Pfeifer J, Palmada Fenés M, Harzer O, et al. (2022) The importance of effect sizes when comparing cycle threshold values of SARS-CoV-2 variants. PLoS ONE 17(7): e0271808. https://doi.org/10.1371/journal.pone.0271808

Editor: Etsuro Ito, Waseda University: Waseda Daigaku, JAPAN

Received: March 18, 2022; Accepted: July 7, 2022; Published: July 21, 2022

Copyright: © 2022 Brinkmann et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

The new pandemic coronavirus SARS-CoV-2 has the capacity to select for mutational variants. Particularly variants of concern (VOC) pose a considerable threat because they exhibit increased transmissibility, detrimental change in COVID-19 epidemiology, an increased virulence or change in clinical disease presentation or a decreased effectiveness of public health and social measures or available diagnostics, vaccines and therapeutics [1, 2].

Currently, five VOCs have been described in the European Union: the Alpha, Beta, Gamma, Delta, and Omicron variant of concern [3]. In January 2021, almost exclusively wild-type SARS-CoV-2 (wtV) circulated in Germany. However, in the further course, it was more and more replaced by the Alpha VOC, reaching a 88% prevalence of this variant at the end of March 2021 [4]. During this period, it was possible to compare the properties and kinetics of wild type and mutated SARS-CoV-2 in a direct way.

Cycle threshold (Ct) values that are generated in a real-time polymerase chain reaction (PCR), are a way of estimating the virus concentration in a sample semi-quantitatively. This roughly reflects the viral load in the patient’s material and could allow conclusions to be drawn about the infectiousness of a patient [5].

The Ct values of the individual VOCs were examined and compared in some studies [6–9]. The results of these investigations were that significant differences between Ct values of VOCs could be determined. However, statistical significance is only one part of the equation. While statistical significance shows that an effect exists in a study and is unlikely to be caused by chance, effect sizes describe whether the effect is large enough to be meaningful in the real world [10]. Particularly with large sample sizes it is more likely that significant results with low p-values can be found in quantitative studies [11]. Therefore, reporting both p-values and effect sizes is much more informative and relevant for the situation in vivo than reporting only p-values. For this purpose, effect sizes can be calculated within different statistical models. In ANOVA, eta squared (η2) is well suited for the calculation of the effect size and gives an estimate how strong the variation, e.g. of the viral load, can be accounted to each factor or predictor. This makes it possible to classify which factor is more or less important.

Our study aimed to investigate whether there was a significant difference in Ct values between wild-type SARS-CoV-2 (wtV) and its mutational variants of concern. Particularly the influence of multiple cofactors like age, gender, days from onset of symptoms, and clinical setting on Ct values were analysed with special consideration of the magnitude of the effect.

Material and methods

Subjects and samples

The study period of this retrospective analysis lasted from January 20th until March 17th 2021. All positive samples in the even-numbered weeks, regardless of the measured Ct-values, and a collective of samples with a Ct value <30 in the uneven numbered weeks were included. Dry swabs without transport medium were taken from the patients´ naso- or oropharyngeal area, shipped to our laboratory within 24–48 hours at room temperature and processed within a mean time of 22 hours. For all patients, age, gender, time of sampling, in- or outpatient status, and Ct values were documented. In telephone calls with the institutions that sent positive samples, the time between patient´s onset of symptoms and sampling was determined.

SARS-CoV-2 PCR and determination of SARS-CoV-2 VOCs

PCR was run on a cobas® 8800 instrument (Roche Diagnostics, Mannheim, Germany) using the cobas® SARS-CoV-2 test (also Roche Diagnostics, Mannheim, Germany), which includes specific detection of the ORF1 a/b non-structural region and E-gene. Pool tests were not performed, each sample was tested individually. To ensure (i) the stability of the viral RNA during transport and (ii) the reproducibility of our PCR system we investigated three previously tested samples with different Ct values. These samples were stored at room temperature and retested on three consecutive days. Thereafter, the day-to-day coefficients of variation were calculated for each of these samples.

SARS-CoV-2 mutational variants were analysed with the cobas® SARS-CoV-2 Variant Set-1, which enabels the detection of the key mutations for the Alpha-, Beta-, and Gamma-VOCs (N501Y, E484K, and the 69–70 deletion).

Statistical methods

The Mann-Whitney U-test was applied in a first attempt using Abacus 2.0 (Labanalytics, 2021, Jena, Germany) to analyse Ct values in females and males as well as in- and outpatients and different age groups, as described previously [e.g. references 5 and 6]. Using this test, the Ct values ​​of wtV and the Alpha VOC for the respective groups were compared. Thus, two groups were formed for these demographic factors, each consisting of unpaired samples. Similarly, rates of hospitalized patients with wtV and the Alpha VOC were compared with Fisher´s exact test.

To avoid alpha-error-accumulation which occurs with e.g. multiple testing we ran a multi-factor Analysis of Variance (ANOVA)(IBM SPSS for Windows, Version 27.0, 2020, Armonk, NY, USA) afterwards and compared the results of both analyses. The ANOVA model term contained exclusively the main effects, no interactions were included. The factor “days after onset of symptoms” was excluded due to a low sample size. All the assumptions for the calculation, including equality of variances (F = 1.026; p = .427) were met. Post-hoc-test with Bonferroni-correction was calculated for the age factor, as it had more than two groups. A p-level of <0.05 was considered to be statistically significant.

The effect sizes were calculated with the designated standard function of SPSS (see in S1 Table). Values of η2 between 0.01 and 0.059 are considered as small effects, those between 0.06 and 0.139 correspond to a medium effect, and of 0.14 or greater to a large effect.

Results

A total of 1873 SARS-CoV-2 positive samples were analysed with Ct values ranging between 13–36. The reproducibility and stability of Ct values over a period of three days at room temperature was high: repeated measurements of three samples on three consecutive days showed a coefficient of variation of 3% (sample 1, Ct value 17), of 2.2% (sample 2, Ct value 23), and of 2.3% (sample 3, Ct value 28).

Wild type virus was detected in 1013 samples (54%), 845 (45%) patients carried the Alpha VOC, and 15 (1%) the Beta VOC. The distribution of patients between outpatients and inpatients was 954/59 for wtV, 817/28 for the Alpha VOC, and 15/0 for the Beta VOC, respectively. The prevalence of the Alpha VOC changed from 9% (41/464) in calendar week 4 to 74% (406/552) in calendar week 10 (Fig 1). The Beta VOC played a subordinate role with 0.9% (4/438) in week 8 and 2% in week 10 (11/552) (Fig 1).

thumbnail
Download:
Fig 1. Prevalence of all positives of wild type SARS-CoV-2 (wtV) and the Alpha and Beta VOCs.

https://doi.org/10.1371/journal.pone.0271808.g001

Due to the low case number (n = 15), the Beta VOC was excluded from further analysis. Table 1 shows the results of the remaining samples (n = 1858) using a Mann-Whitney-U two-group comparison as described in other publications [e.g. references 5 and 6]. Significantly, lower Ct values of the Alpha VOC were detected for the factor gender (p<0.0001), outpatients (p<0.0001), and age groups 0–80 years (p<0.004). The rate of inpatients carrying the Alpha VOC (28/845) was significantly lower than those carrying wtV (59/1013, p = 0.011). The time between the onset of symptoms and the day of sampling could be determined cross-sectionally for a subgroup of 242 patients until day 5 of the infection. The results revealed only a weak significance on day one showing slightly lower Ct values of the Alpha VOC compared to wtV (25.0 (SD 3.47) vs. 26.8 (SD 3.54), p = 0.03) (Table 1). Days 2–5 showed no significant differences in Ct values between wtV and the Alpha VOC. Hence, this parameter was excluded from the subsequent multiple factor analysis.

thumbnail
Download:
Table 1. Ct values and standard deviations (SD) of wild type SARS-CoV-2 (wtV) and the Alpha variant of concern (VOC) with p-values derived from the associated Mann-Whitney-U tests.

https://doi.org/10.1371/journal.pone.0271808.t001

To avoid the multiple comparison problem of testing, and further investigate the relevance of all factors together, we employed a multi-factor ANOVA. This model showed significant main effects on Ct values for the factors viral variant (F (1,1850) = 55.841, p < .001), age (F (4,1850) = 3.297, p = .011), and setting (F (1,1850) = 8.520, p = .004). The gender groups did not differ significantly regarding the viral load. The effect sizes showed a small effect for the Alpha VOC (η2 = .029) as well as for the setting (η2 = .004) and age (η2 = .007), meaning that e.g. 2.9% of the variation in the Ct values account for the factor viral variant. Wildtype virus had slightly higher Ct values (mean 26.4, SD 4.27) compared to the Alpha VOC (mean 25.0, SD 3.84) and total inpatients had slightly higher Ct values (mean 27.0, SD 4.2) than total outpatients (mean 25.7, SD 4.12) (Fig 2). Post-hoc tests with Bonferroni-correction showed no significant differences between the age categories, so the small, significant main effect was neglected. Taken together, significant differences of Ct values could be found between wtV and the Alpha VOC and between each total in- and outpatients, however with only very small effect sizes.

thumbnail
Download:
Fig 2. Box-Whisker-Plots of Ct values for SARS-CoV-2.

(Left) Type of SARS-CoV-2. (Right) Clinical setting. Eta squared (η2) indicates the effect size.

https://doi.org/10.1371/journal.pone.0271808.g002

Discussion

The investigation of viral variants of SARS-CoV-2 is becoming increasingly important. There has been a great effort to characterize and compare wild type SARS-CoV-2 and its mutational variants in vitro [12–17]. In these studies, increased infectivity and faster viral entry into target cells have been shown for VOCs compared to wtV. However, studies on the comparison in vivo are difficult to perform as wtV and its variant should circulate in the population at the same time. That is why there are only a few analyses with this approach [6–9, 18–22].

In our analysis, patients carrying the Alpha VOC had a mean Ct value that was one PCR cycle lower than patients carrying wtV, which is in line with previously published studies [6, 7]. Also, in the study by Luo et al. [8] the mean difference in Ct values between the B.1.2, Delta and Alpha variants was in the range of 1–2 PCR cycles (20.61 vs. 19.62 vs. 21.77, respectively). Although these and our results were statistically highly significant, the mean difference of 1–2 PCR cycles is low. P-values, however, are confounded by large sample sizes [11]. This could have been true for at least our study and was the reason why we calculated effect sizes. Also, there is another problem, namely the lack of standardization of the pre-analytical phase prior to the PCR for SARS-CoV-2. For instance, the technique of swabbing or the influence of transport conditions on individual samples could cause a bias in the evaluation of Ct values. One possible solution to balance out preanalytical errors in single samples is to analyse large patient groups, as it was done in our study. However, as mentioned above, this requires the specification of effect sizes in addition to the analysis of p-values.

We found lower Ct values of the Alpha VOC ​​in both sexes, in outpatients and almost all age groups. However, a subsequent multiple-factor analysis led to the result that there was only a significant difference between wtV and the Alpha VOC in the total cohort and between in- and outpatients concerning Ct values and, of great importance, that the effect size was low for these factors. Particularly, the low influence of the viral variant on Ct values was surprising, considering the observation of higher transmissibility of the Alpha VOC compared to wtV [6–9, 18–22]. A plausible explanation could be an increased binding affinity of this variant to the ACE2 receptor [14–16]. This leads to higher infectivity even with comparable virus concentrations [17].

Our study covered a period during which the Alpha VOC had almost completely replaced the wtV. The prevalence rates in our cohort correlate well to the general German prevalence rates determined by the Robert Koch Institute [4]. In addition, hardly any people were vaccinated at that time in Germany. Therefore, a direct comparison of Ct values in currently taken samples was possible and the results were not biased by different immune protection rates of infected persons.

Because the time since symptom onset is a key determinant of viral load in SARS-CoV-2 infections [23, 24], we tried to control for this cofactor. A limitation was, that we could only retrospectively record the days between onset of symptoms and sampling in a cross-sectional manner and that no longitudinal examination of the Ct values could be carried out in individuals during the acute phase of the infection. Nevertheless, our results implicate that apart from slightly lower Ct values of the Alpha VOC at day one there were no significant differences between wtV and VOC in the Ct values even during the first five days of the disease. This strengthens the hypothesis that the increased infectiousness of the Alpha VOC in vivo relies on other factors rather than high viral concentrations during the initial phase of the infection process. In vivo, factors like receptor-affinity [14–16] or enhanced target cell entry [12, 15] e.g. associated with altered spike cleavage sites [12] of the Alpha VOC seem to play a more important role.

Conclusions

For future studies elaborating Ct values of e.g. the Delta, Omicron (in particular BA.2 or BA.5) or other emerging VOCs, statistical analyzes with multiple factors seem to be essential. Additionally, we propose the calculation of effect sizes, which would allow an estimation of the magnitude of differences in Ct values ​​and of their true meaning in vivo. This could provide clues to the really decisive factors in such a complicated, multifactorial process as the infectivity and transmission of SARS-CoV-2.

Supporting information

S1 Table. Output of effect sizes in SPSS for the data of our study.

https://doi.org/10.1371/journal.pone.0271808.s001

(PNG)

S1 Dataset. Anonymized original dataset.

https://doi.org/10.1371/journal.pone.0271808.s002

(PDF)

References

  1. 1. World Health Organisation. Tracking SARS-CoV-2 variants [Online]. 2020. Available: https://www.who.int/en/activities/tracking-SARS-CoV-2-variants/ (accessed on 7 September 2021).
  2. 2. European Centre for Disease Prevention and Control (ECDC). 2021. SARS-CoV-2 variants of concern as of 8 July 2021 [Online]. Available: https://www.ecdc.europa.eu/en/covid-19/variants-concern (accessed on 7 September 2021).
  3. 3. European Center for Disease Prevention and Control (ECDC). SARS-CoV-2 variants of concern as of 13 January 2022. https://www.ecdc.europa.eu/en/covid-19/variants-concern (accessed on 19 January 2022).
  4. 4. Robert-Koch-Institut. Aktualisierter Bericht zu Virusvarianten von SARS-CoV-2 in Deutschland, insbesondere zur Variant of Concern (VOC) B.1.1.7. https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/DESH/Bericht_VOC_2021-04-14.pdf?__blob=publicationFile, [Accessed 03.September 2021].
  5. 5. Bullard J, Dust K, Funk D, Strong JE,Alexander D, Garnett L et al. Predicting infectious Severe Acute Respiratory Syndrome Coronavirus 2 from diagnostic samples. Clin Infect Dis. 2020;71(10):2663–2666. pmid:32442256
  6. 6. Calistri P, Amato L, Puglia I, Cito F, Di Giuseppe A, Danzetta ML, et al. Infection sustained by lineage B.1.1.7 of SARS-CoV-2 is characterised by longer persistence and higher viral RNA loads in nasopharyngeal swabs, Int J Infect Dis. 2021;105:753–755. pmid:33684558
  7. 7. Roquebert B, Haim-Boukobza S, Trombert-Paolantoni S, Lecorche E, Verdurme L, Foulonge V, et al. SARS-CoV-2 variants of concern are associated with lower RT-PCR amplification cycles between January and March 2021 in France. Int J Infect Dis. 2021;113:12–14. pmid:34601145
  8. 8. Luo CH, Morris CP, Sachithanandham J, Amadi A, Gaston DC, Li M, et al. Infection with the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Delta variant is associated with higher recovery of infectious virus compared to the Alpha variant in both unvaccinated and vaccinated individuals. Clin Infect Dis. 2021;ciab986. pmid:34922338
  9. 9. Kidd M, Richter A, Best A, Cumley N, Mirza J, Percival B, et al. S-variant SARS-CoV-2 lineage B1.1.7 is associated with significantly higher viral loads in samples tested by ThermoFisher TaqPath RT-qPCR. J Infect Dis. 2021;223(10):1666–1670. pmid:33580259
  10. 10. Bhandari P. What is effect size and why does it matter? Scribbr. 2022. [published 2020 Dec 22, revised 2022 Jun 01, cited 2022 Jun 08]. Available from: https://www.scribbr.com/statistics/effect-size/
  11. 11. Bakker A, Cai J, English L, Kaiser G, Mesa V, van Dooren W. Beyond small, medium, or large: points of consideration when interpreting effect sizes. Educ Stud Math. 2019;102:1–8. https://doi.org/10.1007/s10649-019-09908-4
  12. 12. Meng B, Kemp SA, Papa G, Datir R, Ferreira IATM, Marelli S, et al. Recurrent emergence of SARS-CoV-2 spike deletion H69/V70 and its role in the Alpha variant B.1.1.7, Cell Rep. 2021;35(13):109292. pmid:34166617
  13. 13. Zhang J, Xiao T, Cai Y, Lavine CL, Peng H, Zhu H, et al. Membrane fusion and immune evasion by the spike protein of SARS-CoV-2 Delta variant. Science. 2021;374:1353–1360. pmid:34698504
  14. 14. Kim S, Liu Y, Lei Z, Dicker J, Cao Y, Zhang XF, et al. Differential interactions between human ACE2 and spike RBD of SARS-CoV-2 variants of concern. J Chem Theory Comput. 2021;17:7972–7979. pmid:34856802
  15. 15. Kim Y, Gaudreault NN, Meekins DA, Perera KD, Bold D, Trujillo JD, et al. Effects of spike mutations in SARS-CoV-2 variants of concern on human or animal ACE2-mediated virus entry and neutralization. Microbiol Spectr. 2022; ahead of print. pmid:35638818
  16. 16. Starr TN, Greaney AJ, Hilton SK, Ellis D, Crawford KHD, Dingens AS, et al. Deep mutational scanning of SARS-CoV-2 receptor binding domain reveals constraints on folding and ACE2 binding. Cell. 2020;182:1295–1310. pmid:32841599
  17. 17. Hu J, He CL, Gao QZ, Zhang GJ, Cao XX, Long QX, et al. The D614G mutation of SARS-CoV-2 spike protein enhances viral infectivity and decreases neutralization sensitivity to individual convalescent sera. BioRrixv preprint https://doi.org/10.1101/2020.06.20.161323
  18. 18. Tanaka H, Hirayama A, Nagai H, Shirai C, Takahashi Y, Shinomiya H, et al. Increased transmissibility of the SARS-CoV-2 Alpha variant in a Japanese population. Int J Environ Res Public Health. 2021;18:7752. pmid:34360046
  19. 19. Graham MS, Sudre CH, May A, Antonelli M, Murray B, Varsavsky T, et al. Changes in symptomatology, reinfection, and transmissibility associated with the SARS-CoV-2 variant B.1.1.7: An ecological study. Lancet Public Health. 2021;6:e335–e345. pmid:33857453
  20. 20. Davies NG, Abbott S, Barnard RC, Jarvis CI, Kucharski AJ, Munday JD, et al. Estimated transmissibility and impact of SARS-CoV-2 lineage B.1.1.7 in England. Science. 2021;372:6538. pmid:33658326
  21. 21. Volz E, Mishra S, Chand M, Barrett JC, Johnson R, Geidelberg L, et al. Assessing transmissibility of SARS-CoV-2 lineage B.1.1.7 in England. Nature. 2021;593:266–269. pmid:33767447
  22. 22. Leung K, Shum MH, Leung GM, Lam TT, Wu JT. Early transmissibility assessment of the N501Y mutant strains of SARS-CoV-2 in the United Kingdom, October to November 2020. Eurosurveillance. 2021;26:2002106. pmid:33413740
  23. 23. Biguenet A, Bouiller K, Marty-Quinternet S, Brunel AS, Chirouze C, Lepiller Q. SARS-CoV-2 respiratory viral loads and association with clinical and biological features. J Med Virol. 2021;93:1761–1765. pmid:32889755
  24. 24. Salvatore PP, Dawson P, Wadhwa A, Rabold EM, Buono S, Dietrich EA, et al. Epidemiological correlates of PCR cycle threshold values in the detection of SARS-CoV-2. Clin Infect Dis. 2021;72:e761–e767. pmid:32986120