https://orcid.org/0000-0002-8162-5090
Figures
Abstract
Public health policy impact evaluation is challenging to study because randomized controlled experiments are infeasible to conduct, and policy changes often coincide with non-policy events. Quasi-experiments do not use randomization and can provide useful knowledge for causal inference. Here we demonstrate how longitudinal wastewater monitoring of viruses at a small geographic scale may be used in a quasi-experimental design to evaluate the impact of COVID-19 public health policies on the spread of COVID-19 among a university population. We first evaluated the correlation between incident, reported COVID-19 cases and wastewater SARS-CoV-2 RNA concentrations and observed changes to the correlation over time, likely due to changes in testing requirements and testing options. Using a difference-in-differences approach, we then evaluated the association between university COVID-19 public health policy changes and levels of SARS-CoV-2 RNA concentrations in wastewater. We did not observe changes in SARS-CoV-2 RNA concentrations associated with most policy changes. Policy changes associated with a significant change in campus wastewater SARS-CoV-2 RNA concentrations included changes to face covering recommendations, indoor gathering bans, and routine surveillance testing requirements and availability.
Citation: Chan EMG, Bidwell A, Li Z, Tilmans S, Boehm AB (2024) Public health policy impact evaluation: A potential use case for longitudinal monitoring of viruses in wastewater at small geographic scales. PLOS Water 3(6): e0000242. https://doi.org/10.1371/journal.pwat.0000242
Editor: Ricardo Santos, Universidade Lisboa, Instituto superior Técnico, PORTUGAL
Received: February 2, 2024; Accepted: May 5, 2024; Published: June 3, 2024
Copyright: © 2024 Chan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Wastewater data are publicly available through the Stanford Digital Repository (https://doi.org/10.25740/ch598gf0783).
Funding: This work was supported by the Provost’s Office of Stanford University to ABB with additional support from the Sergey Brin Family Foundation to ABB. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Nonpharmaceutical interventions (NPIs) aim to reduce the spread of an infectious disease in a community, especially when the community has little immunity to the pathogen or a vaccine is not yet available [1]. Examples of NPIs implemented in the United States at the start of the coronavirus disease 2019 (COVID-19) pandemic include face mask mandates, stay-at-home orders, non-essential business closures, and large gathering bans [2]. Although NPIs intended to benefit communities by flattening the epidemic curve—that is by reducing the peak number of cases and burden on the health care system—the implementation of NPIs also led to economic consequences and tolls on social well-being [3–5]. Governments and institutional leadership are tasked with balancing public health, social well-being, and economic prospects in the face of epidemics. Causal evidence can help policymakers and leaders make better-informed decisions in dire situations.
Following the initial wave of the pandemic, several studies empirically assessed the impact of NPIs on health-related outcomes. These studies suggested that NPIs reduced the spread of severe acute respiratory disease syndrome coronavirus 2 (SARS-CoV-2) virus, with school and workplace closures, business restrictions, large gathering bans, and mask mandates among the most impactful [6–10]. A review of the methodologies used by these studies found that around half analyzed raw outcome data and half analyzed computed outcome data (i.e., raw outcome data was used to compute another outcome) [11]. The most common raw outcomes analyzed were clinical surveillance reports (e.g., confirmed cases or deaths) and human mobility (e.g., tracking of mobile phones) [11]. The most common computed outcomes analyzed were COVID-19 growth rate and effective reproduction number [11].
Although clinical surveillance and mobile phone tracking are the most common sources of data used to evaluate NPIs, these data are not without biases and limitations. Counts of confirmed cases depend on clinical testing capacity and clinical testing rates, and deaths that occur outside of hospitals may be underreported [6–8,10,11]. Furthermore, clinical testing behaviors have drastically changed with the availability of self-administered antigen tests which are not reported to health departments [12]. Mobility data through tracking of mobile phones are unaffected by changes in clinical testing, but these data are not always publicly accessible, biased towards individuals who opt into location history sharing, and may not be a reliable proxy for SARS-CoV-2 transmission dynamics [13,14]. Wastewater monitoring, which gained heightened attention during the COVID-19 pandemic, is a promising data source because it does not suffer some of the limitations of clinical surveillance and mobility data for epidemiological inference.
Wastewater monitoring refers to the analysis of a sample of wastewater, which represents a pooled biological sample of the contributing population, for concentrations of infectious disease markers. Wastewater monitoring data capture contributions from both symptomatic and asymptomatic individuals and are not influenced by clinical testing availability or clinical test-seeking behaviors [15]. Studies reported that concentrations of SARS-CoV-2 RNA in wastewater solids are temporally correlated with laboratory-confirmed incident COVID-19 cases [16–19]. Several studies also demonstrated that wastewater monitoring can be used at geographic scales smaller than a sewershed (i.e., the population serviced by a wastewater treatment plant) to gain insight about COVID-19 incidence [20–32]. A potential use case for wastewater monitoring at subsewershed scales is to assess the impact of public health policies.
The World Health Organization (WHO) suggests sampling at finer spatial scales when using wastewater monitoring data to inform targeted control interventions [15]. Previous studies evaluating NPIs using clinical surveillance or mobility data were mostly conducted at national or subnational scales, and few of these studies investigated variation in the impact of NPIs on health-related outcomes among subpopulations [11]. NPIs may be more or less impactful in a subpopulation compared with the general population (e.g., due to different interaction patterns) or public health goals may differ among subpopulations (e.g., universities aim to maximize on-campus activity) [33]. Wastewater monitoring data may be well-suited to objectively assess NPIs, particularly among subpopulations and when clinical testing rates are low.
In this study, we evaluate the potential use case of wastewater monitoring data to empirically assess the impact of NPIs on the spread of COVID-19 among a university population. We begin by assessing the correlation between wastewater concentrations of SARS-CoV-2 RNA and reported COVID-19 incidence at Stanford University and evaluate changes to this correlation over time. Next, we evaluate the association between COVID-19 public health policies implemented at Stanford University and changes in wastewater concentrations of SARS-CoV-2 RNA using a difference-in-differences (DiD) approach. DiD is a quasi-experimental design commonly used in econometrics—although it was first used in 1854 by the English physician John Snow for epidemiologic purposes—that assesses the impact of an intervention on an outcome without the use of randomization [34–36]. DiD designs have been used by previous studies to empirically evaluate the causal effects of COVID-19 policies on clinical or mobility outcomes [37–44].
2. Methods
We used wastewater SARS-CoV-2 RNA monitoring data, COVID-19 case surveillance data, and dates associated with changes to campus COVID-19 public health policies between 29 July 2021 to 9 August 2023. During this timeframe, the residential communities for undergraduate and graduate students at Stanford University were open for all students to physically reside on campus. All calculations and statistical analyses were conducted in R (R Foundation for Statistical Computing version 4.1.3). This study was approved by the Stanford Institutional Review Board (IRB) for human subject research (IRB-59746). We did not obtain consent from individuals to preserve anonymity, and we did not have access to personally identifiable information during or after data collection.
2.1 Wastewater monitoring data
We used wastewater monitoring data from the Codiga Resource Recovery Center (CR2C) and the Palo Alto Regional Water Quality Control Plant (RWQCP) for this analysis. CR2C is a pilot scale wastewater treatment facility that services a portion of the Stanford University campus (California, USA). Buildings serviced include academic buildings and student and faculty housing; hospitals and clinics affiliated with the medical school are not serviced by CR2C (Fig A in S1 Text) [45,46]. CR2C services approximately 10,000 people with an estimated daily flow of approximately 0.5 million gallons of wastewater each day [20,46]. CR2C is a subsewershed of the sewershed serviced by RWQCP which is operated by the City of Palo Alto (California, USA). RWQCP services approximately 236,000 people and treats approximately 20 million gallons of wastewater each day for Los Altos, Los Altos Hills, Mountain View, Palo Alto, Stanford University, and the East Palo Alto Sanitary District (Fig A in S1 Text) [47].
Prospective, longitudinal wastewater sampling from CR2C and RWQCP began July 2021 and October 2020, respectively, and is currently ongoing. Briefly, wastewater settled solids are collected from both CR2C and RWQCP for laboratory processing. Settled solids samples at CR2C are generated from a 24-hour time proportional composite sample of the wastewater influent that is allowed to settle. Settled solids samples at RWQCP are “grab” samples from the primary clarifier; these samples are essentially composite samples because solids in the primary clarifier collect over 12–24 hours [48]. Six samples per week are collected from CR2C; seven samples per week are collected from RWQCP. Sampling from CR2C was temporarily reduced to two samples per week between 1 November 2022 and 31 December 2022. Details about sampling and processing methods used to measure the RNA targets, including quality assurance and quality control metrics, are registered in protocols.io [49–51] and have been described previously by Wolfe et al. [52] and Boehm et al. [53], so they are not repeated herein. Measurements and reporting in those other publications follow Environmental Microbiology Minimal Information (EMMI) guidelines. For this analysis, we used concentrations of the SARS-CoV-2 RNA N gene in wastewater settled solids in gene copies (gc) per gram (g) dry weight (gc/g), both unnormalized (N) and normalized by pepper mild mottle virus (PMMoV) RNA concentrations in wastewater settled solids in gc/g (N/PMMoV). The N gene target is located near the frequently used N2 assay target [54], and we have confirmed no mutation in the genomic target over the course of the pandemic [19]. PMMoV is a commonly used marker of wastewater fecal strength, and based on a mass balance model N/PMMoV should scale with disease incidence rate [18,55,56]. We used data between 29 July 2021 and 9 August 2023 (CR2C: 590 days; RWQCP: 736 days). The measured N gene concentration was below the limit of detection (approximately 1,000 gc/g) in 29 samples from CR2C. No samples from RWQCP were below the limit of detection. We imputed half the limit of detection (500 gc/g) for the N gene concentration for samples below the limit of detection. There were no non-detects for PMMoV in the dataset. Data from RWQCP between 16 November 2020 and 31 December 2022 have been published previously by Boehm et al. [53] and are publicly available through the Stanford Digital Repository (https://doi.org/10.25740/cx529np1130) [57]. Data from CR2C are novel and not published elsewhere. All wastewater monitoring data used in this study are publicly available through the Stanford Digital Repository (https://doi.org/10.25740/ch598gf0783) [58].
2.2 COVID-19 case surveillance data
Reported COVID-19 cases (hereafter “case data”) among students residing in the CR2C subsewershed are available from Stanford University. The date assigned to the positive test result is the date of specimen collection. We used case data between 29 July 2021 and 9 August 2023 for this analysis. The campus case data include positive test results from both student-reported self-administered antigen tests and laboratory-based PCR tests through the university’s surveillance testing program. The university’s surveillance testing program required vaccinated students to test once per week (twice per week for unvaccinated students) through 7 April 2022. Free, optional laboratory-based PCR testing continued to be available for students through 18 June 2023, so any cases thereafter were exclusively from student-reported self-administered tests. The CR2C subsewershed includes faculty and staff housing, but nonstudents residing in the CR2C subsewershed are not included in the university’s case data. Data provided by the state of California did not identify any COVID-19 cases in nonstudent housing areas during our entire analysis period.
2.3 Campus COVID-19 public health policies
Dates and details of changes to Stanford University’s COVID-19 public health policies were obtained from Stanford COVID-19 Health Alerts [59]. There were 15 unique dates on which campus COVID-19 public health policies changed during the study period (Table 1). We categorized policies into three groups: masking (i.e., those involving the use of face coverings), mobility (i.e., those involving movement or gathering of individuals), and testing (i.e., those relating to laboratory-based surveillance testing). We included testing policies because we hypothesize that surveillance testing requirements and availability affect the number of asymptomatic cases interacting with the general university population and, in turn, SARS-CoV-2 transmission on campus. We further differentiated policies between those that enforced rules (i.e., restrictions) and those that relaxed existing rules (i.e., relaxations). More information about each policy is included in Table A in the S1 Text.
2.4 Correlation analysis
Incident COVID-19 cases within the CR2C subsewershed were reported daily, whereas between 2–6 wastewater samples per week were collected and analyzed from CR2C during the analysis period. Clinical case surveillance data may further contain reporting biases on weekends. To compare the two time series, we calculated weekly average N concentrations, N/PMMoV concentrations, and incident COVID-19 cases for each epidemiological week (Sunday through Saturday). Neither raw nor log10-transformed weekly average N or N/PMMoV concentrations from CR2C were normally distributed (Shapiro-Wilk normality test, p < 0.01), so we used Kendall’s tau correlation to test the null hypothesis that weekly average wastewater SARS-CoV-2 RNA concentrations and weekly average incident COVID-19 cases in the CR2C subsewershed are not temporally correlated. We tested this null hypothesis using both unnormalized (N) and normalized (N/PMMoV) wastewater concentrations. We used the KendallTauB function from the DescTools R package to compute the 95% confidence interval for each tau estimate [60].
We further conducted three subgroup correlation analyses. First, we grouped the data by whether wastewater sample or clinical specimen collection occurred during the academic year (autumn, winter, or spring quarter) or nonacademic year (summer quarter). We used the date halfway between the last day of classes of the previous quarter and first day of classes of the following quarter to define the start and end of quarters [61]. Second, we grouped the data by whether wastewater sample or clinical specimen collection occurred before or after the requirement for laboratory-based surveillance testing was suspended for vaccinated and boosted students (7 April 2022) (Table 1). The laboratory-based surveillance testing program required fully vaccinated students to test once a week (twice a week for unvaccinated students) and therefore intended to capture both symptomatic and asymptomatic cases through routine testing. Third, we grouped the data by whether wastewater sample or clinical specimen collection occurred before or after 1 May 2022 [19]. This date represents a point in time when self-administered COVID-19 antigen tests, the results of which are not reportable to health departments, were widely available [12,19]. For each subgroup, we grouped weekly average wastewater concentrations and incident case counts based on the end date of the epidemiological week. In total, we conducted 14 correlation analyses using subsets of the same datasets to test the same null hypothesis, so we used an alpha value of 0.05 / 14 = 0.004 to account for multiple hypothesis testing when interpreting the p-value associated with each tau estimate.
2.5 Policy impact evaluation
We used PMMoV-normalized wastewater concentrations for the remainder of the analysis as the correlation between incident COVID-19 cases and wastewater SARS-CoV-2 RNA concentrations were similar using N and N/PMMoV, and a mass balance model suggests the N/PMMoV ratio should scale with incidence rate [56]. PMMoV is also a conceptually valid normalization approach because (1) PMMoV is an indigenous wastewater virus and therefore may better correct for differences in virus recovery than an exogenous recovery control that is seeded into the sample such as bovine coronavirus (BCoV) and (2) PMMoV is of dietary origin and therefore can control for differences in the fecal strength of the wastestream [55,56]. To assess the association between campus COVID-19 public health policies and changes in N/PMMoV measurements at CR2C, we used a difference-in-differences (DiD) approach. For the DiD design, we assumed policies went into effect at midnight on the date of implementation (day = 0). We defined the pre-treatment period as the 14 days before a policy was implemented (days -14 to -1) and the post-treatment period as the 14 days after a policy was implemented (days 0 to 13). We chose 14 days because 14 days is the maximum incubation period for SARS-CoV-2 and people who shed SARS-CoV-2 RNA typically do so at the start of infection [62–66]. We assumed the RWQCP sewershed represents a reasonable comparison group for the CR2C subsewershed. With the exception of the East Palo Alto Sanitary District, RWQCP services cities in Santa Clara County, which is the same county that Stanford University is located in. Santa Clara County entered the least restrictive “Yellow Tier” of California’s Blueprint for a Safer Economy on 19 May 2021, which lifted most local orders [67]. Moreover, California met the criteria under the Blueprint for a Safer Economy to fully reopen the economy on 15 June 2021 [68]. Regular sampling began at CR2C on 29 July 2021; therefore, we assumed policies implemented by Stanford University thereafter (Table 1) were only applicable to the CR2C subsewershed population and not the greater RWQCP sewershed population. The two exceptions were 3 August 2021 and 2 March 2022 because Santa Clara County also issued the same policies (Table 1) [69,70]. Non-policy events, such as emergence of novel SARS-CoV-2 variants, may also affect SARS-CoV-2 transmission; however, CR2C and RWQCP are in the same geographic area, so we assumed most non-policy events occurred around the same time and are therefore accounted for in the DiD design. Further justification for using RWQCP as a comparison group is included in the S1 Text.
We used a multivariable linear regression model to implement our DiD approach (Eq 1) [71]. A value of 0 for time represents the pre-treatment period (days -14 to -1), and a value of 1 represents the post-treatment period (days 0 to 13). A value of 0 for treated represents the untreated group (RWQCP), and a value of 1 represents the treated group (CR2C). The coefficient of the interaction between time and treated (β3) represents the DiD estimator, or the average treatment effect on the treated (ATT) [35,71]. In this study, a positive ATT value suggests a policy was associated with an increase in wastewater N/PMMoV concentrations; a negative ATT value suggests a policy was associated with a decrease in wastewater N/PMMoV concentrations. We recorded β3 (the ATT) and the p-value associated with β3 for each policy in Table 1 except for the two policies that Santa Clara County also issued (see above). R code for the DiD analysis is available through the Stanford Digital Repository (https://doi.org/10.25740/ch598gf0783) [58].
3. Results and discussion
3.1 Correlation between wastewater concentrations of SARS-CoV-2 RNA and incident COVID-19 cases
Between 29 July 2021 and 9 August 2023, wastewater N gene concentrations from CR2C ranged from not detected to 2.4 x 106 gc/g (mean: 1.3 x 105 gc/g, median: 4.4 x 104 gc/g) (Fig 1A). PMMoV-normalized wastewater concentrations ranged from not detected to 5.0 x 10−3 (mean: 2.4 10−4, median: 6.4 x 10−5) (Fig 1B). Reported daily incident COVID-19 cases within the CR2C subsewershed ranged from 0 cases to 420 cases (mean: 52 cases, median: 15 cases) (Fig 1C). Over the entire analysis period (the week ending on 31 July 2021 through the week ending on 12 August 2023), weekly average wastewater SARS-CoV-2 RNA concentrations were positively and significantly correlated with weekly average incident COVID-19 cases using unnormalized N gene concentrations but not significantly when using normalized N gene concentrations (Table 2). The subgroup analyses suggest the correlation between wastewater SARS-CoV-2 RNA concentrations and incident COVID-19 cases changed over time.
(A) N gene concentrations in gene copies per dry gram dry weight (gc/g), (B) N/PMMoV concentrations, and (C) incident COVID-19 cases over time. Gray circles represent measurements; error bars are one standard deviation. Gray triangles indicate measurements outside of the range shown on the plot. Black lines connect weekly average values. The shaded area corresponds to the nonacademic year. The dashed lines correspond to the date the surveillance testing requirement was suspended (7 April 2022) and the date of widespread availability of self-administered COVID-19 antigen tests in the region (1 May 2022).
Weekly average wastewater SARS-CoV-2 RNA concentrations were positively and significantly correlated with weekly average incident COVID-19 cases during the academic year using both unnormalized and normalized N gene concentrations; this correlation was not statistically significant during the nonacademic portion of the year (Table 2). The decrease in students on campus and increase in nonresidential visitors during the nonacademic portion of the year may explain the lack of a statistically significant correlation during the nonacademic year. The COVID-19 case data only include reported student cases residing within the CR2C subsewershed, but infected, nonresidential visitors may still contribute viral RNA to the wastewater that flows to CR2C.
Weekly average wastewater SARS-CoV-2 RNA concentrations were positively and significantly correlated with weekly average incident COVID-19 cases before the suspension of surveillance testing using both unnormalized and normalized N gene concentrations; this correlation was not statistically significant after the suspension of surveillance testing using normalized N gene concentrations only (Table 2). The required, laboratory-based surveillance testing program intended to capture both symptomatic and asymptomatic cases through routine testing. Thus, fewer asymptomatic cases may have been captured in the case data after surveillance testing was suspended which may explain the lack of a statistically significant correlation after this policy change.
Lastly, weekly average wastewater SARS-CoV-2 RNA concentrations were positively and significantly correlated with weekly average incident COVID-19 cases before the widespread availability of self-administered antigen tests using both unnormalized and normalized N gene concentrations; this correlation was not statistically significant after the widespread availability of self-administered antigen tests (Table 2). Positive, laboratory-based PCR tests are reportable under state-disease reporting laws [72]; however, self-reporting of self-administered antigen test results is voluntary. The widespread availability of self-administered antigen tests may have contributed to underreporting of cases which may explain the lack of a statistically significant correlation after the change in testing options.
It is not possible to deduce the main driver for the change in correlation between wastewater SARS-CoV-2 RNA concentrations and incident COVID-19 cases over time, but we suspect the change is due to several factors including changes in routine COVID-19 surveillance testing requirements, changes in test reporting, and overall decreases in PCR test-seeking behaviors as the pandemic continues [19,73–75]. Studies suggest that virus shedding patterns differ among SARS-CoV-2 variants [76–79], so changes in SARS-CoV-2 variants over time could be another reason for the change in correlation over time. We also did not consider lead-lag time effects between wastewater monitoring and case surveillance data as done in other studies [16,80], so future work could investigate how lead-lag time effects between wastewater monitoring and case surveillance data have changed over the course of the pandemic. Nonetheless, wastewater monitoring data are independent of test-seeking behaviors or test reporting patterns so may be a less biased tool for monitoring public health, particularly in periods characterized by low test-seeking and reporting rates.
3.2 Association between campus COVID-19 public health policies and changes in wastewater concentrations of SARS-CoV-2 RNA
Because the reliability of campus COVID-19 case data changed over the course of the study period at Stanford University, we used campus wastewater monitoring data from CR2C to evaluate the impact of COVID-19 public health policies at Stanford University using a DiD approach. Table 3 summarizes the average treatment effect on the treated (ATT) and associated p-value for each unique date associated with a change in campus COVID-19 public health policies as estimated using Eq 1. The two policies that were also implemented by the greater Santa Clara County were omitted from the analysis. Dates associated with a significant change (p ≤ 0.05) in wastewater N/PMMoV concentrations are shaded (red if ATT > 0 and blue if ATT < 0). A depiction of the DiD approach using the date when indoor events and gatherings were allowed to resume (28 January 2022) as an example is shown in Fig 2. In total, we analyzed 13 unique dates on which at least one change in campus COVID-19 public health policies went into effect. Most policy change dates (n = 8) were not associated with a significant change in wastewater N/PMMoV concentrations at CR2C. Five policy change dates were associated with a significant change in wastewater N/PMMoV concentrations (Table 3 and Fig B in S1 Text). These five dates included policies from all categories (masking, mobility, testing); three dates corresponded to policy relaxations, one corresponded to a policy restriction, and one corresponded to both a policy relaxation and restriction.
(A) Daily N/PMMoV concentrations at the Codiga Resource Recovery Center (CR2C) and Palo Alto Regional Water Quality Control Plant (RWQCP) over the 14 days before and after the policy change (denoted by the dotted line). Concentrations are displayed on a log10 scale. (B) Average log10(N/PMMoV) concentration at CR2C and RWQCP across the 14 days before and after the policy change. The counterfactual average log10(N/PMMoV) concentration at CR2C post-policy was estimated based on the time trend observed at RWQCP. The difference between the observed and counterfactual average log10(N/PMMoV) concentration at CR2C post-policy represents the average treatment effect on the treated (ATT). Here, a positive ATT value suggests that the policy change was associated with an increase in wastewater N/PMMoV concentrations at CR2C.
We did not expect policy relaxations to be associated with a significant change in wastewater N/PMMoV concentrations because these policy types are not intended to curb virus transmission. Eight dates exclusively corresponded to a policy relaxation, and five of them were not associated with a change in N/PMMoV concentrations. However, three of these dates were associated with a significant change in N/PMMoV concentrations. There was a significant increase in N/PMMoV concentrations associated with allowing indoor gatherings to resume (28 January 2022), which suggests that indoor gatherings are high-risk activities for SARS-CoV-2 transmission on campus. Indoor gatherings are known to promote SARS-CoV-2 transmission [81]. There was a significant decrease in N/PMMoV concentrations associated with suspending the surveillance testing requirement for students (7 April 2022), and then a significant increase in N/PMMoV concentrations associated with ending optional, free, laboratory-based PCR testing for employees (24 March 2023). These results are difficult to reconcile with expectations.
We expected policy restrictions to be associated with a significant decrease in wastewater N/PMMoV concentrations because these policy types are intended to curb virus transmission. Four dates exclusively corresponded to a policy restriction, but only one of these dates was associated with a significant change in N/PMMoV concentrations (recommending face coverings outdoors and prohibiting indoor parties on 2 September 2021). This date was associated with a significant increase rather than decrease in N/PMMoV concentrations, which could suggest these restrictive policies did not curb virus transmission on campus. Both a policy relaxation (revised travel guidelines) and policy restriction (surveillance testing required for all faculty, staff, and postdoctoral scholars) were implemented on the remaining date associated with a significant change in N/PMMoV concentrations (20 September 2021). This date was associated with a significant decrease in N/PMMoV concentrations; it is not possible to disentangle the individual causal effects of different policy types implemented on the same day.
Limitations of the DiD analysis may impact the interpretation of results and explain why some results did not align with expectations. First, policies may not be associated with immediate effects on outcomes [82]. The policy restrictions we considered may be associated with long-term effects on N/PMMoV concentrations despite being associated with null short-term effects. We determined that 14 days preceding and succeeding a policy was the most justified time interval for the DiD design given the maximum incubation period for SARS-CoV-2 is 14 days and people who shed SARS-CoV-2 RNA generally do so at the start of infection [62–66]. When using 14 days, the pre- or post-treatment period of one policy sometimes overlapped part of the pre- or post-treatment period of another policy for policies implemented close together which may lead to cumulative impacts on N/PMMoV concentrations that are not possible to disentangle. Second, there were sometimes campus announcements or national news headlines about COVID-19 preceding the implementation of policies, which could impact peoples’ behaviors leading up to the actual policy change date [82]. Peoples’ knowledge about the gravity of the COVID-19 pandemic has been shown to influence the effectiveness of lockdown policies [39]. We used official dates associated with changes to campus policies, but peoples’ behaviors may have started changing before these dates. Alternatively, peoples’ behaviors may have never changed if campus policies were ignored. Third, while the DiD design accounts for non-policy events that affect both CR2C and RWQCP, some non-policy events that affect the CR2C population and not the RWQCP population may have occurred. Co-occurrence of such events with policy changes is unaccounted for in the DiD design. For example, starts of quarters and university commencements may influence N/PMMoV concentrations at CR2C because these events result in large influxes of students and visitors to Stanford’s campus. We conducted the DiD analysis for dates associated with commencements and the first day of classes each quarter (Table B in the S1 Text), and two starts of quarters were associated with a significant change in N/PMMoV concentrations at CR2C (decrease at the start of autumn 2021 and increase at the start of spring 2022). Lastly, the proportion of people with immunity, either from prior infection or vaccination, changes over time. Potential impacts of public health policies on virus transmission may depend on the susceptible fraction of the population; however, the DiD design does not account for changing levels of susceptibility in either population. Importantly though, COVID-19 vaccination rates were similar and high among the Stanford University and greater Santa Clara County populations at the start of the analysis period [69,83].
Notwithstanding these limitations, we compared our results to those of other studies that also assessed COVID-19 public health policies among a vaccinated university population. Yang et al. similarly found that large gatherings are potentially high-risk events on campus [84]. Niu and Scarciotti concluded that mask wearing and social distancing measures were most effective at reducing new infections [33]. Motta et al. [85] and Paltiel and Schwartz [86] determined that routine surveillance testing was associated with a reduction in infections, even as vaccine effectiveness or coverage decreased. These other studies all used modeling approaches to assess COVID-19 public health policies; models are useful tools to evaluate public health measures although they often simplify real-world circumstances.
To our knowledge, there are no other published studies that empirically evaluate public health policies using wastewater monitoring data and a quasi-experimental approach, particularly among a vaccinated university population. The few other empirical studies using wastewater monitoring data for policy impact evaluation, which were conducted at large geographic scales and the beginning of the pandemic, used before-and-after descriptive approaches [87,88] or regression modeling and changepoint analysis [89]. The DiD design used herein aimed to account for co-occurring factors that may also affect the trajectory of N/PMMoV concentrations, such as changing SARS-CoV-2 variants, by using a nearby sewershed as a comparison group. We further considered both policy restrictions and policy relaxations during a period when COVID-19 vaccines were widely available. Previous studies that empirically assessed the impact of NPIs on health-related outcomes generally only focused on restrictions and were most commonly conducted at the start of the pandemic when economies were not fully opened and vaccines were not available. It is not only important to evaluate the implementation of policies but also whether policies are eventually relaxed appropriately, especially because early or rapid relaxation of NPIs may lessen the anticipated benefits of vaccine rollout efforts [90–94]. The quasi-experimental approach demonstrated herein could be useful in other epidemic situations triggering policy interventions, provided the pathogen is shed in human excretions that contribute to wastewater and there exists a reasonable comparison sewershed for the DiD design (e.g., a sewershed in a different state that did not roll out a given intervention).
Causal effects of COVID-19 public health policies are inherently challenging to study given the inability to conduct randomized controlled experiments and concurrence of policy and non-policy events [95]. Policymakers often need to make decisions despite having robust evidence. Quasi-experiments, which are growing in recognition in the health sciences, are a practical alternative to randomized controlled experiments that can still generate causal evidence [34]. In the DiD quasi-experimental design used herein, RWQCP represents a reasonable comparison group for CR2C because both sewersheds are in the same geographic area and policies implemented by Stanford University were only applicable to the CR2C population. We also implemented the DiD analysis using wastewater data from another, similar comparison sewershed because wastewater monitoring data from CR2C and RWQCP are not truly independent—although CR2C comprises only a very small proportion of RWQCP. Using wastewater data from the San José-Santa Clara Regional Wastewater Facility [96], which also services portions of Santa Clara County, as a comparison sewershed generated similar results as provided in Table 3 (Table B in the S1 Text). Similar findings using a different comparison group further strengthens the credibility of our DiD design and affirms the plausibility of the parallel trends assumption [97]. Still, uncertainties regarding wastewater monitoring data affect interpretation of data from any sewershed. Limited knowledge exists about SARS-CoV-2 RNA fecal shedding quantity and duration, especially differences in fecal shedding patterns among demographic groups and vaccination statuses [64]. Studies suggest that SARS-CoV-2 RNA shedding quantity and duration in human excretions that contribute to wastewater differs among SARS-CoV-2 variants, which may affect the interpretation of wastewater monitoring data over time [76–79]. Wastewater monitoring data can also exhibit high day-to-day variability; potential mechanisms for this variability remain yet to be systematically understood but could be due to heterogeneity of the wastestream [98]. Future studies using longitudinal wastewater monitoring data for causal inference may consider analyzing changes in a computed outcome variable, such as a wastewater-based estimation of the effective reproductive number [54] or wastewater-based measure of trend [99], rather than changes in raw wastewater concentrations. Ultimately, the performance of such computed outcomes still depends on understanding the raw wastewater concentration data that are used to generate computed outcomes. Continued work investigating sources of uncertainty and variability in wastewater monitoring data—and particularly the effect size of these sources—is necessary for better interpretation of these data for public health use cases [100,101].
4. Conclusions
We assessed the correlation between wastewater concentrations of SARS-CoV-2 RNA and incident, reported COVID-19 cases at a university and evaluated changes to this correlation over time. Consistent with other studies, we provide evidence that the correlation between wastewater SARS-CoV-2 RNA concentrations and incident COVID-19 cases has changed over time. We further investigated the use of longitudinal wastewater monitoring data for policy impact evaluation. Using a DiD approach, we observed that most campus COVID-19 public health policy changes were not associated with a significant change in wastewater SARS-CoV-2 RNA concentrations on campus. The quasi-experimental design presented herein demonstrates how longitudinal wastewater monitoring of viruses at a small geographic scale may be used for causal inference when randomized controlled experiments are not possible to conduct.
Acknowledgments
We acknowledge feedback on study design, implementation, and interpretation from James Jacobs, Julie Parsonnet, Russell Furr, Rich Wittman, Robyn Tepper, Jorge Salinas, Bonnie Maldonado, Christina Kong, and Stephanie Kalfayan. We acknowledge Palo Alto and San Jose wastewater treatment plant staff and the CR2C Student Operators team for wastewater sample collection.
References
- 1.
U.S. Centers for Disease Control and Prevention. Nonpharmaceutical Interventions (NPIs). 4 Oct 2022 [cited 18 Jul 2023]. Available: https://www.cdc.gov/nonpharmaceutical-interventions/index.html.
- 2.
Kaiser Family Foundation. State Actions to Mitigate the Spread of COVID-19. In: State Health Facts [Internet]. 4 Aug 2021 [cited 18 Jul 2023]. Available: https://www.kff.org/other/state-indicator/state-actions-to-mitigate-the-spread-of-covid-19/.
- 3. Qualls N, Levitt A, Kanade N, Wright-Jegede N, Wright-Jegede S, Biggerstaff M, et al. Community Mitigation Guidelines to Prevent Pandemic Influenza—United States, 2017. MMWR Recomm Rep. 2017;66: 1–34. pmid:28426646
- 4. Deb P, Furceri D, Ostry JD, Tawk N. The Economic Effects of COVID-19 Containment Measures. Open Econ Rev. 2022;33: 1–32.
- 5. Hosseinzadeh P, Zareipour M, Baljani E, Moradali MR. Social Consequences of the COVID-19 Pandemic. A Systematic Review. Investig Educ En Enfermeria. 2022;40: e10. pmid:35485623
- 6. Iezadi S, Gholipour K, Azami-Aghdash S, Ghiasi A, Rezapour A, Pourasghari H, et al. Effectiveness of non-pharmaceutical public health interventions against COVID-19: A systematic review and meta-analysis. PLoS ONE. 2021;16: e0260371. pmid:34813628
- 7. Mendez-Brito A, Bcheraoui CE, Pozo-Martin F. Systematic review of empirical studies comparing the effectiveness of non-pharmaceutical interventions against COVID-19. J Infect. 2021;83: 281–293. pmid:34161818
- 8.
Poeschl J, Larsen RB. How do non-pharmaceutical interventions affect the spread of COVID-19? A literature review. Danmarks Nationalbank; 2021 Apr. Report No.: No. 4. Available: https://www.nationalbanken.dk/en/news-and-knowledge/publications-and-speeches/archive-publications/2021/how-do-non-pharmaceutical-interventions-affect-the-spread-of-covid-19-a-literature-review.
- 9. Rizvi RF, Craig KJT, Hekmat R, Reyes F, South B, Rosario B, et al. Effectiveness of non-pharmaceutical interventions related to social distancing on respiratory viral infectious disease outcomes: A rapid evidence-based review and meta-analysis. SAGE Open Med. 2021;9: 20503121211022972. pmid:34164126
- 10. Talic S, Shah S, Wild H, Gasevic D, Maharaj A, Ademi Z, et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. The BMJ. 2021;375: e068302. pmid:34789505
- 11. Banholzer N, Lison A, Özcelik D, Stadler T, Feuerriegel S, Vach W. The methodologies to assess the effectiveness of non-pharmaceutical interventions during COVID-19: a systematic review. Eur J Epidemiol. 2022;37: 1003–1024. pmid:36152133
- 12. Rubin R. COVID-19 Testing Moves Out of the Clinic and Into the Home. JAMA. 2021;326: 1362–1364. pmid:34550303
- 13. Aktay A, Bavadekar S, Cossoul G, Davis J, Desfontaines D, Fabrikant A, et al. Google COVID-19 Community Mobility Reports: Anonymization Process Description (version 1.1). arXiv; 2020.
- 14. Kishore N, Taylor AR, Jacob PE, Vembar N, Cohen T, Buckee CO, et al. Evaluating the reliability of mobility metrics from aggregated mobile phone data as proxies for SARS-CoV-2 transmission in the USA: a population-based study. Lancet Digit Health. 2022;4: e27–e36. pmid:34740555
- 15.
World Health Organization. Environmental surveillance for SARS-COV-2 to complement public health surveillance: interim guidance. World Health Organization; 2022 Apr. Report No.: WHO/HEP/ECH/WSH/2022.1. Available: https://apps.who.int/iris/bitstream/handle/10665/353158/WHO-HEP-ECH-WSH-2022.1-eng.pdf?sequence=1.
- 16. Peccia J, Zulli A, Brackney DE, Grubaugh ND, Kaplan EH, Casanovas-Massana A, et al. Measurement of SARS-CoV-2 RNA in wastewater tracks community infection dynamics. Nat Biotechnol. 2020;38: 1164–1167. pmid:32948856
- 17. Graham KE, Loeb SK, Wolfe MK, Catoe D, Sinnott-Armstrong N, Kim S, et al. SARS-CoV-2 RNA in Wastewater Settled Solids Is Associated with COVID-19 Cases in a Large Urban Sewershed. Environ Sci Technol. 2021;55: 488–498. pmid:33283515
- 18. D’Aoust PM, Mercier E, Montpetit D, Jia J-J, Alexandrov I, Neault N, et al. Quantitative analysis of SARS-CoV-2 RNA from wastewater solids in communities with low COVID-19 incidence and prevalence. Water Res. 2021;188: 116560. pmid:33137526
- 19. Boehm AB, Wolfe MK, White B, Hughes B, Duong D. Divergence of wastewater SARS-CoV-2 and reported laboratory-confirmed COVID-19 incident case data coincident with wide-spread availability of at-home COVID-19 antigen tests. PeerJ. 2023;11: e15631. pmid:37397016
- 20. Zambrana W, Catoe D, Coffman MM, Kim S, Anand A, Solis D, et al. SARS-CoV-2 RNA and N Antigen Quantification via Wastewater at the Campus Level, Building Cluster Level, and Individual-Building Level. ACS EST Water. 2022; acsestwater.2c00050. pmid:37552722
- 21. Davó L, Seguí R, Botija P, Beltrán MJ, Albert E, Torres I, et al. Early detection of SARS-CoV-2 infection cases or outbreaks at nursing homes by targeted wastewater tracking. Clin Microbiol Infect. 2021;27: 1061–1063. pmid:33601008
- 22. Karthikeyan S, Nguyen A, McDonald D, Zong Y, Ronquillo N, Ren J, et al. Rapid, Large-Scale Wastewater Surveillance and Automated Reporting System Enable Early Detection of Nearly 85% of COVID-19 Cases on a University Campus. mSystems. 2021;6: 10.1128/msystems.00793-21. pmid:34374562
- 23. Acosta N, Bautista MA, Waddell BJ, McCalder J, Beaudet AB, Man L, et al. Longitudinal SARS-CoV-2 RNA wastewater monitoring across a range of scales correlates with total and regional COVID-19 burden in a well-defined urban population. Water Res. 2022;220: 118611. pmid:35661506
- 24. Greenwald HD, Kennedy LC, Hinkle A, Whitney ON, Fan VB, Crits-Christoph A, et al. Tools for interpretation of wastewater SARS-CoV-2 temporal and spatial trends demonstrated with data collected in the San Francisco Bay Area. Water Res X. 2021;12: 100111. pmid:34373850
- 25. Crowe J, Schnaubelt AT, SchmidtBonne S, Angell K, Bai J, Eske T, et al. Assessment of a Program for SARS-CoV-2 Screening and Environmental Monitoring in an Urban Public School District. JAMA Netw Open. 2021;4: e2126447. pmid:34550382
- 26. Betancourt WQ, Schmitz BW, Innes GK, Prasek SM, Pogreba Brown KM, Stark ER, et al. COVID-19 containment on a college campus via wastewater-based epidemiology, targeted clinical testing and an intervention. Sci Total Environ. 2021;779: 146408. pmid:33743467
- 27. Gibas C, Lambirth K, Mittal N, Juel MAI, Barua VB, Roppolo Brazell L, et al. Implementing building-level SARS-CoV-2 wastewater surveillance on a university campus. Sci Total Environ. 2021;782: 146749. pmid:33838367
- 28. Scott LC, Aubee A, Babahaji L, Vigil K, Tims S, Aw TG. Targeted wastewater surveillance of SARS-CoV-2 on a university campus for COVID-19 outbreak detection and mitigation. Environ Res. 2021;200: 111374. pmid:34058182
- 29. Colosi LM, Barry KE, Kotay SM, Porter MD, Poulter MD, Ratliff C, et al. Development of Wastewater Pooled Surveillance of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) from Congregate Living Settings. Appl Environ Microbiol. 2021;87: e00433–21. pmid:33858836
- 30. Wong JCC, Tan J, Lim YX, Arivalan S, Hapuarachchi HC, Mailepessov D, et al. Non-intrusive wastewater surveillance for monitoring of a residential building for COVID-19 cases. Sci Total Environ. 2021;786: 147419. pmid:33964781
- 31. Barrios RE, Lim C, Kelley MS, Li X. SARS-CoV-2 concentrations in a wastewater collection system indicated potential COVID-19 hotspots at the zip code level. Sci Total Environ. 2021;800: 149480. pmid:34392211
- 32. Rubio-Acero R, Beyerl J, Muenchhoff M, Roth MS, Castelletti N, Paunovic I, et al. Spatially resolved qualified sewage spot sampling to track SARS-CoV-2 dynamics in Munich ‐ One year of experience. Sci Total Environ. 2021;797: 149031. pmid:34346361
- 33. Niu Z, Scarciotti G. Ranking the effectiveness of non-pharmaceutical interventions to counter COVID-19 in UK universities with vaccinated population. Sci Rep. 2022;12: 13039. pmid:35906452
- 34. Bärnighausen T, Tugwell P, Røttingen J-A, Shemilt I, Rockers P, Geldsetzer P, et al. Quasi-experimental study designs series—paper 4: uses and value. J Clin Epidemiol. 2017;89: 21–29. pmid:28365303
- 35. Lechner M. The Estimation of Causal Effects by Difference-in-Difference Methods. Found Trends® Econom. 2011;4: 165–224.
- 36.
Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CMJ. Impact Evaluation in Practice, Second Edition. Washington, DC: Inter-American Development Bank and World Bank; 2016. https://doi.org/10.1596/978-1-4648-0779-4
- 37. Fang H, Wang L, Yang. Human Mobility Restrictions and the Spread of the Novel Coronavirus (2019-nCoV) in China. National Bureau of Economic Research; 2020.
- 38. Friedson AI, McNichols D, Sabia JJ, Dave D. Did California’s Shelter-in-Place Order Work? Early Coronavirus-Related Public Health Effects. National Bureau of Economic Research; 2020.
- 39. Loiacono L, Puglisi R, Rizzo L, Secomandi R. Pandemic knowledge and regulation effectiveness: Evidence from COVID-19. J Comp Econ. 2022;50: 768–783. pmid:35221397
- 40. Berry CR, Fowler A, Glazer T, Handel-Meyer S, MacMillen A. Evaluating the effects of shelter-in-place policies during the COVID-19 pandemic. Proc Natl Acad Sci. 2021;118: e2019706118. pmid:33766888
- 41. Courtemanche C, Garuccio J, Le A, Pinkston J, Yelowitz A. Strong Social Distancing Measures In The United States Reduced The COVID-19 Growth Rate. Health Aff (Millwood). 2020;39: 1237–1246. pmid:32407171
- 42. Dave D, Friedson A, Matsuzawa K, Sabia JJ, Safford S. JUE Insight: Were urban cowboys enough to control COVID-19? Local shelter-in-place orders and coronavirus case growth. J Urban Econ. 2022;127: 103294. pmid:33191960
- 43. Dave DM, Friedson AI, Matsuzawa K, McNichols D, Sabia JJ. Are the Effects of Adoption and Termination of Shelter-in-Place Orders Symmetric? Evidence from a Natural Experiment. National Bureau of Economic Research; 2020.
- 44. Juranek S, Zoutman FT. The effect of non-pharmaceutical interventions on the demand for health care and on mortality: evidence from COVID-19 in Scandinavia. J Popul Econ. 2021;34: 1299–1320. pmid:34334957
- 45.
Stanford University. Codiga Resource Recovery Center at Stanford. In: Stanford Engineering [Internet]. [cited 24 Feb 2023]. Available: https://cr2c.stanford.edu/.
- 46.
Stanford University. Wastewater Analysis. In: Stanford COVID-19 Health Alerts [Internet]. [cited 17 Mar 2023]. Available: https://healthalerts.stanford.edu/covid-19/wastewater-dashboard/.
- 47.
City of Palo Alto. Regional Water Quality Control Plant. In: cleanbay.org [Internet]. [cited 24 Feb 2023]. Available: https://cleanbay.org/our-programs/regional-water-quality-control-plant/.
- 48.
Metcalf & Eddy Inc., Tchobanoglous G, Burton FL, Tsuchihashi R, Stensel HD. Processing and Treatment of Sludges. 5th ed. Wastewater Engineering: Treatment and Resource Recovery. 5th ed. New York, NY, USA: McGraw-Hill Professional; 2013. p. 1487.
- 49. Topol A, Wolfe M, White B, Wigginton K, Boehm AB. High Throughput pre-analytical processing of wastewater settled solids for SARS-CoV-2 RNA analyses V.2. In: protocols.io [Internet]. 5 Dec 2021. Available: https://doi.org/10.17504/protocols.io.b2kmqcu6.
- 50. Topol A, Wolfe M, White B, Wigginton K, Boehm AB. High Throughput SARS-COV-2, PMMOV, and BCoV quantification in settled solids using digital RT-PCR V.5. In: protocols.io [Internet]. 2 May 2022. Available:
- 51. Topol A, Wolfe M, Wigginton K, White B, Boehm AB. High Throughput RNA Extraction and PCR Inhibitor Removal of Settled Solids for Wastewater Surveillance of SARS-CoV-2 RNA V.2. In: protocols.io [Internet]. 5 Dec 2021. Available: https://doi.org/10.17504/protocols.io.b2mkqc4w.
- 52. Wolfe MK, Topol A, Knudson A, Simpson A, White B, Vugia DJ, et al. High-Frequency, High-Throughput Quantification of SARS-CoV-2 RNA in Wastewater Settled Solids at Eight Publicly Owned Treatment Works in Northern California Shows Strong Association with COVID-19 Incidence. mSystems. 2021;6: e00829–21. pmid:34519528
- 53. Boehm AB, Wolfe MK, Wigginton KR, Bidwell A, White BJ, Hughes B, et al. Human viral nucleic acids concentrations in wastewater solids from Central and Coastal California USA. Sci Data. 2023;10: 396. pmid:37349355
- 54. Huisman JS, Scire J, Caduff L, Fernandez -Cassi Xavier, Ganesanandamoorthy P, Kull A, et al. Wastewater-Based Estimation of the Effective Reproductive Number of SARS-CoV-2. Environ Health Perspect. 130: 057011. pmid:35617001
- 55. Kitajima M, Sassi HP, Torrey JR. Pepper mild mottle virus as a water quality indicator. Npj Clean Water. 2018;1: 1–9.
- 56. Wolfe MK, Archana A, Catoe D, Coffman MM, Dorevich S, Graham KE, et al. Scaling of SARS-CoV-2 RNA in Settled Solids from Multiple Wastewater Treatment Plants to Compare Incidence Rates of Laboratory-Confirmed COVID-19 in Their Sewersheds. Environ Sci Technol Lett. 2021;8: 398–404. pmid:37566351
- 57. Boehm A, Wolfe M, Wigginton K, Bidwell A, White B, Duong D, et al. Human viral nucleic acids concentrations in wastewater solids from Central and Coastal California, USA. In: Stanford Digital Repository [Internet]. 14 Apr 2023 [cited 12 Sep 2023]. Available: https://purl.stanford.edu/cx529np1130. pmid:37349355
- 58. Chan E, Bidwell A, Li Z, Tilmans S, Boehm A. Data and R code for policy impact evaluation using wastewater SARS-CoV-2 RNA concentrations paper. In: Stanford Digital Repository [Internet]. 2023. Available: https://purl.stanford.edu/ch598gf0783.
- 59.
Stanford University. COVID-19 resources for the Stanford community. In: Stanford COVID-19 Health Alerts [Internet]. [cited 24 Feb 2023]. Available: https://healthalerts.stanford.edu/covid-19.
- 60. Signorell A, Aho K, Alfons A, Anderegg N, Aragon T, Arachchige C, et al. DescTools: Tools for Descriptive Statistics. 2023. Available: https://cran.r-project.org/web/packages/DescTools/index.html.
- 61.
Stanford University. Academic Dates. In: Stanford Student Services [Internet]. [cited 24 Feb 2023]. Available: https://studentservices.stanford.edu/calendar/academic-dates.
- 62. Lauer SA, Grantz KH, Bi Q, Jones FK, Zheng Q, Meredith HR, et al. The Incubation Period of Coronavirus Disease 2019 (COVID-19) From Publicly Reported Confirmed Cases: Estimation and Application. Ann Intern Med. 2020;172: 577–582. pmid:32150748
- 63.
U.S. Centers for Disease Control and Prevention. Symptoms of COVID-19. 26 Oct 2022 [cited 22 Feb 2023]. Available: https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html.
- 64. Arts PJ, Kelly JD, Midgley CM, Anglin K, Lu S, Abedi GR, et al. Longitudinal and quantitative fecal shedding dynamics of SARS-CoV-2, pepper mild mottle virus, and crAssphage. mSphere. 2023;0: e00132–23. pmid:37338211
- 65. Wölfel R, Corman VM, Guggemos W, Seilmaier M, Zange S, Müller MA, et al. Virological assessment of hospitalized patients with COVID-2019. Nature. 2020;581: 465–469. pmid:32235945
- 66. Natarajan A, Zlitni S, Brooks EF, Vance SE, Dahlen A, Hedlin H, et al. Gastrointestinal symptoms and fecal shedding of SARS-CoV-2 RNA suggest prolonged gastrointestinal infection. Med N Y N. 2022;3: 371–387.e9. pmid:35434682
- 67.
County of Santa Clara. Success in Vaccinating Most County Residents and Low COVID-19 Case Rates Allow Health Officer to Adopt New, More Focused Local Order. In: Santa Clara County Public Health [Internet]. 18 May 2021 [cited 3 Aug 2023]. Available: https://covid19.sccgov.org/news-releases/pr-05-18-2021-scc-adopts-more-focused-local-health-order.
- 68.
California Department of Public Health. Blueprint for a Safer Economy. In: California Department of Public Health [Internet]. 15 Jun 2021 [cited 3 Aug 2023]. Available: https://www.cdph.ca.gov/Programs/CID/DCDC/Pages/COVID-19/COVID19CountyMonitoringOverview.aspx.
- 69.
County of Santa Clara. Bay Area Health Officials Urge Immediate Vaccination and Issue Orders Requiring Use of Face Coverings Indoors to Prevent the Spread of COVID-19 ‐ Emergency Operations Center ‐ County of Santa Clara. In: Santa Clara County Public Health [Internet]. 2 Aug 2021 [cited 3 Aug 2023]. Available: https://covid19.sccgov.org/news-releases/pr-08-02-2021-health-officials-urge-immediate-vaccination-and-require-face-covering-indoors.
- 70.
County of Santa Clara. County of Santa Clara Universal Indoor Mask Requirement to Transition to a Recommendation March 2. In: Santa Clara County Public Health [Internet]. 1 Mar 2022 [cited 3 Aug 2023]. Available: https://covid19.sccgov.org/news-releases/pr-03-01-2022-indoor-mask-requirement-transitions-to-recommendation.
- 71. Craig P, Katikireddi SV, Leyland A, Popham F. Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research. Annu Rev Public Health. 2017;38: 39–56. pmid:28125392
- 72.
U.S. Centers for Disease Control and Prevention. What is Case Surveillance? 27 Mar 2023 [cited 31 Aug 2023]. Available: https://www.cdc.gov/nndss/about/index.html.
- 73. Kadonsky KF, Naughton CC, Susa M, Olson R, Singh GL, Daza-Torres ML, et al. Expansion of wastewater-based disease surveillance to improve health equity in California’s Central Valley: sequential shifts in case-to-wastewater and hospitalization-to-wastewater ratios. Front Public Health. 2023;11. pmid:37457240
- 74. Varkila MRJ, Montez-Rath ME, Salomon JA, Yu X, Block GA, Owens DK, et al. Use of Wastewater Metrics to Track COVID-19 in the US. JAMA Netw Open. 2023;6: e2325591. pmid:37494040
- 75. Rader B. Use of At-Home COVID-19 Tests—United States, August 23, 2021–March 12, 2022. MMWR Morb Mortal Wkly Rep. 2022;71. pmid:35358168
- 76. Anjos D, Fiaccadori FS, Servian C do P, da Fonseca SG, Guilarde AO, Borges MASB, et al. SARS-CoV-2 loads in urine, sera and stool specimens in association with clinical features of COVID-19 patients. J Clin Virol Plus. 2022;2: 100059. pmid:35262032
- 77. Daou M, Kannout H, Khalili M, Almarei M, Alhashami M, Alhalwachi Z, et al. Analysis of SARS-CoV-2 viral loads in stool samples and nasopharyngeal swabs from COVID-19 patients in the United Arab Emirates. PLOS ONE. 2022;17: e0274961. pmid:36137134
- 78. Puhach O, Meyer B, Eckerle I. SARS-CoV-2 viral load and shedding kinetics. Nat Rev Microbiol. 2023;21: 147–161. pmid:36460930
- 79. Wannigama DL, Amarasiri M, Phattharapornjaroen P, Hurst C, Modchang C, Chadsuthi S, et al. Increased faecal shedding in SARS-CoV-2 variants BA.2.86 and JN.1. Lancet Infect Dis. 2024;0. pmid:38522445
- 80. Keshaviah A, Huff I, Hu XC, Guidry V, Christensen A, Berkowitz S, et al. Separating signal from noise in wastewater data: An algorithm to identify community-level COVID-19 surges in real time. Proc Natl Acad Sci. 2023;120: e2216021120. pmid:37490532
- 81. Qian H, Miao T, Liu L, Zheng X, Luo D, Li Y. Indoor transmission of SARS-CoV-2. Indoor Air. 2021;31: 639–645. pmid:33131151
- 82. Goodman-Bacon A, Marcus J. Using Difference-in-Differences to Identify Causal Effects of COVID-19 Policies. Surv Res Methods. 2020;14: 153–158.
- 83.
Stanford University. Update for Stanford community on unvaccinated employee testing and face coverings. In: Stanford COVID-19 Health Alerts [Internet]. 28 Jul 2021 [cited 22 Aug 2023]. Available: https://healthalerts.stanford.edu/covid-19/2021/07/28/update-for-stanford-community-on-unvaccinated-employee-testing-and-face-coverings/.
- 84. Yang Q, Gruenbacher DM, Scoglio CM. Estimating data-driven coronavirus disease 2019 mitigation strategies for safe university reopening. J R Soc Interface. 2022;19: 20210920. pmid:35285285
- 85. Motta FC, McGoff KA, Deckard A, Wolfe CR, Bonsignori M, Moody MA, et al. Assessment of Simulated Surveillance Testing and Quarantine in a SARS-CoV-2–Vaccinated Population of Students on a University Campus. JAMA Health Forum. 2021;2: e213035. pmid:35977169
- 86. Paltiel AD, Schwartz JL. Assessing COVID-19 Prevention Strategies to Permit the Safe Opening of Residential Colleges in Fall 2021. Ann Intern Med. 2021;174: 1563–1571. pmid:34461034
- 87. Hillary LS, Farkas K, Maher KH, Lucaci A, Thorpe J, Distaso MA, et al. Monitoring SARS-CoV-2 in municipal wastewater to evaluate the success of lockdown measures for controlling COVID-19 in the UK. Water Res. 2021;200: 117214. pmid:34058486
- 88. Wurtz N, Lacoste A, Jardot P, Delache A, Fontaine X, Verlande M, et al. Viral RNA in City Wastewater as a Key Indicator of COVID-19 Recrudescence and Containment Measures Effectiveness. Front Microbiol. 2021;12. Available: https://www.frontiersin.org/articles/10.3389/fmicb.2021.664477. pmid:34079532
- 89. Stephens N, Béen F, Savic D. An Analysis of SARS-CoV-2 in Wastewater to Evaluate the Effectiveness of Nonpharmaceutical Interventions against COVID-19 in The Netherlands. ACS EST Water. 2022;2: 2158–2166. pmid:37552733
- 90. Althobaity Y, Wu J, Tildesley MJ. Non-pharmaceutical interventions and their relevance in the COVID-19 vaccine rollout in Saudi Arabia and Arab Gulf countries. Infect Dis Model. 2022;7: 545–560. pmid:36035780
- 91. Li H, Wang L, Zhang M, Lu Y, Wang W. Effects of vaccination and non-pharmaceutical interventions and their lag times on the COVID-19 pandemic: Comparison of eight countries. PLoS Negl Trop Dis. 2022;16: e0010101. pmid:35025865
- 92. Moore S, Hill EM, Tildesley MJ, Dyson L, Keeling MJ. Vaccination and non-pharmaceutical interventions for COVID-19: a mathematical modelling study. Lancet Infect Dis. 2021;21: 793–802. pmid:33743847
- 93. Patel MD, Rosenstrom E, Ivy JS, Mayorga ME, Keskinocak P, Boyce RM, et al. Association of Simulated COVID-19 Vaccination and Nonpharmaceutical Interventions With Infections, Hospitalizations, and Mortality. JAMA Netw Open. 2021;4: e2110782. pmid:34061203
- 94. Gozzi N, Bajardi P, Perra N. The importance of non-pharmaceutical interventions during the COVID-19 vaccine rollout. PLOS Comput Biol. 2021;17: e1009346. pmid:34506478
- 95. Haber NA, Clarke-Deelder E, Feller A, Smith ER, Salomon JA, MacCormack-Gelles B, et al. Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength. BMJ Open. 2022;12: e053820. pmid:35017250
- 96.
San José-Santa Clara Regional Wastewater Facility. In: City of San Jose [Internet]. [cited 16 Jun 2022]. Available: https://www.sanjoseca.gov/your-government/environment/water-utilities/regional-wastewater-facility.
- 97. Bärnighausen T, Oldenburg C, Tugwell P, Bommer C, Ebert C, Barreto M, et al. Quasi-experimental study designs series—paper 7: assessing the assumptions. J Clin Epidemiol. 2017;89: 53–66. pmid:28365306
- 98. Bivins A, North D, Wu Z, Shaffer M, Ahmed W, Bibby K. Within- and between-Day Variability of SARS-CoV-2 RNA in Municipal Wastewater during Periods of Varying COVID-19 Prevalence and Positivity. ACS EST Water. 2021;1: 2097–2108.
- 99. Chan EMG, Kennedy LC, Wolfe MK, Boehm AB. Identifying trends in SARS-CoV-2 RNA in wastewater to infer changing COVID-19 incidence: Effect of sampling frequency. PLOS Water. 2023;2: e0000088.
- 100. Wade MJ, Lo Jacomo A, Armenise E, Brown MR, Bunce JT, Cameron GJ, et al. Understanding and managing uncertainty and variability for wastewater monitoring beyond the pandemic: Lessons learned from the United Kingdom national COVID-19 surveillance programmes. J Hazard Mater. 2022;424: 127456. pmid:34655869
- 101. Li X, Zhang S, Shi J, Luby SP, Jiang G. Uncertainties in estimating SARS-CoV-2 prevalence by wastewater-based epidemiology. Chem Eng J. 2021;415: 129039. pmid:33642938