Advertisement

  • Loading metrics

Open Access

Peer-reviewed

Research Article

From bench to beach: Assessing the reliability of community-based qPCR monitoring for recreational water quality

Abstract

Quantitative polymerase chain reaction (qPCR) is increasingly used in recreational water quality monitoring, yet the temporal variability of indicator concentrations as well as the breadth of locations and biological hazards to be monitored continues to present a challenge. Participatory approaches such as community-based monitoring (CBM) are valued in environmental research but the potential for the integration of DNA-based methods has yet to be realized. This study assessed the reliability of a decentralized, community-based qPCR monitoring program for fecal indicator bacteria, Enterococcus spp., in recreational waters. Non-expert community partners were responsible for DNA extraction and qPCR analysis of samples at a satellite laboratory; training, protocols, and materials were provided and standardized by our research team. Comparison of community partner results to those from duplicate samples analyzed by our research team following U.S. EPA Method 1611 revealed a high level of reliability, with 72.8% of community partner results indicating the same beach management decision as Method 1611. Median coefficient of variation between community partner and Method 1611 results ranged from 7.07% to 10.29%. In this study, we demonstrate the ability of non-expert community partners to independently carry out protocols and to generate reliable qPCR monitoring data for water quality indicators and the strong relationship between the results of this community-based approach and gold standard methods. As the employment of DNA-based testing expands, incorporation of these techniques into a CBM framework presents a means to advance and expand traditional monitoring and research approaches by increasing capacity, addressing gaps, fostering greater inclusivity and community engagement in monitoring and management, and improving the accessibility of environmental research.

Citation: Welch CP, Rudko SP, Peter B, Klimchuk S, Gill K, Lu RX, et al. (2025) From bench to beach: Assessing the reliability of community-based qPCR monitoring for recreational water quality. PLOS Water 4(5): e0000309. https://doi.org/10.1371/journal.pwat.0000309

Editor: Guillaume Wright, PLOS: Public Library of Science, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND

Received: October 4, 2024; Accepted: April 24, 2025; Published: May 27, 2025

Copyright: © 2025 Welch et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and related supporting information files.

Funding: This work was supported by funding from the Natural Sciences and Engineering Council of Canada (NSERC) #2018-05209 and #2018-522661 (PCH), by Alberta Innovates #2615 (PCH). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interest exist.

Introduction

Recreational water quality monitoring is vital for protecting public health and for responsible environmental management. Fecal contamination of recreational waters presents a serious threat to human health due to the number and diversity of pathogenic microbes which may be transmitted [1,2]. Levels of fecal indicator bacteria (FIB) are a key water quality metric for assessing fecal-associated health risks. FIB, while typically not harmful themselves, indicate the presence of fecal contamination and, thus, the potential presence of fecal-associated pathogens. Currently recommended fecal indicators in freshwater monitoring are Escherichia coli and Enterococcus spp. [3,4]. Monitoring FIB levels provides essential insight regarding the level of health risk to water recreators and alerts those responsible for beach management to potential fecal pollution.

Traditional culture-based methods for FIB monitoring require at least 24 hours after sampling to obtain results, with follow-up testing requiring more time, precluding the ability to assess current risk level [2]. As a result, risk assessments and beach management decisions are made based on the previous day’s water quality. FIB density can change drastically from one day to the next, such that yesterday’s data may not show any correlation with today’s indicator density [5–8]. Thus, delays between sample collection and availability of results incurred from culture testing increase the potential for unnecessary beach closures and, more concerningly, impede a timely response to unsatisfactory water quality, resulting in a potential lack of warning to beach-goers regarding current health risk [5,6,8,9].

The ability to generate same-day results makes qPCR an attractive, rapid alternative to traditional culture-based testing. The National Epidemiological and Environmental Assessment of Recreational (NEEAR) Waters Study investigated the use of qPCR in water quality monitoring in an effort to explore methodologies that would enable a decrease in the time required between sampling and results [1,2,10]. NEEAR epidemiologic studies assessed the predictive relationship between density of Enterococcus spp. determined by qPCR and swimming-associated health risks. They revealed that qPCR-based measurements were better predictors of the risk of gastrointestinal illness compared to those obtained by culture methods [2]. Following the results of the NEEAR Study, the U.S. Environmental Protection Agency (U.S. EPA) published Method 1611: Enterococci in Water by TaqMan Quantitative Polymerase Chain Reaction (qPCR) Assay and, later, Method 1609 [11,12]. Both Method 1611 and 1609 enable the rapid detection and quantification of enterococci by qPCR in fresh and marine recreational waters. Water quality criteria values for Enterococcus densities determined by Method 1611 were developed which reflected an equivalent health risk to those set for culture methods. The 2012 U.S. EPA [4] Recreational Water Quality Criteria included, for the first time, beach action and criteria values for Enterococcus densities determined by both culture and qPCR methods.

Despite the advantages and effectiveness of qPCR for beach monitoring, and the limitations of culture-based testing, qPCR has yet to become the primary method for fecal indicator monitoring. While some beach monitoring programs such as those in Alberta, Canada, have made the transition to qPCR for FIB monitoring [13,14], many programs still rely on culture methods. In 2020, it was estimated that, of all assessments of microbial indicators funded through the United States BEACH Act, less than 1% were determined by qPCR [15]. Barriers to implementation of qPCR methods within existing monitoring programs have been reported to include startup costs associated with specialized equipment, cost of sample processing, time required for training, and the expectation of increased monitoring [16]. qPCR is traditionally limited to use by highly trained and experienced personnel in centralized laboratories, but the recent development of small, portable qPCR machines has increased the accessibility and portability of qPCR-based testing. This advance in qPCR technology has also given rise to rapid, in-field qPCR analysis of water samples [17–20].

While real-time monitoring of FIB densities is made possible by the implementation of qPCR, this may not always be achieved. Delays between sampling and dissemination of results may occur due to the time required for sample transport to a central laboratory, capacity for timely sample processing and analysis, and communication of results [21], or sampling frequency may be limited [14]. These scenarios again give rise to the same issue faced with the use of culture-based methods: risk assessment and beach management decisions are being informed by outdated data which may no longer reflect current water quality. Decreasing time to results and expanding monitoring capacity within qPCR programs would facilitate further realization of the advantages of molecular methods and enable more effective beach monitoring and protection of public health.

Community-based monitoring

Community-based monitoring (CBM) is a participatory research approach that serves as a powerful tool in environmental monitoring and research. CBM is cost-effective, enables surveillance and data collection across a vast spatial and temporal range, and facilitates a large scope of monitoring targets [22–25]. CBM has been hailed as a means of promoting local sustainable development, increasing monitoring capacity, empowering and engaging community members, and fostering greater inclusivity in environmental management and policy [26–28]. Importantly, co-creation of knowledge through a CBM approach also facilitates incorporation and appreciation of local knowledge and Indigenous expertise, and can support Indigenous sovereignty and environmental governance [26,28,29]. Recognition of the strengths of a community-based approach has resulted in increasing numbers of CBM programs and expansion of the variety of monitoring targets [27,28,30].

Community-based qPCR monitoring

Community-based qPCR monitoring (CBqM) is a novel idea which addresses current challenges in water quality monitoring and stands to increase the efficiency and effectiveness of surveillance and improve protection of public health. Rather than transporting samples to a core laboratory, CBqM decentralizes testing as partners perform qPCR locally, reducing time to results and increasing notification of hazardous water quality. Sample collection by CBM partners promotes increased geographic reach and frequency of sampling [24,31–33].

Putting qPCR analysis into the hands of inexperienced volunteers or citizen scientists is a significant departure from the typical restriction of molecular methods to use by highly trained professionals and has, to our knowledge, only been explored in three publications. Rudko et al. [18] demonstrated the implementation of a CBqM program for health hazards in recreational waters wherein trained community partners carried out qPCR testing of samples in the field. Participants were involved in every stage of the monitoring process including program design, sampling, DNA extraction, and qPCR analysis, while protocols, materials, reagents, and data interpretation were under control of a central research laboratory [18]. Both Tøttrup et al. [34] and Knudsen et al. [35] explored a citizen science approach to eDNA monitoring of a variety of species in marine and fresh waters, respectively. Water samples were collected by high school students and submitted to researchers for DNA extraction [34,35]. The students were later provided a comprehensive training session before conducting qPCR analysis at a central laboratory to determine the presence or absence of target DNA [34,35].

While the efficacy of a community-based qPCR monitoring program has been demonstrated, there remains a dearth of research which has rigorously investigated the quality of qPCR data generated by CBM partners against traditional methods. In this study, we assessed the reliability of a decentralized, CBqM program for enterococci in recreational waters over three years, wherein community partners were responsible for DNA extraction and qPCR analysis of recreational water samples in a satellite laboratory. Duplicate water samples enabled comparison of CBqM results to those of U.S. EPA Method 1611. CBM partner-extracted samples were also analyzed by the overseeing research team at the University of Alberta via the same CBqM protocols. In this study, we demonstrate the reliability of CBqM data: the ability of CBM partners to carry out protocols and the strong relationship between data generated by the CBqM method and by the gold standard, Method 1611.

Methods

Study design

Water samples were collected as part of the Province of Alberta Beach Monitoring program in accordance with provincial standardized protocols from priority beach access points at recreational lakes in Alberta. Volunteers of the Provincial sampling program collected composite water samples that were delivered to the University of Alberta research team within 48 hours of sample collection. Two equivalent filtered samples were prepared from each water sample: one filter was processed and analyzed following U.S. EPA Method 1611 by the research team at the University of Alberta; the duplicate filter underwent DNA extraction and qPCR analysis by a CBM partner following a CBqM protocol at a satellite laboratory. Partner-extracted samples then underwent qPCR analysis following the same CBqM protocol by the research team at the University of Alberta (Fig 1). Community-based qPCR monitoring methodology followed Rudko et al. (2020) and is detailed below.

thumbnail
Download:
Fig 1. Study design. Two equivalent filters were prepared from each water sample: one for analysis by Method 1611 and one for analysis by CBqM methods by CBM partner and by the central University of Alberta laboratory.

https://doi.org/10.1371/journal.pwat.0000309.g001

Training of CBM partners

CBM Partners recruited for this study were undergraduate students from the University of Alberta Faculty of Science with little to no prior laboratory training and no previous experience with qPCR. CBM Partners were provided with a written protocol for DNA extraction and qPCR analysis of samples and three in-person training sessions (S1 Text). Training included familiarization with protocols, use of equipment, workflow, and proper laboratory techniques. DNA extraction and qPCR analysis methods were demonstrated during the initial training session while the second and third sessions allowed partners to run samples with guidance as needed. Upon completion of training, partners were permitted to begin working independently in the decentralized laboratory, extracting and analyzing samples without supervision.

Sample collection, transport, and filtration

All water samples were collected by the Alberta Lake Management Society according to provincial recreational water monitoring protocols [36,37] with some deviations as detailed below.

Water samples were collected from recreational use access points (i.e., beaches). At each lake, either five or ten sampling points were selected along the beach where the water was approximately 1 meter deep. A 50mL sample was collected from each sampling point 0.1-0.5m below the surface of the water using sterile 50mL conical tubes. All 50mL samples were then combined into a single composite sample.

Water samples were transported in coolers on ice packs to the satellite laboratory for filtration. Any water samples not immediately filtered were stored at 4°C for a maximum of 48 hours from the time of collection. From each water sample, two 25mL or 20mL aliquots were filtered separately, creating two equivalent filtered samples for DNA extraction. Samples were vacuum filtered through a 0.4 μm polycarbonate filter (Pall FMFNL1050). Filters were stored at -20°C until DNA extraction.

Community-based qPCR monitoring methods

DNA extraction.

DNA extraction was performed using the Qiagen DNeasy Blood & Tissue Kit (cat. no. 69506). Extraction was performed as per manufacturer instructions with the addition of a bead beating step, as described in Rudko et al. [38].

Filters were transferred to 1mL tubes containing 0.2-0.3mm glass beads (Generite cat. no. S0205-50) before adding 200uL lysis buffer AL (Qiagen) and 20uL proteinase K solution (Qiagen). Tubes containing filter and beads were vortexed on high for 5 min using a vortex adapter or for 1 minute using a bead beater. Tubes were then centrifuged to collect the sample at the bottom of the tube before incubation for 10 minutes at 56°C. A negative extraction control was included with each batch of samples extracted. Samples not immediately analyzed by qPCR were stored at -20°C.

qPCR reagents.

qPCR master mix (enzyme mix, primers, and probes) was prepared at the University of Alberta in a designated cleanroom. Master mix for CBM partners’ use was aliquoted into 5mL tubes (Axygen SCT5MLS), each tube containing a sufficient volume for one complete qPCR run. All DNA standards were prepared at the University of Alberta in a UV dead box to prevent contamination. In 2020, our CBM partner was provided with serially diluted standards from which they prepared their own reaction tubes. In 2021 and 2022, reaction tubes containing master mix and plasmid standards were fully prepared at the University of Alberta and provided to CBM partners. Partners were instructed not to open reaction tubes containing standards. This shift was made to further mitigate risk of contamination of the satellite laboratory workspace with high concentration DNA. Master mix and standards were prepared and provided to CBM partners monthly, transported on ice, and stored at -20°C (Table 1).

thumbnail
Download:
Table 1. Comparison of methodological differences behind CBM Partner, Laboratory, and Method 1611 results.

https://doi.org/10.1371/journal.pwat.0000309.t001

IDT PrimeTime Gene Expression Master Mix was used. qPCR primers and probe sequences utilized were consistent with those employed in Method 1611, validated and published by the U.S. EPA [11]. This assay targets the 23S rRNA gene of Enterococcus spp. qPCR master mix contained 1x master mix, 1uM forward and reverse primers (Integrated DNA Technologies), and 0.08uM fluorescein-labeled probe (Integrated DNA Technologies). Forward and reverse primer sequences, respectively: 5’-GAGAAATTCCAAACGAACTTG; 5’-CAGTGCTCTACCTCCATCATT. TaqMan probe sequence: 5’-TGGTTCTCTCCGAAATAGCTTTAGGGCTA.

Samples were quantitated to a standard curve of synthesized pJET 1.2 plasmid containing the 23S rRNA gene sequence. Quantification of plasmid standard was performed via Qubit Fluorometer (ThermoFisher Scientific). Standard curve concentrations were run in duplicate by Partners and in triplicate by the research team at the University of Alberta and consisted of 50000, 5000, 500, 50, and 5 copies of the target sequence per reaction. A minimum of two no-template controls with molecular-grade water were included in each instrument run. Samples were run in duplicate, and the average value was reported. Each reaction contained 15uL of master mix and 5uL of DNA template.

qPCR thermocycling.

CBM partners performed qPCR using the portable Quantabio Q while the ThermoFisher QuantStudio 3 was utilized at the overseeing University of Alberta laboratory (Table 1). The thermocycling profile used was identical: a 2-step reaction consisting of a 5-minute initial hold at 95°C followed by 40 cycles of a 5-second denaturation step at 95°C and a 30-second annealing step at 60°C.

qPCR reporting units.

Data generated by community partners and at the central laboratory following the community-based qPCR methodology (S1 Data) are presented in cell equivalents (CE) per 100mL (Table 1). Cell equivalents were calculated as the total number of target sequences detected per 100mL divided by 15 to align with U.S. EPA Method 1611.1’s [39] recommended reporting units which reflect a ratio of 15 target sequences per calibrator cell. Partner- and Laboratory-generated qPCR data was adjusted for 50% extraction efficiency relative to that of Method 1611.

U.S. EPA method 1611 DNA extraction and qPCR analysis methods

Duplicate sets of filtered samples underwent DNA extraction and qPCR analysis according to Method 1611 [11]. qPCR analysis was conducted using the Applied Biosystems 7500 Fast Real-Time PCR System. Calibrator cell samples were prepared in-house, as described in Method 1611.

Reporting units.

Data generated by U.S. EPA Method 1611 are presented in units of RWQC-adjusted calibrator cell equivalents (CCE) (Table 1), calculated as described in U.S. EPA Method 1611.1 [39]. Briefly, total target sequences detected per sample are divided by 15 to reflect a ratio of 15 target sequences per calibrator cell [39].

Statistics

Limit of detection was calculated using Wilrich & Wilrich [40] POD/LOD calculator. Statistical analyses were performed in IBM SPSS (Version 28 & 29). Data showed normality following log10-transformation. Scatterplots were created in IBM SPSS (Version 29). Positive predictive values were calculated as the number of samples considered to be ‘true exceedances’ (indicated to exceed a 1280 CCE/100mL statistical threshold value [STV] by both Partner and Method 1611) divided by the sum of ‘true exceedances’ and ‘false exceedances’ (samples for which CBM Partner results indicated exceedance of the STV but Method 1611 did not). Negative predictive values were calculated as the number of samples considered to be ‘true non-exceedances’ (indicated to be below the STV by both Partner and Method 1611) divided by the sum of ‘true non-exceedances’ and ‘false non-exceedances’ (samples for which CBM Partner results indicated were below the STV, but Method 1611 identified as exceeding the STV). Spearman correlation analysis was performed on log10-transformed CE/100mL and log10-transformed CCE/100mL values. Intraclass correlation coefficients were calculated on log10-transformed CE/100mL and log10-transformed CCE/100mL values using a two-way random effects model with a consistency definition and single measures to assess interrater reliability. Maximum expected difference (with 95% probability) between CBM Partner and Method 1611 values for matched sample results was calculated as described in Ebentier et al. [41] using nested ANOVA with variance component analysis. Model factors were defined as qPCR Method (CBqM or Method 1611), Sampling Location, and Sample (nested within Sampling Location). Maximum expected difference (with 95% probability) between methods was calculated using the sum of variance component estimates of Method and Error. Percent coefficients of variation (CV) were calculated as the standard deviation divided by the mean of CBM Partner and Method 1611 results for each sample, multiplied by one hundred, using log10-transformed CE/100mL and CCE/100mL values respectively. Bland-Altman plots were created in Prism 10 (GraphPad, USA) using log10-transformed CE/100mL and log10-transformed CCE/100mL values.

Ethic statement

Ethics approval for this study was waived by the University of Alberta Research Ethics Board. It was determined that this study meets one of the conditions outlined under Chapter 2 of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans – TCPS 2 (2022) as an activity that does not require Research Ethics Board review. No human subjects or data associated with human subjects or animals were involved in this research.

Results

A total of 380 samples were processed and analyzed for enterococci by community partners at a satellite laboratory over the course of this study.

Data quality

Average R2 across all CBM partner qPCR runs was 0.99. Average E across all CBM partner qPCR runs was 0.92. Acceptance criteria for data from CBM partners included R2 ≥ 0.97 and E 0.75-1.2. Samples displaying inhibition as described in Method 1611 [11] were omitted from analysis. The 95% limit of detection (LOD95) of the Enterococcus assay is 3 target sequence copies per 5μL using the QuantStudio 3 [42] and 29 target sequence copies per 5μL using the Quantabio Q.

Comparison of beach management decisions by method

According to the Alberta Safe Beach Protocol, any recreational water sample with a detected Enterococcus concentration above the statistical threshold value (STV) of 1280 CCE/100mL triggers follow-up microbial source tracking and possible issuance of a water quality advisory [14]. Sample results obtained from each analysis method were categorized as either exceeding or falling below the STV (Fig 2). Percent agreement was subsequently determined as the proportion of samples categorized consistently across methods. Variability in agreement was observed across different years and between different method comparisons.

thumbnail
Download:
Fig 2. Scatterplots of paired log10-transformed Partner and Method 1611 values in 2020 (panel A), 2021 (panel B), and 2022 (panel C).

The result of analysis by Method 1611 is displayed on the X-axis while the corresponding Partner result is displayed on the Y-axis. Each filled circle represents one sample. Dotted lines indicate the STV of 1280 CCE/100mL and 1280 CE/100mL respectively. Samples within the upper-right quadrant of each plot were found to exceed the STV by both Method 1611 and by the Partner; samples within the lower-right quadrant were found to exceed the STV by Method 1611 but not by the Partner; samples within the lower-left quadrant were found not to exceed the STV by both Method 1611 and the Partner; samples in the upper-left quadrant were found to exceed the STV by the Partner but not by Method 1611.

https://doi.org/10.1371/journal.pwat.0000309.g002

Percent agreement between Partner and Laboratory values in relation to a 1280 CE/100mL guideline value was 74.1% in 2020, 71.2% in 2021, and 68.4% in 2022. Percent agreement between Partner and Method 1611 values in relation to a 1280 CE/100mL or 1280 CCE/100mL guideline value respectively was 72.8% in 2020, 80.0% in 2021, and 67.3% in 2022. Compiling data from all years of study showed 71.0% agreement between Partner and Laboratory results (n = 341) and 72.8% agreement between Partner and Method 1611 results (n = 279). False non-exceedances, defined as exceedance of the STV according to the Method 1611 sample result but not according to the Partner result, accounted for 11.8% (33 of 279 samples) of all Partner results or 43.4% (33 of 76 samples) of all disagreement between Partner and Method 1611 results (Table 2). False exceedances, defined as a Partner result exceeding the STV when the Method 1611 result did not, accounted for 15.4% (43 of 279 samples) of all Partner results or 56.6% (43 of 76 samples) of disagreement between Partner and Method 1611 results (Table 2). Partner results correctly identified 72.0% (85 of 113 samples) of samples which exceeded the guideline STV according to their Method 1611 result and 73.3% (118 of 161 samples) of samples which did not exceed the STV according to Method 1611 results.

thumbnail
Download:
Table 2. Agreement between Partner and Method 1611 results as to whether or not a sample exceeded the 1280 CCE/100mL STV.

https://doi.org/10.1371/journal.pwat.0000309.t002

Positive predictive values (PPV) indicate the percentage of Partners’ ‘exceeds STV’ results that are considered ‘true exceedances’, while negative predictive values (NPV) indicate the percentage of Partners’ ‘does not exceed STV’ results that are considered ‘true non-exceedances’. A result is considered a ‘true exceedance’ or ‘true non-exceedance’ when the corresponding Method 1611 result also indicates STV exceedance or non-exceedance, respectively. PPVs were found to range between 44.2% and 85.7% (Table 2) across different years, indicating the proportion of Partners’ ‘exceeds STV’ results for which exceedance was also indicated by Method 1611. NPVs were found to improve following the change in preparation of Partners’ DNA standards: NPVs for 2021 and 2022 indicate that, 81.3% and 89.1% (Table 2) of the time, respectively, if Partners reported a sample to be below the STV and thus of acceptable water quality, Method 1611 revealed the same result, indicating that the sample truly was of acceptable quality.

Reliability of CBqM results

Correlation of enterococci values between users and across methodologies was assessed by Spearman’s rank-order correlation using log-transformed CE/100mL and log-transformed RWQC-adjusted CCE/100mL. Spearman’s correlation coefficient between Partner and Laboratory values was 0.677 (ρ(79)=0.677, p < 0.001) in 2020, 0.812 (ρ(144)=0.812, p < 0.001) in 2021, and 0.726 (ρ(112)=0.726, p < 0.001) in 2022. Spearman’s correlation coefficient between Partner and Method 1611 values was 0.766 (ρ(90)=0.766, p < 0.001) in 2020, 0.771 (ρ(78)=0.771, p < 0.001) in 2021, and 0.567 (ρ(105)=0.567, p < 0.001) in 2022. Compiling results from all three years, Spearman’s correlation coefficient was 0.704 (ρ(339)=0.704, p < 0.001) between Partner and Laboratory values and 0.635 (ρ(277)=0.635, p < 0.001) between Partner and Method 1611 values (Table 3).

thumbnail
Download:
Table 3. Spearman’s correlation and ICC values for comparison of Partner and Laboratory and of Partner and Method 1611 results.

https://doi.org/10.1371/journal.pwat.0000309.t003

Correlation and agreement of log-transformed CE/100mL and log-transformed CCE/100mL values between users and methodologies was assessed by intraclass correlation coefficient (ICC). ICC values between users varied across the years of study. ICC between Partner and Laboratory values was 0.686 (p < 0.001, 95% CI: 0.551, 0.786) in 2020, 0.842 (p < 0.001, 95% CI: 0.788, 0.884) in 2021, and 0.765 (p < 0.001, 95% CI: 0.676, 0.831) in 2022. ICC between Partner and Method 1611 values was 0.731 (p < 0.001, 95% CI: 0.619, 0.813) in 2020, 0.780 (p < 0.001, 95% CI: 0.677, 0.853) in 2021, 0.604 (p < 0.001, 95% CI: 0.468, 0.711) in 2022. Compiling results from all three years, ICC was 0.728 (p < 0.001, 95% CI: 0.674, 0.774) between Partner and Laboratory values and 0.677 (p < 0.001, 95% CI: 0.608, 0.736) between Partner and Method 1611 values (Table 3).

Variability between method results

Maximum expected difference (with 95% probability) between CBqM method and Method 1611 Enterococcus values ranged from 0.845 log10 copies/filter in 2020, 0.669 log10 copies/filter in 2021, to 0.704 log10 copies/filter in 2022.

Percent CV between CBM Partner and Method 1611 log10-transformed values varied by year. Median CV was 10.29% in 2020, 7.07% in 2021, and 8.10% in 2022. 80% of samples across all years of study had a %CV below 15.32.

Agreement and bias between methods

Bland-Altman plots A-C (Fig 3) display the average of paired Partner log10 CE/100mL and Method 1611 log10 CCE/100mL values on the X-axis and the difference between these values (Method 1611 Value - Partner Value) on the Y-axis. 95% limits of agreement were (-0.72, 1.33) in 2020, (-0.89, 0.98) in 2021, and (-1.11, 0.61) in 2022. Mean difference between duplicate sample log10-transformed values ranged from 0.30 (2020), 0.042 (2021), to -0.25 (2022). Bland-Altman plots D-F (Fig 3) display the average of paired Partner and Laboratory log10 CE/100mL on the X-axis and the difference of these values (Laboratory Value - Partner Value) on the Y-axis. 95% limits of agreement were (-0.89, 1.22) in 2020, (-1.10, 0.37) in 2021, and (-0.91, 0.35) in 2022. Mean difference between Partner and Laboratory log10-transformed values for the same extract ranged from 0.17 (2020), -0.37 (2021), to -0.28 (2022).

thumbnail
Download:
Fig 3. Bland-Altman plots of paired log10-transformed Partner and Method 1611 values (panels A-C) and of paired log10-transformed Partner and Laboratory values (panels D-F).

The average of each set of paired values is displayed on the X-axis while the difference between values is plotted on the Y-axis. Each filled circle represents one sample. Dotted lines indicate 95% limits of agreement (mean difference ± 1.96*standard deviation of differences).

https://doi.org/10.1371/journal.pwat.0000309.g003

Discussion

This study’s demonstration of community partners’ ability to produce reliable qPCR data supports further investigation and application of community-based citizen science approaches to qPCR water monitoring and research. We evaluated the ability of trained community partners to reliably conduct qPCR-based water quality monitoring for Enterococcus at satellite laboratories and assessed the relationship between data derived from this decentralized CBqM approach and from U.S. EPA Method 1611. Partners were responsible for DNA extraction and qPCR analysis of recreational water samples using methodology designed to be appropriate for safe use by novices in the field. Partners’ results were compared to those generated by the research team at the central University of Alberta laboratory, both of partner-extracted samples analyzed via the same CBqM assay and of duplicate water samples extracted and analyzed according to Method 1611.

Methodological differences of the CBqM method

The CBqM methods demonstrated here were developed to closely parallel those of U.S. EPA Method 1611 while being suitable for use by novice partners without prior research experience or access to standard laboratory facilities and equipment. CBqM methods must additionally be designed for safe usage in satellite laboratories. While these requirements necessitated deviations from Method 1611, the impact of each methodological difference on the relationship between results of both methods was considered.

Sample & filtration.

While each recreational water sample was mixed and divided across two filters to create two duplicate samples, potential exists for variation in the concentration of target material (bacterial cells, extracellular DNA, organic or fecal material, etc.) and contaminants between filters. While not directly experimentally measured, this source of potential variability likely contributes to observed variation between results of Partner-extracted samples and Method 1611.

DNA extraction.

Unlike the crude extraction in Method 1611, the kit-based DNA extraction protocol used by Partners includes steps for denaturation of nucleases and the removal of contaminants. This purification reduces the risk of qPCR inhibition and improves DNA stability but may result in a decreased yield [43,44]. The Qiagen DNEasy Blood & Tissue kit used for Partner extractions was found to have the greatest extraction efficiency out of those assessed in a number of studies [45–47]. The average extraction efficiency of similar kits has been found to range from 2.4% ± 0.1% to 28.3% ± 10.5% [44,48]. Validation studies of Method 1611 revealed an average extraction efficiency of 83% from fresh water samples [11]. It is important to note, however, that there is potential for variation in extraction efficiency between samples as well as across different extraction kits or protocols and matrices [11,44–46,49,50]. For example, Method 1611 [11] states that validation of the protocol revealed variation in extraction efficiency across laboratories, samples, and matrices; average recovery ranged between 68%-841% and 7%-156% across different laboratories for marine and freshwater samples, respectively, that were spiked with known quantities of E. faecalis prior to extraction. Since the extraction efficiencies of both Method 1611 and the CBqM protocol are variable, the difference in DNA recovery between methods is likely inconsistent across different samples. This variability contributes to the differing levels of agreement observed between CBqM and Method 1611 results.

To account for the observed difference in average extraction efficiency between kit-based extraction and Method 1611, Partner and Laboratory values were adjusted by a constant factor, assuming 50% extraction efficiency. While this adjustment does not mitigate the impact of variable DNA recoveries, it addresses the lower efficiency of kit-based extractions relative to Method 1611 to provide a more conservative estimate of water quality by the CBqM method. If CBqM values were not adjusted for lower extraction efficiency, the overall agreement between Partner results and Method 1611 regarding beach management decisions based on the 1280 CCE/100mL STV would improve, increasing from 72.8% (Table 2) to 78.1%. However, the number of samples for which Partner results fail to capture an exceedance of the STV that is indicated by Method 1611 would increase from 33 samples (Table 2) to 55 samples, decreasing the method’s observed sensitivity in capturing STV exceedances from 72.03% to 53.4% and decreasing the negative predictive value from 78.1% (Table 2) to 73.8%. These findings highlight the importance of understanding the impact of differences in DNA recovery between methods on the comparability of results.

Quantification method & reference material.

Perhaps the most important difference between the CBqM protocol employed in this study and U.S. EPA Method 1611 is the quantification method and reference material. While both methods utilize the same qPCR chemistry, primer and probe sequences and concentrations, they differ in quantification type. The CBqM Enterococcus assay is an absolute quantification method that quantifies samples based on a standard curve of plasmid DNA containing known quantities of the target sequence. Method 1611 utilizes the comparative cycle threshold (ΔΔCT) relative quantitation method in which samples are quantified against calibrator samples composed of known quantities of Enterococcus faecalis cells [11]. A plasmid standard curve method was selected because use of E. faecalis calibrator samples was deemed inappropriate for use in a decentralized CBM context due to biosafety requirements. Both methods quantify the concentration of target sequences in experimental samples through comparison of the qPCR cycle threshold (CT) value of the sample to that of the reference material. In the standard curve method, the CT values and concentrations of the plasmid standards are used to plot a standard curve against which the CT value of the experimental sample is compared, revealing the concentration of target sequences in the experimental sample. In the ΔΔCT method, comparison of CT values provides a ratio of target sequences detected in the calibrator cell samples to the experimental sample. As the number of cells present in the calibrator samples is known, the ratio of target sequences detected in calibrator samples to those detected in the experimental sample is multiplied by the calibrator sample cell count to estimate the number of cells, expressed in calibrator cell equivalents (CCEs), present in the experimental sample.

Comparison of sample results to water quality standards, as well as comparison of data across methods, requires the use of standardized units of measure. Haugland et al. [51] displayed that the number of target sequences detected per E. faecalis calibrator cell can vary significantly. As the number of target sequences detected per calibrator cell is intrinsically tied to the derivation of CCEs, changes in the number of target sequences detected per calibrator cell effectively alters the size of CCEs as a unit [51]. This has a potentially large and significant impact on the calculated number of CCEs in experimental samples. The number of cells in the calibrator sample is used to estimate the number of cells in the experimental sample as a function of the ratio of target sequences. Thus, if the calibrator cell count were to remain the same, but the number of target sequences detected per calibrator cell changed, the number of CCEs calculated to be in the experimental sample would be altered.

Updates to Method 1611, published in U.S. EPA Method 1611.1, describe methods for the calculation of assay results in new, standardized units referred to as RWQC-adjusted CCEs [39]. Calculation of RWQC-adjusted CCEs is similar to that of unstandardized CCEs, but corrects values for variation in the ratio of target sequences to calibrator cells and standardizes units to reflect a ratio of fifteen target sequences per calibrator cell. To facilitate comparison of data across methods in units that were as similar as possible, RWQC-adjusted CCEs were used in this study. Laboratory and CBM Partner results, originally expressed as target sequence copies per 100mL, were divided by fifteen to reflect the same target sequence to cell ratio used in the calculation of RWQC-adjusted CCEs per 100mL units used to express Method 1611 data.

The ΔΔCT method used in U.S. EPA [11] Method 1611 also includes the use of salmon DNA as a sample processing control (SPC), added to both water samples and calibrator cell samples prior to extraction. The SPC enables detection of inhibition and adjustment for potential variation in extraction efficiency between calibrator and experimental samples [11]. In future applications, the addition of a SPC within the CBqM protocol may reduce variability and improve reliability of CBqM data by enabling normalization of extraction efficiency between samples extracted by different users or by different methods and across different sampling locations, as well as facilitate detection of qPCR inhibition.

qPCR thermocycler.

Different qPCR thermocyclers were employed in each method, presenting an additional source of potential variability. Ease of use and portability were key factors in the selection of the Quantabio Q for use by CBM partners. The Quantabio Q has a higher LOD compared to thermocyclers used in the central laboratory and in Method 1611, resulting in decreased sensitivity at low Enterococcus concentrations. Differences in LOD is likely a contributor to variability observed between methods amongst low concentration samples.

Data quality

Tracking of experimental control and data quality metrics was performed as an important aspect of assessing the reliability of a CBM approach to qPCR monitoring. Two Partner qPCR runs were deemed unacceptable due to amplification failure. Data from these runs were excluded and the samples were re-analyzed by partners. Average run parameters across all years of study fell within acceptable parameters [52], though inclusion criteria for individual runs was broadened to both maximize the number of Partner data points and to assess the reliability of Partner results under real-world conditions which likely increased variability between methods [53].

10.38% of samples analyzed via Method 1611 displayed qPCR inhibition and were excluded from further analysis. While Partners’ qPCR analyses did not include inhibition controls, future applications of CBqM could incorporate inhibition testing with relative ease via an internal amplification control [18]. The Qiagen DNeasy kit-based extraction method selected for use by Partners includes multiple steps for the removal of contaminants, including potential inhibitors [54]. Utilization of an environmental master mix in lieu of universal master mix would further mitigate risk of inhibition amongst recreational water samples [55].

A centralized source of qPCR reagents, reference material, and protocols enabled standardization between Partners. Prepared DNA standards were provided to Partners to reduce variability and minimize risk of contamination. Workflow and separation of workspace was established at the satellite laboratory for the handling of qPCR master mix and experimental samples to further reduce risk of contamination.

Agreement between CBqM and method 1611 results for beach management decisions

The 2012 U.S. EPA Recreational Water Quality Criteria and 2022 Alberta Safe Beach Protocol recommend use of a STV of 1280 CCE/100mL as exceedance of this value is indicative of poor water quality and significant health risk to recreators [4,14]. Under the Alberta Safe Beach Protocol [14], samples found to have an Enterococcus concentration in excess of the 1280 CCE/100mL STV but below 6400 CCE/100mL are sent for microbial source tracking (MST), the result of which determines whether or not a beach advisory is issued. Percent agreement was defined as the percentage of samples for which both results (either Partner and Laboratory or Partner and Method 1611) agreed as to whether the sample did or did not exceed the STV. While percent agreement does not provide a full depiction of concordance between methods, we wanted to assess how CBM Partner results would compare to those of Method 1611 if they were used in a real-world scenario to inform beach management decisions as indicated by the guideline STV. For example, if our CBM program had been used to flag priority samples for MST based on the guideline STV, how many samples with unsatisfactory water quality would have been missed?

The percentage of samples for which Partner and Laboratory or Partner and Method 1611 results agreed varied by year (Table 2). Compiling data across all years, percent agreement between the results of Partner and Method 1611 analyses (72.8%, n = 279) was slightly greater than that of Partner and Laboratory (71.0%, n = 341), despite the latter’s decreased number of methodological differences, though 95% confidence intervals revealed this difference to be non-significant (Table 2). CBM Partner results indicated a different beach management decision than Method 1611 for 27.2% (95% CI: 22.0%, 32.5%) of samples. Partners reported false non-exceedances (Method 1611 results indicated the sample exceeded the 1280 CCE/100mL STV, but Partner results did not) for 11.8% (95% CI: 8.0%, 15.6%) of samples and reported false exceedances (Partner results indicated the sample exceeded the 1280 CCE/100mL STV, but Method 1611 results did not) for 15.4% (95% CI: 11.2%, 19.7%) of samples.

The percent agreement between Partners and Method 1611 regarding STV exceedance was similar across years of study. However, partner results were inconsistent in whether this disagreement resulted from over- or under-estimation of Enterococcus concentrations relative to Method 1611 from year to year (Table 2). In 2020, 84.0% of the disagreement observed resulted from false non-exceedances. However, in 2021 and 2022, the majority of disagreement (62.5% and 82.9%, respectively) observed was the result of false exceedances. Similarly, the rate of false non-exceedance was greater than the false exceedance rate for 2020 Partner data, while the opposite was found in 2021 and 2022. In 2020, our CBM partner was responsible for preparing their own standards from a provided serial dilution of plasmid DNA, while, in following years, we provided partners with fully prepared standards in reaction tubes to minimize risk of contamination of the satellite laboratory with high concentration DNA. This inconsistency is a limitation of this study but presents a potential explanation for this observed change in the proportion of false non-exceedance results in 2020 compared to subsequent years. Regardless, this finding highlights the potential for variability in the type of disagreement between methods, resulting from relative over- or underestimation of sample concentrations, and the impact this may have on the interpretation of the data. Within equal overall levels of agreement, a greater proportion of false exceedance results would indicate that the method could be considered a more conservative approach; flagging a greater proportion of samples as risky, while a greater proportion of false non-exceedance results might elicit concern that potentially hazardous samples are not being captured by the method. This difference in sensitivity and specificity of the method in categorizing samples relative to a guideline value alter how data may be interpreted and applied in a real-world monitoring scenario. It is thus crucial to understand and consider any potential methodological bias in the application of CBqM data.

The impact of false guideline exceedances and false non-exceedances is dependent on the interpretation and application of results. In a scenario wherein samples reported by CBM partners to have an Enterococcus concentration of 1280 CE/100mL or greater would be flagged for either confirmatory testing by Method 1611 or MST by the overseeing laboratory, false exceedances would result in additional samples unnecessarily undergoing follow-up testing while false non-exceedances would lead to a failure to flag samples with unacceptable water quality and potential exposure of beachgoers to increased health risk. While the rate of false exceedances must remain low enough so as not to be prohibitive, minimization of false non-exceedances should be prioritized to better protect public health. We therefore sought to further analyze Partners’ false non-exceedance results, particularly those observed in 2021 and 2022. Six false non-exceedances were observed in Partner results in each of 2021 and 2022 out of 44 and 29 STV exceedances, respectively, indicated by Method 1611. All twelve of these samples were found to have Laboratory results which also indicated no STV exceedance. This suggests that the occurrence of Partners’ false non-exceedances in 2021 and 2022 was the result of differences between Partner- and Method 1611-extracted samples due to either variation between the filtered samples prior to extraction or excess differences in extraction efficiency between the methods for these samples.

Positive predictive (PPV) and negative predictive values (NPV) provide a measure of the reliability of results and how they may best be applied. PPVs and NPVs reflect the probability that, if a Partner result is ‘exceeds guideline’ or ‘does not exceed guideline’, respectively, the corresponding Method 1611 result is the same. A high false non-exceedance rate contributed to 2020’s lower NPV, while the lower PPV observed for 2022 reflects the increased proportion of false exceedances. Strong NPVs following the switch to providing Partners with fully prepared DNA standards suggest that CBqM data could be used to identify samples of acceptable water quality with reasonable confidence. It is important to note that, unlike measures of sensitivity and specificity, positive and negative predictive values are influenced by the prevalence of the outcome being tested and may thus vary when the same testing method is applied to different sample populations (Monaghan et al., 2021).

While these results provide valuable insight into the agreement between methods, it is essential to recognize that they do not fully define the reliability of CBqM data. The CBqM methodology under study is not presented as a replacement for traditional testing by gold standard methods, nor is it suggested that solely Partner data be used for issuance of beach closures. Estimates of agreement offer a simplified binary perspective that does not capture the continuous nature of the data produced by each method. The value and application of partners’ results is not limited to categorizing samples relative to the specified STV. Additional water quality data produced through CBqM is informative for addressing temporal and geographic gaps in monitoring, the prioritization of monitoring locations, and the identification and study of both short- and long-term trends. Furthermore, different STVs might be applied to CBqM data depending on the specific goals of the monitoring program. While estimates of agreement and predictive value are informative, they should be interpreted alongside other measures of method performance to provide a comprehensive assessment of reliability.

Consistency and reliability of CBqM data

Strength of the correlation between sample results across methods was assessed to elucidate the relationship between data obtained by different analysis methodologies and user experience levels. Observation of a stronger correlation between Partner and Laboratory data (ρ(339)=0.704, p < 0.001) than between Partner and Method 1611 (ρ(277)=0.635, p < 0.001) (Table 3) is expected due to the methodological differences between the CBqM protocol and Method 1611 and potential variation between duplicate filters, but highlights the importance of evaluating the impact of these differences on the relationship between different methods and suggests that CBqM data should be interpreted with a degree of caution if used alongside Method 1611-based monitoring. Similarly, a greater level of agreement as indicated by ICC was observed between Partner and Laboratory data compared to Partner and Method 1611 data when results from all years of study were compiled (0.728 and 0.677, respectively) as well as in results from 2021 and 2022 alone, though the inverse was seen in results from 2020 (Table 3). However, confidence intervals reveal that the ICC values for 2020 comparisons were not significantly different from one another.

As Partner and Laboratory results stem from the same community partner-extracted sample, the potential for differences in extraction efficiency and filter-to-filter variation to impact the relationship between these data sets is precluded. Conversely, these sources of variability, as well as differences in extraction method, are relevant in the relationship between Partner and Method 1611 data. For example, the presence of inhibitors in water samples may differentially affect Partner and Method 1611 results due to the purification only of Partner-extracted samples as part of the kit-based extraction method. Additionally, differences between the LOD of the CBqM assay conducted on the portable Q thermocycler and of Method 1611 may increase variability between methods for low-concentration samples. As such, observation of a stronger correlation between Partner and Laboratory data than between Partner and Method 1611 data was expected.

Fernández-Baca et al. [17] assessed correlation between different qPCR methods for Enterococcus monitoring at various recreational water locations. Several methods were tested against a U.S. EPA Method 1611.1-based assay, all utilizing the same U.S. EPA Enterococcus spp. 23S rRNA gene primer and probe sequences. Methods varied in the extraction protocol, master mix, and thermocycler used, as well as volume of DNA extract and water sample. Fernández-Baca et al. [17] found that strength of the correlation to the Method 1611.1-based assay results varied by method and sampling location. Significant Pearson correlation coefficients between r = 0.522 and r = 0.807 were reported when samples were limited to a small number of localized sampling sites, while no significant correlation was observed when the number and diversity of sampling sites was increased [17]. Results of Fernández-Baca et al. [17] highlight the considerable impact that sampling location can have on the relationship between results of different qPCR methods. This points to an important consideration when employing CBqM-adapted protocols across different locations. Correlations reported in this paper stem from results of samples from more than fifty different lakes. Despite this, correlations observed between results of our CBqM protocol and Method 1611 were found to be significant (p < 0.001) and fall within the range of the significant correlations reported by Fernández-Baca et al. [17].

Rudko et al. [18] reported ICCs as a measure of agreement between results of avian schistosome- and cyanobacteria-targeting qPCR assays generated by community partners and experts. Results were compared under two scenarios: when partners and experts analyzed the same set of partner-extracted samples and when partners and experts each extracted and analyzed one half of split water samples following identical protocols [18]. In both scenarios, analyses by partners and experts were performed at separate locations and utilized different qPCR thermocyclers [18]. ICC values were reported to vary by year and by assay, ranging from 0.57 (95% CI: 0.1, 0.86) to 0.88 (95% CI: 0.85, 0.90) for the comparison of results of the same extracts and from 0.54 (95% CI: 0.32, 0.68) to 0.67 (95% CI: 0.37, 0.83) for split samples [18].

The comparison conducted by Rudko et al. [18] for results stemming from analysis of the same set of partner-extracted samples is similar to the comparison of Partner to Laboratory results reported here. As may then be expected, ICC values for Partner and Laboratory results (Table 3) fall within the range reported by Rudko et al. [18] for partner-extracted samples. Comparison of Partner to Method 1611 results reported here carries all of the same sources of variability as in the comparison of split samples by Rudko et al. [18], but additionally includes different DNA extraction and qPCR assay protocols as major potential contributors. Despite this, ICC values for the comparison of Partner and Method 1611 results for 2020, 2021, and when all years’ data are combined is greater than those reported by Rudko et al. [18] for the comparison of split samples, indicating a higher degree of reliability.

The findings of Rudko et al. [18] suggest that assay and user experience contribute to the level of agreement observed between CBM partner and core laboratory results. Rudko et al. [18] further discussed agreement between results from the portable thermocycler used by partners and the core thermocycler used by experts. It follows that choice of thermocycler plays a role in concordance between results and should thus be considered in the employment of CBqM methods. While not measured directly, it is possible that differences between the portable thermocyclers selected for use in this study and by Rudko et al. [18], as well as the difference in qPCR assays tested, contributed to the increased agreement observed in this study.

Variability & reproducibility between CBqM and gold standard results

Ebentier et al. [41] reported reproducibility values as the maximum expected difference (with 95% probability) between the results of qPCR analyses conducted in different laboratories on identical samples when following the same protocols and using standardized reagents. They assessed reproducibility in nine different MST assays and one Enterococcus-targeting assay, displaying maximum expected differences of between 0.09-0.37 log10 copies per filter for MST assays and 0.66 log10 copies per filter for the Enterococcus assay. Rudko et al. [18] similarly reported reproducibility values as the upper limit of the 95% confidence interval of the mean of the log difference of all paired results. These values ranged between 1.0-1.5 log for results of the same DNA extract analyzed by CBM partners and experts following the same protocol but using different thermocyclers, and between 1.3-1.4 log for results of split samples which were divided and then extracted and analyzed by partners and experts separately [18]. We found the maximum expected difference (with 95% confidence) in log10 copies per filter between CBM Partner, Laboratory, and Method 1611 sample results was 0.845 in 2020, 0.669 in 2021, and 0.704 in 2022.

Ebentier et al. [41] found that some assays displayed significantly lower variability than others across laboratories employing identical protocols and reagents. Furthermore, Ebentier et al. [41] observed greater variability among low-concentration samples in MST assays, pointing to the influence of target concentration on observed reproducibility and variability between methods. These findings demonstrate important considerations when comparing reproducibility values across studies.

Ebentier et al. [41] also assessed variability in qPCR results across laboratories when protocols and reagents were not standardized. Interlaboratory %CV values for each assay studied were calculated from the results of both core laboratories which used standardized reagents and protocols and non-core laboratories which used non-standardized reagents and protocols [41]. Ratios of the number of core to non-core laboratories ranged from between 3:2–5:1 across assays studied [41]. Resulting assay-specific median %CV values for data stemming from these mixed methods were found to be greater than those of standardized methods alone, ranging from 3.4% to 17.6%, with four out of six assays having a median CV between 5% and 10% [41]. Given that these results stemmed from data which was pooled from both laboratories utilizing identical, standardized protocols and reagents and laboratories following differing protocols and reagents [41], a greater degree of variability as indicated by the median CV might be expected when comparing data solely from distinct methods. However, median %CV values between CBM Partner and Method 1611 results across all years fall between 7.07% and 10.29%, within the range reported by Ebentier et al. [41], with a CV below 15.32% for 80% of total samples.

Shanks et al. [53] assessed interlaboratory variability across different qPCR methods for Bacteroidales and Enterococcus. Protocols, qPCR thermocyclers, and reagents were consistent across all eight laboratories studied. Significant variability between results from different laboratories was observed, though overall %CV values remained low; the majority of results (average log10 cell equivalents) for identical samples across laboratories exhibited a CV below 10% [53]. The coefficient of variation was found to be below 10% for 90.9% (ten of eleven) samples in two Enterococcus assay variations and for 100% of samples in another two Enterococcus assay variations [53]. Compiling Partner and Method 1611 data from all three years of study, we found the coefficient of variation to be below 10% for 58.4% of samples. As sample analyses reported by Shanks et al. [53] were conducted by professionals in research laboratories with standardized protocols and reagents, it is not surprising that the reported interlaboratory %CV is lower than the inter-method variation observed between CBM Partner and Method 1611 results. Shanks et al. [53] found that departures from study protocols were associated with increased variability between laboratories’ results. Greater variability may then reasonably be expected from more significant methodological differences, such as those described in this paper.

Bias and agreement between methods

Any potential bias and the agreement between methods is graphically displayed by Bland-Altman plots. Scattering of data within the limits of agreement and the absence of a consistent trend or pattern indicates uniform variability in agreement between methods and a lack of proportional bias (Fig 3). Across comparisons of CBM Partner data to both Laboratory and Method 1611 results, 95% limits of agreement narrow by year, indicating decreasing variability and increased agreement [56], potentially due to improvement in the application of CBqM methods and honing of training tools. As expected, 95% limits of agreement tended to be smaller for the comparison of Partner to Laboratory data rather than to Method 1611, indicating more consistent agreement between Partner and Laboratory values. Mean differences, however, varied by year.

It is important to note that CBM partners in this study had only a short period of time within which to gain practice and competency in the protocols, receiving three training sessions prior to working independently and without supervision, and subsequently analyzing all samples over the course of only five to eight weeks. It would be expected that, as partners gain more experience with the methodology, their proficiency would improve. It may therefore be advantageous to establish CBqM partnerships where long-term commitment can be expected.

Future studies should seek to elucidate the variability in target DNA between duplicate filters from split samples to enable better understanding of what proportion of disagreement observed between data sets is attributable to variation between replicate filters and inherent variability in extraction efficiency compared to variation resulting from methodological differences or user experience.

Future applications of CBqM

As the employment of DNA-based testing expands, community-based qPCR monitoring (CBqM) emerges as a pivotal tool for enhancing monitoring program capacities, suitable for both water management and research objectives. This study underscores the reliability of qPCR monitoring data generated by community partners. A single laboratory group responsible for protocol validation, training, equipment calibration, reagent preparation, and data analysis, could effectively oversee and manage numerous CBqM programs and satellite labs. Collaborative partnerships fostered through CBqM pave the way for heightened sampling and testing frequency, an expanded scope of monitoring targets, and incorporation of testing for location-specific targets and community concerns. Importantly, these partnerships enable routine monitoring in remote areas which may otherwise not be feasible for standard, centralized programs to reach. Furthermore, CBqM applications may be tailored to suit partner commitment and program needs, with the potential for partner involvement to range from sample collection alone to participation in all aspects of program design and execution. Due to its adaptable nature, with appropriate program design, quality control, and validation of novel methods, CBqM stands poised to bolster provincial or state beach monitoring programs for recreational water quality indicators such as FIB, support broader environmental monitoring initiatives, and empower communities to engage in environmental research and management.

It is essential that program design is based in monitoring objectives [18]. Selection of sampling and DNA extraction method, qPCR thermocycler, reagents, and assay must be informed by context-specific considerations such as relevant limits of detection, risk of inhibition, and data interpretation. A project aimed at early detection of invasive species may only assess results for the presence or absence of the target but would necessitate reliable detection of the target sequence at low concentrations while greater variability at higher copy numbers may be acceptable, requiring a sufficiently low limit of detection and validation that partners’ reliability in capturing low copy numbers falls within a predetermined acceptable range. In contrast, greater emphasis may be placed on the need for precise quantification of higher concentration samples and a greater limit of detection may be acceptable when assessing FIB levels.

Method 1611 includes a number of quality control measures to be conducted prior to implementation of the protocol and on a regular, ongoing basis to assess competency in carrying out the protocol as well as method, reagent, and equipment performance [11]. It is also recommended that such analyses be performed each time a novel location is sampled to evaluate matrix-specific impacts on method performance [11]. Similarly, validation and analysis of quality control metrics of CBqM partner results is crucial both initially and throughout the program. Acceptance criteria for partner-generated data should be determined in advance and should be used to assess CBM partner proficiency and adequacy of training prior to allowing partners to begin working independently, as well as to continually monitor adherence to protocols and method performance.

Validation of CBqM methods is crucial prior to implementation. CBqM programs should be sure to assess efficiency of chosen DNA-extraction methods in a realistic application: when performed by partners in the field on sample matrices intended to be monitored. Each new assay introduced to a CBqM program should be assessed for variability when conducted both by the central, overseeing laboratory and by community partners as it has been demonstrated that repeatability and reproducibility of results within the same lab and when compared across laboratories may differ from one assay to the next [18,41]. Additionally, protocol performance should be validated whenever a new sampling location is added as sample matrix and location may impact the relationship between results of different methods (such as a CBqM method and gold standard method) and the level of variability within a method [11,17]. These sources of potential variation and their impact on method reliability must be understood and considered in the application of CBqM methods and the interpretation of resulting data.

We see CBqM strengthening water monitoring and management in four primary ways. The first is in supporting traditional, centralized programs by directing monitoring efforts. Shifting the burden of sentinel monitoring to a CBqM network would free up resources, enabling more focused testing by central programs and the concentration of efforts where most valuable. Widespread monitoring could be accomplished by community partners, using an online reporting system to communicate results to a central body in real time. A threshold value for partner results, either for a single sample or a rolling mean, could be established, exceedance of which would trigger targeted, confirmatory sampling and analysis by gold standard methods. CBqM data could also be used to prioritize locations for more in-depth monitoring, for example by identifying those with highly variable water quality [15,16] which might then be subjected to more frequent centralized testing by gold standard methods or MST. Second is the application of CBqM to enhance the geographic scope and number of monitoring locations or frequency of sampling and analysis to address gaps in monitoring and data. Community partnerships facilitate increased program capacity in regions which may lack routine or long-term monitoring, such as water bodies in remote areas [57]. Expansion of monitoring efforts in underserved communities and understudied waters through CBqM would promote sustainable local management and provide a more comprehensive understanding of water quality and its implications for human and ecological health. Third is the potential to increase and diversify the number of monitoring targets. Once a program has been established, any number of additional qPCR assays can be incorporated, such as those used for MST to elucidate contributors to poor water quality, monitoring for the presence of aquatic invasive species, or the detection of antimicrobial resistance genes in environmental waters. This flexibility enables the modification of CBqM programs to respond to evolving risks and to support varied research areas. Furthermore, CBqM programs may be tailored to address location-specific concerns which might not be feasible for a central program with finite resources to prioritize. Lastly is the ability to archive samples collected and analyzed through CBqM for future research, developing a repository of environmental samples for retroactive assessment of biological targets.

Conclusion

This study supports and expands on the findings of prior studies [18,34,35], demonstrating that community partners, with appropriate training and a central source of standardized protocols and reagents, are able to produce reliable qPCR data for biological indicators of water quality. Community-based methods engage and empower communities to be active participants, or indeed leaders, in scientific research and environmental management. Community-based qPCR monitoring stands to support and expand traditional monitoring programs for any number of biological targets, from FIB to invasive species, addressing data gaps and facilitating informed decision making for the protection of public health and sustainable water management.

Supporting information

S1 Text. The protocol utilized by CBqM Partners for qPCR analysis of recreational water

samples at the satellite laboratory in this study.

https://doi.org/10.1371/journal.pwat.0000309.s001

(DOCX)

S1 Data. Raw data file displaying the year of sample collection, sample identifier, and Laboratory, Partner, and Method 1611 qPCR results for each sample analyzed in this study.

Values for both the average sequence copy number detected (indicating the total number of target sequences detected in each reaction averaged across the two replicates for each sample) and the extraction efficiency-adjusted cell equivalents per 100mL are shown for both Laboratory and Partner results.

https://doi.org/10.1371/journal.pwat.0000309.s002

(XLSX)

References

  1. 1. Wade TJ, Calderon RL, Sams E, Beach M, Brenner KP, Williams AH, et al. Rapidly measured indicators of recreational water quality are predictive of swimming-associated gastrointestinal illness. Environ Health Perspect. 2006;114(1):24–8. pmid:16393653
  2. 2. Wade TJ, Calderon RL, Brenner KP, Sams E, Beach M, Haugland R, et al. High sensitivity of children to swimming-associated gastrointestinal illness: results using a rapid assay of recreational water quality. Epidemiology. 2008;19(3):375–83. pmid:18379427
  3. 3. Health Canada. Guidelines for Canadian Recreational Water Quality: Indicators of fecal contamination [Internet]. 2023. Available from: https://www.canada.ca/content/dam/hc-sc/documents/services/publications/healthy-living/recreational-water-quality-guidelines-indicators-fecal-contamination/recreational-water-quality-guidelines-indicators-fecal-contamination.pdf
  4. 4. U.S. EPA. Recreational Water Quality Criteria [Internet]. U.S. EPA; 2012. Available from: https://www.epa.gov/sites/default/files/2015-10/documents/rwqc2012.pdf
  5. 5. Wymer LJ, Brenner KP, Martinson JW, Stutts WR, Schaub SA, Dufour AP. The EMPACT Beaches Project: Results from a Study on Microbiological Monitoring in Recreational Waters [Internet]. U.S. EPA; 2005 Aug [cited 2024 Apr 17]. Report No.: EPA 600/R-04/023. Available from: https://cfpub.epa.gov/si/si_public_record_report.cfm?Lab=NERL&dirEntryId=309397
  6. 6. Dorevitch S, Shrestha A, DeFlorio-Barker S, Breitenbach C, Heimler I. Monitoring urban beaches with qPCR vs. culture measures of fecal indicator bacteria: Implications for public notification. Environ Health. 2017;16(1):45. pmid:28499453
  7. 7. Boehm A, Grant S, Kim J, Mowbray S, McGee C, Clark C, et al. Decadal and shorter period variability of surf zone water quality at Huntington Beach, California. Environ Sci Technol. 2002;36(18):3885–92.
  8. 8. Kim J, Grant S. Public mis-notification of coastal water quality: a probabilistic evaluation of posting errors at Huntington Beach, California. Environ Sci Technol. 2004;38(9):2497–504.
  9. 9. Saleem F, Schellhorn HE, Simhon A, Edge TA. Same-day Enterococcus qPCR results of recreational water quality at two Toronto beaches provide added public health protection and reduced beach days lost. Can J Public Health. 2023;114(4):676–87. pmid:37069453
  10. 10. Wade TJ, Sams E, Brenner KP, Haugland R, Chern E, Beach M, et al. Rapidly measured indicators of recreational water quality and swimming-associated illness at marine beaches: a prospective cohort study. Environ Health. 2010;9(1):66.
  11. 11. U.S. EPA. Method 1611: Enterococci in Water by TaqMan® Quantitative Polymerase Chain Reaction (qPCR) Assay [Internet]. 2012. Available from: https://www.epa.gov/sites/default/files/2015-08/documents/method_1611_2012.pdf
  12. 12. U.S. EPA. Method 1609: Enterococci in Water by TaqMan® Quantitative Polymerase Chain Reaction (qPCR) with Internal Amplification Control (IAC) Assay [Internet]. 2013. Available from: https://www.epa.gov/sites/default/files/2015-08/documents/method_1609_2013.pdf
  13. 13. Alberta Health. Alberta Safe Beach Protocol [Internet]. 2019. Available from: https://open.alberta.ca/dataset/71f0b5ea-b295-4677-afc6-0905641f0694/resource/372d1058-9c90-4da6-a56e-98395dad4a59/download/alberta-safe-beach-protocol.pdf
  14. 14. Alberta Health. Alberta Safe Beach Protocol. 2022 Edition. [Internet]. 2022. Available from: https://open.alberta.ca/dataset/71f0b5ea-b295-4677-afc6-0905641f0694/resource/c1451587-4af3-4595-95a6-2375d77fba20/download/health-alberta-safe-beach-protocol-2022-05.pdf
  15. 15. Shrestha A, Dorevitch S. Slow adoption of rapid testing: beach monitoring and notification using qPCR. J Microbiol Methods. 2020;174:105947.
  16. 16. Griffith JF, Weisberg SB. Challenges in implementing new technology for beach water quality monitoring: lessons from a California demonstration project. Mar Technol Soc J. 2011;45(2):65–73.
  17. 17. Fernández-Baca C, Spirito C, Bae J, Szegletes Z, Barott N, Sausele D. Rapid qPCR-based water quality monitoring in New York State recreational waters. Front Water. 2021;3.
  18. 18. Rudko SP, Reimink RL, Peter B, White J, Hanington PC. Democratizing water monitoring: Implementation of a community-based qPCR monitoring program for recreational water hazards. PLoS One. 2020;15(5):e0229701. pmid:32401786
  19. 19. Thomas AC, Tank S, Nguyen PL, Ponce J, Sinnesael M, Goldberg CS. A system for rapid eDNA detection of aquatic invasive species. Environmental DNA. 2019;2(3):261–70.
  20. 20. Nguyen PL, Sudheesh PS, Thomas AC, Sinnesael M, Haman K, Cain KD. Rapid Detection and Monitoring of Flavobacterium psychrophilum in Water by Using a Handheld, Field-Portable Quantitative PCR System. J Aquat Anim Health. 2018;30(4):302–11. pmid:30269364
  21. 21. Health Canada. Guidelines for Canadian Recreational Water Quality Technical Document - Microbiological Sampling and Analysis [Internet]. 2024 [cited 2024 Apr 25]. Available from: https://www.canada.ca/en/services/health/publications/healthy-living/guidelines-canadian-recreational-water-quality-technical-document-microbiological-sampling-analysis.html
  22. 22. Sharpe A, Conrad C. Community based ecological monitoring in Nova Scotia: challenges and opportunities. Environ Monit Assess. 2006;113(1–3):395–409. pmid:16502035
  23. 23. Theobald E, Ettinger A, Burgess H, DeBey L, Schmidt N, Froehlich H, et al. Global change and local solutions: tapping the unrealized potential of citizen science for biodiversity research. Biol Conserv. 2015;181:236–44.
  24. 24. Pocock MJO, Tweddle JC, Savage J, Robinson LD, Roy HE. The diversity and evolution of ecological and environmental citizen science. PLoS One. 2017;12(4):e0172579. pmid:28369087
  25. 25. SoIB. State of India’s Birds, 2023: Range, trends, and conservation status. [Internet]. The SoIB Partnership; 2023. Available from:
  26. 26. Sidorova E, Virla L. Community-based environmental monitoring (CBEM) for meaningful incorporation of indigenous and local knowledge within the context of the Canadian northern corridor program. Sch Public Policy Publ. 2022;15(15).
  27. 27. Stepenuck K, Green L. Individual- and community-level impacts of volunteer environmental monitoring: a synthesis of peer-reviewed literature. Ecol Soc. 2015;20(3).
  28. 28. Pollock RM, Whitelaw GS. Community-Based Monitoring in Support of Local Sustainability. Local Environment. 2005;10(3):211–28.
  29. 29. Wilson NJ, Mutter E, Inkster J, Satterfield T. Community-based monitoring as the practice of Indigenous governance: A case study of Indigenous-led water quality monitoring in the Yukon River Basin. J Environ Manage. 2018;210:290–8.
  30. 30. Kouril D, Furgal C, Whillans T. Trends and key elements in community-based monitoring: a systematic review of the literature with an emphasis on Arctic and Subarctic regions. Environ Rev. 2016;24(2):151–63.
  31. 31. Ruppen D, Chituri OA, Meck ML, Pfenninger N, Wehrli B. Community-Based Monitoring Detects Sources and Risks of Mining-Related Water Pollution in Zimbabwe. Front Environ Sci. 2021;9.
  32. 32. Peeters ETHM, Gerritsen AAM, Seelen LMS, Begheyn M, Rienks F, Teurlincx S. Monitoring biological water quality by volunteers complements professional assessments. PLoS One. 2022;17(2):e0263899. pmid:35213583
  33. 33. Biggs J, Ewald N, Valentini A, Gaboriaud C, Dejean T, Griffiths R, et al. Using edna to develop a national citizen science-based monitoring programme for the great crested newt (Triturus cristatus). Biol Conserv. 2015;183:19–28.
  34. 34. Tøttrup A, Svenningsen L, Rytter M, Lillemark M, Møller P, Knudsen S. Citizens in the lab: performance and validation of edna results. Citiz Sci Theory Pract. 2021;6(1):1–12.
  35. 35. Knudsen SW, Hesselsøe M, Rytter M, Lillemark MR, Tøttrup AP, Rahbek C, et al. Detection of environmental DNA from amphibians in Northern Europe applied in citizen science. Environmental DNA. 2023;5(6):1429–48.
  36. 36. Alberta Health Services. How to Collect a Grab Enterococcus Beach Water Sample [Internet]. 2022 [cited 2024 Apr 26]. Available from: https://www.albertahealthservices.ca/assets/wf/eph/wf-eph-grab-enterococcus.pdf
  37. 37. Alberta Health Services. How to Collect Cyanobacteria (Blue-Green Algae) Water Samples [Internet]. 2022 [cited 2024 Apr 26]. Available from: https://www.albertahealthservices.ca/assets/wf/eph/wf-eph-cyanobacteria-sample.pdf
  38. 38. Rudko SP, Reimink RL, Froelich K, Gordy MA, Blankespoor CL, Hanington PC. Use of qPCR-Based Cercariometry to Assess Swimmer’s Itch in Recreational Lakes. Ecohealth. 2018;15(4):827–39. pmid:30120669
  39. 39. U.S. EPA. Method 1611.1: Enterococci in Water by TaqMan® Quantitative Polymerase Chain Reaction (qPCR) [Internet]. 2015. Available from: https://www.epa.gov/sites/default/files/2015-08/documents/method_1611-1-enterococcus_2015.pdf
  40. 40. Wilrich C, Wilrich P-T. Estimation of the POD function and the LOD of a qualitative microbiological measurement method. J AOAC Int. 2009;92(6):1763–72. pmid:20166595
  41. 41. Ebentier DL, Hanley KT, Cao Y, Badgley BD, Boehm AB, Ervin JS, et al. Evaluation of the repeatability and reproducibility of a suite of qPCR-based microbial source tracking methods. Water Res. 2013;47(18):6839–48.
  42. 42. Rudko SP. Development and Implementation of a Community Based qPCR Monitoring Program for Biological Hazards of Recreational Water [Internet]. University of Alberta, School of Public Health; 2020 [cited 2024 Apr 17]. Available from: https://doi-org.login.ezproxy.library.ualberta.ca/10.7939/r3-6qtj-m847
  43. 43. Wang Y-S, Dai T-M, Tian H, Wan F-H, Zhang G-F. Comparative analysis of eight DNA extraction methods for molecular research in mealybugs. PLoS One. 2019;14(12):e0226818. pmid:31891602
  44. 44. Mumy K, Findlay R. Convenient determination of DNA extraction efficiency using an external DNA recovery standard and quantitative-competitive PCR. J Microbiol Methods. 2004;57(2):259–68.
  45. 45. Bletz MC, Rebollar EA, Harris RN. Differential efficiency among DNA extraction methods influences detection of the amphibian pathogen Batrachochytrium dendrobatidis. Dis Aquat Organ. 2015;113(1):1–8. pmid:25667331
  46. 46. Karstens L, Siddiqui NY, Zaza T, Barstad A, Amundsen CL, Sysoeva TA. Benchmarking DNA isolation kits used in analyses of the urinary microbiome. Sci Rep. 2021;11:6186.
  47. 47. Djurhuus A, Port J, Closek CJ, Yamahara KM, Romero-Maraccini O, Walz KR, et al. Evaluation of Filtration and DNA Extraction Methods for Environmental DNA Biodiversity Assessments across Multiple Trophic Levels. Front Mar Sci. 2017;4. https://www.frontiersin.org/articles/10.3389/fmars.2017.00314
  48. 48. Barry D, Veillard M, James C, Brummelhuis L, Pila E, Turnbull A, et al. qPCR-based environmental monitoring of Myxobolus cerebralis and phylogenetic analysis of its tubificid hosts in Alberta, Canada. Dis Aquat Organ. 2021;145:119–37.
  49. 49. Haugland RA, Siefring SC, Wymer LJ, Brenner KP, Dufour AP. Comparison of enterococcus measurements in freshwater at two recreational beaches by quantitative polymerase chain reaction and membrane filter culture analysis. Water Res. 2005;39(4):559–68.
  50. 50. Galla G, Praeg N, Rzehak T, Sprecher E, Colla F, Seeber J, et al. Comparison of DNA extraction methods on different sample matrices within the same terrestrial ecosystem. Sci Rep. 2024;14(1):8715. pmid:38622248
  51. 51. Haugland RA, Siefring SD, Varma M, Dufour AP, Brenner KP, Wade TJ, et al. Standardization of enterococci density estimates by EPA qPCR methods and comparison of beach action value exceedances in river waters with culture methods. J Microbiol Methods. 2014;105:59–66. pmid:25038459
  52. 52. Bustin SA, Benes V, Garson JA, Hellemans J, Huggett J, Kubista M, et al. The MIQE guidelines: minimum information for publication of quantitative real-time PCR experiments. Clin Chem. 2009;55(4):611–22. pmid:19246619
  53. 53. Shanks OC, Sivaganesan M, Peed L, Kelty CA, Blackwood AD, Greene MR, et al. Interlaboratory comparison of real-time pcr protocols for quantification of general fecal indicator bacteria. Environ Sci Technol. 2012;46(2):945–53.
  54. 54. Qiagen. DNeasy Blood & Tissue Handbook. 2023.
    • 55. Haugland RA, Siefring S, Varma M, Oshima KH, Sivaganesan M, Cao Y, et al. Multi-laboratory survey of qPCR enterococci analysis method performance in U.S. coastal and inland surface waters. J Microbiol Methods. 2016;123:114–25.
    • 56. Myles P, Cui J. Using the Bland–Altman method to measure agreement with repeated measures. Br J Anaesth. 2007;99(3):309–11.
    • 57. Canadian Polar Commission. State of environmental monitoring in Northern Canada [Internet]. Ottawa; 2015 [cited 2024 May 23]. Available from: publications.gc.ca/pub?id=9.808248&sl=0