2.1.1. Research goal.

Categorization of studies by their research goal is an important first step to understanding whether data from these studies are comparable [23]. For example, if the analysis of aggregated data is focused on large-scale assessment of biodiversity of the fish assemblage, it will be necessary to retain only datasets that were collected with a goal to sample the full assemblage, rather than datasets that targeted individual species (e.g., only sampling Atlantic salmon Salmo salar) (Table 1). Even if datasets are focused on the same ecosystem components (e.g., aspects of the ecosystem that are monitored, including biotic assemblages or abiotic measurements), differences in research goals among datasets may introduce variability in spatial and temporal scales, site selection and integrity, measured parameters (i.e., the aspects of an ecosystem component that are actually measured/collected or portion of an assemblage sampled), sample collection effort and/or processing methods, and potentially data integrity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of major challenges and considerations in analysis of aggregated freshwater data from different sources, specifically related to the research goal(s), spatial scale, temporal scale, site selection, site integrity, and sampling effort of data sources.

https://doi.org/10.1371/journal.pwat.0000502.t001

2.1.2. Spatial scale.

Differences in the spatial coverage of datasets often reflect contrasts in sampling effort [23] (Table 1). For example, spatially restricted investigations may have higher sample site density [27] or may allow resources to be allocated to repeated sampling over time (refer to sec. 2.1.3). These forms of spatially restricted studies with higher sample effort can increase the detectability of species or patterns that may be missed by large spatially extensive surveys [28] and may lead to higher and more accurate estimates of diversity or population size over the sampled area [27,29].

Spatially extensive sampling may not allow for the same concentrated sampling effort across space and time due to the resource constraints associated with sampling a larger geographic area. Consequently, spatially extensive sampling data may not be able to detect seasonal or inter-annual trends or episodic perturbations; though they may sample more regions, habitats, and environmental conditions, thereby increasing the number of species detected [28]. Spatially extensive sampling designs also have the potential to estimate species distributions, though care should be taken not to treat lack of presence at a site as a true species absence because of the high risk of a type II error [30].

Within-site variation in spatial scale can also be an issue because comparisons of data that were collected in different microhabitats may not be valid for all analytical purposes [31]. Abiotic and biotic conditions may vary between microhabitats, potentially confounding the patterns of interest [32,33]. Furthermore, community studies that focus on a larger spatial scale within a site (i.e., sampling across multiple microhabitats) are expected to have higher diversity than those that focus on a single microhabitat [29]. Stratification of samples by habitat type may be necessary to control for this variability (Fig 1).

2.1.3. Temporal scale.

Temporal aspects of datasets that may affect comparability in analysis of aggregated data include the years and seasons of sampling and the time between sampling events (for repeated sampling; Table 1). When comparing across regions, the temporal scope of sample collection should be considered. Data collected during the same time frame at multiple locations will prevent confusion over whether differences are due to space or time. This is particularly relevant due to ongoing climate change contributing to rapid shifts in freshwater ecosystems [34–36]. Samples collected at the same locations over decades may consequently differ solely due to natural inter-annual variability and/or the influence of changing climate. Caution must therefore be used when comparing data from different regions that were collected over very different time frames because spatial differences could be confounded by temporal changes.

The season in which sampling occurs plays an important role in determining the natural range of variability in habitat conditions (Table 1), including water chemistry, temperature, snow/ice cover, and hydrology [16,37,38]. For several biotic components, seasonal abundance and composition relate to phenological traits that are driven by habitat conditions such as temperature and flow, including the timing of aquatic insect emergence and spawning and migration of fish [39]. For example, the timing of sampling in relation to insect life cycles and emergence periods can result in different dominant taxa and decreased capture of early instars, and samples of benthic macroinvertebrates should be collected in the same season(s) to be comparable among sites or among years [31]. If the full annual cycle is not the focus of research questions, analysis of aggregated biotic or abiotic data should apply a seasonal classification to ensure natural seasonality and population cycles do not introduce variability into the analysis [16] (Fig 1).

Sampling frequency and how it relates to the expected rate of change in measured ecosystem components must also be considered [27,40]. Supporting variables, which are environmental measurements, such as water chemistry that are used to relate to or better understand variability in biotic measurements, can vary over short time scales, including diurnal cycles for some parameters [40,41]. Impacts of perturbation may be evident over different temporal scales depending on whether it is a lentic or lotic system and depending on water residence time [42]. While seasonal spot measurements may be effective for characterizing water chemistry, albeit with errors [40], spot measurements taken once per year may not be an accurate reflection of habitat conditions, particularly for parameters that are highly affected by flow and lake thermal stratification [42].

Algal communities, though less variable than water chemistry, can change over a matter of days or weeks and will thus integrate environmental changes over a relatively short period of time [43,44]. In contrast, benthic macroinvertebrates and fish will integrate environmental change over longer periods of time [45] and therefore require less frequent sampling, although sampling of fish must recognize differences in habitat use, life history, and catchability at different times [46]. The differences in rates of change for different organism groups and environmental variables influence the level of sampling effort (including temporal frequency) that is required to effectively characterize sites or answer research questions, and they also have implications for the temporal variability captured by the data. Therefore, single-sampling events for water chemistry data provide a less time-integrated measure of condition than single-sampling events for benthic macroinvertebrate or fish. Repeated sampling can detect seasonal and inter-annual trends and increase the chances of detection of episodic perturbation in organism groups or supporting variables with a fast turnover rate. Repeated sampling can also be used in hierarchical modelling frameworks that yield species detection probabilities for individuals in communities [e.g., multi-species occupancy models; 47–49].

2.1.4. Sample selection.

Prior to any analysis of aggregated data, it is important to determine the questions and hypotheses to be addressed, as this will act as a guide for the types of data that are needed [19]. For example, will the analysis investigate genetic diversity [e.g., 50], intraspecific diversity [e.g., 51], richness [i.e., taxonomic diversity, e.g., 22], or ecosystem diversity [e.g., 52]? Datasets from multiple studies may apply or may be devoid of the information necessary to address a particular question. If the study aims to examine the population-level metrics of diversity, then it could be necessary for the collected datasets to include abundance, age, sex, or size information. Conversely, if the study focuses on species richness, then incidence or abundance information would likely suffice.

Considerations for sample selection in analyses of aggregated data require controlling variability that is unrelated to the questions or hypotheses (Table 1). The ability to detect patterns and differences in response variables is dependent on both the magnitude of response and the amount of variability in the data [23]. If the combined data come from sites that differ naturally in terms of climate, ecoregion, underlying geology, system size, or other potentially confounding factors, then variability in the response to the stressor in question may be obscured or covary with multiple environmental gradients. For example, underlying geology should be taken into consideration when selecting diatom and benthic macroinvertebrate samples or analyzing data because geology plays a large role in determining driving factors like water and sediment chemistry (Fig 1) [53–55].

Due to the need to control for confounding factors and ensure similar sampling approaches, metadata (i.e., information about the data) and supporting data must offer sufficient detail to provide the context for the data and indicate comparability [56–58]. Metadata should include detailed site and habitat descriptions as well as specifics of sampling and sample processing methodology, and information about who collected the data [56,57]. Furthermore, data on relevant supporting variables that characterize sample habitats and seasons (e.g., flow, physical habitat measurements such as depth and width, substrate composition, and water chemistry, if not already included as part of the main sampling design) make it possible to control for confounding factors.

2.1.5. Site integrity.

Site integrity indicates to what extent sites have been impacted by natural or anthropogenic stressors, thereby providing information about comparability of sites (Table 1). For example, a site in an area of resource development with active construction likely differs from unimpacted sites with respect to water chemistry, sediment load, physical habitat, and biotic community structure [59, 60]. Selection of samples for analysis therefore requires that sites can at minimum be classified as unimpacted (i.e., representing reference conditions) or impacted to ensure that human disturbance does not confound the assessment if it is not the focus of the research question. If possible, more detailed descriptions of site integrity may be provided by classifying sites in a hierarchy of reference conditions, such as natural condition, minimally impacted, least disturbed, or best attainable [61,62]. In addition to site integrity classification, information about potential stressors should ideally be present in the metadata, ensuring that analysis is focused on sites/samples with a similar level of integrity.

Although not directly a measure of site integrity, it is also necessary to consider whether sampled water bodies are natural or artificial (e.g., canals, reservoirs, and other human-made water bodies). This has implications for aspects of habitat such as channel structure (e.g., tortuosity), flow, water body shape, and habitat complexity, all of which may contribute to differences in biotic and abiotic ecosystem components [63,64].

2.1.6. Sampling effort.

Sampling effort depends on factors such as the number of sample sites or reaches within a water body (often affected by ecosystem size), the number of replicate samples collected at a site during a sampling event, the length of time spent during a sampling event, the different sampling methods used at a site, and the temporal frequency of sampling events (Table 1). Analysis of aggregated data must control for these differences in sampling effort because they naturally yield different results [65]. Sampling effort can be affected by the spatial and temporal extent of sampling [23] and often differs among datasets.

Approaches to control for sampling effort in analysis of aggregated data depend on the extent of differences, the sampled parameters, and the measures of interest. For example, it may be necessary to stratify the data and omit data for sites that include sampling across multiple seasons while most other sites were sampled in only one or two seasons (Fig 1). If multiple sampling methods were used to collect taxa, it may be necessary to select a subset of data based on sampling methods that are common among studies in the aggregated data. Alternatively, use of integrated community models or other hierarchical frameworks (e.g., joint species distribution models) may allow for analyses that simultaneously model multiple data sources for multiple species [66,67]. In biotic studies, there are specific approaches to control for differences in sampling effort when estimating taxonomic richness. Sampling effort can be standardized for estimates of taxonomic richness through rarefaction [29,68] when the number of sample sites differs among studies or regions [e.g., refer to [21] and other papers in the same issue]. Also, abundance can be reported as catch per unit effort (CPUE) to control for differences in time or area sampled [69]. When these measures are not appropriate or possible, scaling the data (to a mean of 0 and standard deviation of 1) or converting counts/abundance to incidence data (presence = 1, absence = 0) allows for the comparison of general trends across sites/studies while controlling for differences in magnitude. Options that are specific to particular freshwater ecosystem components or taxonomic groups, whether to control for differences in sampling or sample processing, are discussed in more detail in the next section.

2.2.1. Measured parameters.

Many challenges related to measured parameters for biotic sampling involve studies that focus on different subsets of an assemblage (Table 2). For example, analyses that aim to estimate taxonomic diversity should include datasets on the full assemblage of species within a particular organism group rather than data from studies that target specific or limited taxa [e.g., species-targeted sampling of fish; 70]. This is also a challenge for measures of abundance, biomass, and habitat use or occupancy by species. Similar issues exist when sampling equipment or study goals (e.g., a focus on certain size fractions) do not allow for the identification and enumeration of all species, which is a common concern for data on zooplankton [e.g., exclusion of rotifers; 71], phytoplankton [e.g., exclusion of picoplankton and nanoplankton; 72], and benthic algae [e.g., focus on only diatoms; 73]. To avoid introducing analytical bias in all biotic components, only samples that have measured a similar portion/group of the assemblage should be used for the analysis (Fig 1).

Water and sediment chemistry data can provide important supporting information to understand habitat conditions in freshwaters, but data cannot be compared if the same parameters have not been measured for all samples [74] (Table 2). Virro et al. [75] compared five different water quality databases that ranged from national to global scale and found variable degrees of overlap in parameters across databases. For nutrient data, such differences in comparability among studies depend upon whether elemental/molecular forms, organic or inorganic forms, and dissolved and particulate phases have been measured [74]. Similarly, the comparison of metal concentrations in water can include dissolved, total, or free metals, and data may not always be clear on the sampled fraction. Incomplete or incorrect metadata, including missing or unclear reporting of units, combined with ambiguous terminology can further contribute to a lack of clarity regarding the comparability of water chemistry data [57,74]. Such issues highlight the need for collection of a standard suite of water chemistry parameters to ensure comparability, and the need for detailed information about which parameters, fractions, and units are being reported. Some data repositories attempt to circumvent such issues through the use of a strict data schema that must be followed when providing data, including both required fields and controlled vocabulary [e.g., the Gordon Foundation DataStream in Canada; 76]. These efforts facilitate greater comparability between datasets.

2.2.2. Sampled habitat.

Lake limnologic zone (littoral, pelagic, profundal), stream morphology (riffle, run, pool), tortuosity, depth, water body size, and connectivity are relevant characteristics of the habitat that was chosen for sampling because each can affect biotic composition and environmental characteristics [77] (Table 2). The distinction between lake limnologic zones is one of the most important considerations for analyses of aggregated lake biotic data. Habitat conditions in the shallow littoral zone are vastly different from those in the deeper profundal zone, resulting in different communities. For example, whereas littoral zones may often be oxygenated, with high primary productivity and a large variability of microhabitats (e.g., different size substrata, water plants, wave action), profundal zones of lakes generally have soft sediments, with low (or no) primary productivity and potentially low oxygen or anoxic conditions below the light-compensation level [77,78]. Benthic macroinvertebrate assemblages of soft sediments in the profundal zone therefore tend to be dominated by Diptera and worms that are adapted to low oxygen conditions, whereas those in the littoral zone hold a higher diversity [79] that is more similar to those of stream habitats [22]. Zooplankton assemblages are usually more diverse in littoral than in pelagic zones; however, most emphasis has been on sampling of pelagic habitats [80]. Fish are known to rely on energy resources in the littoral zone [81], but some species may shift to a reliance on pelagic food sources in the summer when zooplankton abundance is high [82]. As a result of differences in habitat use, some studies have focused on characterising fish assemblages in a single zone (e.g., only littoral or only pelagic) to circumvent the difficulties associated with trying to efficiently sample across all lake habitats [46]. When conducting analysis of aggregated data for lake biotic communities, it may therefore be important to stratify the data based on the sampled habitat (Fig 1). Furthermore, for pelagic data, it may be important to have information about lake mixing regimes and thermal stratification in relation to sampling depth because this may affect composition of biotic samples [83].

Many water chemistry parameters vary strongly along the vertical profile of a lake, including temperature, oxygen, and nutrients [77]. Therefore, it is important to check the metadata to determine how/where the samples were collected, from which depth(s), and the sample position relative to thermally stratified layers, if applicable [57,58,84]. For example, water samples may be collected from just below the surface, as discrete samples collected from a specific depth(s), or as composite samples that integrate water over multiple depths in part of or the entire water column, which can influence the comparability of the data [85]. The location at which samples were collected (e.g., closer to the shore or in mid-lake) can also impact results [85] and should be taken into consideration. In rivers and streams, representative samples should be collected from the thalweg (deepest and fastest flowing location) or another area where water is flowing and mixing because results for water chemistry parameters might be affected by sampling in stagnant waters or pools that are not well mixed. Metadata for water chemistry samples should be used to subset based on the collection depth.

2.2.3. Sample collection and processing methods.

Differences in gear function, mesh (pore) size, effort, and reported units of sampling lead to differences in either the capture of organisms (e.g., missing small size classes of zooplankton) or differences in reporting (e.g., individuals/unit or biomass) [23,72,86,87] (Table 2). It may be necessary to omit samples collected with very different methods or to conduct a qualitative assessment (e.g., presence/absence) rather than a quantitative assessment (e.g., abundance, density) if methods are not comparable [21]. For example, fish are often collected in lakes using multiple sampling methods because methods are size- or habitat-specific and will differ in effectiveness at capturing different species/sizes of fish [46], which complicates the harmonisation of data. In locations where replication of sampling with a single gear is sufficient to determine species detection probabilities, the addition of data from supplemental gear types can be used in hierarchical frameworks to improve understanding of community dynamics and processes that structure assemblages [67,88]. The conversion of fish data collected with different methods to presence/absence may be used to account for variability in abundance estimates, but even here, caution should be used when estimating diversity across studies, particularly for methods that are biased toward particular taxa or sizes [89]. For microscopic organisms, samples should be selected for analysis based on whether collection methods were quantitative or qualitative and based on comparable mesh/filter size. Some mesh sizes of zooplankton nets exclude small organisms (e.g., rotifers), and this may result in lower diversity and abundance estimates [71,87]. Larger mesh sizes may also exclude young zooplankton (e.g., nauplii and early copepodites) [87]. For phytoplankton, the exclusion of small size classes (e.g., picoplankton) due to filter pore size may lead to reduced estimates of diversity, biomass, and primary productivity [72]. It is important to ensure all samples combined for analysis are effectively sampling the same portion of the assemblage to avoid introducing sampling bias (Fig 1).

Different sample processing methods can introduce bias if they vary with respect to sub-sampling, enumeration, and species identification (Table 2). For example, benthic macroinvertebrate samples may be subsampled or fully enumerated, and identification may be at different taxonomic levels, including family, genus, or lowest practical taxonomic unit (species or coarser, as possible) [31,90]. Where taxonomic level differs, taxonomic harmonisation will be necessary, potentially by using a standardized list of taxa or applying a coarser taxonomic resolution and dropping some taxa that are redundant with finer resolution data [91]. Microscopic counting procedures for zooplankton can affect abundance and density estimates [71]. Similarly, information on how colonies are enumerated in benthic algae data is helpful to understand potential biases in abundance estimates. For both phytoplankton and benthic algae, it may be necessary to harmonise data on (relative) abundance/density/biomass if these data are reported in different ways in different studies. However, due to the level of expertise required to identify algal taxa, caution is necessary to ensure confidence in taxonomic identification because differences among taxonomists can have a large effect on sample comparability [92,93]. For all organism groups, information on sample processing must be discerned from the metadata to ensure comparable data are retained for analysis.

Ambiguities can also be introduced into analysis of aggregated water chemistry data when information about sample filtration (whether it was done, and with what pore size) is lacking in the metadata [74], and this can result in increasing the unexplained variance in the analysis [75]. The issue of differences in processing methods, equipment, and sensitivity levels among laboratories is also difficult to control in analysis of aggregated chemistry samples [58]. In some cases, issues that arise due to among-laboratory differences can be handled and harmonized by simple conversion of units and values [75]. When lower detection limits or reporting limits differ among samples, decisions must be made for how to deal with these discrepancies [58]. Simple removal of non-detects (values below detection limit or reporting limit) biases the data towards higher values [94]. Common practice is to replace non-detects with other values (for example, some fraction of the detection limit), but this may have a particularly strong effect on the results and may introduce erroneous patterns into the data [94]. Statistical techniques such as maximum likelihood estimates offer a potential solution that may provide less biased options for replacing non-detects [94], but these approaches should still be used with caution when detection limits vary across studies [58].

2.2.4. Data integrity.

Missing or ambiguous metadata is a common data concern that affects the comparability and integrity of data [56,58,74] (Table 2). For example, in their review of 25 million records of nutrient data from water chemistry samples, Sprague, Oelsner and Argue [74] found that over half had missing or ambiguous metadata that compromised the user’s ability to analyse the data. The minimum detection limit or reporting limit for water chemistry and sediment chemistry analyses are required to determine whether low values are comparable among studies, but this information is not consistently reported [58]. Furthermore, chemistry data may be provided after some pre-treatment of the data (e.g., adjustment of data below detection limit/reporting limit), and it is critical that the information about the chosen pre-treatment be provided with the data [58]. In the absence of the required metadata, it is often necessary to omit the data to avoid making assumptions about the missing/ambiguous information (Fig 1).

Nomenclature issues are a common concern for data integrity of both abiotic and biotic data (Table 2). Sprague, Oelsner and Argue [74] found that the 10 most common nutrient parameters reported in water chemistry data from the USA were reported with 1046 different parameter names, of which 115 could not be clearly assigned to a single parameter. The fact that a single parameter can be labelled and reported with more than 100 different names (e.g., nitrite + nitrate was reported with 130 different names), leads to issues both in data harmonisation and in the potential for ambiguity and error. The same is true for biotic data [74,91]. Fish are often known by a wide variety of common or regional/local names [95] which complicates joining information across large spatial scales or among regions. This is particularly problematic if the same name is used across regions/communities to refer to different genotypes, or if a single fish species is known by multiple common names [96].

When combining data collected across a wide range of years, further effort must be made to ensure that nomenclature for biotic data is recently updated. Changes to taxonomic nomenclature occur as new species are discovered and existing species are re-classified (e.g., taxonomic groups are split or joined, or taxa are moved to different taxonomic groups) [91]. An early step in an analysis of aggregated biotic data should therefore be to examine the taxonomic list and ensure that all taxonomic names are current and updated [91]. Across broader regions (e.g., when comparing data between continents), it may be necessary to consider whether regional naming conventions exist and require additional harmonisation steps. For example, there are thousands of diatom species [97], and nomenclature is rapidly changing. Harmonization of nomenclature requires expert review of taxonomic lists and should make use of accepted taxonomic backbones, such as the Freshwater Animal Diversity Assessment (FADA) project [98]. New tools are emerging to simplify this process [91].