River rehabilitation aims at alleviating negative effects of human impacts such as loss of biodiversity and reduction of ecosystem services. Such interventions entail difficult trade-offs between different ecological and often socio-economic objectives. Multi-Criteria Decision Analysis (MCDA) is a very suitable approach that helps assessing the current ecological state and prioritizing river rehabilitation measures in a standardized way, based on stakeholder or expert preferences. Applications of MCDA in river rehabilitation projects are often simplified, i.e. using a limited number of objectives and indicators, assuming linear value functions, aggregating individual indicator assessments additively, and/or assuming risk neutrality of experts. Here, we demonstrate an implementation of MCDA expert preference assessments to river rehabilitation and provide ample material for other applications. To test whether the above simplifications reflect common expert opinion, we carried out very detailed interviews with five river ecologists and a hydraulic engineer. We defined essential objectives and measurable quality indicators (attributes), elicited the experts´ preferences for objectives on a standardized scale (value functions) and their risk attitude, and identified suitable aggregation methods. The experts recommended an extensive objectives hierarchy including between 54 and 93 essential objectives and between 37 to 61 essential attributes. For 81% of these, they defined non-linear value functions and in 76% recommended multiplicative aggregation. The experts were risk averse or risk prone (but never risk neutral), depending on the current ecological state of the river, and the experts´ personal importance of objectives. We conclude that the four commonly applied simplifications clearly do not reflect the opinion of river rehabilitation experts. The optimal level of model complexity, however, remains highly case-study specific depending on data and resource availability, the context, and the complexity of the decision problem.
Citation: Langhans SD, Lienert J (2016) Four Common Simplifications of Multi-Criteria Decision Analysis do not hold for River Rehabilitation. PLoS ONE 11(3): e0150695. doi:10.1371/journal.pone.0150695
Editor: Igor Linkov, US Army Engineer Research and Development Center, UNITED STATES
Received: August 23, 2015; Accepted: February 18, 2016; Published: March 8, 2016
Copyright: © 2016 Langhans, Lienert. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The publication of this article was funded by the Open Access Fund of the Leibniz Association. Simone D. Langhans was supported by the Alexander von Humboldt-Foundation through a fellowship for experienced researchers. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Due to human impact, freshwaters are among the most seriously threatened and modified environments on the planet . Habitat destruction and fragmentation, physical and chemical alterations, and the introduction of exotic species or overexploitation of freshwater resources, have severely altered the ecology of freshwaters . During the next decades, the pressure on freshwater ecosystems is expected to increase, driven by the combined effects of population growth and climate change .
In response to the poor conservation status of freshwater biodiversity and the increasing loss of services that humans receive from freshwater ecosystems, river rehabilitation is a global priority [3, 4]. The success of a rehabilitation that aims at improving the overall ecological state of a degraded river system depends on concurrently achieving quality targets for biological, chemical, and physical elements. The valuation of these elements, however, is case-specific and therefore difficult to base on empirical data, which is often sketchy. To satisfy this complexity, river rehabilitation often relies on the knowledge of experts, who deal with the assessment of and the trade-offs between the quality elements. Additionally, socio-economic considerations might be included , in which case the assessment also depends on the preferences of stakeholders or the public. However, socio-economic preference elicitation from stakeholders is not the focus of this paper; for a more extensive discussion see .
Expert knowledge has gained momentum as a source of information for decision making, particularly in contexts where empirical information is sparse or unobtainable [7–9]. In such situations, previous learning and experience provide valuable insight into the behavior of environmental systems . Multi-Criteria Decision Analysis (MCDA; [11–14]) is naturally suited to incorporate expert knowledge through value functions. These are expert preferences for objectives (see explanation in Table 1) on a standardized scale. MCDA helps to structure and facilitate complex projects with multi-stakeholder involvement, which is typical for river rehabilitation. During the last two decades, MCDA has been increasingly adopted to support decision making in the environmental field [15, 16]. However, in river rehabilitation it is still rarely applied. For example, a recent meta-analysis found that of 60 river rehabilitation projects 26 would benefit from MCDA, but only four actually used it .
Terminologies used in Multi-Criteria Decision Analysis (MCDA; see also Table 1 in ).
In this paper, we focus on Multi-Attribute Value and Utility Theory (MAVT/MAUT), two methods within the MCDA family that are grounded on axioms of rational choice (e.g. [12, 13]). These methods can easily deal with a large number of alternatives (Table 1) and new or compromise alternatives can be included at any stage of the decision process . This has been found to be important in the case of conflicting interests in river rehabilitations in Switzerland . MAVT and MAUT can also consider uncertainty of predictions inherent in environmental data, and MAUT includes the risk attitude of the decision makers, i.e., their preferences in face of uncertain outcomes (e.g. [6, 12, 13]. MAVT/MAUT allow assessing different objectives targeted by river rehabilitation, e.g. reaching a good physical, chemical or biological state of the river, and aggregating these assessments into an overall score. A value function represents the degree of fulfilment of the overall objective on a scale from zero to unity as a function of scientifically measurable system properties, the attributes. To facilitate the construction of such a value function, the overall objective is broken down into objectives at lower hierarchical levels. The value function for the overall objective is constructed by formulating individual value functions for each lowest-level objective, as a function of a number of attributes, and aggregating the values to the next-higher levels. Value functions representing expert preferences for ecological objectives and their aggregation may be elicited from scratch following a pre-defined, rigorous protocol to avoid biases; as demonstrated in this paper. To decrease elicitations costs, value functions can also be translated from existing assessment protocols. Thereby, the assessments of the individual physical, chemical, or biological elements defined in the protocols, be it in a qualitative or quantitative way, are standardized onto a common scale from 0 to 1 (please see [20, 21] for detailed information on this procedure). If an outcome of a decision, such as the consequences of implementing a rehabilitation measure is uncertain, the experts’ preferences can change. In this case, risk attitudes have to be elicited additionally to value functions, to transform the values to utilities.
Experts should be selected in a way to cover all aspects of the respective MCDA. If stakeholders are included, this can be done with the help of a stakeholder or social network analysis . For river rehabilitation that aims at improving the ecological state, different experts concerning the functioning of river ecosystems such as ecologists, hydraulic engineers, water chemists, or physicists should be involved.
MCDA-applications often use a relatively pragmatic approach to elicit expert knowledge, because quantifying preferences is a laborious and time-consuming process. The most frequent simplification in on-the-ground river rehabilitation is to include only a limited number of objectives and attributes. We found a restriction to some five to ten objectives or attributes in MCDA-approaches [17, 19, 23–26] and non-MCDA approaches that strived to set standards for successful river rehabilitation .
Another simplification is assuming that all value functions are linear , thus circumventing their laborious elicitation [19, 24]. Although intuitively compelling, linear value functions may not reflect the preferences of decision makers. To give an example, for an attribute ranging from 0 to 100 units (e.g. diversity index), linearity implies that an improvement from 0 to 50 units is equally valuable as an improvement from 50 to 100 units. In reality, experts may value the improvement of a heavily degraded river higher than of one which is already in a good condition (or vice versa) resulting in non-linear value functions.
Additivity (i.e. weighted averaging) is often assumed when aggregating values of different objectives [17, 19, 24], despite its use being problematic. Most importantly, this is due to the fact that additivity entails the assumption of preferential independence of objectives, i.e. that preferences for the level of one objective can be specified independently of the levels of other objectives [12, 13]. The additive model also implies that a low value of one objective can be compensated by a high value of another one. As an example: if fish are in a very bad state, the additive model implies that the bad state of fish can be fully compensated by benthic organisms being in a very good state (given that both receive equal weights). The result is that at the next higher level of the objectives hierarchy, biodiversity will be in a medium state when the two sub-objectives fish and benthic organisms are aggregated. However, experts may not want to allow for any compensation or for only some, which is possible for example by using minimum or multiplicative aggregation, respectively (S4 File).
Finally in the face of uncertain (rehabilitation) outcomes, it is necessary to consider the risk attitude of decision makers. The risk attitude is needed to convert value functions into utility functions . When eliciting utility functions, decision makers have to express preferences for probabilistic outcomes, typically presented as lotteries which is more demanding than asking for value functions . Thus, in many applications, risk neutrality is assumed with the implication that the shape of the value function is identical to that of the utility function. Giving again an example: risk neutrality implies that a decision maker is indifferent between two options A and B. In option A, the ecological state of the river is 50% for sure. In option B there is uncertainty, and there is a 50% probability that the river is in a very bad state, and a 50% probability that it is in a very good state. In real-world applications, many decision makers might prefer the sure outcome A to the uncertain one B, implying risk aversion . This risk attitude, be it risk prone or risk averse, depends on an individual person’s preferences. Mathematically, it implies a convex or concave shift in the shape of the value function, respectively .
The main aim of this study was to investigate whether the simplifications discussed above, i.e. concerning the number of objectives and attributes, the shape of the value functions, the choice of the aggregation method, and the assumption of risk neutrality reflect expert opinion, specifically for the evaluation of the success of ecological river rehabilitation. For this, we base our study on intensive and very detailed face-to-face interviews with river rehabilitation experts. We hypothesized that 1) if given the choice, experts prefer to include a broad range of objectives and attributes, 2) most of the elicited value functions will be non-linear, 3) experts prefer other aggregation methods than additivity, and 4) the risk attitude of the experts will vary. Additionally, we demonstrate in detail, how an expert-interview-based MCDA-approach can be implemented and provide ample (supplementary) material for other applications. Based on our findings, we aim to formulate recommendations for the elicitation of expert knowledge, particularly in the context of river rehabilitation.
2. Material and Methods
2.1 Interview procedure
We started the knowledge elicitation process with a fish ecologist (hereafter called Fish-expert), a biologist/physicist (BioPhys-expert) and a river ecologist (BioA; Table 2). These experts suggested additional experts needed to cover the whole range of objectives that are usually targeted in small-scale river rehabilitation: two river ecologists with different expertise (BioB, and BioC) and a hydraulic engineer (Phys). This method of expert identification is often referred to as snowball sampling . All experts had proven expertise in river rehabilitation.
Expert nickname (Exp.), main research fields of expert and approx. years of experience (Exper.), the objectives for which each expert provided information (Elicitation), the method to elicit weights (Weight), and whether the expert’s risk attitude was elicited (Risk).
We carried out two series of face-to-face interviews with each of the six experts. Face-to-face interviews are among the most likely to produce high-quality responses [33, 34] compared to remote (online) surveys or group methods. The main aim of the first interview series was to find the minimum set of objectives and attributes to adequately assess the potential ecological improvement after the implementation of a river management action. To facilitate these interviews, we adapted an existing objectives hierarchy, which was constructed previously for the support of river rehabilitation [5, 19, 24]. In the second interview series, we elicited the value functions, the weights of the objectives, the additional scaling factor if experts chose multiplicative (and not additive) aggregation, and the experts’ attitude towards risk. Each interview lasted between two to five hours, was recorded, transcribed, and sent to the experts for feedback.
We followed a rigorous interview protocol including a structured, documented, and systematic protocol for the quantification of expert knowledge (S4 File). It is well known in decision analysis that the elicitation of expert knowledge is prone to subjective biases; with the use of such a protocol, we countered these, including the splitting bias, framing, overconfidence, anchoring, halo effects, dominance, or linguistic uncertainty (e.g. [35, 36–39]).
Compared to unstructured questions, structured elicitation procedures produce more reliable and better-calibrated estimates of facts (e.g. ), and thus presumably also of preferences.
2.2 Ethics statement
Five of the six experts are active scientists themselves that publish in the field; one expert now works in practice, but has a scientific background (Phys; Table 2). All experts were fully aware of the fact that the information from these interviews would be used to construct an assessment procedure for river rehabilitation, with the aim of a scientific publication. They fully agreed with this, and they received a detailed information package about the content before each interview.
After each interview, the experts received the transcribed interview protocols, which were corrected according to their written feedback, and they agreed in their Email responses that we could use the information to construct an assessment procedure for river rehabilitation. We did not regard it as necessary to pass this study about the quantification of expert opinion (also see e.g.  and Bayesian approaches) through an ethics committee beforehand, since we did not ask the experts for any sort of personal, medical, or psychological information. To protect the expert’s anonymity, we do not disclose their names or affiliations (Table 2).
2.3 Study site
Assessing the ecological state of a river, including the selection of objectives and attributes as well as their evaluation, strongly depends on the river type. We selected the lower reaches of the Wigger as a study site to test the elicitation simplifications, because it represents a common river type in Switzerland and is well studied (e.g. ; see S6 File). The Wigger is a 41 km long, gravel-bed river located in the Swiss Midlands with a catchment area of 368 km2. It originates at 1300 m a.s.l., flowing through steep and narrow valleys in its upper reaches. The middle and lower reaches are wider and would naturally be of a braided river type, structured by gravel bars. However, large parts of the lower reaches are channelized, cleared from most of the riparian vegetation, and impacted by surrounding agriculture. The Wigger is a typical trout brook characterized by brown trout (Salmo trutta), barbel (Barbus barbus), nase (Chondrostoma nasus), spirlin (Albumoides bipunctatus), and stone loach (Barbatula barbatula).
2.4 Objectives and attributes
The procedure to identify the objectives and attributes consisted of four steps. In each step, we moved from higher-level to lower-level objectives to avoid the well-known splitting bias (e.g. [18, 41, 42]). We asked the experts i) to come up with necessary objectives and attributes to include, ii) to decide on the final objectives and attributes (now also considering the objectives and attributes elicited in a previous study ) and iii) to classify each chosen attribute as “essential”, “very valuable”, or “desirable”. To consolidate the information from all experts, we built the final objectives hierarchy including only those objectives and attributes that were classified as “essential” by at least one expert. All objectives were chosen adhering to the formal requirements of decision theory including comprehensive coverage of the decision problem, being fundamental, complete, concise, and simple [11–13].
The objectives may not necessarily seem non-ambiguous in the sense that they may be difficult to understand for lay persons. However, since the objectives hierarchy in our example was built for expert use, based on expert knowledge in the respective field, this requirement is fulfilled. Hence, we assumed that any biodiversity-expert would understand the goal of having a good fish status and would agree to measure this goal with the suggested attributes. To validate that the proposal of an expert would also match the opinion of another, we confronted each expert with the objectives and attributes identified as essential by one or several of the other experts. Hence, this cross-validation allowed us to be sure that all objectives were well covered by the sub-objectives and attributes and that the attributes were unambiguous and defined in sufficient detail to be easily understood by any other expert in the field. However, in case of strong disagreement, we kept the proposition of each expert, resulting in an individual objectives hierarchy for each expert (S3 File).
Although decision theory also asks for non-redundancy, we allowed for (partly) redundant lower-level objectives and especially attributes. In the context of environmental management this is favourable [6, 21], and probably a rather unique property of environmental decision support compared to other application fields. In any case where an objectives hierarchy is used, an objective is divided into lower-level sub-objectives, whose values are aggregated to the next higher level. If redundant sub-objectives are available, averaging across these data may likely be beneficial: It decreases the uncertainty associated with environmental assessment and the measurement procedure of any environmental indicator and therewith increases statistical significance. Additionally, due to the redundancy of sub-objectives, an assessment can even be performed if not all values are available which is often the case in real-world applications. In this case, simply the weights for the remaining sub-objectives need to be renormalized to prevent biases. To exemplify this: there are several indices available with which we can assess the status of macroinvertebrates in rivers. These redundant indices should give similar assessment results. Averaging these results will provide a value with higher statistical significance than using a single result of only one of the indices. Hence, considering redundant lower-level objectives accounts for the facts that ecosystem valuation is typically based on measurable ecological attributes, that it is uncertain, and also that data are often scarce.
2.5 Value functions
In the second interview, we first presented the consolidated objectives hierarchy and adapted objectives and attributes if the expert did not agree with them. Then, we asked each expert for possible measurement methods of the attributes, the measurement units, and the range of each attribute from the worst- to the best-possible state considering a near-natural state of our study river. Based on this information, we elicited the value functions using the bisection method ([11, 12]; Table 1). Each expert elicited the value functions that corresponded with her/his competences (Table 2). Examples of typical questions in an interview to ask for preferences are given in S7 File.
2.6 Aggregation methods
We first explained hierarchical aggregation to the experts: objectives at the lowest level of the objectives hierarchy are combined and build the basis for the aggregation at each next-higher level. We then introduced three different methods including additive aggregation, minimum aggregation (also called one-out, all-out; ), and multiplicative aggregation (see also S4 File). We chose additive and minimum aggregation as both are widely applied in ecological assessments , and multiplicative aggregation which requires weaker preference independence conditions [12, 13, 31] and therefore is a popular alternative to the additive model. We discussed the implications of the three different aggregations for the assessment outcomes  and asked the experts to choose a method for each branch of the objectives hierarchy. Depending on the method, we had to elicit different parameters: For additive aggregation [12, 13, 31], we had to identify value functions and weights [that sum up to one), for minimum aggregation value functions, and for the multiplicative aggregation value functions, weights and an additional scaling constant, the synergy factor k (see below).
2.6.1. Additive aggregation.
If the weights are equal for all elements (wi = 1/n), the result is identical to the (unweighted) arithmetic mean which is often referred to as unweighted averaging. In decision science, the weighted arithmetic mean is called additive aggregation, which is by far the most widely used aggregation function for multi-criteria decision support [12, 13].
2.6.2. Minimum aggregation.
The minimum aggregation method indicates a pessimistic attitude that may even hold the potential of a pessimism bias , since the aggregated value only reflects the worst sub-objective and does not account for an improvement of a better-performing sub-objective.
2.6.3. Multiplicative aggregation.
Despite requiring weaker preference independence, multiplicative aggregation is rarely applied in practice (but see e.g. [46–51]), which could be due to the complexity of the model including the necessity of identifying an additional parameter (the synergy factor k): (3)
In our application, this synergy factor could take one of four values including 0.25 (large synergy effect of several attributes having a rather good state), 0.5 (medium synergy), 0.75 (small synergy), and 1 (no synergy effect = additive aggregation), which we assessed directly from the experts  with the help of an illustration (Fig 1 and Figures A and B in S4 File). Multiplicative aggregation penalizes options with extremely divergent attribute scores (i.e. some attributes being in a poor state, and some in a high state), while attributes with similar, medium level-outcomes are assessed superiorly (see discussion).
Diagram used in the second interviews to discuss implications of multiplicative aggregation using the example of four objectives with equal weights. These objectives can achieve different values between 0 (worst-possible state) and 1 (best-possible state). Solid line: all objectives are increased from a value of 0 to 1 together. Dashed line: value of only one objective at a time is increased from 0 to 1, i.e. this is done for each objective one after the other, and resulting values are summed up. A) No synergy effect (same as additive aggregation), B) small synergy effect of having all objectives on a similarly good level, C) medium synergy effect, and D) large synergy effect. B-D) A better overall value is achieved if all objectives are increased together (solid lines) than if only one individual objective at a time is improved, but not the others (dashed lines).
2.7 Scaling constants: weights and synergy factor
With each expert, we identified the weights for those lowest-level objectives for which the expert was able to give value functions as well as for all higher-level objectives. To assess the weights, we used the Swing, Reversed-Swing, or the direct rating method (Table 1). Because we were working with scientists who understood the implications, we assessed some of the scaling constants directly although direct rating is not generally recommended (but it is a recommended procedure by ). Whenever experts chose multiplicative aggregation, we elicited a synergy factor to define the importance of synergistic effects, i.e. when several objectives are improved simultaneously (Fig 1 and S1 File). For those cases where several experts proposed different weights or different synergy factors for the same objective, we calculated the median over all contributing experts. To exclude unwanted objectives from the assessment, experts were allowed to assign a weight of 0.
2.8 Risk attitudes
We only asked about the experts´ risk attitude on the highest-level objective (i.e. "high level of ecological integrity"). Identifying risk attitudes on the lower hierarchical levels requires expertise in the respective branches. However our experts had “only” deep knowledge of a few branches, why we assumed that they would agree with the risk attitude of their fellow expert who does have the required expertise. We elicited the risk attitude with the bisection version of the variable certainty equivalent method (e.g. ) from three of the six experts, including the BioA-, BioB-, and BioPhys-expert (Table 2). The risk attitude is needed to translate the values of the state of the river (between 0 = worst state and 1 = best state) to utilities. In a rehabilitation project, one would determine whether the ranking of the possible rehabilitation actions changes when using values or utilities, i.e. without or with risk attitude. If the ranking of the best-performing action(s) does not change, there is no need to go through the extra effort to identify risk attitudes.
To assess the risk attitude, we presented 50–50 lotteries to the experts and asked for the respective certainty equivalent (CE). In our case, the CE is that ecological state of our study-river (defined as a value between 0 and 1) the expert would be willing to accept instead of the presented lottery. First, we asked for the CE x0.5. The CE x0.5 is the state of the river (given as a value between 0 and 1) that the expert would equally likely accept as a 50–50 lottery (xmin, 0.5; xmax, 0.5). Thereby, the 50–50 lottery represents a 50% chance of the river being in its worst state (value = 0) and a 50% chance of being in its best state (value = 1). A risk neutral expert is indifferent between the 50–50 lottery and the river being in a medium state for sure (value = 0.5; corresponding to a linear utility function), and a risk averse expert between the lottery and the river being in a poorer state for sure (i.e. CE < 0.5; for risk prone CE > 0.5). Following the elicitation of the CE x0.5 we asked for x0.75 and x0.25. We carried out consistency checks for some intermediate points (see interview example in S7 File).
3.1 Objectives and attributes
The final objectives hierarchy integrates the objectives and attributes that the experts defined individually as essential to assess a change in the ecological river status after the implementation of a rehabilitation action along the lower reaches of our study stream. Together, the experts identified 102 essential objectives: three on level 1, seven on level 2, 18 on level 3, 51 on level 4, and 23 on level 5 (Fig 2). An individual objectives hierarchy for each expert is given in the S3 File.
This hierarchy shows all the objectives (arranged at levels 1 to 5) and respective measurable system attributes that were considered to be essential by the experts to assess whether a good "ecological state" (main objective) was reached after rehabilitation. Objectives and attributes were identified for the lower, braided reach of the river Wigger in Switzerland, and may therefore change when considering a different river type (abbreviations see Table 3 and S3 Table).
Five of the six experts agreed to quantify the main objective on the basis of the three objectives "good physical", "good chemical", and "good biological state" (level 1 of the objectives hierarchy, Fig 2). On the consecutive levels, these experts (Fish, BioB, BioC, BioPhys, and Phys) included 75, 78, 89, 76, and 90 essential objectives, respectively. The BioA-expert included only one objective, namely the "biological state" on the first level, and 53 objectives on the levels 3 to 6.
All experts together identified a total of 70 essential attributes to quantify the lowest-level objectives (level 5, Fig 2 and Table 3). Thereof, the Fish-expert included 50, BioA 37, BioB 51, BioC 60, Phys 61, and the BioPhys-expert 49 attributes in their individual objectives hierarchies (S3 File).
Explanation, units, and ranges (worst possible to best possible level) of the 70 essential attributes included in the objectives hierarchy of Fig 2. Reference river refers to the lower reach of the Wigger. L: attribute and value function from the literature; BA: BioA-expert; BB: BioB-expert; BC: BioC-expert BP; BioPhys-expert; F: fish-expert; P: Phys-expert (see Table 2; more explanations see S3 Table).
3.2 Value functions
We established value functions for 69 of the 70 total attributes that were assessed to be essential. The experts provided value functions for 65 of them (S3 File), and we extracted another four from the literature (numbers 2; , 11 , 18 , and 28 [54, 55]; Table 3, S2 File, S2 Table). Neither the experts nor the literature could help us finding a value function for "mean suspended solid concentration at low discharge" (number 30, Fig 2). Since we explicitly excluded the entire branch "good water quality", we did not identify attributes and value functions for this objective.
For each of the eight attributes "flowamp", "floodbed", "refug", "shorelength", "tributar", "softw", "gravel", and "periph", two or three experts gave a value function resulting in a total of 80 value functions (see S1 Table for our suggestion of value functions for a generalized model). The Fish-expert also gave value functions and weights for the mid-reaches and headwaters of the reference river (S2 Table), additionally to the lower reaches presented here. This information is accessible in the Supporting Information, but we do not further discuss it.
Of the 76 individual value functions from the experts, 74 were continuous (two were discrete) and could therefore be analysed for their shape: 19% were linear and 81% were shaped differently (Fig 3 and S5 File).
We show different exemplary value functions for the attributes (A) substrate clogging (n° 18; discrete; Table 3) and (B—D) shoreline length per river length (n° 38; continuous). The BioPhys-expert assessed shoreline length as a linear value function (B), whereas the BioPhys- and the BioC-experts opted for a non-linear one (C and D). x-axis: attribute scale from worst-possible (left) to best-possible state (right). y-axis: value scale, where the attribute level is translated to a neutral value between 0 (worst-possible state) to 1 (best-possible state) with help of the value function. Color-coding according to Swiss Modular Concept of stream assessment in five quality classes: bad, unsatisfactory, moderate, good, and very good (http://www.modul-stufen-konzept.ch/index_EN).
3.3 Aggregation methods
In 76% of all aggregations (37 cases), the experts preferred multiplicative aggregation. For the remaining twelve cases, they choose additivity (Table 4 and S1 File). The Phys-expert preferred the additive model throughout the entire objectives hierarchy. On the highest-level of the hierarchy, two experts (Fish and Phys) averaged the objectives "physical", "chemical", and "biological state", while two others preferred multiplicative aggregation (BioB and BioPhys; Table 4). BioA did not require aggregation at the highest level, opting to base the entire assessment on the biological branch alone.
Weights for the objectives on level 1 "physical state", "chemical state", and "biological state", preferred aggregation method (mult = multiplicative, add = additive), and synergy factor elicited from five experts (Fish, BioA, BioB, BioPhys, and Phys). The summary shows the median, minimum, and maximum weight for all of the experts. Symbol Ø: no aggregation required, since the full weight of 1 was given to the "biological state".
In the biological branch, only one out of 26 aggregations was defined to be additive (mean density of ground and rove beetles; BioA; Table C in S1 File). In the chemical branch, two aggregations were additive ("physico-chemical state" and "water quality"; "total suspended solids", "mean suspended solids concentration at low flow" and "suspended solids deposition in floodplain"; both by the Phys-expert; Table B in S1 File). In the physical branch, seven out of the 14 aggregations were additive, including five assigned by the Phys-expert (aggregation of the three lower-level objectives of "physical state"; the four lower-level objectives of "morphology and hydraulics"; the three lower-level objectives of "discharge"; "bed-moving flood" and "floodplain flooding"; and the three lower-level objectives of "connectivity") and our own assumption for aggregating "barrier height" and "no power stations" (Table A in S1 File).
3.4 Scaling constants: weights and synergy factor
We assessed the weights and the synergy factors from at least one, but mostly from more experts with two exceptions: Due to the lack of expert input, we assumed equal weights for the objectives "barrier height" and "no hydropower stations" (Table A in S1 File), and the three objectives "total suspended solids", "suspended solids deposition in floodplain", and "mean suspended solids concentration at low flow" (Table B in S1 File).
We found large variation in weights (min—max ≥ 0.4) elicited for the same objective from different experts in 12 of 85 cases (14%; Table 4 and S1 File). The highest-level weights of the physical, chemical, and biological state covered the whole range from 0 to 1 (Table 4). Four of five experts weighted the biological state (median: 0.45) more than the physical (median: 0.28) and the chemical state (median: 0.23). Contrarily, the Phys-expert weighted the "chemical state" the highest (Table 4). The BioA-expert discarded the chemical and the physical state from the objectives hierarchy, assigning weights of 0 to those objectives.
Weights of the second and lower-level objectives also varied between 0 and 1 with a median weight of 0.31 for the objectives in the physical branch, of 0.20 the chemical branch, and of 0.32 in the biological branch (S1 File).
Also in the lower branches, experts sometimes assigned 0 weights; namely to 34 objectives, therewith excluding them from the assessment: to seven in the physical, three in the chemical, and to 24 in the biological branch (S1 File). For instance in the physical branch, the BioPhys-expert assigned a weight of 0 to the objectives "deposition/erosion", "substrate diversity" and "channel geometry", and "substrate clogging" and "substrate armoring" (Table A in S1 File). Likewise in the chemical branch, "total suspended solids" and "mean suspended solids concentration" (BioPhys and Phys, respectively; Table B in S1 File), and in the biological branch "floodplain vegetation", "benthic organisms", and "shoreline fauna" (Fish; Table C in S1 File) received a weight of 0 from some experts. Hence, weights for these objectives (among some others) differed most (min—max ≥ 0.4). Additionally, there were relatively large weight differences between 0.2 and 0.4 in seven of 29 cases in the physical branch (24%; Table A in S1 File) and in ten of 39 cases (26%) in the biological branch (Table C in S1 File).
There were much larger variations between the experts concerning the additional scaling constant, the synergy factor, which is needed for multiplicative aggregation: In 11 of 27 cases (41%; Table 4 and S1 File), this variation was very large (min—max ≥ 0.4). For aggregating the highest-level objectives (physical, chemical, and biological state), experts chose synergy factors between 0.25 and 1 (median: 0.75; Table 4). For the second and lower-level synergy factors, for the physical branch, experts assigned the largest differences in the synergy effect to the objectives "morphology & hydraulics", "discharge", and "connectivity" at the second level, to the lower-level objectives of "discharge" and "connectivity", and to the lower-level objectives of "flood dynamics". For the biological branch, the largest differences arose from experts preferring small or large synergy effects when aggregating for instance "ecosystem function" and "biodiversity" on the second level, or the different benthic functional feeding groups, and "respiration" and "production" on level five (Table C in S1 File).
3.5 Risk attitudes
The three experts BioA, BioB, and BioPhys showed a similar trend in their risk attitudes, i.e. slight risk aversion in the case of both a poor (value = 0.25) and a good ecological state of the river (value = 0.75) (Fig 4 and S4 Table). If the river was considered to be in a moderate state (value = 0.5), BioB and BioPhys were slightly risk averse, while BioA was slightly risk prone (utility = 0.6; slightly convex function; S4 Table).
We showed that four commonly applied simplifications in the process of quantifying expert knowledge for MCDA-river rehabilitation did not reflect the opinion of most of our experienced experts, i.e. including only a small number of objectives and attributes and assuming linear value functions, additive aggregation, and risk neutrality. The experts recommended assessing a rehabilitated river with an extensive objectives hierarchy including between 54 and 93 essential objectives, which they decided to aggregate multiplicatively in 37 of the 49 cases (76%). They further proposed between 37 to 61 essential attributes for 81% of which they gave non-linear value functions. We also demonstrated that the experts were either risk averse or risk prone (but never risk neutral) depending on the ecological state of the river (for higher level objectives), and the experts’ personal importance of objectives (at lower levels). Considering these results, we conclude that the sensitivity of these four simplifications on the outcome should either be tested in each case, or that these simplifications should be avoided in preference elicitation if their effect on the assessment outcome is unknown.
4.1 Objectives and attributes
Despite the fact that the experts reduced the number of initially selected objectives and attributes (keeping only the essential ones), most of the individual objectives hierarchies remained large (S3 File). The smallest objectives hierarchy was defined by the BioA-expert, who included only biological objectives (Figure B in S3 File). Large objectives hierarchies have two main disadvantages. First, they are time-consuming to develop. However, such an effort may be legitimate if an objectives hierarchy is used repeatedly, for example within a national program. Second, a large hierarchy appears complex and the evaluation in an MCDA may be very time consuming. Therefore, practitioners may be discouraged from applying it. However, as already mentioned in the methods (Section 2.4), when assessing the success of rehabilitations complexity may be advantageous: Redundant attributes increase the statistical significance of assessment outcomes if more than one of them is measured, and they increase flexibility since different users can select their favorite ones to work with . Additionally, complex objectives hierarchies allow identifying the link between an implemented rehabilitation action (i.e. the cause) and its effect(s) on several levels of objectives, therewith enabling learning for future rehabilitations.
The number of objectives does not necessarily need to be large in any MCDA for river rehabilitation. It may vary considerably depending on the complexity of the main objective that has to be quantified: A more complex main objective, such as for example reaching a good ecological river state will always call for more objectives than a simpler main objective such as increasing the rivers’ attractiveness to excursionists. In such cases, multiple interest groups and not only experts might need to be involved. With them, additional socio-economic objectives may need to be discussed (e.g. ).
4.2 Value functions
Despite linearity being commonly assumed in MCDA-applications , 81% of all the value functions that the experts identified were non-linear. Similarly, in a recent study on water infrastructure planning only 23 (14%) of 172 elicited value functions were linear ; and in a study on pharmaceuticals in hospital wastewater, 22.5% were linear (18 of 80 value functions elicited from 26 stakeholders; [18, 57]). Hence, non-linearity may be a characteristic of value functions in diverse application types. However, in both of these applications, we found that assuming linearity would not have substantially changed most rank-orders of alternatives.
In our study, differences in the shape of value functions evolved due to individual preferences among experts. For example, the BioPhys-expert preferred linear relationships arguing that this is the most conservative approach, least prone to errors and biases. One of the stakeholders in the hospital wastewater case study argued similarly [18, 57]. This view contrasted the opinion of the Fish-expert, who argued that it is more important to improve an impacted river than one that is already in quite a good state. Many of the value functions related to fish objectives are therefore asymptotic curves with a steeper value improvement from a bad to a moderate state than from a moderate to a good state. For example, an increase in the population of Chondrostoma nasus from 0 to 300 individuals is very steep, because according to our fish expert it is important to have at least some individuals present to sustain a population, while the improvement from 300 to 2,000 individuals (300 = x0.5) follows a much flatter curve (Fig 5). Hence, the first few individuals are extremely valuable.
Value function assessed from the fish-expert to describe a natural population of adult nase (Chondrostoma nasus). x-axis: attribute scale from left: worst-possible state (0 individuals) to right: best-possible state (2,000 individuals). y-axis: value scale, where the attribute level is translated to a neutral value between 0 and 1, with help of the value function.
Five of the six experts insisted on directly drawing the value function rather than answering the questions that led to the final curve, arguing that this was easier (exception: BioFish and in some cases BioA and BioB). However, applying a direct rating method is not recommended . To ensure that the shape of the curves represented the preferences of the experts, we posed follow-up questions about the relative preferences using the bisection method (; typical questions see S7 File). In many cases, the curve they had drawn reflected their preferences well. This may be due to the fact that scientists are familiar to the mathematical representation of values. However, since there were exceptions, we strongly recommend following textbook elicitation protocols or if not, carrying out careful consistency checks.
When answering our questions about the shape of value functions, the experts had problems to envisage a corresponding reference condition for some objectives. This is important because the reference defines the upper limit of the attribute (and the worst-possible state the lower limit; ). Switzerland is densely populated and a pristine situation (e.g. meandering river) can rarely be achieved with rehabilitation. Therefore, we defined the reference condition to be the best-possible state that can be achieved with an optimal rehabilitation measure, given existing restrictions such as towns, roads, and agriculture .
4.3 Aggregation methods
Due to the inter-dependencies between objectives, the experts chose multiplicative aggregation in 76% of the cases. On the first level of the objectives hierarchy, two experts (BioB and BioPhys) preferred multiplicative aggregation (Table 4) because the physical, chemical, and biological objectives are strongly connected. Contrarily, two experts (Fish and Phys) argued that these objectives depict largely independent parts of the ecological river system, and therefore favored additive aggregation. As an example, the Fish-expert argued that even in a chemically impaired river, one might not find effects on fish (e.g. changed vitellogenin levels; ). This contrasts the possibility of interactions among objectives on the lower levels of the objectives hierarchy, where the Fish-expert opted for multiplicative aggregation.
The Phys-expert preferred additive aggregation throughout all hierarchical levels and branches, even in cases with strong dependencies between the objectives (and preferential dependency). In principle, the additive mathematical model is incorrect to reflect this interdependency and a multiplicative one should be favored. The Phys-expert was aware of the mathematical background (see methods), however argued that for real-world applications a simple model like additive aggregation is preferable: It is more transparent, and thus less viable to manipulation, which is more important than using a model that better reflects interdependencies of objectives.
From a scientific point of view, the Phys-expert considered a mixed minimum-additive model to presumably be most promising . Such an aggregation would account for the undesired properties of the purely additive and purely minimum methods to allow deficits in some objectives being compensated by good scores in others and to only reflect an improvement of the worst attribute in the aggregated value, respectively (see S4 File).
4.4 Scaling constants: weights and synergy factor
At the highest level of objectives, four of the five experts weighted the biological state higher than the physical or the chemical state (Table 4). They argued that any physical improvement is irrelevant, if it does not translate into an improvement of the biological condition of the river (see European Commission ). Along these lines, BioA suggested that the ecological state of a river could be assessed with biological indicators only (giving zero weight to the chemical and the physical objectives). This view mirrors the assessment scheme adopted by the Water Framework Directive , where a rehabilitation action is considered to be unsuccessful if the biological condition is not improved. Contrarily, the Phys-expert considered reaching a good water chemistry to be most important, arguing that a chemically intact river is the basis for the biological elements reaching a good quality.
Such varying views were not limited to the highest-level objectives. The Fish-expert, for example, gave a weight of one to the objective "fish", being the only expert who considered the three other objectives within this same branch to be unnecessary (Figure A in S3 File). He argued that fish are sensitive and can mirror the condition of other species. Similarly in the "morphology & hydraulics"-branch, the BioPhys-expert gave a weight of one to "flow diversity" therewith ignoring the remaining three objectives, while the Phys-expert evaluated all four objectives (i.e. "flow diversity", "deposition/erosion", "substrate diversity", and "channel geometry") almost equally important (Table A in S1 File).
Two camps of opinion are not uncommon in expert elicitation and may arise from ambiguity in framing the problem or because experts have different backgrounds [60, 61]. In our case, it could be attributed to different perspectives: The BioPhys-expert based his opinion on information from the scientific literature, whereas the Phys-expert, who is involved in the implementation of projects on-ground, emphasized the feasibility of the assessment.
Despite these varying views, in only 14% of all aggregations weights differed substantially among the experts (min—max ≥ 0.4), suggesting that the elicited weights are quite robust. However, this did not apply to the synergy factor, which is needed for multiplicative aggregation: In 41%, the synergy factor varied substantially when assessed by different experts. We speculate that this variation resulted from the fact that multiplicative aggregation was new to all experts and therefore they had no pre-established opinions. It may also indicate that we have to improve the protocol for identifying synergy factors from experts to reach better results. Contrary to the assessment of weights, which is routine in practical decision analysis, assessing the additional scaling constant for multiplicative aggregation is very rarely done in MCDA.
4.5 Risk attitudes
The three experts from whom we elicited the risk attitude at the highest level of the objectives hierarchy were generally moderately risk averse or risk prone, but never risk neutral (Fig 4). For example, the BioA-expert explained that if the river were in a poor state (value = 0.25) the main objective would be to protect the few existing resources and the functionality of the river, and he would not want to take risks on that (i.e. confirming risk aversion). If the river were in a good state (value = 0.75), he would not want to jeopardize this state (again confirming risk aversion). However, at a medium state of the river (value = 0.5), BioA would be willing to trade a higher risk for a possible (but uncertain) higher ecological state of the river, being slightly risk prone in this case. The other two experts expressed similar opinions.
With the BioPhys-expert, we discussed the assumption that the risk attitude can be different for different objectives . Our expert confirmed this by stating that he would be more conservative for those objectives that are most important to him (i.e. that received higher weights) while for less-important ones he would be more risk prone. He thus confirmed that on the highest level of the objectives hierarchy, he would be rather risk averse, resulting in a concave shape of the utility function for "ecological river integrity" (Fig 4). Therefore, the chosen approach to elicit the risk attitude only at the highest level is justified. It is the most conservative approach, because a change of the risk attitude towards being more risk prone would only occur for some less important lower-level objectives.
Finally, we wish to point out that correctly eliciting preferences is always demanding, but eliciting the utility function with lotteries is an especially difficult task. Below, we recommend several ways to deal with this problem.
4.6 The uncertainty in expert knowledge
In our study, value functions, weights, synergy factors, and risk attitudes from different experts (for the same objective) differed from slightly to considerably. This may be due to true differences in the opinion of the experts (see section 4.4), but may also include some systematic bias, such as experts producing systematically optimistic or pessimistic responses (e.g. [36, 37]). Hence, quantifying the uncertainty of expert opinion is an important feature of any elicitation exercise that informs decision makers. It can either be done by quantifying the opinion of multiple experts and combining them mathematically, e.g. by taking a (weighted) group average [63, 64], or by using Bayesian approaches [38, 60, 65, 66]. If only one expert is available, one can ask about the level of precision or certainty about the given weights (best in a face-to-face interview with a feedback cycle; [56, 66]). Thus, we wish to emphasize that large differences in expert judgment, some of which we have observed in our study, are not necessarily bad news. Such differences can even be seen as an advantage of expert elicitation, since they may display different schools of thought in an expert community [60, 61, 67]. For decision making, such varying judgments may best not be combined, but rather used to separately analyze how they affect the assessments .
4.7 Reality check
Our study shows that the four commonly applied MCDA-simplifications do not reflect expert knowledge in the case of river rehabilitation. However, how much expert preferences influence MCDA-outcomes (compared to the simplifications) will be highly case-specific and depend on i) the selection of objectives and attributes, ii) how much the value functions of these objectives deviate from linearity, and iii) the attribute levels measured in the field. Additionally, MCDA-outcomes will be influenced by the choice of the aggregation method, particularly when objectives with significantly different qualities are aggregated . Whether it is worthwhile to carry out an elaborate MCDA to support an application to river rehabilitation, or simply apply the assumptions we have tested, needs to be decided for each case individually. Such a decision may be supported by estimating the relative costs of developing an elaborate MCDA compared to the costs of the rehabilitation implementation. We showcase this idea in the following section for two rehabilitations in Switzerland that differ in river size and complexity regarding the considered objectives, i.e. for the Thur River (large, complex) and the Töss River (mid-size, less complex).
4.7.1 A note on the costs.
The objectives hierarchy and the value functions which we have identified in this study build the basis for developing MCDAs for river rehabilitation in other cases. If additional objectives and value functions need to be considered, information may be derived from the literature [6, 20, 69, 70] or from additional expert judgments where needed. We can reasonably assume that the data to predict the outcomes of different rehabilitation actions (i.e. alternatives), e.g. regarding the ecological state or the costs, are available. However, these data might not be in such a form suitable for MCDA and might thus need additional data collection, for example as expert interviews. Based on these assumptions, we estimate that a consultant with some MCDA-expertise can develop an MCDA for the Töss River within four months, which would cost approximately 101’623 CHF including 8% VAT (salary = 170 CHF/hr; 150 hrs/month; based on official Swiss government recommendations; ). The rehabilitation project along the Töss River, which included widening 200 m of the river course and revegetating the river banks, cost 0.5 Mio CHF . Thus, the costs for developing the MCDA are 20% of the 0.5 Mio investment costs, which is considerable. However for the more complex case, i.e. the Thur River, the costs for developing an MCDA would only be approximately 3–6% of the total costs of the rehabilitation project of 6 Mio CHF (widening 1500 m of the river course plus rehabilitating the river banks), which probably makes the MCDA worthwhile. Hereby, we assumed that a skilled senior consultant (197 CHF/hour incl. 8% VAT; 150 hrs/month; ) works on the case full time for six months (176’707 CHF) to one year (353’415).
Costs of river rehabilitation vary strongly depending on the number of objectives that should be improved (ecological state, flood protection, recreational and aesthetic value) and the magnitude of improvement targeted (e.g. small or large improvement of ecological state), the length and the location of the stretch to be restored (e.g. rehabilitation within cities are more expensive than outside of populated areas due to higher land prices), the chosen measure (e.g. one-sided or two-sided bank rehabilitation etc.; see  for costs of different measures), the stakeholders involved and the magnitude of their (dis-)agreement, the size of the involved river , and the country of the implementation. Also the costs for developing an MCDA vary among countries due to different salary levels: It is therefore impossible to give general cost estimates. However, relative costs of developing an MCDA compared to the implementation costs are most likely in the same order of magnitude. As stated above, whether it is worthwhile to carry out an elaborate MCDA from scratch depends very much on the case itself. Hence we fully agree with : “What exactly is done at each step of an SDM (“Structured Decision Making”) process, to what level of rigor and complexity, will depend on the nature of the decision, the stakes and the resources, and timeline available.” Finally, costs for developing MCDAs in future river rehabilitation projects will decrease considerably, since they will benefit from information gathered in previous applications.
4.7.2 Minimizing costs.
Despite the above remarks, we are aware that eliciting individual preferences from experts, instead of adopting simplifications, may considerably increase the efforts and costs of MCDA-applications. Hence to minimize these costs, we recommend the following:
- Decide whether it is justified to develop a large, comprehensive objectives hierarchy, e.g. if it is repeatedly or extensively used. In other cases, rely on existing objectives hierarchies (e.g. this one for similar rivers as the Wigger) and adapt to the specific case. Ask experts to rank their objectives according to different categories (e.g. essential, very valuable, or desirable) and only keep essential ones to reduce the number of objectives.
- Reduce the effort for determining the entire shape of the value function by asking for only the mid-value point x0.5 [18, 57]. The mid-value point will indicate whether the value function is concave, convex or linear. Use sensitivity analyses to determine whether more-detailed follow-up questions are required. Letting experts/scientists draw the value functions instead of answering questions to identify their preferences will also reduce the elicitation effort. If done so, ensure the accuracy of their shape with consistency checks.
- To ease the difficult elicitation of utilities with lotteries, (a) elicit value functions and transform them to utilities at the highest level of the objectives hierarchy, (b) assume risk aversion as a conservative approach , (c) elicit only value functions but carry out extensive sensitivity analysis to determine whether the risk attitude affects the ranking of alternatives , (d) only roughly determine the risk attitude of the expert which should suffice to allow ordering the alternatives , or (e) elicit risk attitudes for different parts of the highest-level value function and for objectives at different hierarchical levels from only a few experts.
Optimizing river rehabilitation through the prioritization of rehabilitation measures has gained momentum and will become a common tool in river rehabilitation planning . The here presented MCDA-approach is based on a solid, conceptual foundation, which is a crucial aspect of a decision support methodology for environmental management . Practical implementation is another key element, but transferring insights from theory to practice is by no means trivial. In this paper, we document an extensive expert elicitation exercise, and provide ample material for discussion and use in applied river rehabilitation projects (documented in the Supporting Information). Eliciting preferences from experts instead of applying simplified assumptions will improve the representation of ecological expert knowledge and will contribute towards an informed optimization of holistic on-the-ground river rehabilitation. The optimal level of model complexity, however, will always be highly case-study specific depending on data and resource availability, the context, and the complexity of the decision problem.
S1 File. Weights and aggregation.
Information identified from the experts for the A) physical, B) chemical, and C) biological objectives.
S2 File. Value functions translated from the literature.
Information from the literature for the attributes A) “diversity of sediment patches”, B) “frequency of floodplain flooding”, C) “substrate clogging”, and D) for “total suspended solids”, translated into value functions (E).
S3 File. Objectives hierarchies for the individual experts.
Objectives hierarchies identified by the A) Fish-, B) BioA-, C) BioB-, D) BioC-, E) BioPhys-, and the F) Phys-expert.
S4 File. Illustrations used to support the interviews.
A) Aggregation methods for value functions, and B) synergy effect for multiplicative aggregation.
S5 File. Value functions for the different objectives.
S6 File. Description of the reference river, the Wigger (Switzerland).
S7 File. Example of an interview to elicit preferences.
S1 Table. Selection of expert value functions for the generalized model.
S2 Table. Value functions elicited from six river experts.
Identified attribute levels for the worst-possible state (value = 0), for the values 0.1, 0.25, 0.5, 0.75, and 0.9, and the best-possible state (value = 1) are shown.
S3 Table. Comments value functions.
Attribute name, abbreviation, and details concerning the attribute or shape of the value function.
S4 Table. Risk attitudes.
Certainty Equivalents (CE) elicited from the BioA-, BioB-, and BioPhys-expert.
The authors thank all the experts who devoted their time and expertise to participate in this exercise. We highly appreciate the support in data processing of Judith Ellens and the fruitful contributions of Peter Reichert. We also thank the editor and two anonymous reviewers for their comments which considerably improved the manuscript.
Conceived and designed the experiments: JL SDL. Performed the experiments: JL SDL. Analyzed the data: SDL JL. Contributed reagents/materials/analysis tools: JL SDL. Wrote the paper: SDL JL.
- 1. Sala OE, Chapin FS, Armesto JJ, Berlow E, Bloomfield J, Dirzo R, et al. Biodiversity—Global biodiversity scenarios for the year 2100. Science. 2000;287: 1770–4. pmid:10710299
- 2. Collares-Pereira MJ, Cowx IG. The role of catchment scale environmental management in freshwater fish conservation. Fisheries Manag Ecol. 2004;11: 303–12.
- 3. Bates BC, Kundzewicz ZW, Wu S, Palutikof JP. Climate Change and Water. Technical Paper of the Intergovernmental Panel on Climate Change. Geneva, Switzerland: IPCC Secretariat; 2008.
- 4. Roni P, Hanson K, Beechie T. Global review of the physical and biological effectiveness of stream habitat rehabilitation techniques. North Am J Fish Mana. 2008;28: 856–90.
- 5. Reichert P, Borsuk M, Hostmann M, Schweizer S, Sporri C, Tockner K, et al. Concepts of decision support for river rehabilitation. Environ Modell Softw. 2007;22: 188–201.
- 6. Reichert P, Langhans SD, Lienert J, Schuwirth N. The conceptual foundation of environmental decision support. J Environ Manage. 2015;154: 316–32. doi: 10.1016/j.jenvman.2015.01.053. pmid:25748599
- 7. Sutherland WJ. Predicting the ecological consequences of environmental change: a review of the methods. J Appl Ecol. 2006;43: 599–616.
- 8. Martin TG, Burgman MA, Fidler F, Kuhnert PM, Low-Choy S, McBride M, et al. Eliciting expert knowledge in conservation science. Conserv Biol. 2012;26: 29–38. doi: 10.1111/j.1523-1739.2011.01806.x. pmid:22280323
- 9. McBride M, Garnett ST, Szabo JK, Burbidge AH, Butchart SHM, Christidis L, et al. Structured elicitation of expert judgments for threatened species assessment: a case study on a continental scale using email. Methods Ecol Evol. 2012;3: 906–20.
- 10. Fazey I, Fazey JA, Salisbury JG, Lindenmayer DB, Dovers S. The nature and role of experiential knowledge for environmental conservation. Environ Conserv. 2006;33: 1–10.
- 11. Belton V, Stewart TS. Multiple Criteria Decision Analysis: An Integrated Approach. Dordrecht, The Netherlands: Kluwer Academic Publishers; 2003.
- 12. Eisenführ F, Weber M, Langer T. Rational Decision Making. Berlin, Heidelberg, New York: Springer; 2010.
- 13. Keeney RL, Raiffa H. Decisions with Multiple Objectives: Preferences and Value Tradeoffs. New York: Wiley; 1976.
- 14. Linkov I, Moberg E. Multi-Criteria Decision Analysis: Environmental Applications and Case Studies: CRC Press; 2011.
- 15. Huang IB, Keisler J, Linkov I. Multi-criteria decision analysis in environmental sciences: Ten years of applications and trends. Sci Total Environ. 2011;409: 3578–94. doi: 10.1016/j.scitotenv.2011.06.022. pmid:21764422
- 16. Keisler J, Linkov I. Environment models and decisions. Environ Syst Decis. 2014;34: 369–72.
- 17. Corsair HJ, Bassmann Ruch J, Zheng PQ, Hobbs BF, Koonce JF. Multicriteria decision analysis of stream restoration: potential and examples. Group Decis Negot. 2009;4: 387–417.
- 18. Schuwirth N, Reichert P, Lienert J. Methodological aspects of multi-criteria decision analysis for policy support: A case study on pharmaceutical removal from hospital wastewater. Eur J Oper Res. 2012;220: 472–83.
- 19. Hostmann M, Borsuk M, Reichert P. Stakeholder values in decision support for river rehabilitation. Large Rivers. 2005;15: 491–505.
- 20. Langhans SD, Lienert J, Schuwirth N, Reichert P. How to make river assessments comparable: A demonstration for hydromorphology. Ecol Indic. 2013;32: 264–75.
- 21. Langhans SD, Schuwirth N, Reichert P. The method matters: guide to indicator aggregation in ecological assessments. Ecol Indic. 2014;45: 494–507.
- 22. Lienert J, Schnetzer F, Ingold K. Stakeholder analysis combined with social network analysis provides fine-grained insights into water infrastructure planning processes. J Environ Manage. 2013;125: 134–48. doi: 10.1016/j.jenvman.2013.03.052. pmid:23660534
- 23. Hermans C, Erickson J, Noordewier T, Sheldon A, Kline M. Collaborative environmental planning in river management: An application of multicriteria decision analysis in the White River Watershed in Vermont. J Environ Manage. 2007;84: 534–54. pmid:17029759
- 24. Hostmann M, Bernauer T, Mosler HJ, Reichert P, Truffer B. Multi-attribute value theory as a framework for conflict resolution in river rehabilitation. Journal of Multi-Criteria Decision Analysis. 2005;13: 91–102.
- 25. Prato T. Multiple-attribute evaluation of ecosystem management for the Missouri River system. Ecol Econ. 2003;45: 297–309.
- 26. Woolsey S, Capelli F, Gonser T, Hoehn E, Hostmann M, Junker B, et al. A strategy to assess river restoration success. Freshwater Biol. 2007;52: 752–69.
- 27. Palmer MA, Bernhardt ES, Allan JD, Lake PS, Alexander G, Brooks S, et al. Standards for ecologically successful river restoration. J Appl Ecol. 2005;42: 208–17.
- 28. Dyer JS, Sarin RK. Relative risk-aversion. Manage Sci. 1982;28: 875–86.
- 29. Ananda J, Herath G. Evaluating public risk preferences in forest land-use choices using multi-attribute utility theory. Ecol Econ. 2005;55: 408–19.
- 30. Eisenführ F, Weber M, Langer T. Rational Decision Making. Berlin, Heidelberg: Springer-Verlag; 2010.
- 31. Dyer JS, Sarin RK. Measurable multiattribute value functions. Oper Res. 1979;27: 810–22.
- 32. Bortz J, Döring N. Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler / J. 4., überarb. Nachdruck Aufl. ed. Heidelberg, Germany: Springer Medizin Verlag; 2009. 897 p.
- 33. Choy SL, O'Leary R, Mengersen K. Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. Ecology. 2009;90: 265–77. pmid:19294931
- 34. Clemen RT, Reilly T. Making Hard Decisions with Decision Tools. Pacific Grove: Duxbury Press; 2001.
- 35. Hamalainen RP, Luoma J, Saarinen E. On the importance of behavioral operational research: The case of understanding and communicating about dynamic systems. Eur J Oper Res. 2013;228: 623–34.
- 36. Montibeller G, von Winterfeldt D. Cognitive and motivational biases in decision and risk analysis. Risk Anal. 2015;35: 1230–51. doi: 10.1111/risa.12360. pmid:25873355
- 37. Morton A, Fasolo B. Behavioural decision theory for multi-criteria decision analysis: a guided tour. J Oper Res Soc. 2009;60: 268–75.
- 38. O´Hagan A, Buck CE, Daneshkhah A, Eiser R, Garthwaite P, Jenkinson D, et al. Uncertain Judgements: Eliciting Experts´ Probabilities (Statistics in Practice). Chichester, West Sussex, England: John Wiley & Sons; 2006.
- 39. Regan HM, Colyvan M, Burgman MA. A taxonomy and treatment of uncertainty for ecology and conservation biology. Ecol Appl. 2002;12: 618–28.
- 40. Peter A. Untersuchungen über die Populationsdynamik der Bachforelle (Salmo trutta fario) im System der Wigger, mit besonderer Berücksichtigung der Besatzproblematik. (Population dynamics of brown trout (Salmo trutta fario) in the River Wigger, considering fish stocking) (in German). PhD thesis, ETH Zurich. 1987.
- 41. Hamalainen RP, Alaja S. The threat of weighting biases in environmental decision analysis. Ecol Econ. 2008;68: 556–69.
- 42. Weber M, Eisenfuhr F, Vonwinterfeldt D. The effects of splitting attributes on weights in multiattribute utility measurement. Manage Sci. 1988;34: 431–45.
- 43. Hering D, Borja A, Carstensen J, Carvalho L, Elliott M, Feld CK, et al. The European Water Framework Directive at the age of 10: A critical review of the achievements with recommendations for the future. Sci Total Environ. 2010;408: 4007–19. doi: 10.1016/j.scitotenv.2010.05.031. pmid:20557924
- 44. European Commission. Directive 2000/60/EC of the European Parliament and of the Council of 23 October 2000 establishing a framework for community action in the field of water policy; 2000.
- 45. Cunningham R. Glass half full or half empty? Why 2009 Water Framework Directive classification results are over-optimistic about the state of rivers despite the One-out, All-out rule. Royal Society for the Protection of Birds; 2012. Available: http://www.wfduk.org/sites/default/files/Media/Wildlife%20and%20Countryside%20Link%20The%20One-Out%20All-Out%20pessimism%20myth%20Final.pdf. Accessed December 2015.
- 46. Delforce RJ, Hardaker JB. An experiment in multiattribute utility-theory. Aust J Agr Econ. 1985;29: 179–98.
- 47. Duckstein L, Treichel W, Elmagnouni S. Ranking groundwater-management alternatives by multicriterion analysis. J Water Res Pl-Asce. 1994;120: 546–65.
- 48. Keeney RL, Wood EF. Illustrative example of use of multiattribute utility theory for water-resource planning. Water Resour Res. 1977;13: 705–12.
- 49. Raju KS, Pillai CRS. Multicriterion decision making in performance evaluation of an irrigation system. Eur J Oper Res. 1999;112: 479–88.
- 50. Raju KS, Vasan A. Multi attribute utility theory for irrigation system evaluation. Water Resour Manag. 2007; 21: 717–28.
- 51. Torrance GW, Feeny DH, Furlong WJ, Barr RD, Zhang YM, Wang QN. Multiattribute utility function for a comprehensive health status classification system—Health Utilities Index Mark 2. Med Care. 1996;34: 702–22. pmid:8676608
- 52. LAWA. Gewässerstrukturgütekartierung in der Bundesrepublik Deutschland—Verfahren für kleine und mittelgroße Fließgewässer. Empfehlung: Länderarbeitsgemeinschaft Wasser; 2000.
- 53. LAWA. Gewässerstrukturgütekartierung in der Bundesrepublik Deutschland–Übersichtsverfahren. Empfehlungen Oberirdische Gewässer: Länderarbeitsgemeinschaft Wasser; 2002.
- 54. Strager MP, Fletcher JJ, Yuill CB, Strager JM. Not in My Watershed! An Interactive Tool to Evaluate Land Use Changes on Stream Water Quality. Prodeedings of the ESRI International User Conference. Available: http://proceedingsesricom/library/userconf/proc00/professional/papers/PAP663/p663htm. Accessed March 2012.
- 55. USEPA. Developing water quality criteria for suspended and bedded sediments (SABS), Potential approaches. US EPA Office of Water, Office of Science and Technology, Draft; 2003. Available: http://water.epa.gov/scitech/swguidance/standards/criteria/aqlife/pollutants/sediment/upload/2004_08_17_criteria_sediment_sab-discussion-paper.pdf. Accessed August 2014.
- 56. Scholten L, Schuwirth N, Reichert P, Lienert J. Tackling uncertainty in multi-criteria decision analysis—An application to water supply infrastructure planning. Eur J Oper Res. 2015;242: 243–60.
- 57. Lienert J, Koller M, Konrad J, McArdell CS, Schuwirth N. Multiple-criteria decision analysis reveals high stakeholder preference to remove pharmaceuticals from hospital wastewater. Environ Sci Technol. 2011;45: 3848–57. doi: 10.1021/es1031294. pmid:21417333
- 58. Bundi U, Peter A, Frutiger A, Hutte M, Liechti P, Sieber U. Scientific base and modular concept for comprehensive assessment of streams in Switzerland. Hydrobiologia. 2000; 422: 477–87.
- 59. Sumpter JP, Jobling S. Vitellogenesis as a biomarker for estrogenic contamination of the aquatic environment. Environ Health Persp. 1995; 103: 173–8, Supplement: 7.
- 60. Aspinall W. A route to more tractable expert advice. Nature. 2010;463: 294–5. doi: 10.1038/463294a. pmid:20090733
- 61. US Environmental Protection Agency. Expert elicitation task force white paper. Washinigton, DC; 2011. Available: www.epa.gov/osa/pdfs/elicitation/Expert_Elicitation_White_Paper-January_06_2009.pdf. Accessed April 2015.
- 62. Ananda J, Herath G. A critical review of multi-criteria decision making methods with special reference to forest management and planning. Ecol Econ. 2009;68: 2535–48.
- 63. Keeney RL. Foundations for Group Decision Analysis. Decision Anal. 2013;10: 103–20.
- 64. Kim SH, Ahn BS. Interactive group decision making procedure under incomplete information. Eur J Oper Res. 1999;116: 498–507.
- 65. Clemen RT, Winkler RL. Combining probability distributions from experts in risk analysis. Risk Anal. 1999;19: 187–203.
- 66. Kuhnert PM, Martin TG, Griffiths SP. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett. 2010;13: 900–14. doi: 10.1111/j.1461-0248.2010.01477.x. pmid:20497209
- 67. Oppenheimer M, O'Neill BC, Webster M, Agrawala S. The limits of consensus. Science. 2007;317: 1505–6. pmid:17872430
- 68. Morgan MG. Use (and abuse) of expert elicitation in support of decision making for public policy. P Natl Acad Sci USA. 2014;111: 7176–84.
- 69. Langhans SD, Hermoso V, Linke S, Bunn SE, Possingham H. J Environ Manage. 2014;132: 296–303. doi: 10.1016/j.jenvman.2013.11.021. pmid:24325822
- 70. Reichert P, Paillex A, Schuwirth N, Schirmer M, Brouwer R, García de Jalón D, et al. D5.4: Risks and uncertainty in river rehabilitation. REFORM REstoring rivers FOR effective catchment Management (project funded by the European Commission within the 7th Framework Programme 2007–2013). Eawag (and others). 2015. Available: http://www.reformrivers.eu/system/files/5.4%20Risks%20and%20Uncertainty%20in%20River%20Rehabilitation%20-%20FINAL.pdf. Accessed 16 February 2016.
- 71. KBOB. Empfehlungen zur Honorierung von Architekten und Ingenieuren. Schweizerische Eidgenossenschaft. 2016. Available: https://www.kbob.admin.ch/kbob/de/home/publikationen/dienstleistungen-planer/empfehlungen-zur-honorierung-von-architekten-und-ingenieuren.html. Accessed 16 February 2016.
- 72. Gregory R, Failing L, Harstone M, Long G, McDaniels T, Ohlson D. Structured decision making: A practical guide to environmental management choices. Wiley-Blackwell Publishing; 2012.
- 73. European Commission. River Basin Management Plans. Report from the commission to the European Parliament and the Council on the implementation of the Water Framework Directive (2000/60/EC); 2012.