Survey methodology

Multiple trained coders with many years of experience implemented the methodology for the first-round survey. The second round (R2) follows this method for consistency reasons. All persons administering the survey were trained for 2–3 weeks and had a college education. Each team consisted of two members, at least one of which was a civil engineer.

At least three users answered the questionnaire. The first person was a user who was willing to answer questions. The second person is someone who was pointed out by others as being very knowledgeable about the irrigation system. The third informant was someone who was economically underprivileged on average, as determined by plot size and the condition of their assets. If one of the respondents answered “poor” to a question about a physical condition, the system was coded as poor in that dimension. For all other answers, the most frequent response was entered into the database and group interviews were also conducted. The performance variables of water supply adequacy, predictability and reliability at the end of the process are based on the respondents’ answers. The survey teams were trained to assess these metrics consistently.

The sampling method, distribution and characteristics of irrigation systems, and survey methodology are described in length in [43]. Further information about the data collection, summary statistics about all variables, e.g., size, who built these systems, etc. can be found in the supplementary materials (S1 Table S13 in S1 File).

Machine learning

Machine learning algorithms typically outperform typical statistical approaches like multivariate linear regressions on predictive performance. This is especially true for this particular data set, since one study has shown through a direct method comparison [41] that predicting out-of-sample success of for these irrigation systems works much better with machine learning approaches. Therefore, we use these state-of-the-art algorithms for the best predictive model performance.

However, even very similar algorithms such as random forests and gradient boosting sometimes yield conflicting results about the significance of variables [47]. Therefore, for robust conclusions, we compare 5 different algorithms – decision trees, random forests, gradient boosting, general linear models and support vector machines, both tuned and “untuned”, i.e., with the default parameters, serving as benchmark (S1 Section 3 in S1 File), resulting in 10 variants. All algorithms were implemented using the R package MLR3 [48]. These algorithms were chosen for their superior performance and we provide the machine learning code in the supporting material. The models were tested for accurate classification of the seven performance variables listed in Table 1. The list of independent variables can be found in the supplementary materials (S1 Table S1 in S1 File). In addition, analyses were performed using the H2O package [49] to rule out any possible model artifacts resulting from the package implementation. No such artifacts were detected.

Training was performed on 80% of the data sets, with 10% held back for three-fold cross-validation and the remaining 10% used for testing. Due to the small data set size, we opted for 80% instead of 70% [41]. The test set, which the algorithm does not see, was then used to evaluate prediction accuracy, expressed as the percentage of correct classifications (i.e., deterioration or no deterioration). A different metric (log loss) was chosen for the survived variable, since accuracy is not suitable for highly skewed variables (88% of systems survived). Log loss is a measure of how well predicted probabilities match the true outcomes, penalizing confident wrong predictions more strongly.

Both a random grid search method and Bayesian optimization were used for hyperparameter tuning, as these have been shown to be superior to other tuning methods across various domains. Each algorithm has parameters that require tuning, and the ranges for these are provided in the supplementary materials (S1 Section 3 in S1 File). However, the default parameters often proved to be on par with their tuned counterparts. To ensure comparability between the machine learning algorithms, the duration of each search was set to the same value for each run. Search times varied between 10 and 150 minutes. For decision trees, gradient boosting and random forests, the relative importance of each variable was extracted.

With a multitude of variables and models, spurious correlations become more likely. It is therefore important to note that the variables included in our analysis were strictly those identified by prior studies and theory as potentially significant for survival and performance over time. Due to the number and complexity of these variables, we refer to [33] where all these factors are defined, their significance and pathways for success, and their relationships are described in detail. Thus, our study can be understood as measuring the relative significance of factors hypothesized to be important for long-term survival. One limitation of this approach is that we miss the opportunity to identify novel factors, despite having constructed a thorough longitudinal dataset. However, we also reduce the possibility of identifying spurious factors. We were also able to identify the machine learning models and algorithms that performed best for this particular application.

Predicting performance from data 30 years ago

In line with previous research [27,50], we consider survival and performance. When analyzing performance, we distinguish between the physical condition of the headworks and the canals, and between tail equity as well as tail adequacy (Table 2). However, unlike existing studies, the following results predict performance in 2013 using just the data from 30 years ago. Hence, long-term results (LT) are not comparable to snapshot studies.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Best models for each of the six performance variables.

https://doi.org/10.1371/journal.pone.0353875.t002

Across the board, both untuned Decision Trees and tuned Gradient Boosting perform best, but there is no clear overall winner. The sweet spot for training time seems to be around 40–60 minutes – models trained for three hours do not perform better, possibly due to overfitting.

None of the models for a specific period, whether R1, R2 or LT, perform consistently better across all variables. Thus, models for 30 years reach about the same accuracy as models for either round 1 or 2.

The most difficult performance variables to predict are maintenance of headworks and maintenance of canals. This is plausible, since these depend on the time the systems have been surveyed – repairs might have been scheduled, but not yet realized. Here, the best models correctly predict 67% (SVM) and 59% (SVM) of all cases, respectively. It is comparatively easier to predict the deterioration of canals (89%, DT) and headworks (77%, DT).

In contrast, tail adequacy (84%, GBM), equity (86%, SVM) and predictability (92%, many models) can be predicted rather accurately, even with data from 30 years ago. Given that the majority of schemes have indeed survived, and that institutional design changes slowly in successful schemes because there is little need, this seems plausible.

The causal pathways between influencing factors and outcomes are complex [33]. For example, a high participation rate can lead to quick and targeted adaptation of rules. In turn, a high dependency on the resource system increases the likelihood of participation, e.g., in user associations. Tail equity relies on fair and open processes which benefit from high participation levels.

Comparing model performance across algorithms

Fig 1 shows an overview of the accuracy of all model runs for the test sets (= out-of-sample) across all seven performance variables (for details, see S1 section 4 in S1 File), without delving into the importance of contributing factors (see section 1.4 and Table 1.3 for that).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Accuracies across machine learning algorithms and all performance variables in both R1 and R2 (higher = better predictive accuracy).

https://doi.org/10.1371/journal.pone.0353875.g001

Median accuracy across all seven performance variables is relatively stable at about 0.65, except for the Bayesian models. The two tuned gradient boosted models (XGBoost) perform slightly better than the others. In each case, the tuned models outperform their untuned counterparts, albeit sometimes only slightly. This consistent performance indicates that a large number of models accurately estimate the importance of contributing factors.

The mean accuracy is highest (0.75) for predictability at the tail, and lowest for maintenance of the headworks (0.39). The best models (see Table 2 below) are fairly accurate, often classifying systems with 80–90% accuracy. ML-results always have to be compared to a benchmark and in this case, benchmark models are featureless, i.e., they always predict the majority of the cases. In the case of skewed outcomes, e.g., survival, where 88% of systems survive, this makes it especially hard to outperform them, since they predict survival correctly 88% of the time. In general, benchmark models are often only a few percentage points below the best ML-models.

Survival

Can we predict 30 years in advance which systems will survive? This would be of immense value for the design of irrigation systems. It would also come as no surprise if this were not possible, given the long time and – in this case – the many major external changes in Nepal like war or revolutions. Yet, some models predict survival rather precisely. Here, survival refers to whether a canal was in use at the time of the second survey. A system that falls into disuse but is then revived and becomes functional once again is still considered a surviving system if the users are largely the same and the system was used at the time of the second survey.

As the sample is highly skewed, with 88% of all systems surviving, accuracy is not an appropriate measure of error. Instead, log loss is used. The best models for predicting system survival are those that include only institutional design variables. All models that include system size (area and number of users) are worse by ~ 6% in accuracy and 0.05 in log loss. Correlations between the number of users and the length of main channel are only moderately strong (see S1 Section 5). This is likely because of variations in geography – long canals in hilly areas typically serve fewer users than in the plains.

The best algorithm (tuned decision trees) predicts survival with a log loss (error of prediction probability) of 0.17 and is thus better than the baseline featureless benchmark model with a log loss of 0.23.

Which factors are most predictive of performance and persistence?

There is little agreement in the literature on which characteristics are associated with the performance of irrigation systems [39]. There are several reasons for this. On the one hand, differences may be due to factors being conceptualized, measured and/or modeled differently [33]. In addition, the choice of method itself plays a role, as machine learning algorithms assign quite different importance to variables, even when using the same dataset [47].

A major contribution of this paper is its ability to address these issues. First, only results that are robust across ten different machine learning algorithms are reported. Second, seven performance variables, drawn from existing studies and theory, are used to capture nuances in performance. Third, three explanatory models (R1, R2 and LT) complement each other (see S1 Tables S3 and S4 in S1 File). Fourth, we distinguish between three sets of variables: one set considers only institutional variables as independent variables; one set considers only physical variables (e.g., number of users, system area or rainfall); and one set considers all variables (see S1 Tables S3 and S4 in S1 File). This separation allows the stability of the influence to be estimated more accurately across different sets of variables. Finally, the second-round data (R2) were collected using the same protocols as the first (R1) round to maintain the internal validity of the concepts and their measurement.

We calculated the importance of each performance variable (see Table 3) and survival across all models (n = 218). The three factors with the highest mean per performance variable across machine learning models are reported. Therefore, variables that are consistently associated with success in the majority of these variations should be highly relevant to a large number of irrigation systems in general.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. The three most important variables for LT (relative factor importance) with directions in parentheses.

https://doi.org/10.1371/journal.pone.0353875.t003

The following section discusses the main influences for each performance variable.

The condition of the headworks of irrigation systems is related to outside assistance with penalties that are enforced, and only a small perceived economic gap between rich and poor members. Participation of users in the decision-making process also shows a strong predictive association with success, likely because it is a critical factor for rule enforcement [12,51]. However, the many possible causal interrelationships between factors are not the topic of this analysis (but see [33,52]) and have not been modelled quantitively yet.

For the canals, outside assistance by the government or NGOs and stable rules are equally predictive for success. Systems in which users themselves are in charge — including construction, operation of canals, and management — tend to show higher persistence.

The predictability of water at the tail end is associated with good communication in arenas where problems can be discussed. Frequent user meetings and fair, user-adjustable, user-enforced rules are likewise associated with survival. It is also relevant that not too many members spend time in non-agricultural activities.

For water adequacy at the tail end, the relevant factors are outside support, but also that users can manage their system themselves. It is also relevant that users do not take more than their share water from the canals than before.

Stable and fair rules are a precondition for equity of water at the tail. Again, this is a good validation of our models, as these variables are expected to be closely related. The presence of frequently used dispute-resolution arenas and of penalties is also associated with better outcomes. Perceived rule fairness is among the characteristics most strongly associated with persistence. Outside assistance in critical periods is also important. In addition, it is critical that the users can manage the system themselves and address the rules according to changing needs by using appropriate arenas.

Overall, interventions from non-governmental organizations or government agencies show among the strongest associations with performance and persistence in our data. Having a leader from within the group, regular meetings, and stable rules are also among the most predictive characteristics overall [53,54]. A wide gap between poor and rich group members is detrimental to success, and enforcement is positively associated possibly because good institutional design only matters if rules are enforced [6,7].

Surprisingly, enhancing the system or avoiding harm to it seems not to be particularly relevant for success. Physical variables like variation of water over time, for example due to rainfall, have some short-term relevance, but do not play a major role long-term. Penalties and their enforcement are important in some models, but not in others.

A comparison with the design principles of Elinor Ostrom [6] finds that three design principles – graduated sanctions, collective-choice arrangements, and minimal recognition of rights to organize – play the most important role if matched to the performance variables. A detailed table can be found in the supplementary materials (S1 Table S14 in S1 File). This ties in with the work of Cox et al. [7] who also find support for these design principles while pointing out that other important factors are missing, like fairness in our case.

Finally, we remain unsure about the role of system size measured by the number of users and the length of canals (see S1 Section 5 in S1 File). On the one hand, when included in the models, size dominates the importance ranking, like in community forestry [55]. On the other hand, if left out, models are about the same quality. This is in line with previous results with this dataset, which also found no clear effect of size [50]. A dedicated study is needed to tease these complex relationships apart.

Successful versus unsuccessful systems

In the dataset, there is a wealth of information on what is characteristic of stable, well performing versus unstable or failing systems. Since five of the seven performance variables are binary, they naturally fall into two groups that can be tested for differences with t-tests. The two variables for the maintenance status of the headworks and the canals are transformed into binary ones by lumping “good” and “fair” together as opposed to “poor”. All reported tests in the text are significant at the 0.05 level with Bonferroni correction. The exact means per group, t-values and degrees of freedom can be found in the supplementary materials (S1 Section 7 in S1 File).

Good maintenance of headworks and canals is related to the availability of arenas for users to meet and discuss, equity and adequacy of water distribution at the tail end, a decreasing water use rate, and interventions by the government or NGOs.

Deterioration of headworks and canals is associated with less maintenance, and no penalties. Deterioration is also less severe if there is no other canal available as an alternative and if users can participate in system governance.

Predictability of the waterflow at the tail is associated with less variability of water over time, is better if the system is built by the government or NGOs (instead of the users themselves), and, surprisingly, surviving systems have less predictable flow at the tail end. It is independent of other performance variables.

More equity at the tail end is coupled with much higher maintenance of headworks and canals, a decreasing use rate, no alternatives, and better enforcement of the rules.

An adequate water supply is also tied to decreasing use, equity, good communication between government and water user associations, and good maintenance of the physical hardware.

Finally, survival is related to having established penalties, predictability of the water supply, and only small differences between the rich and the poor families of the group.