A Multi-Criteria Index for Ecological Evaluation of Tropical Agriculture in Southeastern Mexico

The aim of this study was to generate an easy to use index to evaluate the ecological state of agricultural land from a sustainability perspective. We selected environmental indicators, such as the use of organic soil amendments (green manure) versus chemical fertilizers, plant biodiversity (including crop associations), variables which characterize soil conservation of conventional agricultural systems, pesticide use, method and frequency of tillage. We monitored the ecological state of 52 agricultural plots to test the performance of the index. The variables were hierarchically aggregated with simple mathematical algorithms, if-then rules, and rule-based fuzzy models, yielding the final multi-criteria index with values from 0 (worst) to 1 (best conditions). We validated the model through independent evaluation by experts, and we obtained a linear regression with an r2 = 0.61 (p = 2.4e-06, d.f. = 49) between index output and the experts’ evaluation.


Introduction
Since the report of the World Commission of Environment and Development (1987), better known as the ''Brundtland Report'', and the UN Conference on Environment and Development (1992) in Rio de Janeiro, the concept of sustainability has gained worldwide acceptance and popularity. It is generally agreed that sustainability indicators are necessary to put the sustainability concept into effect and to introduce it into government policies. There is an abundance of literature on holistic indicators for sustainable development as well as on indicators for specific dimensions of sustainability, for example, the environmentally sound management of a variety of natural resources such as forestry, agriculture and animal production (the works of Bü chs, 2003;Mendoza and Prabhu, 2003;Karlen et al., 2004;Gamborg and Sandøe, 2005;OECD, 2008 are but a few).
However, it is only recently that practical tools which can help local users to apply these general concepts at the local to regional levels have emerged (Hurni, 2000). Land evaluation and land-use decisions on a local scale are instead often based on the opinion of local experts, as opposed to the formal application of indicators or decision support systems for environmentally sound resource management. This is particularly evident in the tropics where, due to restricted access to modern communication and information technologies, local decision makers rely on the expertise of local academics, forest managers, cattle breeders and/or farmers. Fuzzy rule-based model Indicator development Machine-learning Regression tree Sustainable land use Tropical agriculture Tropical cattle-farming Tropical forest Tropical soils A B S T R A C T Despite abundant literature on indicators for sustainable resource management, practical tools to help local users to apply its general concepts at a local to regional level are scarce. This means that decisions over land evaluation and land use at a local level are often not based on the formal application of indicators or decision support systems for environmentally sound management but instead on the opinion of local expertise, for instance forest managers, cattle breeders, farmers and/or academics. This is particularly seen to be the case in the tropics where access to modern communication and information technologies is restricted.
As the opinions of experts are often based on and influenced by personal experience, intuition, heuristics and bias, their evaluations and decision are often unclear to the non-expert working at a local level. In order to make their reasoning more comprehensible to the non-expert, the ecological condition of 176 plots in the tropical Southeast of Mexico were evaluated by experts on soil fertility, forest management, cattle breeding and agriculture. With the assistance of a knowledge engineer (one who converts expert knowledge and reasoning into a model), these expert opinions and reasoning were then translated into a formal computer model.
As an alternative approach we applied a knowledge discovery technique, namely the induction of regression trees and automatically developed models using the expert evaluations as training data. Where knowledge engineering was tedious and time consuming, regression models could be rapidly generated. Moreover, the correspondence between regression trees and expert opinions was considerably higher than the correspondence between expert opinion and their own models. The regression trees used less explicative variables than the models generated by the experts. The minimisation of sampling effort due to variable space reduction means that the application of regression tree induction has a high potential for a rapid development of indicators for narrowly defined ecological assessments, needed for decision making on a local or regional scale.
However, similar problems arise even in developed countries; Sutherland et al. (2004) reported that more than 75% of the knowledge sources consulted for conservation management actions in a UK nature reserve are based on common sense and personal experience, rather than scientific evidence. If the domain expert's reasoning is to be understood by decision-makers and users it must first be translated into a formal model which will typically, but not necessarily take the form of a computer programme. Two key parts in the development of a formal model are 'knowledge acquisition' (Finlay and Dix, 1996), the gleaning of knowledge from domain experts which will be required by the knowledge engineer in the formation of a formal model and 'knowledge engineering' (Medsker and Liebowitz, 1994), the process of designing, developing, testing and implementing a computer programme for a formal model. The acquisition of knowledge from experts has proved to be a difficult task as many expert system projects have shown, experts are notoriously bad in communicating what they know in a formalised manner. Also, checking the represented experts' knowledge for consistency and completeness, for example as a set of IF-THEN rules, can be a laborious and time consuming task.
The development of a formal model will require several interviews between the knowledge engineer and domain expert in order to fill in any gaps in knowledge, resolve doubts and confirm the model structure. Also, to facilitate clearer communication the expert must become familiar with at least a very basic knowledge of the inference methods applied, fuzzy logic being one example. This can lead to what is referred to in expert systems literature as a 'knowledge acquisition bottleneck' (Medsker and Liebowitz, 1994). To remedy this problem, ways in which expert opinions can be automatically translated into a formal model have been sought, for example, the application of learning algorithms to automatically generate models from training data, a process called machine learning or knowledge discovery. Recently, frameworks have been proposed to aid researchers in designing fuzzy indices of environmental conditions (Marchini et al., 2009); unfortunately these were published too late to be applied in this study.
For the purpose of this study all three strategies were applied, expert opinion for evaluating land use types, knowledge engineering by converting the reasoning of the domain experts into a formalised model and knowledge discovery by automatically inducing a model using the domain expert's opinion as training cases.
First of all, using one domain expert for each land use and soil type, evaluations were made on a number of sites based on a set of primary indicators chosen by each expert. The experts then tried to explicitly represent their reasoning in a hierarchical model that should predict the ecological state of each site. This posed the question of whether the end results would correspond with one another, as a synthetic domain experts' evaluation (based on experience and heuristics) will not necessarily coincide with an analytical explanation, even when both are provided by the same expert.
Finally, to discover how far an automated time saving procedure could replace the difficult time consuming process of knowledge acquisition, the evaluations made by each domain expert were used as training data in the automated induction of regression trees. It was also necessary to find out whether the automatically induced model would contain every primary indicator chosen by the domain expert, or if some of them would become redundant.
The two questions addressed in this paper are: (1) Do the predictions of formal models, based on the knowledge and experience of domain experts, actually correspond to the evaluations made by the same experts? (2) Can time-saving machine learning methods be used to develop evaluation models and, if so, how do they correspond to the models developed by the domain experts?

Materials and methods
We evaluated the ecological state of forested sites, cattlebreeding ranches and agricultural areas, as well as the soils in the State of Tabasco, SE Mexico, a region which has undergone drastic land use changes in the last few decades, such as the conversion of tropical forest to cattle-breeding areas (Challenger, 1998). We randomly chose 176 sites distributed in the municipalities Balancá n and Tenosique in the east of Tabasco (17815 0 -18810 0 N, 91801 0 -91846 0 W) ( Fig. 1) comprising an area of 5474 km 2 . The region is characterized by a warm (mean annual temperature 26 8C) and humid climate with precipitation throughout the whole year (1750 mm annual precipitation) (INEGI, 2001). The region is mainly a plain (67% of area, elevation <20 m a.s.l.) with hills (29%, 20-200 m a.s.l.) and mountains (4%, max. 640 m a.s.l.) located in the southern part. The dominant soils are Gleysols over alluvial sediments (plain), Vertisols, Cambisols, Luvisols and Acrisols over Miocene or Oligocene sediments (hills) and Leptosols and Regosols over limestone (mountains) (INEGI, 1985). The two municipalities have suffered dramatic changes of land cover during the last 60 years. While in 1950 the very sparsely populated region still was dominated by various types of tropical rainforest, by 2003 only about 10% of forests (including secondary forests) were left (Isaac-Má rquez, 2008), mainly in the less accessible mountainous areas. Due to governmental programs of subsidies for cattle ranching, the percentage of grassland has increased to 78% (Casco, 1980;Tudela, 1989;Isaac-Má rquez et al., 2008). Annual and perennial crops cover approximately 4% of the study region; the remainder consists mainly of wetlands and temporarily flooded areas.
Each site corresponded to a plot of homogenous land use. Four domain experts of forestry (SO-G, BdJ 1 ), cattle-breeding (SH-D), agriculture (EH-L) and pedology (VG) had the task to evaluate land use on these sites. In particular, forested plots were evaluated with regard to structure, composition and perturbations, grazing cattle and agricultural plots with regard to the ecological sustainability of the actual site management, and soils on all plots with regard to soil fertility. For ease of communication, we refer to each of the four evaluations as an index of ecological condition in the remainder of this paper.
Based on personal knowledge and experience, each expert chose a number of primary indicators that in his or her opinion would satisfy the purpose of evaluating ecological condition. Their values were determined in the field between March and December 2004. Non-numeric primary indicators were scaled from 0 to 1, where 0 represented ''disastrous'' and 1 represented ''excellent''. Based on the primary indicators and expertise, each domain expert evaluated ecological condition on each site on a scale from 0 to 1 (0: very bad; 1: very good).
Subsequently, each domain expert was asked to represent his reasoning in an explicit model, supported by a knowledgeengineer who himself is an ecologist (CK 1 ). The experts should mould their ways of evaluating the sites into a hierarchical structure, aggregating two or more primary indicators at a time to higher level intermediate variables, and finally aggregating the intermediate variables into an index of ecological condition. The hierarchical structure was chosen (a) in order to support the domain experts in structuring their ecological reasoning, (b) in order to reduce rule-complexity at each aggregation level (see below). The methods applied in indicator aggregation were:  Table 1).
While in classic set theory an object can only either be a member (membership = 1) or not be a member (membership = 0) of a given set, the central idea of fuzzy set theory is that a member of a set may have partial membership, which consequently may possess all possible values between 0 and 1. The closer the membership of an element is to 1, the more it belongs to the set; the closer the membership of an element is to 0, the less it belongs to the set. The value a = 0.4 of primary indicator A in Fig. 2, for example, has higher membership in the set ''a is medium'' than in the set ''a is low''; it does not belong at all to the set ''a is high''. The determination of the model output consists of three steps: First, the observed values of the primary indicators are translated into membership values in the fuzzy sets (called fuzzification); second, the memberships of the consequence of the applying rules in the fuzzy sets of the intermediate variable X are calculated (called fuzzy inference); third, the fuzzy result is converted into a discrete numerical output (called defuzzification) (see Bothe, 1995or Zimmermann, 1996 for an introduction to fuzzy models). Fuzzy rule-based models have become comparably popular in ecological modelling (Li and Rykiel, 1996;Salski, 1996) and there exist various examples in the context of evaluation, bioindication and sustainable management (for example, Mendoza and Prabhu, 2003;Kampichler and Platen, 2004).
The hierarchical structure of the model helps to keep the rulebases as simple as possible. Assume three primary indicators, A, B and C, with three levels each, that are to be aggregated to an index I: direct aggregation needs 3 3 = 27 rules with three antecedents each, whereas the aggregation of A and B to intermediate variable X and the aggrgeation of X and C to index I requires only 3 2 + 3 2 = 18 rules with only two antecedents each.
The expert evaluations of soil quality and ecological condition of forest, agricultural and cattle-ranching systems were used as training cases in order to induce regression trees (Breiman et al., 1998) using the primary indicators as independent variables and expert evaluation as the dependent variable. Regression trees are based on the assumption that the relationship between independent and dependent variables is not constant over the entire range   of possible variable values, but can be approximated in smaller subdomains. Regression trees are thus constructed iteratively splitting the data into homogeneous subsets, which results in a tree-like structure where each branch is defined by a certain range of values of the independent variables and the terminal nodes of the tree consist of a predicted constant. Regression trees are ideally suited for exploring data that exhibit nonlinearity and high-order interactions; in contrast to classification trees they are, however, only occasionally used in ecological data analysis (for example, Andersen et al., 2000;De'ath and Fabricius, 2000;Kampichler et al., 2000;Džeroski and Drumm, 2003).
Personal expert evaluations and model outputs were compared by calculating Spearman rank correlations as well as their mean absolute deviations (MAD) calculated as where x i,expert is the ecological condition of the ith site as evaluated by the expert domain, x i,model is the ecological condition of the ith site as predicted by the model, and n is the number of sites. Personal expert evaluations and the regression trees were compared in the same way. All statistical analyses were performed with R (R Development Core Team, 2007); for regression tree induction we used the R package tree (Ripley, 2007).

Results
The domain experts chose between 8 (soils) and 20 (primary and secondary forests) primary variables for their personal evaluations of the sites. They aggregated the primary variables to an index of ecological condition by means of between 5 (soils) and 15 (primary and secondary forests) nodes generating between 4 and 14 intermediate variables (Fig. 3). Detailed descriptions of each model (justification for variable choice and variable aggregation, specific algorithms used in each model node, rule sets, fuzzy sets) will be published elsewhere (Herná ndez-Daumá s et al., submitted for publication; Ochoa-Gaona et al., submitted for publication).
There were noteworthy differences between the experts' opinions and the outputs of their models (Fig. 4). The Spearman correlation coefficient could be as low as 0.417 and the mean absolute deviation between expert opinion and the ecological condition index as calculated by the model attained values as high as 0.279 (Table 2).
All regression trees (Fig. 5) included considerably less primary variables than chosen by the domain experts. Numbers of variables used dropped from 20 to 5 (primary and secondary forests), from 9 to 5 (cattle ranching), from 12 to 4 (agriculture) and from 8 to 5 (soils) ( Table 2). Nonetheless, there was higher correspondence between expert opinion and regression trees (Fig. 6) than between expert opinion and their own models with considerable higher correlation coefficients and considerable lower MADs (Table 2).

Discussion
Human experts are human beings, and as with all humans, their decisions are influenced by personal experience, intuition, heuristics and bias. Thus, experts of the same knowledge domain often disagree on the relative importance of different evaluation criteria (Weisberg et al., 2008). Uncertainty or vagueness of how to rate the importance of every possible explanatory variable might motivate domain experts to choose a high number of primary indicators, obviously obeying a rule of thumb such as ''better measure a few variables too many than to miss an important factor''. Regression trees reduced the number of primary indicators considerably, at least by 38%. Since expert opinion served as database for their training, regression trees yielded results corresponding closely to the expert's evaluation but were more parsimonious in terms of necessary variables. Thus, the automated induction of regression trees seems to be a promising way to develop indicators for narrowly defined ecological assessments as needed for decision-making at the local or regional scale. A twostage procedure -first stage: primary indicator selection by a local domain expert, preliminary sampling, expert evaluation, induction of a regression-tree trained by the expert evaluation results; second stage: comprehensive sampling taking into account only the narrowed indicator set -could reduce considerably the time . Ecological condition is scaled between 0 (very bad) and 1 (very good). a DBH, diameter in breast height; b relative value ranging from 0 to 5 considering presence of at least two types from (i) scattered, (ii) riparian, (iii) live fences and (iv) other important trees, one of them has to be (i) or (ii) in a well developed canopy; c CEC, cation exchange capacity. spent in the field for sampling and data acquisition. A reduction of variables by 75% such as in the case of primary and secondary forests could allow for the multiplication of sample size without increasing project expenditure, thus producing more data and higher statistical power.
Model development by the domain experts themselves obviously had a serious drawback: in all cases, the model outputs were not as close to their personal evaluations as were the results of regression tree induction. Initial expert doubts on the validity of the trees could easily be resolved. For example, the expert on cattle-ranching systems observed the condensation of the values of the ecological condition index to a narrower range from [0.22 0.73] in the expert's evaluation to [0.28 0.63] in the regression tree (Fig. 6B). This reflects the problem of assigning values to a few extreme plots (very good and very bad ones); while the tree yields values that corresponded well with the experts evaluation, the experts evaluation and the model did not as shown by their quartiles (expert evaluation: 0.39, 0.46, 0.56; expert model: 0.20, 0.35, 0.49; regression tree: 0.40, 0.46, 0.58).
Moreover, machine learning methods have the potential to improve ecological knowledge. Džeroski et al. (1997), for example, reported how classification tree induction enhanced knowledge on the role of several bioindicator taxons in Slovenian rivers. The experts' opinions on the regression trees generated in this study were conflicting. Some aspects of the trees were positively recognised, others caused contradiction. For example, the forest experts acknowledged the primary bifurcation in the corresponding regression tree (Dominance of trees with diameter in breast height >11-20 cm; Fig. 5A), but doubted the validity of the second bifurcation (shrub layer <50% vs. >50%). Also, the expert on cattle-ranching systems was surprised by the fact that the sole presence of electric fences should be sufficient to qualify a plot as comparably good (index of ecological condition = 0.63, see Fig. 5B). This conflict most probably arises due to the fact that experts intent to find relationships between primary variables and ecological condition which are generally valid and not restricted to the study area, whereas tree induction yields relationships valid specifically for the training set of cases. So while the dominance of the shrub layer is generally not a valid approach to distinguish between better and worse ecological condition (for example, mature rainforests with a high and closed canopy normally have a little developed shrub layer, so a lack of the shrub layer does not always mean low quality), it may well be so for the restricted number of observations made in the study area. Likewise, the presence of electric fences is not a necessary trait of ecologically sound cattle-ranching systems; in the study area they might be a valuable indicator of the application of best practices in rangeland management. This underpins the potential of rapidly developed regression trees for variable selection and minimisation of time and money expenditure for sampling at the local or regional scale; however, for the development of generalisable models, expert knowledge and experience is indispensable (except when a huge database collected at a larger scale should be available).
The standard for the evaluation of the experts' models and the regression trees in this study were the personal evaluations made by the experts themselves. Experts may be wrong, however, and there exists the possibility that their models, based on explicit reasoning, reflect the situation in the real world better than their synthetic expert opinion does which might be susceptible to subjectivity and bias. The suggested procedure ''expert evaluation -regression-tree induction -sampling based on narrowed indicator-set'' could be made more robust by consulting a group of local experts and applying Delphi methodology (Bowles, 1999;Rowe and Wright, 1999;Linstone and Turoff, 2002) in the stages of primary variable selection and personal plot evaluation; regression-trees based on personal evaluation averaged across a number of experts will suffer less danger of being biased due to expert subjectivity which as we know can be considerable (Weisberg et al., 2008).

Conclusions
Knowledge engineering, i.e., the conversion of expert opinion into a formal model of decision making, was tedious and time consuming in comparison with the knowledge discovery approach, i.e., the automated induction of regression-trees. Moreover, the correspondence between regression tree output and expert opinion was considerably higher than the correspondence between expert opinion and the experts' own models, despite the trees used being only a subset of the primary variables. The application of regression tree induction, thus, has a high potential for the rapid development of indicators for narrowly defined ecological assessments as needed for decision-making at the local or regional scale and for the minimisation of sampling effort due to variable space reduction. Expert knowledge will still be indispensable for the development of generalisable models valid at larger spatial scales.