Skip to main content
Browse Subject Areas

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A decision support system for assessing management interventions in a mental health ecosystem: The case of Bizkaia (Basque Country, Spain)

  • Carlos R. García-Alonso ,

    Contributed equally to this work with: Carlos R. García-Alonso, Nerea Almeda, José A. Salinas-Pérez, Mencía R. Gutiérrez-Colosía, José J. Uriarte-Uriarte, Luis Salvador-Carulla

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Writing – review & editing

    Current address: Department of Psychology, Universidad Loyola Andalucía, Seville, Spain

    Affiliation Universidad Loyola Andalucía, Seville, Spain

  • Nerea Almeda ,

    Contributed equally to this work with: Carlos R. García-Alonso, Nerea Almeda, José A. Salinas-Pérez, Mencía R. Gutiérrez-Colosía, José J. Uriarte-Uriarte, Luis Salvador-Carulla

    Roles Conceptualization, Data curation, Formal analysis, Investigation, Writing – review & editing

    Current address: Department of Psychology, Universidad Loyola Andalucía, Seville, Spain

    Affiliation Universidad Loyola Andalucía, Seville, Spain

  • José A. Salinas-Pérez ,

    Contributed equally to this work with: Carlos R. García-Alonso, Nerea Almeda, José A. Salinas-Pérez, Mencía R. Gutiérrez-Colosía, José J. Uriarte-Uriarte, Luis Salvador-Carulla

    Roles Data curation, Investigation, Writing – review & editing

    Affiliation Universidad Loyola Andalucía, Seville, Spain

  • Mencía R. Gutiérrez-Colosía ,

    Contributed equally to this work with: Carlos R. García-Alonso, Nerea Almeda, José A. Salinas-Pérez, Mencía R. Gutiérrez-Colosía, José J. Uriarte-Uriarte, Luis Salvador-Carulla

    Roles Data curation, Investigation, Writing – review & editing

    Affiliation Universidad Loyola Andalucía, Seville, Spain

  • José J. Uriarte-Uriarte ,

    Contributed equally to this work with: Carlos R. García-Alonso, Nerea Almeda, José A. Salinas-Pérez, Mencía R. Gutiérrez-Colosía, José J. Uriarte-Uriarte, Luis Salvador-Carulla

    Roles Investigation, Writing – review & editing

    Affiliation Bizkaia Mental Health Services, Osakidetza-Basque Health Service, Biocruces Health Research Institute, Bilbao, Spain

  • Luis Salvador-Carulla

    Contributed equally to this work with: Carlos R. García-Alonso, Nerea Almeda, José A. Salinas-Pérez, Mencía R. Gutiérrez-Colosía, José J. Uriarte-Uriarte, Luis Salvador-Carulla

    Roles Conceptualization, Investigation, Writing – review & editing

    Affiliation ANU College of Health and Medicine, Australian National University, Canberra, Australia


Evidence-informed strategic planning is a top priority in Mental Health (MH) due to the burden associated with this group of disorders and its societal costs. However, MH systems are highly complex, and decision support tools should follow a systems thinking approach that incorporates expert knowledge. The aim of this paper is to introduce a new Decision Support System (DSS) to improve knowledge on the health ecosystem, resource allocation and management in regional MH planning. The Efficient Decision Support-Mental Health (EDeS-MH) is a DSS that integrates an operational model to assess the Relative Technical Efficiency (RTE) of small health areas, a Monte-Carlo simulation engine (that carries out the Monte-Carlo simulation technique), a fuzzy inference engine prototype and basic statistics as well as system stability and entropy indicators. The stability indicator assesses the sensitivity of the model results due to data variations (derived from structural changes). The entropy indicator assesses the inner uncertainty of the results. RTE is multidimensional, that is, it was evaluated by using 15 variable combinations called scenarios. Each scenario, designed by experts in MH planning, has its own meaning based on different types of care. Three management interventions on the MH system in Bizkaia were analysed using key performance indicators of the service availability, placement capacity in day care, health care workforce capacity, and resource utilisation data of hospital and community care. The potential impact of these interventions has been assessed at both local and system levels. The system reacts positively to the proposals by a slight increase in its efficiency and stability (and its corresponding decrease in the entropy). However, depending on the analysed scenario, RTE, stability and entropy statistics can have a positive, neutral or negative behaviour. Using this information, decision makers can design new specific interventions/policies. EDeS-MH has been tested and face-validated in a real management situation in the Bizkaia MH system.


Decision Analysis (DA) is a technique to support decision making avoiding pitfalls. DA aims to understand the decision-making processes, the factors involved in the way people make decisions and the procedures involved in choosing alternatives for maximizing the expected utility, the probability of achieving some specific goals or minimizing the decisional uncertainty [1].

The recommendations proposed by the World Health Organization (2018) [2] for Mental Health (MH) policy development highlight the need to help policy makers to reach deeper knowledge on service planning, but traditional approaches focusing on systematic reviews of evidence, policy briefs and better accessibility to data [3] do not suffice to guide decision making. In real world conditions, decision makers might approximate the consequences of a policy plan or specific management interventions (e.g., reallocation of workforce capacity or beds) in a very limited way. Additionally, the complexity, uncertainty, non-linearity, dimensionality and multiscalarity of the questions posed in mental healthcare planning [4] make it necessary to integrate an entire array of different disciplines, research fields and analysis techniques to develop usable and interoperable Decision Support Systems (DSS) in the real world.

DSS are interactive computer-based tools for improving decision-making processes and guiding decision makers in semi-structured or, sometimes, unstructured problem solving [5,6]. DSS integrate techniques from: statistics, clinical information systems, computational modelling, simulation and machine learning, among others [7]. Nevertheless, the capacitation processes for guiding users to make decisions based on DSS results are always complex in dynamic environments [8,9].

DSS based on artificial intelligence techniques [10], Bayesian networks [11] and simulation modelling [12] have been applied for decision-making in health care, in general, and in MH care, in particular, [1315]. Finally, DSS can assess the impact and/or effectiveness of MH policies [16], improving both the management of MH services [1719] and care provision [20].

The Efficient Decision Support—Mental Health (EDeS-MH) is a DSS based on Relative Technical Efficiency (RTE), Monte-Carlo simulation and artificial intelligence (a fuzzy inference engine prototype for interpreting standard “IF … Then” production rules) that has been developed following the systems thinking approach requirements.

RTE analyses the relationship that exists between the inputs (usually resources) consumed and the outputs (resource utilisation and outcomes) produced by a set of comparable Decision Making Units (DMUs) [21]. These DMUs interact in complex systems, such as the MH ones, in uncertain environments. The RTE is “relative” because it is obtained by comparing every DMU to each other. The DMU that shows the best input/output rates has a RTE = 1 (best efficiency), while the others have RTE scores between a [0, 1) range where an RTE = 0 means a complete inefficiency. The RTE techniques can analyse Small Health Areas (SHAs) (meso-level) as well as the entire MH systems, considering both the real (initial) situation “A” and the consequential potential situation “B” after acting the designed interventions.

Due to the environmental and the inner MH systems’ uncertainties, RTE is probabilistic. Recent findings have evidenced that stochastic RTE is a good indicator to assess the performance of MH services [22].

Regarding to the methodology used for analysing RTE, Data Envelopment Analysis (DEA) is a robust non-parametric technique for assessing performance of a set of comparable DMUs, introduced by Charnes, Cooper and Rhodes in 1978. Till now, DEA has been widely applied in many research fields [2325] including healthcare [2628]. Nevertheless, the RTE assessment of the performance of MH services is still a challenge [22,2931].

In order to solve main drawbacks of DEA (deterministic, non-standard input/output treatment, super-efficiency interpretation and the limitation of the number of inputs/outputs depending on the number of observations), original models were improved by Imprecise-DEA [32], order-alpha and order-m models [33], and Monte-Carlo simulation engines for introducing probabilistic risks into DEA models [22,34,35].

Monte-Carlo simulation is a statistical technique specifically designed to introduce the risk (as an approximation of the real system uncertainty) in almost any analysis [36]. This methodology uses pseudorandom numbers that have been algorithmically produced by computers to reproduce the predefined probability distributions that represent the behaviour of system variables [37].

Although it is stated that Monte-Carlo simulation can resolve two relevant DEA drawbacks–it is a discrete model (RTE is probabilistic) and the limitation on the number of inputs/outputs depending on the number of observations- [38,39] it has not been frequently applied.

This project is part of a long-lasting collaboration between the Bizkaia department of health (Osakidetza) and an international research consortium made by the PSICOST research association (Spain), the Universidad Loyola Andalucía, and the Australian National University to improve the knowledge base for regional evidence-informed planning in MH care. This collaboration has produced the Atlases of MH Care of Bizkaia [40], and Gipuzkoa [41], and different modelling strategies for DSS to assess different patterns of care in small areas using neuronal network analysis [42] and to produce league tables of the RTE of SHAs in the Basque Country based on Monte-Carlo DEA [22].

This study presents a novel hybrid DSS (EDeS-MH) in order to support consequential priority setting of shifted/relocated resources in a regional health care system, as a demonstration of its applicability for the assessment of management interventions and its usability to generate evidence-informed recommendations at the different levels of the health care ecosystem (micro, meso and macro).



Bizkaia (1,156,447 inhabitants) is one of the three provinces of the Basque Country (Spain). The health department in each province has full governance capacity and centralizes health care planning and provision [43]. The MH ecosystem in Bizkaia is organised in 19 SHAs with a reference community MH centre. These SHAs are arranged in four districts with a reference general hospital for acute admissions.

Vocabulary for the DSS

To overcome the impact of terminological variability in health system research and in the design of DSS, we have used a standard vocabulary based on the REFINEMENT glossary of terms for MH systems analysis [44]. We have added some new terms to define management interventions. Management is defined in the glossary as “the responsibility for and control of a company or organisation”.

A “management intervention” is here defined as an act of planning, organising, staffing, directing and/or controlling components of a health care ecosystem that utilises financial, human and material resources to achieve a defined health care goal. From a priority setting perspective [45], these management interventions can be classified as procedural (rules and steps to set priorities such as engagement, transparency and empowerment) and consequential (implementation of decisions and results of the priority setting such as efficiency analysis, reduction of resource waste and increase satisfaction). We have analysed consequential management interventions for shifting/reallocating resources at the micro (changes in an individual service), meso (SHAs) and macro levels (the Bizkaia MH care ecosystem). These management interventions can involve changes in the availability of services, placement capacity or workforce capacity. The obtained results can be considered relevant as a piece of knowledge for a better understanding of the evolution of the MH ecosystem.


The sociodemographic characteristics, the local service provision and the resource utilisation pattern have first been described for every SHA. To design useful and operable DSS, a series of steps should be followed within a systems thinking approach [46]: (a) describe the framework/model/values of the system being assessed; (b) describe the environmental characteristics of the health care ecosystem; (c) describe the agents operating in this ecosystem (consumers, professionals, teams and organisations); (d) describe the main interactions of the agents within the system; (e) apply advanced methods for knowledge discovery from data; and (f) use hybrid models that integrate operational and statistical techniques (for data analysis) as well as artificial intelligence-based techniques in order to incorporate previously formalized expert knowledge (production rules).

We combined qualitative and quantitative approaches to generate the critical knowledge to develop the EDeS-MH following these recommendations. First, we developed a Basic Mental Health Community Care Model (B-MHCCM) that can be applied to benchmarking and efficiency analysis in Spain [47].

Second, we conducted a detailed analysis of the MH care ecosystem of Bizkaia that was incorporated into the Atlas de Salud Mental de Bizkaia [40]. This included a description of the areas and their social and demographic characteristics. We also provided a general description of the main agents operating in this system: 1) consumers: population with mental disorders in contact with the MH services; 2) professionals; 3) care teams; and 4) organisations. The DESDE-LTC system (Description and Evaluation of Services and DirectoriEs for Long Term Care) [48] was used for the standard description of the MH service provision (professionals, care teams and organisations). DESDE-LTC uses an international classification for coding care teams in services [49] to allow comparisons across different jurisdictions. Care Teams or “Basic Stable Inputs of Care” (BSICs) are coded according to the following main types of care: “R” Residential services, “D” Day care services and “O” Outpatient services. The description included the service availability, its placement capacity (number of beds or places in day care) and the workforce capacity measured in full-time equivalents. This information was aggregated at meso (19 SHAs) and macro (Bizkaia province) levels.

Third, the interactions of different agents in the system were analysed by the resource utilization of hospital and outpatient services obtained from databases of the department of health in this jurisdiction [40].

Fourth, we incorporated experts in the knowledge discovery from data. This expert group included, for Bizkaia, the senior managers (2 people, psychiatrists), MH services managers (depending on the meeting, from 3 to 5 people; psychologists and psychiatrists) and information system managers (depending on the meeting, from 1 person to 3 people). For the design of the scenarios and for the interpretation of the variable values, experts from the Catalonia Autonomous Community and Gipuzkoa (Basque Country) were also included in the discussion groups (only MH managers; psychologists and psychiatrists). In a previous work [50] we proposed an approach called “Expert-based Collaborative Analysis” (EbCA) to generate a minimum metadata set, the EDeS-MH core, that included 57 variables for describing the structure of the MH system in 19 DMUs (SHAs). The experts followed an information-guided approach using visualisations of these variables to generate estimates of the range of appropriateness for every indicator. Using this information, 43 variables are identified as inputs (Group C-Quality of care in the REFINEMENT glossary): availability, placement capacity and workforce capacity. The remaining 14 variables were labelled as outputs (Group B-Service Utilization): hospital discharges, readmissions, average length of stay, utilization of acute and non-acute day care and, finally, utilization of non-acute outpatient care (prevalence, incidence and frequency of visits). The minimum dataset for this study is available in the Dryad Digital Repository:

Design of the scenarios

The concept of scenario refers to different meaningful combinations of variables (inputs and outputs). By using the EbCA model (for checking the model structure and details see [50]), experts in MH care planning and management and service researchers designed 15 scenarios for assessing the RTE of SHAs. The first 9 described specific main types of care for each DMU: Scenario 1 (S1) acute hospital care; Scenario 2 (S2) non-acute hospital care; Scenario 3 (S3) residential community care; Scenario 4 (S4) day care health related; Scenario 5 (S5) day care; Scenario 6 (S6) outpatient care; Scenario 7 (S7) placement capacity; Scenario 8 (S8) placement capacity health related; and Scenario 9 (S9) health related workforce capacity. The remaining scenarios were designed as combinations of different main types of care according to the B-MHCCM [47]: Scenario 10 (S10) included the workforce capacity for residential, day and outpatient care; Scenario 11 (S11) included a mix of residential and health related day care with outpatient care; Scenario 12 (S12) combined acute hospital, community residential, day and outpatient types of care; Scenario 13 (S13) combined placement capacity of acute hospital, day health-non related health and outpatient care; Scenario 14 (S14) was a mix of day and residential placement and workforce capacity of residential-outpatient care; finally, Scenario 15 (S15) included a combination of health/community residential placement capacity and workforce capacity of residential-outpatient care (see Table 1 for an in-depth description of selected scenarios). For each DMU and for the MH system, the RTE was probabilistically assessed from these 15 different points of view (scenarios) in order to provide a holistic perspective of the system performance.

Table 1. Descriptions of the scenarios directly affected by the proposed interventions.

Design of the management interventions

To assess the usability of the EDeS-MH, three management interventions that involved shifting staff across different services in different SHAs of the Bizkaia MH system have been tested. These mesomanagement interventions were designed by senior managers of the Bizkaia MH system to provide a better community MH care.

Mesomanagement intervention 1: Reassigning a psychologist from an outpatient care service (coded by letter O, DESDE-LTC) located in SHA-1 (Uribe) to a day hospital (code D41) located in SHA-2 (Durango). This mesomanagement intervention modifies the structure of a day hospital that provides MH care not only to the Durango SHA but also to the Basauri, Bermeo, Galdakao and Gernika SHAs.

Mesomanagement intervention 2: Reassigning a psychiatrist from an outpatient care service (code O) located in SHA-1 (Uribe) to an outpatient service located in SHA-3 (Sestao).

Mesomanagement intervention 3: Reassigning a psychiatrist from an outpatient care service (code O8-O10) located in SHA-4 (Ercilla) to an outpatient service located in SHA-5 (Barakaldo).

The mesomanagement interventions 1, 2 and 3 directly modified many variable values in specific SHAs (Table 2). Due to the structural modification of the day hospital located in Durango, the mesomanagement intervention 1 modified some secondary variable values (causal effect) in other SHAs. In addition to these variables, the budget in all of the affected areas had to be correspondingly modified.

These modifications generated real expectations (in terms of expected outcomes) for decision makers (Table 2).

Randomization of the original data (understanding the environmental and structural uncertainties)

To integrate into the model the uncertainties of the MH systems, the majority of the original dataset was randomized by using standard triangular statistical distributions (1). (1) being, xi the original value of the variable and wtl and wtr the weights that define the range of the triangular distribution (wtl = 0.9 and wtr = 1.1). For example, if the variable value of TD1 is 0.4398 in Bermeo SHA, the corresponding triangular distribution is: minimum value 0.9x0.4398 = 0.3958, modal value 0.4398 and maximum value 1.1x0.4398 = 0.4838, T[0.3958, 0.4398, 0.4838]. This procedure is deeply explained in a previous work [22].

For the variable UFrecO8+O10 (output in many scenarios) in Barakaldo, Ercilla, Sestao and Uribe, uniform (rectangular) statistical distributions (2) were used: (2) being, wur the weight that define the range of the uniform distribution (wur = 1.1). For example, if the variable value of UFrecO8+O10 in Bermeo is 439.12, the corresponding rectangular (uniform) is: minimum value 439.12 and maximum value 1.1x439.12 = 483.03, U[439.12, 483.03]. This procedure is deeply explained in a previous work [22].

In the end, the original 19×57 dataset (19 SHAs and 57 variables) was transformed into a 19×57 statistical distributions (triangular or uniform) matrix. This structure fed, in an iterative process, the Monte-Carlo simulation engine for the design of the final dataset before carrying out the variable values interpretation (fuzzy inference engine) and RTE assessment. Once RTE scores were calculated for a specific dataset, results were stored in a solution pool and the process started again by selecting at random a new dataset using the Monte-Carlo simulation engine.

The proposed mesomanagement interventions directly involved 47 variable values (statistical distributions): 4.34%, minor changes in the dataset structure. Nevertheless, the numbers of affected SHAs 8/19 (42.11%) and scenarios 11/15 (73.33%, S4, S6-S9, S11-S15) were very relevant. Regarding the workforce capacity, only three persons (2 psychiatrists and 1 psychologist) were reassigned (0.32% of the total personnel of Bizkaia’s MH system).

The fuzzy inference engine prototype

Once the Monte-Carlo simulation engine selects at random a variable value xi according to the statistical distribution selected for Xi (triangular or uniform), it is interpreted according to the B-MHCCM. So, xi is transformed into as a result of a function: that represents a recursive relationship (the interpreted value depends on the original value, once and are defined: the range for interpreting the appropriateness degree of xi). This function f is different at the left hand side of , within the range and at the right hand side of , but it is always linear and monotone and it increases/decreases the original value xi according the explicit expert knowledge formalized in the B-MHCCM. For example, a variable value for TR12 (input, availability of community residential care–rate per 100.000 inhabitants) can be considered “appropriate” within the range (0.9753, 1.0979) and inside it the more centred the value the more appropriate. In this specific case, the original value is transformed by the linear monotone function: , being and .

Under real conditions, and depend on the values of other variables in the dataset according to a causal model that represents the B-MHCCM. This circumstance means that the interpretation of the variable Xi depends on the value of the rest of the variables. The fuzzy inference engine prototype takes into consideration that Residential care (R) provision depends on Day care (D) and Outpatient care (O) provision. This relationship is defined through linking and for each variable in R to the values of D and O calculated by the fuzzy inference engine in a range [0, 100] (0: no availability of the corresponding services and 100: the availability is maximum). These D and O values are determined by using the values of the respective professional, places/beds and availability variables (rates per 100.000 inhabitants).

Each variable in the fuzzy inference engine prototype is limited to 5 normal fuzzy sets with sum equal to 1 in a range defined by a minimum (usually 0) and a maximum (defined by the experts) specific for each variable. The semantic labels associated to each fuzzy set are: very little (Z function), little (triangular), standard (triangular), much (triangular) and very much (Z function). The fuzzification process is standard once the original value xi is known. Due to the structure of the fuzzy sets, each xi can only have 1 or 2 membership degrees greater than cero and their sum is always equal to 1. In all the cases, the standard product-sum gravity method is used for the defuzzyfication process [51].

The structure of the production (fuzzy) rules is standard. For example, IF O is “Very little” and D is “Little” THEN should be “Much”. According to this rule, and will be displaced to the right. The fuzzy inference engine firstly calculates in a range defined by the experts. In this range, the prototype takes into consideration 5 fuzzy sets (with the same semantic labels: very little, little, standard, much and very much). Once the new value is calculated (defuzzyfication process) the new right limit for the variable value interpretation is calculated as follows: . In conclusion, the interpretation of each variable in R depends on D and O provision.

Relative Technical Efficiency (RTE)

RTE can be assessed from both a constant or a Variable Returns to Scale (VRS). The former approach implies that any input/output variation has a proportional output/input change [21]. In the latter, which is more realistic, this does not occur [52].

RTE can also be calculated from an input or output point of view (input-oriented or output-oriented techniques) [53]. In the first approach, the model tries to minimize the input consumption when maintaining the output production constant and is used when the situation is focused on input management. In the second, it tries to maximize the output production when maintaining the input consumption constant and is used for output management purposes.

The hybrid EDeS-MH DSS combines the following methodologies:

  • Monte-Carlo simulation (in order to carry out this technique a Monte-Carlo simulation engine was designed and developed) for analysing the environmental and structural uncertainties of the MH system.
  • Artificial intelligence by using “IF…Then” rule-based criteria to introduce the expert´s opinion, based on the B-MHCCM.
  • DEA, both input and output-oriented and variable returns to scale for RTE assessment.
  • Basic Statistics: Kolmogorov-Smirnov test, T-test and Levene´s Test for the assessment of the intervention impact.
  • Stability and Entropy for analysing the pre and post-intervention stability and uncertainty of the MH system.

DSS structure (pseudocode)

In order to assess the RTE of each scenario, both the fuzzy inference engine (to interpret the variable values according to the B-MHCCM) and the operational model (to calculate RTE scores) are embedded in the Monte-Carlo simulation engine (to integrate the inner uncertainty of the environment). The process is iterative and RTE scores are saved in a solution pool. The proposed pseudocode is:

1: DSS procedure

2:  The solution pool (SP) is cleaned

3:  For j = 1 to nsim do → being j the simulation number and nsim the maximum number of simulations

4:   For i = 1 to nsce do → being i the scenario number and nsce the maximum number of scenarios

5:    For k = 1 to ndmu do → being k the SHA (DMU) number and ndmu the number of SHA

6:     Original dataset Sijk → the Monte-Carlo simulation engine selects at random a value for each simulation, variable and SHA k according to their specific statistical distributions.

7:     Calculate Dijk and Oijk → the fuzzy inference engine determines the day care and outpatient care fuzzy sets and membership degrees for each SHA

8:     Calculate and → the fuzzy inference engine determines the range for variable value interpretation according to the B-MHCCM

9:     Interpreted dataset Iijk → the original dataset Sijk are interpreted by the fuzzy inference engine according to the B-MHCCM

10:      Calculate RTEijk scores → variable returns to scale Data Envelopment Analysis (DEA) model (both input and output-oriented)

11:      Save RTEijk scores in the solution pool

12:    end for

13:   end for

14: end for

15: end procedure

This pseudocode is published in:

For each SHA and scenario (15), 500 simulations were carried out. Therefore, each scenario was analysed by 9,500 simulations (19×500) and the global RTE of the Bizkaia MH system was assessed by 142,500 simulations (19×15×500). This process was repeated 4 times using the DEA-VRS assessing (i) pre-management interventions and input-oriented RTE, (ii) pre-management interventions and output-oriented RTE, (iii) post-management interventions and input-oriented RTE and (iv) post-management interventions and output-oriented RTE (the whole process is described in Fig 1).

Fig 1. Analysis of an organizational intervention: A flowchart.

In all the assessed cases, the calculated RTE is a probabilistic distribution (Fig 2) that can be analysed by both a frequency analysis and statistical estimators:

  • Probability of being efficient (PRTE⩵1).
  • Probability of being weakly efficient (PRTE≅1)
  • Probability of being inefficient (PRTE<1).
  • Efficiency average .
  • Efficiency standard deviation .
  • Efficiency error .
  • Efficiency error .
  • Probability of having an RTE score greater than rte1 and lower than rte2.
  • Probability of having an RTE score greater than rte3.
Fig 2. An example of resulting RTE statistical distribution for a specific DMU.

The first interval (at the left hand side) is [0, 0.05), the second is [0.05, 0.1), …, the antepenultimate is [0.95, 1), the penultimate represents the simulations where RTE = 1 but the sum of slacks are different to zero (weakly efficient) and, finally, the last interval represents the simulations where RTE = 1 and the sum of the slacks is equal to zero (completely efficient).

After an intervention or policy, the initial RTE statistical distribution can vary, significantly or not, depending on the resulting impact.

Stability and entropy analysis

The resulting RTE statistical distributions were additionally analysed using the stability and entropy indicators. Stability aims to assess (in a [0, 100] range; 0: completely unstable and 100: completely stable) if the RTE statistical distribution is more or less sensitive to change when the input/output values change. A system can be considered stable when data changes (that represent structural changes) do not significantly modify RTE scores (the efficiency of the system is not sensitive to structural changes). In contrast, the system is unstable when small data changes vary the RTE scores dramatically (the efficiency of the system is highly dependent on small structural changes). DMU final stability (stab) has two components.

  1. Stability due to the number of RTE intervals with probability (interval stability of DMU i, intstabi). A DMU is completely stable (intstabi = 100) if all the probability is concentrated in only one RTE interval. A DMU is completely unstable (intstabi = 0) when the probability is distributed in all the RTE intervals or it is concentrated in the first and the last interval.
  2. Stability due to the concentration of the density of probability (density stability of DMU i, denstabi). A DMU is completely stable (denstabi = 100) when all (or the majority) of the probability is concentrated in only one RTE interval, independent of where the rest is distributed. A DMU is completely unstable (denstabi = 0) when all (or the majority) of the probability is concentrated in all or a much separated RTE intervals.

The interval stability of the DMU i (intstabi) is calculated in the following way (3): (3) where ninti is the number of RTE intervals in the frequency analysis that have probability greater than zero and inttot is the total number of intervals defined in the frequency analysis.

The density stability of the DMU i (denstabi) is calculated in the following way (4): (4) where acprobi is the first accumulated probability strictly greater than a predefined probability (prob), nintprobi is the number of intervals needed to reach acprobi, minln is the minimum feasible value for the logarithm, and maxln is the maximum feasible value for the logarithm.

The final stability of the DMU i (stabi) results of a weighted sum (5) of interval stability and density stability is as follows: (5) where wint is the weight selected for the interval stability and wden is the corresponding weight for the density stability. For a better understanding of the results (Figs 3 and 4), it could be necessary (but it is not mandatory) that: wint + wden = 1.

Fig 3. Some examples of different RTE statistical distributions and their corresponding stabilities (inttot = 22, prob = 0.8, minln = 1.2909 and maxln = 4.6051).

For the example in E: ninti = 5, acprobi = 0.88, nintprobi = 3, wint = 0.5 and wden = 0.5. Finally, the resulting stabE = 72.04 (likely stable).

Fig 4. Interval stability (intstabE = 80.95, circle with the darker background), density stability (denstabE = 63.13, circle with the light grey) and final stability (stabE = 72.04, greater circle without any background colour) for the DMU shown in the example (DMU E) in Fig 3 (wint = 0.5 and wden = 0.5).

(1) Small changes in data values (generated by the Monte Carlo simulation engine or real ones) can change RTE scores dramatically. Decision makers should be awarded. (2) Small changes in data values (generated by the Monte Carlo simulation engine or real ones) can change RTE scores a lot. Decision makers should be awarded. (3) Changes in data values (generated by the Monte Carlo simulation engine or real ones) could change RTE scores not so much. Decision makers should be awarded. (4) Changes in data values (generated by the Monte Carlo simulation engine or real ones) do not change RTE scores a lot. (5) Changes in data values (generated by the Monte Carlo simulation engine or real ones) do not change RTE (changes, if exist, are very small).

The DMU entropy (entri) is calculated using (6), the Shannon’s entropy, and represents the degree of uncertainty included in the RTE statistical distribution [54]: (6) where nint is the number of intervals assessed in the frequency analysis and freqh is the relative frequency in the interval h.


Whole RTE assessment: Initial situation (“A”)

RTE results.

Entire System (pre-management interventions). From an input management perspective (input-oriented DEA-VRS), the Bizkaia MH system is likely efficient. The probability of being efficient (PRTE⩵1) is 0.2040, the RTE average is 0.7831, and the probability of having an RTE greater than 0.75 is 0.6240 (Table 3, “Pre-Interventions” column).

Table 3. Impact of the meso-interventions (input-oriented results).

From an output management perspective (output-oriented DEA-VRS), the system is efficient. The PRTE⩵1 = 0.2188, , and the probability of having an RTE greater than 0.75 is 0.8805 (Table 4, “Pre-Interventions” column).

Table 4. Impact of the meso-interventions (output-oriented results).

Fifteen Scenarios (pre-management interventions). Concerning the input management, PRTE⩵1 varies in a [0.1447, 0.3113] range for workforce capacity health related (S9) and placement capacity (S13). The oscillates within [0.6128, 0.9357] for placement and workforce capacity (S15) and day care health related (S4). The (S4) shows the greatest probability (0.9260) of having an RTE score higher than 0.75, while the (S15) has the lowest one (0.3289) (Table 3, “Post-Interventions” column). From an output perspective, the mix placement capacity (S13) scenario has the highest PRTE⩵1 = 0.3445, while the lowest corresponds to workforce capacity health related (S9) PRTE⩵1 = 0.1616. The maximum is reached by (S11), which combines residential and day care health related plus outpatient care. In contrast, (S6) outpatient care has the lowest . Concerning the probability of having an RTE higher than 0.75, (S6) shows the lowest value of 0.7464, while (S12) residential and health related placement capacity and outpatient availability has the highest at 0.9573 (Table 4, “Post-Interventions” column).

Stability and entropy of the system.

Entire System (pre-management interventions). According to the input management, the interval stability (intstab) is very poor (small changes in data values can change RTE scores dramatically). Its density stability (denstab) is poor (small changes in data values can change RTE scores a lot). Finally, the stability of the entire system (stab), considering wint = 0.5 and wden = 0.5, is very poor (Fig 5 and Table 5, “pre” column).

Fig 5. Global stability of the system: Initial situation (stab = 0.5 × intstab + 0.5 × denstab).

Table 5. Scenario stability and entropy (in brackets the entropy in percentage up to its maximum).

The output management results show that the interval stability (intstab) is very poor, while the density stability (denstab) is intermediate (small changes in data values do not change RTE too much). The final stability (stab) is poor considering again that wint = 0.5 and wden = 0.5 (Fig 5 and Table 5, “pre” column).

According to the input management, the entropy of the global system is very high, 3.83 (85.82% up to the feasible maximum). In the output management analysis, it is also high, 3.21 (72.01% up to the feasible maximum). These results indicate that the Bizkaia MH system can be easily improved by selecting appropriate interventions (Table 5, “pre” column).

Fifteen Scenarios (pre-management interventions). From an input management perspective, the greatest stab = 24.08 corresponds to (S4) delivery of day care health related, while the lowest stab = 5.87 is for (S15) combination of placement capacity and workforce capacity for residential, day and outpatient care. In contrast, output management (S8) placement capacity health related care has the highest stab = 23.61, while (S6) outpatient care has the lowest stab = 12.50 (Table 5, “pre” column).

From an input management point of view, the Shannon’s entropy oscillates within [4.11 (92.09%), 2.37 (53.21%)], respectively, for (S15) places in the hospital and community and workforce capacity for both residential and outpatient care and (S4) day care health related. From an output management perspective, it varies within [2.58, 3.25] for (S12) hospital-community residential, day health related and outpatient care and (S6) outpatient care. Again, there are many opportunities to improve the MH System of Bizkaia (Table 5, “pre” column).

RTE assessment: The impact of the mesomanagement interventions (“B”)

RTE results.

Impact on the entire System (post-management interventions. Focusing our attention on input management, after the proposed three interventions, the increases slightly, and the difference (pre-post) is statistically significant (two-tailed p = 0.002, α = 0.05) while the variances cannot be considered equal (Levene´s test p < .001). RTE statistical distributions (pre and post) can be considered equal (Kolmogorov-Smirnov, α = 0.05 and 0.01; Table 3, “Post-Interventions” column). The output-oriented results indicate that decreases slightly, but the difference is not statistically significant (two-tailed p = 0.387, α = 0.05) and the variances can be considered equal (Levene´s test p = 0.356). Finally, the statistical distributions can be considered equal (Kolmogorov-Smirnov, α = 0.05 and 0.01; Table 4, “Post-Interventions” column).

The three management interventions cause a small but positive RTE increase in the service performance oriented to the input management. However, the situation from an output management perspective remains constant.

Impact on the fifteen scenarios (post-management interventions). Regarding the input management, the PRTE⩵1 increases in (S7), (S11), (S12) and (S13). The first one, placement capacity, shows the highest one (5.30%). However, in (S4), (S8), (S9), (S10), (S14) and (S15), PRTE⩵1 decreases, with (S15) places of hospital and community in addition to workforce capacity of outpatient care having the worst one (-2.63%). In (S6) outpatient care, PRTE⩵1 remains constant.

The increases in (S4), (S7), (S11), (S13), (S14) and (S15); in (S7) placement capacity of residential and day care, it increases to 2.34%. In (S10) workforce capacity total, remains constant, while in (S6), (S8), (S9) and (S12), it decreases, with (S12) placement capacity for residential care in the hospital and community plus day and outpatient care having the worst one.

Looking at the probability of having an RTE score higher than 0.75, it increases in all of the scenarios, except (S8), (S9), (S10) and (S12). (S14) shows the highest percentage (3.06%) and (S8) the worst after meso-interventions (-2.11%; Table 3).

Taking into account the output-oriented results, PRTE⩵1 is lower after the interventions in all of the scenarios, except (S6) outpatient care (3.49%) and (S13) placement capacity (2.51%). (S9) shows the highest negative impact (-5.08%).

increases in (S6) outpatient care and (S14) placement capacity and decreases in the remaining scenarios, with (S9) workforce capacity health related having the worst (-0.37%).

The probability of having an RTE score higher than 0.75 decreases in six of the eleven scenarios (S4, S7, S8, S9, S11 and S12), with (S12) being the most affected (-0.51%). In the remaining scenarios, it increases, and (S14) shows the highest percentage (0.66%; Table 4).

Stability and entropy of the system.

Impact on the entire System (post-management interventions). In the input orientation and after the proposed interventions, the interval stability (intstab) remains constant, but the density stability (denstab) and stab have a slight increase (0.1%) that is not statistically significant. From an output management perspective, intstab, denstab and final stability (stab) remain constant (Table 5). The input-oriented results show that the entropy decreases (-0.26%), which is a positive impact. However, from an output management view, the entropy increases a bit (0.1%) (Table 5).

Impact on the fifteen scenarios (post-management interventions). The results oriented to input management show that (S11) places for hospital and community care and availability of outpatient care has the highest stab increase (12.80%), while (S8) places health related shows the highest decrease (-8.37%). However, output management analysis evidences that (S14) placement and workforce capacity for residential, day and outpatient care has the highest stab increase (0.65%), while (S8) reaches a relevant decrease (-0.42%; Table 5).

The input-oriented results show that the greatest negative variation (it increases) of the entropy corresponds to (S8) with 1.23%. However, in (S7) placement capacity, it decreases (positive impact) by -2.30%. Form an output orientation perspective, (S8) shows the highest increase (1.61%) again, while in (S14), the entropy significantly decreases (-1.63%; Table 5).


EDeS-MH allows decision makers to assess the RTE of the MH system for analysing the impact of potential management interventions and policies prior to their becoming real. This DSS is able to guide MH care managers and planners in designing evidenced-informed interventions as well as policies, reducing the risk associated with decision-making.

From a hybrid methodological approach, the Monte-Carlo simulation engine allows for the incorporation of the uncertainty of the real system into the DSS [22]. By using DEA models, both the resource use and the outcome production of SHA can be optimized. The utilization of the DESDE-LTC codification system [48] allows for the standardization, comparison and evaluation of MH systems, considering their main types of care provided and avoiding the terminology variability [44]. Finally, the fuzzy inference engine prototype integrates expert knowledge in the operational model [22].

In the initial situation of the entire Bizkaia MH system, the RTE indicators show that there is a relevant opportunity for improving the system performance from both input (resources) and output (outcomes) management points of view. On average, its RTE is 0.7831 and 0.8812, respectively, but it does not exactly mean that input management can be improved by 21.69% (maintaining constant the output production) or, otherwise, by 11.88% (maintaining constant the resource consumption). This is because of the inner variability of the system that is described by the proposed scenarios. In this case, the model implications reveal that planners and managers of the MH system of Bizkaia took into consideration the results obtained in the present study. They have approved the feasibility of the model for supporting decision-making processes and resource allocation.

The EDeS-MH allows decision makers to study the most likely specific consequences of interventions and policies on selected scenarios as well as to design new potential ones. This procedure is critical because, for example, a potentially neutral intervention at a global scale can be the result of a balance of negative and positive effects on real scenarios. By using this knowledge, decision makers can modify their proposals or design additional and corrective interventions on specific scenarios in a creative way, overcoming old ideas, e.g., for increasing the efficiency of a MH system, it is required to reduce the amount of resources used [5557]. Expert knowledge (included in EDeS-MH) highly penalizes dramatic decreases in the inputs away from a specific minimum established by the B-MHCCM. Another old fashioned idea is that the efficiency of MH systems can be improved by increasing the level of outputs produced [55,58]. Again, expert knowledge identified non-standard outcomes where an uncontrolled increase away from another specific level–defined by the B-MHCCM- must also be penalized. The B-MHCCM understands these phenomena and can be used for a better understanding of the behaviour of complex MH systems under uncertainty.

From the perspective of the entire system, the proposed management interventions can be considered slightly positive (input management) and neutral (output management). Obviously, they imply very small changes in our real system (only a few professionals could be shifted and reassigned), so no dramatic effects were expected. However, the stability and Shannon’s entropy are very poor, so small changes in data values (due to an intervention or policy) can have a relevant impact, sometimes negative, in the efficiency of the entire system.

The analysis of the specific scenarios that are directly impacted by the management interventions showed that, in the input orientation, (S4) showed the highest and probability of having an RTE higher than 0.75 as well as a relative adequate stability and entropy. The performance of this scenario increased as a consequence of deinstitutionalization policies and the change in care provision from the hospital to the community, promoting the importance of day care [59,60].

The present findings support that it is possible to deliver an efficient day care in the community.

In (S7), (S11) and (S13), all of the indicators become better after the management interventions. These scenarios combine the placement capacity and availability of residential, day and outpatient care with utilization indicators (outcomes). For this reason, these decisions can be considered positive because the three main types of care in MH are combined in a balanced way.

From an output management perspective, (S11) showed the highest , while (S12) has the highest probability of having an RTE higher than 0.75 as well as the best stability and entropy. These scenarios combine residential, day and outpatient care, so it is possible to provide an efficient delivery, supported by an appropriate balance between hospital and community care [61,62].

However, (S6) outpatient care showed an improved situation after the mesomanagement interventions. In addition, (S14) evidenced the better stability and entropy improvements. Therefore, after carrying out the interventions, the provision of outpatient care improved.

Currently, the EDeS-MH has some drawbacks related to the difficulty of modelling interventions/policies in complex and interrelated MH systems under conditions of uncertainty. Decisions have to be translated into data variations (usually in the input variables) and expectations (usually in the output variables). For a specific intervention, decision makers can easily identify first level variations for the inputs (the variables that are directly impacted by the intervention; for this paper: the workforce variations due to the reassign process), but they have difficulties in identifying and assessing the expected consequences in both the outcomes and second, third, etc., level variables–inputs or outputs- (that modify their values as a cause of the primary changes, for example, data variations due to the hospital structural changes) according to a causal model. The EbCA model [50] is used to guide this process.

There were some limitations in our study. The analysis did not included 65 beds for long-term care financed by the Department of health of Bizkaia but placed in another province. These beds where part of a nested system within the MH system of Gipuzkoa so it was agreed with the experts from the public agency to exclude this facility from the counting of service availability and placement capacity in the Bizkaia MH care system. Bizkaia is a relative small province in the Spanish MH system with a centralised management and, due to that, the procedure have to be repeated for other provinces or regions in order to increase the variability. In the RTE assessment the majority of the statistical distributions were triangular (see section “Randomization of the original data” for a better understanding of their structure), this distribution can be considered very appropriate for this case but other (trapezoidal, uniform, etc.) can be also used, also with different weights. The expert-based ranges for data values interpretation (according to the B-MHCCM) as well as other parameters can be modified in order to check the DSS sensitivity to these changes. The DSS cannot yet interpret complex relationships between variables in day and outpatient care but it is ready to manage causal relationships when residential care variables are considered the “effect” of day and outpatient provision. All the variables as well as D (day care) and O (outpatient care) constructs are analysed taking into consideration only 5 fuzzy sets. In order to compare RTE statistical distributions, additional non-parametric tests can be also included in the DSS. In order to analyse the viability of the DSS, only the impact of meso-management interventions was assessed; macro-management interventions (or complex policies) can also be taken in consideration for complex viability analysis.

In conclusion, both the EDeS-MH and the procedure followed work properly and allow decision and policy makers to analyse and propose new alternatives that will have an impact on a MH system of selected interventions and policies prior to their becoming real.


Decision makers reported that the EDeS-MH is useful for helping them to (i) structure their minds in designing new interventions (i.e., to describe real problems, to: identify critical variables that are directly concerned, to identify secondary variables involved by using causal reasoning and, finally, to assess their expected outcomes–variables and amounts); (ii) understand the behaviour of the global system as well as of the corresponding SHAs in specific scenarios; (iii) identify critical scenarios that could have a negative reaction; (iv) identify post-intervention improvements in specific areas and scenarios; and, finally, (v) design nested-interventions (policies) from a long-term perspective.


We would like to thank the managers of the MH system of Bizkaia, particularly Carlos Pereira, Enrique Pinilla and José Uriarte, and the planners of the MH system of Gipuzkoa, highlighting Álvaro Iruin and Andrea Gabilondo, for the provision of data and support to develop the present study. The present research study is frame-worked in the REFINEMENT Spain project (Project PI15/01986), Carlos III Health Institute (Health Ministry).


  1. 1. Parnell GS, Bresnick TA, Tani SN, Johnson ER. Handbook of Decision Analysis [Internet]. Hoboken, NJ, USA: John Wiley & Sons, Inc.; 2013.
  2. 2. World Health Organization. Mental health policy, planning & service development [Internet]. 2018 [cited 1 Jan 2018].
  3. 3. Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). 13. Preparing and using policy briefs to support evidence-informed policymaking. Heal Res Policy Syst. 2009;7. pmid:20018103
  4. 4. Salvador-Carulla L, Haro JM, Ayuso-Mateos JL. A framework for evidence-based mental health care and policy. Acta Psychiatr Scand. 2006;111: 5–11.
  5. 5. Kaklauskas A. Biometric and intelligent decision making support. Kacprzyk J, Jain LC, editors. Springer; 2015.
  6. 6. Keen PGW. Decision support systems: The next decade. Decis Support Syst. 1987;3: 253–265.
  7. 7. Martin-Sanchez F, Aguiar-Pulido V. Chapter 9—Analytics and Decision Support Systems in Global Health Informatics. In: de Fátima Marin H, Massad E, Gutierrez MA, Rodrigues RJ, Sigulem DBT-GHI, editors. Academic Press; 2017. pp. 195–217.
  8. 8. Martins DML, de Lima Neto FB. Hybrid Intelligent Decision Support Using a Semiotic Case-Based Reasoning and Self-Organizing Maps. IEEE Trans Syst Man, Cybern Syst. 2017; 1–8.
  9. 9. Chan SH, Song Q, Sarker S, Plumlee RD. Decision support system (DSS) use and decision performance: DSS motivation and its antecedents. Inf Manag. 2017;54: 934–947.
  10. 10. Suzuki K, Chen Y. Artificial Intelligence in Decision Support Systems for Diagnosis in Medical Imaging. 1st ed. Springer Publishing Company, Incorporated; 2018.
  11. 11. Hargrave C, Deegan T, Bednarz T, Poulsen M, Harden F, Mengersen K. An image-guided radiotherapy decision support framework incorporating a Bayesian network and visualization tool. Med Phys. United States; 2018;45: 2884–2897. pmid:29772061
  12. 12. Mielczarek B, Uzialko-Mydlikowska J. Application of computer simulation modeling in the health care sector: A survey. Simulation. 2012.
  13. 13. Valenza G, Gentili C, Lanata A, Scilingo EP. Mood recognition in bipolar patients through the PSYCHE platform: preliminary evaluations and perspectives. Artif Intell Med. Netherlands; 2013;57: 49–58. pmid:23332576
  14. 14. Constantinou AC, Freestone M, Marsh W, Coid J. Causal inference for violence risk management and decision support in forensic psychiatry. Decis Support Syst. Elsevier B.V.; 2015;80: 42–55.
  15. 15. Heeg B, Buskens E, Knapp M, van Aalst G, Dries PJT, de Haan L, et al. Modelling the treated course of schizophrenia: development of a discrete event simulation model. Pharmacoeconomics. New Zealand; 2005;23 Suppl 1: 17–33. pmid:16416759
  16. 16. Silverman BG, Hanrahan N, Bharathy G, Gordon K, Johnson D. A systems approach to healthcare: agent-based modeling, community mental health, and population well-being. Artif Intell Med. Netherlands; 2015;63: 61–71. pmid:25801593
  17. 17. Bernard JC, Amir FC, Hosios J, Rousseau JM. Experimental: A simulation of the distribution of services to mentally deficient children. Behav Sci. John Wiley & Sons, Ltd; 1977;22: 356–366.
  18. 18. Kuno E, Koizumi N, Rothbard AB, Greenwald J. A Service System Planning Model for Individuals With Serious Mental Illness. 2005;7. pmid:16193999
  19. 19. Mohan L, Muse L, McInerney C. Managing smarter: a decision support system for mental health providers. J Behav Heal Serv Res. Springer; 1998;25: 446–455.
  20. 20. Young AS, Mintz J, Cohen AN, Chinman MJ. A Network-Based System to Improve Care for Schizophrenia: The Medical Informatics Network Tool (MINT). J Am Med Inform Assoc. American Medical Informatics Association; 2004;11: 358–367. pmid:15187072
  21. 21. Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. Eur J Oper Res. 1978;2: 429–444.
  22. 22. Torres-Jiménez M, García-Alonso CR, Salvador-Carulla L, Fernández-Rodríguez V. Evaluation of system efficiency using the Monte Carlo DEA: The case of small health areas. Eur J Oper Res. 2015;242: 525–535.
  23. 23. Emrouznejad A, Parker BR, Tavares G. Evaluation of research in efficiency and productivity: A survey and analysis of the first 30 years of scholarly literature in DEA. Socioecon Plann Sci. 2008;42: 151–157.
  24. 24. Emrouznejad A, G liang Yang. A survey and analysis of the first 40 years of scholarly literature in DEA: 1978–2016. Socioecon Plann Sci. Elsevier Ltd; 2017; 1–5.
  25. 25. Tavares G. A bibliography of Data Envelopment Analysis (1978–2001). RRR (Rutgers Res Report). 2002; 186.
  26. 26. Pelone F, Kringos DS, Romaniello A, Archibugi M, Salsiri C, Ricciardi W. Primary Care Efficiency Measurement Using Data Envelopment Analysis: A Systematic Review. J Med Syst. 2015;39. pmid:25486892
  27. 27. Hussey PS, De Vries H, Romley J, Wang MC, Chen SS, Shekelle PG, et al. A systematic review of health care efficiency measures: Health care efficiency. Health Serv Res. 2009;44: 784–805. pmid:19187184
  28. 28. Cantor VJM, Poh KL. Integrated Analysis of Healthcare Efficiency: A Systematic Review. J Med Syst. 2017;42: 8. pmid:29167999
  29. 29. Moran V, Jacobs R. An international comparison of efficiency of inpatient mental health care systems. Health Policy (New York). Elsevier Ireland Ltd; 2013;112: 88–99. pmid:23891192
  30. 30. Corredoira RA, Chilingerian JA, Kimberly JR. Analyzing performance in addiction treatment: An application of data envelopment analysis to the state of Maryland system. J Subst Abuse Treat. Elsevier Inc.; 2011;41: 1–13. pmid:21353439
  31. 31. Nahra TA, Mendez D, Alexander JA. Employing super-efficiency analysis as an alternative to DEA: An application in outpatient substance abuse treatment. Eur J Oper Res. Department of Health Management and Policy, School of Public Health, University of Michigan, 109 Observatory Road, Ann Arbor, MI 48109–2029, United States; 2009;196: 1097–1106.
  32. 32. Zhu J. Imprecise data envelopment analysis (IDEA): A review and improvement with an application. Eur J Oper Res. 2003;144: 513–529.
  33. 33. Wheelock DC, Wilson PW. Consolidation in US banking: Which banks engage in mergers? Rev Financ Econ. 2004;13: 7–39.
  34. 34. Cordero J., Santín D, Sicilia G. Testing the accuracy of DEA estimates under endogeneity through a Monte Carlo simulation. Eur J Oper Res. 2015;244: 511–518.
  35. 35. Di Giorgio L, Flaxman AD, Moses MW, Fullman N, Hanlon M, Conner RO, et al. Efficiency of Health Care Production in Low-Resource Settings: A Monte-Carlo Simulation to Compare the Performance of Data Envelopment Analysis, Stochastic Distance Functions, and an Ensemble Model. PLoS One. Public Library of Science; 2016;11: e0147261. pmid:26812685
  36. 36. Fishman GS. Monte Carlo. Concepts, Algorithms, and Applications. 1996.
  37. 37. Farrance I, Frenkel R. Uncertainty in measurement: a review of monte carlo simulation using microsoft excel for the calculation of uncertainties through functional relationships, including uncertainties in empirically derived constants. Clin Biochem Rev. Australia; 2014;35: 37–61.
  38. 38. Perelman S, Santín D. How to generate regularly behaved production data? A Monte Carlo experimentation on DEA scale efficiency measurement. Eur J Oper Res. 2009;199: 303–310.
  39. 39. Krüger JJ. A Monte Carlo study of old and new frontier methods for efficiency measurement. Eur J Oper Res. 2012;222: 137–148.
  40. 40. Pereira C, Salinas-Perez JA, Gutiérrez MR. Atlas de Salud Mental de Bizkaia. [Internet]. Bilbao: Osakidetza; 2013.
  41. 41. Iruin A, Gutiérrez-Colosía MR, Salinas-Pérez JA. Atlas de Salud Mental de Gipuzkoa [Internet]. Bilbao: Osakidetza; 2012.
  42. 42. Chung Y, Salvador-Carulla L, Salinas-Perez JA, Uriarte-Uriarte JJ, Iruin-Sanz A, Garcia-Alonso CR. Use of the self-organising map network (SOMNet) as a decision support system for regional mental health planning. Heal Res policy Syst. England; 2018;16: 35. pmid:29695248
  43. 43. Salvador-Carulla L, Costa-Font J, Cabases J, McDaid D, Alonso J. Evaluating mental health care and policy in Spain. J Ment Health Policy Econ. 2010;13: 73–86. pmid:20919594
  44. 44. Montagni I, Salvador-Carulla L, Mcdaid D, Straßmayr C, Endel F, Näätänen P, et al. The REFINEMENT Glossary of Terms: An International Terminology for Mental Health Systems Assessment. Adm Policy Ment Heal Ment Heal Serv Res. Springer US; 2017; 1–10. pmid:28918502
  45. 45. Barasa EW, Molyneux S, English M, Cleary S. Setting Healthcare Priorities at the Macro and Meso Levels: A Framework for Evaluation. Int J Heal Policy Manag. Kerman University of Medical Sciences; 2015;4: 719–732. pmid:26673332
  46. 46. Savigny D, Adam T. Systems thinking for health systems strengthening [Internet]. Geneva: World Health Organization; 2009.
  47. 47. Salvador-Carulla L, García-Alonso C, Gonzalez-Caballero JL, Garrido-Cumbrera M. Use of an operational model of community care to assess technical efficiency and benchmarking of small mental health areas in Spain. J Ment Health Policy Econ. 2007;10: 87–100. pmid:17603149
  48. 48. Salvador-Carulla L, Alvarez-Galvez J, Romero C, Gutiérrez-Colosía MR, Weber G, McDaid D, et al. Evaluation of an integrated system for classification, assessment and comparison of services for long-term care in Europe: the eDESDE-LTC study. BMC Health Serv Res. 2013;13: 218. pmid:23768163
  49. 49. Salvador-Carulla L, Amaddeo F, Gutiérrez-Colosía MR, Salazzari D, Gonzalez-Caballero JL, Montagni I, et al. Developing a tool for mapping adult mental health care provision in Europe: the REMAST research protocol and its contribution to better integrated care. Int J Integr Care. Utrecht, The Netherlands: Uopen Journals; 2015;15: e042–e042.
  50. 50. Gibert K, García-Alonso C, Salvador-Carulla L. Integrating clinicians, knowledge and data: expert-based cooperative analysis in healthcare decision support. Health Res Policy Syst. 2010;8: 28. pmid:20920289
  51. 51. Cox E. Fuzzy Modeling and Genetic Algorithms for Data-Mining and Exploration. San Francisco: Elsevier; 2005.
  52. 52. Banker ARD, Charnes A, Cooper WW. Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis Some Models For Estimating Technical And Scale Inefficiencies In Data Envelopment Analysis *. Manage Sci. 1984;30: 1078–1092.
  53. 53. Farrell MJ. The Measurement of Productive Efficiency [Internet]. Journal of the Royal Statistical Society. Series A (General). 1957. p. 253.
  54. 54. Klir GJ. Trends in general systems theory. Wiley-Interscience; 1972.
  55. 55. Tyler LH, Ozcan YA, Wogen SE. Mental health case management and technical efficiency. J Med Syst. UNITED STATES; 1995;19: 413–423.
  56. 56. Ozcan YA, McCue MJ, Okasha AA. Measuring the technical efficiency of psychiatric hospitals. J Med Syst. UNITED STATES; 1996;20: 141–150.
  57. 57. Kontodimopoulos N, Bellali T, Labiris G, Niakas D. Investigating sources of inefficiency in residential mental health facilities. J Med Syst. United States; 2006;30: 169–176.
  58. 58. Yeh J, White KR, Ozcan YA. Efficiency evaluation of community-based youth services in Virginia. Community Ment Health J. UNITED STATES; 1997;33: 487–499.
  59. 59. Marshall M, Crowther R, Sledge WH, Rathbone J, Soares-Weiser K. Day hospital versus admission for acute psychiatric disorders. Cochrane database Syst Rev. 2011; CD004026–CD004026. pmid:22161384
  60. 60. Marshall M, Crowther R, Almaraz-Serrano A, Creed F, Sledge W, Kluiter H, et al. Systematic reviews of the effectiveness of day care for people with severe mental disorders: (1) Acute day hospital versus admission; (2) Vocational rehabilitation; (3) Day hospital versus outpatient care. Health Technol Assess (Rockv). 2001;5.
  61. 61. Thornicroft G, Tansella M. Components of A Modern mental Health Services : A Pragmatic Balance of Community and Hospital Care. Br journals Psychiatr. 2004;185: 283–290. pmid:15458987
  62. 62. Thornicroft G, Tansella M. Better mental health care. Cambridge, editor. Cambridge University Press; 2009.