https://orcid.org/0000-0001-7018-9716
Figures
Abstract
The yearly increase in government R&D investment and top-down national R&D investment allocation requires a more quantitative decision-making system that maximizes R&D performance and efficient budget allocation. Sound decision-making is necessary at both the selection stage and the pursuit stage in order to maximize limited national R&D resources. We study Korean smart farms as an example to examine national R&D investment from the various R&D actors (academia, industry, and research institutes) perspectives. The objective of our research is to evaluate the theoretical efficiency of R&D investment on specific technologies in smart farms and compare the results with expert opinions where the reality is reflected. To be specific, our study is to provide the quantitative approach in making decision among policymakers by reflecting the field experiences and opinions. We use a data envelopment analysis with an assurance region model, which integrates an analytic hierarchy process and a data envelopment analysis. The weights of output in DEA model by the R&D actors are similar to the overall weight by all actors, implying that investment allocation decisions in the smart farm sector are not significantly affected by the R&D actors. We realized that the relative efficiency of some R&D technologies increases after reflecting qualitative ideas of experts. In reality, it is necessary to invest in these technology groups, but they excluded from top-down decision-making. This also shows that a government’s top-down decision-making can distort its investment allocation. This study proposes a new approach to compensate for the difference between theoretical virtual prices and actual prices in data envelopment analysis. In particular, when comparing the only quantitative results on investment priorities with analysis results by additionally reflecting the opinions of experts in each sector, we found that the Korean government’s investment priorities in the smart farm field are considerably distorted. Therefore, this study is expected to be used as an alternative for policy makers to compensate for the quantitative distortion might be caused by top-down national R&D investment decisions.
Citation: Park W, Kim S-G (2022) Integrating quantitative and qualitative methodologies to build a national R&D plan using data envelopment analysis based on R&D stakeholders’ perspectives. PLoS ONE 17(3): e0265058. https://doi.org/10.1371/journal.pone.0265058
Editor: Dragan Pamucar, University of Defence in Belgrade, SERBIA
Received: November 3, 2021; Accepted: February 22, 2022; Published: March 11, 2022
Copyright: © 2022 Park, Kim. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Professor Paul M. Romer, a laureate of the 2018 Nobel Prize in Economic Sciences, argues that technological progress does not fit the classical growth model, which is determined by factors external to the model, that is, actions (e.g., Research and Development (R&D)) encouraged through economic incentives. This is referred to as the endogenous growth model [1]. According to Romer [2], unlike public and other existing goods, technology (or knowledge) is non-rival, which means it can be shared by many people without depleting its availability. Nevertheless, it is also excludable, and can be protected by laws or institutions to limit its use. The higher the excludability, the greater the private return from R&D, which further encourages such activities in the private sector (or industry). However, knowledge that is less likely to be excluded, such as basic science, is usually examined in universities or public research institutions. These characteristics allow for appropriate government intervention in order to achieve an efficient resource allocation. Therefore, the main purpose of national investment in R&D is to compensate for private R&D activities, support the development of future core technologies, and develop technologies in the basic, public, and welfare sectors [3]. As the national investment resources are limited, it is critical to make a priority which research needs support or technologies are developed. It requires government to challenge how to utilize the available but limited resources best for its science and technology research. In this regards, the priority setting is important when it comes to national R&D investment. According to OECD [4], OECD member governments are tackling decision making in priority setting for R&D. The report defines priority setting as a complex political process involving many people from various fields. It includes from thematic to structural priorities such as training researchers or balancing the funding options. The decision-making for setting the R&D investment priority is very complicated process which involves various stakeholders from scientific experts to policy, business and community representatives.
Korea’s investment in R&D has increased significantly, with the country ranked fourth largest in the world in 2021, and top in terms of percentage of GDP. Although investment in national R&D is increasing, research performance remains poor, given the size of the budget. In addition, the innovation performance in R&D system in Korea is still below the average among OECD countries. The reason is that the R&D resource allocation is mainly under control of government and the decisions are made by top-down process (e.g., government designs the project and the companies or institutions are applied the project) qualitatively. However, it is difficult to change the current decision-making process immediately. Therefore, this increase, and the top-down national R&D planning it needs, requires a more quantitative R&D investment decision-making system that can maximize technology transfer and diffusion and yield an efficient budget allocation. Given the country’s limited resources, national R&D projects should undergo a selection process to maximize their effectiveness.
Policymakers responsible for allocating the national R&D budget need to make optimal investment decisions in order to maximize research performance. Several studies have examined the performance and efficiency of R&D investment, and proposed methodologies to solve these problems. Jang [5] develops and applies a national R&D budget allocation model based on machine learning and optimization theory. Lee et al. [6] use a data envelopment analysis (DEA) to analyze the relative efficiency of investment in the field of precision medicine. In addition, Kim et al. [7] analyze the effect on investment performance of R&D actors, development levels, and periods, using technology transfer and diffusion as their performance measure for national R&D investment. Wang and Huang [8] evaluate the relative efficiency of R&D activities in 30 countries using DEA method. They show that 15 countries are fully efficient in R&D activities and more than 20 countries are increasing their returns to scale. They also find out that there is chance to improve the effectiveness of policies and efficiency in R&D performance. Kyung [9] proposes a national R&D investment priority model based on a technology roadmap. At the national level, the roadmap is used to secure the competitiveness of domestic technology, and to determine the directions and priorities for resource allocations. In the process of identifying technologies in which to invest, the development of convergent technologies can be reflected in the opinions of R&D actors. As a result, governments can effectively allocate and use R&D resources by selecting and focusing on certain technologies. In recent years, the DEA method has become popular for analyzing the efficiency and performance of R&D investments, at both the national and the industry level. Liu et al. [10] evaluate the R&D performance of industries in China with two-stage DEA model. Contreras and Lozano [11] solve the problem of additional resources allocation among public universities by utilizing centralized DEA with bargaining theory. Khoshnevis and Teirlinck [12] evaluate the performance of R&D active firms to allocate R&D resources efficiently. They utilize the DEA model with ratio inputs and outputs. Ozbugday et al. [13] analyze OECD countries to distinguish the efficiency changes in long-term. They implement input oriented DEA with Malmquist Productivity Index. Most of previous literatures utilize DEA as a tool to analyze the performance of various sectors including R&D performance. They focus on the R&D performance according to DEA results.
In this study, we develop a scale that can measure the efficiency in producing various outcomes compared to the R&D expenditure input and the relative importance (or preference) qualitatively considered for individual performance indicators by industry, academia, and research institutes. Therefore, we present a new model in the form of a mixture of qualitative and quantitative analysis that can measure the efficiency of input versus performance within the importance range for the intended output variables by preemptively measuring them. At this time, evaluating in advance the importance (or preference) of performance indicators by each innovation entities (i.e. industry, academia, and research institutes) in the smart farm sector in Korea through AHP and reflecting the additional constraints on the coefficients of the output variables in the DEA model with an assurance. Lastly, in order to secure the reliability of the optimal investment alternatives based on the newly proposed DEA-AR model, sensitivity analysis is performed to analyze the change in the appropriate amount of input that will not change the optimal investment priority. As a result, we evaluate whether national R&D investment decisions reflect reality rationally and objectively. And we use the results to derive policy implications. This study in this regard aims to provide better quantitative decision making for policymakers when they allocate resources on national R&D.
The remainder of this paper is organized as follows. Section 2 outlines the analytical framework for R&D investment priorities with the research method. Section 3 gives a description of the data and the mathematical modelling that we use in the paper. Section 4 presents the empirical results and analysis of Korea’s smart farm R&D investment priorities. We apply the proposed model to compare the current investment priorities with investment areas considered important to performance, as identified by relevant R&D actors. The final two section provides the discussion and conclusion of the results which show the empirical, practical implications and academic contributions for future policymakers.
The analytical framework for R&D investment priorities
An overall summary of the methodology.
To quantitatively evaluate investment decision-making related to national R&D planning, we apply an integrated an analytic hierarchy process (AHP) and DEA model to analyze a DEA-AR, with weights constrained using the results of the AHP. The DEA is a nonparametric decision-making methodology that objectively determines relative efficiency based on minimal information. One of the biggest advantages of the DEA is that the model generates the values or weights of the inputs and outputs automatically, rather than relying on analysts to provide such information in advance [14]. However, most observations have an efficiency value of one, making it impossible to compare the efficiency of the evaluation or to limit the number of projects determined to be the most efficient [15]. Therefore, once an absolute value judgment is available from analysts, the results can be enhanced using real data by reflecting the value judgment in advance. Various methods have been proposed to improve the discriminating power of the DEA model. One such method restricts the range of weights used to indicate the relative importance of the values in the DEA model. Thompson et al. [16] recognize the weaknesses of the DEA model, explicitly applying an AR in their efficiency analysis of the 100 largest US banks. R.G.Dyson and E.Thanassoulis [17] formulate a weight-restricted AR model, and suggest various types of weight restrictions and value judgments for the AR. Allen et al. [18] summarize the development of weight restrictions and provide a basic overview of the model. They also discuss in detail direct weight constraints, adjusting the input and output variables, and actual input and output variable constraints.
In order to calculate the weight for each R&D actor in terms of R&D performance, we categorized the actors as belonging to industry, academia, or a research institute, as in previous studies. Each R&D actor’s AHP value was then used as a weight constraint in the performance indicators. The conceptual structure of the analytical model is shown in Fig 1.
Measuring key indicators to evaluate the performance of government decision-making.
The AHP is a decision-making method that uses pairwise comparisons to select or prioritize one of several alternatives. The computational model was originally devised by Prof. Thomas L. Satty to analyze problems hierarchically, similarly to the way people do. The model is widely used in the field of decision-making because it provides a quantitative measure (a ratio) of relative importance. Here, we employ an AHP to obtain key indicators, which we use to evaluate the performance of government decision-making related to prioritizing R&D investment in specific sectors.
DEA model with restricting the weight ranges.
Efficiency can be measured in several ways. The DEA model measures the efficiency of output over input nonparametrically. Based on linear programming, the empirical efficiency frontier is derived from the outputs and inputs, and then each Decision Making Unit (DMU) measures its inefficiency as its distance from the efficient frontier [19]. The DEA model was developed in 1978 by Charnes et al. [20] combining Farrell’s efficiency analysis theory from 1957 and Shephard’s distance function theory from 1970. Various extensions to the DEA model have been proposed, including the Charnes–Cooper–Rhodes (CCR) model and the Banker–Charnes–Cooper (BCC) model [21]. The CCR model assumes constant returns to scale. The BCC model, which extends the CCR model, assumes variable returns to scale, which change with the scale of inputs and outputs. In addition, the DEA model is divided into input- and output-oriented models. The input-oriented model derives an improvement plan that minimizes inputs and maximizes output efficiency. The output-oriented model maximizes output for optimal input efficiency [14].
The DEA model determines the relative efficiency of various areas, based on minimal information. However, the conventional DEA model has practical limitations: it lacks discriminating power, and it cannot limit the number of efficient projects. This may be problematic in practice, where the number of government R&D projects is usually limited by budgetary constraints. If the number of efficient projects identified by the DEA analysis is larger than the budget, there is no way of determining which projects are superior [15]. To solve this problem, methods have been proposed that restrict the range of a weight. In contrast to the conventional DEA model, the DEA-AR model, developed by. Thompson et al. [22] performs an additional analysis on the DMUs on the efficiency frontier. The difference between this and the DEA model is the weight constraint formula. The weight range sets upper and lower limits for the ratio between the input and output weights.
Fig 2 is a two-dimensional illustration of the DEA-AR model. It considers one input variable (X) and two output variables (Y) and assumes that each DMU is given by the points in the figure. Then, the efficient DMUs are P2, P3, P4, and P5. Because there is only one input variable, only the weight range of the output variables needs to be considered. This weight range is measured from the origin, and these lines become the constraint range, or the AR. When constraining the AR range from R1 to R3, DMUs P3 and P4 (within the range of R1 to R3) become efficient. Similarly, if we constrain the AR range from R1 to R2, P3 becomes the only efficient DMU. As described above, by constraining the AR, we obtain the most efficient DMU, thus solving the problem of a lack of discriminating power. Several methods exist for setting the AR range by incorporating expert opinions directly, using a conventional decision-making analysis or a multiple regression analysis. In this study, we set the weight range using expert opinions obtained from an AHP analysis.
Integrated models with assurance region.
The integrated AHP/DEA model overcomes the weaknesses of the individual models, constraining the AR range using the AHP. Ho [23] reviews four literature articles regarding integrated AHP/DEA approach and discovers that AHP/DEA model has less studied comparing with other AHP combined model, such as mathematical programming techniques (i.e. linear programming). Several studies have introduced the empirical applications of the integrated AHP/DEA methodology. Wang et al. [24] apply the integrated model to assess the bridge risk in the UK. AHP is applied to calculate the weights of four criteria (safety, functionality, sustainability and environment) with linguistic terms (high, medium, low and none) by top manager of bridge maintenance projects and then DEA utilizes to determine the values of the linguistic terms. Yang and Kuo [25] propose AHP for evaluating qualitative DMUs and DEA for calculating both qualitative and quantitative data in order to design the efficient plant layout. Sueyoshi et al. [26] combine AHP and DEA to help making efficient decision for internal auditing. AHP is used to deal with qualitative information from professionals and DEA is applied to compute the quantitative data from the accounting department and the fleet information. Lin et al. [27] propose an integrating DEA and AHP model to evaluate the economic performance in China. Among ten variables, two qualitative variables are measured by AHP and used in DEA analysis while other quantitative variables are used in the DEA analysis. Tapia et al. [28] develop a hybrid AHP and DEA framework to select efficient carbon dioxide utilization options. They apply the model to two case studies by combining quantitative data and expert opinions on qualitative criteria as a priority weights. Jang and Lee [19] verify the practical application of an analytical hybrid model to select an optimal promising technology, given limited national R&D resources. Kim and Choi [29, 30] and Park [31] evaluate the efficiency of container terminals using a hybrid AHP/DEA model. Lee et al. [32] use an integrated AHP/DEA model to develop a realistic mathematical model that can be applied to select R&D projects. Sinuany-Stern et al. [33] complement the limitations of each decision model by ranking DMUs using an AHP, combining the AHP models with a DEA. In addition, Rhim et al. [15] propose a method for determining the AHP model in a hierarchical structure, and then using the values as weight constraints in a DEA model. They present a case study that uses an integrated AHP/DEA model to evaluate and select R&D projects. Ramanathan [34] derives a comparative matrix using a DEA to design a method for ranking alternatives using an AHP. Wang et al. [35] set weight ranges for the inputs and outputs and suggest an alternative model that compensates for the limitations of the AHP/DEA model. Lee et al. [36] compares the performance of national R&D programs using DEA and DEA-AR model to incorporate the importance of variables in reality. The pairwise comparison in the AHP for the upper and lower bound of AR was introduced to measure the priority weights of output variables. Lee and Lee [37]also introduce AHP to make the pairwise comparison for DEA-AR model, setting weights of upper and lower bounds. Lai et al. [38] use AHP/DEA-AR model for evaluating the efficiency of international airports. AHP results are used to calculating the upper and lower bound of DEA-AR. There are various application studies with integrated model from business sector to transportation. Most of studies use AHP as evaluating qualitative DMUs or ranking the variables of input and output.
Sensitivity analysis for confirming optimal alternatives.
The calculation cost issue in the DEA model is highly related to the number of input and output variables. In the investment efficiency analysis model for national R&D projects proposed in this study, the number of input and output variables is generally not large. Similarly in the case of Korean smart farm, the input variable is R&D funds, and the output variable is composed of seven performance indicators representing technological and economic performance. Due to the nature of the DEA model, there may be a stability issue regarding the optimal efficiency score according to the amount of change in the input variable that tried in this research. There is a need to confirm the maintainability of the optimal solution of the DEA-AR proposed model within a specific change range of the input. What this means is that it is necessary to check whether the complementary slackness theory in the DEA liner programing is satisfied according to the change in the input amount. Within the range of the amount of change of the input that can satisfy the complementary slackness condition, the conversion between the primal and the dual in the linear programing becomes easy, and there is a possibility that the calculation cost of the linear programming method will be lowered. Therefore, in this study, sensitivity analysis on the DEA model is additionally performed to solve these two issues.
Data and mathematical modelling
DMU.
The DMUs in this study are 26 smart farm technology groups, which we selected by grouping similar technologies using document clustering, based on SCOPUS data. Then, the final technologies were selected by collating the expert opinions. The smart farm DMUs are given in Table 1.
AHP modelling.
For the AHP survey, we selected two categories, containing seven indicators in total, given the availability of data and their ability to calculate the weights for the experts. Government investment in R&D is important to the effectiveness of core technologies in industry, and causes economic ripple effects in investment; therefore, we categorize the criteria as technical and economic effects. Technical effects include papers, patents, and technical link indicators; economic effects include the employment inducement effect, value-added inducement effect, forward and backward industry link effects from the Bank of Korea, and input-output statistics. Technical and economic effects are located directly below the primary hierarchy target, that is, “priorities of national R&D investment,” followed by a further seven sub-criteria. The AHP model and criteria for this study are given in Table 2.
DEA input and output variables.
The inputs and outputs are based on AHP criteria derived in previous works on R&D investment efficiency. The model is implemented with one input variable (the cumulative amount of R&D investment from 2013 to 2018) and seven output variables (papers, patents, employment inducement effects, value-added inducement effects, and forward and backward industry effects). The inputs and outputs identified by previous studies on R&D investment efficiency are shown in Table 3.
Mathematical linear programming: DEA and proposed DEA-AR model.
This study analyzes the input-oriented CCR model using the benchmarking package in R. The assumption of DEA model is that there are i = 1,…,m inputs and r = 1,…,s outputs. The overall efficiency value of a particular DMU can be evaluated using following equation where it should be noted that the variables are the ur’s and the vi‘s and the yrk’s and xik’s are the observed output and input values, respectively, of DMUk.
The DEA-AR model of this study is expressed as follows:
(2)
(3)
where Zk represents efficiency of DMUk to be evaluated and ur is the weight of output r (paper, patent, …, backward industry link effect) while v is the weight of input (cumulative R&D budget). For the value of output and input, we denote yrj is the value of output r (paper, patent, …, backward industry link effect) for DMUj and xj is the value of input (cumulative R&D budget) for DMUj. The DEA-AR model put restrictions on the upper bound (Ui) and lower bound (Li). Lower bound for the ratio of virtual price in output variable is ui/u1 while upper bound for the ratio of virtual price in output variable is ui/u1.
In this study, we apply AHP to set the relative weights for DEA assurance range by mirroring real world opinions. First, experts from each fields (industries, academia, and research institutes) independently make their own pairwise comparison matrices and derive the relative weights for output variables. Then, the upper (U1i) and lower (L1i) bounds for assurance region are determined by the relative weight ratio of output variables as
(4)
where u1 is the weight vectors of the 1st output indicating paper while ui is the weight vectors of the ith output which is the one of the remaining output variables but paper. When the absolute values of the upper bound (U1i) and the lower bound (L1i) analyzed through AHP analysis are greater than or equal to 1 and less than 1, the consistency of relative weight is broken. In this case, when adding the AR constraints, there may be a case where it is necessary to guarantee the consistency between the upper bound and lower bound, and a case where it is not. AR constraints were set without correcting consistency in this study to reflect all opinions of individual experts in the smart farm sector.
Sensitivity analysis.
Variation of the input within the range that maintains an optimal solution in DEA (namely, the radius of stability) does not change the efficiency of the DMU. If the complementary slackness condition is satisfied, the newly proposed DEA-AR model can have a dual linear problem, and the optimal solution in this case is the same. Therefore, if the complementary slackness condition is satisfied, there a chance for the computation cost to be reduced as much as the number of variables of the objective function in the dual problem. In this study, sensitivity analysis was performed to check whether the complementarity slackness condition was satisfied. According to a previous study [44], it was pointed out that the sensitivity analysis method used in the existing linear programming method is not suitable for use in DEA. It suggests the need to perform sensitivity analysis while checking whether the efficiency of each DMU belonging to the efficiency group and the inefficiency group changes according to the amount of change in the input variable. Therefore, in this study, by referring to previous studies, a sensitivity analysis was performed to show how stable the analysis results are while changing the input values little by little. That is, the efficiency and inefficiency DMU were classified and the stability of the analysis result was confirmed by changing it from 5% to 20% based on the optimal input amount. As a result, even in the smart farm case of this study, the complementary slackness condition was satisfied within the range of 5% to 20% of the optimal input amount, confirming the possibility of retaining the dual problem. In this case, the number of variables in the given constraint and DEA-AR problem is not large, and the additional constraint constrained by the assurance region only limits the feasible range, so the calculation cost is not high even for the primal problem, not the dual problem.
Empirical results of the R&D investment priorities by R&D actors
Weight range of key performance indicators considered significant in R&D planning and strategies by R&D actors (total, industry, academia, research institutes).
Table 4 shows the weights of the indicators viewed as important to R&D planning by experts representing industry, academia, and research institutes. In general, we can suppose that academia is more interested in the fundamental or basic research, whereas industry is more interested in development research; research institutes bridge the two. Given the importance of the key indicators identified by actors in terms of their technical effect, papers and patents are considered relatively essential by industry, and technical links are considered important by academia. In terms of the economic effect, research institutes consider the employment inducement and value-added inducement effects to be substantial, whereas academia values the forward and backward industry link effects. These results indicate that academia and research institutes consider the weights of application and development to be higher than those of basic research, because smart farms tend to implement convergence technologies.
The range of the DEA-AR weight constraint is as follows (Table 5). The range for each R&D actor is set to 0.5 to 33.6, which was derived in a similar way to the overall weight, including all R&D actors. Once again, the AHP weight results by R&D actor reflect characteristics of the smart farm industry, as shown in Table 4. Therefore, we conclude that because most smart farm technologies are convergent technologies, they do not vary between individual R&D actors, as expected.
Comparative analysis of unconstrained DEA and weighted DEA-AR.
Prior to the comparative analysis, organic water reclamation (T17) and printed farm (T24) had no input values; therefore, we exclude them from the DEA and DEA-AR analysis. Fig 3 indicates the results of efficiency scores in the conventional DEA model without weight constraints, and the DEA-AR model with weight constraints, applying each R&D actor’s weight and the overall weight. Once the AR constraints are added, the efficiencies of DMUs can be lower than that of the unconstrained DEA. In this study, most DMUs, reflected in the weight of each R&D actor, were less efficient than before. This is because as the weight constraints are added, the efficiency area narrows, becoming smaller than the maximum value of the objective function of the general CCR model. The DEA-AR results indicate that there is no significant difference in the weight range for each R&D actor. This means that they have similar views in practice given the technologies of smart farm that applied. As noted, because the smart farm sector mainly implements convergent and applied technology, R&D actors’ features are not prominent. Therefore, we conduct a comparative analysis using the overall weight range, rather than differentiating between individual weights by type of actor.
Note: DMUs T17 and T24 are excluded because their input values are zero. R.I represents Research Institutes.
Under the unconstrained DEA model, multi-sensing AI control (T13) and hybrid renewable energy (T16) have the highest efficiency (1.000). In contrast, in the weighted DEA-AR model, only hybrid renewable energy (T16) has a high efficiency of one. Furthermore, after setting the constraints, most DMUs are less efficient than they were in the unconstrained DEA. Given efficiency scores between the DEA and DEA-AR model in Fig 3, the following 8 DMUs get worse by applying the constraints: smart machinery (T7), smart product sorting (T10), long-term storage packaging technology (T11), Multi-sensing AI control (T13), artificial synthesis (T14), Nano-bio sensing (T19), smart food (T26). It implies that a DMU in the unrestricted virtual price model identified as efficient can be turned out to be inefficient in the reality that analysts can assign subjective value judgements on the relative proportions of virtual prices. For example, Multi-sensing AI control (T13) is the most efficient technology in basic DEA model, but its efficiency gets worse after reflecting opinions in the related field. If policymakers decide R&D resource allocation only by efficiency, Multi-sensing AI control (T13) and hybrid renewable energy (T16) would be selected. However, Multi-sensing AI control (T13) is actually less efficient in reality and it is possible that other efficient technologies have been excluded. Therefore, it implies that reflecting field opinions is crucial when making resource allocation decision for policymakers.
Comparing the rankings of the DMUs (Table 6), the following 10 rank higher after applying the weights: environmental condition monitoring (T1), watering system measurement (T3), smart irrigation system (T4), cooling/heating AC system control (T5), biometric information measurement (T6), pest and disease diagnosis (T8), smart product sorting (T10), customized functional ingredient control (T18), smart quality control and distribution platform (T23), and advanced production (T25). Multi-sensing AI control (T13) ranks first when is no restriction; however, drops to seventh when using the DEA-AR model. The following DMUs do not change rank: pest and disease management (T9), vertical farm building design (T12), artificial synthesis (T14), integrated environmental information processing platform (T20), crop growth big data platform (T21), and productivity optimizing management platform (T22).
As mentioned earlier, in this study, there is a case where there is no consistency between the upper and lower bounds for the ratio of relative prices added as constraints of AR, and the consistency is corrected (the lower bound is adjusted from 0.5 to 1) to perform a sensitivity analysis. As a result of carrying out the sensitivity analysis, almost the same outcomes were obtained as in the case without correction. This result was interpreted to show that the level of lack of consistency does not show much difference compared to the size of the relative upper bound.
Comparative analysis in terms of government investment.
Table 6 shows the technology groups selected as investment priorities for two years (2019–2020), as well as the DEA and DEA-AR efficiency rankings estimated in this study. According to Table 6, of the 14 technologies selected as investment priorities, five ranked higher in the DEA-AR model: cooling/heating AC system control (T5), from 16 to 14; smart product sorting (T10), from 3 to 2; customized functional ingredient control (T18), from 17 to 13; smart quality control and distribution platform (T23), from 19 to 18; and advanced production (T25), from 15 to 12. The investment priorities of AI farmbot (T15) and hybrid renewable energy (T16) do not change. AI farmbot (T15) has a lower efficiency ranking, owing to the weight constraints, but hybrid renewable energy (T16) remains unchanged. These results are assumed to reflect reality when aligning the weight to the AHP. Hybrid renewable energy (T16), integrated environmental information processing platform (T20), and productivity optimizing management platform (T22) remain unchanged after the weight constraint is applied. Furthermore, organic water reclamation (T17) and printed farm (T24), which have no cumulative investment (zero input value), were included in the investment area. The fact that hybrid renewable energy (T16) was selected as the most efficient technology group in the DEA and DEA-AR models indicates that working-level opinions are considered when making investment decisions.
Sensitivity analysis results.
We performed sensitivity analysis as well to show how stable the analysis are while gradually changing the inputs. As suggested in the previous study, firstly, DMUs were classified by efficiency (efficient or inefficient) and then the input value was changed form 5% to 20%. We introduce 5% to 20% increases in each of input for T16. Simultaneously, we decreases these inputs by 5% to 20% for the other (inefficient) DMUs. Although T16, an efficient DMU, changed from 5% to 20%, it was always an efficient DMU while the inefficient DMUs did not change. Figs 4–7 show the results of sensitivity and stability analysis in DEA-AR model by professional group. As stated in previous study, these results significantly increase the robustness of the proposed DEA model. In other words, it was confirmed whether the results of this research were empirically robust through sensitivity analysis.
Discussion
The 26 smart farming technology groups use convergence technologies, in which application technology is employed rather than basic technology. These characteristics are also common to the weighting results of the R&D actors. The basic research-oriented academia and research institutes that serve as a bridge between basic research and applied development emphasize high weights on patents, the employment inducement effect, the value-added inducement effect, and the forward and backward industry link effects. That is, even for academia and research institutes, where basic technology is considered theoretically important, the performance of technology application and development is as significant as it is in industry. This implies that reflecting the opinions of such actors on R&D performance indicators is crucial to selecting the appropriate investment decision model. In the DEA-AR model, which applies the opinions of the R&D actors, the opinions for the individual R&D actors are similar to the overall viewpoint, indicating that the subjective judgements of each R&D actor do not have a significant effect on investment decision-making in the smart farm sector.
The government selected investment priorities for two consecutive years (from 2019 to 2020), mainly consisting of IoT platform technologies and AI control and sensing technologies. Thus, when the government decides to allocate R&D investments, they recognize the importance of core technologies in the field. Furthermore, a comparison of the investment priorities for the two years from the DEA-AR model (reflecting the views of experts) and the DEA model (virtual price is calculated automatically) shows that technologies are selected that have a relative higher ranking or that are at the same level. Among the 26 technology groups, hybrid renewable energy (T16) was the most efficient DMU in both models. Therefore, hybrid renewable energy (T16) is a decision that meets the investment needs of reality. However, other technology groups are not considered investment priorities, owing to a low theoretical efficiency; however, their relative efficiency increases after reflecting the opinions of experts, that is T1 (environmental condition monitoring), T3 (watering system measurement), T4 (smart irrigation system), T6 (Biometric information measurement), and T8 (pest and disease diagnosis). In reality, it is necessary to invest in these groups, but they are excluded from top-down national R&D investment decisions, which distorts the investment allocations. In addition, additional detailed analyses are needed to determine whether the technology groups selected for investment are relatively efficient compared with those that are not selected. Moreover, we need to reconsider the policy of which technology groups are selected for investment from among relatively low- and high-rank technology groups. This will improve the investment efficiency when making investment allocation decisions from policymakers’ points of view. Therefore, we need a methodology that quantitatively reflects experts’ views in the field related to performance measurement and decision-making for national R&D investment efficiency. Previous studies regarding DEA mainly compare the DEA results among DMUs. However, our study reflect the characteristics of smart farm industry based on R&D stakeholders’ opinions. Therefore, the proposed DEA-AR model can be used as a methodology to bridge the gap between the theory and the reality, and can compensate for quantitative distortions caused by top-down national R&D investment decisions.
The difference from previous studies is that we made the most of the advantages of the DEA model by preemptively reflecting the importance of performance indicators for each R&D actor in the process of allocating the national R&D budget. This study also contributes to the fact that it objectifies national R&D investment allocation decisions and secures reliability for investment alternative preferences through additional sensitivity analysis.
Conclusion
This study suggests a methodology for modeling R&D investment efficiency that incorporates the opinions of R&D actors and can be used for government decision-making related to R&D planning and investments. A conventional DEA analysis is meaningful in recommending the most efficient and optimal DMU, automatically and theoretically. However, in addition to providing theoretical results, the DEA-AR model presented here uses actors’ opinions to constrain the range of the weights, thus providing reasonable and objective conclusions. The AHP, based on the principles of human behavior, is a problem-solving decision-making method that combines subjective judgment with systemic approaches. That is, we calculate the weights based on experts’ opinions by considering important key performance R&D indicators that adequately reflect their opinions. Moreover, the DEA ensures the objectivity of the analysis of the top-down national R&D investment decision-making process, which mainly reflects the subjective opinions of policymakers. In particular, by applying a DEA-AR model instead of a general DEA model, we solve the latter’s problem of low discriminating power by restricting the weights. In other words, it is possible to objectively judge whether national R&D investment allocations and planning are consistent with reality using rational weight values from experts in each R&D sector in an AHP analysis. Therefore, the DEA-AR model provides flexible and unbiased national R&D investment decisions that reflect the actual situation and intention of internal and external R&D planning. The model can also be used to implement policies for strategic decisions, as well as to improve future investment efficiency.
We examined the importance of key performance indicators for each R&D actor, considering the case of Korean smart farms, which represent one of 10 key areas identified by the Korean government. The study also proposes a national R&D investment decision-making model that reflects the industrial smart farm characteristics of each R&D actor (i.e., academia, industry, and research institutes) which is a new qualitative and quantitative investment alternative analysis model. The range of weights (or preferences) for performance indicators was analyzed using AHP by collecting R&D actor’s opinions. In addition, efficiency analysis was performed by reflecting the range of weights for performance indicators as additional constraints. This can be said to have suggested a method to objectively quantify and reflect expert opinions in national R&D investment decision making. Finally, in order to analyze the category in which the efficiency score for each investment alternative does not change, sensitivity analysis was performed to secure the reliability of the proposed model. The model we proposed emphasizes practicality so that it can be used in the actual decision making process. AHP was applied as a method to systematically supplement the problem of discriminating power that may be pointed out as a limitation of the conventional DEA model while making the most of the objectivity of the DEA model. Our approach can be extended to various policy decision making issues that should properly reflect the decision maker’s willingness to evaluate while maintaining the objectivity of the decision making process in a special situation such as a top down manner. In addition, in practical terms, this model is basically based on the linear programming, so it is considered relatively easy to apply to other cases.
This study is conducted for the short term (i.e., two years), owing to a lack of available data. However, national R&D investment efficiency and its performance need to be analyzed over a longer period, and the smart farm technology groups (DMUs) used in this analysis may change over time. Therefore, additional variables are required. We can also apply a model other than the AHP to constrain the weight ranges. Furthermore, there is room to extend this model to other major national R&D investment fields. These topics are left to future research.
Supporting information
S1 Table. Comparison of efficiency scores and ranks between the DEA and DEA-AR models.
https://doi.org/10.1371/journal.pone.0265058.s001
(PDF)
S2 Table. Results of the DEA and DEA-AR model analyses by R&D stakeholders.
https://doi.org/10.1371/journal.pone.0265058.s002
(PDF)
Acknowledgments
The authors would like to thank the Editor-in-Chief and anonymous reviewers for their helpful comments and suggestions.
References
- 1.
Han WY, Lee J, Kim J. Government Research and Development Budget Analysis of the FY 2019. Seoul; 2019. https://www.kistep.re.kr/board.es?mid=a10305080000&bid=0002&act=view&list_no=25203&tag=&nPage=19.
- 2. Romer PM. Endogenous technological change. J Polit Econ. 1990;98: S71–S102.
- 3.
Han WY, Kim J. Government Research and Development Budget Analysis in the FY 2018. Seoul; 2018. https://www.kistep.re.kr/board.es?mid=a10305080000&bid=0002&act=view&list_no=25178.
- 4.
OECD. Priority Setting: Issues and Recent Trends. Governance of Public Research:Toward Better Practices. OECD; 2003. pp. 61–76.
- 5. Jang H. A decision support framework for robust R&D budget allocation using machine learning and optimization. Decis Support Syst. 2019;121: 1–12.
- 6. Lee BR, Sohn ES, Won DK, Yeo WD, Kim KH, Kim S. The efficiency analysis of national R & D planning for the field of precision medicine in Korea. J Open Innov Technol Mark Complex. 2018;4: 1–17.
- 7. Kim M-S, Yeon S-M, Kim J-S, Lee B-H. Analysis of Factors Influencing the Performance of Technology Transfer on National R&D by Research Actors. J Korea Contents Assoc. 2015;15: 559–570.
- 8. Wang EC, Huang W. Relative efficiency of R&D activities: A cross-country study accounting for environmental factors in the DEA approach. Res Policy. 2007;36: 260–273.
- 9. Kyung J. A Study on R & D Investment Decision Making Model by Using Small-Medium Enterprises Strategic Technology Roadmap. J Korea Acad Coop Soc. 2018;19: 786–794.
- 10. hui Liu H, liang Yang G, xiao Liu X, yao Song Y. R&D performance assessment of industrial enterprises in China: A two-stage DEA approach. Socioecon Plann Sci. 2019; 100753.
- 11. Contreras I, Lozano S. Allocating additional resources to public universities. A DEA bargaining approach. Socioecon Plann Sci. 2019; 100752.
- 12. Khoshnevis P, Teirlinck P. Performance evaluation of R&D active firms. Socioecon Plann Sci. 2018;61: 16–28.
- 13. Ozbugday FC, Tirgil A, Kose EG. Efficiency changes in long-term care in OECD countries: A non-parametric Malmquist Index approach. Socioecon Plann Sci. 2020;70: 100733.
- 14.
Lee J-D, Oh D. Theory of efficiency analysis: Data Envelopment Analysis. Jiphil Media; 2012. https://books.google.co.kr/books?id=i7hvuQAACAAJ.
- 15. Rhim H, Yoo S-C, Kim Y-S. A DEA/AHP Hybrid Model for Evaluation & Selection of R&D Projects. J Korean Oper Res Manag Sci Soc. 1999;24.
- 16. Thompson RG, Singleton FD, Thrall RM, Smith BA. Comparative Site Evaluations for Locating a High-Energy Physics Lab in Texas. Interfaces (Providence). 1986;16: 35–49.
- 17. Dyson R.G., Thanassoulis E. Reducing Weight Flexibility in Data Envelopment Analysis. J Oper Res Soc. 1988;39: 563–576.
- 18. Allen R, Athanassopoulos A, Dyson RG, Thanassoulis E. Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions. Ann Oper Res. 1997;73: 13–34.
- 19. Jang H-D, Lee J-H. An Efficient Selection of Promising Technology Using DEA and AHP. Korea Corp Manag Rev. 2011;18: 67–85.
- 20. Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. Eur J Oper Res. 1978;2: 429–444.
- 21. Banker RD, Charnes A, Cooper WW. Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis. Manage Sci. 1984;30: 1078–1092.
- 22. Thompson RG, Brinkmann EJ, Dharmapala PS, Gonzalez-Lima MD, Thrall RM. DEA/AR profit ratios and sensitivity of 100 large U.S. banks. Eur J Oper Res. 1997;98: 213–229.
- 23. Ho W. Integrated analytic hierarchy process and its applications—A literature review. Eur J Oper Res. 2008;186: 211–228.
- 24. Wang YM, Liu J, Elhag TMS. An integrated AHP-DEA methodology for bridge risk assessment. Comput Ind Eng. 2008;54: 513–525.
- 25. Yang T, Kuo C. A hierarchical AHP/DEA methodology for the facilities layout design problem. Eur J Oper Res. 2003;147: 128–136.
- 26. Sueyoshi T, Shang J, Chiang WC. A decision support framework for internal audit prioritization in a rental car company: A combined use between DEA and AHP. Eur J Oper Res. 2009;199: 219–231.
- 27. Lin MI, Lee YD, Ho TN. Applying integrated DEA/AHP to evaluate the economic performance of local governments in China. Eur J Oper Res. 2011;209: 129–140.
- 28. Tapia JFD, Promentilla MAB, Tseng ML, Tan RR. Screening of carbon dioxide utilization options using hybrid Analytic Hierarchy Process-Data Envelopment Analysis method. J Clean Prod. 2017;165: 1361–1370.
- 29. Kim S, Choi Y. An AHP / DEA Hybrid Model for Efficiency Evaluation of Container Terminal. J Korea Port Econ Assoc. 2012;28: 179–194.
- 30. Kim S, Choi Y. An Efficiency Evaluation of Employees Organization in Container Terminal. J Ind Econ Bus. 2012;25: 3697–3711.
- 31. Park R. A Brief Empirical Verification Using Multiple Regression Analysis on the Measurement Results of Seaport Efficiency of AHP/DEA-AR. J Korea Port Econ Assoc. 2016;32: 73–87.
- 32. Lee D-J, Bae S, Kang J. Development of R&D Project Selection Model and Web-based R&D Project Selection System using Hybrid DEA/AHP Model. J Korean Inst Ind Eng. 2006;32: 18–28.
- 33. Sinuany-Stern Z, Mehrez A, Hadad Y. An AHP/DEA methodology for ranking decision making units. Int Trans Oper Res. 2000;7: 109–124.
- 34. Ramanathan R. Data envelopment analysis for weight derivation and aggregation in the analytic hierarchy process. Comput Oper Res. 2006;33: 1289–1307.
- 35. Wang YM, Chin KS, Poon GKK. A data envelopment analysis method with assurance region for weight generation in the analytic hierarchy process. Decis Support Syst. 2008;45: 913–921.
- 36. Lee H, Park Y, Choi H. Comparative evaluation of performance of national R&D programs with heterogeneous objectives: A DEA approach. Eur J Oper Res. 2009;196: 847–855.
- 37. Lee S, Lee H. Measuring and comparing the R&D performance of government research institutes: A bottom-up data envelopment analysis approach. J Informetr. 2015;9: 942–953.
- 38. Lai PL, Potter A, Beynon M, Beresford A. Evaluating the efficiency performance of airports using an integrated AHP/DEA-AR technique. Transp Policy. 2015;42: 75–85.
- 39. Eilat H, Golany B, Shtub A. R&D project evaluation: An integrated DEA and balanced scorecard approach. Omega. 2008;36: 895–912.
- 40. Chun D, Hong S, Chung Y, Woo C, Seo H. Influencing factors on hydrogen energy R&D projects: An ex-post performance evaluation. Renew Sustain Energy Rev. 2016;53: 1252–1258.
- 41. Kim H-H, Lee D-J, Kim K-T, Park S-J. Measuring the Efficiency of Investment in the Deployment and Technology Development of Renewable Energy in Korea Using the DEA. J Korean Inst Ind Eng. 2014;40: 358–365.
- 42. Han U, Asmild M, Kunc M. Regional R&D Efficiency in Korea from Static and Dynamic Perspectives. Reg Stud. 2014;50: 1170–1184.
- 43. Zhong W, Yuan W, Li SX, Huang Z. The performance evaluation of regional R&D investments in China: An application of DEA based on the first official China economic census data. Omega. 2011;39: 447–455.
- 44. Cooper WW, Li S, Seiford LM, Tone K, Thrall RM, Zhu J. Sensitivity and Stability Analysis in DEA: Some Recent Developments. J Product Anal. 2001;15: 217–246.