In an effort to deal with more complicated evaluation situations, scientists have focused their efforts on dynamic comprehensive evaluation research. How to make full use of the subjective and objective information has become one of the noteworthy content. In this paper, a dynamic comprehensive evaluation method with subjective and objective information is proposed. We use the combination weighting method to determine the index weight. Analysis hierarchy process method is applied to dispose the subjective information, and criteria importance through intercriteria correlation method is used to handle the objective information. And for the time weight determination, we consider both time distance and information size to embody the principle of esteeming the present over the past. And then the linear weighted average model is constructed to make the evaluation process more practicable. Finally, an example is presented to illustrate the effectiveness of this method. Overall, the results suggest that the proposed method is reasonable and effective.
Citation: Liu D, Zhao X (2013) Method and Application for Dynamic Comprehensive Evaluation with Subjective and Objective Information. PLoS ONE 8(12): e83323. https://doi.org/10.1371/journal.pone.0083323
Editor: Alejandro Raul Hernandez Montoya, Universidad Veracruzana, Mexico
Received: June 13, 2013; Accepted: November 1, 2013; Published: December 26, 2013
Copyright: © 2013 Liu, Zhao. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was partially supported by the key project of Jiangsu Social Science Research (Grant NO. 2013ZDIXM015),the Fundamental Research Funds for the Central Universities Funding of Jiangsu Innovation Program for Graduate Education (Grant NO. CXLX12_0181) and the Production and Research Open Fund Project of Nanjing University of Aeronautics and Astronautics (Grant NO. 56XZA12074). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Over the years, dynamic comprehensive evaluation has been of great importance in the comprehensive evaluation theory. Compared with the static comprehensive evaluation, dynamic comprehensive evaluation, which studies on the performances of evaluated objects in a certain time period, faces with more complicated situations . Due to its advantages, dynamic comprehensive evaluation has become increasingly attractive for applications such as economic and management . Furthermore, dynamic comprehensive evaluation is a complex process in which a variety of information needs to be processed. So how to make full use of the information during the dynamic evaluation process has become the worth studying area at present.
In the specialized literature there is a considerable amount of research on the methods or applications of the dynamic comprehensive evaluation. The early studies focused on dealing with the real closure of an order field , . And then the evaluation process has improved through different ways, such as considering value fluctuation , object gain  or background , scattering degree , , as well as programming . At the same time, how to calculate the index or time weight more accurately has become central to optimize the dynamic comprehensive evaluation –. Moreover, the trend of methods developing toward multiplicity has also promoted the application research of dynamic comprehensive evaluation –. However, as clearly studied on index weight, many researchers have mainly ascertained it by applying subjective or objective information, but few researchers have used both . For the time weight many studies have been published concerning on the time distance, while little attention has been paid to the weight changes caused by the index information , .
This study therefore aims to present a dynamic comprehensive evaluation method to make heave use of subjective and objective information. In particular, (1) a combination weighting method is adopted to determine the index weight, including analysis hierarchy process (AHP) method to deal with subjective preference information, and criteria importance through intercriteria correlation (CRITIC) method to dispose the objective data; (2) both time distance and information size are applied to embody the different importance of each time point; (3) furthermore, a linear weighted average model is constructed by stating the optimal ordinal method; (4) and finally, comparing the results with the previous ones in ref. , the effectiveness of this method has been illustrated clearly.
The organization of the rest of the paper is as follows. In section 2 the research problem is briefly described. The index and time weights confirmation are presented in section 3. In section 4 we briefly describe the linear weighted average model for dynamic comprehensive evaluation. We illustrate the effectiveness of the method in this study through an example in section 5. The final section concludes the research results and puts forward the future work.
Let be a set which contains indicators including . The set is a set which has evaluation objects . The index weight set is , and with . The set is on behalf of the evaluation period from , to , and the time weight set is with and . Under the index , the evaluation research about objects during time period forms a dynamic comprehensive evaluation problem.
stands for the observed value of object () under indicator () at time (). The data unification and dimensionless should be stated in the first place –, because there may be distinct in type, unit or order of magnitudes among the indicators. After the data preprocessing, we assume that presents the standard data used in this study.
Index weight determination
Part 1, the initial weight determination: with AHP method.
AHP method developed by T. L. Satty et al. –, constructs a hierarchical structure model to determine the importance degree of index. In this study, evaluators apply 1–9 scale method to compare the importance of each indicator . And the data reflects the subjective preference information of evaluators. By using the subjective preference information, the analytical procedures are as follows:
- The analytic hierarchy structure is constructed. The index set is decomposed into different subsets including certain indicators. And its subordinate relations are confirmed according to the relationship between indicators.
- The evaluation index comparison judgment matrix is confirmed. (1)where is the important value after comparing with , and, with .
- Single hierarchy sorting and consistency check are affirmed. The characteristic roots of judgment matrix are calculated. After normalizing the characteristic roots, the weights of each indictor in the same evaluation level can be concluded. And then the random consistency ratio is confirmed. (2), . is the average random consistency index. If , the sorting has satisfactory consistency.
- The initial weight, which reflects the subjective information, is determined. The weight of every indicator is calculated at all levels; then the initial weight of indicator is determined. And , with .
Part 2, the secondary weight determination: with CRITIC method.
CRITIC method, which proposed by Diakoulaki in 1995 , reflects the relative importance by applying the comparative and conflict information among the indicators. So CRITIC method is chosen to dispose the objective information that refers to the observed value of evaluation object under the index. And its procedures are as follows:
- The conflicts between and at time are quantized. (3)where is the correlation coefficient between and at .
- The information quantity of at time is confirmed. (4)where is the standard deviation of at .
- The weight of at time is calculated. (5)
- The secondary weight is determined. The average index weight of in time phase is (6)where , and .
Part 3, the index weight determination: with combination weighting method.
In this process, we combine the weights, which are confirmed by both AHP and CRITIC methods, to determine the index weight.
The index weight on the one hand contains the subjective preference of evaluators, and on the other hand reflects the objective information of each indicator. If is close to 0, indicator is less significant, while if is close to 1, the importance of is larger. Above all, the index weight realizes effectively about the combination of both subjective and objective information.
Time weight determination
We deem that the time weight () significantly associates with two factors: time distance and information size. The former one means that new information has greater importance than the old one, and the latter one implies that the lager the information size of each indicator at time , the more important is.
The time weight reflects both the time distance and index information. If is close to 0, time becomes less crucial; while if is close to 1, the importance of is greater. In a word the time weight embodies the principle of esteeming the present over the past and reflects the importance of information.
The linear weighted average model
To make the evaluation process more practicable, we construct the dynamic comprehensive evaluation model by introducing the optimal ordinal method.
For , we set (11)where is the standardized data of object under indicator at time , and , , with .
From the formula (14), we know that is the comprehensive evaluation value which is determined by the linear weighted average model. Whether plays well or not in the time phase is known clearly by comparing with other objects.
We evaluate the regional environmental risk in China from 2003 to 2007 by using the same evaluation index, information, data pretreatment method and so on in ref. . And furthermore, we compare the evaluation results with the ones in ref.  to illustrate the effectiveness of the above method. The original data is collected in China Statistical Yearbook (2004–2008) and China Environmental Yearbook (2004–2008). The specific calculation processes are as follows:
- The same extremum method in ref.  is adopted to standardize the initial data;
- The index weight is determined. For the subjective information, we apply the same one which was calculated by AHP method in ref. . The objective information is computed by formulae (3)–(6) of CRITIC method. The combination of subjective and objective weights is determined finally by using the formula (7).
- The time weight of each evaluation year is confirmed by utilizing formulae (8)–(10). .
- The comprehensive evaluation values are calculated. We apply the formulae (11)–(14) to calculate the comprehensive evaluation value, and finally sort the evaluation objects.
- Compared the evaluation results with the ones in ref. .
Table 1 shows that there are many differences between subjective and objective weights. Subjective weight has on behalf of the evaluators, and objective weight reflects the data information of index. From table 1, it is observed that the combination of subject and object weights effectively states the significance of index.
Table 2 displays the dynamic comprehensive evaluation results in this study. Basing on the evaluation values, it is concluded the rank ordering of environmental risk of each provinces; then 31 provinces in China are divided into four risk types. Shanghai, Tianjin, Shandong, Beijing, Jiangsu and Liaoning fall into the category of type IV which has the high environment risk. The evaluation values of 10 provinces between 3.00 and 4.00, and they have a less high risk. And 11 provinces including Hunan, Gansu, Chungking and so forth are a form of type II which means a less low risk. Yunnan, Sinkiang, Chinghai and Tibet, whose evaluation values are much lower than others', are classified as type I which refers to the low risk.
Table 3 exhibits the dynamic comprehensive evaluation results in ref. . Contracted with the evaluation results in ref. , the environment risk classification of 31 provinces in China has different degrees of variation in our study. Further analysis of the results, we suggest that the division of environment risk is much more detailed than the one in ref. . Beijing had the high risk and in the third place in ref. , while the order is higher than the one in this research. In our study, the rank of Beijing is much more coincided with the actual environment improvement condition from 2003 to 2007. In addition, Jiangsu, Shandong and Liaoning developed rapidly and the environment destructions had increased and so they should be in a high risk at that period. For Yunnan, Sinkiang, Chinghai and Tibet, the damages of the environment were low and they were in the low risk level. While for the other 12 provinces in low risk type in ref. , the environment had been destroyed much more than the remaining four provinces' - Yunnan, Sinkiang, Chinghai and Tibet. So it is not reasonable to put the remaining 12 provinces in the low risk type in ref. .
The data provide evidence that the results of this study are more in line with reality than the ones in ref. . The reason why the evaluation results are more elaboration is because both subjective and objective information, which embody in the index and time weights, are applied. Consequently, we consider that the above method makes up for the information insufficient in ref. , and to some extent it is more reasonable and effective.
Conclusions and future work
In this paper, we propose a dynamic comprehensive evaluation method with subjective and objective information. The combination weighting method (AHP and CRITIC methods), which applies much more data information, has improved the accuracy of index weight. The time weight has reflected the principle of esteeming the present over the past by considering both time distance and information size in each time point. We construct the dynamic comprehensive evaluation model by introducing the optimal ordinal method. And the advantage of corresponding to reality of the proposed method has known clearly after compared with the results in ref. . In all, we argue that the dynamic comprehensive evaluation method with subjective and objective information may have an effective and reasonable evaluation results.
However, there are still some limitations in this research. Possible future research topics can be stating nonlinear programming method into the dynamic comprehensive evaluation process.
We are very grateful to the following people for help in collecting data: Yuehong Tian, Hao Wu, Qingyuan Xue, Yan Zhang, Zhanglong Yao and Yan Wang. We also thank Alejandro Raul Hernandez Montoya, Diva Gallo, Jennifer Cao, Susan Winn and two anonymous reviewers for their comments which helped to improve the manuscript.
Conceived and designed the experiments: DL XZ. Performed the experiments: DL. Analyzed the data: DL. Contributed reagents/materials/analysis tools: DL XZ. Wrote the paper: DL.
- 1. Guo Y (2007) The theory, method and application of comprehensive evaluation. Beijing: Science Press Ltd.
- 2. Guo Y (2012) Comprenhensive Evaluation Theory, Method and Extensions. Beijing: Science Press Ltd.
- 3. Duval D, González-Vega L (1996) Dynamic evaluation and real closure. Mathematics and computers in Simulation 42: 551–560.
- 4. Grammatikoulos V, Koustelios A, Tsigilis N, Theodorakis Y (2004) Applying Dynamic evaluation approach in education. Studies in Educational Evaluation 30: 255–263.
- 5. Guo Y, Hu L, Wang Z (2011) Method of dynamic comprehensive evaluation based on three stage difference driving features. Journal of Systems Engineering 26: 546–550.
- 6. Ma Z, Guo Y, Zhang F, Pan Y (2009) Method of dynamic comprehensive evaluation based on gain level inspiriting. Journal of Systems Engineering 24: 243–247.
- 7. Guo S, Li W, Guo Y (2012) A Dynamic Comprehensive Evaluation Method Reflecting Resource Background Difference. Journal of Northeastern University (Natural Science) 33: 296–299.
- 8. Guo Y, Pan J, Cao Z (2001) Dynamic Comprehensive Evaluation Method Supported by Multi-Dimensional Time Series. Journal of Northeastern University (Natural Science) 22: 465–467.
- 9. Guo Y (2002) New theory and method of dynamic comprehensive evaluation. Journal of Management Sciences in China 5: 49–54.
- 10. Otero MC, Dolado JJ (2004) Evaluation of the comprehension of the dynamic modeling in UML. Information and Software Technology 46: 35–53.
- 11. Guo Y (1995) Doubly Weighting Method for Dynamic Synthetical Evaluation. Journal of Northeastern University (Natural Science) 16: 547–550.
- 12. Guo Y, Tang H, Qu D (2010) Dynamic comprehensive evaluation method and its application based on minimal variability. Systems Engineering and Electronics 32: 1225–1228.
- 13. Huang W, Yao S, Guo Y (2011) Dynamic Comprehensive Evaluation Model of Incomplete Index Preference Information and its Application. Journal of Northeastern University (Natural Science) 32: 891–894.
- 14. Zhou J, Liu F (2008) The Dynamic Comprehensive Evaluation of Regional Economic Development in China. Journal of Xi'an Jiao tong University (Social Sciences) 28: 9–15.
- 15. Qu C, Bi J, Huang L, Li F, Yang J (2010) Dynamic Comprehensive Evaluation on Regional Environmental Risk. Acta Scientiarum Naturalium Universitatis Pekinensis 46: 477–482.
- 16. Chen H, Liu S, Hu H (2011) Dynamic Comprehensive Evaluation of Higher Institutions Research Achievements Based on Dual-Excitation Control Lines. Science of Science and Management of S.&T. 32: 129–133.
- 17. Wang X, Guo Y, Zhao L (2005) The Method of Dynamic Constitution Evaluation and Its application in Supplier Selection. Management Review 17: 41–43.
- 18. Guo Y, Yao Y, Yi P (2007) Method and Application of Dynamic Comprehensive Evaluation. Systems Engineering -Theory & Practice 27: 154–158.
- 19. Liu W, Shi C, Zhao S (2013) Dynamic Comprehensive evaluation model with the feature of speed. Systems Engineering- Theory & Practice 33: 705–710.
- 20. Johnston J (1972) Economic methods (2nd Edition). New York: MeGraw-Hill.
- 21. Yue C (2003) Decision theory and method. Beijing: High Education Press.
- 22. Yi P, Zhang D, Guo Y, Gao L (2009) Study on Dimensionless Methods in Dynamic Comprehensive Evaluation. Journal of Northeastern University (Natural Science) 30: 891–894.
- 23. Guo Y, Ma F, Dong Q (2011) Analysis of influence of dimensionless methods on deviation maximization method. Journal of Management Sciences in China 14: 19–28.
- 24. Zhao H, Xu S, He J (1986) Analytic Hierarchy Process method: A simple new decision method. Beijing: Science Press.
- 25. Parra-López C, Calatrava-Requena J, de-Haro-Giménez T (2008) A systemic comparative assessment of the multifunctional performance of alternative olive systems in Spain within an AHP-extended framework. Ecological Economics 64: 820–834.
- 26. Vidal L A, Marle F, Bocquet J C (2011) Using a Delphi process and the Analytic Hierarchy Process (AHP) to evaluate the complexity of projects. Expert Systems with Applications 38: 5388–5405.
- 27. Du D (2001) Study on 0.1∼0. 9 Scale in AHP. Systems Engineering and Electronics 23: 36–38.
- 28. Diakoulaki D, Mavrotas G, Papayannakis L (1995) Determining objective weights in multiple criteria problems: the CRITIC method. Compute & Operations Research 22: 763–770.
- 29. Bartlett L M, Du S (2005) New progressive variable ordering for binary decision diagram analysis of fault trees. Quality and Reliability Engineering International 21: 413–425.