https://orcid.org/0000-0003-3051-0581
Figures
Abstract
Objectives
To compared the presentation of research priorities in the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) clinical practice guidelines (CPGs) developed under the guidance of the GRADE working group or its two co-chair, and the Chinese CPGs.
Methods
This was a methodological empirical analysis. We searched PubMed, Embase, and four Chinese databases (Wanfang, VIP Database for Chinese Technical Periodicals, China National Knowledge Infrastructure and Chinese Biomedical Literature Database) and retrieved nine Chinese guideline databases or Society websites as well as GRADE Pro websites. We included all eligible GRADE CPGs and a random sample of double number of Chinese CPGs, published 2018 to 2022. The reviewers independently screened and extracted the data, and we summarized and analyzed the reporting on the research priorities in the CPGs.
Results
Of the 135 eligible CPGs (45 GRADE CPGs and 90 Chinese CPGs), 668, 138 research priorities were identified respectively. More than 70% of the research priorities in GRADE CPGs and Chinese CPGs had population and intervention (PI) structure. 99 (14.8%) of GRADE CPG research priorities had PIC structures, compared with only 4(2.9%) in Chinese. And 28.4% (190) GRADE CPG research priorities reflected comparisons between PICO elements, approximately double those in Chinese. The types of research priorities among GRADE CPGs and Chinese CPGs were mostly focused on the efficacy of interventions, and the type of comparative effectiveness in the GRADE research priorities was double those in Chinese.
Conclusions
There was still considerable room for improvement in the developing and reporting of research priorities in Chinese CPGs. Key PICO elements were inadequately presented, with more attention on intervention efficacy and insufficient consideration given to values, preferences, health equity, and feasibility. Identifying and reporting of research priorities deserves greater effort in the future.
Citation: Gao Y, Liu Z, Cao R, Liao Y, Feng Y, Su C, et al. (2024) Emphasis should be placed on identifying and reporting research priorities to increase research value: An empirical analysis. PLoS ONE 19(3): e0300841. https://doi.org/10.1371/journal.pone.0300841
Editor: Jose A. Calvache, Universidad del Cauca, COLOMBIA
Received: October 5, 2023; Accepted: March 4, 2024; Published: March 22, 2024
Copyright: © 2024 Gao et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the article and its Supporting Information files.
Funding: This project was supported by the Key Research and Development Program of Xinjiang Uygur Autonomous Region, URL: http://kjt.xinjiang.gov.cn/ (grant 2021B03006-4 to YT,Fei). Funding sources have no involvement in any of the activities conducted in this article.
Competing interests: The authors declare that they have no competing interests.
Introduction
In the development of clinical practice guidelines (CPGs), the formation of recommendations requires a comprehensive consideration of the quality of evidence, desirable and undesirable anticipated effects, health equity and other dimensions. The quality of the evidence was heavily considered as a key determinant dimension [1, 2]. However, in practice, evidence retrieval, particularly of high-quality evidence, often proves challenging [3–5]. Despite these limitations, recommendations were typically formulated based on the best available evidence. For the research gap, guideline developers tend to report research priorities through summarization and prioritization.
Simultaneously, another contrary situation arises where high-quality evidence has already existed and is robust enough that would unlikely be overturned in the future. In such cases, research saturations, a special kind of priority, should be presented to discourage further research, thus averting research waste. Furthermore, some guideline manuals also include research priorities as a form of research recommendations [6–10]. To be more precise, research priorities are not solely evidence gaps or knowledge gaps, but rather proposals for prioritized future research based on the current evidence.
The development of guidelines requires huge intelligence and financial input. Therefore, methodological rigor is crucial, and the guideline development group is also in the most suitable position to determine the research priorities.
Since the 1990s, China has developed a considerable number of CPGs [11], and the number has been increasing rapidly over time [12]. Serving as statement documents to guide Chinese clinicians in decision-making [13–15], CPGs has been playing a crucial and distinct role. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approaches, developed by the GRADE working group that is led by world-leading evidence-based medicine experts, published in 2016, were considered a gold standard in guideline development, often provides a paradigm of high-quality methodology. Their structured and transparent approaches, including a fixed research priority dimension embedded in the framwork, offered valuable guidance for guideline development groups [16–18].
Although there were some studies on research gaps [19–23], the exploration of the research priorities in the guidelines was still inadequate, especially the systematic analysis of the form and content of the research priorities [24]. This study systematically investigated and compared the CPGs that were developed under guidance of the GRADE working group and the Chinese CPGs, to offer reference and guidance for the development and reporting of research priorities in the future.
Methods
Identification of research priorities
We defined the research priorities as the focus of the most important future research topics addressing the needs of developing guidelines or recommendations. We usually find them in the EtD (evidence to decision) framework, discussion, evidence summary, etc., and identify their common terms including: research priorities, future research, evidence gaps, further research, etc.
Guidelines sources and searches
We searched two English databases (PubMed, Embase) and four Chinese databases (Chinese Biomedical Literature Database, China National Knowledge Infrastructure, Wanfang, and VIP Database for Chinese Technical Periodicals). We also searched the website (https://www.GRADEpro.org/) and methodological papers published by two co-chairs of the GRADE working group to find more GRADE CPGs. And we searched nine guideline databases or official websites of authoritative Chinese societies to find more Chinese guidelines (S1 File for the detailed search strategy).
Eligibility criteria
The GRADE CPGs and the Chinese CPGs that were published from the 1st January 2018 to the 31st December 2022 were included. GRADE CPGs are defined as CPGs developed under the guidance of the GRADE working group or its two co-chairs. Chinese CPGs are CPGs published in Chinese. Number of included Chinese CPGs were designed to be a random sample with twice as much as the included GRADE CPGs. Older versions and duplicate published CPGs were ineligible.
Data extraction and analysis
Two reviewers worked independently in pairs, the methods of systematic reviews of screening titles, abstracts, full text, and data extraction were strictly implemented. Unresolved differences were settled through consultation until consensus was achieved or by a third reviewer. We extracted the following information based on the standardized data extraction form designed in advance: (1) the basic characteristics of the guidelines, such as the type, scope and publication year, whether reported the research priorities, etc.; (2) the relevant information of the research recommendations, including reason, type (e.g., PICO, population, intervention, comparison, and outcome), structure, dimensions related to recommendations of research priorities, etc. Descriptive statistical analysis was used.
Results
Search results
The retrieval process for the GRADE CPGs and the Chinese CPGs were performed separately. For the GRADE CPGs, 523 records were retrieved from the databases, and an additional 23 records were obtained from other sources, resulting in 64 remaining records after removing duplicates and ineligible entries. Among these, four older versions and 15 records not meeting guideline criteria were excluded, leaving 45 CPGs included. Regarding Chinese CPGs, 225782 and 201 records were retrieved from databases and other sources respectively, 595 remained after removing duplicate and unqualified records. 3 older versions, 27 duplicate publications, 22 not guidelines were excluded. Finally, 543 Chinese CPGs were identified, of which 90 were randomly selected for inclusion (S2 File and Fig 1).
Guideline characteristics
A total of 135 CPGs were included (45 GRADE CPGs and 90 Chinese CPGs). Among the GRADE CPGs, about 90% (40, 88.9%) reported research priorities, with more than half (22, 55%) orienting to the whole guidelines. In Chinese CPGs, only less than 30% (26, 28.9%) of the guidelines reported research priorities, of which more than 70% (20, 76.9%) were for the whole guidelines. Furthermore, ten (25.0%) of the GRADE CPGs reported more than 20 research priorities, while the Chinese CPGs all presented fewer than 20 (Table 1).
Structure and presentation form of research priorities in CPGs
The GRADE CPGs reported a total of 668 research priorities, while the Chinese CPGs reported only 138. As for the reasons for developing the research priorities, 515 (77.1%) GRADE CPG research priorities reported the lack of evidence, while the Chinese CPG research priorities further reported the lack of high-quality evidence (71, 51.4%). Analyzing the structure of research priorities, we found that over 70% of both GRADE CPGs and Chinese CPGs had structures (P, I), comprising 475 (71.1%) and 101 (73.2%) respectively. In contrast, there were 99 (14.8%) GRADE research priorities with PIC structures compared to only 4 (2.9%) in the Chinese CPGs. Notably, 190 (28.4%) GRADE CPG research priorities involved the comparison between PICO elements, approximately double those in Chinese CPG research priorities (Table 2).
We identified 1076 and 128 descriptions related to the research priorities in the GRADE CPGs and Chinese CPGs, respectively. The descriptions of the research priorities in the GRADE CPGs were concentrated, with approximately 60% (635, 59.0%) described as “research priorities/needs/agenda/are needed/question”, with "research priorities" alone accounting for 51.4% (553). In contrast, descriptions of Chinese research priorities were more scattered and relatively simple. The maximum number of “questions to be solved/studied/addressed” described was 43 (33.6%), followed by “further research / studies / clarified” and “future research / research should prioritize to address”, 28 (21.9%) and 20 (15.6%), respectively (S1 and S2 Figs).
Focus of research priorities
Both the research priorities of the GRADE CPGs and the Chinese CPGs were clustered around efficacy of interventions, accounting for 42.1% (281) and 52.2% (72), respectively. Regarding the proportion of research priorities for comparative effectiveness, that is the condition if different interventions would need to be compared to address the research priority, GRADE research priorities were approximately double those in Chinese (accounted for 30.2% versus 15.2%). More than half of the research priorities in the GRADE CPGs and the Chinese CPGs reflected the dimension of desirable or undesirable anticipated effects, which are in the GRADE EtD framework for formulation of recommendations. The GRADE CPG research priorities considered a higher proportion of health equity and acceptability, while the Chinese is more concerned about the feasibility and quality of evidence (Table 3).
Recommendations for the study methods and details of intervention implementation in the research priorities
The number of research priorities concerning recommended research methods among GRADE CPGs and Chinese CPGs was relatively small, with randomized controlled trials being the primary research method, accounting for 6.4% (43) and 8.0% (11), respectively. GRADE CPGs also recommends observational studies (14. 2.1%) and risk assessment model or tools (11, 1.6%), while Chinese CPGs mentioned multi-center (4, 2.9%), large sample (10, 7.2%) and rigorous methodology (4, 2.9%) more frequently. Concerning the details of intervention implementation, GRADE CPG research priorities were more commonly related to dose (25, 3.7%), treatment time interval (19, 2.8%) and duration of therapy (16, 2.4%), while Chinese CPG research priorities were mostly related to dose (6, 4.3%), course of treatment (4, 2.9%) and treatment prescription (3, 2.2%) (Figs 2 and 3).
Discussion
Principal findings
This study reviewed 45 GRADE CPGs and 90 Chinese CPGs published in 2018 to 2022, and identified 668 and 138 research priorities, respectively. While 88.9% (40) of GRADE CPGs reported research priorities, only 28.9% (26) of Chinese CPGs did so. Analysis of the PICO elements of research priorities revealed that about 70% of both GRADE and Chinese research priorities had a PI structure. Furthermore, 14.8% (99) of GRADE CPG research priorities had PIC structures, compared to only 2.9% (4) in Chinese. Of notice, 28.4% (190) of GRADE CPG research priorities reflected comparisons between PICO elements, approximately double the proportion observed in Chinese research priorities. More than half of both GRADE and Chinese CPG research priorities focused on the dimensions of desirable or undesirable anticipated effects of the GRADE EtD framework. However, for the other dimensions, the concerns were different: the former was more focused on health equity and acceptability, while the latter was more concerned on feasibility and quality of evidence.
Strength and limitations
In our study, we conducted a detailed analysis of the presentation form, structure and type of research priorities in GRADE CPGs and Chinese CPGs. To our knowledge, no comprehensive and systematic research of such research priorities before.
There were limitations in our study. We accepted and analyzed the research priorities reported in the CPGs without further investigating the appropriateness of the underlying logic of the prioritization. Additionally, in most cases, the presented research recommendations were due to the lack of evidence or the low-quality of evidence. Different guideline development groups may have differences or confusion about the definition of the lack of evidence and quality of evidence.
Relation to previous work
A previous study formulated research priorities by developing a process, benefiting from considering the evidence base while identifying current knowledge gaps as well as any uncertainty [25]. Other studies focused on the structure of the research priorities, which summarized EPICOT (evidence, population, intervention, comparison, outcome, and time) as an essential structural element and emphasized the burden of disease and type of study as details that should be further considered [26–28]. Robinson and his colleagues identified 62 research gaps in five guidelines published by the Cystic Fibrosis Foundation and found that only 20% of the evidence gap were identified as research priorities by the guideline development group. They suggested that guideline developers should articulate research priorities more explicitly and systematically [24]. Although these studies explored research gaps or research priorities, they often conflated the two, which may not accurately reflect the connotation of the research priorities.
Implications for research and guidelines
We emphasize the distinction between evidence gaps and research priorities as an independent dimension. Not all evidence gaps indicate a reason to be prioritized. Significant effort in enhancing the methods and standards of identifying and reporting of research priorities should be taken.
First of all, the key structure elements of research priorities should be reported as comprehensive and standardized as possible. Clinical question formulation frameworks such as the PICO or EPICOT models can facilitate the transformation of research priorities into clinical questions efficiently and accurately, ensuring consistency with the actual needs of clinical practice and future research. Secondly, attention should be directed towards addressing research priorities within the dimensions of the GRADE EtD framework, including health equity, feasibility, values and preferences. Thirdly, guideline development groups should be encouraged to standardize the description of research priorities in guideline reporting, positioning them consistently and independently whenever feasible to aid identification by guideline users. Lastly, more attention should be paid to identify research priorities on details of intervention implementation, which are essential for developing really practical clinical recommendations.
Besides the above, another issue needs to be clarified. The rationale for proposing research priorities should not be solely tied to evidence absence or low-quality. It would be of great value of we could point out research saturation areas to avoid research waste when there is high-quality evidence that is robust enough that very unlikely to be overturned in the future.
Conclusion
There was still considerable room for improvement in the developing and reporting of research priorities in Chinese CPGs. Key PICO elements were inadequately presented, with more attention on intervention efficacy and insufficient consideration given to values, preferences, health equity, and feasibility. Identifying and reporting of research priorities deserves greater effort in the future.
Supporting information
S1 Fig. Reported descriptions of research priorities in GRADE CPGs.
https://doi.org/10.1371/journal.pone.0300841.s003
(TIF)
S2 Fig. Reported descriptions of research priorities in Chinese CPGs.
https://doi.org/10.1371/journal.pone.0300841.s004
(TIF)
Acknowledgments
The authors would to thank Cheng-Wei Si, Yu-Jie Wang, Peng-Cheng Wang, Xin-Yan Zhuang all from Beijing University of Chinese Medicine, for contributing to the literature screening, data extraction and visualization.
References
- 1. Djulbegovic B, Hozo I, Li SA, Razavi M, Cuker A, Guyatt G. Certainty of evidence and intervention’s benefits and harms are key determinants of guidelines’ recommendations. Journal of clinical epidemiology. 2021;136:1–9.
- 2. Djulbegovic B, Kumar A, Kaufman RM, Tobian A, Guyatt GH. Quality of evidence is a key determinant for making a strong GRADE guidelines recommendation. Journal of clinical epidemiology. 2015;68(7):727–32. pmid:25766057
- 3. Hazlehurst JM, Armstrong MJ, Sherlock M, Rowe IA, O’Reilly MW, Franklyn JA, et al. A comparative quality assessment of evidence-based clinical guidelines in endocrinology. Clinical endocrinology. 2013;78(2):183–90. pmid:22624723
- 4. Lee DH, Vielemeyer O. Analysis of overall level of evidence behind Infectious Diseases Society of America practice guidelines. Archives of internal medicine. 2011;171(1):18–22. pmid:21220656
- 5. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr., Scientific evidence underlying the ACC/AHA clinical practice guidelines. Jama. 2009;301(8):831–41.
- 6. Lotfi T, Hajizadeh A, Moja L, Akl EA, Piggott T, Kredo T, et al. A taxonomy and framework for identifying and developing actionable statements in guidelines suggests avoiding informal recommendations. Journal of clinical epidemiology. 2022;141:161–71. pmid:34562579
- 7.
National Health and Medical Research Council. Procedures and requirements for meeting the 2011 NHMRC standard for clinical practice guidelines-Summary for developers[M]. Melbourne:National Health and Medical Research Council,2011.
- 8. Andrews J, Guyatt G, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. Journal of clinical epidemiology. 2013;66(7):719–25. pmid:23312392
- 9.
Scottish Intercollegiate Guidelines Network. A guideline developer’s handbook[M]. Edinburgh:Scottish Intercollegiate Guidelines Network,2019.
- 10.
World Health Organization. WHO handbook for guideline development[M].2nd ed. Vienna:World Health Organization,2014.
- 11. Zhou Q, Wang Z, Shi Q, Zhao S, Xun Y, Liu H, et al. Clinical Epidemiology in China series. Paper 4: The reporting and methodological quality of Chinese clinical practice guidelines published between 2014 and 2018: A systematic review. Journal of clinical epidemiology. 2021;140:189–99. pmid:34416326
- 12. Chen Y, Wang C, Shang H, Yang K, Norris SL. Clinical practice guidelines in China. BMJ (Clinical research ed). 2018;360:j5158. pmid:29437564
- 13.
Graham R, Mancher M, Wolman DM, Greenfifield S, Steinberg E, editors. Clinical practice guidelines we can trust. Washington, DC: Institute of Medicine, National Academies Press; 2011.
- 14. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. Journal of clinical epidemiology. 2011;64(4):383–94. pmid:21195583
- 15. Brozek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW, et al. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy. 2009;64(5):669–77. pmid:19210357
- 16. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. Journal of clinical epidemiology. 2011;64(4):395–400. pmid:21194891
- 17. Wiercioch W, Nieuwlaat R, Zhang Y, Alonso-Coello P, Dahm P, Iorio A, et al. New methods facilitated the process of prioritizing questions and health outcomes in guideline development. Journal of clinical epidemiology. 2022;143:91–104. pmid:34843861
- 18. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ (Clinical research ed). 2008;336(7650):924–6. pmid:18436948
- 19. Ruxrungtham K. Guidelines to improve urticarial care and the remaining research gaps. Asian Pacific journal of allergy and immunology. 2016;34(3):179–80. pmid:27690470
- 20. Singh Ospina N, Rodriguez-Gutierrez R, Brito JP, Young WF Jr., Montori VM. Is the endocrine research pipeline broken? A systematic evaluation of the Endocrine Society clinical practice guidelines and trial registration. BMC medicine. 2015;13:187. pmid:26265226
- 21. Chou R, Ballantyne JC, Fanciullo GJ, Fine PG, Miaskowski C. Research gaps on use of opioids for chronic noncancer pain: findings from a review of the evidence for an American Pain Society and American Academy of Pain Medicine clinical practice guideline. The journal of pain. 2009;10(2):147–59. pmid:19187891
- 22. Zhang W, Moskowitz RW, Nuki G, Abramson S, Altman RD, Arden N, et al. OARSI recommendations for the management of hip and knee osteoarthritis, part I: critical appraisal of existing treatment guidelines and systematic review of current research evidence. Osteoarthritis and cartilage. 2007;15(9):981–1000. pmid:17719803
- 23.
Research gaps from evidence-based contraception guidance: the US Medical Eligibility Criteria for Contraceptive Use, 2016, and the US Selected Practice Recommendations for Contraceptive Use, 2016.
- 24. Robinson KA, Saldanha IJ, McKoy NA. Identification of research gaps from evidence-based guidelines: a pilot study in cystic fibrosis. International journal of technology assessment in health care. 2011;27(3):247–52. pmid:21756412
- 25.
National Institute for Health and Care Excellence. Research Recommendations Process and Methods Guide [Internet]. London: National Institute for Health and Care Excellence (NICE); 2015.
- 26. Brown P, Brunnhuber K, Chalkidou K, Chalmers I, Clarke M, Fenton M, et al. How to formulate research recommendations. BMJ (Clinical research ed). 2006;333(7572):804–6. pmid:17038740
- 27. Vlassov V. How to formulate research recommendations: format is not enough. BMJ (Clinical research ed). 2006;333(7574):917. pmid:17068045
- 28. Greenhalgh T. How to formulate research recommendations: the pie or the slice? BMJ (Clinical research ed). 2006;333(7574):917. pmid:17068044