There are multiple organisations in Russia that publish clinical practice guidelines (CPGs). The demand for CPGs and appreciation of their role in healthcare provision has been steadily growing. However, quality and methodology of development of CPGs have not been systematically addressed.
To analyse the quality of Russian-produced CPGs for surgical treatment of hepatic-pancreatic-biliary diseases.
We searched Russian databases for CPGs, published between 2013 and 2017. We identified 6 relevant documents that met our inclusion criteria. We approached four experts in the field with written and verbal instructions on the use of the AGREE II questionnaire.
All six CPGs received the highest domain scores for the domain Clarity of Presentation (46%–80%). The lowest domain scores were for the domain Editorial Independence (6%-25%). Overall, the experts put the highest total sum scores to the CPG for treating chronic pancreatitis (70%), while the lowest total sum score was attributed to the CPG for treating acute cholangitis (22%).
The overall quality of CPGs, as assessed by the four experts with the AGREE II instrument, was low. The highest scoring, best organized and most comprehensive and straightforward CPG was the one for chronic pancreatitis. The AGREE II instrument should be considered for use in Russia by guideline developers to assess existing CPGs and inform the creation of new guidelines.
Citation: Lienhard DA, Kisser LV, Ziganshina LE (2018) Assessing methodological quality of Russian clinical practice guidelines and introducing AGREE II instrument in Russia. PLoS ONE 13(9): e0203328. https://doi.org/10.1371/journal.pone.0203328
Editor: Gregory W.J. Hawryluk, University of Utah Hospital, UNITED STATES
Received: April 16, 2018; Accepted: August 17, 2018; Published: September 11, 2018
Copyright: © 2018 Lienhard et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work. Liliya E. Ziganshina received partial salary from the Russian Government Program of Competitive Growth of Kazan Federal University (https://kpfu.ru/eng/strategy-5-top- 100/roadmap-in-details) for development of Cochrane Russia. Another part of Liliya E. Ziganshina’s salary came from the Kazan Federal University government budget for teaching pharmacology. The Kazan Federal University receives federal government budgeting through the Ministry of Education and Science of the Russian Federation (https://kpfu.ru/sveden; https://xn--80abucjiibhv9a.xn--p1ai/) based on the number of students taught. Dina Lienhard (Ziganshina) received an hourly wage from the Arizona State University for working as an Instructional Aide. She taught BIO181/182 (General Biology) and CHM237/238 (General Organic Chemistry) labs at the ASU Downtown Campus between Fall 2015 and Spring 2018. She also received a travel grant from Susan and Marc Mulzet National Travel Award Program to attend the American Association for the Advancement of Science (AAAS) meeting in Austin, TX in February of 2018, as well as AAAS Section L funding. There she presented her Honors Thesis research on the history of mammography guidelines in the United States. (https://www.cbs.asu.edu/news/biology-and-society-graduate-begins-her-phd-journey). Lidiya V. Kisser received her salary from the Ministry of Health of Kaliningrad oblast (https://www.infomed39.ru/; https://www.infomed39.ru/ministry/structures/) and PhD student stipend from the Kazan Federal University. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. There was no additional external funding received for this study.
Competing interests: The authors have read the journal's policy and the authors of this manuscript have the following competing interests: Liliya E. Ziganshina is the Director of Cochrane Russia.LEZ has received remuneration from Cochrane for Russian translation work, and numerous travel and accommodation support for Cochrane events. LEZ is a member of the WHO Expert Advisory Panel on Drug Evaluation and a member of the WHO European Advisory Committee on Health Research. LEZ received travel and accommodation support from the WHO on multiple occasions. LEZ is an editorial board member of the International Journal of Risk & Safety in Medicine and has reviewed manuscripts for PLOS ONE. Dina A. Lienhard (formerly Ziganshina) is a graduate of Barrett, The Honors College at Arizona State University (ASU). She graduated with her Bachelor of Science degree in May 2018 and started a PhD program in Biology and Society in June 2018. Dina is a recipient of 2018 Moeur Award of the ASU Alumni Association for perfect record and other academic achievements. Dina is the daughter of Liliya Ziganshina. Lidiya V. Kisser is a practicing medical doctor, a clinical pharmacologist at the Municipal clinical hospital for emergency care of Kaliningrad belonging to the Ministry of Health of Kaliningrad oblast. She has been working for the Ministry of Health of Kaliningrad oblast with development of medicine reimbursement lists and guidelines and has been promoted during the editorial process to the position of Lead consultant at the Department of medicines and medical devices provision at the Ministry of Health of Kaliningrad oblast. Lidiya is a PhD student at the Department of Basic and Clinical Pharmacology at the Kazan Federal University. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
Clinical practice guidelines (CPG) are a useful tool for evidence-based healthcare provision. The World Health Organization (WHO), governmental health agencies and civil society organizations at the country level work to introduce them into clinical practice. The main goal of CPGs is to provide to practicing physicians the tools for quality evidence-based health care . However, in general, physician adherence to CPGs is low . In Russian Federation, the majority of official guideline documents have been developed in the past two decades. However, health care professionals in Russia mostly use these guidelines to make insurance and hospital financing decisions [3, 4, 5], not to inform clinical decisions.
In some instances, conflicting CPGs for treating the same condition may be available, originating from different professional organizations. In these cases, physicians need to make an informed decision about which guidelines to use. The quality and reliability of CPGs is very important to physicians, as the health outcomes of their patients directly depend on CPGs implementation. In this study we examine the quality of Russian CPGs for various conditions affecting the hepatopancreatobilliary system. In particular, we analysed CPGs for acute cholangitis, acute cholecystitis, acute pancreatitis, chronic pancreatitis and cholelithiasis by using AGREE II assessment tool.
AGREE instrument is an internationally recognized, validated tool for guideline development and assessment. It was introduced in 2003, and later updated in 2009 to become AGREE II. It allows assessing methodological rigour and transparency in which a guideline was developed . Ideally, AGREE II is to be used in the process of guideline development, as it provides a checklist for the most important issues that must be considered when developing any guideline universally. AGREE II consists of 6 domains with several questions each, which makes for a total of 23 items . AGREE II is often used in the Western world, but it is rarely used in Russia for guideline assessment and has never been used for guideline development. The use and importance of the AGREE II instrument are further discussed in the Methods section. For the convenience of our experts, the team of Cochrane Russia translated AGREE II into Russian (S3 Text).
Pathologies of the hepatopancreatobiliary system present the most common disorders of the digestive tract, managed surgically . Due to the complex anatomical structure and topography of this zone, severity of complications with high mortality rates and new costly technologies used for their management, hepatobiliary and pancreatic disorders increasingly present challenges to health systems . In Russia, diseases of digestive system affect 5% of total population, or 66.1 per 100 thousand people and present the third most common cause for hospital admissions with consistent increase from year to year .
Acute cholangitis is a bacterial infection that causes inflammation and obstruction of the biliary tract, which, if left untreated, it is potentially lethal . As the amount of bacteria in bile grows exponentially, the bacteria gets into the bloodstream, cause sepsis and, eventually, death . In Russia acute suppurative cholangitis complicated up to 27% of patients with cholelithiasis . The Russian Society of Surgeons created CPGs for treating acute cholangitis in 2016 . A brief description of the content of this CPG can be found in S1 Text.
Acute cholecystitis is an inflammation of the gall bladder. It is caused mainly by gallstones blocking the cystic duct . It is characterized by sharp pain in the upper right abdomen and may lead to gallbladder rupture if left untreated . The Russian Society of Emergency Medicine published CPGs for treating acute cholecystitis in 2014 . The Russian Society of Surgeons created national CPGs for treating acute cholecystitis in 2015 . A brief description of the contents of these CPGs, along with the major differences between them, can be found in S1 Text.
Cholelithiasis is a condition where gallstones are present in the gallbladder. Gallstones may block the bile duct and cause severe dull pain to the patient . The Russian Gastrointestinal Association created a set of CPGs on choleliathisis in 2015 . A brief description of the content of this CPG can be found in S1 Text.
Acute pancreatitis happens when the pancreas quickly becomes inflamed and the patient experiences sharp pain in the abdomen [11, 17]. Its prevalence in Russia ranges from 32 to 389 per million population, with death rate ranging from 6 to 12 people per 100 . The Russian Society of Surgeons, the Association of Hepatopancreatic Surgeons of CIS Countries, and the Russian Society of Emergency Medicine created a collective set of CPGs for treating acute pancreatitis in 2014 . A brief description of the content of this CPG can be found in S1 Text.
On the other hand, chronic pancreatitis is a condition where inflammation of the pancreas recurs over many years . In Russia chronic pancreatitis accounts for about five to 9% of all digestive diseases, considerably reducing patient quality of life . The Russian Society of Surgeons and the Association of Hepatopancreatic Surgeons of CIS Countries created a collective set of CPGs for treating chronic pancreatitis in 2014 . A brief description of the content of this CPG can be found in S1 Text.
Noteworthy is the recent consistent growth of benign and malignant pathology of pancreatic-biliary system with consequent growth of the prevalence of the syndrome of mechanical jaundice . Despite the introduction of modern sophisticated diagnostic procedures and minimally invasive endoscopic and percutaneous trans-hepatic interventions, there are many complications with lethality rates reaching 15–30%. .
One of the best tools used to analyse CPGs is a questionnaire-based tool called Appraisal of Guidelines for Research & Evaluation (AGREE). Now AGREE II is available for use. The AGREE II tool provides an opportunity to analyse the guidelines based on six distinct domains. Those domains are Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability, and Editorial Independence . We asked four experts in gastrointestinal surgery to assess the identified guidelines using this tool.
We included CPGs on acute cholangitis, acute cholecystitis, acute pancreatitis, chronic pancreatitis and cholelithiasis published between 2013 and 2017 in Russian language that were meant for health care professionals, specialising in surgery, providing care for adult patients in a hospital setting. The CPGs had to be available online.
Search methods for identification of guidelines
We searched for all relevant guidelines (CPGs) regardless of the level of their acceptance by either professional associations or health systems managerial bodies (approved / accepted for implementation, draft consultations / in development).
We used a PRISMA flow checklist to create the PRISMA diagrams for our literature searches (S1 Checklist). We conducted the primary search in October—December 2016 (S1 Fig), then repeated and expanded our search in April 2018, with the date of the last search 12th April 2018 (S2 Fig).
In 2016 we searched the following Russian databases:
- Federal Electronic Medical Library of the Ministry of Health of the Russian Federation (http://www.femb.ru/feml or http://feml.scsml.rssi.ru/feml)
- Websites of professional associations:
In 2018 we repeated the search in the databases we searched in 2016 and added the following two sources to the list:
Selection and analysis of CPGs with description of the databases and website
1. Federal Electronic Medical Library of the Ministry of Health of the Russian Federation.
Searched in 2016 and 2018. It was established by the Ministry of Health in 2012 on the basis the Central Scientific Medical Library of the First Moscow State Medical University named after I.M. Sechenov. It consists of seven databases, one of which is Clinical Guidelines (Protocols of treatment) with 1202 records on the 12th of April 2018. It is the primary electronic database for clinical practice guidelines. It operates in Russian only, allows simple and advanced search with the use of Boolean operators and filters for adults and children, and for the level of healthcare (emergency care, in-patient care, out-patient care and day-time in-patient care). All searches with results are stored in history. We used simple search terms, filtering for adults and not filtering for the level of care. Here we found four CPGs meeting our inclusion criteria (all approved):
- on acute cholecystitis—2 CPGs:
- 2015 from the Russian Society of Surgeons, and
- 2014 from the Russian Society of Emergency Medicine
- on acute cholangitis—no results
- on acute pancreatitis– 1 and the same CPG with 2 records:
- 2014 jointly from the Russian Society of Surgeons, the Association of Hepatopancreatic Surgeons of CIS Countries, and the Russian Society of Emergency Medicine
- 2014 from the Russian Society of Emergency Medicine
- on chronic pancreatitis—1 CPG jointly from the Russian Society of Surgeons, the Association of Hepatopancreatic Surgeons of CIS Countries, 2014
- on cholelithiasis—no results
2. The website of the Russian Society of Surgeons.
Searched in 2016 and 2018. It has a series of special pages for CPGs: approved for use Clinical Guidelines grouped in 8 sections, including abdominal surgery and emergency abdominal surgery; draft guidelines for comments and suggestions grouped in 4 sections; drafts to be approved at the 12th National meeting of the surgeons of Russia (acute appendicitis); and methodology papers on the process of guideline development. The website uses only Russian language; allows for eye screening of guidelines. Here we found four CPGs meeting our inclusion criteria:
- approved CPGs on acute cholecystitis (2015),
- approved CPGs on acute pancreatitis (2014),
- approved CPGs on chronic pancreatitis (2014), and
- a draft of CPGs on acute cholangitis (2016).
3. The website of the Russian Society of Emergency Medicine.
Searched in 2016 and 2018. It does not contain any clinical guidelines. It provides the website visitors with description of organisational structure and leadership of the Society, relevant governmental decrees and orders, information on events; it also provides platform for questions and answers, and the news. The website uses only Russian language. The CPGs, developed and approved by this Society are located on the Federal Electronic Medical Library of the Ministry of Health of the Russian Federation and on the website of the Russian Society of Surgeons.
4. The website of the Russian Association of Gastroenterologists.
Searched in 2016 and 2018. It consists of five major pages, delivering information on upcoming conferences, the association’s clinical practice guidelines, about the association, announcements and specifics for continuing medical education. The website uses only Russian language. There are 24 CPGs available for cursory screening. We identified two CPGs, meeting our inclusion criteria:
- on cholelithiasis (2015) and
- on chronic pancreatitis (produced in 2013, published 2014 identified in both searches of 2016 and 2018).
5. The official collection of clinical practice guidelines of the Ministry of Health (MoH).
Rubriсator–Рубрикатор, searched in 2018. It contains searchable collection of CPGs, grouped according to the International Classification of Diseases 10. It was launched in October—November 2017, establishing the system of National Clinical Guidelines. This database operates in Russian only, allows simple search by terms, registered number of a CPG, year of publication, ICD position, status (approved in action, draft in development and outdated) with filters for adults and children. Here we found three CPGs meeting our inclusion criteria (all approved):
- on acute cholecystitis (2015 from the Russian Society of Surgeons), the same as 2014, formatted to a new style
- on acute pancreatitis (2014 jointly from the Russian Society of Surgeons and the Association of Hepatopancreatic Surgeons of CIS Countries and the Russian Society of Emergency Medicine), and
- on chronic pancreatitis (2016 from Russian Association of Gastroenterologists).
It is the largest Russian electronic database of scientific publications; it is searchable. The database is integrated with the Russian Scientific Citation Index (RINC), which is a free publicly available tool for measuring publication activity of individual researchers and organizations, commissioned by the Ministry of Education and Science of the Russian Federation. The eLIBRARY.RU and RINC have been developed and supported by the Scientific Electronic Library Company. The platform eLIBRARY.RU was created in 1999 on the initiative of the Russian Foundation for Basic Research to provide Russian scientists with electronic access to leading foreign scientific publications. Since 2005, eLIBRARY.RU has started working with Russian-language publications and is now the leading electronic database of scientific periodicals in Russian in the world. Abstracts and full texts of more than 26 million research papers and publications, including electronic versions of more than 5300 Russian scientific and technical journals are available at eLIBRARY.RU. The total number of registered institutional users (organizations) is more than 2800. The system registered 1.7 million individual users from 125 countries. Annually readers receive from the library more than 12 million full-text papers and view over 90 million annotations. More than 4,500 Russian scientific journals are placed in free public access. For access to other publications, it is possible to subscribe or order individual publications. eLIBRARY.RU operates in Russian only, though it contains some publications in other languages, including English. We ran a series of searches by surgical condition / pathology of interest, which yielded finally 2 potential records, which were excluded after assessing full texts for eligibility (S1 Table).
eLIBRARY search strategy
This strategy is based on search specifics of the database: search by term morphology, meaning by the roots of the words typed in as full words (с учетом морфологии); search by combination of terms, working as Boolean operator OR; searching within results of the previous search working as Boolean operator AND; combining searches in the automated way to remove duplicates.
- acute cholecystitis AND clinical guidelines (острый холецистит И клинические рекомендации)
- acute cholangitis AND clinical guidelines (острый холангит И клинические рекомендации)
- acute pancreatitis AND clinical guidelines (острый панкреатит И клинические рекомендации)
- chronic pancreatitis AND clinical guidelines (хронический панкреатит И клинические рекомендации)
- cholelithiasis AND clinical guidelines (желчнокаменная болезнь И клинические рекомендации)
We identified a total of 6 CPGs (Table 1).
The four experts performed the assessment independently of each other. Each expert had a different specialization in the field of abdominal surgery (S1 Appendix), and thus presented a different approach to assessing the guidelines, which was most useful in obtaining a well-rounded, balanced analysis of CPGs. We equipped all four experts with the AGREE II instrument and with detailed information on how to use it. But we did not provide any formal training on the AGREE instrument.
The AGREE II is a questionnaire designed to assess the quality of health care guidelines and to be used as a tool to develop clinical practice guidelines . AGREE II has been available in English since 2010; the team of Cochrane Russia translated it into the Russian language (S3 Text).
AGREE II includes 23 questions in 6 different domains that address different aspects of guidelines. Those domains are Scope and Purpose (3 questions), Stakeholder Involvement (3 questions), Rigour of Development (8 questions), Clarity of Presentation (3 questions), Applicability (4 questions), and Editorial Independence (4 questions) . For each question, the expert is asked to provide a numerical score between 1 and 7, 1 being the minimum score and 7 being the maximum score. Each question is presented as a statement and the experts should answer the questions based on how much they agree with the statement (1 –strongly disagree, 7 –strongly agree). Each CPG receives a raw score between 23 and 161 points from a single expert. It is recommended that at least two experts analyse the same guidelines using the AGREE II tool. Additionally, experts may leave comments for each question and state whether they would recommend the use of the guideline at the end of the questionnaire.
Next, derived data is calculated by compiling the scores of all experts for each domain and scaling them as a percentage of a maximum possible score for that specific domain. That is done by using the formula (Eq 1): (1)
Table 2 below provides a sample set of data for Scope and Purpose domain for acute cholangitis CPG by the Russian Society of Surgeons (RSS), in which we show sample calculations to illustrate how the AGREE II scoring was used in our study:
The total obtained score is calculated by summing the individual scores of all experts (Eq 2): (2)
The minimum possible score is calculated by attributing 1 to each question, as the minimum possible score is 1 for each question and multiplying it by the number of experts (Eq 3): (3)
The maximum possible score is calculated by attributing 7 to each question, as the maximum possible score is 7 for each question and multiplying it by the number of experts (Eq 4): (4)
The original formula is used to calculate the domain score (Eq 5): (5)
We performed this calculation in Microsoft Excel. We created a separate data sheet for each CPG and another sheet—for the comparison of all guidelines. First, we created a table for each CPG with all 23 questions organized in the 6 domains. Each table included the data from four experts. We added the scores for each question from the four experts. Next, we calculated minimum and maximum scores for each domain. Then, we calculated the difference between the maximum and the minimum score for each domain. We used the above formula to calculate a domain score for each domain and the total sum of assessment for each CPG. Since we analyzed two CPGs for treatment of acute cholecystitis, we also created a separate comparison table for these two CPGs. Finally, we created a comparison table for all CPGs and made a bar graph to visualize the difference in total sum of assessment between all the CPGs studied.
The domain scores differed greatly between different CPGs. The domains Scope and Purpose, and Clarity of Presentation consistently received high scores across all CPGs. However, the domains Applicability and Editorial Independence consistently received low scores.
The domain Scope and Purpose deals with the main objective of CPGs, the health question and the population to whom the guideline is meant to apply . This domain includes 3 items (items 1–3). The range of scores for Scope and Purpose was 35%–100% (S9 Table). Most CPGs received high scores for this domain (50%–100%), while two CPGs received low scores of 35% and 36%. The CPG for treating acute cholecystitis by the Russian Society of Surgeons obtained the lowest score for this domain (S5 Table). The experts attributed it to the guidelines having a vague and not clearly defined objective. They provided similar critique about the CPG for treating acute pancreatitis (S7 Table). The CPG for treating chronic pancreatitis obtained the highest score for Scope and Purposes as the population was clearly described and the purpose was well defined (S8 Table).
The domain Stakeholder Involvement is focused on assessing whether all relevant clinical professionals participated in the development of the CPG and whether the target audience who would use the CPG is specified . This domain includes three items (items 3–6). The range of scores was 24%–76% (S9 Table). The CPG for treating acute cholangitis received the lowest score and the CPG for treating acute cholecystitis by the Russian Society of Emergency Medicine received the highest score (S2 and S5 Tables).
The experts assigned such a low score (24%) to the CPG for treating acute cholangitis for the domain Stakeholder Involvement as it did not include any specifications about the target audience and only surgical specialists participated in the development of the guidelines. All experts pointed out that clinical pharmacologists, general physicians, epidemiologists, and medical statisticians should have been included in the development of CPG for treating acute cholangitis.
The CPG for treating acute cholecystitis by the Russian Society of Emergency Medicine received the highest score (76%) for the domain Stakeholder Involvement (S5 Table). The experts were unanimous in their comments regarding the fact that more stakeholders could have been involved in the development of these guidelines, but overall the variety of specialists involved in the development of these guidelines was better than in all other CPGs. A similar score (74%) for the domain Stakeholder Involvement was obtained by the CPG for treating chronic pancreatitis (S8 Table). That set of CPGs received similar comments from the experts.
The domain Rigour of Development deals with the actual methods used to compose the CPGs . It is the largest domain in the AGREE II and consists of eight items (items 7–14). The range of scores for this domain was 9%–81% (S9 Table). Rigour and Development had the largest variation between the highest and the lowest scores, as it is the biggest and arguably the most important domain in the AGREE II. The scores for all CPGs varied greatly.
The lowest score for the domain Rigour and Development was obtained by the CPG for treating acute cholangitis (S2 Table). Experts supported their decision of giving this CPG such a low score by providing extensive comments. They pointed out that the CPG lacked a procedure of evidence search, strengths and limitations of this evidence and most other items of this domain. The only item in the Rigour and Development that received a score above 1 from three of the experts was the item 10, which deals with methods for creating the recommendations.
The highest score (81%) for the domain Rigour and Development was obtained by the CPG for treating chronic pancreatitis (S8 Table). Even though the experts gave this CPG a high score, they left many comments for this domain. Most experts were unanimous in saying that the CPG for treating chronic pancreatitis did not include a method for reviewing the strengths and limitations of the evidence (such as GRADE), or a Delphi method for formulating the recommendations.
The fourth domain is called Clarity of Presentation and it consists of three items (items 15–17). Clarity of presentation is focused on the way the CPG is written, specifically its language, format, and structure . This domain consistently received a high score in each of the CPGs (47%–81%) (S9 Table). The CPG for treating acute cholangitis received the lowest score (47%; S2 Table). The experts attributed it to the CPG missing crucial information, such as practical clinical questions and hiding the main recommendations in the body text, instead of pointing it out in the form of a table. The highest scores for this domain were achieved by the CPG for treating cholelithiasis (81%) and chronic pancreatitis (80%). The experts were unanimous in their comments, stating that the recommendations were unambiguous and clear (S6 and S8 Tables).
The next domain is called Applicability and it involves four items (items 18–21). Applicability mostly assesses the way that the CPG can be used in the society, including crucial factors such as assessment of cost, barriers and facilitators of implementation, as well as in possible tactics for uptake of the recommendations . Overall, Applicability received a low score from the experts throughout all CPGs (8%–55%) (S9 Table).
The lowest score (8%) for the Applicability domain was received by the CPG for treating acute cholangitis (S2 Table). The experts simply stated that there was not enough information about the implementation factors in the CPG. On the other hand, the CPG for treating cholelithiasis received the highest score for the domain Applicability, even though the experts indicated that they could not locate some of the information needed for this domain, such as the criteria for auditing and the possible facilitators and barriers to the application of the CPG (S6 Table).
The last domain in the AGREE II is called Editorial Independence and it consists of two items (items 22–23). This domain score reflects whether the authors could have any conflict of interest in the matter of creating the CPG . This domain received the lowest scores from all the experts. The range of the scores was 6%–25% (S9 Table).
The lowest score of 6% was obtained by the CPG for treating acute pancreatitis (S7 Table). All experts gave it the lowest rating (1 point) by stating that no information about the conflict of interest was provided, but Expert 4 gave this CPG 4 points for the item 22, which assesses whether the funding influenced the recommendations. The Expert 4 said that he simply did not know the answer to that question, and gave the item 4 points out of 7 possible points, which led to the CPG receiving 6% overall score for the domain Editorial Independence, instead of 0%.
The highest score (even though it was considered low as a separate score) was received by the CPG for treating chronic pancreatitis and the CPG for treating acute cholecystitis by the Russian Society of Emergency Medicine (both received 25%; S4 and S8 Tables). The experts provided no particular comments for their rating. Experts 1, 2 and 3 gave the lowest possible rating to this CPG (1 point), but expert 4 gave it the highest rating without any explanation, which was the reason that the CPGs for treating chronic pancreatitis and acute cholecystitis have a 25% rating.
Total sum of assessment scores
The total sum of scores, or the overall calculation was performed for each of the CPGs. Their comparison can be viewed in S9 Table. It was used to assess the overall quality of the CPGs. Overall, the highest score (70%) was received by the CPG for treating chronic pancreatitis (S8 Table). The experts only gave low scores to the domains Applicability and Editorial Independence, but those domains received low scores in all CPGs.
The lowest total sum score was obtained by the CPG for treating acute cholangitis (22%; S2 Table). It received scores of 50% or less for all domains. The authors attributed this to the CPG still being under development and suggested that many changes should be made before the CPG for treating acute cholangitis should be introduced into clinical practice.
Two different CPGs for treating acute cholecystitis were analysed (S3 and S4 Tables). The CPG for treating acute cholecystitis by the Russian Society of Emergency Medicine received a total sum score of 41%, while the CPG for treating acute cholecystitis by the Russian Society of Surgeons received a total sum score of 32% (S5 Table). Even though both scores were low, the experts attributed higher quality recommendations to the CPG from the Russian Society of Emergency Medicine.
Evidence-based medicine has become the mainstream and key to the development of clinical medicine and health care systems over the last thirty years . Different teams developed multiple CPGs over time and there are no set standards as to which guidelines to use, so physicians have to use their best judgment in deciding which CPGs to implement. Although there are many papers describing how to determine whether a certain CPG is of good quality, qualitative and quantitative data gathered with AGREE II instrument currently is the best validated tool for a physician or a health setting / system to decide whether to implement guidelines or not . CPG is a relatively new concept for the Russian health community, thus, difficulties in creating high quality CPGs still exist. There are no yet fully agreed upon and accepted official standards on how the CPGs are to be formulated and formatted or what information must be included. Our findings of low overall scores for all studied CPGs confirm this.
We found the combination of two problems—variability in assessment scores across the people using the AGREE tool and variability in quality of the assessed CPGs. We think that there are many reasons for this variability, including the following.
As the concept of CPGs is relatively new, most organizations that create CPGs get formal approval of a CPG at their conferences or other important professional meetings, where many experts in the field are present. Every CPG is different in terms of its structure, content organization, use of evidence, etc. Often, it is unclear who the authors are or what their clinical specialty is, as only their degrees, such as PhD or DSc are mentioned.
Two CPGs (chronic pancreatitis and acute pancreatitis) were developed by a coalition of organizations, while all other CPGs were created by single organizations (Table 1). Interestingly, the CPG for treating chronic pancreatitis created by the coalition of the Russian Society of Surgeons and the Association of Hepatopancreatic Surgeons of CIS Countries received the highest total assessment score among all studied CPGs (70%), while the CPG for treating acute pancreatitis by the same organization in partnership with the Russian Society of Emergency Medicine received the second lowest total assessment score (24%) (S7 and S8 Tables). The discrepancy of scores is especially noteworthy as the acute pancreatitis CPG was created by three organizations, and the one for chronic pancreatitis was developed by the coalition of two of those organizations.
Two CPGs for treating acute cholecystitis created by two different organizations were assessed. Their total assessment scores were both low, but the CPG by the Russian Society of Emergency Medicine had a higher score than the one created by the Russia Society of Surgeons (S5 Table). Noteworthy is the lack of consistency in quality even among CPGs created by the same organization, specifically the Russian Society of Surgeons.
The quality of some CPGs should indeed be questioned due to complete absence of in-text citations. Only two of six CPGs (for acute cholangitis, and for chronic pancreatitis) consistently cited references throughout the text; the CPG for cholelithiasis only referred to literature citations in the history section. Also, the majority of CPGs except for acute pancreatitis and for acute cholecystitis by the Russian Society of Emergency medicine, used mostly sources published in English and not in Russian. This may be another reason for variable quality of CPGs as assessed by the experts, as many recommendations that are easily implemented in Western countries may not be practical in some regions of Russia due to technological and financial constraints. We did not find any interrelation between the quality of a CPG and the provided references.
Overall, the scores in each domain varied greatly from one expert to the other. One of the reasons, why the answers of the experts differed so much, was that each of them specialized in a different aspect of abdominal surgery, ranging from endoscopic and endocrine surgery to emergency general surgery.
We think that one more reason for these variations might be that Russian medical education is very different from that of Western countries. Our healthcare professionals are not trained to critically appraise literature, to apply methodology concepts, or to assess any health policy documents. From our experts’ comments we notice that they may be biased towards certain parts of the CPGs depending on whether they agree with the proposed treatment option or not, rather than assessing the methodological aspects of the CPGs per se. In addition to this, some of the CPGs more closely resemble reviews of Western (mostly British or American) CPGs rather than an analysis of evidence in creating CPGs appropriate for Russia specifically.
It is interesting to note that Expert 4 gave very high ratings to most CPGs, while the other three experts gave consistently low and similar scores and comments to all CPGs. This could possibly be attributed to the fact that the first three experts were located in one city (Kaliningrad), while expert 4 was located in Kazan. Experts 1, 2, and 3 could have communicated among themselves or had been influenced by the same factors, while expert 4 was not influenced by those factors. Also, Kaliningrad is the only city in Russia that is located away from the rest of the country. It is located in Europe, so the first three experts may have had more exposure to Western methodology assessment techniques than the expert in Kazan did. It is also a possibility that the instructions for the use of AGREE II were better explained to the first three experts, as they were handed the CPGs in person, while expert 4 only received them over e-mail with little guidance on how to analyse them.
None of the experts received practical training on how to use the AGREE II tool. This explains the fact that many times experts gave a low sum score to a particular CPG, but rated it 7/7 for the overall quality (last question on the AGREE II questionnaire, not used in the calculation of total points) and said they would recommend the use of guidelines with some changes. Often, the experts questioned the validity of the guidelines due to absence of factors that they deemed important, but still said that they would use those CPGs. The AGREE II instrument is not meant to test the validity or reliability of guidelines, simply the methods of their creation and the potential to be implemented in the society .
We also assessed the Russian CPGs independently to see whether our results would be different from those given by the experts (S10–S17 Tables). Each author analyzed the CPGs with the AGREE II tool separately and independently. We did not refer to each other’s scores for comparison. After each of the authors scored the CPGs and provided comments, we followed the same protocol as described in the Methods section to calculate domain scores and total sum of assessment scores for each CPG. We also made summarizing tables (S9 and S17 Tables) and bar graphs to easily visualize the difference in scores between the experts and the authors.
Interestingly, as the figures below show, we gave overall lower total sum of assessment scores to all guidelines, yet the general trend of which CPGs got higher / lower scores was similar to that of the experts (Figs 1 and 2). The abbreviations in those figures correlate to those in Table 1. Our scores were about 1.5 to 4 times lower than those of the experts. We made every effort to only analyse methodological quality of guidelines and the possibility of being practically implemented in Russia, as intended by AGREE instrument . We gave a 0% score to all CPGs for the last domain, Editorial Independence, as not a single CPG specified any funding or competing interests in their text. It is possible that our experts were more familiar with the authoring organizations of the CPGs and thus assessed the last domain based on their prior knowledge of possible funding and competing interests. The detailed scores for each domain of each CPG can be found in S10–S16 Tables.
Even though there was a large difference in the scores, we rated the CPG for treating chronic pancreatitis as the highest quality one, just as the experts did. Thus we think that these our findings would not change much if some formal cursory training for experts was performed. In fact, we plan for the trainings with pre- and post-training assessment, which we will hopefully be able to perform in the design of a randomized controlled trial (AGREE assessments by experts after training versus no training). However, by the time we will be able to have training performed there will be new editions of the CPGs, which we hope will be better due to the current momentum in the country with high governmental attention to guideline process, including our humble contribution with this manuscript.
The appreciation of the value of evidence-based guidelines by the Russian health system leaders and the high level governmental leadership has been growing recently, with the launch of the official collection of clinical practice guidelines of the Ministry of Health of the Russian Federation (Rubriсator–Рубрикатор) in late 2017 thus establishing the system of the National Clinical Guidelines. Our repeated searches of April 2018 showed that this has not yet solved the problem of multiple guideline documents developed and endorsed by various professional organisations or health policy makers. We hypothesize that our results of AGREE II assessment of this small collection of guidelines may be generalised to other clinical fields. However, this needs to be studied in a special project of applying the AGREE II tool to various sets of guidelines.
The AGREE II instrument is rarely used in Russia to assess guidelines, and there is no information whether any of the developers of the CPGs, assessed in this study, used AGREE II to develop them. The Russian situation with CPGs’ development seems to be typical for other upper middle-income countries, like China , or Turkey , as assessed with AGREE II instrument. The current Russian situation calls for urgent energetic action for guideline assessment, development and application.
We plan to undertake a follow-up study in two-three years’ time to determine whether the overall quality of Russian CPGs for treating hepatopancreatobiliary pathologies improved.
The overall quality of the CPGs was low as the total sum score ranged between 22% and 70% (S9 Table). All CPGs needed improvements, especially in the Rigour of Development domain. The highest scoring domain of the AGREE II was Clarity of Presentation among all CPGs and the lowest scoring domains were Applicability and Editorial Independence. The highest scoring, best organized and most comprehensive and straightforward CPG was the one for chronic pancreatitis.
These findings and the limitations of this research are important to the Russian health community as a whole, as they show that Russian medical professionals are not skilled enough in the development of Evidence-Based Clinical Practice Guidelines or use of the AGREE II instrument. The results also show that the AGREE II instrument can be used efficiently in assessing existing Russian CPGs and producing better quality clinical recommendations in the future.
S3 Text. AGREE II instrument translation into Russian.
S2 Table. Assessment of CPG for treating acute cholangitis with AGREE II.
S3 Table. Assessment of a CPG for treating acute cholecystitis with AGREE II (by the Russian Society of Surgeons).
S4 Table. Assessment of a CPG for treating acute cholecystitis with AGREE II (by the Russian Society of Emergency Medicine).
S5 Table. Comparison of the assessment of two CPGs for treating acute cholecystitis with AGREE II.
S6 Table. Assessment of CPG for treating cholelithiasis with AGREE II.
S7 Table. Assessment of CPG for treating acute pancreatitis with AGREE II.
S8 Table. Assessment of CPG for treating chronic pancreatitis with AGREE II.
S9 Table. Total sum of assessment for all CPGs with AGREE II.
S10 Table. Authors’ assessment of CPG for treating acute cholangitis with AGREE II.
S11 Table. Authors’ assessment of a CPG for treating acute cholecystitis with AGREE II (by the Russian Society of Surgeons).
S12 Table. Authors’ assessment of a CPG for treating acute cholecystitis with AGREE II (by the Russian Society of Emergency Medicine).
S13 Table. Authors’ comparison of the assessment of two CPGs for treating acute cholecystitis with AGREE II.
S14 Table. Authors’ assessment of CPG for treating cholelithiasis with AGREE II.
S15 Table. Authors assessment of CPG for treating acute pancreatitis with AGREE II.
S16 Table. Authors’ assessment of CPG for treating chronic pancreatitis with AGREE II.
This project was developed partly according to the Russian Government Program of Competitive Growth of Kazan Federal University
The authors would like to acknowledge support and assistance of Professor L.E. Slavin, Professor L.V. Kim, Dr. G.I. Kuznetsov, Assoc.Professor I.Z. Weissbein.
- 1. Jarošová D, Žitníková P. Assessing the methodological quality of clinical practice guidelines for preventing intravascular catheter-related infections. Cent Eur J Nurs Midw. 2014; 5(4):176–183.
- 2. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don't physicians follow clinical practice guidelines?: A framework for improvement. JAMA. 1999 Oct 20;282(15):1458–1465. pmid:10535437
- 3. Law Branch of the Medical Rights Department. Clinical recommendations or protocols of care. Kormed. http://www.kormed.ru/kontrol-kachestva-medicinskoi-pomoschi/klinicheskie-rekomendatsii-ili-protokoly-lecheniya
- 4. New clinical recommendations: Inform physicians. Zdrav.ru. 2017. https://www.zdrav.ru/articles/4293657969-17-m11-03-novye-klinicheskie-rekomendatsi
- 5. Shevchenko R. Amendements to the law will change the role of clinical recommendations and standards. Medvestnik, Russian Doctors' Portal. 2017. https://www.medvestnik.ru/content/news/Popravki-v-zakon-izmenyat-rol-klinicheskih-rekomendacii-i-standartov.html
- 6. Yadav D, Lowenfels AB. The Epidemiology of pancreatitis and pancreatic cancer. Gastroenterol. 2013;144(6): 1252–1261. http://dx.doi.org/10.1053/j.gastro.2013.01.068
- 7. Russian Ministry of Health Care. 2014 Report on public health status and organization of health care in regards to the actions of the executive authority in subjects of Russian Federation. Rosminzdrav.ru. 2014. https://static-0.rosminzdrav.ru/system/attachments/attaches/000/026/627/original/Doklad_o_sostojanii_zdorovja_naselenija_2014.pdf?1434640648
- 8. Mosler P. Diagnosis and management of acute cholangitis. Current gastroenterology reports. 2011 Apr 1;13(2): 166–172. pmid:21207254
- 9. Malakhanov V, Seliverstov P, Gumerov R. Abnormality of biliary patency. Siberian Medical Journal. 2016;8: 5–10.
- 10. Russian Society of Surgeons. Draft national clinical guidelines for acute cholangitis. Russian Society of Surgeons. 2016. http://xn----9sbdbejx7bdduahou3a5d.xn--p1ai/stranica-pravlenija/klinicheskie-rekomendaci/urgentnaja-abdominalnaja-hirurgija/ostryi-holangit.html.
- 11. Indar AA, Beckingham IJ. Acute cholecystitis. BMJ. 2002;325(7365): 639–643. pmid:12242178
- 12. Mayo Clinic. Cholecystitis (symptoms and causes). Mayo Clinic. 2017. https://www.mayoclinic.org/diseases-conditions/cholecystitis/symptoms-causes/syc-20364867
- 13. Kabanov MY, Yakovleva DM, Degtyarev DB, Sementsov KV, Lyichev AB, Dimnikov DA, et al. Clinical recommendations (protocol) of emergency medical care for acute cholacystitis. Russian Society of Emergency Medicine. 2014. https://cyberpedia.su/1x54cc.html
- 14. Russian Society of Surgeons. Clinical recommendations—acute cholecystitis. Rubricator of clinical recommendations. 2015. http://cr.rosminzdrav.ru/schema.html?id=732#/text
- 15. Heuman DM. Gallstones (Cholelithiasis): Practice Essentials, Background, Pathophysiology. Medscape. 2017. https://emedicine.medscape.com/article/175667-overview.
- 16. Ivashkin VT, Mayev IV, Baranskaya EK, Okhlobystin AV, Shulpekova YO, Trukhmanov IS, et al. Gallstone disease diagnosis and treatment: guidelines of the Russian gastroenterological association. Russian Gastrointestinal Society. 2015. http://www.gastro.ru/userfiles/R_ZKB_2017.pdf
- 17. Banks PA, Conwell DL, Toskes PP. The management of acute and chronic pancreatitis. Gastroenterol hepatol. 2010 Feb;6(2): 1–16.
- 18. Spirin AN. Differential treatment of acute mild pancreatitis. PhD Dissertation, Saint-Petersburg I.I. Dzhanelidze Research Institute of Emergency Medicine. 2015. http://medical-diss.com/medicina/differentsirovannoe-lechenie-ostrogo-pankreatita-legkoy-stepeni
- 19. Russian Society of Surgeons, Association of Hepatopancreatic Surgeons of CIS Countries, Russian Society of Emergency Medicine. Diagnosis and treatment of acute pancreatitis (Russian clinical recommendations). 2014. http://pandia.ru/text/80/140/50065.php
- 20. Zakharova AV. Improvements of diagnostic and surgical tactic for patients with mechanical jaundice. Practical Medicine. 2013;2: 156–159.
- 21. Russian Society of Surgeons, Association of Hepatopancreatic Surgeons of CIS Countries. Clinical recommendations for surgical treatment of patients with chronic pancreatitis. 2014. http://stomatologo.ru/associaciya-gepatopankreatobiliarnih-hirurgov-stran-sng-klinic.html
- 22. Natalsky AA. Contemporary methods of diagnosis and treatment of mechanical jaundice. PhD dissertation, Ryazan State Medical University named after I.P. Pavlov. 2017. http://docplayer.ru/26409792-Sovremennye-principy-diagnostiki-i-lecheniya-sindroma-mehanicheskoy-zheltuhi.html
- 23. Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. for the AGREE Next Steps Consortium. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J. 2010. Dec 2010; 182:E839–842;
- 24. Sackett DL. Evidence-based medicine. Semin Perinatol. 1997 Feb 28; 21: 3–5. http://dx.doi.org/10.1016/S0146-0005(97)80013-4 pmid:9190027
- 25. Hayward RS, Wilson MC, Tunis SR, Bass EB, Guyatt G. Users' Guides to the Medical Literature: VIII. How to Use Clinical Practice Guidelines A. Are the Recommendations Valid?. JAMA. 1995 Aug 16;274(7):570–4. pmid:7629987
- 26. Zhang Z-w, Liu X-w, Xu B-c, Wang S-y, Li L, Kang Y-y, et al. (2013) Analysis of Quality of Clinical Practice Guidelines for Otorhinolaryngology in China. PLoS ONE 8(1): e53566. https://doi.org/10.1371/journal.pone.0053566 pmid:23349719
- 27. Yaşar I, Kahveci R, Baydar Artantaş A, Ayhan Başer D, Gökşin Cihan F, Şencan I, et al. (2016) Quality Assessment of Clinical Practice Guidelines Developed by Professional Societies in Turkey. PLoS ONE 11(6): e0156483. https://doi.org/10.1371/journal.pone.0156483 pmid:27295303