https://orcid.org/0000-0003-3890-7735
Figures
Abstract
Objective
To describe a protocol to achieve consensus on valid and important indicators to assess primary health care (PHC) quality regarding all clinical contexts of PHC in European health systems.
Study design
Qualitative study using the Delphi technique to gain consensus among European panels of experts comprising a heterogeneous professional background.
Methods
Potential candidate indicators were extracted and translated according to a set of informative elements (i.e., name, description, formula, unit of analysis, and sources). This list was then independently reviewed, and duplicates were removed totaling 1726 indicators. To guarantee a good response rate, indicators were distributed across 57 Delphi panels organized by clinical context. Each panel is a Delphi process, assessing between 23 to 33 indicators. Experts’ opinions on the validity and importance of the extracted indicators will be obtained through two rounds of online questionnaires, using a 9-point Likert scale and free-text boxes. To prevent biased responses, participation will be anonymous to other participants and to the team administrating panels. Consensus will be considered if at least 70% of ratings (≥7 assuming 10 participants) lie within the 7–9 range and less than 15% of ratings (<2 assuming 10 participants) are in the 1–3 range. Analysis of results will be streamlined and generalizable across panels using scripts.
Conclusions
This protocol will contribute to improve the quality of PHC in Europe by achieving a consensual and concise list of PHC quality indicators retrieved from the scientific literature that fit current clinical guidelines and populations’ needs in countries from the European region according to the World Health Organization.
Citation: Lobo M, Pinto A, Conceição G, Escadas S, de Medeiros AM, Dias B, et al. (2024) Selection of primary health care quality indicators in Europe: A Delphi study protocol. PLoS ONE 19(10): e0309395. https://doi.org/10.1371/journal.pone.0309395
Editor: Vivian Aline Mininel, Federal University of Sao Carlos: Universidade Federal de Sao Carlos, BRAZIL
Received: November 9, 2023; Accepted: August 12, 2024; Published: October 24, 2024
Copyright: © 2024 Lobo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting information files.
Funding: This article was supported by National Funds through FCT – Fundação para a Ciência e a Tecnologia , I.P., within Cintesis, R&D Unit (reference UIDB/4255/2020).
Competing interests: The authors have declared that no competing interests exist.
Introduction
Primary care plays a central role in providing healthcare, contributing to health promotion and prevention. It differs from other care settings by providing continuous care in an integrated manner focused on the whole person, according to each patient’s unique needs, culture, values, “and preferences in a particular family and community environment. Being the patient’s first contact with the health system, primary health care (PHC) practices coordinate care within the health system across other specialty care, hospitals, among others as deemed necessary by PHC providers [1–3]. Previous studies indicate that good quality PHC systems are associated with lower hospitalization rates, fewer emergency visits, lower total costs, improved satisfaction both from patient and healthcare providers, improved adherence to clinicians’ recommendations, improved health and better self-reported health outcomes, greater equity, less service duplication, higher overall efficiency, improved patient self-management for chronic conditions, among others [2]. Due to its ability to maximize the quality of health care delivered and the well-being of populations, enhancing PHC is increasingly a policy priority for health systems around the globe [4].
Health indicators are an important tool for any healthcare system to improve performance and the quality of the healthcare provided to patients. These consist of measurable elements of the structures, processes, and outcomes of care and are usually operationalized as a ratio between a numerator and a denominator specifying the population at risk [5].This definition is in keeping with the Donabedian framework for quality improvement and with the definition of quality of care as “the degree to which health services increase the likelihood of desired results and are consistent with current professional knowledge,” proposed by the National Academy of Medicine, former Institute of Medicine in 1999 [6].
Monitoring health indicators has become indispensable in health policy activities to raise awareness, foster transparency and accountability for the quality of healthcare delivered, and incentivize improving healthcare quality. Additionally, health policies defining financial incentives based on health quality indicators, such as pay-for-performance programs or penalties based on performance level targets, are becoming more common [7, 8]. As of 2018, thirteen OECD countries have introduced new payment models to encourage high-quality primary health care. However, this requires the right set of measures covering all health care provision and coordination aspects, including patient-reported experiences and outcomes measures [9].
There is a vast collection of studies in the scientific literature proposing health indicators to assess the quality of primary care. Ramalho et al. have recently performed an overview of systematic reviews assembling 727 indicators proposed to measure the quality of PHC [10]. Monitoring so many indicators is complex and time-consuming. Hence a shorter setlist of indicators could facilitate this task and would be of great value to health policymakers, healthcare providers, and patients to inform about the quality of PHC. Moreover, indicators retrieved from the scientific literature may no longer fit the current clinical guidelines and populations’ needs [11, 12]. Specifically, internationally developed indicators may not apply or have little relevance within the context of a health system of a specific country since health systems worldwide vary in their organization, financing systems, and population needs [13]. However, international comparisons are important instruments to public health and health policy, exposing areas of health care that need improvement and readily elucidating how to achieve better healthcare quality and health. Such comparisons became ubiquitous, especially due to efforts of international organizations such as the OECD Statistics (https://stats.oecd.org/), the WHO Observatory on Health Systems and Policies (https://www.euro.who.int/en/about-us/partners/observatory), the GBD project (http://www.healthdata.org/gbd), among others. Sound comparisons across countries require a common set of valid measures that are relevant to and feasible in those countries [14]. Particularly, countries that share global strategic synergies to improve the health of citizens, sharing many common features regarding PHC, such as the fact that it is either the filter for access to specialists or users have advantages in using it for that end, can greatly benefit from utilizing a standard list of PHC quality indicators.
Study objectives
This study will use the Delphi technique to produce a validated selection of quality indicators to assess primary health care in countries from the European region (https://www.who.int/countries). Specifically, we will derive consensus among experts regarding the validity, including face and content validity, and importance of quality indicators previously identified through an overview of systematic reviews [10]. Based on the patterns of response across countries and experts’ profiles, we will discuss the applicability of the selected indicators for within and between countries comparisons.
Methods
Preparation of candidate indicators
Instead of developing indicators for PHC, these have been abstracted from the literature by our research team. This process has considered previous work carried out in an umbrella review of systematic reviews covering PHC quality indicators and the primary studies of the reviews included in this umbrella [10]. In total, 1726 indicators (S1 Table) were identified, covering different types of indicators (e.g., outcome, process, and structure), capturing different types (e.g., acute, chronic, preventive) and purposes (e.g., screening, diagnosis, treatment, follow up) of care, as well as, different domains of quality of care, and covering several clinical contexts (i.e., WHO ICPC-2 chapters categorization) [15].
Our team examined and translated indicators extracted according to a set of informative elements (i.e., name, description, formula, unit of analysis, and sources). The list of indicators was then reviewed by five independent medical doctors, who also removed any remaining duplicates. This process was carried out because we believe that enforcing this consistency among indicators would improve the interpretability and comparison of indicators by the Delphi participants.
Study design
This protocol proposes the Delphi process to select PHC quality indicators [16], which will comprise 57 Delphi panels covering indicators of different clinical areas to be administered to experts in PHC (comprising different PHC stakeholders) working in European countries.
The Delphi process seeks to obtain consensus from the opinions of experts. Traditionally, the process consists of a series of structured questionnaires (also known as rounds), that each participant responds anonymously and iteratively. Anonymizing individual responses is essential to prevent bias, as participants may be influenced if the identity of other respondents is known. At the end of each round, the panel responses are summarized and shared with the participants, who are then allowed to respond in light of both the individual and the group’s responses. The process ends when a consensus is achieved [17–19]. More recently, the real-time Delphi process has been proposed as a more efficient variation of the traditional Delphi process. Essentially, it allows participants to review their answers in a questionnaire considering the real-time group’s response [20].
The Delphi study will be conducted online using a software to aid with the communication with participants, improving access, data collection and results sharing. Two rounds of questionnaires will be administered anonymously to participants via the eDelphi program (www.edelphi.org). The rounds will use 9-point Likert scales to evaluate the level of importance and validity of quality indicators, alongside free text boxes. The first round will be carried out as a traditional Delphi round to ensure anonymity and prevent biased ratings. The second round will be carried out in real-time to support an interactive and efficient process that simultaneously ensures effective discussions between participants.
All rounds will allow for asynchronous answering so that one participant can take part several times and change their answers until the end of a given time frame (expected time of two weeks) is reached. This time frame may be expanded if a low response rate is experienced.
Definitions of the selection criteria
The selection criteria that participants will have to consider are importance and validity. Definitions of these criteria will be provided at the beginning of each round as follows:
- Importance: The indicator should capture important performance aspects of primary health care. It should address areas of concern of policymakers where there is a clear gap and potential for improvement through identifiable events associated with the healthcare system. It should also permit useful comparisons within and across countries [21, 22].
- Validity: The indicator is valid if it is clinically logical and supported by consensus (i.e., face validity); represents a valid measure supported by evidence demonstrating a correlation with the quality of care (i.e., content validity); is reproducible by and comparable across different countries, organizations, and providers, as well as using different data sources over time (i.e., reliable/consistent); detects important changes in the quality of care discriminating well different levels healthcare quality (i.e., sensitivity and specificity) [5, 23, 24].
Development of the Delphi questionnaires/panels
The final list of candidate indicators has been ordered by clinical context and similarity and then distributed across fifty-seven Delphi questionnaires/panels. The number of panels per clinical context varied between 1 to 11 panels, depending on the number of existing indicators for each of the contexts, and each panel included between 23 and 33 indicators. Since previous studies using the Delphi technique to select healthcare quality indicators varied greatly in the number of indicators included in the first questionnaire [16], the number of indicators per panel was determined based on a compromise between the response/drop-out rate and the number of Delphi panels. The ideal number of indicators estimated to minimize dropouts was 25–30 indicators per panel. In most cases, all the indicators assigned to one panel correspond to a single clinical context to make the evaluation more meaningful by comparison of context-wise indicators. However, some panels combined indicators from different clinical contexts because the number of indicators regarding some contexts was small (S1 File).
eDelphi.
The eDelphi (eDelphi.org) is a subscription license online program specifically designed for the qualitative use of the Delphi method affording complete anonymous participation (responses are tagged with an anonymized id). It has plenty of options that allows to streamline the entire panel management of a Delphi process, which was a key consideration in this study, given the large number of panels to implement. Specifically, 1) managing invitations and reminders are simplified with the eDelphi; 2) the acceptance link that is sent to participants is personal, hence allowing them to respond to a questionnaire in the eDelphi without the need to register/login in the program; 3) exporting results as an image (bubble plot) or a google sheet. It is a convenient feature to feedback results with participants since plots contain descriptive measures of the group responses (Q1, Q2, Q3) over two dimensions; and 4) to analyze consensus on a regular basis with a statistical software of our choice.
Several limitations of the eDelphi have also been identified. First, since participation in a panel is anonymous, it will be virtually impossible to distinguish between participants that complete the questionnaire from those that respond to some questions (or even enter the questionnaire but do not respond to any question). Therefore, reminders will be sent to all participants who have accepted to participate in the round of a panel, unless participants disclose their identity (e.g., using the link to reject participation after having accepted or sending us an email). The only way we could overcome this limitation would be to export results regularly to link new answers with new participants. Second, because experts participate anonymously, the individual responses made in the first round can only be presented to the respective participant in the second round if the same type of question is chosen in round 2. Third, the Delphi does not provide a design mode to implement mandatory questions; hence participants can submit blank questionnaires. Since the participation is anonymous, we won’t be able to ask participants to complete specific information. Participants with all questions blanks will be removed from the analysis at the end of each round.
Study population
Participants.
We will invite general practitioners and public health physicians, nurses, and other healthcare professionals, as well as researchers and other stakeholders in healthcare quality assessment living or working in European countries (S1 File). Panel members will be selected from different fields relating to clinical practice, health policy, and academia. We will recruit as many experts as required to achieve a 10-rating minimum per quality indicator in each Delphi panel. This is in line with the Rand UCLA criteria method that advocates that nine experts consist of a sufficiently large panel to achieve consensus and to incorporate the necessary heterogeneity needed to ensure study quality [25]. For a total of 57 Delphi panels, the expectation is that we will need to invite a minimum of 2300 participants (approximately 40 participants per panel) based on an acceptance rate of 30% (based on a related pilot study) and a drop-out rate of 10% [26]
Recruitment.
To reach the targeted experts, we will use contacts in the public domain of members of European and National Professional Associations and related working networks and groups (e.g., EUPHA, WONCA, Nurses associations), editors of journals classified as PHC according to the Journal Citation Report (https://jcr.clarivate.com/jcr/browse-category-list), as well as experts identified in PHC conferences. If a contact is not immediately available on the website of the organization considered, scientific articles published by target experts will be searched using the name and affiliation information to identify a (potential) useable email contact.
Formal invitations to participate in the study will be sent by email (S1 File), which will include a link to a consent form containing information about this study and a brief sociodemographic questionnaire (S2 File). To increase sample size, we will also use a “exponential non-discriminative snowball sampling” approach by asking the expert contacted initially to forward the invitation email to their peers. Experts consenting to participate in the study will constitute the pool of potential participants to be distributed across the different Delphi panels. Participants completing a specific panel will be asked if they would like to participate in other panels. (Fig 1).
*Refers to Decision matrix in Table 1.
Study procedure
Fig 1 summarizes the study procedure. We will use the eDelphi program to develop online questionnaires for each round of each panel and to invite potential participants from the pool of experts consenting to participate in the study. Experts will be invited to respond to a round of a panel via eDelphi by email with a link to accept and a link to reject participating in that round (S1 File). The email will include information about the study aim, the expected time for completion of the questionnaire, the ethics approval of the study, a request to complete each round within two weeks, and our commitment to share results at the end of the study. Reminders will be sent on a weekly basis. Participants rejecting or explicitly withdrawing (after accepting participating) from the panel will be emailed a link to a form requesting to provide a reason for declining participation (S3 File).
Round 1.
The first round will be carried as a traditional Delphi round, where experts participate anonymously. An initial set of 40 invitations per panel will be sent assuming that this number will be enough to guarantee ten ratings per indicator. To promote a diverse and inclusive panel, invitations will be addressed to experts from at least two (ideally more) European countries and will comprise at least one general practitioner/family doctor, one public health doctor and one nurse.
In round 1, participants will initially be informed about the study and how to answer the questionnaire through a brief description of the structure of the questionnaire and selection criteria (S1 File), an information sheet (S4 File) and video tutorial (https://youtu.be/SaKSkyrYq9M) as well as they will be asked to provide demographic information including their gender, age, country of residence/work, level of education, professional area, and type of work institution. This will be necessary to assess experts’ profiles in each round and panel.
Participants will then be asked to use radio buttons to rate indicators according to two criteria: 1) Importance and 2) Validity, expressing how much they agree with a statement defining each criterion on a scale from 1 –highly disagree to 9 –highly agree (Fig 2). Participants will be encouraged to leave comments in a free-text box to justify ratings between 1 and 3, if the indicators are not clearly expressed.
Participants who do not start answering the questionnaire (only login) while the round is open (after clicking the link to accept to participate in the round/panel) will not be invited to round 2.
An indicator will be removed from the second round depending on the number of criteria that reached consensus and the type of consensus reached, according to Table 1. Specifically, indicators that either reached consensus regarding the lack of “Validity” or “Importance” will be considered inadequate for assessing PHC quality and therefore will not pass to the second round. Additionally, indicators that in round 1 reach consensus in both “Validity” and “Importance”, will be assumed appropriate to assess PHC quality, and for this reason will also not pass to round 2. Finally, indicators for which consensus regarding the uncertainty of both their “Validity” and “Importance” of indicators to assess PHC quality in round 1 will not pass to round 2 as well. Consensus in this case is more stringent (1 or none out of 10 participants rating outside the 4–6 range of values, see definition below). This is justified by the fact that we want an efficient process to identify indicators that are clearly good candidates to assess quality of PHC.
Round 2.
Indicators with no consensus in either dimension will pass to round 2, except when consensus in the other dimension is either “Not Valid” or “Not Important”. Indicators with consensus on importance but uncertain validity and vice-versa will also pass to round 2. (Table 1).
In round 2, the questionnaire will additionally include the results from round 1. To this end, we will use the eDelphi results export tool to plot aggregate responses for each indicator evaluated. The plots will be inserted in the corresponding question, along with anonymous quotes of all the comments received in round 1 (S1 File). One week will be needed to assemble the questionnaire for round 2. Provided that low response rates do not prove to be a problem, each second round will open three weeks after the first round.
The second round will be carried out in real-time. This means that the group response 2 will be updated in real-time and will be made visible to the participant immediately after giving their response. In practice, this allows experts to respond each question twice requiring a single interaction with the software (Fig 3) [20, 27].
A) before start answering/clicking on the grid for the first time; B) right after answering/clicking on the grid for the first time, the group’s response becomes visible, and the participant has the chance to review the initial response.
In round two, participants will also initially have access to the same information about the study; the only difference will be that the information will include additional explanations about the real-time approach Delphi. Participants will also have to provide the same demographic information as in round 1. This will allow us to invite supplementary experts in round 2 if we experience dropouts.
Participants will then be asked to rate their level of agreement with the clinical indicators that have been passed on to round 2 regarding their validity and importance. The rating should consider the distribution of participants’ ratings and comments from the previous round which will be visible in the description of each indicator (S1 File). To rate the indicators, participants must point and click in a nine-by-nine grid representing the 2-dimension space of possible rating combinations in round 2. Ratings will be rounded to the nearest integer number (and this will be informed to participates at the beginning of round 2). After clicking/responding, de-identified ratings of other participants will appear on the grid. (Fig 3), which may elicit a change in the response.
Data analysis plan
De-identified ratings will be summarized using eDelphi tools into quartiles 1, 2 and 3 and as the number of ratings in each indicator, which will then be reported to panel members as feedback after round 1 of each panel.
Consensus targets for this study are defined ‘a priori’ based on other studies [16, 25]. Consensus for each criterion will be determined based on the distribution of participants’ ratings and the level of disagreement among the participants according to three categories.
- Indicators with 70% or more ratings in the 7–9 range and less than 15% of ratings in the 1–3 range will be classified as “Important” or”Valid”.
- Indicators with 70% or more ratings in the 1–3 range and less than 15% of ratings in the 7–9 range will be classified as “Not important” or”Not valid”.
- Indicators with 85% or more ratings in the 4–6 range will be classified as “Uncertain”.
A minimum of 10 answers per indicator will be required to close a round unless consensus is reached with fewer answers and two weeks pass since starting the round. (Table 2).
Descriptive statistics (min, Q1, Q2, Q3, max) of the distribution of the number of responses, number of indicators that either have at least 10 answers or reached consensus, time a round is open, number of drop-outs, number of invitations sent will be monitored regularly to determine the progress of a round. This information may be used in the reminders to motivate experts to answer and close a round. Descriptive statistics of participants’ profiles will be determined to check heterogeneity of participants’ profiles within and between panels.
Any free text responses provided by participants to specific indicators during the two rounds will be analyzed thematically [28]. The thematic analysis of qualitative open responses will not be shared with participants, it will only be used to discuss the final results of panels for dissemination purposes of this work.
R Studio will be used to develop R scripts to implement repetitive and automatized analyses. Scripts will be distributed through the team for consistency of analysis across panels.
Steering committee/project governance
A Steering Committee at the Faculty of Medicine, University of Porto, was formed to develop and conduct this project. The Committee includes representatives of various disciplines (Medicine, Nursing, and Data analysis) and research expertise (qualitative, quantitative, and Delphi methods). The research methods were established in face-to-face meetings and email communications. Different members of the committee were responsible for: A) extracting and processing indicators available in the literature, B) reviewing indicators definitions and removing indicator duplicates, C) identifying (contacts of) potential participants, recruitment, and developing the questionnaires, and D) defining a framework and choosing tools to manage Delphi panels. Several pilot tests were carried out to establish the organization, structure, and amount of information provided in questionnaires.
Ethics and data management
This project has been approved by the ethics committee of the São João Hospital University Center/Faculty of Medicine, University of Porto [Ref. 338–19].
Personal data (names, email addresses, profession, working institution) will only be collected to approach potential participants, send out links to the questionnaire, and send out reminders. These data will be publicly available and as such will be stored unencrypted. However, all responses will be de-identified through the eDelphi and the team will not have access to the key to re-identify participants.
Discussion
This protocol describes the research design for a Delphi process to obtain consensus from experts on a core set of valid and important health indicators to assess PHC quality selected from a list of indicators abstracted from the scientific literature. This protocol will add to the current instruments aimed at incentivizing PHC quality. Specifically, the implementation of this protocol will arrive to a concise list of health quality indicators validated for European countries, and as such it will be essential for comparisons of PHC quality across countries [9]. Therefore, this research will contribute to overcoming the issue regarding the vast collection of indicators to assess PHC quality [10], identifying indicators retrieved from the scientific literature that may no longer fit the current clinical guidelines and populations’ needs in European countries.
The Delphi technique is well suited for our purpose since it is commonly used in health research, particularly to select health indicators [16]. Unlike methods that rely on face-to-face meetings to form a group consensus, such as the nominal group technique [21, 29] and the UCLA RAND method [25, 30], the Delphi is a soloistic, anonymous process avoiding direct confrontation of the experts. Thus, participation is less susceptible to dominant opinions, and controlled feedback allows for gradually forming a considered opinion [18, 31]. Nevertheless, face-to-face discussions could be important to allow the debate of potentially different viewpoints, but they are unlikely feasible in this study due to the large list of indicators to assess. It should be noted, however, that in Round 2, experts will interact as they will have access to the results of other participants in real time and can change their classification accordingly.
Moreover, we believe that the use of an online program specially developed to carry out Delphi studies, such as the eDelphi, will be key to manage a large amount of Delphi panels consistently over time, offering easy participation to experts, and achieving a reasonable response rate. Panel members will be drawn from a variety of backgrounds ensuring a multidisciplinary perspective and validity across European countries [25]. However, the lack of clear guidelines about implementing Delphi remains objectionable [31, 32], publishing this protocol should strengthen the research design of this study.
This study will disregard any feasibility and cost implications associated with the implementation of the selected indicators. This study will also disregard potential valid and/or important indicators that are not described in the literature. Additionally, limited time and resources available preclude at present to carry a final Delphi process to further select from the final consensual valid and important indicators that will be derived in this study. However, the present work will allow us to discern between the best and worst candidate indicators described in the literature within a specific clinical context. Future work will be needed to address these limitations.
This paper describes the design of a European Delphi study to derive a consensual set of quality indicators of PHC. The impact of this study will ultimately aid standardization of PHC and benefit the quality of care delivered to patients across European countries.
Supporting information
S3 File. Reasons for declining participation—Google forms.
https://doi.org/10.1371/journal.pone.0309395.s004
(PDF)
References
- 1. Espinosa-González AB, Delaney BC, Marti J, Darzi A. The impact of governance in primary health care delivery: a systems thinking approach with a European panel. Health research policy and systems 2019; 17:1–16. pmid:31272472
- 2. O’Malley AS, Rich EC, Maccarone A, DesRoches CM, Reid RJ. Disentangling the linkage of primary care features to patient outcomes: a review of current literature, data sources, and measurement needs. Journal of general internal medicine 2015; 30:576–85. pmid:26105671
- 3.
WHO. A vision for primary health care in the 21st century: towards universal health coverage and the Sustainable Development Goals. World Health Organization, 2018. https://iris.who.int/bitstream/handle/10665/328065/WHO-HIS-SDS-2018.15-eng.pdf
- 4.
OECD. Realising the potential of primary health care. OECD Publishing, 2020.
- 5. Mainz J. Defining and classifying clinical indicators for quality improvement. International journal for quality in health care 2003; 15:523–30.
- 6. Wolfe A. Institute of Medicine report: crossing the quality chasm: a new health care system for the 21st century. Policy, Politics, & Nursing Practice 2001; 2:233–5.
- 7.
Eckhardt H, Smith P, Quentin W. Pay for Quality: using financial incentives to improve quality of care. In: Busse R, Klazinga N, Panteli D, et al., editors. Health Policy Series, No. 53. Copenhagen (Denmark): European Observatory on Health Systems and Policies. Improving healthcare quality in Europe: Characteristics, effectiveness and implementation of different strategies; 2019. pp: 357–396.
- 8. Groene O, Skau JK, Frølich A. An international review of projects on hospital performance assessment. International journal for quality in health care 2008; 20:162–71.
- 9.
OECD. Policy Brief: Realising the potential of primary health care.. 2019. https://www.oecd.org/health/health-systems/OECD-Policy-Brief-Primary-Health-Care-May-2019.pdf
- 10. Ramalho A, Castro P, Goncalves-Pinho M, Teixeira J, Santos JV, Viana J, et al. Primary health care quality indicators: an umbrella review. PloS one 2019; 14:e0220888. pmid:31419235
- 11. Cantrill JA, Sibbald B, Buetow S. Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability. BMJ Quality & Safety 1998; 7:130–5. pmid:10185138
- 12. Caughey GE, Ellett LMK, Wong TY. Development of evidence-based Australian medication-related indicators of potentially preventable hospitalisations: a modified RAND appropriateness method. BMJ open 2014; 4:e004625. pmid:24776711
- 13. Morris C, Cantrill J, Hepler C, Noyce P. Preventing drug-related morbidity—determining valid indicators. International Journal for Quality in Health Care 2002; 14:183–98.
- 14. Campbell SM, Ludt S, Lieshout JV, Boffin N, Wensing M, Petek D, et al. Quality indicators for the prevention and management of cardiovascular disease in primary care in nine European countries. European Journal of Preventive Cardiology 2008; 15:509–15.
- 15.
WONCA. An Introduction to the International Classification of Primary Care—Version 2.http://www.ph3c.org/PH3C/docs/27/000098/0000054.pdf
- 16. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PloS one 2011; 6:e20476. pmid:21694759
- 17. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. Journal of advanced nursing 2000; 32:1008–15.
- 18.
Helmer O. Analysis of the future: The Delphi method. Rand Corp Santa Monica CA, 1967. https://mn.gov/web/prod/static/mnddc/live/parallels2/pdf/70s/72/72-MRA-RRT.pdf#page=11
- 19. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Management science 1963; 9:458–67.
- 20. Aengenheyster S, Cuhls K, Gerhold L, Heiskanen-Schüttler M, Huck J, Muszynska M. Real-Time Delphi in practice—A comparative analysis of existing software-based tools. Technological Forecasting and Social Change 2017; 118:15–27.
- 21. Chrusch CA, Martin CM. Quality improvement in critical care: selection and development of quality indicators. Canadian respiratory journal 2016; 2016. pmid:27493476
- 22.
Marshall M, Leatherman S, Mattke S. Selecting indicators for the quality of health promotion, prevention and primary care at the health systems level in OECD countries. 2004.
- 23. Campbell SM, Braspenning Ja, Hutchinson A, Marshall M. Research methods used in developing and applying quality indicators in primary care. Quality and Safety in Health Care 2002; 11:358–64.
- 24. Schwemlein S, Cronk R, Bartram J. Indicators for monitoring water, sanitation, and hygiene: a systematic review of indicator selection methods. International journal of environmental research and public health 2016; 13:333. pmid:26999180
- 25.
Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR. The RAND/UCLA appropriateness method user’s manual. Rand Corp Santa Monica CA, 2001. https://apps.dtic.mil/sti/tr/pdf/ADA393235.pdf
- 26. Tetzlaff JM, Moher D, Chan A-W. Developing a guideline for clinical trial protocol content: Delphi consensus survey. Trials 2012; 13:1–10. pmid:23006145
- 27. Gnatzy T, Warth J, von der Gracht H, Darkow I-L. Validating an innovative real-time Delphi approach-A methodological comparison between real-time and conventional Delphi studies. Technological Forecasting and Social Change 2011; 78:1681–94.
- 28. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative research in psychology 2006; 3:77–101.
- 29. Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. American journal of public health 1972; 62:337–42.
- 30. van der Ploeg E, Depla M, Shekelle P, Rigter H, Mackenbach JP. Developing quality indicators for general practice care for vulnerable elders; transfer from US to The Netherlands. BMJ Quality & Safety 2008; 17:291–5.
- 31. Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Information & management 2004; 42:15–29.
- 32. Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, et al. Internet versus mailed questionnaires: a controlled comparison (2). Journal of medical Internet research 2004; 6:e912. https://www.jmir.org/2004/3/e30/