https://orcid.org/0000-0002-0665-6348
Figures
Abstract
Background
High-quality general practice has been demonstrated to provide cost-effective, equitable health care and improve health outcomes. Yet there is currently not a set of agreed comprehensive indicators in Australia. We have developed 79 evidence-based indicators and their corresponding 129 measures of high-quality general practice. This study aims to achieve consensus on relevant and feasible indicators and measures for the Australian context.
Methods
This Delphi consensus study, approved by WSU Human Research Ethics Committee, consists of three rounds of online survey with general practice experts including general practitioners, practice nurses and primary health network staff. The identified indicators and measures are grouped under an attribute framework aligned with the Quadruple Aim, and further grouped under structures, processes and outcomes according to the Donabedian framework. Participants will rate each indicator and measure for relevance and feasibility, and provide comments and recommendations of additional indicators or measures. In the last round, participants will also be asked their views on the implementation of a quality indicator tool. Each indicator and measure will require ≥70% agreement in both relevance and feasibility to achieve consensus. Aggregated ratings will be statistically analysed for response rates, level of agreement, medians, interquartile ranges and group rankings. Qualitative responses will be analysed thematically using a mixed inductive and deductive approach.
Discussion
This protocol will add to the current knowledge of the translation of performance guidelines into quality practice across complex clinical settings and in a variety of different contexts in Australian general practice. The Delphi technique is appropriate to develop consensus between the diverse experts because of its ability to offer anonymity to other participants and minimise bias. Findings will contribute to the design of an assessment tool of high-quality general practice that would enable future primary health care reforms in Australia.
Citation: Lau P, Ryan S, Abbott P, Tannous K, Trankle S, Peters K, et al. (2022) Protocol for a Delphi consensus study to select indicators of high-quality general practice to achieve Quality Equity and Systems Transformation in Primary Health Care (QUEST-PHC) in Australia. PLoS ONE 17(5): e0268096. https://doi.org/10.1371/journal.pone.0268096
Editor: Marie-Pascale Pomey, University of Montreal, CANADA
Received: October 28, 2021; Accepted: April 21, 2022; Published: May 24, 2022
Copyright: © 2022 Lau et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.
Funding: This study is funded by the Digital Health Cooperative Research Centre https://www.digitalhealthcrc.com/. The funding body is part of the Project Control Group which oversees the conduct of the study, including the design of the study and collection, analysis, and interpretation of data and the writing of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The collection, robust analysis and appropriate use of general practice data are critical to informing continuous quality improvement and ensure high-quality primary health care (PHC). Many countries around the world collect PHC data for quality improvement purposes such as monitoring health and utilisation of health services like the Nivel’s Primary Care Database in The Netherlands, [1] and for research purpose such as the Clinical Practice Research Datalink (CPRD) in the UK [2].
Australian general practice care data is routinely collected by primary health networks (PHNs) for quality improvement purposes and more recently through the Practice Incentives Program Quality Improvement (PIP QI) initiative, launched in August 2019 by the Australian Government to improve patient outcomes [3–6]. Practices registered for PIP QI are required to regularly submit data collected against ten specified improvement measures to their local PHNs and commit to implementing continuous quality improvement activities in partnership with the PHNs [4]. The general practice data, known as the PIP Eligible Data Set, collected through this process is then uploaded to a national portal and the Australian Institute of Health and Welfare (AIHW) oversees access by researchers and other interested parties to the deidentified PIP Eligible Data Set [4]. The 31 PHNs were established in 2015 across Australia for supporting PHC to identify and meet local health needs, building PHC workforce capacity and the delivery of high-quality care, and integrating local health services to improve patient experience and better use of health resources [5]. They also collect and collate other general practice data from extraction of de-identified data from individual practice-based patient records [5]. However, there is a lack of consistency across PHNs in data content, variability in the quality of the data collected and also in the quality improvement outcomes achieved through this process [7].
A general practice indicator is “a measurable element of practice performance for which there is evidence or consensus that it can be used to assess the quality, and hence change in the quality, of care provided” [8]. A set of standardized and evidence-based indicators to measure and track high quality clinical performance and outcomes in general practice is necessary, not only for the profession’s accountability, but also for identifying population needs and gaps in the quality of care received by patients across Australia [7, 9]. Whilst the Royal Australian College of General Practitioners (RACGP) Standards for General Practices underpin accreditation, they are minimum standards for benchmarking purposes [4, 6]. Previous work to develop PHC quality indicators have focused on specific areas of interest, for instance, the Primary Care Practice Improvement Tool (PC-PIT) which is an organisational performance improvement tool focusing only on systems and processes [10]. Even though the collection of data is key to the evaluation of health care quality and services and clinical decision-making, there is currently not a set of universally agreed comprehensive high-quality indicators in Australia that would identify, measure and reward high-quality general practice.
In 2020, Western Sydney University, in partnership with PHNs in the western Sydney region, conducted a literature review to identify evidence-based indicators and measures, then assessed these in three workshops with general practitioners (GPs), practice managers, nurses, consumers and PHN staff in the western Sydney region [11]. A suite of 79 evidence-based indicators and their corresponding 129 measures of high-quality general practice was subsequently developed. The measures specifically included outcome measures as these are rarely addressed in frameworks of quality PHC [12, 13]. Key literature was also analysed to identify four attributes of high-quality general practice and construct a suitable framework for the indicators and measures [11]. The attributes are expressed as ‘accountabilities’: accountability to our patients; professionally accountable; accountability to the community and accountability to society [14, 15]. They align with the elements of the Quadruple Aim which states that effective healthcare improvement must take into account the care of individual patients, the health of populations, health care costs and the wellbeing of health care providers, [16] and is increasingly used to monitor and evaluate primary health system performance in Australia and countries like the UK and US [13, 17, 18]. The indicators and measures identified are further grouped under structures, processes and outcomes of high-quality general practice according to a Donabedian framework, [19, 20] and include some “blue sky” measures considered difficult to currently implement but are nonetheless important.
This study extends the previous work by Western Sydney University [11]. Wider consultations with Australian stakeholders will be conducted using a Delphi consensus study with experts to explore the relevance and feasibility of the identified suite of indicators and measures. Experts will include Australian general practices and PHNs involved in quality improvement initiatives. Consultations have been held with consumers, with regards to key patient-reported measures (PRMs). Aboriginal and Torres Strait Islander health and justice health sectors will also be consulted with regard to relevant indictors unique for those populations. These will be detailed elsewhere.
Materials and methods
Aim
The overall aim is to establish consensus with experts to contribute to the development of the first comprehensive, evidence-based, professionally endorsed tool for analysing and reporting across all components of high-quality general practice in Australia.
Study design
This study will use a survey to achieve consensus across an expert group of general practice and PHN staff. The Delphi technique has been selected due to its flexibility and anonymity provided to participants [21, 22]. The survey will consist of three rounds to obtain opinions on a suite of indicators and measures previously developed by the research team to reach consensus on a core set of relevant and feasible high-quality performance indicators [11].
Project governance
A Project Control Group has been established with the responsibility for overseeing the conduct of the project. The group consists of representatives from the Digital Health Cooperative Research Centre (CRC), and eight primary health organisations: Brisbane North PHN, Central and Eastern Sydney PHN, Nepean Blue Mountains PHN, North Western Melbourne PHN, South Western Sydney PHN, Western Sydney and then (WentWest), Western Australia Primary Health Alliance, and Western NSW PHN.
A Steering Committee, that meets more frequently, has also been established to provide strategic direction and advice to the research team on dissemination and collaboration with relevant stakeholder groups. This committee consists of the representatives of the primary health organisations and the RACGP, Australian College of Rural and Remote Medicine (ACRRM), Justice Health NSW (New South Wales) and SA (South Australia) Prison Health Service.
Setting
The study will be undertaken in four states in Australia across regions of the eight primary health organisations: seven PHNs and one primary health alliance comprising three PHNs which support primary care across a less populous state of Australia. These organisations cover a total area of 2,942,817km2 in metropolitan and rural Australia, and a diverse population of over 9.6 million with over 3,000 general practices. The characteristics of the PHNs, their geographical locations and the populations in their regions are summarised in Table 1.
Sample size
The study will aim to recruit a minimum of 80 participants. A minimum of 17 participants is the recommended minimum sample size for content validity in Delphi studies involving the selection of healthcare quality indicators [23]. In order for this Delphi study to meet the minimum sample size requirement, we must achieve a minimum of 47% retention rate in rounds 2 and 3.
Participants and recruitment
Participants will include GPs, practice nurses, practice managers and key PHN staff who are familiar with quality improvement initiatives in the context of Australian general practice. People under 18 years old will be excluded.
A purposive and convenience sampling approach will be used. Each of the eight primary health organisations will assist in recruiting eight to ten general practices in their region and nominate two to three key PHN staff. Practices will be purposively recruited to maximize diversity in regard to geographic location, practice size, and socio-economic status based on the Socio-Economic Index for Areas (SEIFA). An Invitation Pack containing an invitation letter, project information and consent form will be emailed by the PHN to their nominated staff and recruited practices. Each practice will nominate one to two practice staff to participate in the survey. All survey participants will be anonymised to their PHNs and other participants with allocation of a random identification number. A password protected file will be maintained by the research team with participants’ identifying information.
Criteria for the Delphi participants to consider
A total of 79 indicators with 129 measures that had been developed and finalised by the QUEST PHC team in 2020 [11] will be assessed by participants in the Delphi study. (Table 2) They are grouped under the four attributes of high-quality general practice framework aligned with the four elements of the Quadruple Aim [14–16]. Table 3 outlines the four high-quality general practice attributes, their definitions and alignment with the Quadruple Aim and the number of indicators and measures identified under each attribute.
Survey format
Three rounds of online surveys will be administered using the Qualtrics platform. (Qualtrics, Provo, UT, USA. https://www.qualtrics.com). The online survey has been constructed and pilot-tested for comprehension and adequate functioning of the survey set up. Unique links to each round will be emailed to participants on the morning that it is officially opened. Each round will take around 20 to 30 minutes to complete and will remain opened for three weeks. Results will be analysed at least two weeks in between rounds. Participants will receive up to three email reminders to complete each round before it closes.
Rating process
Participants will be asked to rate each indicator and measure for relevance and feasibility in three rounds of the online survey. Relevance is defined as the value and appropriateness of an indicator/measure in Australian general practice. Participants will be asked to rate on a 4-point Likert scale: 1 irrelevant; 2 somewhat irrelevant; 3 somewhat relevant; 4 relevant. Feasibility is defined as the applicability and implementability of an indicator/measure in Australian general practice. Participants will be asked to rate each indicator/measure on a 4-point Likert scale– 1 infeasible; 2 somewhat infeasible; 3 somewhat feasible; 4 feasible. Text boxes will be available for participants to provide comments, including recommendations for additional indicators or measures, for each subgroup of indicators.
The flow of the Delphi study rating process is shown in Fig 1. In Round 1, participants will initially be asked to provide demographic information including their name, age, gender, job position, and number of years of experience. They will then be asked to rate the indicators and measures under Attribute 1. In subsequent rounds, only names will be requested to match participants’ responses in the various rounds. In Round 2, they will be presented with items that did not reach consensus in Round 1, and given the opportunity to change their previous responses if they wish to do so. They will then be asked to rate the indicators and measures under Attributes 2, 3 and 4 and to provide comments as before for each subgroup of indicators. In Round 3, they will similarly be presented with items that did not reach consensus in Round 2, and given the opportunity to change their previous responses if they wish to do so. In this last round, as the final list of indicators and measures emerges, participants will be presented with a summary of any suggestions or qualitative responses from rounds 1 and 2, and asked open questions regarding their views and suggestions on the implementation of a quality indicator tool in Australian general practice.
The levels of consensus in the Delphi methodology vary depending on size of the expert panel and the aim of the research [24, 25]. Consensus target for this study are defined ‘a priori’ based on previous research experience [26]. Each indicator (average score of its measures) and measure will require a minimum of 70% agreement (combined scores of 3 and 4) in both relevance and feasibility to achieve consensus. We determined that this threshold target and approach to be pragmatic and reasonable for establishing consensus across diverse and complex general practice settings.
Data analysis
Quantitative data.
Participants’ demographics will be analysed descriptively using Microsoft Excel software. The aggregate results of the participants’ responses will be analysed for percentage response rates, percentages for each level of agreement for each measure, medians, interquartile ranges and their associated group rankings [27].
A measure will require at least 70% in both relevance and feasibility to achieve consensus. Score of 1 and 2 will be collapsed as irrelevant or infeasible, and scores of 3 and 4 will be collapsed as relevant or feasible. If an indicator or measure achieves ≥70% in relevance but not feasibility, it will be included in a ‘blue skies’ category for future consideration. If an indicator or measure achieves ≥70% in feasibility but not in relevance, it will be discarded. Sub-analysis of the individual scores 1, 2, 3 and 4 will also be conducted to help us understand better the strength of the consensus.
Qualitative data.
Participants’ responses in the text boxes will be analysed thematically. They will be imported into the NVivo analysis software and coded using a mix of inductive and deductive approaches [28, 29]. Patterns will then be identified from the codes and grouped according to the accountability attributes (deductive approach) as well as to elicit new themes (inductive approach). The research team will separately and collectively analyse the data and resolve any differences in interpretation.
Data management plans
The types of data that will be produced include demographic data collected on participant consent forms in MS Word/PDF format and electronic survey data. A MS Excel spreadsheet will be created in which participant names will be assigned a number. Participant numbers will be used in place of participant names in naming participant data files for the duration of the project. Survey files will be named using the participant’s number, the survey number and the date e.g. Participant1_survery1_190521.
Digital data will be stored on the Western Sydney University’s OneDrive system. PL is the administrator and the only person able to provide access to other team members. The only team members with access are PL, SR and JR.
Non-digital data, if any, will be scanned and stored with the digital data. The original hardcopy documents will be stored in a locked filing cabinet in a locked office at Western Sydney University Campbelltown Campus.
All research data and primary materials will be stored for 15 years and then destroyed in accordance to Western Sydney University protocols.
Potential risks and risk management
Potential risks related to this project include those that may be internal or external. Survey participants may feel inconvenienced by the process required in the study. This includes being required to read the project information, sign the consent form and complete three rounds of survey. To manage these risks the project aims and purpose will be clearly explained to the participants who are experts familiar with Australian general practice quality improvement initiatives. The study will be of inherent interest to them. It will also be styled to allow easy completion and participants will be able to save their responses and return to them later.
Some participants may be concerned about confidentiality. Although participants will be asked to provide demographic information, their identity and information will be blinded to other survey participants and the PHNs. As detailed above, they will be provided anonymity with a random participant project number that will only be able to be linked to their identifiable information by the research team.
External to the project, risks include the current COVID-19 pandemic and government restrictions on movements. These restrictions and the workload of vaccine roll-out may potentially affect recruitment and participation as PHNs and general practices are directly involved in pandemic prevention and control. The recruitment and survey timeline will be flexible to accommodate any unforeseen interruptions. Each round may also be opened for a longer period if necessary.
Ethical considerations
This research has ethics approval from Western Sydney University Human Research Ethics Committee (ID H14460). Participants will be required to provide written consent before round 1 of the survey.
Status and timeline
At the time of manuscript submission, the research has just commenced recruitment of participants. Tentative timeline is outlined in Table 4.
Discussion
This study protocol describes the research design for a Delphi study to obtain opinions and reach consensus from experts on a core set of relevant and feasible high-quality performance indicators and measures from a suite of indicators and measures previously developed by the researchers in partnership with PHNs in Western Sydney [11]. This protocol will add to the current knowledge of the translation of performance guidelines into quality practice and how best to measure and promote high quality in Australian general practice.
Whilst many PHNs work with general practices to collect data for quality assurance purposes, there is no agreed comprehensive tool that could identify, measure and reward high-quality general practice. Some work has been done in PHNs supporting Patient-Centred Medical Home model of care, but the indicators that were used revolved around processes and system requirement for a team-based approach to deliver this model of care [30, 31]. Although very useful, these indicators are specific to the PCMH models and are dependent on the continuation of funding and evolving policies to support this model of care. Australian general practice requires practical and evidence-based indicators and measures of high quality if funding models move to incorporate payment for quality in addition to current throughput payment [32]. Findings from this Delphi consensus study will address the gaps in the literature around establishing consensus on high-quality structural, process and outcome indicators and measures for use across diverse and complex general practice settings, and contribute to the design of an assessment tool that would change how high-quality general practice can be measured and enable future PHC reforms in Australia.
The suite of 79 indicators and their corresponding 129 measures to be evaluated in this Delphi consensus study were derived from robust interrogation of existing literature and extensive consultations with key stakeholders [11]. They are focused on structures, processes and outcomes of care. This Delphi study will enable consideration of their relevance and feasibility within different general practice clinical settings where multi-morbidities and complex interventions are common and the constraints of providing health services are unique. Opinions from our participants will inform and guide the implementation of the developed tool in the real world.
There is growing interest in the processes required to establish assessment tools to identify high-quality health care and service performance. The Delphi technique is appropriate to develop consensus between the diverse stakeholders and experts in the Australian general practice setting because of its flexibility and ability to offer anonymity to participants. It has the benefit of being able to minimise bias from dominant experts compared to other consensus development methods. It provides a platform to canvass suggestions and opinions on implementation of the tool to measure improvement in individual practices and considerations required for specific contexts including cultural and socio-economic factors that may impact achievement of quality indicators. Additionally, the provision of opportunities for participants to review results from previous rounds and to revise their responses is a unique characteristic of the Delphi technique to enable the determination of consensus. A disadvantage of the Delphi technique, however, is that it does not involve direct interactions with the participants and may limit their ability to generate ideas during the consensus process [33]. Another limitation of this study is that it is designed specifically for the Australian context and may not represent the setting and conditions of other countries.
Using four high-quality general practice attributes that reflect the Quadruple Aim as a framework in this Delphi consensus study will help us to focus on the design of an assessment tool that will facilitate high-quality general practice delivery. The application of scoring criteria for approval for each consensus statement is also expected to ensure the relevance and feasibility of the final core set of indicators and measures.
Another strength of the study is the broad representation of Australian primary health organisations and diverse backgrounds of the participants involved. However, the diverse medical and non-medical participant populations with different perspectives and priorities may confound the results. If that is the case, we will be able to differentiate the stakeholder groups and analyse accordingly to identify and understand the different perspectives.
Although we have involved only PHN and general practice experts in this consensus development process, we plan to engage with primary health care consumers and Aboriginal and Torres Strait Islander health and justice health sectors separately in focus groups to explore their views on indicators and measures applicable to the final quality improvement tool. Through this Delphi consensus study, QUEST PHC will provide valuable information to guide future research and quality improvement activities in these diverse settings.
Acknowledgments
The authors would like to acknowledge the Project Control Group: Digital Health CRC, Brisbane North PHN, Central and Eastern Sydney PHN, Nepean Blue Mountains PHN, North Western Melbourne PHN, South Western Sydney PHN, WentWest, Western Australia Primary Health Alliance, and Western NSW PHN. The authors would also like to acknowledge the RACGP, ACRRM, Justice Health NSW and SA Prison Health Service for their contribution to the Steering Committee.
References
- 1. Smeets HM, Kortekaas MF, Rutten FH, Bots ML, van der Kraan W, Daggelders G, et al. Routine primary care data for scientific research, quality of care programs and educational purposes: the Julius General Practitioners’ Network (JGPN). BMC Health Services Research. 2018;18(1):735. pmid:30253760
- 2. Ghosh RE, Crellin E, Beatty S, Donegan K, Myles P, Williams R. How Clinical Practice Research Datalink data are used to support pharmacovigilance. Ther Adv Drug Saf. 2019;10:2042098619854010. pmid:31210923
- 3.
Australian Government Department of Health. PIP QI incentive guidance Canberra, ACT: Commonwealth Government; 2021 [updated 28 April 2021]. Available from: https://www1.health.gov.au/internet/main/publishing.nsf/Content/PIP-QI_Incentive_guidance.
- 4.
Royal Australian College of General Practitioners. Practice incentives program quality improvement incentive (PIP QI) fact sheet East Melbourne, VIC: RACGP; 2021. Available from: https://www.racgp.org.au/running-a-practice/security/managing-practice-information/secondary-use-of-general-practice-data/pip-qi-factsheet.
- 5. Australian Government Department of Health. Fact Sheet: PHN Practice Support: Commonwealth of Australia; 2018 [updated 18 June 2018]. Available from: https://www1.health.gov.au/internet/main/publishing.nsf/Content/Fact+Sheet-PHN-Practice-Support+.
- 6.
Royal Australian College of General Practitioners. Standards for general practices. 5th edn. East Melbourne, VIC: RACGP; 2020.
- 7. Ghosh A, Halcomb E, McCarthy S, Ashley C. Structured yet simple approaches to primary care data quality improvements can indeed strike gold. Australian Journal of Primary Health. 2021;27(2):143–51. pmid:33689677
- 8.
Marshall M, Roland M, Campbell S, Kirk S, Reeves D, Brook R, et al. Measuring general practice: a demonstration project to develop and test a set of primary care clinical quality indicators. London, UK: The Nuffield Trust; 2003. 68 p.
- 9. Exworthy M, Wilkinson EK, McColl A, Moore M, Roderick P, Smith H, et al. The role of Performance Indicators in Changing the Autonomy of the General Practice Profession in the UK. Social science & medicine. 2003;56(7):1493–504. pmid:12614700.
- 10. Crossland L, Upham SJ, Janamian T, Siskind V, Sheehan M, Jackson CL. Trial of the Primary Care Practice Improvement Tool: building organisational performance in Australian general practice and primary health care. Med J Aust 2016;204(7):S15. pmid:27078787
- 11. Metusela C, Cochrane N, van Werven H, Usherwood T, Ferdousi S, Messom R, et al. (2021). Developing indicators and measures of high-quality for Australian general practice. Australian Journal of Primary Health. (In press).
- 12.
Australian Government Department of Health. Evaluation of the GP Super Clinic Program 2007–2008: 7.3.7 Quality in Primary Health Care Canberra, ACT: Commonwealth of Australia; 2012. Available from: https://www1.health.gov.au/internet/publications/publishing.nsf/Content/GPSuperClinicsEvaluation-toc~discussion~progresstowardsachieving~qualityinprimaryhealthcare.
- 13. Australian Medical Association. AMA Position Statement: Measuring Clinical Outcomes in General Practice– 2016: AMA; 2016. Available from: https://www.ama.com.au/position-statement/measuring-clinical-outcomes-general-practice-2016.
- 14. Sikka R, Morath JM, Leape L. The Quadruple Aim: care, health, cost and meaning in work. BMJ Quality & Safety. 2015;24(10):608. pmid:26038586
- 15. Rathert C, Williams ES, Linhart H. Evidence for the Quadruple Aim. Medical Care. 2018;56(12):976–84. pmid:30339573
- 16. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–6. pmid:25384822
- 17. Young RA, Roberts RG, Holden RJ. The challenges of measuring, improving, and reporting quality in primary care. Ann Fam Med. 2017;15(2):175–82. pmid:28289120
- 18. Mutter JB, Liaw W, Moore MA, Etz RS, Howe A, Bazemore A. Core Principles to Improve Primary Care Quality Management. The Journal of the American Board of Family Medicine. 2018;31(6):931. pmid:30413549
- 19. Donabedian A. The Quality of Care: How Can It Be Assessed? JAMA. 1988;260(12):1743–8. pmid:3045356
- 20. Berwick D, Fox DM. "Evaluating the Quality of Medical Care": Donabedian’s Classic Article 50 Years Later. Milbank Q. 2016;94(2):237–41. pmid:27265554
- 21. Hsu C-C, Sandford BA. The Delphi Technique: Making Sense of Consensus. Practical Assessment, Research, and Evaluation. 2007;12(Article 10).
- 22. von der Gracht HA. Consensus measurement in Delphi studies: Review and implications for future quality assurance. Technological Forecasting and Social Change. 2012;79(8):1525–36.
- 23. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. pmid:21694759
- 24. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9. pmid:24581294
- 25. English B, Tickle A, das Nair R, Moore K. Consensus‐based good practice guidelines for clinical psychologists to support care staff in enabling sexual expression in people with intellectual disabilities—A Delphi study. Journal of Applied Research in Intellectual Disabilities. 2019;33. pmid:31578807
- 26. Ekawati FM, Licqurish S, Gunn J, Brennecke S, Lau P. Hypertensive disorders of pregnancy (HDP) management pathways: results of a Delphi survey to contextualise international recommendations for Indonesian primary care settings. BMC Pregnancy and Childbirth. 2021;21(1):269. pmid:33794799
- 27. Holey EA, Feeley JL, Dixon J, Whittaker VJ. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Medical Research Methodology. 2007;7(1):52. pmid:18045508
- 28. Clarke V, Braun V. Thematic analysis. The Journal of Positive Psychology. 2017;12(3):297–8.
- 29. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology. 2013;13(1):117. pmid:24047204
- 30. Borg S, Crossland L, Risk J, Porritt J, Jackson C. The Primary Care Practice Improvement Tool (PC-PIT) process for organisational improvement in primary care: Application by Australian Primary Health Networks. Australian Journal of Primary Health. 2019;25. pmid:30819323
- 31. Borg SJ, Donald M, Totsidis K, Quinn N, Jackson CL. Improving quality in general practice using the Primary Care Practice Improvement Tool (PC-PIT) with Primary Health Network support. Australian Journal of Primary Health. 2020;26(6):484–91. pmid:33296624
- 32. Oliver-Baxter J, Brown L, Dawda P. Should the healthcare compass in Australia point towards value-based primary healthcare? Aust Health Rev. 2017;41(1):98–103. pmid:27209074
- 33. Keeney S, McKenna H, Hasson F. The Delphi Technique in Nursing and Health Research: Wiley-Blackwell; 2010.
- 34.
Western Sydney Primary Health Network. Annual Review 2019/20: Changing Lanscapes. Sydney, Australia: WentWest; 2020.
- 35.
Nepean Blue Mountains Primary Health Network. Annual Report 2020.: NPMPHN. Nepean Blue Mountains, Australia: NBMPHN; 2020.
- 36.
South Western Sydney Primary Health Network. Annual Report 2019/20. Sydney, Australia: SWSPHN; 2020.
- 37.
Network. CaESPH. Annual Report 2019/20. Sydney, Australia: CESPHN; 2020.
- 38.
Western NSW Primary Health Networks. Health Needs Assessment: Shaping how we serve the community. New South Wales, Australia: WNSWPHN; 2020.
- 39.
North Western Melbourne Primary Health Network. Annual Report 2019/20. Melbourne, Australia: NWMPHN; 2020.
- 40.
Brisbane North Primary Health Network, Metro North Hospital and Health Service. Population Health Report 2019. Brisbane, Australia: BNPHN; 2019.
- 41.
Perth North Primary Health Network. Primary Health Network Program Needs Assessment Reporting 2019–2022. Perth, Australia: PNPHN; 2019.
- 42.
Curtin University Health System & Health Economics School of Public Health. Population Health Needs Assessment Perth North PHN. Perth, Australia: WA Primary Health Alliance, Perth North Primary Health Network; 2016.
- 43.
Perth South Primary Health Network. Primary Health Network Program Needs Assessment Reporting 2019–2022. Perth, Australia: PSPHN; 2019.
- 44.
Curtin University Health System & Health Economics School of Public Health. Population Health Needs Assessment Perth South PHN. WA Primary Health Alliance, Perth South PHN; 2016.
- 45.
Curtin University Health System & Health Economics School of Public Health. Population Health Needs Assessment Country WA PHN. Western Australia: WA Primary Health Alliance, Country WA PHN; 2016.