https://orcid.org/0000-0002-0399-3584
Figures
Abstract
Neonatal sepsis is a serious public health problem; however, there is substantial heterogeneity in the outcomes measured and reported in research evaluating the effectiveness of the treatments. Therefore, we aim to develop a Core Outcome Set (COS) for studies evaluating the effectiveness of treatments for neonatal sepsis. Since a systematic review of key outcomes from randomised trials of therapeutic interventions in neonatal sepsis was published recently, we will complement this with a qualitative systematic review of the key outcomes of neonatal sepsis identified by parents, other family members, parent representatives, healthcare providers, policymakers, and researchers. We will interpret the outcomes of both studies using a previously established framework. Stakeholders across three different groups i.e., (1) researchers, (2) healthcare providers, and (3) patients’ parents/family members and parent representatives will rate the importance of the outcomes in an online Real-Time Delphi Survey. Afterwards, consensus meetings will be held to agree on the final COS through online discussions with key stakeholders. This COS is expected to minimize outcome heterogeneity in measurements and publications, improve comparability and synthesis, and decrease research waste.
Citation: Taneri PE, Kirkham JJ, Molloy EJ, Biesty L, Polin RA, Wynn JL, et al. (2023) Protocol for the development of a core outcome set for neonatal sepsis (NESCOS). PLoS ONE 18(12): e0295325. https://doi.org/10.1371/journal.pone.0295325
Editor: Andrew Harding, Lancaster University, UNITED KINGDOM
Received: April 14, 2023; Accepted: November 17, 2023; Published: December 5, 2023
Copyright: © 2023 Taneri et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: No datasets were generated or analysed during the current study. All relevant data from this study will be made available upon study completion.
Funding: This research was funded by the Health Research Board (HRB, Ireland) through funding to the HRB Irish Network for Children’s Clinical Trials (In4Kids) [Grant number: CTN-2021-007] Funders did not play any role in the study design, data collection (when applicable) and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Sepsis is a major global public health problem. Children account for more than half of all sepsis cases, most of whom are neonates [1]. The global incidence of neonatal sepsis has been estimated at 2824 (95% CI 1892 to 4194) cases per 100,000 live births, with a mortality rate of 17.6% (95% CI 10.3% to 28.6%) and a higher burden in low-middle income countries [2]. In addition to the increased risk of early death, neonatal sepsis may cause long-term neurodevelopmental delay [3] and neurodevelopmental impairments such as cerebral palsy and neurosensory deficits [4]. Neonatal sepsis is also associated with impaired physical growth, increased risk of bronchopulmonary dysplasia and multiple organ failure [3].
There is significant variation in definitions of neonatal sepsis [5], clinical signs of neonatal sepsis [6], and antibiotic use [7]. The lack of a universally accepted definition for neonatal sepsis inhibits efforts to improve accurate diagnostic and prognostic testing methods for this vulnerable population [8]. In addition, there is substantial heterogeneity in the outcomes measured and reported in studies evaluating the effectiveness of treatments for neonatal sepsis, with Henry et al. identifying 88 outcomes across 90 studies [9]. This heterogeneity in outcomes limits the ability to compare, contrast, and synthesise [10] the findings of individual studies and contributes to research waste.
Furthermore, the challenges experienced by those who use research to inform decisions about their health care are exacerbated further by outcome-reporting bias [11]. Outcomes in which the intervention has a statistically significant effect are more likely to be fully reported [12]. Developing a Core Outcome Set (COS) for treatments for neonatal sepsis could help minimise these current shortcomings in reporting challenges.
A COS is ‘the minimum that should be measured and reported in all clinical trials of a specific condition and could also be suitable for use in other types of research and clinical audit’ [13]. The COS should always be measured, collected, and reported. Researchers may measure and report outcomes additional to the COS, but the COS should be reported in its entirety [11]. The advantages of the widespread use of a COS includes increased outcome consistency across trials, a likely decrease in selective reporting, and increased opportunity for a study to contribute to syntheses across outcomes included in the COS [14]. Importantly, using a COS helps ensure that the outcomes measured and reported are those that matter to stakeholders, including patients and/or their carers [15].
This study aims to develop a COS for studies evaluating the effectiveness of treatments for neonatal sepsis.
Materials and methods
Overview
The University of Galway, Research Ethics Committee provided ethical approval for this project (Reference Number: 2022.10.002). During the Delphi survey and consensus meeting phases, participants were provided with detailed project information via participant information leaflets. They were then cordially invited to partake in the study, contingent upon their agreement as evidenced by completing the standardized consent forms. These forms, which included a thorough briefing, were utilized to obtain written consent from the participants. Integral to our adherence to ethical standards, these documents were incorporated into our submission to the ethical committee and were subsequently granted approval.
This protocol has been prepared according to COS-STAP Statement recommendations [16]. We are not aware of any published COS evaluating the effectiveness of interventions for treating neonatal sepsis. The protocol of this study has been registered with the Core Outcome Measures in Effectiveness Trials (COMET) initiative (https://www.comet-initiative.org/Studies/Details/2118).
This study will be conducted in four stages (Fig 1):
- Qualitative systematic review
- Delphi survey
- Consensus meetings
- Dissemination and implementation
Stage 1: Qualitative systematic review
Recently, Henry et al. published a systematic review of core outcomes from randomised trials of therapeutic interventions in neonatal sepsis. They reported 88 unique outcomes from 90 included studies [9].
The outcomes chosen and reported by researchers or clinicians alone may not be important and/or relevant for other stakeholders, such as patients, caregivers, or other decision-makers [17]. The outcomes considered during the consensus process should represent the views of all relevant stakeholders. Therefore, it has been advised that COS developers consider different approaches to establishing an initial list of outcomes for prioritisation, such as collecting data from patient interviews or analysing qualitative research studies focusing on patients’ opinions [18]. Based on this recommendation, several qualitative systematic reviews have been published to support the COS development processes in, for example, female chronic pelvic pain [17], type 2 diabetes [18], and neonatal care [19].
Our proposed qualitative systematic review was registered with The International Prospective Register of Systematic Reviews (PROSPERO ID: CRD42022344485). The review will be conducted and reported following the guidelines of the ENTREQ statement and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [19, 20].
Review/synthesis question: What are the key outcomes of neonatal sepsis identified by parents, other family members, parent representatives, healthcare providers, policymakers, and researchers that should be included in a ’long list’ of outcomes in a Delphi Survey as part of the development of a COS on treatments for neonatal sepsis?
Inclusion criteria
The PerSPEcTiF framework was used to structure the description of inclusion criteria [21] as per qualitative review guidelines [22, 23] (Table 1).
We will include studies with a qualitative study design such as ethnographic, grounded theory, historical, and case study; those that used qualitative data-collection methods, including observations, textual or visual analysis, and interviews (individual or group). We will also include mixed-method studies with a qualitative component where participant experiences are reported separately. We will exclude abstracts, randomised trials, clinical trials, intervention studies, cross-sectional studies, case-control studies, prospective and retrospective cohort studies, letters to the editor, conference proceedings, and systematic reviews with or without meta-analyses.
Search strategy.
With no date restrictions, a comprehensive literature search will be carried out using MEDLINE, EMBASE, CINAHL, and PsycINFO databases. The complete search strategy is presented in S1 File.
The reference lists of included studies will be hand-searched to identify other potentially relevant studies [24–27]. Findings of the literature search will be transferred to Endnote, and after deduplication, abstract and title screening will be done via Rayyan [28].
Study selection.
The titles and abstracts will be screened by two reviewers independently using the eligibility criteria, and discrepancies will be resolved through discussion with a third review author where necessary. For all studies that appear to fulfil the eligibility criteria based on the abstract, two authors will examine full-text articles independently. Discrepancies will be resolved again through discussion with a third review author where necessary.
Quality assessment.
Two reviewers will independently assess the quality of the included studies using the Critical Appraisal Skills Programme (CASP) tool [29]. While studies will not be excluded based on this assessment of methodological limitations, the assessment will be reported in the final write-up of our systematic review. The results will be reported in the manuscript to inform readers about the methodological quality of the included studies.
Data extraction.
One review author will extract the data using a pre-piloted data extraction form to obtain study characteristics from qualifying studies. One additional review author will double‐check the data extraction performed by the first review author and verify that all relevant data are extracted. Author information, publication year, study design, number of participants, data collection and analysis methods, and text excerpts relevant to outcomes will be noted on the form.
Data analysis.
Drawing on the principles of thematic synthesis [30], extracted data related to outcomes will be coded line by line, and similarities will be identified. Codes will be grouped to define distinct, descriptive themes related to outcomes of therapeutic interventions in neonatal sepsis. Two reviewers will review and interpret the outcomes noted in the themes and will map these to a framework of outcome domains presented by Webbe et. al’s “Core outcomes in neonatology: development of a core outcome set for neonatal research.” [31]. The following domain headings will be used to guide this mapping: survival, respiratory, cardiovascular, gastrointestinal, neurological, genitourinary, skin, surgical, development (gross motor/fine motor/cognitive/special senses/speech and social), psychosocial, healthcare utilisation, outcomes related to parents, outcomes related to healthcare workers, general outcomes and miscellaneous. If the identified outcomes do not map to any domains, the reviewers will develop additional relevant domains.
The outcomes determined at the end of this phase will be brought forward to Stage 2.
Stage 2: Delphi survey
Delphi studies were used first in the 1960s by Dalkey and Helmer of the Rand Corporation to obtain consensus on the views of a group of experts [32]. Traditionally, the Delphi procedure helps develops consensus among experts by conducting controlled feedback on the findings of multiple survey rounds (usually) or interviews [33]. It can foster collective ownership and promote unity among people with different points of view [34]. The traditional Delphi technique is used extensively in health care to achieve consensus on various topics such as mental health [35–37], cancer [38, 39], and primary health care [40, 41].
The traditional Delphi technique’s recurring and multiple feedback nature necessitates a significant amount of time, which increases the risk of participant attrition across rounds [42]. In 2006, Gordon & Pease introduced the Real-Time Delphi (RTD) as an alternative to multi-round Delphi surveys, aiming to improve the process’s efficiency and shorten the required time. In this method, participants can view the group’s responses for each question in real time and the total number of responses simultaneously [43]. RTD is round-less; however, in light of changing feedback, participants can be recommended to revisit and re-rate questions. The responses participants will see on visiting an RTD survey reflect those who have participated in the survey up to that point rather than reflecting all survey respondents after each round as in traditional Delphi.
The number of papers using RTD is increasing. The RTD has been used to develop consensus across a range of settings, including economics, [43, 44] transportation [44], education [45, 46], religion [47], professional development [48], and sustainability [49]. RTD has been used in healthcare studies that aim to develop a medication adherence technologies repository [50] and determine the negative impacts of substance use disorders among people with HIV in the United States [51]. Several researchers have incorporated an RTD approach into the COS development processes [52, 53].
A comparison of real-time and traditional Delphi approaches for the future of the logistics industry in the year 2025 showed no statistically significant differences in participants’ projections on the future of logistics between the two studies [42]. A study comparing online Delphi and Real-Time Delphi surveys suggested that the RTD may reduce survey time and expenses and increase tester convergence and consensus. However, the study is small (n = 12) and lacks clarity on experimental methods [54]. A randomised trial comparing a three-round with a Real-Time Delphi approach on outcomes prioritised as part of a core outcome set development is currently awaiting reporting [52].
Participants.
We aim to include at least 100 stakeholders across three different stakeholder groups i.e., (1) researchers, (2) healthcare providers, and (3) patients’ parents/family members and parent representatives. We plan to include participants from high, middle, and low-income countries. Participants will be sought through professional associations and groups representing different stakeholders. Through the encouragement of the participants to forward the invitation, especially to stakeholders from low-income countries, snowball sampling will also be encouraged.
Procedures.
Previously, 88 outcomes were identified from 90 randomised trials [9]. Those will be mapped to Webbe et al.’s framework [31]. If a domain and/or outcome cannot be mapped, a new domain/outcome will be created. This will be combined with the domains and outcomes identified in Stage 1. Duplicate domains and outcomes will be removed, leaving a list of unique domains and outcomes for presentation as the ‘long list’ outcomes for use in the Delphi survey.
We will develop a survey in which participants will be invited to rate the importance of each outcome. Participants will rate the importance of each outcome for inclusion in the COS on a 9-point Likert scale (i.e., 1–3 limited importance, 4–6 important but not critical, and 7–9 critical) [11].
Once a participant rates an outcome, they will be presented with ‘real-time’ feedback on the portion of participants from each stakeholder group and all participants’ overall rating for each point on the 9-point scale for each outcome. Participants will then consider their ratings based on real-time feedback on responses from other participant groups and the group overall.
Before the survey goes live, we want to ensure that the survey is populated with feedback responses representative of the stakeholder groups. For this, we will recruit at least 5 participants from each stakeholder group (researchers, healthcare providers, patients’ parents/family members and parent representatives). These participants will also be included in the final survey numbers. This first group of participants will be invited to revisit the survey when it is live, to see stakeholder responses and amend their rating for each outcome if they wish.
Once the main survey goes live, participants will have a 3-week window during which they will be invited to re-visit the survey and review responses if they wish. They can choose to retain their original rating for each outcome or alter their rating based on how others have responded. Based on a comparison of software-based tools for RTD [55, 56], we will use Surveylet (https://calibrum.com) platform for our RTD.
At the end of the three weeks, outcomes that scored 7–9 by 70%, and 1–3 by less than 15% of participants across all stakeholder groups will be brought forward to the consensus meetings. When 50% or fewer participants score 7–9 in each stakeholder group that outcome will be excluded. This consensus definition was created to ensure that a measure could not achieve consensus if a minority stakeholder group mostly rated it as ‘limited importance’ and was used previously during the development of other COSs [57–59].
Stage 3: Consensus meetings
The consensus meetings aim to agree on the final COS through online discussions with key stakeholders.
Participants.
Consensus meetings will be conducted with an international panel of stakeholders representing patients’ parents/family members and parent representatives, researchers, and healthcare providers from high, low, and middle-income countries.
Schedule.
We will hold two independent online consensus meetings, including at least three people from each stakeholder group, to make it feasible for the participants living in different time zones. The unique outcomes identified for inclusion in the COS from each of the two consensus meetings will be discussed at a third, final online consensus meeting attended by stakeholders representing patients’ parents/family members and parent representatives, researchers, and healthcare providers from high, low, and middle-income countries.
During the meetings, the outcomes emerging from the RTD will be presented to the participants, along with the voting patterns of the stakeholder groups for each outcome. An experienced facilitator who will not participate in voting will chair each consensus meeting. There will be anonymous, computerised voting after a discussion of each outcome. If at least 80% of participants, including at least one representative from each stakeholder group, vote in favour of an outcome, it will be included in the COS.
Stage 4: Dissemination and implementation strategy
Once the consensus meeting is over and the final COS has been approved, a paper will be written, published, and distributed in accordance with the COS-STAP recommendations [16]. The final COS will be made available through the COMET database. We will present our results at national and international conferences to encourage researchers and clinicians to use the COS. We will target the audience/key stakeholders (e.g., all participants in the survey, known maternity and neonatal care researchers, the Cochrane Pregnancy and Childbirth Group, and the Cochrane Neonatal Group.
Discussion
Despite advancements in the quality of neonatal care, sepsis-related morbidity and mortality are growing concerns, particularly in low-middle-income countries [60]. Currently, there is inconsistency in the reporting of outcomes in studies on treatments of neonatal sepsis. A COS can help standardize outcomes measured and reported in studies and enable comparisons between and synthesis across them [61]. By conducting a systematic review, RTD survey, and consensus meetings, we will develop a COS for studies evaluating the effects of treatments for neonatal sepsis. We anticipate that this COS will minimise heterogeneity in outcomes measured and reported, enable better comparison and synthesis, and reduce research waste.
Supporting information
S1 Checklist. PRISMA-P (Preferred Reporting Items for Systematic review and Meta-Analysis Protocols) 2015 checklist: Recommended items to address in a systematic review protocol*.
https://doi.org/10.1371/journal.pone.0295325.s001
(DOC)
S1 File. NESCOS search strategy.
The complete search strategy of the qualitative systematic review of the NESCOS project.
https://doi.org/10.1371/journal.pone.0295325.s002
(DOCX)
References
- 1. Rudd KE, Johnson SC, Agesa KM, Shackelford KA, Tsoi D, Kievlan DR, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet. 2020;395(10219):200–11. pmid:31954465; PubMed Central PMCID: PMC6970225.
- 2. Fleischmann C, Reichert F, Cassini A, Horner R, Harder T, Markwart R, et al. Global incidence and mortality of neonatal sepsis: a systematic review and meta-analysis. Arch Dis Child. 2021. Epub 20210122. pmid:33483376; PubMed Central PMCID: PMC8311109.
- 3. Bakhuizen SE, de Haan TR, Teune MJ, van Wassenaer-Leemhuis AG, van der Heyden JL, van der Ham DP, et al. Meta-analysis shows that infants who have suffered neonatal sepsis face an increased risk of mortality and severe complications. Acta Paediatr. 2014;103(12):1211–8. Epub 2014/07/31. pmid:25073543.
- 4. Cai S, Thompson DK, Anderson PJ, Yang JY. Short- and Long-Term Neurodevelopmental Outcomes of Very Preterm Infants with Neonatal Sepsis: A Systematic Review and Meta-Analysis. Children (Basel). 2019;6(12). Epub 2019/12/07. pmid:31805647; PubMed Central PMCID: PMC6956113.
- 5. McGovern M, Giannoni E, Kuester H, Turner MA, van den Hoogen A, Bliss JM, et al. Challenges in developing a consensus definition of neonatal sepsis. Pediatr Res. 2020;88(1):14–26. Epub 2020/03/04. pmid:32126571.
- 6. Wynn JL, Wong HR, Shanley TP, Bizzarro MJ, Saiman L, Polin RA. Time for a neonatal-specific consensus definition for sepsis. Pediatr Crit Care Med. 2014;15(6):523–8. Epub 2014/04/23. pmid:24751791; PubMed Central PMCID: PMC4087075.
- 7. McMullan B, Cooper C, Spotswood N, James R, Jones C, Konecny P, et al. Antibiotic prescribing in neonatal sepsis: an Australian nationwide survey. BMJ Paediatr Open. 2020;4(1):e000643. Epub 2020/04/02. pmid:32232180; PubMed Central PMCID: PMC7101048.
- 8. Wynn JL. Defining neonatal sepsis. Curr Opin Pediatr. 2016;28(2):135–40. Epub 2016/01/15. pmid:26766602; PubMed Central PMCID: PMC4786443.
- 9. Henry CJ, Semova G, Barnes E, Cotter I, Devers T, Rafaee A, et al. Neonatal sepsis: a systematic review of core outcomes from randomised clinical trials. Pediatr Res. 2022;91(4):735–42. Epub 2022/01/09. pmid:34997225; PubMed Central PMCID: PMC9064797.
- 10. Bellucci C, Hughes K, Toomey E, Williamson PR, Matvienko-Sikar K. A survey of knowledge, perceptions and use of core outcome sets among clinical trialists. Trials. 2021;22(1):1–13.
- 11. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280. Epub 2017/07/07. pmid:28681707; PubMed Central PMCID: PMC5499094.
- 12.
Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Assessing risk of bias in a randomized trial. In: Higgins J, Thomas J, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.4, 2023.
- 13. Clarke M. Standardising outcomes for clinical trials and systematic reviews. Trials. 2007;8:39. Epub 2007/11/28. pmid:18039365; PubMed Central PMCID: PMC2169261.
- 14. Saldanha IJ, Lindsley KB, Money S, Kimmel HJ, Smith BT, Dickersin K. Outcome choice and definition in systematic reviews leads to few eligible studies included in meta-analyses: a case study. BMC Med Res Methodol. 2020;20(1):30. Epub 2020/02/13. pmid:32046643; PubMed Central PMCID: PMC7014938.
- 15. Williamson PR, de Avila Oliveira R, Clarke M, Gorst SL, Hughes K, Kirkham JJ, et al. Assessing the relevance and uptake of core outcome sets (an agreed minimum collection of outcomes to measure in research studies) in Cochrane systematic reviews: a review. BMJ Open. 2020;10(9):e036562. Epub 2020/09/09. pmid:32895272; PubMed Central PMCID: PMC7476465.
- 16. Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndardised Protocol Items: the COS-STAP Statement. Trials. 2019;20(1):116. Epub 2019/02/13. pmid:30744706; PubMed Central PMCID: PMC6371434.
- 17. Keeley T, Williamson P, Callery P, Jones LL, Mathers J, Jones J, et al. The use of qualitative methods to inform Delphi surveys in core outcome set development. Trials. 2016;17(1):230. Epub 2016/05/05. pmid:27142835; PubMed Central PMCID: PMC4855446.
- 18. Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med. 2017;14(11):e1002447. Epub 20171116. pmid:29145404; PubMed Central PMCID: PMC5689835.
- 19. Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology. 2012;12(1):1–8. pmid:23185978
- 20. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9, W64. Epub 20090720. pmid:19622511.
- 21. Booth A, Noyes J, Flemming K, Moore G, Tunçalp Ö, Shakibazadeh E. Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ global health. 2019;4(Suppl 1):e001107. pmid:30775019
- 22. Flemming K, Noyes J. Qualitative evidence synthesis: Where are we at? International Journal of Qualitative Methods. 2021;20:1609406921993276.
- 23. Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, et al. Qualitative evidence. Cochrane Handbook for Systematic Reviews of Interventions version 63: Cochrane; 2022.
- 24. Pearson M, Moxham T, Ashton K. Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Evaluation & the health professions. 2011;34(3):297–308. pmid:21224267
- 25. Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information & Libraries Journal. 2010;27(2):114–22.
- 26. Evans D. Database searches for qualitative research. J Med Libr Assoc. 2002;90(3):290–3. pmid:12113512; PubMed Central PMCID: PMC116400.
- 27. Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a methodology for cluster searching to provide conceptual and contextual “richness” for systematic reviews of complex interventions: case study (CLUSTER). BMC medical research methodology. 2013;13(1):1–14. pmid:24073615
- 28. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Systematic reviews. 2016;5(1):1–10. pmid:27919275
- 29.
Critical Appraisal Skills Programme. CASP for Qualitative Studies Checklist 2018. Available from: https://casp-uk.net/wp-content/uploads/2018/03/CASP-Qualitative-Checklist-2018_fillable_form.pdf.
- 30.
Thomas J, Harden A., Newman M. Synthesis: combining results systematically and appropriately. An Introduction to Systematic Reviews Sage Publications; 2012.
- 31. Webbe JWH, Duffy JMN, Afonso E, Al-Muzaffar I, Brunton G, Greenough A, et al. Core outcomes in neonatology: development of a core outcome set for neonatal research. Arch Dis Child Fetal Neonatal Ed. 2020;105(4):425–31. Epub 20191115. pmid:31732683; PubMed Central PMCID: PMC7363790.
- 32.
Williamson K. Research methods for students, academics and professionals: Information management and systems: Elsevier; 2002.
- 33.
Sproull NL. Handbook of research methods: A guide for practitioners and students in the social sciences: Scarecrow press; 2002.
- 34. Thangaratinam S, Redman CW. The delphi technique. The obstetrician & gynaecologist. 2005;7(2):120–5.
- 35. Cleverley K, McCann E, O’Brien D, Davies J, Bennett K, Brennenstuhl S, et al. Prioritizing core components of successful transitions from child to adult mental health care: a national Delphi survey with youth, caregivers, and health professionals. European Child & Adolescent Psychiatry. 2021:1–14. pmid:34089382
- 36. Hultsjö S, Berterö C, Arvidsson H, Hjelm K. Core components in the care of immigrants with psychoses: A Delphi survey of patients, families, and health‐care staff. International journal of mental health nursing. 2011;20(3):174–84. pmid:21352448
- 37. Campbell S, Shield T, Rogers A, Gask L. How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings? BMJ Quality & Safety. 2004;13(6):428–34.
- 38. Lambert SD, Ould Brahim L, Morrison M, Girgis A, Yaffe M, Belzile E, et al. Priorities for caregiver research in cancer care: an international Delphi survey of caregivers, clinicians, managers, and researchers. Supportive Care in Cancer. 2019;27(3):805–17. pmid:30062587
- 39. Efstathiou N, Ameen J, Coll A-M. Healthcare providers’ priorities for cancer care: a Delphi study in Greece. European Journal of Oncology Nursing. 2007;11(2):141–50. pmid:16914376
- 40. Santaguida P, Dolovich L, Oliver D, Lamarche L, Gilsing A, Griffith LE, et al. Protocol for a Delphi consensus exercise to identify a core set of criteria for selecting health related outcome measures (HROM) to be used in primary health care. BMC family practice. 2018;19(1):1–14.
- 41. Gharibi F, Tabrizi JS. Development of an accreditation model for health education and promotion programs in the Iranian primary healthcare system: a Delphi study. Health Promotion Perspectives. 2018;8(2):155. pmid:29744312
- 42. Gnatzy T, Warth J, von der Gracht H, Darkow IL. Validating an innovative real-time Delphi approach—A methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc. 2011;78(9):1681–94. WOS:000297564200016.
- 43. Gordon T, Pease A. RT Delphi: An efficient, "round-less" almost real time Delphi method. Technol Forecast Soc. 2006;73(4):321–33. WOS:000236821400001.
- 44. Meyer T, von der Gracht HA, Hartmann E. Technology foresight for sustainable road freight transportation: Insights from a global real‐time Delphi study. Futures & Foresight Science. 2022;4(1):e101.
- 45. Sunderji N, Waddell A. Using real-time Delphi to develop a consensus on competencies. Med Educ. 2015;49(11):1151–2. WOS:000363514600026. pmid:26494082
- 46.
Genova JP. Futuring in high performing, high poverty, and high minority schools: A real-time Delphi study: St. John’s University (New York), School of Education and Human Services; 2018.
- 47. Gary JE. Outlook 2020: Results from a Real-Time Delphi Survey of Global Pentecostal Leaders. Pneuma. 2012;34(3):383–414.
- 48. Gary JE, von der Gracht HA. The future of foresight professionals: Results from a global Delphi study. Futures. 2015;71:132–45. WOS:000361407800012.
- 49. Serra M, Antonio N, Henriques C, Afonso CM. Promoting Sustainability through Regional Food and Wine Pairing. Sustainability. 2021;13(24):13759.
- 50. Makovec UN, Goetzinger C, Ribaut J, Barnestein-Fonseca P, Haupenthal F, Herdeiro MT, et al. Developing a medication adherence technologies repository: proposed structure and protocol for an online real-time Delphi study. BMJ open. 2022;12(4):e059674. pmid:35459677
- 51. Garner BR, Gotham HJ, Knudsen HK, Zulkiewicz BA, Tueller SJ, Berzofsky M, et al. The prevalence and negative impacts of substance use disorders among people with HIV in the United States: A real-time delphi survey of key stakeholders. AIDS and Behavior. 2022;26(4):1183–96. pmid:34586532
- 52. Quirke FA, Healy P, Bhraonáin EN, Daly M, Biesty L, Hurley T, et al. Multi-Round compared to real-time Delphi for consensus in core outcome set (COS) development: a randomised trial. Trials. 2021;22(1):1–9.
- 53. Dube K, Ayebare E, Bakhbakhi D, Bedwell C, Chandriah S, Chaudhry N, et al. Protocol for the development of a meta-core outcome set for stillbirth prevention and bereavement care following stillbirth. medRxiv. [Preprint]. 2022 [cited 2023 August 22]. Available from: https://www.medrxiv.org/content/10.1101/2022.10.13.22281030v1
- 54. Hsieh CH, Tzeng FM, Wu CG, Kao JS, Lai YY. The Comparison of Online Delphi and Real-Time Delphi. 2011 Proceedings of Picmet 11: Technology Management in the Energy-Smart World (Picmet). 2011. WOS:000298384400042.
- 55. Aengenheyster S, Cuhls K, Gerhold L, Heiskanen-Schüttler M, Huck J, Muszynska M. Real-Time Delphi in practice—A comparative analysis of existing software-based tools. Technol Forecast Soc. 2017;118:15–27.
- 56. Varndell W, Fry M, Elliott D. Applying real-time Delphi methods: development of a pain management survey in emergency nursing. BMC nursing. 2021;20(1):1–11.
- 57. Needham DM, Sepulveda KA, Dinglas VD, Chessare CM, Friedman LA, Bingham III CO, et al. Core outcome measures for clinical research in acute respiratory failure survivors. An international modified Delphi consensus study. American journal of respiratory and critical care medicine. 2017;196(9):1122–30. pmid:28537429
- 58. Prinsen CA, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to select outcome measurement instruments for outcomes included in a “Core Outcome Set”–a practical guideline. Trials. 2016;17(1):1–10.
- 59. Bartlett SJ, Hewlett S, Bingham CO, Woodworth TG, Alten R, Pohl C, et al. Identifying core domains to assess flare in rheumatoid arthritis: an OMERACT international patient and provider combined Delphi consensus. Annals of the rheumatic diseases. 2012;71(11):1855–60. pmid:22772326
- 60. Satar M, Ozlu F. Neonatal sepsis: a continuing disease burden. Turk J Pediatr. 2012;54(5):449–57. Epub 2013/02/23. pmid:23427506.
- 61. Molloy EJ, Wynn JL, Bliss J, Koenig JM, Keij FM, McGovern M, et al. Neonatal sepsis: need for consensus definition, collaboration and core outcomes. Pediatr Res. 2020;88(1):2–4. Epub 2020/03/21. pmid:32193517.