https://orcid.org/0000-0002-6379-8780
Figures
Abstract
Introduction
The Nominal Group Technique (NGT) is a consensus group method used to synthesize expert opinions. Given the global shift to virtual meetings, the extent to which researchers leveraged virtual platforms is unclear. This scoping review explores the use of the vNGT in healthcare research during the COVID-19 pandemic.
Methods
Following the Arksey and O’Malley’s framework, eight cross-disciplinary databases were searched (January 2020-July 2022). Research articles that reported all four vNGT stages (idea generation, round robin sharing, clarification, voting) were included. Media Synchronicity Theory informed analysis. Corresponding authors were surveyed for additional information.
Results
Of 2,589 citations, 32 references were included. Articles covered healthcare (27/32) and healthcare education (4/32). Platforms used most were Zoom, MS Teams and GoTo but was not reported in 44% of studies. Only 22% commented on the benefits/challenges of moving the NGT virtually. Among authors who responded to our survey (16/32), 80% felt that the vNGT was comparable or superior.
Conclusions
The vNGT provides several advantages such as the inclusion of geographically dispersed participants, scheduling flexibility and cost savings. It is a promising alternative to the traditional in-person meeting, but researchers should carefully describe modifications, potential limitations, and impact on results.
Citation: Lee SH, ten Cate O, Gottlieb M, Horsley T, Shea B, Fournier K, et al. (2024) The use of virtual nominal groups in healthcare research: An extended scoping review. PLoS ONE 19(6): e0302437. https://doi.org/10.1371/journal.pone.0302437
Editor: Nabeel Al-Yateem, University of Sharjah, UNITED ARAB EMIRATES
Received: June 30, 2023; Accepted: April 3, 2024; Published: June 12, 2024
Copyright: © 2024 Lee et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The entirety of the quantitative data collected and analyzed for this review is within the manuscript and its Supporting information files. Interview data is restricted due to the participants only consenting to aggregate data sharing. Interview data will be made available upon request and approval from the University of Ottawa’s Department of Innovation in Medical Education (DIME) at dime@uottawa.ca.
Funding: This project is being funded by a medical education research grant, Department of Medicine, University of Ottawa. The funders will not have any role in the study design, data collection, and analysis, decision to publish or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Consensus group methods are often used to synthesize expert opinions when evidence is lacking or contradictory. They have been increasingly utilized in healthcare, business, engineering, and education [1–4].
These methods have been used to inform and evaluate a variety of healthcare-related activities (e.g. defining diagnostic criteria, informing management guidelines [2, 5, 6]; course evaluation [7]; amongst other uses [8–12]–Foundational principles of consensus methods include anonymity, iteration, controlled feedback, statistical group response, and structured interaction [1].
The NGT was developed for idea generation and group decision-making [3]. The structured format allows for effective generation and prioritization of ideas. The NGT has several key steps: presentation of the nominal question, silent generation of ideas in writing, round-robin sharing of ideas from participants, group discussion and clarification of each idea, followed by anonymous individual voting to rank priority areas. Feedback of results followed by more discussion and re-rating may occur [13, 14].
The NGT differs from other consensus techniques by employing an in-person meeting among 5–12 participants [15]. This is touted by many to be a strength, as it not only allows discordant ideas on topics of mutual interest to be freely expressed and synthesized, but it also affords the opportunity to explore any differences in opinions. Further, the collaborative feature of the NGT may increase ownership of research among stakeholders and enhance the potential of informing policy or practice [3]. However, the small number of participants is also regarded as a limitation, and the potential for dominant members to unduly influence group decision-making cannot be dismissed, even if NGT aims to counteract this effect in its structured procedure.
The coronavirus disease of 2019 (COVID-19) pandemic has fundamentally changed the way we work, learn, and conduct research [16] and NGT has not been exempt from the shift; many researchers transitioned the in-person NGT meeting to a virtual environment [17–20]. For instance, Nelson et al. (2022) employed 3 nominal groups to identify burnout strategies in resident physicians, one being the traditional in-person meeting followed by two of which were held virtually via Zoom [17]. Timmermans et al. (2022) conducted three virtual NGTs (vNGT) and provided recommendations on the transition to the synchronous, online environment, but their claims were drawn solely from the authors’ subjective experiences [20].
There remains ambiguity regarding the extent to which other researchers have shifted the NGT to a virtual format, the types of platforms used, and modifications made to the technique. In addition to this exploratory inquisition, it is also important to consider whether researchers voiced any challenges, or perceived advantages or disadvantages. Taken collectively, these lessons learned would provide guidance for future users of vNGT.
The COVID-19 pandemic has supported an upsurge in information and communication technologies, which fundamentally changed how individuals interact. The Media Synchronicity Theory (MST) is a conceptual framework that considers the effectiveness of information and communication technologies in facilitating group work [21]. Through the lens of MST, all communication activities are grouped into two simple processes: conveyance (transmission of new information to the receiver that enables the creation and revision of individual understanding of a problem) and convergence (mutual process that governs how individuals understand and negotiate a common ground for a problem), the latter of which may involve more authentic, rapid back and forth transmission of information. For the NGT, the idea generation phase would presumably require more conveyance, while the discussion and clarification phase emphasize more convergence. Both processes are necessary, but the proportion depends on the complexity and inherent characteristics of the research.
In summary, the NGT has been used in a variety of research settings, to inform important decisions. Although interaction between panel members is fundamental to the process, many in-person meetings have moved online due to the pandemic. At present, it is unclear to what extent the NGT has been undertaken virtually, and what implications this may have for the decision process.
Objective
The overarching purpose of this study is to explore the use of the virtual Nominal Group Technique (vNGT) in healthcare research. Specific objectives are to answer the following questions: 1) To what extent has the NGT been used virtually? 2) What virtual communication platforms are used? 3) What modifications to the technique were made to accommodate this online format? And 4) What advantages and disadvantages were noted by authors?
Methods
Following a study protocol that has been published [22], a scoping review was conducted. A scoping review was considered appropriate since this topic is poorly defined, and the purpose was to map the literature, find key concepts, types and sources of evidence and identify gaps in the literature. The study followed the Arksey and O’Malley framework [23] and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Review [24]. Written ethics approval was granted from the University of Ottawa Research Ethics Board on September 26, 2022. All participants answering the survey provided written informed consent.
Step 1: Identifying the research question
Our study began with a broad objective: to explore the use of the vNGT in healthcare research. This study included all English-language published research in healthcare and healthcare education that used the NGT in a virtual format. This included using any non-in-person format, such as videoconferencing. The outcomes of interest included the author’s description of how the vNGT was used, and the perceived success of the process, benefits, risks, and challenges. Informed by a preliminary search of the literature, we decided upon the following questions:
- To what extent has the NGT been used virtually?
- What virtual communication platforms were used?
- What modifications to the technique were made to accommodate this online format?
- What advantages and disadvantages were noted by authors?
Step 2 & 3: Identifying relevant articles and article selection
We started with the following framework: Population, Concept, Context. The population included any published research studies using the nominal group technique, the concept entailed the use of virtual modalities to execute the nominal group technique, and context involved any study topic.
Several pilot searches were undertaken to define the search strategy. Details can be found in the published protocol [22]. Final inclusion and exclusion criteria are noted in Table 1.
The final search strategies were executed by an information specialist (KF) and peer reviewed using the PRESS guideline [25]. The searches were conducted July 15th, 2022, in: MEDLINE(R) ALL (OvidSP), Embase (OvidSP), CINAHL (EBSCOHost), ERIC (OvidSP), Education Source (EBSCOHost), APA PsycInfo (OvidSP), Web of Science, and Scopus to retrieve references published January 2020 to July 2022. This time period was selected to detect studies conducted during the COVID-19 pandemic to reflect the more rapid shift to virtual formats. No search filters, or language limits were used, but conference abstracts were removed since only full papers were of interest. The search strategies are included in S1 File.
Using the Covidence software [26], all titles and abstracts were reviewed in duplicate with varying pair combinations by three co- authors (SHL, SHM, MG and a research assistant). Those meeting inclusion criteria, or if not clearly conducted in person were pulled for full text review. Duplicate review of full text articles against inclusion/exclusion criteria was completed. All conflicts were resolved via consensus discussion with a third member of the team.
Step 4: Charting the data
Through an iterative process the data extraction form was developed. In addition to publication-level information and demographics, concepts related to the MST were explored. Details can be found in the published protocol [22].
Following best practices from the manual for evidence synthesis [27] at least two members (SL, SHM) independently reviewed 20% of articles applying the final data extraction form. Thereafter, one member of the research team carried out data extraction with verification from a second member. Any ambiguous items that arose were resolved through discussion with the senior author (SHM).
Step 5: Collating, summarizing, and reporting the results
Both quantitative and thematic analyses were used to synthesize study results. Quantitative analysis focused on the nature (e.g., education, clinical research, guideline development) and distribution of relevant articles. Two members of the research team (SL, SHM) independently reviewed the data to identify preliminary themes as informed by the MST. Several group meetings with all team members were held to review the data and to agree on a final summary of findings.
Step 6: Survey of the authors
Upon review of several studies, it became apparent that many of the articles did not comment on the “virtual” aspect of the NGT. As a result, we disseminated an online survey to the corresponding authors for articles included in the study. The survey sought to confirm which virtual platform was used and for which steps of the NGT, if additional functions were used (e.g., chats features) or modifications made to the NGT to accommodate the virtual platform, why the virtual platform was used, their general impressions and perceived benefits and challenges, comparing their experience with in-person NGTs, and any lessons learned. The survey is available in S2 File. The Survey. The Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) Checklist can be found in S3 File.
Results
A total of 2,589 records were identified through database searching, of which 1,754 were removed as duplicates (Fig 1). Excluding irrelevant titles and abstracts, 598 full-text articles were assessed for eligibility based on our exclusion criteria. Resulting in 32 full-text articles meeting inclusion criteria.
Abbreviation: NGT, nominal group technique.
Study demographics
Demographic information can be found in Table 2. The largest number of studies were published in 2022 (50%; 16/32) [6, 28–42], followed by 2021 [43–54] and 2020 (13%; 4/32) [19, 55–57]. As seen in Table 2 geographic distribution, virtual studies were predominantly carried out at the national level (53%; 17/32), followed by intercontinental (19%; 6/32), international (16%; 5/32), and local settings (6%; 2/32).
Research questions were more related to healthcare (87.5%; 28/32) than to healthcare education (12.5%; 4/32). Within healthcare, topics were varied and included items such as improving care for inmate dementia [30], home rehabilitation for stroke survivors [49], COVID-19 vaccine rollout in pharmacies [51] barriers to testing lipids and achieving disease control in rheumatoid arthritis [56, 57]. Patients were often central to the vNGT with topics including patient views on the treatment of osteoporosis [28], multiple myeloma [50] or osteoarthritis [43] and patient reported outcomes in heart failure [33].
vNGT participants
Five studies did not report the number of participants (15.6%) [6, 38, 41, 46, 47]. For those that did, the total number of participants per NGT group ranged from 2–20, however it was not always clear how many participants were in each NGT group as some studies ran several NGT groups in parallel (Table 2).
Virtual modality
Table 3 provides details of the technique itself. All studies used an online conferencing modality, but many did not report which platform was used (43.7%; 14/32). Of those that did (n = 18), the most common software noted was Zoom (66.6%; 12/18), followed by Microsoft Teams (11.1%; 2/18), GoTo (11.1%; 2/18), Crisco (5.6%; 1/18) and Skype (5.6%%; 1/18). Half of the articles also added an email or e-survey modality to the study design (50%; 16/32) either before, during, or after participants convened online. The rest used telephone communication in addition to online conferencing (9%; 3/32). Some studies held virtual sessions in parallel with in-person offerings to conduct the NGT simultaneously among various groups (16%; 5/32).
vNGT modifications for virtual
Modifications to accommodate the technique virtually were often implied to have taken place, but rarely commented upon (25%; 8/32). Notably, the online break-out room functionality replaced physical meeting rooms to conduct multiple nominal group sessions [51]. Investigators also implemented online sheets [51], interactive whiteboards, and slideshows [39] to complement the process.
Asynchronous communication modalities, defined as non-interactive, non-simultaneous platforms such as email threads and e-survey, were noted in a quarter of the articles (25%; 8/32). The idea generation step was conducted asynchronously in 8 of 32 articles (25%). At the discussion and clarification stage, Bavelaar et al., 2022 [29] and Hoops et al., 2022 [31] offered asynchronous participation in conjunction with the interactive discussion in the form of email threads.
Five studies held in-person and virtual sessions in parallel, two of which allowed for both in-person and virtual within the same meeting based on context-dependent needs (e.g., geographic difference, member preference, technical challenges) [29, 31].
vNGT method
A minority of studies defined consensus a priori (16%; 5/32). Four of the articles used a percentage score ranging from 50% to 75% agreement as the cut-off value [31, 48, 49, 54].
Documentation
In many instances, the discussion phase of the NGT was formally documented either via audio or video recording (41%; 13/32) or transcription in the form of written notes during the meeting (69%; 22/32). Some opted to using both recording and live transcription (34%; 11/32).
Other findings
Our inclusion criteria required that all stages of the vNGT be virtual (i.e., no in-person components). However, during the screening we did note that 52 articles incorporated at least one virtual component.
Descriptive characteristics
Benefits and challenges of vNGT.
Only eight articles (25%;8/32) provided comments on the benefits or challenges (see Table 4). Online participation afforded geographical and scheduling flexibility, allowing a more representative sample of participants to convene in nominal group sessions [29]. One study suggested that the online nature of discussions may also have promoted the sharing of ideas among participants [51]. Janssens et al., (2021) suggested for older participants that the telephone was preferable [50]. The addition of asynchronous platforms was advantageous for one study as it provided more time for researchers to procure stakeholder perspectives, participants to understand study material through repeated review, and to efficiently prioritize and rank ideas.
In contrast, some comments directly addressed the challenges of adapting onsite activities to videoconferencing, email threads, and phone calls, which could have affected participant engagement. Two articles suggested that virtual modalities could have acted as a barrier to participation to some individuals, such as older participants [50, 57]. Prospective participants may have declined invitations to the study due to the extra effort required to partake in online meetings and asynchronous activities [44]. Furthermore, differences in geographical time zones posed time constraints for participants, which could have influenced group discussions, and therefore the results of the study [51]. One study mentioned that results could have been further compromised via asynchronous modalities as participants would be directed to a priori information online [44].
Participant feedback.
One article openly gathered feedback from participants regarding the virtual nature of the NGT [44]. Participants expressed that the asynchronous format provided ample time to prepare ideas before of convening online.
Corresponding author survey results.
The survey was completed by half of corresponding authors (50%; 16/32). Authors reported having been involved in in-person sessions 0 to 4 times in the past, and virtual sessions from 1 to 10 times. Participants were also asked their general impression of how the virtual worked compared to in-person NGT. Overall, 44% (7/16) felt that the vNGT was better, 36% (6/16) felt that both media performed similarly, and 19% (3/16) felt that vNGT was inferior to in-person NGT. Thematic analysis across the questions fell under benefits and concerns of using the vNGT as seen in Table 5.
Discussion
This study set out to explore the use of the vNGT. Our findings demonstrate researchers use virtual platforms considering restrictions for travel and the requirements for social distancing. Although 8 databases were included in our study, the vast majority of articles reporting use of a virtual platform were from medical journals and related to healthcare.
The pandemic has imposed unprecedented sanctions in the way research is conducted, necessitating the transition from physical to virtual interactive modalities. Recent guidelines have suggested that researchers should report any modifications made as a result of extenuating circumstances (e.g., COVID-19), report important modifications to the methods, including mitigating strategies and impact on the results [58]. Our review demonstrated that most published research did not adhere to these guidelines; approximately a quarter of studies reported modifications to the method and just under a half which virtual platform was used. None considered the impact on outcomes. Overall, this is consistent with poor reporting seen in consensus research [4, 59, 60].
A few articles discussed the benefits and challenges of conducting the NGT virtually. Commonly cited benefits were increased accessibility for participants by overcoming barriers to participation, such as psychological discomfort in group settings [19], transportation issues [29, 49], or time constraints [32]. This advantage may have been amplified for healthcare professionals seeking to actively participate in research in the midst of battling the COVID-19 pandemic and has been previously reported [58]. However, lack of access or ability to use digital platforms may have reduced accessibility for some participants, due to socioeconomic disadvantage [28] or advanced age [50, 57]. Previous literature has certainly described age, race and literacy-related disparities in the use of technology with online populations being younger and more affluent [61, 62]. However, some report that online focus group participants were more likely to be non-white, less educated, and less healthy than the in-person sample [63]. Thus, online methods may lead to less representative participant demographics.
Several other limitations were noted such as timing of sessions if participants are from different time zones [51], and technical difficulties [28]. Testing the technology prior to use and having designated staff to address technical issues has been recommended [20], but is not always sufficient [28].
From the perspective of MST, the virtual platforms offered a variety of options to enhance communication. The capacity of communication modalities to support synchronicity is further informed by qualities intrinsic to the media, including, but not limited to transmission velocity, which refers to the speed at which a message reaches the recipient; parallelism, the number of simultaneous transmissions at any given time, and symbol sets, meaning different ways in which a message can be encoded. In general, symbol sets and transmission velocity enhance synchronicity, whereas parallelism decreases it [21, 64, 65].
The MST helped to inform our understanding of vNGT in pragmatic terms. Studies employed a diverse array of modalities depending on the NGT stage. In idea generation, which predominantly comprises one-way input, several studies [29, 31, 35, 36, 44, 48, 53, 54] adopted asynchronous modalities such as e-surveys or emails. On the other hand, discussion and clarification phases were conducted synchronously as demonstrated in all studies in our review with two also providing asynchronous additional options [29, 31]. This is in keeping with the idea that convergent communication benefits from higher transmission velocity. See Fig 2 for considerations at each stage.
Concerns about engagement and interactivity were raised by several corresponding authors in our study, but none explored this in detail. Media richness theory suggests that computer-mediated group communication has lower social presence and less task focus [66] but this theory predates the current advances in platforms, which have improved our ability to see and interact with colleagues. One study that directly compared the web-based NGT to a traditional in-person format was using asynchronous technology with no visual capabilities [66]. Not surprisingly, the online participants were significantly less satisfied with the decision process compared to the traditional session, but only marginal differences were found in the outcomes [66]. Another study compared video to in person interviews. Interestingly Skype interviews yielded some of the most unguarded responses and richest data [67].
For this review, the vNGT was defined as having all stages in a virtual format. Several authors used a combination of virtual and in-person, but these were not included and are worthy of further exploration.
Conclusion
The vNGT offers several potential benefits; inclusion of geographically dispersed participants, reduced time and expense for travel/meeting accommodations, and flexible scheduling. Reduced engagement and reduced participation for technology-challenge individuals are considerations. Since research on the vNGT itself is lacking, authors should clearly report modifications made and risks/benefits as well as potential impact on the consensus decision. Any minor adjustments in research protocols can introduce unanticipated vulnerabilities, thus compromising results and improved reporting is essential. In addition, more research is required to directly compare in-person to vNGTs including potential implications for cognitive biases [68].
Acknowledgments
Dr. Humphrey-Murto would like to acknowledge the work of Amanda Pace and Kate Scowcroft from the Department of Innovation in Medical Education for assisting with the study.
References
- 1. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ: British Medical Journal 1995; 311: 376. pmid:7640549
- 2. Murphy M, Black N, Lamping D, et al. Consensus development methods, and their use in clinical guideline development. Health Technology Assessment (Winchester, England) 1998; 2: i–88. pmid:9561895
- 3. Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. International journal of nursing practice 2012; 18: 188–194. pmid:22435983
- 4. Humphrey-Murto S, Varpio L, Wood TJ, et al. The use of the Delphi and other consensus group methods in medical education research: a review. Academic Medicine 2017; 92: 1491–1498. pmid:28678098
- 5. Salvi A, Nitti C, Fabbri A, et al. Diagnosis and Treatment of Deep Vein Thrombosis in the Emergency Department: Results of an Italian Nominal Group Technique Study. Clinical and Applied Thrombosis/Hemostasis 2020; 26: 1076029620959720. pmid:33112649
- 6. Ridgway A, Cotterill N, Dawson S, et al. Nocturia and chronic kidney disease: systematic review and nominal group technique consensus on primary care assessment and treatment. European Urology Focus. pmid:35031353
- 7. Coker J, Castiglioni A, Massie FS, et al. Evaluation of an advanced physical diagnosis course using consumer preferences methods: The nominal group technique. The American Journal of the Medical Sciences 2014; 347: 199–205. pmid:23552288
- 8. Myong Y, Shin H-I, Lee J-E, et al. Development and Validation of a New Scale to Assess Attitudes and Perspectives Toward Persons With Disabilities. Annals of Rehabilitation Medicine 2021; 45: 331–340. pmid:34496476
- 9. Tran C, Archibald D, Humphrey-Murto S, et al. What makes a high-quality electronic consultation (eConsult)? A nominal group study. Journal of Telemedicine and Telecare 2020; 26: 239–247. pmid:30717627
- 10. Halman S, Dudek N, Wood T, et al. Direct observation of clinical skills feedback scale: development and validity evidence. Teaching and Learning in Medicine 2016; 28: 385–394. pmid:27285377
- 11. Kottmann A, Krüger AJ, Sunde GA, et al. Establishing quality indicators for pre-hospital advanced airway management: a modified nominal group technique consensus process. British journal of anaesthesia 2022; 128: e143–e150. pmid:34674835
- 12. Virumbrales M, Elorduy M, Graell M, et al. COVID-19: Making the Best out of a Forced Transition to Online Medical Teaching—a Mixed Methods Study. Medical science educator 2022; 32: 337–347. pmid:35223143
- 13. Van De A, Delbecq AL. Nominal versus interacting group processes for committee decision-making effectiveness. Academy of management Journal 1971; 14: 203–212.
- 14. Campbell S, Cantrill J. Consensus methods in prescribing research. Journal of clinical pharmacy and therapeutics 2001; 26: 5–14. pmid:11286603
- 15. Humphrey-Murto S, Varpio L, Gonsalves C, et al. Using consensus group methods such as Delphi and Nominal Group in medical education research. Medical teacher 2017; 39: 14–19. pmid:27841062
- 16. Shamsuddin A, Sheikh A, Keers RN. Conducting research using online workshops during COVID-19: Lessons for and beyond the pandemic. International Journal of Qualitative Methods 2021; 20: 16094069211043744.
- 17. Nelson V, Dubov A, Morton K, et al. Using nominal group technique among resident physicians to identify key attributes of a burnout prevention program. Plos one 2022; 17: e0264921. pmid:35303009
- 18. Alkwai HM, Mirza A, Abdwani R, et al. Consensus clinical approach for a newly diagnosed systemic juvenile idiopathic arthritis among members of the pediatric rheumatology Arab group. International Journal of Pediatrics and Adolescent Medicine 2021; 8: 129–133. pmid:34350323
- 19. Johnson BJ, Zarnowiecki D, Hutchinson CL, et al. Stakeholder generated ideas for alternative school food provision models in Australia using the nominal group technique. International journal of environmental research and public health 2020; 17: 7935. pmid:33138030
- 20. Timmermans L, Huybrechts I, Decat P, et al. Recommendations for Researchers on Synchronous, Online, Nominal Group Sessions in Times of COVID-19: Fishbone Analysis. JMIR Formative Research 2022; 6: e34539. pmid:35225814
- 21. Dennis AR, Fuller RM, Valacich JS. Media, tasks, and communication processes: A theory of media synchronicity. MIS quarterly 2008; 575–600.
- 22. Humphrey-Murto S, Ho Lee S, Gottlieb M, et al. Protocol for an extended scoping review on the use of virtual nominal group technique in research. PloS one 2023; 18: e0280764. pmid:36662907
- 23. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International journal of social research methodology 2005; 8: 19–32.
- 24. Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of internal medicine 2018; 169: 467–473. pmid:30178033
- 25. McGowan J, Sampson M, Salzwedel DM, et al. PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of clinical epidemiology 2016; 75: 40–46.
- 26.
Covidence systematic review software, Veritas Health Innovation.
- 27. Peters MD, Godfrey C, McInerney P, et al. Scoping reviews. Joanna Briggs Institute reviewer’s manual 2017; 2015: 1–24.
- 28. Beaudart C, Silverman S, Gold DT, et al. A Qualitative Study to Assess US Patient Preferences between new Transdermal System and Injectable Anabolic Therapies for Osteoporosis Treatment. Archives of Osteoporosis 2022; 17: 57. pmid:35378644
- 29. Bavelaar L, Nicula M, Morris S, et al. Developing country-specific questions about end-of-life care for nursing home residents with advanced dementia using the nominal group technique with family caregivers. Patient Education and Counseling 2022; 105: 965–973. pmid:34376304
- 30. du Toit S, Ng S. Improving care for older prisoners living with dementia in Australian prisons: perspectives of external organizations. The Gerontologist 2022; 62: 543–555. pmid:34570214
- 31. Hoops K, Fahimi J, Khoeur L, et al. Consensus-driven priorities for firearm injury education among medical professionals. Academic medicine 2022; 97: 93–104. pmid:34232149
- 32. Klaic M, Kapp S, Hudson P, et al. Implementability of healthcare interventions: an overview of reviews and development of a conceptual framework. Implementation Science 2022; 17: 10. pmid:35086538
- 33. Lawson CA, Lam C, Jaarsma T, et al. Developing a core outcome set for patient-reported symptom monitoring to reduce hospital admissions for patients with heart failure. European Journal of Cardiovascular Nursing 2022; 21: 830–839. pmid:35404418
- 34. Li S, Honey A, Coniglio F, et al. Mental Health Peer Worker Perspectives on Resources Developed from Lived Experience Research Findings: A Delphi Study. International Journal of Environmental Research and Public Health 2022; 19: 3881. pmid:35409560
- 35. Liem SI, Vliet Vlieland TP, van de Ende CH, et al. Consensus-based recommendations on communication and education regarding primary care physical therapy for patients with systemic sclerosis. Musculoskeletal Care 2023; 21: 45–55. pmid:35689435
- 36. Love A, Cornwell P, Hewetson R, et al. Test item priorities for a screening tool to identify cognitive-communication disorder after right hemisphere stroke. Aphasiology 2022; 36: 669–686.
- 37. Mcclunie-Trust P, Jones V, Winnington R, et al. Doing case study research collaboratively: the benefits for researchers. International Journal of Qualitative Methods 2022; 21: 16094069221096296.
- 38. Ryan N, Vieira D, Gyamfi J, et al. Development of the ASSESS tool: a comprehenSive tool to Support rEporting and critical appraiSal of qualitative, quantitative, and mixed methods implementation reSearch outcomes. Implementation Science Communications 2022; 3: 1–13.
- 39. Singh JA. Disease modification in gout: a qualitative study of gout expert rheumatologists. Rheumatology Advances in Practice 2022; 6: rkab107.
- 40. Smith D, Cartwright M, Dyson J, et al. Selecting intervention content to target barriers and enablers of recognition and response to deteriorating patients: an online nominal group study. BMC Health Services Research 2022; 22: 766. pmid:35689227
- 41. van Merode NA, Dawson S, Coulthard E, et al. Assessment and treatment of nocturia in neurological disease in a primary care setting: systematic review and nominal group technique consensus. European Urology Focus.
- 42. Volkmer A, Cartwright J, Ruggero L, et al. Principles and philosophies for speech and language therapists working with people with primary progressive aphasia: An international expert consensus. Disability and Rehabilitation 2023; 45: 1063–1078. pmid:35352609
- 43. Singh JA. “I wish it had a place to go”: a nominal group study of barriers to the effectiveness of non-surgical treatments for knee osteoarthritis inclusive of minority populations. Arthritis Research & Therapy 2021; 23: 1–13.
- 44. Al-Yateem N, Ahmad AM, McCreaddie M, et al. Synthesizing core nursing skills to support behavioural-based interviews for nurses in the UAE: a nominal group study. Journal of Nursing Management 2021; 29: 953–961. pmid:33301631
- 45. Aronson KI, O’Beirne R, Martinez FJ, et al. Barriers to antigen detection and avoidance in chronic hypersensitivity pneumonitis in the United States. Respiratory Research 2021; 22: 225. pmid:34376172
- 46. Choquette D, Chan J, Bardi M, et al. Monitoring the transition of patients on biologics in rheumatoid arthritis: Consensus guidance for pharmacists. Pharmacy Practice (Granada); 19. pmid:34621449
- 47. Dawson S, Duncan L, Ahmed A, et al. Assessment and treatment of nocturia in endocrine disease in a primary care setting: systematic review and nominal group technique consensus. European Urology Focus.
- 48. Evangelista CB, Larsen KL, Cervero RM, et al. Corneal laser refractive surgery curriculum development in the military: using the nominal group technique. Military Medicine 2023; 188: e145–e151. pmid:34191032
- 49. Fisher RJ, Riley-Bennett F, Russell L, et al. Nominal group technique to establish the core components of home-based rehabilitation for survivors of stroke with severe disability. BMJ open 2021; 11: e052593. pmid:34857570
- 50. Janssens R, Lang T, Vallejo A, et al. Patient preferences for multiple myeloma treatments: a multinational qualitative study. Frontiers in medicine 2021; 8: 686165. pmid:34295912
- 51. Michel DE, Iqbal A, Faehrmann L, et al. Using an online nominal group technique to determine key implementation factors for COVID-19 vaccination programmes in community pharmacies. International journal of clinical pharmacy 2021; 43: 1705–1717. pmid:34633625
- 52. Occomore-Kent LC, Hatch E, Cruice M. Scoping opinion: Speech and language therapists’ views on extending their role to the urgent ear, nose and throat pathway. International Journal of Language & Communication Disorders 2021; 56: 975–988.
- 53. Østbø N, Jimenez EY, Harb S, et al. Nutrition Information Resources Used by People With Systemic Sclerosis and Perceived Advantages and Disadvantages: A Nominal Group Technique Study. ACR Open Rheumatology 2021; 3: 540–549. pmid:34196508
- 54. Wood L, Bishop A, Lewis M, et al. Treatment targets of exercise for persistent non-specific low back pain: a consensus study. Physiotherapy 2021; 112: 78–86. pmid:34029781
- 55. Lee RE, O’Neal A, Cameron C, et al. Developing content for the Food Environment Assessment Survey Tool (FEAST): A systematic mixed methods study with people with disabilities. International journal of environmental research and public health 2020; 17: 7781. pmid:33114296
- 56. Navarro-Millán I, Cornelius-Schecter A, O’Beirne RJ, et al. Views of primary care physicians and rheumatologists regarding screening and treatment of hyperlipidemia among patients with rheumatoid arthritis. BMC rheumatology 2020; 4: 1–7.
- 57. Owensby JK, Chen L, O’Beirne R, et al. Patient and rheumatologist perspectives regarding challenges to achieving optimal disease control in rheumatoid arthritis. Arthritis care & research 2020; 72: 933–941. pmid:31008566
- 58. O’Brien EC, Sugarman J, Weinfurt KP, et al. The impact of COVID-19 on pragmatic clinical trials: lessons learned from the NIH Health Care Systems Research Collaboratory. Trials 2022; 23: 1–9.
- 59. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS medicine 2011; 8: e1000393. pmid:21283604
- 60. Boulkedid R, Abdoul H, Loustau M, et al. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PloS one 2011; 6: e20476. pmid:21694759
- 61. Bailey SC, O’conor R, Bojarski EA, et al. Literacy disparities in patient access and health-related use of I nternet and mobile technologies. Health Expectations 2015; 18: 3079–3087. pmid:25363660
- 62. Davies L, LeClair KL, Bagley P, et al. Face-to-face compared with online collected accounts of health and illness experiences: a scoping review. Qualitative Health Research 2020; 30: 2092–2102. pmid:32667257
- 63. Rupert DJ, Poehlman JA, Hayes JJ, et al. Virtual versus in-person focus groups: Comparison of costs, recruitment, and participant logistics. Journal of Medical Internet Research 2017; 19: e80. pmid:28330832
- 64. Daft RL, Lengel RH. Organizational information requirements, media richness and structural design. Management science 1986; 32: 554–571.
- 65.
Dennis AR, Valacich JS. Rethinking media richness: Towards a theory of media synchronicity. In: Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers. IEEE, 1999, pp. 10-pp.
- 66. Lago PP, Beruvides MG, Jian J-Y, et al. Structuring group decision making in a web-based environment by using the nominal group technique. Computers & Industrial Engineering 2007; 52: 277–295.
- 67. Jenner BM, Myers KC. Intimacy, rapport, and exceptional disclosure: A comparison of in-person and mediated interview contexts. International Journal of Social Research Methodology 2019; 22: 165–177.
- 68. Bhandari S, Hallowell MR. Identifying and controlling biases in expert-opinion research: Guidelines for variations of Delphi, nominal group technique, and focus groups. Journal of Management in Engineering 2021; 37: 04021015.