https://orcid.org/0000-0001-7623-1639
Figures
Abstract
Purpose
This is a study protocol to co-create with knowledge users a core outcome set focused on middle-aged and older adults (40 years+) for use in social prescribing research.
Methods
We will follow the Core Outcome Measures in Effectiveness Trials (COMET) guide and use modified Delphi methods, including collating outcomes reported in social prescribing publications, online surveys, and discussion with our team to finalize the core outcome set. We intentionally center this work on people who deliver and receive social prescribing and include methods to evaluate collaboration. Our three-part process includes: (1) identifying published systematic reviews on social prescribing for adults to extract reported outcomes; and (2) up to three rounds of online surveys to rate the importance of outcomes for social prescribing. For this part, we will invite people (n = 240) who represent the population experienced in social prescribing, including researchers, members of social prescribing organizations, and people who receive social prescribing and their caregivers. Finally, we will (3) convene a virtual team meeting to discuss and rank the findings and finalize the core outcome set and our knowledge mobilization plan.
Conclusion
To our knowledge, this is the first study designed to use a modified Delphi method to co-create core outcomes for social prescribing. Development of a core outcome set contributes to improved knowledge synthesis via consistency in measures and terminology. We aim to develop guidance for future research, and specifically on the use of core outcomes for social prescribing at the person/patient, provider, program, and societal-level.
Citation: Esfandiari E, Chudyk AM, Grover S, Lau EY, Hoppmann C, Mortenson WB, et al. (2023) Social Prescribing Outcomes for Trials (SPOT): Protocol for a modified Delphi study on core outcomes. PLoS ONE 18(5): e0285182. https://doi.org/10.1371/journal.pone.0285182
Editor: Vanessa Carels, PLoS ONE, UNITED STATES
Received: April 7, 2023; Accepted: April 18, 2023; Published: May 16, 2023
Copyright: © 2023 Esfandiari et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: No datasets were generated or analyzed during the current study.
Funding: The University of British Columbia Health Innovation Funding Investment (HIFI) Award and Social Sciences and Humanities Research Council (SSHRC). The funders did not and will not have a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Social prescribing is a model of health and social care which focuses on aspects of people’s unmet social needs [1]. The approach connects people with public and non-profit or volunteer community resources, such as physical activity, arts-based, or housing resources [2]. It aims to address, for example, loneliness or management of long-term health conditions, like diabetes [3]. This is an especially salient issue because of the COVID-19 pandemic and subsequent social isolation, which may have undermined the health and well-being of people across the life span [4].
For too long, health has been defined as the absence of disease and is less aligned with the World Health Organization’s definition as “a state of complete physical, mental, and social well-being, and not merely the absence of disease…” [5], or an updated version proposed as “structural, functional and emotional state that is compatible with effective life as an individual and as a member of society.” [6]. In the past several decades, a new health and social model of care, social prescribing, has emerged. The United Kingdom (UK)’s public healthcare system is leading the way with social prescribing with an objective to refocus the healthcare system on non-medical concerns [1]. While globally there is no accepted definition for social prescribing, a recent pre-print used a Delphi method to report social prescribing as “a means for trusted individuals in clinical and community settings to identify that a person has non-medical, health-related social needs and to subsequently connect them to non-clinical supports and services within the community by co-producing a social prescription…” [7].
There are a number of systematic reviews on social prescribing and its effect on health and well-being [2, 8–16]. Although results to date are inconclusive, there are some promising findings. However, there are several considerations for future research. For example, our systematic review of social prescribing for older adults [2], noted considerable heterogeneity in outcomes used among studies. Second, synthesizing evidence on social prescribing requires knowledge on outcomes at multiple levels of person, provider, and system. Systematically identifying and synthesizing patient (person)-important outcomes [17] for social prescribing is essential for research and practice. In particular, a focus on “patient-important” outcomes helps to better understand people’s preferences and values [18] and improve patient relevance by contributing to a consistent understanding [17]. Furthermore, evidence lacks sufficient detail about social prescribing implementation to be assessed for their effectiveness and value [8]. Trials sometimes need system-level considerations to explore factors that go beyond the person-level, for example, effective access, efficient resource utilization [19], and implementation outcomes at the system level [8]. These knowledge gaps (and possibly others) make it difficult to synthesize evidence on social prescribing outcomes at multiple levels (person, provider, system).
The Core Outcome Measures in Effectiveness Trials (COMET) is a research initiative to guide the identification and selection of measures [20] defined as “an agreed standardized collection of outcomes … which should be measured and reported in all trials for a specific clinical area” (p.1) [21]. Sets of core outcomes have been developed for several health conditions [22–27], and may reduce outcome reporting bias, enhance comparability and enable syntheses of results across different studies [21]. Core outcome sets can also provide a list of meaningful measures across relevant groups, such as patients/people, caregivers, healthcare providers, and/or policy makers, who affect or are affected by an intervention [28], which ultimately could improve care decisions [29, 30]. There are several approaches or methods used in the development of core outcome sets, such as conducting systematic reviews (to identify outcomes used in practice or research), and to reach consensus on core outcome measures, using methods such as the Delphi method [20, 31].
The Delphi method is frequently used in healthcare settings to reach consensus for decision-making [32–37]. It usually involves an iterative process with systematic rounds of (online) rating/ranking (of questions/concepts) to reach group agreement, and it is especially useful when there are few data available [34], which is the case for social prescribing. It is likely multiple outcomes will not reach agreement [23], therefore a modified version of the Delphi method may be more effective in the synthesis process [34, 38, 39]. Many modified approaches request participants first respond to survey questions (individually), followed by a group discussion to finalize the results [39]. Further, to increase the generalizability of the core outcomes set, it has been recommended to involve a variety of knowledge users in the surveys and group discussion [40]. Therefore, a modified Delphi method will provide opportunities to resolve potential disagreements between different knowledge users and ensure that the population with experience in social prescribing are represented in the process [23].
It is rare in practice for knowledge users, such as patients, to contribute early to the development of core outcomes sets [40]. Incorporating different knowledge users’ perspectives (especially patients) on outcomes is important for health care resource utilization [29, 41]. Patients as research team members are not “just patients” in the traditional sense [42]. In addition to their health conditions and needs, they have a wide variety of backgrounds, skills, and interests to be incorporated through an intersectional approach that acknowledges the relationship between different social locations and experiences [42, 43]. Participatory action research is a collaborative approach to engaging knowledge users, who can contribute to the resolution of a problem and benefit from the solution in a “real-world setting” [44, 45]. Core outcome set development could be strengthened with a participatory approach to support identification and consensus building for important outcomes, and in particular, for patient-important outcomes.
To advance the science and practice of social prescribing, a logical next step is to co-create a core outcome set. To our knowledge, there has not been a core outcome set developed for social prescribing, but there are two projects on core outcomes in social care [46, 47]. Specifically, there is one report of an umbrella review (review of reviews) on integrated health and social care for people living with chronic health conditions [46]; and the second project is the development of core outcomes for people receiving social care, related to the delivery of specific services such as home care, day centres, equipment, or home adaptations [47]. Our objective is to develop core outcomes for adults aged 40 years and older, when the perception of aging starts [48, 49], using a three-part modified Delphi method. We anticipate the core outcome set could be used in planning and testing future social prescribing research studies [32–34, 38, 39]. We further aim to extend Delphi methods by using a participatory approach [44, 45] and collaborate with a diverse group of people before (in team composition), during (recruiting participants) and after (within our knowledge mobilization strategies) the research study [50]. Therefore, the purpose of this work is to report on the protocol that will be used to guide our work of co-creating a core outcome set with knowledge users focused on middle-aged and older adults (40 years+) for use in social prescribing research.
Materials and methods
We will use a three-part modified Delphi method [32–34, 38, 39, 51] to identify social prescribing core outcomes for middle-aged and older adults (40 years and older) at the person, provider, program, and societal-levels.
We used the Conducting and REporting DElphi Studies (CREDES) guideline to design this study [52]. Operationally, we will use methods based on the COMET Handbook [53], including, (1) reviewing evidence to identify types of outcomes for social prescribing (via a review of the published literature); (2) collecting feedback from online surveys to rate outcomes; and (3) holding a virtual team meeting to finalize the core outcome set. We modified the Delphi method by prioritizing the perspectives of middle-aged and older adults (as team members and survey participants) and using a virtual team meeting to finalize the outcomes within our large interdisciplinary team. Fig 1 is an overview of this study protocol and Fig 2 is an outline of the steps used within the Delphi method.
The figure outlines the steps within the process (the “what”) and people involved with each step (“the who”).
We obtained ethics approval from the Behavioral Research Ethics Board at The University of British Columbia (H22-03569); and we registered the protocol with the COMET database (https://www.comet-initiative.org/Studies/Details/2364).
Composition and role of research team
The research team comprises people from some of the groups summarized in Fig 3, including people who receive social prescribing (or their caregivers), trainees, health and social providers, community-based organizations, and researchers. The International Association for Public Participation (IAP2)’s Public Participation Spectrum [54] guides how we conceptualize our approach to engaging diverse members of our interdisciplinary research team. Members will be invited to take part in all components of this study and will “collaborate” in decision making as defined by the IAP2 Spectrum [e.g., “partner with the public in each aspect of the decision including the development of alternatives and the identification of the preferred solution.” [54]]. Our engagement activities will include regular virtual team meetings aimed at planning and discussing the study’s progress across its research cycle and contributing to the development of study-related materials (e.g., patient-facing materials, knowledge mobilization products). An operations team (composed of trainees, and the first, second, and last author) formed from the larger group will closely monitor the study progress and evaluate its methodological aspects for consistency and rigor (e.g., via field notes and regular check-ins). An operations team member will also be appointed as a liaison to the knowledge users team members to ensure that they have a point person with any concerns and to uphold the integrity of the engagement process. We will establish and follow a terms of reference document co-developed by the team at the study’s outset [55]. Team members will participate in the larger online Delphi surveys (described below) but will be asked to self-identify as a team member within the pre-survey questions, so we can conduct a sensitivity analysis. We will present results separately for the research team and the larger group of invited participants. Finally, to ensure team members feel they can collaborate (as outlined by the IAP2 Spectrum) we will use a questionnaire, the Public and Patient Engagement Evaluation Tool [56], twice over the consensus study’s duration (mid-point and final) to learn from and evaluate our engagement process. We will also regularly check in with all team members to ensure they feel they can collaborate (as outlined by the IAP2 Spectrum).
In this work, a link worker or community connector is a role held by people who support people to (re)engage with community resources or other opportunities based on identified needs.
Part 1: Identifying types of outcomes used for social prescribing
We will identify potential outcomes from systematically reviewing the existing literature and feedback from our research team [23]. We recently published two systematic reviews on social prescribing and older adults [2, 57] and will review outcomes identified in studies within these syntheses. We will also identify other published systematic reviews of social prescribing studies via searching the following databases: MEDLINE and Embase (Ovid), EBSCO (APA Psych Articles, APA PsycInfo, CINAHL Complete, Social Work Abstracts, SPORTDiscus), Cochrane Database of Systematic Reviews, and Epistemonikos. One author (MCA) will use keywords “social prescribing” OR “social prescription” and filter the search with “review” to develop the search strategy. We will include reviews focussed on adults or older adults (without a specific health condition). We will include publications from all languages (using Google Translate, when able, to read the publication in English) and years. Two team members (EE and MCA) will independently review the titles, abstract and full texts to identify the reviews. Once systematic reviews are identified, both team members will independently review outcomes measured across studies. They will also extract the following information: author, publication year, systematic review question(s), population, setting, and outcome measures (instruments, such as the name of the questionnaire or performance-based measure).
For each outcome identified, we will categorized them into one of the six categories defined according to the taxonomy developed for outcomes in medical research [58]: (1) mortality/survival; (2) physiological/clinical; (3) functioning; (4) delivery of care; (5) resource use; and (6) adverse events/effects [58]. This approach differs from the core outcomes for social care previously noted, where the study categorized outcomes based on what was identified in the literature [47]. The two team members who extract the outcomes will categorize them using the taxonomy, and this list will be reviewed by two different team members (first independently then in a virtual discussion) to decide on the proposed synthesized list of outcomes. During the categorization process, it is also possible additional categories may emerge; or we need to reconsider the categorization of outcomes. In this case, two team members will discuss any items and present their suggestions to the team.
Part 2. Online surveys to rate outcomes
Survey Development and Piloting: We will use DelphiManager (Software, DelphiManager, Core Outcome Measure for Effectiveness Trials Initiatives, Liverpool, UK) to design and administer survey rounds. We will provide participants access to the DelphiManager via email and assign each participant a unique number. For each survey question (on types of outcomes), we will ask participants to rate the importance of each outcome to social prescribing on a scale from one (which indicates “not important”) to nine (which indicates “essential”). Two authors will draft the first surveys and send them to all team members who will review and provide feedback in advance of starting the survey process.
Study participants.
Selection of the participants is essential for conducting a robust Delphi study [51]. Participants should represent and/or have experience with the topic [20]; here, participants would be people "affected by or who affect change” within social prescribing [28]. In particular, for people receiving social prescribing we are focused on adults 40 years and older, when the perception of aging starts [48, 49], and their caregivers. Available evidence highlights variability in the size of the number of participants (i.e., the panel) which can range between four and 3,000 people [34]. The recommended number for panel membership is 10 to 18 people [52] to facilitate the consensus-building process [52]. Thus, we will engage a larger group of participants to brainstorm and provide feedback on core outcomes within the surveys. However, our team will be the panel who reviews and finalises the core outcome set.
Acceptance rates to participate in a Delphi study varies from about 25% [20] to 75% [59]. Although Junger and colleagues suggested a panel size of 10–18 was beneficial for group dynamics, we used this guidance to estimate a sample size for subgroups of participants who will brainstorm and rate outcomes for social prescribing. Specifically, we propose 12 subgroups of participants based on our previous framework for older adults’ community mobility [60] and listed in Fig 3. We will also include data scientists and ethicists as study participants to consider the role of data integration within social prescribing, and the importance of ethics in digital health [61], a possible delivery mode. Therefore, we aim to enroll up to 10 people for each of the 12 sub-groups of people (n = 120) [52]. However, we recognize we may not be able to enrol 10 data scientists or ethicists. Conservatively assuming a 50% rate of enrollment [20] we will invite 240 people to contribute to the Delphi process.
For our recruitment strategies, we will create a list of people with social prescribing experience (research, practice, or membership in a social prescribing organizations) through a number of methods. For example, we will (i) search related publications located in the systematic reviews identified in Part 1; (ii) approach social prescribing organizations to send out information on the project to their membership; (iii) identify providers experienced with social prescribing in Canada (via websites, related organisations, and publications); and (iv) via adults/older adults and/or caregivers who receive social prescribing in Canada [via posters and informational presentations (community engagement) at social prescribing sites in Canada]. Once identified, we will contact people via email and invite them to take part in the study. We will inform potential participants about the aim of the study, methods, confidentiality, and the time needed to complete surveys. We will limit information provided to potential participants, to reduce the risk of creating (unintentional) bias.
Strategies to engage participants.
To improve the response rate, we will use different strategies, including: (i) providing a clear explanation (and/or creating infographics or a video) of the study process, timeline, and commitment throughout the process (e.g., a priori communication plan); (ii) being responsive to participants’ emails and feedback, and in particular, sending study details in a timely manner (within 2–3 business days); (iii) sending an email reminder at identified times if a response is not received at the anticipated date; and (iv) as mentioned previously, we will provide in-person presentations on social prescribing and the study (if possible). We will not provide an honorarium to participants.
Delphi survey rounds.
Round 1: We will use an online (up to) three round Delphi method to brainstorm and rate the importance of outcomes in social prescribing. Participants will be asked to complete each survey round within two weeks. In the first-round, participants will ask to complete some background information for the team to better understand their perspectives. We will also ask them to self-select (if possible) from the list of groups provided in Fig 3. We will not ask participants to provide any identifying information. The list of outcomes will be presented randomly to participants within surveys order to reduce the risk of bias [62]. Participants will also have the opportunity to suggest additional items. Finally, there will be space for participants’ written feedback about the selection of rating scores.
We will summarize the number of respondents and scores for each outcome using frequencies and means. We will also calculate the frequency of participants scoring one to three, four to six, and seven to nine [23]. We will review data in text boxes to identify possible additional outcomes to include in the subsequent round; and to understand why people made their rating choices.
Round 2: We will invite the participants from the first round to take part in the second-round survey. We will present the first-round findings to participants including a summary of their responses and average scores for each question. We will ask participants to rate each outcome (listed in a random order), where no consensus was reached during the first round, within a two-week time period. We will summarize the number of respondents and scores for each outcome using frequencies and means. We will also calculate the frequency of participants scoring one to three, four to six, and seven to nine [23]. We will analyze findings between groups to clarify perspectives and responses.
Round 3: A third and final round survey may be sent to rate outcomes.
Guidance for developing agreement (consensus).
During the multiple rounds of surveys, we will categorize outcomes into one of three groups. We will reach consensus to include an outcome when at least 75% of participants rate an outcome between seven and nine (i.e., the outcome is “essential”) and fewer than 20% of participants rate the outcome as “not important” (rated between one and three). Similarly, we will reach consensus to exclude an outcome if more than 75% of participants indicate the outcome is “not important” and fewer than 20% of participants rate the outcome essential. If either of these two conditions are not met, consensus will not be met, and participants will be asked to review and possibly revise their ratings in light of the group scores.
Part 3: Virtual team meeting to finalize the core outcome set
Following the identification of outcomes and survey parts of this study, we will hold a virtual meeting to permit the research team representing selected knowledge user groups (based on Fig 3) to finalize and rank the core set of outcome measures. The meeting is a key element of the modified Delphi method: it will permit the research team to provide further clarification and justify their viewpoints [63]. We will send the team a summary of the results prior to the meeting.
The operations team will start the meeting by providing a summary of the process and findings. The team will also discuss any issues or “non-consensus” items which rise arose within the Delphi process. We will discuss all “excluded” and “no consensus” outcomes from the Delphi process to ensure key outcomes have not been excluded from the set for policy decisions and intervention adoption (like cost-effectiveness outcomes) or which may be important to patients and caregivers. We will also review responses from subgroups to make sure there is consistency across knowledge user groups.
External validation of core outcome set
It has been suggested an external panel or board review the results from the core outcome set to support the validity of findings. After finalizing the core set of outcomes, we will request posting the synthesized data on social prescribing related websites for a two-week period to seek feedback. We will also hold presentations in the community to seek feedback on the study and its findings.
Knowledge mobilization
Our team includes a range of people and perspectives, and we adopt an integrated and participatory approach from the beginning. For our end of project knowledge mobilization, we will use a variety of strategies to engage different audiences, as guided by our interdisciplinary research team. We will publish a manuscript and submit conference abstracts. We will create a presentation for disseminating information and community engagement. We will also use this current synthesis process to inform our next research phase–to review the specific instruments (and their psychometric properties) used with social prescribing.
Discussion
Social prescribing is becoming more prevalent within the literature and (publicly funded) health care systems. However, despite its potential to have a positive impact at multiple levels, there remain opportunities to integrate robust research approaches within its delivery and evaluation. Our previous evidence syntheses [2, 57] underscored the need to identify important and relevant outcome measures for social prescribing. Therefore, we will address this knowledge gap by using robust Delphi methods [53] to engage a large diverse group of people and organizations to co-create a core outcome set for social prescribing. This study will advance the field by taking a comprehensive and participatory approach to include perspectives from a wide range of knowledge users. We anticipate the findings of this study will provide researchers and knowledge users with outcomes at the person, provider, program, and societal-level for future studies.
Potential limitations and strength of the proposed study
The main limitation with this work is the methodology. Although the Delphi method has many strengths, findings are based on consensus and reflect the perspectives of the people who contributed [64]. However, following the guidance from CREDES [52] we have included several steps to strengthen our work. For example, we registered the protocol with COMET; provided a detailed description of our methods (including additional components beyond the traditional Delphi method) in this protocol; defined the level of agreement required before starting the consensus process; and we will report the findings within the context of the methods (i.e., results are based on perspectives, report on topics which did not receive group consensus) [64].
We seek to be inclusive within team composition and invited participants. However, we recognize it is often difficult to recruit participants for research studies in general. As a result, findings will be guided by who takes part in the consensus process. However, we will also ask team members to provide feedback on how to identify people from hard-to-reach populations and how outcomes may vary by factors such as age, ethnicity, culture, gender, gender identity and expression, and sexual orientation; living in low resource communities; from small urban or rural settings; living with disabilities, and other factors which contribute to a person’s identity.
We will make every effort to recruit a consistent number of participants for each subgroup. However, based on our initial search of social prescribing researchers and organizations, we anticipate some challenges identifying data scientists and ethicists with social prescribing experience/knowledge. If needed, we will expand the inclusion criteria for underrepresented groups to participants with experience in middle-age/older adult health and social care.
We will review responses from subgroups to make sure there is consistency across knowledge user groups. The results may be affected by heterogeneity within and between participant groups. However, we intentionally adopted a modified Delphi method and will review the findings at the virtual meeting. Specifically, the team will discuss which items were excluded, and the distribution of data for subgroups and overall.
Further, the use of an online survey will limit the participation to people who are familiar with, and have access to, a computer and the internet. In addition, we will only include adults/older adults and/or caregivers who receive social prescribing in Canada. Although we will aim to recruit participants from distinct parts of Canada, the importance of outcomes may vary between locations and across settings.
Conclusion
We provide a protocol for a comprehensive and participatory approach for the development of core outcomes for social prescribing. A key strength of this work is the inclusion of the focus on the perspectives of “people who receive” social prescribing and begin to shift the dynamic which often place people on the “receiving end” of care. As with most evidence synthesis, this work will be a living document, and findings will change as social prescribing evolves and develops for different settings and populations. However, our goal is to create a conversation and a foundation to ensure what is measured responds to the needs of people, communities, and society.
References
- 1.
Social prescribing: applying all our health [Internet]. 2022 [cited April 5, 2023]. Available from: https://www.gov.uk/government/publications/social-prescribing-applying-all-our-health/social-prescribing-applying-all-our-health.
- 2. Percival A, Newton C, Mulligan K, Petrella R, Ashe M. Systematic Review of Social Prescribing and Older Adults: Where To From Here? Family Medicine and Community Health. 2022;10(Suppl 1). pmid:36207017; PubMed Central PMCID: PMC9557282.
- 3. Calderón-Larrañaga S, Greenhalgh T, Clinch M, Robson J, Dostal I, Eto F, et al. Unravelling the potential of social prescribing in individual-level Type 2 Diabetes prevention: A mixed-methods realist evaluation. BMC Med. 2023;21(1):91. Epub 20230313. pmid:36907857; PubMed Central PMCID: PMC10008720.
- 4. Hwang T-J, Rabheru K, Peisah C, Reichman W, Ikeda M. Loneliness and social isolation during the COVID-19 pandemic. Int Psychogeriatr. 2020;32(10):1217–20. Epub 2020/05/26. pmid:32450943; PubMed Central PMCID: PMC7306546.
- 5. Grad FP. The preamble of the constitution of the World Health Organization. Bull World Health Organ. 2002;80:981–. pmid:12571728
- 6. McCartney G, Popham F, McMaster R, Cumbers A. Defining health and health inequalities. Public Health. 2019;172:22–30. pmid:31154234
- 7. Muhl C, Mulligan K, Bayoumi I, Ashcroft R, Godfrey C. Establishing Internationally Accepted Conceptual and Operational Definitions of Social Prescribing Through Expert Consensus: A Delphi Study. 2022.
- 8. Bickerdike L, Booth A, Wilson PM, Farley K, Wright K. Social prescribing: less rhetoric and more reality. A systematic review of the evidence. BMJ Open. 2017;7(4):e013384. Epub 20170407. pmid:28389486; PubMed Central PMCID: PMC5558801.
- 9. Cooper M, Avery L, Scott J, Ashley K, Jordan C, Errington L, et al. Effectiveness and active ingredients of social prescribing interventions targeting mental health: a systematic review. BMJ Open. 2022;12(7):e060214. Epub 20220725. pmid:35879011; PubMed Central PMCID: PMC9328101.
- 10. Kiely B, Croke A, O’Shea M, Boland F, O’Shea E, Connolly D, et al. Effect of social prescribing link workers on health outcomes and costs for adults in primary care and community settings: a systematic review. BMJ Open. 2022;12(10):e062951. Epub 20221017. pmid:36253037; PubMed Central PMCID: PMC9644316.
- 11. Liebmann M, Pitman A, Hsueh YC, Bertotti M, Pearce E. Do people perceive benefits in the use of social prescribing to address loneliness and/or social isolation? A qualitative meta-synthesis of the literature. BMC Health Serv Res. 2022;22(1):1264. pmid:36261835
- 12. Napierala H, Krüger K, Kuschick D, Heintze C, Herrmann WJ, Holzinger F. Social Prescribing: Systematic Review of the Effectiveness of Psychosocial Community Referral Interventions in Primary Care. International Journal of Integrated Care. 2022;22(3):11. Epub 20220819. pmid:36060831; PubMed Central PMCID: PMC9389950.
- 13. Pescheny JV, Randhawa G, Pappas Y. The impact of social prescribing services on service users: a systematic review of the evidence. Eur J Public Health. 2020;30(4):664–73. pmid:31199436.
- 14. Reinhardt GY, Vidovic D, Hammerton C. Understanding loneliness: a systematic review of the impact of social prescribing initiatives on loneliness. Perspectives in Public Health. 2021;141(4):204–13. Epub 20210623. pmid:34159848; PubMed Central PMCID: PMC8295963.
- 15. Vidovic D, Reinhardt GY, Hammerton C. Can Social Prescribing Foster Individual and Community Well-Being? A Systematic Review of the Evidence. Int J Environ Res Public Health. 2021;18(10). Epub 20210515. pmid:34063543; PubMed Central PMCID: PMC8156788.
- 16. Zhang CX, Wurie F, Browne A, Haworth S, Burns R, Aldridge R, et al. Social prescribing for migrants in the United Kingdom: A systematic review and call for evidence. Journal of Migration and Health. 2021;4:100067. Epub 20211009. pmid:34746902; PubMed Central PMCID: PMC8556515.
- 17. Kersting C, Kneer M, Barzel A. Patient-relevant outcomes: what are we talking about? A scoping review to improve conceptual clarity. BMC Health Serv Res. 2020;20(1):596. pmid:32600321
- 18. Guyatt G, Montori V, Devereaux P, Schunemann H, Bhandari M. Patients at the center: In our practice, and in our use of language. ACP J Club. 2004;140(1):A11. pmid:14711297
- 19.
Roberts M, Hsiao W, Berman P, Reich M. Getting Health Reform Right: A Guide to Improving Performance and Equity: Oxford University Press; 2008 01 Sep 2009.
- 20. Prinsen CA, Vohra S, Rose MR, King-Jones S, Ishaque S, Bhaloo Z, et al. Core Outcome Measures in Effectiveness Trials (COMET) initiative: protocol for an international Delphi study to achieve consensus on how to select outcome measurement instruments for outcomes included in a ‘core outcome set’. Trials. 2014;15(1):1–7.
- 21. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(1):132. Epub 20120806. pmid:22867278; PubMed Central PMCID: PMC3472231.
- 22. Wallace SJ, Sullivan B, Rose TA, Worrall L, Dorze GL, Shrubsole K. Core Outcome Set Use in Poststroke Aphasia Treatment Research: Examining Barriers and Facilitators to Implementation Using the Theoretical Domains Framework. J Speech Lang Hear Res. 2021;64(10):3969–82. Epub 20210907. pmid:34491769.
- 23. Taylor J, Bohnke JR, Wright J, Kellar I, Alderson SL, Hughes T, et al. A core outcome set for evaluating self-management interventions in people with comorbid diabetes and severe mental illness: study protocol for a modified Delphi study and systematic review. Trials. 2017;18(1):70. Epub 20170214. pmid:28196512; PubMed Central PMCID: PMC5309993.
- 24. Moza A, Benstoem C, Autschbach R, Stoppe C, Goetzenich A. A core outcome set for all types of cardiac surgery effectiveness trials: a study protocol for an international eDelphi survey to achieve consensus on what to measure and the subsequent selection of measurement instruments. Trials. 2015;16:545. Epub 20151202. pmid:26625730; PubMed Central PMCID: PMC4667508.
- 25. Harding AJE, Morbey H, Ahmed F, Opdebeeck C, Wang YY, Williamson P, et al. Developing a core outcome set for people living with dementia at home in their neighbourhoods and communities: study protocol for use in the evaluation of non-pharmacological community-based health and social care interventions. Trials. 2018;19(1):247. Epub 20180424. pmid:29690920; PubMed Central PMCID: PMC5916721.
- 26. Ambler GK, Brookes-Howell L, Jones JAR, Verma N, Bosanquet DC, Thomas-Jones E, et al. Development of Core Outcome Sets for People Undergoing Major Lower Limb Amputation for Complications of Peripheral Vascular Disease. Eur J Vasc Endovasc Surg. 2020;60(5):730–8. Epub 2020/08/17. pmid:32798206.
- 27. Harman NL, Wilding JPH, Curry D, Harris J, Logue J, Pemberton RJ, et al. Selecting Core Outcomes for Randomised Effectiveness trials In Type 2 diabetes (SCORE-IT): a patient and healthcare professional consensus on a core outcome set for type 2 diabetes. BMJ Open Diabetes & Research Care. 2019;7(1):e000700. Epub 20191229. pmid:31908789; PubMed Central PMCID: PMC6936506.
- 28.
Freeman RE. Strategic management: A stakeholder approach. Cambridge Core EBA eBooks Complete Collection, Cambridge Core All Books, editors: Pitman, Indiana University; 1984 February 25, 2023. 276 p.
- 29. Kirkham JJ, Williamson P. Core outcome sets in medical research. BMJ Medicine. 2022;1(1):e000284. Epub 20221017. pmid:36936568; PubMed Central PMCID: PMC9951367.
- 30. Kearney A, Gargon E, Mitchell JW, Callaghan S, Yameen F, Williamson PR, et al. A systematic review of studies reporting the development of Core Outcome Sets for use in routine care. J Clin Epidemiol. 2023. Epub 20230320. pmid:36948407.
- 31. Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, et al. Choosing Important Health Outcomes for Comparative Effectiveness Research: A Systematic Review. PLoS One. 2014;9(6):e99111. Epub 20140616. pmid:24932522; PubMed Central PMCID: PMC4059640.
- 32. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Management Science. 1963;9(3):458–67.
- 33. Dalkey N. An experimental study of group opinion: The Delphi method. Futures. 1969;1(5):408–26. https://doi.org/10.1016/S0016-3287(69)80025-X.
- 34. Cantrill J, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;4(2):67–74.
- 35. Custer RL, Scarcella JA, Stewart BR. The Modified Delphi Technique—A Rotational Modification. Journal of Career and Technical Education. 1999;15(2).
- 36. Murphy M, Black N, Lamping D, McKee C, Sanderson C, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technology Assessment (Winchester, England). 1998;2(3):i–88. pmid:9561895
- 37. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376–82. pmid:12581103.
- 38. Graefe A, Armstrong JS. Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. International Journal of Forecasting. 2011;27(1):183–95.
- 39. Gustafson DH, Shukla RK, Delbecq A, Walster GW. A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance. 1973;9(2):280–91.
- 40. Chevance A, Tran V-T, Ravaud P. Controversy and Debate Series on Core Outcome Sets. Paper 1: Improving the generalizability and credibility of core outcome sets (COS) by a large and international participation of diverse stakeholders. J Clin Epidemiol. 2020;125:206-12.e1. Epub 20200113. pmid:31945480.
- 41. Perfetto EM, Oehrlein EM, Love TR, Schoch S, Kennedy A, Bright J. Patient-centered Core Impact Sets: What they are and why we need them. The Patient-Patient-Centered Outcomes Research. 2022:1–9.
- 42. Chudyk AM, Horrill T, Waldman C, Demczuk L, Shimmin C, Stoddard R, et al. Scoping review of models and frameworks of patient engagement in health services research. BMJ Open. 2022;12(8):e063507. Epub 20220819. pmid:35985787; PubMed Central PMCID: PMC9396146.
- 43. Roche P, Shimmin C, Hickes S, Khan M, Sherzoi O, Wicklund E, et al. Valuing all voices: Refining a trauma-informed, intersectional and critical reflexive framework for patient engagement in health research using a qualitative descriptive approach. Research involvement and engagement. 2020;6(1):1–13. pmid:32699647
- 44. Leask CF, Sandlund M, Skelton DA, Altenburg TM, Cardon G, Chinapaw MJM, et al. Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Research Involvement and Engagement. 2019;5:2. Epub 20190109. pmid:30652027; PubMed Central PMCID: PMC6327557.
- 45. Tossavainen PJ. Co-create with stakeholders: Action research approach in service development. Action Research. 2017;15(3):276–93.
- 46. Kelly L, Harlock J, Peters M, Fitzpatrick R, Crocker H. Measures for the integration of health and social care services for long-term health conditions: a systematic review of reviews. BMC Health Serv Res. 2020;20(1):358. Epub 2020/04/28. pmid:32336288; PubMed Central PMCID: PMC7183623.
- 47. Sugavanam T, Caiels J, Peters M, Fitzpatrick R. The Development of a Core Outcome Set for Trials and Evaluative Studies in Adult Social Care. Journal of Long Term Care. 2021:438–45.
- 48. Dolberg P, Ayalon L. Subjective meanings and identification with middle age. International Journal of Aging and Human Development 2018;87(1):52–76. Epub 20170817. pmid:28817946.
- 49. Abrams D, Vauclair M, Swift HJ. Predictors of attitudes to age across Europe. 2011.
- 50. Chudyk AM, Stoddard R, McCleary N, Duhamel TA, Shimmin C, Hickes S, et al. Activities and impacts of patient engagement in CIHR SPOR funded research: A cross-sectional survey of academic researcher and patient partner experiences. Research Involvement and Engagement. 2022;8(1):44. Epub 20220829. pmid:36038887; PubMed Central PMCID: PMC9423700.
- 51. Veugelers R, Gaakeer MI, Patka P, Huijsman R. Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response. BMC Med Res Methodol. 2020;20(1):156. Epub 2020/06/17. pmid:32539717; PubMed Central PMCID: PMC7294633.
- 52. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. Epub 20170213. pmid:28190381.
- 53. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280. Epub 2017/07/07. pmid:28681707; PubMed Central PMCID: PMC5499094.
- 54.
International Association for Public Participation. Public Participation Spectrum [April 5, 2023]. Available from: https://iap2.org.au/resources/spectrum/.
- 55. Chudyk AM, Ragheb S, Kent D, Duhamel TA, Hyra C, Dave MG, et al. Patient Engagement in the Design of a Mobile Health App That Supports Enhanced Recovery Protocols for Cardiac Surgery: Development Study. JMIR Perioper Med. 2021;4(2):e26597. Epub 20211130. pmid:34851299; PubMed Central PMCID: PMC8672287.
- 56. Abelson J, Tripp L, Kandasamy S, Burrows K, Team PIS. Supporting the evaluation of public and patient engagement in health system organizations: Results from an implementation research study. Health Expect. 2019;22(5):1132–43. Epub 20190802. pmid:31373754; PubMed Central PMCID: PMC6803403.
- 57. Grover S, Sandhu P, Nijjar GS, Percival P, Chudyk AM, Liang J, et al. Older Adults and Social Prescribing Experience, Outcomes, and Processes A Meta-Aggregation Systematic Review. Public Health. 2023;In Press February 2023. pmid:37060740
- 58. Dodd S, Clarke M, Becker L, Mavergames C, Fish R, Williamson PR. A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. Journal of Clinical Epidemiology 2018;96:84–92. Epub 2017/12/31. pmid:29288712; PubMed Central PMCID: PMC5854263.
- 59. Hung HL, Altschuld JW, Lee YF. Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Eval Program Plann. 2008;31(2):191–8. Epub 20080304. pmid:18403018.
- 60. Schiller C, Winters M, Hanson HM, Ashe MC. A framework for stakeholder identification in concept mapping and health research: A novel process and its application to older adult mobility and the built environment. BMC Public Health. 2013;13:428. Epub 2013/05/04. pmid:23639179; PubMed Central PMCID: PMC3653754.
- 61. Rivera-Romero O, Gabarron E, Miron-Shatz T, Petersen C, Denecke K. Social Media, Digital Health Literacy, and Digital Ethics in the Light of Health Equity. IMIA Participatory Health and Social Media Working Group. 2022;31(01):082–7. pmid:35654433
- 62. Schmidt RC. Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences. 1997;28(3):763–74.
- 63. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476. Epub 20110609. pmid:21694759; PubMed Central PMCID: PMC3111406.
- 64. Blazey P, Crossley KM, Ardern CL, van Middelkoop M, Scott A, Khan KM. It is time for consensus on ’consensus statements’. Br J Sports Med. 2022;56(6):306–7. Epub 20210923. pmid:34556467; PubMed Central PMCID: PMC8899487.