Current research suggests that while patients are becoming more engaged across the health delivery spectrum, this involvement occurs most often at the pre-preparation stage to identify ‘high-level’ priorities in health ecosystem priority setting, and at the preparation phase for health research.
The purpose of this systematic rapid review of the literature is to describe the evidence that does exist in relation to patient and public engagement priority setting in both health ecosystem and health research.
Study eligibility criteria
i) published in English; ii) published within the timeframe of 2007—Current (10 years) unless the report/article was formative in synthesizing key considerations of patient engagement in health ecosystem and health research priority setting; iii) conducted in Canada, the US, Europe, UK, Australia/New Zealand, or Scandinavian countries.
Study appraisal and synthesis
i) Is the research valid, sound, and applicable?; ii) what outcomes can we potentially expect if we implement the findings from this research?; iii) will the target population (i.e., health researchers and practitioners) be able to use this research?. A summary of findings from each of the respective processes was synthesized to highlight key information that would support decision-making for researchers when determining the best priority setting process to apply for their specific patient-oriented research.
Seventy articles from the UK, US, Canada, Netherlands and Australia were selected for review. Results were organized into two tiers of public and patient engagement in prioritization: Tier 1—Deliberative and Tier 2—Consultative. Highly structured patient and public engagement planning activities include the James Lind Alliance Priority Setting Partnerships (UK), Dialogue Method (Netherlands), Global Evidence Mapping (Australia), and the Deep Inclusion Method/CHoosing All Together (US).
The critical study limitations include challenges in comprehensively identifying the patient engagement literature for review, bias in article selection due to the identified scope, missed information due to a more limited use of exhaustive search strategies (e.g., in-depth hand searching), and the heterogeneity of reported study findings.
The four public and patient engagement priority setting processes identified were successful in setting priorities that are inclusive and objectively based, specific to the priorities of stakeholders engaged in the process. The processes were robust, strategic and aimed to promote equity in patient voices. Key limitations identified a lack of evaluation data on the success and extent in which patients were engaged. Issues pertaining to feasibility of stakeholder engagement, coordination, communication and limited resources were also considered.
Citation: Manafò E, Petermann L, Vandall-Walker V, Mason-Lai P (2018) Patient and public engagement in priority setting: A systematic rapid review of the literature. PLoS ONE 13(3): e0193579. https://doi.org/10.1371/journal.pone.0193579
Editor: Jo Thompson Coon, University of Exeter, UNITED KINGDOM
Received: September 19, 2017; Accepted: February 14, 2018; Published: March 2, 2018
Copyright: © 2018 Manafò et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: AbSPORU gratefully acknowledges the funding received from the Canadian Institute for Health Research (CIHR) - Strategy for Patient-Oriented Research (SPOR) and Alberta Innovates.
Competing interests: The authors have declared that no competing interests exist.
In Canada, the Strategy for Patient-Oriented Research (SPOR) was initiated to foster evidence-informed health care by bringing innovative approaches to the point of care for greater quality, accountability and accessibility . The coalition of federal, provincial, and territorial partners is dedicated to integrating the patient voice into the research process to better ensure the patient voice and perspective is incorporated into policy and practice. Each SPOR SUPPORT (Support for People and Patient-Oriented Research and Trials) Unit offers locally accessible, multidisciplinary clusters of specialized research services, knowledge, and patient engagement to cultivate patient-oriented research and to facilitate a research culture change in response to the local needs and infrastructure gaps.
There is a critical opportunity to validate methodologies and frameworks for meaningful patient and public engagement in prioritization across the spectrum of research and decision-making activities. Given its relative infancy, there is a need to grow an evidentiary base about what works in achieving and sustaining productive patient engagement overall  and what does not . This evidence will ultimately help to evaluate whether meaningful public and patient engagement priority setting impacts enhanced patient and family-centered care, service delivery, and health outcomes. There is a growing consensus identified in the literature that consulting with the public and patients is the necessary link between decision-makers and potential knowledge users [4, 5]. Indeed, without such engagement from the earliest stages, researchers and clinicians may ultimately miss the needs deemed as high priority by the end users .
Much of the current literature focusses on patient engagement across the health care delivery spectrum , although there is increased interest in implementing engagement opportunities in planning activities. Evidence suggests involvement in planning occurs most often at the ‘pre-preparation’ stage for identifying ‘high-level’ priorities in the health ecosystem (i.e., decision-making about health systems agendae and strategic planning opportunities)  and at the ‘preparation’ phase for research (i.e., identifying research topics, prioritizing topics, and developing/refining research questions) . However, as evidenced by a recent scoping review conducted by this paper’s authors about patient engagement in health research, the literature is lacking in the provision of practical guidance on how to engage with the public and patients to obtain their perspectives. Based on these findings, it can also be inferred that a limited understanding of the best practices that focus on engaging patients and the public at the earlier stages of health ecosystem and health research priority setting also exists.
As such, members of the Patient Engagement Platform of the Alberta SPOR SUPPORT Unit (AbSPORU) conducted a rapid systematic review to describe the existing evidence about engaging patients and the public in the ‘pre-preparation’ and ‘preparation’ phases of health ecosystem and research priority setting. In this report the methods, findings, summary of key learnings, and suggested opportunities to support patient engagement in health planning and research are described. In particular, based on the findings, we will describe how patient engagement is used to promote collaboration and decision-making among patients, carers, researchers, health practitioners and decision-makers. The emphasis is on utilizing appropriate tools based on purpose, scope, and capacity particularly at the highest level of patient engagement, known as ‘deliberative engagement’ [9, 10].
Definitions in public and patient engagement in prioritization
Similar to the findings of the previous scoping review led by the AbSPORU PE Platform  the language used to describe the spectrum of engagement across priority-setting activities is inconsistent. For example, Amba et al.  suggested that while ‘agenda setting’ and ‘priority setting’ are used interchangeably, they refer to different models and strategies to engage patients, carers, and clinicians in health research and beyond. Therefore, for the purpose of this review, the following definitions were operationalized and used in structuring the review:
1. Health ecosystem priority setting
Engaging the public and patients in priority setting in the health ecosystem was initially framed by principles of Participatory Action Research (PAR) which is defined by the engagement of public and patients in visioning or goal setting exercises . Priority setting in the health ecosystem is framed by Khodyakov’s  definition of patients who are engaged in decision-making concerned with the planning and designing of programs. This implies patients being engaged in macro- or meso-level decisions more so than in clinical decision-making .
2. Research priority setting
Priority setting in health research is specifically guided by the Patient Outcome Centered Research Initiative’s (PCORI) definition of engaging patients in topic solicitation, prioritization, and framing of the research question [8, 15]. Specifically, this area of engagement focusses on what knowledge is (or questions are) valued most by patients and the public as they become experts in their health care experiences. They should have a say in determining research priorities and informing clinical decision-making .
At the same time, it is critical to note that despite the articulated definitions, the findings suggested an overlapping of characteristics for both priority setting activities i.e., health ecosystem and research, making a separate analysis of the results ineffective, limiting meaningful application to the end user. As such, the authors chose to organize the results using the distinguishing characteristics of engagement activities, which are the types and levels of engagement. These range from highly structured approaches with a high level of engagement to one-off consultations that are more limited in the opportunity for meaningful engagement. To operationalize the degree of public and patient engagement priority setting within the scope of this paper, Rowe and Frewer’s public engagement framework  was adapted to outline the three levels of public and patient engagement priority setting from least to most interactive . The levels of engagement are integrated with Khodyakov’s  and Szelest’s  definitions of roles for patients in ‘health ecosystem priority setting’ and ‘health research priority setting’ respectively. This implies patients being engaged in decisions that inform direct clinical decision-making more so at the micro-level.
As noted by the International Association for Public Participation (IAP2) , engagement is broadly defined and can therefore encompass distinct levels of interactivity. The IAP2 Spectrum of Engagement (2014) clearly outlines five levels of engagement, which denote degrees of public participation. The spectrum suggests that the greater the degree of participation, the more influence the community has in its decision-making Vandall-Walker’s  Levels of Patient and Researcher Engagement in Health Research, adapted from the IAP2 spectrum, involves six levels of engagement, and further identifies the relative time, knowledge and monetary investment required by bothresearchers and patients, and mechanisms by which this can be addressed
Pratt et al.  assert that the deeper degrees of participation are better framed as ‘partnerships’. Indeed, deliberative processes of engagement extend beyond ‘focus group’ methods and instead add a range of opportunities for stakeholders to provide input . This partnership level of engagement occurs when “the act of dialogue and negotiation serves to transform opinion in the members of both parties” .
The public and patient engagement priority setting matrix developed for the specific purpose of this review is depicted in Table 1. The different levels of engagement are further categorized into ‘tiers’ of public and patient engagement priority setting. To note, as identified in the IAP2 Spectrum, patients and the public can also participate in a ‘one way’ communication process (i.e., Communicative), where they can simply learn about the priority setting processes. Examples include a public hearing or meeting, drop-in Centre or online information. This level of engagement is not included in Table 1 since the emphasis in this emergent area is to move away from one-way information transfer from decision-makers and researchers to the public, to a two-way process, as depicted by the Learn/Inform level of engagement noted by  (Fig 1). Furthermore, the role of the public and patient is to move beyond sharing of information amongst themselves and instead to seek out roles with researchers from the Consultation level to the Lead-level in priority-setting processes.
A literature review was conducted using a systematic rapid review process, which is a more accelerated method of reviewing the literature compared with traditional systematic reviews. The PRISMA 2009 Checklist was used to endorse good reporting on the methods (S1 File). The advantages of this method are that it is quick, requires limited time and resources, and is functional in supporting informed decision-making. Emphasis is placed on locating and using synthesized research evidence and where not available, high-quality or recent primary studies . A study protocol was first developed to describe the rationale and planned methods of the review (S2 File). This was prepared before the review and used when in its execution. A seven step process was used to undertake the review. All authors contributed and refined the review’s search strategy. Given its limited scope and timeline, the principal researcher (EM) conducted the literature search, applied the selection and critical appraisal criteria, selected the final article yield and extracted the relevant data. The established process provided the principal researcher (EM) with a framework to use in discussion with the other authors particularly when determining if an article met the selection or quality criteria.
Step 1—Identify research questions
The following research questions were identified i) what are the methods of engaging the public and patients in ‘health ecosystem priority setting’ and ‘health research priority setting’ activities?; ii) What are the outcomes of engaging the public and patients in ‘health ecosystem priority setting’ and ‘health research priority setting’ activities?
Step 2—Identify and select search terms
Next, given that this paper focusses on priority setting specifically, a review of key articles related to priority setting were identified and reviewed from the initial scoping review  to determine specific language that targetted the priority setting literature.
As well, to focus the search to health and health care literature, additional search terms were used to filter studies for inclusion that pertained to priority setting within the context of health and the health care system.
A combination of approaches was used to ensure that nuances in language were captured and documented as part of this research study. Search terms were adapted as needed to best meet the requirements of each database. Additional filters (e.g., geography, year of publication, language) were applied when available in the databases, to further refine the search and yield appropriate articles to the inclusion and exclusion criteria.
The search terms were as follows: patient engagement OR patient participation OR Involve AND research priority setting OR consensus build* OR research consult* AND health OR healthcare. (S1 Table)
Step 3—Identify and select data sources
A three-pronged approach was used to collect data. First, specific and appropriate electronic databases were identified to explore the research questions with the assistance of Ryerson University Library Services accessible to the principle researcher (EM). Given the stated methods, the search strategy was narrowed to four relevant databases using all authors’ experiences in extracting relevant article yields and the principal researcher’s accessibility via the stated university’s library system: HealthStar (via OVID); CINAHL; Proquest Databases; and Scholar’s Portal.
Second, a targeted grey literature search was conducted. Each of the formal databases provided the option to also scan grey literature sources, which was a search included to broaden the reach of literature reviewed. Article titles that met the study selection criteria and clearly outlined the process, methods, and/or outcomes of priority setting, were inputted into the Ryerson University Library Services “Search Everything” database and PubMed’s “Single Citation Matcher”. Articles that were similar, relevant, or had been cited from the article title inputted, were reviewed for inclusion. Third, based on the identified process and methods of patient engagement, additional hand searching of the specific tools and/or frameworks was conducted (e.g., James Lind Alliance Priority Setting Partnership Manual).
The author feel confident in the search yield because of their combined use of various data sources as well as relying on literature from formal and manual search strategies.
Step 4—Establish eligibility criteria
The following selection criteria were established based on the purpose and scope of the search. To be included in the review, articles had to address research priority setting at any or all points of the priority setting process and be: i) published in English; ii) published within the timeframe of 2007—Current (10 years) unless the report/article was formative in synthesizing key considerations of patient engagement in health ecosystem and health research priority setting; iii) conducted in Canada, the US, Europe, UK, Australia/New Zealand, or Scandinavian countries. Of those articles meeting the inclusion criteria, the following exclusion criteria were applied: does not clearly identify process, methods, and/or outcomes related to the health ecosystem or health research priority setting.
Step 5—Apply critical appraisal criteria
To keep the research article yield focused within the predefined scope, the following quality selection criteria questions  were applied before selecting the article for data extraction: i) is the research valid, sound, and applicable?; ii) what outcomes can we potentially expect if we implement the findings from this research?; iii) will the target population (i.e., health researchers and practitioners) be able to use this research?. Articles that met all three of these criteria were selected for data extraction.
Step 6—Search input and data extraction
Articles from this search were compiled into a comprehensive data extraction table. The data extracted included the resource citation, study methodology, study setting, technique, key activities, study outcomes, author considerations for barriers, and author consideration for facilitators,. From this information, the processes and activities described in the included articles were organized across the two aforementioned tiers of engagement i.e. Tier 1—Deliberative and Tier 2—Consultative. Few authors identified the level of participation explicitly using language such as ‘deliberative’ or ‘consultative’. As expected, most authors implicitly described levels of public and patient engagement. Therefore, articles were manually categorized based on the characteristics noted in the data extraction table.
Step 7—Summarize key findings
A summary of findings from each of the respective processes was synthesized to highlight key information that would support decision-making for researchers when determining the best priority setting process to apply for their specific patient-oriented research. The data is reported in table format in the Findings to facilitate ease of comparison across priority setting processes.
Overview of findings
‘Health ecosystem priority setting’ activities were most often identified at the ‘Consultative’ level of engagement (Tier 2). ‘Health research priority setting’ activities were most often identified at the ‘Deliberative’ (Tier 1) and ‘Consultative’ (Tier 2) engagement levels. In Table 3, examples of engagement processes and activities are organized according to the two identified levels of engagement.
Tier 1 public and patient engagement priority setting processes
A description of Tier 1 deliberative engagement activities in ‘health ecosystem priority setting’ were limited to Khodyakov’s  literature review, and were limited in description. The literature better emphasized highly-structured patient and public engagement planning processes and activities for research, including the James Lind Alliance Priority Setting Partnerships (UK) [10, 16, 24–35], the Dialogue Method (Netherlands) [25, 36–39], Global Evidence Mapping (Australia) , and the Deep Inclusion Method/CHoosing All Together (US) [21, 40]. While these research planning processes and activities differed, a common approach across Tier 1 public and patient engagement priority setting in research planning included gathering and analyzing identified research priorities by engaging patients and the public along with clinicians and researchers, followed by prioritization of topics through dialogue between all stakeholders. In Table 4, a summary of the Tier 1 public and patient engagement priority setting processes for health research is provided, including the aims, key characteristics, barriers, and facilitators of each of the identified models.
Overall, Tier 1 public and patient engagement priority setting research planning processes and activities demonstrated positive outcomes in setting priorities that were inclusive and objectively validated in the literature. Using the JLA process, for example, evidence of success was demonstrated by recent examples in acne treatments , prostate cancer research , wound care , kidney transplantation and dialysis [16, 32, 41], and dementia  among others. However, reporting of outcomes of the process was predominantly descriptive rather than evaluative. A mechanism for formal evaluation to measure the impact of these different planning processes on the quality of partnership and subsequent outputs is needed . To note, the Dialogue Model process was tested over a 5-year period (2003–2007) for research priority setting based on seven case studies related to spinal cord injuries, neuromuscular diseases, renal failures, asthma/COPD, burns, diabetes, and intellectual disabilities . While each case study demonstrated success in prioritizing research questions specific to the conditions and population, Amba et al.  noted that translating the Dialogue Model process to the particulars of the specific context was important, providing a useful opportunity to expand implementation and evaluation efforts of this process. This implies that ‘one size does not fit all’ in implementing public and patient engagement priority setting processes and activities
Tier 2 public and patient engagement priority setting processes
Articles in which several different data collection activities were identified to engage patients were reviewed, such as through surveys and key informant interviews [44–51], focus groups [13, 14, 52–66], and patient panels . While data collection activities differed, a common approach across Tier 2 included a two-step process. First, reviewing the literature and using expert opinion to determine a set of potential topics of interest; and second, using the topics determined as the basis for a survey for input by health professionals and the public . Open-ended responses were critical . Online surveys, key informant interviews, use of World Café or Dotmocracy facilitation tools provided the data .
Overall, evidence reviewed for Tier 2 public and patient engagement priority setting was more limited in describing comprehensive examples of facilitating patient engagement in health ecosystem and health research priority setting planning activities, although detailed techniques for structuring focus groups, surveys, and other group facilitation activities are available in the literature, but were deemed out-of-scope for this review. A lack of practical guidance on how to integrate public input with other forms of evidence, such as scientific articles, remains . Reporting of operational details including cost, infrastructure, and timeline was limited or non-existent. Furthermore, there were few documented studies describing the effectiveness of the different data collection activities, as evaluation was typically informal and based on the perspectives of those organizing the engagement activities [14, 68].
Most often, success was not reliant on clear evaluation metrics but on the authors’ reviews of the process  or on opinions formed in reaching objectives that were not explicitly stated a priori [14, 56]. For example, Rideout et al.  identified that engaging patients at the macro level provided ample opportunity for shared research collaboration related to CVD and mental health. Working groups were formed with leadership provided by co-chairs: one from specific content areas and one from the public/community leaders. Conklin et al.  underscored that outcomes were inadequate due to insufficient periods of observation to measure the impact on policy and practice. However, authors were clear that local-level initiatives (vs. regional or national) were most likely to ultimately impact patient-centeredness and quality of care .
Tier 1 and Tier 2 public and patient engagement priority setting limitations and opportunities
The key findings from this systematic rapid review of priority setting are supportive of the existing but limited recommendations outlined in the public and patient engagement priority setting literature [6, 10, 60, 63, 69, 70]. However, two predominant themes are worth discussing beyond the operational details of public and patient engagement priority setting activities, including the role of ethics in involving patients and the public as well as the opportunity for evaluation of engagement.
As referenced across the public and patient engagement priority setting and patient and public engagement literature, the risk of tokenism is high across all patient-engagement opportunities, limiting the output of the benefits of true engagement. In response, Pandya-Wood et al.  outlined the importance of developing an ‘ethically conscious’ framework for public and patient engagement priority setting. Addressing ethical considerations is critical for avoiding inadequate allocation of time that may result in a lesser impact on public engagement, tokenism, and potentially leading to public members being disenfranchised and unable to contribute fully to the study and as a result, disengaging. Ways to embed an ethical perspective in any engagement opportunity be it for patient and public engagement or public and patient engagement priority setting include: Timeframe: i) build realistic timeframes, including a research design stage which offers public involvement for at least two weeks to read, clarify, and provide feedback on the process; ii) Visibility: Make any public contributions visible; be transparent about how the public has been involved; iii) Accessibility: The public members involved in the research design stage require information, as do the rest of the research team, but communicated in way that they can access and understand, using plain language.
Finally, of critical importance to patient and public engagement in general and public and patient engagement priority setting in particular, is the role of evaluation in patient engagement in decision-making activities. Limited evaluative evidence may hinder future uptake of patient and public engagement in prioritization and decision-making exercises [10, 25, 43]. This includes the patient perspective about how, when, and why they are engaged and underscores the importance of continuing to catalogue public and patient engagement activities, including evaluation using validated tools to solicit the perspective of the public/patients, researchers, policy-makers, and organizations . For example, Abelson et al.  reported on the development of the Public and Patient Engagement Evaluation Tool (PPEET) which includes three questionnaires designed to elicit feedback from: i) those who participate in Public and Patient engagement activities; ii) those who plan, execute, or sponsor Public and Patient engagement activities within an organization; and iii) those who provide leadership and capacity for Public and Patient engagement within their organizations. The questionnaires are guided by the key public and patient engagement principles of: Integrity of design and process; Influence and impact; Participatory culture; and Collaboration and Common Purpose. Chung et al.  also developed a publically available database for cataloguing current community engagement activities that can be leveraged for future engagement opportunities. This may also help with planning future steps to address engagement opportunities by supporting decision-makers with the tools needed to adequately, meaningfully, and effectively engage the public and patients, and in particular in public and patient engagement priority setting. With no consensus on the ‘gold standard’ of engagement , decision-makers require support for informed decision-making on effective public and patient engagement techniques. This may also help foster the much needed ‘buy in’ from patients, carers, researchers, and decision-makers.
The critical study limitations include challenges in comprehensively identifying the patient engagement literature for review, bias in article selection due to the identified scope, missed information due to a more limited use of exhaustive search strategies (e.g., in-depth hand searching), and the heterogeneity of reported study findings. These limitations can somewhat decrease the ability of the review to inform practical recommendations to support patient engagement in health research .
While the authors made the effort to focus the search for an article yield that was useful and meaningful, it is expected that not all articles that meet this review’s selection criteria were identified and selected. which is an inherent limitation of the stated methods. A lack of uniform reporting on indexing methods for an emerging area like patient engagement is a documented challenge in this emergent research area [73–74]. In itself, a wide range of terms are used across the patient engagement literature, to describe patient engagement in health research. This consideration was included in the design and execution of this systematic rapid review, although it is expected it may have impacted the yield.
In addition, the total number of articles selected for this review may have been limited by i) search terms that were not included as part of the search strategy; ii) limiting the number of databases used to conduct the search; and iii) limiting the grey literature search through databases instead of conducting this search manually (e.g., Google, Google Scholar).
Lastly, due to the heterogeneity of studies included, extracting specifically desired data from studies was a challenge, limiting the interpretation and generalizability of the results within the appropriate context of this review. With a longer timeline, additional resources, and a second researcher to conduct, review/appraise and select the relevant articles independently from the primary researcher the study’s yield may have been further strengthened. Additional research that emphasizes useful and transparent findings of patient engagement using the GRIPP2 (Guidance for Reporting Involvement of Patients and Public)  checklist criteria for patient and public involvement may better ensure that this research focus continues to evolve in a manner that is more meaningful to patients and health outcomes.
At the same time, these limitations are mitigated by: i) inclusion of articles from AbSPORU PE Platform-led scoping review of patient engagement in health research ; ii) Inclusion of articles of relevance sent by staff from the Alberta SPOR Support Unit; iii) review of reference lists and input of single citation matchers for identified articles; iv) widening of the typical timeframe for selected articles (i.e., 3–5 years) given the emergent nature of the patient engagement enterprise.
Additional opportunities to leverage the existing recommendations are offered for consideration. The authors suggest the following to focus efforts on cataloguing public and patient engagement priority setting initiatives:
- Maintain adequate data monitoring, collection, and evaluation to support continued interest in public and patient engagement priority setting and to encourage ‘buy in’ from public/patients, researchers, and decision-makers;
- Describe operational details of public and patient engagement priority setting ‘best practices’: There is need for better reporting of operational details of public and patient engagement priority setting that both worked and did not work, including process activities, sampling, cost, time, outputs and evaluation. This will help inform decision-making on which processes and activities will best suit the given circumstance or context for public and patient engagement priority setting activities. An example of one reporting tool includes creating and managing a data collection template for ongoing public and patient engagement priority setting activities;
- Prioritize evaluation of public and patient engagement priority setting initiatives: To supplement existing descriptive outcomes, evaluative outcomes of the impact of patient and public engagement in health ecosystem and health research priority setting are needed to build a robust evidence-base for patient and public engagement specifically in priority setting activities. Priority setting activities are a useful entry for patient and public engagement activities across the research spectrum; and
- Integrate knowledge mobilization opportunities: While evaluation of findings is important, it is equally important to ensure a knowledge mobilization plan is integrated into the public and patient engagement priority setting process to ensure patient and public engagement priorities are translated into action.
It is intended that the AbSPORU PE Platform will consider and take action on these points, as well as on the evidence synthesized in this review, to support future decision-making with research teams about engaging in Public and Patient Engagement activities in health research priority setting.
- 1. Research CIHI. Strategy for Patient-Oriented Research. 2016 [http://www.cihr-irsc.gc.ca/e/41204.html [Accessed 17th March 2017].
- 2. Hibbard JH. What the evidence shows about patient activation: Better health outcomes and care experiences; fewer data on costs. Health Affairs. 2013;32(2).
- 3. Baker G. Evidence boost: A review of Research highlighting how patient engagement contributes to improved care. [http://www.cfhi-fcass.ca/SearchResultsNews/2014/08/14/evidence-boost-a-review-of-research-highlighting-how-patient-engagement-contributes-to-improved-care]. [Accessed 17th March 2017]
- 4. Carman N. The ROI of engaged patients. Healthcare Financial Management. 2013;67(8):8.
- 5. Clavisi O, Bragge P, Tavender E, Turner T, Gruen RL. Effective stakeholder participation in setting research priorities using a Global Evidence Mapping approach. J Clin Epidemiol. 2013;66(5):496–502.e2. pmid:22819249
- 6. Domecq JP P G, Elraiyah T, Elraiyah T, Wang Z, Nabhan M Shippee N et al. Patient engagement in research: a systematic review. BMC Health Services Research. 2014;14(89).
- 7. Patten S, Mitton C, Donaldson C. Using participatory action research to build a priority setting process in a Canadian Regional Health Authority. Soc Sci Med. 2006;63(5):1121–34. pmid:16540221
- 8. Sofolahan-Oladeinde Y, Newhouse RP, Lavallee DC, Huang JC, Mullins CD. Early assessment of the 10-step patient engagement framework for patient-centred outcomes research studies: the first three steps. Fam Pract. 2017;34(3):272–7. pmid:28334775
- 9. Solomon S, Abelson J. Why and when should we use public deliberation? Hastings Cent Rep. 2012;42(2):17–20. pmid:22733325
- 10. Madden M, Morley R. Exploring the challenge of health research priority setting in partnership: reflections on the methodology used by the James Lind Alliance Pressure Ulcer Priority Setting Partnership. Research Involvement and Engagement. 2016;2(1).
- 11. Manafo E, Petermann L, Mason-Lai P, Vandall-Walker V.Patient Engagement in Canada: A scoping review of the ‘how’ and ‘what’ of patient engagement in health research. Health Research Policy and Systems. Health Research Policy and Systems 2018 16:5 https://doi.org/10.1186/s12961-018-0282-4 pmid:29415734
- 12. Tineke A. Abma JB. Patient participation as dialogue: setting research agendas. Health Expectations. 2010;13:160–73. pmid:20536537
- 13. Khodyakov D, Stockdale SE, Smith N, Booth M, Altman L, Rubenstein LV. Patient engagement in the process of planning and designing outpatient care improvements at the Veterans Administration Health-care System: findings from an online expert panel. Health Expect. 2017;20(1):130–45. pmid:26914249
- 14. Mitton C, Smith N, Peacock S, Evoy B, Abelson J. Public participation in health care priority setting: A scoping review. Health Policy. 2009;91(3):219–28. pmid:19261347
- 15. Forsythe LP, Ellis LE, Edmundson L, Sabharwal R, Rein A, Konopka K, et al. Patient and Stakeholder Engagement in the PCORI Pilot Projects: Description and Lessons Learned. J Gen Intern Med. 2016;31(1):13–21. pmid:26160480
- 16. Manns B, Hemmelgarn B, Lillie E, Dip SC, Cyr A, Gladish M, et al. Setting research priorities for patients on or nearing dialysis. Clin J Am Soc Nephrol. 2014;9(10):1813–21. pmid:24832095
- 17. Rowe G, Frewer LJ. A Typology of Public Engagement Mechanisms. Science, Technology, & Human Values. 2005;30(2):251–90.
- 18. Szelest E. Role for Patients in Research. n.d.
- 19. Participation IAfP. IAP2 Spectrum of Participation. [https://c.ymcdn.com/sites/www.iap2.org/resource/resmgr/foundations_course/IAP2_P2_Spectrum_FINAL.pdf. [Accessed 17th May 2017]
- 20. Amirav I, Vandall-Walker V, Rasiah J, Saunders L. Patient-Researcher Engagement in Health Research—A Parent’s Perspective. Pediatrics 2017;1403(3). pmid:28851740
- 21. Pratt B, Merritt M, Hyder AA. Towards deep inclusion for equity-oriented health research priority-setting: A working model. Soc Sci Med. 2016;151:215–24. pmid:26812416
- 22. Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Syst Rev. 2012;1:10. pmid:22587960
- 23. Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56(2):119–27. pmid:11812811
- 24. Brocklehurst PR, Mackay L, Goldthorpe J, Pretty IA. Older people and oral health: setting a patient-centred research agenda. Gerodontology. 2015;32(3):222–8. pmid:26009973
- 25. Broerse JE, Zweekhorst MB, van Rensen AJ, de Haan MJ. Involving burn survivors in agenda setting on burn research: an added value? Burns. 2010;36(2):217–31. pmid:19577849
- 26. Batchelor JM, Ridd MJ, Clarke T, Ahmed A, Cox M, Crowe S, et al. The Eczema Priority Setting Partnership: a collaboration between patients, carers, clinicians and researchers to identify and prioritize important research questions for the treatment of eczema. Br J Dermatol. 2013;168(3):577–82. pmid:22963149
- 27. Deane KH, Flaherty H, Daley DJ, Pascoe R, Penhale B, Clarke CE, et al. Priority setting partnership to identify the top 10 research priorities for the management of Parkinson’s disease. BMJ Open. 2014;4(12):e006434. pmid:25500772
- 28. Chalmers I, Essali A, Rezk E, Crowe S. Is academia meeting the needs of non-academic users of the results of research? The Lancet.380:S43.
- 29. Gadsby R, Snow R, Daly AC, Crowe S, Matyka K, Hall B, et al. Setting research priorities for Type 1 diabetes. Diabet Med. 2012;29(10):1321–6. pmid:22823450
- 30. Ingram JR, Abbott R, Ghazavi M, Alexandroff AB, McPhee M, Burton T, et al. The Hidradenitis Suppurativa Priority Setting Partnership. Br J Dermatol. 2014;171(6):1422–7. pmid:24903313
- 31. Kelly S, Lafortune L, Hart N, Cowan K, Fenton M, Brayne C, et al. Dementia priority setting partnership with the James Lind Alliance: using patient and public involvement and the evidence base to inform the research agenda. Age Ageing. 2015;44(6):985–93. pmid:26504119
- 32. Knight SR, Metcalfe L, O’Donoghue K, Ball ST, Beale A, Beale W, et al. Defining Priorities for Future Research: Results of the UK Kidney Transplant Priority Setting Partnership. PLoS One. 2016;11(10):e0162136. pmid:27776143
- 33. Layton A, Eady EA, Peat M, Whitehouse H, Levell N, Ridd M, et al. Identifying acne treatment uncertainties via a James Lind Alliance Priority Setting Partnership. BMJ Open. 2015;5(7):e008085. pmid:26187120
- 34. Lophatananon A, Tyndale-Biscoe S, Malcolm E, Rippon HJ, Holmes K, Firkins LA, et al. The James Lind Alliance approach to priority setting for prostate cancer research: an integrative methodology based on patient and clinician participation. BJU Int. 2011;108(7):1040–3. pmid:21914107
- 35. Rees SE, Chadha R, Donovan LE, Guitard AL, Koppula S, Laupacis A, et al. Engaging Patients and Clinicians in Establishing Research Priorities for Gestational Diabetes Mellitus. Can J Diabetes. 2017;41(2):156–63. pmid:27881298
- 36. Elberse JE, Caron-Flinterman JF, Broerse JE. Patient-expert partnerships in research: how to stimulate inclusion of patient perspectives. Health Expect. 2011;14(3):225–39. pmid:21176013
- 37. Pittens CACM, Elberse JE, Visse M, Abma TA, Broerse JEW. Research agendas involving patients: Factors that facilitate or impede translation of patients’ perspectives in programming and implementation. Science and Public Policy. 2014;41(6):809–20.
- 38. van Merode T, Bours S, van Steenkiste B, Sijbers T, van der Hoek G, Vos C, et al. Describing patients’ needs in the context of research priorities in patients with multiple myeloma or Waldenstrom’s disease: A truly patient-driven study. Z Evid Fortbild Qual Gesundhwes. 2016;112:11–8. pmid:27172780
- 39. van Middendorp JJ, Allison HC, Ahuja S, Bracher D, Dyson C, Fairbank J, et al. Top ten research priorities for spinal cord injury: the methodology and results of a British priority setting partnership. Spinal Cord. 2016;54(5):341–6. pmid:26554273
- 40. Goold SD, Myers CD, Szymecko L, Cunningham Collins C, Martinez S, Ledon C, et al. Priorities for Patient-Centered Outcomes Research: The Views of Minority and Underserved Communities. Health Serv Res. 2017;52(2):599–615. pmid:27206519
- 41. Jones JM, Bhatt J, Avery J, Laupacis A, Cowan K, Basappa NS, et al. Setting Research Priorities for Kidney Cancer. Eur Urol. 2017.
- 42. Boney O, Bell M, Bell N, Conquest A, Cumbers M, Drake S, et al. Identifying research priorities in anaesthesia and perioperative care: final report of the joint National Institute of Academic Anaesthesia/James Lind Alliance Research Priority Setting Partnership. BMJ Open. 2015;5(12):e010006. pmid:26674506
- 43. Elwyn G, Crowe S, Fenton M, Firkins L, Versnel J, Walker S, et al. Identifying and prioritizing uncertainties: patient and clinician engagement in the identification of research questions. J Eval Clin Pract. 2010;16(3):627–31. pmid:20482747
- 44. Canadian Mental Health Association. Facts about mental illness 2013 [http://www.cmha.ca/media/fast-facts-about-mental-illness/ -.WV_vSIRuK70.] [Accessed 17th May 2017]
- 45. Boivin A, L P, Lacombe R, Burgers J, Grol R. Involving patients in setting prioroities for healthcare improvement: a cluster randomized trial. Implementation Science. 2014;9(24).
- 46. Brooks F. Nursing and public participation in health: an ethnographic study of a patient council. Int J Nurs Stud. 2008;45(1):3–13. pmid:17046769
- 47. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? The Journal of continuing education in the health professions. 2006;26(1):13–24. pmid:16557505
- 48. Medlow S, Patterson P. Determining research priorities for adolescent and young adult cancer in Australia. Eur J Cancer Care (Engl). 2015;24(4):590–9.
- 49. Nierse CJ, Abma TA. Developing voice and empowerment: the first step towards a broad consultation in research agenda setting. J Intellect Disabil Res. 2011;55(4):411–21. pmid:21323783
- 50. Truitt AR, Monsell SE, Avins AL, Nerenz DR, Lawrence SO, Bauer Z, et al. Prioritizing research topics: a comparison of crowdsourcing and patient registry. Qual Life Res. 2017.
- 51. Walton NA, Martin DK, Peter EH, Pringle DM, Singer PA. Priority setting and cardiac surgery: a qualitative case study. Health Policy. 2007;80(3):444–58. pmid:16757057
- 52. Rideout C, Gil R, Browne R, Calhoon C, Rey M, Gourevitch M, et al. Using the Delphi and snow card techniques to build consensus among diverse community and academic stakeholders. Prog Community Health Partnersh. 2013;7(3):331–9. pmid:24056515
- 53. Cox A, Arber A, Gallagher A, MacKenzie M, Ream E. Establishing Priorities for Oncology Nursing Research: Nurse and Patient Collaboration. Oncol Nurs Forum. 2017;44(2):192–203. pmid:28222079
- 54. Boote J, Baird W, Beecroft C. Public involvement at the design stage of primary health research: a narrative review of case examples. Health Policy. 2010;95(1):10–23. pmid:19963299
- 55. Boote J, Baird W, Sutton A. Public involvement in the systematic review process in health and social care: a narrative review of case examples. Health Policy. 2011;102(2–3):105–16. pmid:21641075
- 56. Conklin A, Morris Z, Nolte E. What is the evidence base for public involvement in health-care policy?: results of a systematic scoping review. Health Expect. 2015;18(2):153–65. pmid:23252574
- 57. Costa-Font J, Forns JR, Sato A. Participatory health system priority setting: Evidence from a budget experiment. Soc Sci Med. 2015;146:182–90. pmid:26517295
- 58. Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: development and usability testing of the Public and Patient Engagement Evaluation Tool. Health Expect. 2016;19(4):817–27. pmid:26113295
- 59. Armstrong MJ, Mullins CD, Gronseth GS, Gagliardi AR. Recommendations for patient engagement in guideline development panels: A qualitative focus group study of guideline-naive patients. PLoS One. 2017;12(3):e0174329. pmid:28319201
- 60. Etchegary H, Bishop L, Street C, Aubrey-Bassler K, Humphries D, Vat LE, et al. Engaging patients in health research: identifying research priorities through community town halls. BMC Health Serv Res. 2017;17(1):192. pmid:28284232
- 61. Kielmann T, Huby G, Powell A, Sheikh A, Price D, Williams S, et al. From awareness to involvement? A qualitative study of respiratory patients’ awareness of health service change. Health Expect. 2011;14(3):321–33. pmid:21029282
- 62. Piil K, Jarden M. Patient involvement in research priorities (PIRE): a study protocol. BMJ Open. 2016;6(5):e010615. pmid:27221126
- 63. Flaman LM, Nykiforuk CI, Plotnikoff RC, Raine K. Exploring facilitators and barriers to individual and organizational level capacity building: outcomes of participation in a community priority setting workshop. Glob Health Promot. 2010;17(2):34–43. pmid:20587629
- 64. Staniszewska S, Brett J, Mockford C, Barber R. The GRIPP checklist: strengthening the quality of patient and public involvement reporting in research. Int J Technol Assess Health Care. 2011;27(4):391–9. pmid:22004782
- 65. Northway R, Hurley K, O’Connor C, Thomas H, Howarth J, Langley E, et al. Deciding what to research: an overview of a participatory workshop. British Journal of Learning Disabilities. 2014;42(4):323–7.
- 66. Restall GJ, Carnochan TN, Roger KS, Sullivan TM, Etcheverry EJ, Roddy P. Collaborative priority setting for human immunodeficiency virus rehabilitation research: A case report. Can J Occup Ther. 2016;83(1):7–13. pmid:26755039
- 67. Arkind J, Likumahuwa-Ackman S, Warren N, Dickerson K, Robbins L, Norman K, et al. Lessons Learned from Developing a Patient Engagement Panel: An OCHIN Report. J Am Board Fam Med. 2015;28(5):632–8. pmid:26355135
- 68. Miller CL, Mott K, Cousins M, Miller S, Johnson A, Lawson T, et al. Integrating consumer engagement in health and medical research—an Australian framework. Health Res Policy Syst. 2017;15(1):9. pmid:28187772
- 69. Boivin A, Lehoux P, Lacombe R, Burgers J, Grol R. Involving patients in setting priorities for healthcare improvement: a cluster randomized trial. Implementation science: IS. 2014;9:24. pmid:24555508
- 70. Hewlett S, Wit M, Richards P, Quest E, Hughes R, Heiberg T, et al. Patients and professionals as research partners: challenges, practicalities, and benefits. Arthritis Rheum. 2006;55(4):676–80. pmid:16874772
- 71. Pandya-Wood R, Barron DS, Elliott J. A framework for public involvement at the design stage of NHS health and social care research: time to develop ethically conscious standards. Research Involvement and Engagement. 2017;3(1).
- 72. National Collaborating Centre for Methods and Tools. Rapid reviews: Methods and implications [http://www.nccmt.ca/uploads/media/media/0001/01/c931b32600b7495967fe627c77fbb0cfb00c85a0.pdf. [Accessed 17th of May 2017]
- 73. Forhan M, R C, Solomon P. Contributors to patient engagement in primary health care: perceptions of patients with obesity. Prim Health Care Res Dev. 2012;14(4): 367–372. pmid:23237022
- 74. Gallivan J B K, Bellows M, Eigenseher C. The many faces of patient engagement. J Particip Med. 2012;26(4):e32.
- 75. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ (Clinical research ed). 2017;358:j3453.