https://orcid.org/0000-0002-1326-190X
Figures
Abstract
Background
The Wales COVID-19 Evidence Centre (WCEC) was established from 2021–23 to ensure that the latest coronavirus (COVID-19) relevant research evidence was readily available to inform health and social care policy and practice decision-makers. Although decisions need to be evidence-based, ensuring that accessible and relevant research evidence is available to decision-makers is challenging, especially in a rapidly evolving pandemic environment when timeframes for decision-making are days or weeks rather than months or years. We set up knowledge mobilisation processes to bridge the gap between evidence review and informing decisions, making sure that the right information reaches the right people at the right time.
Aims and objectives
To describe the knowledge mobilisation processes used by the WCEC, evaluate the impact of the WCEC rapid evidence reviews, and share lessons learned.
Methods
Our knowledge mobilisation methods were flexible and tailored to meet stakeholders’ needs. They included stakeholder co-production in our rapid evidence review processes, stakeholder-informed and participatory knowledge mobilisation, wider dissemination of outputs and associated activities including public engagement, capacity building and sharing of methodologies. Feedback on processes and evidence of impact was collected via stakeholder engagement and a stakeholder survey.
Results
Findings indicate that knowledge mobilisation processes successfully enabled use of the WCEC’s rapid evidence reviews to inform policy and practice decision-makers during the COVID-19 pandemic in Wales. Realising actual public and patient benefit from this ‘pathway to impact’ work will take further time and resources.
Discussion and conclusion
The WCEC knowledge mobilisation processes successfully supported co-production and use of rapid evidence review findings by scientific advisors and policy and practice decision-makers during the COVID-19 pandemic. Identified barriers and facilitators are of potential relevance to wider evidence initiatives, for setting up similar Centres during crisis situations, and supporting future evidence-based policy and practice decision-making.
Citation: Gal M, Cooper A, Joseph-Williams N, Doe E, Lewis R, Law RJ, et al. (2024) Knowledge mobilisation of rapid evidence reviews to inform health and social care policy and practice in a public health emergency: Appraisal of the Wales COVID-19 Evidence Centre processes and impact, 2021–23. PLoS ONE 19(11): e0314461. https://doi.org/10.1371/journal.pone.0314461
Editor: Sreeram V. Ramagopalan, University of Oxford, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
Received: July 21, 2023; Accepted: November 11, 2024; Published: November 26, 2024
Copyright: © 2024 Gal et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The anonymised stakeholder survey data (questions and responses (answer selection and free text) has now been made avaialble on the repository Figshare (https://figshare.com). The data file can be downloaded here: https://figshare.com/articles/dataset/Wales_COVID-19_Evidence_Centre_2021-2023_Stakeholder_survey/25912459 Some data and identifiers were removed before adding the data to Figshare (i.e. names. job title, place of work, comments that may identify participants, feedback where participants did not give permission to quote this). This was done to ensure the data was anonymised, as respondents had not consented to have this data shared without anonymisation.
Funding: The following authors; MG, AC, ED, NJW, RL, JG, and AE, received salary funding for this work as part of the Wales COVID-19 Evidence Centre (https://healthandcareresearchwales.org/about-research-community/wales-covid-19-evidence-centre), which was funded by Welsh Government through Health and Care Research Wales (https://healthandcareresearchwales.org/), project identification number 522128. Authors SA and ND received reimbursement for their time from the same award, as members of the Evidence Centre public partnership group. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: Authors RJL, AW, RO, BC, LT, CR, SM and SM are employed by Welsh Government. Author AW is funded by Welsh Government through Health and Care Research Wales.
1.0 Introduction
The COVID-19 pandemic dramatically changed health and social care needs, and the way essential services were delivered in Wales and beyond. Health and care policy and practice decisions had to be rapidly made and informed by evidence. The need for evidence-informed policy and practice decisions is undisputed [1]. However, ensuring that the best research evidence is available, accessible and timely is challenging at the best of times, and even more so in the context of a rapidly evolving public health emergency [2]. The pandemic has resulted in some valuable examples of exemplary practices by clinicians and researchers. These should serve as a valuable resource and lesson for managing future crisis situations, and to inform future health and social care policy and practice [3].
Knowledge mobilisation enables research evidence to be understood and used by health and social care policy and practice decision makers, and to make a difference to patients and society. The process moves beyond ending a research project with conference presentation and a peer-reviewed journal publication. The National Institute of Health and Care Research (NIHR) defines knowledge mobilisation as ‘Getting the right information to the right people in the right format at the right time, so as to influence decision-making’, and has a range of resources, which aim to help researchers with knowledge mobilisation of research findings [4]. Knowledge mobilisation is a dynamic and iterative process that includes engagement, co-production, shared learning, dissemination, communication, and the exchange and use of knowledge. Several frameworks, theories, models and guides for mobilising knowledge have been developed. These include the widely used and adapted ‘Knowledge to Action’ Framework, Developing Evidence Enriched Practice, the ‘What Works Networks’, and the ‘Bridge Building Model’ [5]. The latter two of these being targeted towards the use of research evidence in the policy context [6–9]. In 2021, Social Care Wales (a Welsh Government-Sponsored Body) published its model for knowledge mobilisation, setting out its aims and approach for ‘Evidence-enriched practice, planning and policymaking for social care in Wales’ [10].
There is much published research on the enablers and barriers to the use of research evidence. In the policy making and health and social care practice context, common barriers include a lack of co-production to ensure that the research is actually relevant and needed, inaccessibility of the evidence (scientific language and findings only available in journals), and failing to meet the timeframe of the policy cycle (research can take years and policy makers need information more rapidly), or practice need (e.g. the clinical pathway has not been considered) [1, 11–13]. Conversely, enablers include early engagement and co-production of research together with end users like policy makers, social care workers, clinicians and members of the public. This helps researchers to understand the need, setting and timeframe for evidence uptake, and makes research findings more likely to be used. It should be noted that research evidence will only partly contribute towards the knowledge used by decision-makers, who will also use advice from domain experts, from those with lived experience, and their own knowledge to inform decisions [14]. In addition, the values and preferences of individuals and communities, and available human and financial resources also play an important part in decision making.
Achieving impact from research is supported by knowledge mobilisation processes, and plans on how to achieve accessibility and dissemination of research findings should underpin the research from the start. While impact is often referred to as public and patient benefit, other forms of impact are also valid. Shorter term impact can include increasing knowledge or awareness, or changes in attitudes and motivation, which can contribute to changes in individual practice. Longer-term impact include the changes in policy, behaviour or practice that benefit patients and the public, which can take years [15].
1.1 The Wales COVID-19 Evidence Centre
Several new entities were set up to conduct rapid reviews of research evidence to aid decision-makers during the COVID-19 pandemic. Examples include the UK Health Security Agency COVID-19 Rapid Evidence Service, the COVID-19 Evidence Network to support Decision-making (COVID-END Global), and the WCEC [16–18]. With the remit of ‘Good questions, answered quickly’, the WCEC was funded by Welsh Government and rapidly set-up (March 2021 –March 2023). The purpose of the WCEC was to collect research evidence and ensure this was accessible and rapidly available to the people making decisions for health and social care policy and practice in Wales during the pandemic, and in the later move towards recovery. The WCEC funding covered the WCEC core team, its partner research groups (researchers that conducted the evidence reviews) and public partner members [19], but not stakeholder time or commitment which was given gratis throughout the work.
Knowledge mobilisation and impact activities underpinned all the WCEC processes, which including stakeholder identification, research question prioritisation, rapid evidence reviews, and also rapid primary research and public involvement (the two latter are not included in this paper).
In this paper we have defined collaboration as an interdisciplinary cross-sector partnership that involves working together and coordinating between researchers, and stakeholders from institutions and or organisations, (e.g. WCEC and Welsh Government), which brings distinct and essential expertise to a project. We have defined co-production as working together with our stakeholders throughout a project (e.g. a rapid review), in processes including attending meetings, refining the research questions, identifying outcomes, discussion and interpretation of the findings, interpreting policy and practice implications, reviewing and commentating on all outputs, (e.g. reports, infographics), and involvement in knowledge mobilisation processes.
1.2 Aims
The aims of this paper are to describe, evaluate and reflect on the processes for knowledge mobilisation and evidencing impact from the WCEC rapid research evidence reviews, describe lessons learnt, and make recommendations for best practice. This should serve as a resource for future crisis situations where similar Centres may be set up, and to inform future efforts enabling the use of research evidence by health and social care decision makers.
2.0 Materials and methods
Knowledge mobilisation and evidencing impact were a priority for the WCEC, with two members of staff employed to support this, and a member of the Welsh Government Technical Advisory Cell (TAC) working closely with the WCEC core team to facilitate engagement and communication between Welsh Government teams and the Centre [19].
Knowledge mobilisation processes were iterative, tailored to meet the requirements of stakeholders and included the following activities: 1) co-production and engagement with stakeholders, 2) stakeholder informed knowledge mobilisation, 3) wider dissemination, 4) associated knowledge mobilisation activities, and 5) tracking and evidencing impact (Fig 1). We describe each step below.
Figure based on revised Knowledge-to-Action framework developed in 2006 by Ian Graham and colleagues.
2.1 Co-production and engagement with stakeholders (step 1)
The WCEC identified and reached out to stakeholder groups in Wales to identify research questions of the highest priority for health and social care decision-making during the pandemic and subsequent recovery period [19–21]. Stakeholders engaging with the Centre included teams from Welsh Government’s TAC and Technical Advisory Group (TAG) (providing a role for Wales akin to “UK Government Scientific Advisory Group for Emergencies (SAGE)”) described in their terms of reference [22], which provided coordination of scientific and technical advice to support Welsh Government decision-makers during the pandemic. Potential stakeholders within Welsh Government were identified via a stakeholder mapping exercise [19]. The TAC also had a boundary spanning role to promote communication between the evidence centre and policy-makers.
Other stakeholders included TAG sub-groups (e.g., Risk Communication and Behavioural Insights, Policy Modelling, and Environment) [19], Social Care Wales, NHS and social care leaders, public [21] and professional societies (Table 1).
Stakeholders’ questions were prioritised, with those most urgent and likely to have an impact (e.g., inform health and social care policy or practice decisions) being taken onto the WCEC work programme (Fig 2) [20]. The WCEC contacted stakeholders to invite submission of new questions every 3 months. Additionally, the prioritisation process was flexible and responsive to accommodating new urgent questions outside the 3-monthly invite. The WCEC and stakeholders worked collaboratively to develop focused research questions and agree on outcomes, discuss evidence review findings and relevance to the local setting (Wales), identify policy and practice implications, write and review the final reports, plan and support knowledge mobilisation, and provide evidence of impact. During the review process, 1–3 representative stakeholders from each stakeholder group (that had a question accepted onto the WCEC work programme) were invited and attended a minimum of three on-line meetings with the WCEC. The meetings included an initial meeting to ensure the question was focused and that the outcomes were relevant to the stakeholders needs, a second meeting (one week later) to present initial findings and discuss the next review steps, and a third meeting to present findings and discuss knowledge mobilisation and impact [23]. If a key stakeholder was unable to attend a meeting, they were invited to send a representative from their group to attend on their behalf. These stakeholder meetings were a part of every review undertaken by the WCEC.
In addition to e-mail communication, co-production was enabled through the on-line meetings with key stakeholders during the review process, and involving stakeholders in dissemination of findings (e.g., Welsh Government evidence briefing sessions, and public symposia). The meetings with stakeholders and evidence briefing sessions included discussions about knowledge mobilisation and impact. Feedback to improve WCEC processes was also sought from stakeholders at the meetings, via e-mails and through a stakeholder survey.
2.2 Stakeholder informed knowledge mobilisation (step 2)
2.2.1 Pathway to impact.
Stakeholders submitting questions to the WCEC were asked to identify the potential impact that answering a question would have (e.g., used to inform a particular policy or decision) and the timeframe that the evidence was needed by) [20].
2.2.2 Knowledge mobilisation plans.
Final evidence reports were accompanied by a knowledge mobilisation plan developed with stakeholder input (Fig 3). The plans included ‘stakeholder mapping’ to identify key individuals involved in policy or practice decision-making, and further groups with an interest in the review topic. The plan identified relevant resources and dissemination methods which would be jointly supported by the WCEC team and the stakeholders e.g., Welsh Government ‘evidence briefing’ sessions and themed public-facing symposia.
2.2.3 Timely targeted outputs.
Reports. Reviews were conducted rapidly as required by stakeholders (up to 3 months for rapid reviews, and 1 week for rapid evidence summaries) [23]. The outputs of evidence reviews were presented in reports. Report templates were developed to ensure relevant information could be easily accessed by stakeholders. Reports were full scientific evidence review reports, but to aid access to key information by stakeholders, every report had 3 specific front pages. These included a short information box of review details, followed by a 2-page ‘Topline summary’ containing key information in an accessible language (Table 2). Stakeholders who attended the meetings during the review process, contributed and reviewed the Topline summaries to ensure the 2 pages contained the key information they needed, that the policy and practice implications were accurately reflected, and that the findings were accessible.
Stakeholders, including the WCEC public partnership group members, contributed to report reviewing and identifying the policy and practice implications. Reports were made available to stakeholders for use within their teams at the point of completion (pre-publication) but were watermarked to limit circulation beyond those teams until publication on pre-print servers and in the WCEC library. Feedback on report structure, timeliness and accessibility of the information was collected via the stakeholder survey.
Infographics. When identified by stakeholders as being helpful for accessibility and dissemination purposes, a one-page infographic was produced by LD/FC using Prezi and Affinity Software. The infographics highlighted the main report findings, and the policy and practice implications in a single page, using accessible text and illustrations.
2.2.4 Presenting evidence.
Review findings were shared via presentations to key stakeholders at the final stakeholder meetings, to the Welsh Government TAC and TAG, and at fortnightly on-line Welsh Government evidence briefing sessions. Evidence briefings were invitation-only events where questions could be posed to a panel (key stakeholders, WCEC public partnership group members, review team), and results and knowledge mobilisation discussed.
2.3 Wider dissemination (step 3)
2.3.1 Communications and social media.
Health and Care Research Wales (HCRW) developed a communications strategy (including media and social media management), provided support for the WCEC website and events, and worked to feature WCEC in the media. Communication outputs for the reports including blogs, news stories, social media and event webpages were delivered by HCRW, raising the profile of WCEC and its publications across HCRW channels. A WCEC hashtag (#WalesCovidEvidence) was used to raise social media profile. WCEC’s Public Health Wales and Health Technology Wales partner groups (NHS based) also undertook social media to raise awareness of WCEC outputs within their networks and websites.
2.3.2 Website and report library.
Health and Care Research Wales created, updated, and maintained an online WCEC section on their website in English and Welsh. The WCEC webpage hosted information about its work programme, methods, events, and links to news stories and WCEC newsletters.
Public facing documents including infographics, lay summaries, report Topline summaries, symposium presentations, newsletters, news stories, blogs and the WCEC work programme were translated into Welsh and hosted on the Welsh language version of the website. Full translation of reports into Welsh was available on request.
The website included an easily searchable (by theme, key words), bespoke report library. The library landing page for each report was a lay summary (written by members of the WCEC public partnership group) with a link to the full report and infographics, which could be downloaded.
A link to the WCEC website was included in the websites of other entities undertaking COVID-19 related rapid evidence reviews to raise awareness of the Centre’s work and avoid duplication. This included the UK Health Security Agency Rapid Evidence Service and COVID-END.
2.3.3 Public symposia and engagement.
WCEC held three themed public facing, on-line ‘Evidence into Practice Symposia’ focusing on specific areas of its work including the effects of the pandemic on education, inequalities, and public involvement. Senior Welsh Government ministers or senior policy officials or science advisors opened each of the events. The Symposia programmes included question-and-answer sessions with an expert panel (Welsh Government scientific advisors, stakeholders, and members of the evidence review teams). At one Symposium, breakout groups were included to identify questions that were important to participants, with subsequent voting to rank the top 10 priority questions for inclusion in the WCEC question prioritisation process.
WCEC took part in two public engagement events and a series of focus groups to raise awareness of the Centre and to engage and involve under-represented groups in Wales in its work. Set up, promotion and conduct of these events was supported by HCRW members. (The public engagement is not included in this paper).
2.3.4 Publication of reports.
Initially, completed evidence review reports were published only in the WCEC library. From May 2022, reports were published on pre-print servers including MedRxiv (health and social care relevant reports), EdArXiv (education relevant reports), and Research Square (an environment relevant report). The WCEC library linked to the reports on the pre-print servers.
2.4 Associated activities (step 4)
Associated knowledge mobilisation activities included capacity building, sharing methodologies, and identifying evidence gaps. Evidence gaps and need for further research identified during the review processes were clearly indicated within each report, and collected in a document, with an aim to share with research funders (e.g., Health and Care Research Wales and the National Institute of Health Research). COVID-19 related systematic reviews and preparation of infographics were offered as projects to medical students at Cardiff University. As the timeline of student projects would not have met the rapid turnaround required for many reviews, the questions they answered were less urgent. Stakeholders still received timely findings via a report and presentation, and students subsequently submitted the reports to the University and to peer reviewed journals.
2.5 Track and evidence impact (step 5)
2.5.1 Stakeholder survey.
A stakeholder survey was developed to capture information and feedback including on WCEC processes, report structure, ease of information access, further suggestions for knowledge mobilisation, and using the review evidence. Participants were asked to select from a 5-point Likert scale for questions, and the data was analysed to calculate the frequency percentage. The survey also included open-ended questions to collect stakeholder feedback. The open-ended questions were optional and information was used to gather information for knowledge mobilisation plans and as example quotations (anonymised). No qualitative analysis was planned or conducted. All stakeholders (n = 44) taking part in the three stakeholder meetings during the review process were asked to complete the survey. Stakeholders were asked to completed the survey via e-mail following the report publication, and up to three e-mail reminders were sent.
Where stakeholders were involved in more than one evidence review, they were invited to include more than one report in their survey completion. The survey was open from the 2nd December 2021 to the 31st March 2023. No ethical approval was required for this survey as it was a service evaluation of the WCEC processes and outputs, and not conducted for research purposes.
2.5.2 Collecting additional stakeholder feedback, and metrics.
The WCEC received a total of 223 questions from stakeholders (Fig 2). These included 11 urgent Welsh Government questions. Only questions (n = 58) with a clear pathway-to-impact (e.g., answering a question would help to inform Welsh Government advisors, a policy or plan) were accepted onto the WCEC work programme (i.e. research that would be conducted by the Centre) [19]. Where stakeholder surveys were not completed, information pertaining to impact and use of the report was collected via e-mail requests to the stakeholders involved in the stakeholder meetings, and during discussions at the on-line stakeholder meetings and the evidence briefing sessions, which were both attended by stakeholders. Where further information was needed and stakeholders did not respond to e-mails request or complete the survey, the boundary-spanning Welsh Government team member contacted Welsh Government stakeholders directly to obtain this information.
Reference of reports in stakeholder publications (e.g., Welsh Government TAC/TAG publications) was identified and collected. Identifying reference to WCEC reports in Welsh Government publications was enabled by having a boundary-spanning Welsh Government team member to help identify and search for these. Members of the WCEC core team also searched reports published on the TAC/TAG pages of the Welsh Government website for inclusion of WCEC reports.
3.0 Results
Note that quotes presented in the results section are taken from the stakeholder survey free text feedback, and are for illustrative purposes only.
3.1 Co-production and engagement with stakeholders (step 1)
In two years of operation (March 2021–23), the WCEC engaged and collaborated with 44 stakeholder groups and produced 51 reports from its rapid review work [19].
No stakeholders declined to participate in WCEC’s work owing to lack of direct funding throughout its 2021–23 programme. There was good engagement from all stakeholders involved in the WCEC reviews. We defined good engagement as stakeholders or stakeholder representatives i) attending stakeholder meetings to help refine the research question [usually ‘PICO’ format [24]) and agreeing relevant research outcomes, ii) agreeing deadlines for the evidence outputs, iii) interpreting review findings, identifying policy and practice implications, iv) providing input to knowledge mobilisation plans, v) supporting dissemination (e.g., sharing reports with colleagues) and vi) providing information on how the research evidence was used. Where requested, stakeholders or stakeholder representatives also participated in the additional dissemination activities (evidence briefings and public symposia). Poor engagement would have been defined by failure to attend (personally or via sending a representatives) the stakeholder meetings during the review process without which the review would not have been taken forward, and providing no contribution towards knowledge mobilisation or evidence of how the review findings were used.
Results from the stakeholder survey (n = 21) indicated that 19 (90.5%) stakeholders were ‘very satisfied’ with the engagement process and meetings, and two (9.5%) were ‘satisfied’. (Details on survey sample and responses can be found below in section 3.5). Stakeholders reported value in being part of the review process, committed their time to attend the meetings and understood and trusted the review methods and findings e.g.:
‘…the WCEC representatives involved with the project were highly responsive. Their suggestions were constructive and they worked proactively to make meaningful progress that allowed our Welsh Government project team to quickly and conveniently locate relevant evidence that helped shape our policy proposals’
(S13).
Whilst positive towards the general approach, feedback suggested refining down-stream question prioritisation processes, so that the evidence review processes could be streamlined:
‘….I think that …. there is no perfect solution or approach. There is an evident need to manage demand and expectation at every stage of the process if the WCEC is not to be over-whelmed. ….. there is.. a need to be more rigorous (even brutal) in requiring that questions are specific and narrow (and thus deliverable) from an earlier stage but also understand that this is not easy’
(S10).
3.2 Stakeholder informed knowledge mobilisation (step 2)
3.2.1 Knowledge mobilisation plans.
Gathering information to support a knowledge mobilisation plan for each review worked well during discussions at stakeholder meetings, evidence briefings and via e-mail follow-up. For all the reviews, stakeholders or their representatives provided suggestions about which organisations and persons would be interested in the findings, and shared the reports with their groups.
Additional information for the knowledge mobilisaton plan was also gathered from the stakeholder survey when participants provided this information here.
Stakeholders also shared the reports within their teams and in some cases supported knowledge mobilisation more widely. For example, stakeholders from Social Care Wales arranged for findings from a rapid review of ‘Innovations help to attract, recruit and retain social care workers within the UK context’ to be presented at the 2022 Association of Directors of Social Services (ADSS Cymru) summer seminar and at a meeting of the ADSS Cymru Workforce Leadership Group. They also shared the report with the Social Care Wales workforce task and finish group (community capacity building), and with regional workforce boards (Directors of Social Services, workforce leads, managers and staff in Wales).
A further example is for a review conducted on ‘the effectiveness of community diagnostic centers. Stakeholders including the Welsh Government lead for planned care improvement and recovery enabled the findings to be presented to relevant decision-making groups including at a TAG meeting and at a Welsh Government Health Executive meeting (NHS (Health Boards) Directors of Planning, and Diagnostics leads).
3.2.2 Outputs and presentations.
The results from the stakeholder informed knowledge mobilisation activities are presented in Table 3.
3.3 Wider dissemination activities (Step 3)
The results of wider dissemination activities are shown in Table 4.
3.4 Associated activities (step 4)
Examples of Associated activities carried out by the WCEC are provided in Table 5. Four Cardiff University undergraduate medical students conducted COVID-19 systematic reviews, which have been submitted for peer-reviewed publication. 3 other students completed infographics to support WCEC evidence review reports.
Evidence gaps and need for further research that were identified in the review process are summarised in the topline summaries for each report and being made available on the website as a resource to researchers and research funders.
The WCEC shared its methodology and work programme to raise awareness of the Centre’s work and avoid duplication. The September 2021 work programme was also referenced in a Welsh Government TAC summary of advice [31].
3.5 Track and evidence impact (step 5)
3.5.1 Stakeholder survey.
22 of 51 reviews were evaluated via the survey, and the remainder were evaluated through data collection following e-mail request, feedback at stakeholder meetings, and feedback provided by stakeholders to the WG liaison member of the core team.
All stakeholders (n = 44) who participated in the reviews and stakeholder meetings (1–3 per evidence review) were invited to complete the survey. However, stakeholders were variably and overall less engaged in the survey processes. Twenty-one (48%) stakeholders (mainly from Welsh Government) completed the survey (to December 2022) and two of these completed the survey for more than one evidence review. The participants included representatives from Welsh Government (policy lead and scientific advisors), Public Health Wales, the NHS and social care. Not all stakeholders answered every question. Two respondents dropped off towards the end of the survey. Nevertheless, some useful information was obtained regarding impact, and overall satisfaction with Centre processes including those aligned to knowledge mobilisation was high (Table 6), with feedback including:
‘I have seen excellent teamwork and a strong professional ethic that holds itself to account to provide the highest quality research and analysis’…. ‘The model for delivery is one that is agile and flexible as well as speedy and there is a demand for this—albeit with an understanding of the caveats that come along with it’
(S6).
There were no overtly critical responses, and a general consensus that the approaches used worked well as evidenced in Table 6.
3.5.2 Collecting additional stakeholder feedback and metrics.
In some cases, it was necessary for the Welsh Government liaison person to gather further evidence of how the research evidence was used (if not included in a Welsh Government publication or testimonial received via e-mail or survey) e.g. if a key stakeholder had left or changed department. This was enabled as the liaison person was able to identify and contact relevant people within Welsh Government, which the WCEC was unable to do in some cases.
The evidence reviews informed scientific advisors, health and social care policy and practice decision-makers, and their teams. Where the reviews identified a lack of published evidence, or that evidence was of low quality, the reports indicated the need for robust research and evaluation. In some cases, there was little evidence available in particular areas, and sometimes there was no further information other than that which was already known by stakeholders:
Whilst the evidence of proven innovations was quite weak some of the evidence provided mirrored a lot of the initiatives that are already underway in Wales. The next phase is to further understand what practice is being adopted operationally and we have plans to do just that’
(S14).
Evidencing actual impact from these types of reviews was challenging and often not possible.
Twenty-one review reports were referenced in Welsh Government advice and reports, and informed advisors and policy and practice decisions. The best impact examples of review findings being used to inform policy or practice decisions, are provided in Table 7.
4.0 Discussion
This paper describes the knowledge mobilisation processes of the WCEC, which were set up rapidly, and ensured that evidence was available to health and care decision-makers and advisers in Wales in response to the COVID-19 pandemic. Measures of success include that over 2 years, the WCEC worked closely with >30 stakeholder groups, including members of the public, to produce 51 evidence reviews, underpinned by knowledge mobilisation processes, which led to the evidence informing health and care policy and practice decisions during the pandemic. Below, we reflect on the five steps of our knowledge mobilisation framework, and follow this with lessons learnt by the WCEC.
Step 1: Co-production and engagement with stakeholders
The successful engagement and co-production with a wide range of key health and social care stakeholders in Wales may not have happened outside the urgency and unique requirements of the pandemic. The pandemic directly led to the set-up of the new Welsh Government TAC and TAG, who the WCEC was able to work with closely. Having a liaison person who was embedded in both Welsh Government and the WCEC was crucial, and the benefit of having a person to span boundaries has been evidenced elsewhere [40]. The other health and care stakeholders also had an urgent need for evidence, which helped forge relationships and aided co-production and engagement.
The responsiveness and involvement of stakeholders and members of the public partnership group were key to focusing the research questions, which were often too broad to be answered in the available amount of time [23]. The expertise of the stakeholders and members of the public partnership group was also essential to contextualise the review findings, and support and expedite knowledge mobilisation. Involvement during the review process also helped understanding and built trust in the review results and the WCEC methods.
Moving beyond the pandemic, the need for evidence to address issues like the long health service waiting lists, is ongoing, though the time frames may not have the same immediate urgency. A key challenge will be the maintaining the stakeholder relationships to ensure future research will contribute towards evidence informed policy and practice. Already structures and teams set up during the pandemic have changed. For example, the TAC and TAG have now dissolved, though their strength and contributions were realised, and a new Welsh Government Science, Evidence and Advice division has been set up, which includes members of TAC and TAG.
Following the end of funding of the WCEC, we received 5 further years of funding from Welsh Government via Health and Care Research Wales, and are now funded as the ‘Health and Care Research Wales Evidence Centre (2023–2028) [41]. This enables us to continue longer term close collaborations with the stakeholders we have worked with in the WCEC to enhance use of our report findings, and collect and evidence longer term evidence of impact. We also continue to work with public members and currently the new Evidence Centre has a public partnership group who are involved in all the Centre processes.
To improve engagement and communication with policy and practice decision makers going forward, the new Evidence centre has now implemented ongoing, additional short meetings with stakeholders, which focus on service evaluation, knowledge mobilisation planning and impact tracking. The new Centre also now has additional contact between the review team and stakeholders during the review process (e.g. to discuss the review protocol), and this helps to manage and clarify expectations.
Step 2: Stakeholder informed knowledge mobilisation
Involving stakeholders in knowledge mobilisation processes was crucial to enable evidence dissemination, and track use of the evidence. It enabled much wider sharing of evidence and links to be built with groups outside academia, which may not have been so readily accessible otherwise.
Knowledge mobilisation becomes particularly challenging where organisations are large and diverse and requires domain-specific strategies. Working with stakeholders to disseminate evidence ensures that evidence is seen as coming from a trusted source and reaches appropriate audiences. For example, the WCEC worked together with Social Care Wales knowledge exchange and research teams to disseminate evidence across social care organisations.
Reports of the evidence review findings, included the two-page Topline summary of headline findings. This was directly targeted to address the needs of stakeholder and their feedback was positive. The Topline summary was based on other evidence policy briefs including the BRIDGE Evidence-informed framework for effective information-packaging to support policymaking [9]. Moving forward, it may be useful to include economic considerations, and qualitative research capturing public/patient experiences alongside results of evidence synthesis [1].
In addition to tailored and targeted messages, training may increase evidence uptake by health managers and policymakers [42].
Step 3: Wider dissemination
Wider dissemination processes increased access to our reports and work programme, and potentially avoided duplication of work by other Centres reviewing pandemic related evidence. It is worth noting that the wider dissemination processes required a considerable amount of time (e.g., developing and maintaining website and library, editing and publishing of reports, social media, organising public events and evidence briefing). Further evaluation to understand the meaning of metrics, such as downloads and social media mentions would have been useful. It is recognised that while passive knowledge dissemination increases access to evidence, it may not have any effect on uptake [42]. Future planning of wider dissemination with robust outcome measures and evaluation is needed.
Holding breakout sessions within one of the online public symposia worked well to engage wider public and identify their questions and priorities; public participants engaged in the follow-up processes including ranking their top 10 questions, which were included in the WCEC question prioritisation process.
To make the work of the WCEC more widely accessible, we produced a public facing legacy report [43]. This includes information about the Centre structure, methods, reviews, knowledge mobilisation and impact in an accessible format. The lay summaries accompanying each review were written by members of the WCEC public partnership group, and together with the infographics, increase accessibility of the review findings.
A suggestion for wider dissemination of findings in a different format was made by a public member (e.g., review findings relevant to children and young people could be made into a poem). We acknowledge that wider dissemination methods such as creating stories, poems or videos would likely be valuable to make findings more accessible and engaging to wider groups. However, this was not possible at the time due to resource constraints (available staff time and additional funding). Videos of our methods relating to knowledge mobilisation, impact and rapid reviews are now available on our new Centre website [41].
Step 4: Associated activities
Involving students in the work of the WCEC was valuable for both the WCEC and students. The students gained experience not only of conducting systematic reviews but also experience of knowledge mobilisation processes and of working with and providing evidence that informed stakeholders. Both the reviews and infographics produced by students were a valuable addition to the Centre outputs, and the students presented their review findings at Welsh Government evidence briefing sessions. The more in-depth requirements for academic student projects may not suit evidence synthesis where evidence need is very urgent as projects typically take longer to complete.
Identifying further research needs and gaps in evidence reviews should be valuable for informing both researchers and research funders. The funding organisations have specific requirements for considering such further research and evidence priorities, which should be considered.
Step 5. Track and evidence impact
The best evidence of impact was for questions that came from the Welsh Government TAC and TAG groups, where the evidence was required to inform a specific plan, programme or guidance, and where knowledge mobilisation and gathering impact evidence was supported by the WCEC liaison person and stakeholders from the outset. While impact could be evidenced to the point of informing health and social care decision-makers (a recognised impact outcome) [15], evidence of how the research benefited public and patients would require future tracking and data analysis, which was not achievable within the timeline of the WCEC (2021–23), but which will be ongoing as part of the new Evidence Centre work. The unique environment of the pandemic and the close collaboration and co-production helped outputs to be used to inform decisions. However, moving forward, gaining a better understanding of barriers and facilitators to evidence use, how policymakers process evidence, and how other factors influence their decisions, will be crucial [11, 14].
Closer ‘knowledge brokering’ and embedding people to increase cross-community interactions may increase utilisation of evidence and impact [40]. Trade-offs between the resources required to enable this and increase in impact would be interesting to evaluate.
The framework for our knowledge mobilisation processes
The ‘Knowledge to Action’ (KTA) framework was used to inform our model and processes, as the stages aligned to many of our intended processes, and the model could be adapted to our rapid requirements (Fig1). The original KTA framework includes a knowledge creation process and a seven phase action cycle including 1) identifying the problem or issue to change, selecting knowledge to address the issue, determining the gap between knowledge and practice, 2) adapting the knowledge to your context, 3) assessing barriers and facilitators to knowledge use, 4) selecting an implementation strategy to make changes, 5) monitoring knowledge use, 6) evaluating outcomes and 7) sustaining the change or use of knowledge [6]. The time constraints and volume of evidence reports limited our implementation of some of these actions. However, our experiences and learning, in addition to reflection on other models of knowledge mobilisation and communication will allow us to evolve our processes as we move forward [42].
Lessons learnt by the Evidence Centre
As demonstrated by the example of the WCEC, it is possible to rapidly set up a new evidence unit, which can provide timely evidence to policy and practice decision making. While this was facilitated by the urgency of the pandemic, the model and lessons learnt are valuable to inform future initiatives and good practice during crisis situations.
During the lifetime of the WCEC (2021–2023), we captured challenges, enablers, and lessons learnt in relation to our knowledge mobilisation and impact processes. Our reflections are provided in Table 8.
Lessons that will be taken forward include i) refining our engagement and communication with policy and practice decision makers, and ii) involving stakeholders in the development of the knowledge mobilisation plan from the beginning of the collaboration, developing robust and longer-term evaluation plan from the outset to gather insights and ensure relevance. We will also plan activities to better include and engage members of the public especially from under-served communities and those digitally excluded in Wales. Getting a public voice and stories of personal experiences would likely be a powerful addition to the evidence [7]. This should be feasible when not having to meet the quantity and urgency of work that needed to be undertaken during the public health emergency of the pandemic.
Strengths and limitations
Many of the strengths and limitations are described for each of the knowledge mobilisation steps above and in Table 8.
Strengths included the close involvement and co-production with stakeholders, including public partnership members. This ensured that evidence outputs were targeted, timely and used.
The WCEC process evaluation survey to better understand use and impact of the evidence reviews is a limitation and while providing some insight, could not contribute to robust evaluation. Methods to better evaluate different dissemination methods to assess their impact and effectiveness should have been planned and implemented as part of the processes from the start.
A further limitation warranting wider debate is what is recognised as impact. While rapid reviews could count as impact for informing policy and practice, the impact requirements for academia need to be further considered. Academia necessitates the publication of research in peer reviewed journals for both career development, and to underpin impact cases included in the UK Research Excellence Framework (which evaluates research impact of British Higher Education Institutions). While it is possible to further develop the rapid reviews in line with requirements of peer reviewed journals, this was only possible for 2 reviews during the lifetime of the WCEC, primarily owing to time constraints.
5.0 Conclusion
The WCEC demonstrated that an Evidence Centre could be set up within a rapidly moving pandemic, with knowledge mobilisation processes ensuring that evidence reviews successfully informed health and social care decision-makers. Lessons learnt will be useful for informing future set-ups of such Centres during crisis situations.
The value of the WCEC work to inform evidence based decisions and practice has been recognised with a further five years of funding (from April 2023) as the ‘Health and Care Research Wales Evidence Centre’ [41].
Acknowledgments
We would like to acknowledge the WCEC research partners from the Wales Centre For Evidence Based Care, the Specialist Unit for Research Evaluation, the Public Health Wales Evidence Service, Health Technology Wales and the Bangor Institute for Health and Medical Research, the BioComposites Centre, and the Cedar Health Technology Research Service for conducting the evidence reviews and presenting findings at stakeholder meetings including the Welsh Government evidence briefings and WCEC public symposia. We would like to thank our students Tomi Adewole and Kismet Lalli for their work, and also Peter Bragge, Lauren Elston and Sophie Hughes for their help in designing the stakeholder survey.
We would also like to acknowledge the members of our Public Partnership Group (PPG) and Barbara Harrington and Julie Hepburn from the Wales Primary and Emergency Care (PRIME) Service Users (SUPER) group for writing the lay summaries. Also, to members of the PPG for attending meetings with stakeholders and evidence briefings, Deb Smith for supporting our public engagement event, and the Welsh Government TAG secretariat in arranging evidence briefings.
We would like to thank all our stakeholders for giving their time and expertise and participating in knowledge mobilisation.
Lastly, we would like to thank the Health and Care Research Wales communications team for their support of the WCEC.
References
- 1. Whitty CJM. What makes an academic paper useful for health policy? BMC Med. 2015 Dec 17;13(1):301. pmid:26675206
- 2. Pacheco-Barrios K, Fregni F. Evidence-based decision making during COVID-19 pandemic. Principles and Practice of Clinical Research Journal. 2020 Apr 27;6(1):1–2. pmid:32766450
- 3. Arvanitakis M, Karacic J. 10 Minutes With Prof. Marianna Arvanitakis, The Winner Of The European Award For The Best Medical Practice In The COVID-19 Pandemic. Global Bioethics Enquiry. 2023;11(1):73–5.
- 4.
National Institute for Health and Care Research. https://www.nihr.ac.uk/documents/knowledge-mobilisation-research/22598#introductory-resources-for-knowledge-mobilisation-and-planning-your-pathway-to-impact. 2019. Knowledge mobilisation research.
- 5. Ormstad H, Jamtvedt G, Svege I, Crowe S. The Bridge Building Model: connecting evidence-based practice, evidence-based research, public involvement and needs led research. Res Involv Engagem. 2021 Dec 30;7(1):77. pmid:34717755
- 6. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, et al. Lost in knowledge translation: Time for a map? Journal of Continuing Education in the Health Professions. 2006;26(1):13–24. pmid:16557505
- 7. Andrews N, Gabbay J, Le-May A, Miller E, Petch A, O’Neill M. Story, dialogue and caring about what matters to people: progress towards evidence-enriched policy and practice. Evidence & Policy. 2020 Nov;16(4):597–618.
- 8.
What Works Network. What works? Evidence for decision makers [Internet]. 2014 Nov [cited 2023 Jun 22]. https://www.gov.uk/government/publications/what-works-evidence-for-decision-makers
- 9.
Lavis JN CCPGZA. Communicating clearly: Enhancing information-packaging mechanisms to support knowledge brokering in European health systems. Policy Summary 7 (BRIDGE Series) [Internet]. 2013 [cited 2023 Jun 22]. http://www.euro.who.int/en/about-us/partners/observatory/bridge-series/communicating-clearly-enhancing-information-packaging-mechanisms-to-support-knowledge-brokering-in-european-health-systems.
- 10.
Social Care Wales. https://socialcare.wales/research-and-data/getting-evidence-into-practice/our-evidence-offer. 2022 [cited 2023 Jun 22]. Our evidence offer. https://socialcare.wales/research-and-data/getting-evidence-into-practice/our-evidence-offer#:~:text=We’ll%20support%20local%20authorities,people%20have%20access%20to%20them
- 11. Tricco AC, Cardoso R, Thomas SM, Motiwala S, Sullivan S, Kealey MR, et al. Barriers and facilitators to uptake of systematic reviews by policy makers and health care managers: a scoping review. Implementation Science. 2015 Dec 12;11(1):4.
- 12. Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gøtzsche PC, et al. Increasing value and reducing waste: addressing inaccessible research. The Lancet. 2014 Jan;383(9913):257–66. pmid:24411650
- 13. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014 Dec 3;14(1):2. pmid:24383766
- 14. Cairney P, Kwiatkowski R. How to communicate effectively with policymakers: combine insights from psychology and policy studies. Palgrave Commun. 2017 Nov 28;3(1):37.
- 15. Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrative review. BMC Med. 2016 Dec 23;14(1):78. pmid:27211576
- 16.
UK Health Security Agency. Finding the Evidence: Coronavirus [Internet]. [cited 2023 Jun 22].
- 17.
McMaster Health Forum. COVID-END: COVID-19 evidence network to support decision-making [Internet]. 2021 [cited 2023 Jun 22]. https://www.mcmasterforum.org/networks/covid-end
- 18.
Wales COVID-19 Evidence Centre. Wales COVID-19 Evidence Centre [Internet]. 2021 [cited 2023 Jun 22]. https://healthandcareresearchwales.org/about-research-community/wales-covid-19-evidence-centre
- 19. Cooper A, Lewis R, Gal M, Joseph-Williams N, Greenwell J, Watkins A, et al. Informing evidence-based policy during the COVID-19 pandemic and recovery period: learning from a national evidence centre. Glob Health Res Policy. 2024 May 31;9(1):18. pmid:38822437
- 20.
Joseph-Williams N, Cooper A, Lewis R, Greenwell J, Doe E, Gal M, et al. Working with stakeholders to rapidly identify and prioritise COVID-19 health and social care evidence needs for evidence-informed policy and practice decision making: processes, results, and lessons from the Wales COVID-19 Evidence Centre. ResearchSquare. 2023.
- 21.
Joseph-Williams N, Williams D, Gee P, Hall B, Strong A, Kumar R, et al. Involving the public in research prioritisation and as research partners in the Wales COVID-19 Evidence Centre (2021–23): processes, research priorities, outputs, and lessons. Research Square. 2023.
- 22.
Welsh Government. Terms of Reference: Technical Advisory Cell. Welsh Government [Internet]. 2021 [cited 2023 Jul 12]. https://www.gov.wales/technical-advisory-cell/terms-reference
- 23.
Lewis R, Cooper A, Jarrom D, Mann M, Law RJ, Edwards D, et al. A bespoke rapid evidence review process engaging stakeholders for supporting evolving and time-sensitive policy and clinical decision-making: reflection and lessons learned from the Wales Covid-19 Evidence Centre 2021–23. ResearchSquare. 2024.
- 24. Richardson W.S W MC, N J AH RS. The well-built clinical question: a key to evidence-based decisions. ACP J Club. 1995 Nov 1;123(3):A12. pmid:7582737
- 25.
Wales COVID-19 Evidence Centre. The efficacy, effectiveness and safety of SARS-CoV-2 disinfection methods (including ozone machines) in educational settings for children and young people [Internet]. 2021 [cited 2023 Jun 23]. https://healthandcareresearchwales.org/efficacy-effectiveness-and-safety-sars-cov-2-disinfection-methods-including-ozone-machines
- 26.
Covid: Ozone machine school plans scrapped over safety. BBC News [Internet]. 2021 Oct [cited 2023 Jun 23]; https://www.bbc.co.uk/news/uk-wales-58917096
- 27.
Auskerry H. Covid: Plans to use ‘harmful’ ozone gas machines in Welsh schools scrapped.. ITV News [Internet]. 2021 [cited 2023 Jun 23]; https://www.itv.com/news/wales/2021-10-15/plans-to-use-harmful-ozone-gas-machines-in-schools-scrapped
- 28.
Wales COVID-19 Evidence Centre. What interventions or best practice are there to support people with Long COVID, or similar post-viral conditions or conditions characterised by fatigue, to return to normal activities: a rapid review [Internet]. 2023 [cited 2023 Jun 23]. https://www.medrxiv.org/content/10.1101/2023.01.24.23284947v1
- 29.
Sidharthan C. A review of interventions and support options for long COVID patients returning to work. Medical News [Internet]. 2023 [cited 2023 Jun 23]; https://www.news-medical.net/news/20230202/A-review-of-interventions-and-support-options-for-long-COVID-patients-returning-to-work.aspx
- 30.
Pears K. The ME Association: Research Roundup [Internet]. 2023 [cited 2023 Jun 23]. https://meassociation.org.uk/2023/02/me-cfs-research-published-23-30-january-2023/?utm_source=rss&utm_medium=rss&utm_campaign=me-cfs-research-published-23-30-january-2023
- 31.
Technical Advisory Cell WG. Technical Advisory Cell: Summary of Advice [Internet]. 2021 Sep [cited 2023 Jun 23]. https://www.gov.wales/sites/default/files/publications/2021-10/technical-advisory-cell-summary-of-advice-24-september-2021.pdf
- 32.
Technical Advisory Group. Technical Advisory Group Advice for 22nd April Restriction Review [Internet]. 2021 Apr [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2021-05/technical-advisory-group-advice-for-22-april-restriction-review_0.pdf
- 33.
Technical Advisory Cell. Technical Advisory Cell Advice on ‘Vaccine Passports’ [Internet]. 2021 Sep [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2021-09/technical-advisory-group-advice-on-vaccine-passports.pdf
- 34.
Welsh Government. Coronavirus Control Plan: autumn and winter 2021 update [Internet]. 2021 Oct [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2021-10/coronavirus-control-plan-autumn-and-winter-2021-update.pdf
- 35.
Technical Advisory Group. Technical Advisory Group TAG-E: Evidence review of ozone generators including appropriateness as mitigation within classrooms [Internet]. 2021 Oct [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2021-10/technical-advisory-group-evidence-review-of-ozone-generators-including-appropriateness-as-mitigation-in-classrooms.pdf
- 36.
Technical Advisory Cell. Technical Advisory Cell Summary of Advice 5 [Internet]. 2021 Nov [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2021-11/technical-advisory-cell-summary-of-advice-5-november-2021.pdf
- 37.
Welsh Government. Renew and Reform: Post-16 and Transitions Plan [Internet]. 2022 Mar [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2022-03/renew-and-reform-post-16-and-transitions-plan.pdf
- 38.
Technical Advisory Group. Updated advice from the Technical Advisory Group and Chief Scientific Advisor for Health on the evidence for the use of COVID Passes [Internet]. 2021 Dec [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2021-12/updated-advice-from-the-technical-advisory-group-and-chief-scientific-advisor-for-health-on-the-evidence-for-the-use-of-covid-passes.pdf
- 39.
Technical Advisory Cell. Technical Advisory Cell: Summary of Advice [Internet]. 2022 Jan [cited 2023 Jun 26]. https://www.gov.wales/sites/default/files/publications/2022-02/technical-advisory-cell-summary-of-advice-21-january-2022.pdf
- 40. Wye L, Cramer H, Beckett K, Farr M, le May A, Carey J, et al. Collective knowledge brokering: the model and impact of an embedded team. Evidence & Policy. 2020 Aug;16(3):429–52.
- 41.
Health and Care Research Wales Evidence Centre. Health and Care Research Wales Evidence Centre [Internet]. 2023 [cited 2024 Jul 11]. https://researchwalesevidencecentre.co.uk
- 42. Chapman E, Pantoja T, Kuchenmüller T, Sharma T, Terry RF. Assessing the impact of knowledge communication and dissemination strategies targeted at health policy-makers and managers: an overview of systematic reviews. Health Res Policy Syst. 2021 Dec 6;19(1):140. pmid:34865640
- 43.
Wales COVID-19 Evidence Centre. Wales COVID‐19 Evidence Centre Legacy Report 2023 Wales_COVID‐ 19_Evidence_Centre_Legacy_Report_June23_ENGLISH.pdf [Internet]. 2023 [cited 2024 Jun 23]. https://researchwalesevidencecentre.co.uk/ sites/default/files/2023‐11/Wales_COVID‐19_Evidence_Centre_Legacy_ Report_June23_ENGLISH.pdf.