Design

This study is a qualitative exploration of public health researcher perspectives. It was part of a wider exploratory impact assessment primarily to demonstrate accountability which was undertaken in 2017 on all research awards (n = 113) funded by the NIHR PHR programme between May 2009 and 14^th March 2017 [Lakin K, Baker G, Thomas S, Worswick L, Dorling H. NIHR Public Health Research Programme: Exploring the influence of research on policy & practice. Unpublished internal report; 2018]. The Payback framework underpinned the wider impact assessment. This framework uses a logic model of the research to create a multi-dimensional characterisation of paybacks resulting from research. This framework was selected as it has been used to evaluate the impact of other NIHR funding research programmes [8], and the PHR Programme had a specific interest in exploring two of the Payback categories, namely Benefits to informing policy, and Benefits to [public] health sector. For this study, public health researchers’ experience of the barriers, challenges and facilitators to impact was explored within this context to contribute to a collection of award level case studies building an understanding of the researcher-research user interface within the context of public health. A phenomenological qualitative approach underpinned the study as we were interested in describing and understanding the experiences of our participants. Data were then collected using semi-structured interviews that were informed from the Payback framework (see S1 File).

Recruitment

A purposive and self-selecting sample was used to identify principal investigators from NIHR PHR programme funded research awards which 1) had demonstrated evidence to suggest that the research findings had been influential to policy and/or the [public] health care sector following a desk-analysis of the PHR portfolio based on the Payback framework [9, 10] and 2) were completed and the findings were published within the NIHR PHR Journal. Research awards which had been influential to policy and/or the [public] health care sector were targeted because this allowed an exploration of proven facilitators and barriers to the potential impacts being realised. Award selection based on completed research was most appropriate given the time-lags that often exist between completing the research and realised influence [21, 22].

From a total of 113 PHR awards available, nine principal investigators from different PHR funded projects were eligible and invited to participate in an interview or to nominate another member of their research team. Invitations were sent out by email by a member of the project team.

Ethics approval and consent to participate

Although not standard process within research impact assessments, ethical approval from the Faculty of Medicine University of Southampton Ethics Committee was sought (ID 27460). Study participants were informed about the objectives of the study, any risks and benefits to them, the fact that participation was entirely voluntary and that they could withdraw from the study at any time. Verbal consent was obtained, recorded and stored separately before the start of each interview, as approved in the ethics application. Verbal consent was obtained as this was felt to be the most practical approach given that all interviews were conducted via telephone. Participants were sent a copy of the information sheet and consent form in advance of the arranged interview and were asked to verbally confirm consent. Participant names were anonymised in the interview transcripts. Records linking participant names, consent and transcripts were stored according to the approved protocol.

Data collection

Semi-structured telephone interviews were undertaken (Aug—Sep 2017) with one interviewer and one note-taker and lasted up to one hour. The interviewer and note-taker undertook the interviews together in a private room at the NIHR Evaluation, Trials and Studies Coordinating Centre in Southampton. The interview framework was adapted from [17], which aligned to the Payback Framework, and sought to understand the nature of the influence or impact the research had realised. However, it also sought to understand how change came about or what stopped it, thereby investigating the way in which knowledge from research was mobilised. The initial interview framework was sent to participants in advance of the interviews, and a question was added to the framework following the first interview to reflect new areas for exploration in further interviews.

Analysis

Interviews were transcribed by a transcription service and reviewed for accuracy. NVivo 11 qualitative data analysis software was used to support the thematic analysis. Data were analysed according to [23]. Following familiarisation, transcripts were coded. A second coding exercise led to the identification of common themes and sub-themes. Through an iterative process, themes and interpretations were reviewed and discussed by the research team to refine themes through merging, overlapping and separating out key concepts. The coding process was inductive as no prior themes or framework was considered and the themes were data driven. During the analysis, authors were mindful that they work within the NIHR and have a range of expertise in public health, health services research, health evaluation and commissioning, which strengthened the analysis and minimised bias. No one from the project team had a relationship with the participants prior to study commencement. Quotes were reported verbatim. The COREQ guidelines (COnsolidated criteria for Reporting Qualitative research) [24] for reporting qualitative data were utilised throughout and used as an additional reporting quality check.

Consent for publication

All participants consented for their interview data to be used as part of a thematic analysis which would be published. As good practice, a copy of this manuscript was sent to all study participants to inform them of our plan to publish via a journal article and ensure that the interpretation reflected the discussions during the interview.

1. Public health researcher’s perception of the purpose of the research

This theme suggests that how public health researchers perceive impact and KMb is guided by their views of the purpose of research. There was a variety of views regarding what was considered impact, how to achieve impact and whose responsibility it was to demonstrate impact. Researchers discussed impact and KMb in terms of publications, engagement with the public, policy makers and the media, dissemination, patient and public involvement, changing practice, guidelines or policy, and involvement in the UK REF.

And we’ve tried to illustrate, you know, how big an impact we have had, not just in terms of, you know, the number of publications, which has been, yes, much bigger than I think we had anticipated, but also the wider, kind of, policy impact. Participant 4

These preferences to impact and KMb sat on a continuum that ranged from a fixed pipeline approach to a more integrated, collaborative perspective. Researchers at the pipeline end of the continuum preferred to produce knowledge and evidence via publication that was then available for others to use. This preference tended to give less emphasis to the researcher being involved in how research findings were used in the world after project completion.

I think my job is to produce some knowledge, and then it’s, kind of, not really my skillset to think: well… You know, it’s also none of my business how that gets used in the world, it’s just not what I think I my job is. Participant 1

By contrast, at the opposite end of the continuum, researchers preferred to undertake their research in a more collaborative and engaging way to bring about change within communities or by influencing changes to policy or practice. These researchers felt that undertaking research was a collective responsibility.

Participatory research, characterise impact as change that happens during the process as much as what happens beyond it. And if you involve people, […] the impact will go beyond, be more likely to happen because people own the research. So people who have been involved will own it and, therefore, more likely change what they do. Participant 6

Most researchers seemed aware of their preferences on this continuum. However, they also understood the importance of different activities in facilitating impact and KMb of their research. As such, despite their preferences, researchers still engaged in or employed strategies to ensure different types of activities were undertaken (see theme 2).

Preferences on the continuum were discussed within the context of objectivity and advocacy. It was suggested that the boundary between advocacy and objectivity was sometimes difficult to navigate, with researchers indicating that certain activities aligned them more to one or the other. This could act as a barrier to researchers discussing and planning impact as they sometimes felt uncomfortable attaching themselves to findings or speculating on potential impacts.

Some researchers, I think, tend to lean more on the advocacy side than the objectivity side, but we’ve been really careful to try and make sure we can provide evidence when people want it, but… we remain objective, not advocates. Participant 4

2. Approaches to undertaking knowledge mobilisation activities

This theme encompassed the strategic approaches that researchers used to facilitate impact and KMb. There were three subthemes (2.1.) strategy and opportunism; (2.2.) enduring networks and partnerships; and (2.3.) role and skills of the team.

3. The complex nature of public health research in the wider research context

The nature of public health research is complex. Under this theme there are two subthemes that highlight the facilitators and barriers to KMb and impact delivery within the wider research context: (3.1.) the role of the research funder and (3.2.) the role of wider environmental and societal context. These subthemes interact with the research activities outlined in theme two.

Linking findings to theory

Our findings highlight areas for consideration for the Payback framework with regards to public health research. Fig 1 highlights that for stages 0–3 of the framework (research topic identification and research activity processes) our findings emphasise the need for and role of the research support community, such as HEIs and researcher funders, to support researchers to plan and undertake KMb activities. Interface B (Dissemination) highlights the need to take advantage of windows of opportunity to inform stage 4 (policy developments), which in turn present opportunities for adoption (stage 5). Networks and partnerships play an important role throughout all stages of the framework and are particularly important for maximising the adoption of research findings within public health contexts. Finally, our findings reflect the researcher perspective in terms of preferences, role and skills and how they can influence if and how they plan for and undertake KMb activities.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Study findings mapped onto the Payback Framework Logic Model for assessing research impact.

The Payback Framework Logic Model with areas for consideration resulting from this research highlighted in bold. The framework was adapted from [10].

https://doi.org/10.1371/journal.pone.0268675.g001

Context is important in public health research

Researchers in public health operate within a complex system where health outcomes are related to a multitude of interdependent environmental, social and physical determinants [26, 27]. Factors such as changes in the political landscape, public opinion and organisational restructures present challenges to the research environment. This means that the context in which a public health researcher works (across third sector, multi-government agencies and private sector) is likely different to other types of health research. Context has been recently highlighted as an area that should receive greater consideration to aid understanding of population health interventions [27, 28]. The importance of context has been reiterated by public health researchers in Australia [16] and UK [18, 27]. Our findings mirror this complexity when researchers discuss the need to navigate context and to ensure their research is relevant. For example, researchers reported that evidence did not always follow a linear evidence-based medicine model and there were concerns raised by participants about how to demonstrate and evidence impact, particularly when it is conceptual (i.e. adds to a body of knowledge) in nature.

Whilst [17] found that researchers questioned the type of evidence needed from public health research to inform policy, our findings suggest that researchers were aware of the type of evidence needed and were using non-standard research designs to ensure that the research generated relevant evidence. Evidence derived from realist reviews, natural experiments, non-randomised trials, novel statistical methods and complex systems approaches provide alternative ways to explore complex interventions [26, 29] and some of these approaches were used by the researchers. This is because in complex social interventions, isolating health outcomes in order to measure statistically observable effects is difficult and often not the main goal. Moreover, there are practical challenges in trying to randomise service evaluations and population-level interventions, where other forms of evidence maybe better suited [26, 30–32]. Despite this, we found that researchers feel frustrated by the research funding process and the inability to undertake research that is fit for purpose. Indeed, as pointed out in [30], there is a mismatch between the type of evidence required by public health organisations, those who develop evidence-based guidance, and the types of research that funders are willing to support. The findings in [17] and in our paper seem to evidence this mismatch.

Researchers recognised the need to collaborate and/or engage to ensure that the research is positioned within the right context. However, in line with [16], a lack of capacity (skills and time) to do so was a barrier to effective KMb (and therefore impact). Public health researchers need to engage with a broad range of stakeholders who often have different needs, priorities and entrenched practices. Researchers undertook a range of activities to navigate this space, but our findings suggest that more support is needed to help public health researchers with these activities.

Researcher preferences to knowledge mobilisation activities can facilitate navigation of the public health context

Our findings show that researcher preferences to impact sat on a continuum from a ‘traditional academic’ to a coproduced approach. This type of continuum has been reported by others [17, 20, 33]. Building on this, we found that the continuum for PH researchers is influenced by views of academic rigour, emphasising objective generation of evidence on one end and co-production and engagement activities on the other. Influential PH researchers in Australia rarely just let research speak for itself and instead are aware of the applied nature of PH research [20]. Our findings lend weight to this, and even where a researcher’s preference could be viewed as a more traditional academic (i.e. to produce good quality evidence), their research, by its very nature, had policy goals or was policy-driven i.e. “policy-makers identify and prioritise particular problems that then become the focus of research”, [28 p. 28]. Similar to [16], public health researchers in this study had a strong awareness of professional identity around researcher role and skill-set, and how they viewed their contribution to society. By contrast, we found no evidence that they viewed undertaking KMb activities as a competitive advantage to their peers. In general, however, our findings suggest that researchers were aware of their preference, which often facilitated how they navigated the complexities of the context in which they work. Such awareness drove different approaches and strategies to impact and KMb. However, as concluded in [16], while researchers in applied research are increasingly expected to engage in KMb activities, this has not been followed by funding or other incentives and rewards which help to overcome barriers. Similar findings have been reported for PH research funded in the UK [17, 27]. One way in which funders could support researchers is to allow for flexible impact plans and strategies in order to respond to the changing complex environment. Moreover, our findings suggest that researchers believe that funders could be clearer on the intended changes or impacts of the research they are funding, and what evidence, data or indicators they need from researchers to demonstrate this change.

Public health researchers need to seek out windows of opportunity and engage with a wide range of stakeholders

One way in which almost all researchers navigated the public health context was to be aware of windows of opportunity to engage with policy and key stakeholders, and that this was a facilitator to impact. Changes in the policy and health landscape opened up opportunities for researchers to engage. Such opportunities also presented challenges. Indeed, partnerships can be easier to forge when the policy area is a ‘hot topic’ and therefore high on the political agenda but this makes sustaining partnerships in long term more challenging [34]. Our findings suggest that researchers were aware of the need to expand the composition of the research team to take this into account including collaborating (and coproducing) with key stakeholders, building on skill sets to engage or building on existing networks and partnerships. As found in [18], researchers evaluating public health interventions often had to change/shift their role depending on the context. A systematic review of KMb in the third sector found that strong relationships with academia were important to the mobilisation of research evidence and that having the ability to coproduce research could help to overcome some of the barriers by ensuring that the research was applicable to a local context [35]. Other strategies included using local or national research brokers. Knowledge brokers are being increasingly used to facilitate KMb between the producers and users of evidence to overcome the struggles of mobilising research knowledge into policy and practice. A number of different aims for knowledge brokers have been identified to facilitate KMb [36]. Mobilising knowledge to inform decision-making takes time but an important aspect for a broker is the ability to respond to windows of opportunity (e.g. hot topics, capitalising on chance encounters and creating networking opportunities) while at the same time hold evidence to inform future policies [34]. Despite such strategies, our findings are in line with others [17, 27] and suggest that there is a need to provide more support for researchers such as the provision of training and building of skills to help with the range of stakeholders that relationships need to be forged with and the different ways in which the research is communicated to different audiences.

What this study adds

The recent yet increasing interest around impact to measure research performance is presenting challenges to stakeholders involved in research. This study contributes to the field by shedding light on challenges PH researchers in the UK are struggling with in defining, generating and demonstrating impact, and more importantly, what actions could be taken to develop the skills needed to track and capture impact from research.

This work is presented from the point of view of the PH researchers and provides a snapshot of the landscape of how impact operates in this context. Recommendations arising from the researcher’s perspective here suggest that funders and research organisations should consider how best to support research teams. Such support could include specifying expectations towards impact, being open to the different methodologies that researchers in this field can adopt, provision of funding to support and enable KMb activities, committing to develop the skills that researchers need to engage with a wide range of stakeholders, and to evaluate track and capture impact from their research. This study has been laid out as a platform to encourage further work in the field. To develop a full picture of the views and interactions of all stakeholders, next steps would be to compare the views here with those from evidence-users as well as those who receive the benefits of research, to investigate whether impact is perceived in the same way by all stakeholders. Future research could consider the role of non-academic public health professionals working as knowledge brokers or as part of research team in increasing the impact of research.

Our findings highlight areas for consideration on the Payback framework specifically in the context of researchers undertaking public health research. Our findings have been mapped onto the framework emphasising the additional elements researchers should consider when planning for impact. As reflected in the model, our findings suggest that while the interface between researchers and research users is important, this is more complex than a two-way interface. The role of the research funder and the research institution, is important in this space to ensure that the right risks are being taken to fund the right research, with a team with the right skills in KMb (and provide training and support to achieve this). This will be of interest to anyone who is undertaking a theory driven impact evaluation or assessment of public health research to support what factors could influence the resulting outcomes and impacts of the research.

Since the introduction of the PHR programme, responsibilities for commissioning and providing public health services in England were transferred from the NHS to local authorities [37]. Although such transfers of responsibility may not have resulted in changes to the research environment for the researchers interviewed in this study, it is possible that it may open up new opportunities to engage in evidence-based policy-making, which is applicable to public health [38]. The number of interviews was small, and it is possible that a larger sample size might yield additional findings. We only looked at research which had completed but we acknowledge that impact may have resulted from active research. The research was undertaken from the perspective of the NIHR; participants were chosen from studies funded by the NIHR only and other funders might have different support provision for researchers in public health programmes. In addition, the authors work within the NIHR and although our selection of quotes gave a broad range of opinions, we do not know whether this factor might have influenced the openness of the participant responses. Our findings relate to public health research, specifically research which is undertaken outside of the NHS, and may not be generalisable to public health research conducted in NHS settings.