Introduction

In 2018, for the first time, Australian universities will be assessed on 1) the extent to which they engage with industry, government and other research end-users, and 2) the impact that their research has had on the world beyond academia [1]. The Australian Engagement and Impact Assessment runs in parallel to the established Excellence in Research Australia (ERA) exercise, focused on research funding, publications, citations and peer review [1]. While impact assessment has been on and off the Australian Government agenda for some time, it was the United Kingdom that implemented the world’s first national impact assessment in 2014 [2,3]. The UK Research Excellence Framework (REF) included an impact assessment based on detailed case studies of the impacts research has had beyond academia that could be traced to an underpinning research base [4].

The UK’s action on research impact assessment spurred further Australian developments. In 2015 the Academy of Technology, Science and Engineering (ATSE) ran the Research Engagement Australia (REA) pilot. The REA pilot developed and tested income-based metrics to demonstrate engagement between academia and industry. Focusing on engagement metrics addressed two key criticisms of the UK’s case study approach to impact assessment [5]. First, metrics that use existing data sets minimise burden on the academic sector in regards to collecting evidence of impact [6,7]. Secondly, while impact can take a long time to be achieved and is not in the direct control of academics [3], engagement metrics based on industry-funded research demonstrate vested relationships with industry, and can indicate potential future impact. In 2015, the Australian Government also commissioned a review of research funding which recommended that assessment of research impact be added to national research performance assessments [8].

In December 2015 the Australian Government launched the National Innovation and Science Agenda (NISA) that announced the introduction of a research impact assessment [9]. The purpose of the impact assessment, as outlined in the NISA, was to incentivise higher education institutions to increase engagement with industry and government to increase the positive impacts that research has on the economy and society more broadly. The central hypothesis of this approach is that increased engagement drives increased impact. This premise is supported by research that has identified close collaboration, increased interaction as well as access, trust, relevance and communication as key factors that affect research use by decision-makers in industry and government [10–14].

The problem with focusing solely on engagement as an indicator of future impact is the failure to acknowledge the significant challenge of getting decision-makers to use research evidence. To get from engagement to impact, research must be used in some way. Research must become an input to the decision-making, development, production, implementation or use of policies, programs, practices, products, services, etc., the mechanisms that deliver impact beyond academia. Research use has been defined as the application of research evidence and expertise to inform decision-making within planning, development, implementation and evaluation processes [15–19]. Studies of government decision-makers use of research have shown that different types of research evidence are used. Decision-makers directly commission research relevant to an issue they are facing, search for existing individual studies or literature review or systematic reviews of research, and decision-makers also seek out expert evidence-based advice directly from academics who are recognised experts in their field [16,20–24].

There are also different ways in which research evidence is used. Types of research use have commonly been categorised as: instrumental use, which refers to direct application of research-based evidence; conceptual use, referring to use of research evidence to inform understanding, discussion and/or debate without direct application; and tactical, political or symbolic use, which refers to the use of research evidence to inform a decision or position previously established [10,13,15,25–28].

The definitions and types of use outlined above align with the Australian Research Council definition of impact, which built on the UK’s definition of research impact [3]: ‘The contribution that research makes to the economy, society, environment and culture beyond the contribution to academic research’. This aligns with the academic definitions and types of use outlined above. It suggests that research must be used in some way that contributes to the world beyond academia, but does not specify a specific type of use or contribution. Impact case study templates used in both the UK and Australia require researchers to describe how their research has been used outside of academia how that use represents an important contribution [2, 23].

Engagement has been clearly differentiated from use in the work of Redman et al. [15,29,30], who define engagement as actions ‘that create a bridge between the potential reflected in the capacity to use research and the eventual outcome of research application’ [15]. The definition of engagement adopted for the Australia research impact assessment is ‘The interaction between researchers and research end-users outside of academia, for the mutually beneficial transfer of knowledge, technologies, methods or resources’ [31]. Here, engagement refers to how research-based outputs are transferred to, accessed and shared between researchers and users of research outside of academia. This aligns to Redman et al [15] identification of research access, research appraisal and interactions between decision-makers and researchers as engagement actions.

The 2017 Innovation and Science Australia Performance Review found there are limited mechanisms to support research translation and collaboration between researchers and industry [32]. Research translation and implementation science are fields of research that have developed in response to the substantial challenges faced by those who have attempted to increase use of research in industry and government [33,34]. There is an extensive research base on the factors affecting decision-maker research use [10,11,35–40]. There has also been extensive work in using identified factors and relevant theories and frameworks [41–44] to design interventions aimed at increasing use of research [12,14,18,41–45]. Many of these frameworks are focused on: increasing interaction and collaboration between researchers and decision-makers, for example through co-production [41–44] of research and implementation projects; increasing access to and accessibility of research, e.g. through synthesising research and improving research communication; and also building capacity for decision-makers to engage with research evidence and researchers and vice versa, often though training and facilitation [12,14].

While it has been established that many factors can affect research use and many interventions developed, high quality empirical research testing interventions that aim to increase decision-makers’ research use is more limited [12,14,29,45]. A recent systematic review of research covering the years 1990 to 2015 identified just 16 studies that specifically sought to test interventions aimed at increasing decision-makers’ use of research [12]. Seven of these interventions effectively increased research use, the other nine failed to show any change in behaviour. The authors concluded that ‘skill or the intention to use evidence, in itself, cannot be regarded as a reliable indicator of behaviour change in practice’. A complex interplay of factors can affect research use, which can make the transition from engagement to research use and subsequent impact extremely difficult. Engagement is only a first, albeit necessary, step on a challenging pathway to impact that has not yet been extensively mapped, with no definitive means to effectively navigate.

The systematic review by Langer (2016) [12] showed that interventions with the most reliable evidence of increasing decision-makers’ research use are those that facilitated research access and communication, and built skills for research use [12]. This is supported by the recent review by Sarkies et al. (2017) [14] which identified three studies with experimental designs that tested interventions. Two of the studies were focused on access and communication. The other focused on use of knowledge brokering to support decision-makers to use research. Access mechanisms used in these effective interventions were mainly online evidence portals designed for specific research fields and systematic or other literature reviews that summarised evidence for decision-makers. Targeted and tailored research messages were the most effective communication approaches. While there are many factors affecting research use that are not directly in the control of academia, research access and communication are factors that academics and research institutions can increase and improve.

There has also been a great deal of work on increasing access to research through the open access publishing movement, however there are a limited number of studies that have examined the use of open access academic research by those outside of academia [46,47]. A recent review found just 53 studies published between 2001 and mid 2017 on use of open access research by those outside of academia [46]. The two largest and most comprehensive of these studies were on use of institutional research repositories [48]. These studies showed that there are people outside of academia that seek out and use research across a range of fields for both work-related and personal reasons [46–48].

It is clear from the Engagement and Impact Assessment in Australia and much research in the field of research translation that direct engagement, interaction, collaboration and co-production of research are needed to support increased use and impact of research [12,14,15,31,41,49,50]. For example, the engagement indicators used in the the Engagement and Impact Assessment in Australia are based on establishment of research partnerships, and collaboration and co-production require researchers and decision-makers or end-users to directly work together. The evidence that people are accessing and using research outside of direct engagement and translation efforts [46–48], coupled with the consistent finding that access and accessibility are critical factors enabling research use [11,12], highlights the need to better understand independent research use enabled by open access platforms and identify whether this type of use also has a relationship with indirect engagement actions.

Materials and methods

The Conversation developed The Conversation Annual Survey 2016, with input from Dr Zardo. Dr Zardo led inclusion of questions on use of research that forms the focus of this study (see S1 File). The survey was made available to readers via The Conversation Australian website and through their online newsletter mailed to subscribers. It was also advertised via The Conversation social media. The survey was open from 7 to 15 April 2017. Survey Monkey was used to distribute the survey and collect responses. The method of survey collection was non-random and we cannot calculate a survey response rate or compare the characteristics of respondents and non-respondents.

A low risk human research ethics application was approved by the Queensland University of Technology University Human Research Ethics Committee to analyse the data. The first survey question asked consent for the data to be used in subsequent reporting. Eighteen people (0.23% of total sample) did not give consent, leaving a total sample for analysis of 7,772 consenting participants for analyses.

Data on use of research and academic expertise reported in The Conversation articles were generated from the survey question ‘Have you used articles from The Conversation to do any of the following?’ with the following response options, each asking about a different type of use (tick as many as apply):

• Develop strategy, policy, presentations, decisions and/or directions which have been documented, for example, in policy briefs, papers, projects plans or reports, PowerPoints, etc.
• Further support existing an strategy, policy, program or business decisions
• Inform general understanding, discussion and debate on strategy, policy, project or business topics
• Change my own behaviour and/or attitudes in my personal life

These are the four measures of use that we aimed to predict. The first three types of use align to the instrumental, tactical/symbolic and conceptual types of research use discussed in the introduction and are specifically related to use of articles in relation to work-related decision-making. The fourth seeks to identify use of research to inform one’s personal decision-making.

Data on engagement actions were generated from the question ‘What actions have you taken as a result of reading an article on The Conversation?’ with the following response options (tick as many as apply):

• Republished the article
• Left a comment on the article
• Shared an article on social networks (e.g. Facebook, Twitter) or by email
• Discussed with friends or colleagues
• Printed to read or share
• Contacted the author to discuss their ideas
• Contacted the author to work with them
• Contacted the author to ask about studying with them or at their university
• Used the article in a report
• Used the article as a classroom resource or as basis of discussion with students
• Contacted a local politician or government official
• Undertaken further research
• None

The actions listed here represent dissemination, communication and other activities undertaken by the reader to engage others or the author, or engage withThe Conversation content, or led to engagement with other content. Each response to these questions was automatically made into an individual variable with a binary (yes/no) outcome.

The survey also asked about: age, gender, state of residence, place of residence (city, regional, rural), education, income, sector worked within; employment position; business owner and business type.

Results

Overall results of the regression analyses are presented in Table 1 below. The results of the logistic regression are discussed together with the results of the classification tree below. Please note this table only includes predictors that showed a realtionships to the type of use outcome.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Predictors of use of a The Conversation article.

Positive predictors are in green and negative in orange.

https://doi.org/10.1371/journal.pone.0192290.t001

The total sample of (7772) had an overall probability of:

• 15% for using The Conversation articles to inform development of strategy, policy, programs, etc.
• 17% for using The Conversation articles to support work-related decisions previously made.
• 71% probability of using The Conversation articles to inform work-related discussion and debate.
• 53% of using information read in The Conversation articles to inform personal attitude or behaviour change.

Discussion

Analysis of The Conversation 2016 Annual survey has yielded novel insights on how information in The Conversation articles is being used to inform readers’ decision-making in their work and personal lives. This study makes an important contribution to the high quality quantitative literature on factors that predict decision-maker use of academic research evidence and expertise [11,16,58]. In particular this study draws on data from a broad range of sectors and included clearly defined types of use of The Conversation articles addressing issues identified as challenges to the interpretation of outcomes in previous research [11,12]. Further, this study has included a focus on use of The Conversation articles in personal decision-making and shown that the factors that predict individuals’ use of The Conversation articles in personal decision-making differ from the factors that influence work-related decision-making.

This study set out to answer two questions: do engagement factors predict use?; and what other factors predict use?. The key factors that predicted work-related use of The Conversation articles were related to engagement actions taken after reading an article, being employed in senior-level and policy-related employment positions. Work related use showed much higher growth in predictive capability with additional factors, suggesting it is more dependent on a number of specific factors working in cooperation. This aligns with previous research focused in health policy contexts, which found that the predictive power of factors found to affect research use improved when factors were ‘manipulated simultaneously’ [59]. Of the three key factors that affected personal decision-making regarding attitude and behaviour change, two were related to participants’ main reasons for reading The Conversation, and one was an engagement action. Importantly, this study demonstrates the potential for end-user surveys to form part of the suite of metrics seeking to demonstrate research engagement and impact as part of national research performance assessments [3].

Key predictors of research use

The advantage of using both the logistic regression and classification tree analyses is that logistic regression identifies factors that independently predict use and the classification tree highlights relationships between factors that predict use. This is important as it has been highlighted that the lack of identification of relationships between variables is a limitation of regression analyses [36]. As Table 1 demonstrates, a broad range of factors significantly predicted the each type of article use across the two methods. For sake of brevity, the remainder of the discussion will focus on factors identified as statistically significant predictors across both logistic regression and classification tree analyses. Table 2 provides an overview of the factors that predicted each type of research use across both regression methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Overview of key factors predicting use of The Conversation articles.

https://doi.org/10.1371/journal.pone.0192290.t002

Instrumental use of The Conversation articles.

‘Used the article in a report’ was the strongest predictor of use of The Conversation articles to develop strategy, policy, presentations, decisions and/or directions, etc. across both regression methods. Using research to directly inform development of policy, program, practice and other action has been described as ‘instrumental’ research use [10,13,15,25]. Importantly, this finding indicates that The Conversation articles have been relevant to decision-makers needs and actionable within their work-related context. This means that The Conversation is overcoming some of the most commonly identified barriers to decision-maker use of research: access, relevance, actionable outcomes, and timeliness [11,40,63].

Finding that use of an article in a report predicts use of an article to inform development of policy, strategy, project etc., makes intuitive sense, as these are both direct, instrumental uses of research. In testing assumptions for inclusion of predictors, ‘used article in a report’ was included because it was correlated with the outcome variable but was not highly correlated to other predictors which is a necessary assumption for effective logistic regression analysis [64]. Further, the outcome ‘using article to inform policy, strategy, program, project documents, etc.’ includes a number of different outputs, therefore it is possible that those who have used The Conversation article in a report are also independently more likely to be using research instrumentally in multiple outputs. Qualitative follow up research could provide greater detail to deepen understandings of examples of instrumental use of The Conversation articles.

Working as a politician, policy officer, or government employee also predicted use of The Conversation articles to inform the development of strategy, policy, programs, etc., i.e. instrumental use. This is an important finding that demonstrates that politicians and policy officers are actively seeking out research evidence and academic expertise on The Conversation and using it to inform policy and program development. No other employment type, out of a total of 16 employment types, significantly predicted direct, instrumental use of research. This study shows that publishing in The Conversation offers a unique opportunity to directly inform and influence politicians and policy officers, with real potential for that research to be used in government policy and program decision-making.

‘Discuss with friends and colleagues’ is an engagement action that significantly predicted use of The Conversation articles to support existing strategy, policy and other work-related decisions. ‘To assist with work’ was the only response from the survey question ‘what are the main reasons you read The Conversation’ that predicted instrumental use of a The Conversation article. Together, these results highlight that people are accessing The Conversation for work-related purposes and further strengthens the assertion that participants’ find The Conversation articles content relevant to their work-related decision-making needs and actionable in their context [58,63,65]. The results strongly suggest that The Conversation model is an effective means for both providing access to research, as well as communicating research in a way that enables use of research, a necessary precursor to achieving research impact.

Republishing an article significantly predicted instrumental use of The Conversation articles. The Conversation articles are made freely available using a Creative Commons license, which allows republishing in full at no cost. The finding suggests that participants who republish articles work in roles that have a research or news-related reporting and/or communication and engagement aspect to their work and are using The Conversation as a key source of content. One of the main venues that republish The Conversation is the Australian Broadcasting Corporation (Australia’s national broadcaster). Article republishing significantly increases the potential for engagement and use beyond The Conversation direct readership, therefore also increasing the likelihood of use of The Conversation articles leading to positive impact.

Undertaking further research was another engagement action that predicted instrumental use of The Conversation articles. This suggests that these participants were seeking out information for the purpose of directly informing understanding or decision-making. It also suggests that The Conversation is just one of a number of resources that participants were turning to to inform development of policy, program, strategy, etc. This supports previous research highlighting that research evidence is only ever one of the many inputs that inform policy, program and practice decision-making [27,66–68]. It may also suggest that participants sought to read the underlying research studies often linked to in The Conversation articles, however this is not clear from the survey results. The term ‘research’ can be interpreted in myriad ways and is not always used in reference to academic research literature [69]. Deeper understanding of the motivations of these particular participants could illuminate the types of situations and drivers that are motivating their search for research evidence and uncover the other sources of evidence they rely on.

Tactical use of The Conversation articles.

‘Used the article in a report’ also predicted use of articles to support a work-related decision or action that had already been made. ‘Used article in a report’ was again the strongest predictor for this type of use based on the classification tree. ‘Symbolic’ or ‘tactical’ use of research are terms coined to describe when research is used to support a decision or action made prior to research evidence being sought or examined [13,15,25].

Being employed as a ‘politician, policy officer or government employee’ significantly predicted tactical use of a The Conversation article, supporting the claim that politicians only seek evidence to support their agenda. It is often quipped that politicians only seek information that confirms a predetermined stance or interest, sometimes referred to as ‘policy based evidence’, rather than ‘evidence based policy’, and certainly there are situations where this has occurred [70,71]. However, it has been effectively argued that conceptual and tactical use of research at any stage of the policy decision-making process is a valid use of research. Further, there are often occasions where political staffers and policy officers are not in control of the decision-making process [16,72]. In most cases policy decision-makers, at all levels, are responsible for implementation of the political decision-making of the government of the ruling party [16]. As such, indicating this type of tactical use may be related to use of The Conversation articles to inform planning for implementation of policy, program or strategy decisions that have been made and/or announced [67]. Focusing on different types of research use and recognising that research is only ever one of a multitude of inputs that inform decision-making reflects the complex reality of policy development; which is not a linear, rational process and which is affected by a broad range of interconnected factors [13,25,69,71,73–76].

That policy decision-makers are using The Conversation articles both instrumentally and tactically suggests articles are being used in relation to particular situational and decision-making needs [13]. Previous studies have also demonstrated that policy-decision makers use research both instrumentally and tactically [10,25,35]. A 2016 systematic review [12] has highlighted the pivotal role that motivation and opportunity play in influencing decision-makers use of research. This suggests a more nuanced understanding of the motivations and opportunities that enable research use, potentially through detailed case studies, is urgently needed [69].

Interestingly however, being employed in senior leadership and management positions, regardless of sector, also predicted this type of tactical use of The Conversation articles. And whilst employment position was a significant predictor of research use, employment sector was not. Specific positions in a given context were more important than the context itself in predicting use of The Conversation articles for work-related purposes. This finding links to the existing literature regarding the role that context plays in the design of interventions seeking to increase research use by public health decision-makers [30,77]. The findings suggest that regardless of the sector, identifying and strategically including those in senior, authoritative roles in intervention design will be critical. This is supported by systematic review findings that identified authority to make decisions and engaging opinion leaders and champions as a key enablers of research use [40,63,69].

Finding that those at the most senior levels in organisations across all industries and politicians and other government decision-makers are using The Conversation articles to inform and support work-related decision-making is significant. These results highlight that while partner-based research funding, as evidence of engagement with industry and government, is an important indicator of future potential impact; indirect engagement efforts targeting those in senior-level and government positions also has potential for real impact [59].

Conceptual use of The Conversation articles.

‘To assist with work’ was the only factor that predicted use of The Conversation articles to inform work-related discussion, debate and understanding across both the logistic regression and classification tree analyses. This type of use, where research informs understanding without specific action, is defined as conceptual use [13]. Measuring conceptual use via survey is valuable for various reasons. It is similar to social media analysis that can identify if a particular topic or article of content has been discussed in the public sphere. However, The Conversation survey data can complement social media analytics as it can capture conceptual use that may only take place in personal, face-to-face conversation [53]. Further, this type of use, as asked about in The Conversation survey, is directly focused on whether research has informed research work-related discussion and debate.

Whilst there is no guarantee that conceptual use will shift to an applied type of use, such as tactical or instrumental, it can enable decision-makers to use research to support their business activity when the opportunity becomes available. Through his in-depth research on policy decision-making and development processes Kingdon [72] highlighted that significant policy change and development can occur when the right ‘window of opportunity’ is open. In this way, conceptual use of research and academic expertise as measured in The Conversation survey could be considered as an indicator of research moving along the pathway to impact. Measuring conceptual use of research via survey can augment social media analytics, as it is also able to capture instances of private discussion and debate, which cannot be measured easily on a large scale by other means.

Use of The Conversation to inform one’s own personal attitude and behaviour chnage.

Interestingly, discussing articles with friends and colleagues was the only engagement action that predicted use of The Conversation to inform personal attitude or behaviour change. Together, the findings regarding discussing The Conversation articles with friends and colleagues suggest that this is a very active type of engagement, where one may be seeking to gather opinions to inform their own decision-making or to use the content to influence the opinions or decision-making of others. The finding supports previous studies that have identified discussion and debate as important in enabling research take up [59,78]. Indeed leading research translation focused organisations have established forums to enable deliberative dialogue around research. Many systematic reviews have identified increased interaction, communication and face-to-face engagement as key enablers of research use [11,12,63]. However, the small numbers of interventions that have sought to use this approach show limited evidence of effectiveness [12].

Reading The Conversation ‘to explain the news’ significantly predicted use of a The Conversation article to inform personal attitude and behaviour change. This finding suggests that for these participants The Conversation is providing greater detail and information that helps them understand a particular current issue and inform decision-making. In these instances The Conversation is enabling the use of research and academic expertise in ways that can have very direct and significant impact on people and communities. This opportunity for researchers to have a direct impact on the general public is unique. It is estimated that just 25–50% of all academic research is currently free to access on the Internet via open access journals and articles and institutional repositories [79,80]. Even then much of this research remains inaccessible to the general public, as it written for an expert peer academic audience [65]. As such, The Conversation as a not-for-profit organisation that translates research evidence and academic expertise for non-academic audiences, fills a critical gap; enabling individuals to make evidence-informed decisions that can shape the direction and outcomes of human experience, actions and worldview through translation and open communication of academic research and expertise.

Participants who indicated ‘expert opinion and facts’ as a main reason for reading The Conversation were also predicted to use The Conversation articles to inform personal attitude and behaviour change. Similarly ‘to explain the news’, finding that ‘expert opinion and facts’ significantly predicted use of The Conversation article suggests that The Conversation is providing a reliable source of academic expertise and research evidence that is relevant to their needs and interests [27,69]. In 2015 Alperin [48] completed one of the first comprehensive studies of non-academic use of institutional repositories in South America [46]. Alperin [48] found that those working outside academia represented between 16–25% of users. Of those, between 7.9 and 10.5% were using articles from repositories for personal reasons. Together these results indicate that there is significant capacity to engage with and use of research amongst the general public, and therefore significant potential for research to have impact through open access research communications platforms [47].

Limitations.

This study is based on a survey that was designed by marketing and communications staff within The Conversation. Only the question on use of The Conversation was crafted by an academic and based on academic literature. The Conversation articles are based on research evidence and also on academic expertise and opinion and as the survey responses are not linked to individual outputs we do not what articles were actually used. This is an issue with much research in the field that has surveyed government decision-makers about their use of research [37]. Despite the limitations this data set has produced findings reflected in the academic literature as highlighted throughout the discussion. The strength of this analysis is that the types of use of The Conversation were clearly defined, which has also been an issue in past studies [11,69]. A further limitation is this is based on an Australian survey and therefore the responses reflect use of The Conversation by Australians only. However The Conversation is also available in the UK, which has also introduced Research Impact assessment in their national research performance frameworks and The Conversation is also available in the US, France, Canada, Indonesia and Africa. Future research on use of The Conversation in other countries to provide a comparison to the findings would be valuable.

Conclusions

The Conversation annual survey has provided valuable data demonstrating extensive engagement and use of research and academic expertise by those working in sectors beyond academia. Overall, the major contribution of the study are the findings demonstrating that the majority of factors that predicted the four types of use of The Conversation articles were related to engagement actions undertaken after reading a The Conversation article, participants’ main reasons for reading The Conversation and senior and policy-related employment positions. Importantly however, different factors predicted different types of use, highlighting that interventions seeking to achieve research impact through increased engagement, must be based on a more detailed and nuanced understanding of engagement and use that is currently depicted in the rhetoric surrounding ‘research impact’. These findings suggest that senior and influential decision-makers can play a critical role in development of strategies and interventions seeking to support or increase use of research. Further, this study has shown that The Conversation is providing research-informed articles that are being used by industry and government decision-makers to inform work-related decision-making and discussion and debate. This indicates that decision-makers are finding content on The Conversation that is relevant to their decision-making needs and actionable in their context. This highlights that, for academics, publishing in The Conversation provides an excellent opportunity to increase research use, and therefore potential research impact.

Supporting information

Acknowledgments

We would like to thank The Conversation for the opportunity to contribute to the development of the survey and to analyse and report on survey results.