Figures
Abstract
There are calls for policymakers to make greater use of research when formulating policies. Therefore, it is important that policy organisations have a range of tools and systems to support their staff in using research in their work. The aim of the present study was to measure the extent to which a range of tools and systems to support research use were available within six Australian agencies with a role in health policy, and examine whether this was related to the extent of engagement with, and use of research in policymaking by their staff. The presence of relevant systems and tools was assessed via a structured interview called ORACLe which is conducted with a senior executive from the agency. To measure research use, four policymakers from each agency undertook a structured interview called SAGE, which assesses and scores the extent to which policymakers engaged with (i.e., searched for, appraised, and generated) research, and used research in the development of a specific policy document. The results showed that all agencies had at least a moderate range of tools and systems in place, in particular policy development processes; resources to access and use research (such as journals, databases, libraries, and access to research experts); processes to generate new research; and mechanisms to establish relationships with researchers. Agencies were less likely, however, to provide research training for staff and leaders, or to have evidence-based processes for evaluating existing policies. For the majority of agencies, the availability of tools and systems was related to the extent to which policymakers engaged with, and used research when developing policy documents. However, some agencies did not display this relationship, suggesting that other factors, namely the organisation’s culture towards research use, must also be considered.
Citation: Makkar SR, Haynes A, Williamson A, Redman S (2018) Organisational capacity and its relationship to research use in six Australian health policy agencies. PLoS ONE 13(3): e0192528. https://doi.org/10.1371/journal.pone.0192528
Editor: Pablo Dorta-González, Universidad de las Palmas de Gran Canaria, SPAIN
Received: February 18, 2016; Accepted: January 25, 2018; Published: March 7, 2018
Copyright: © 2018 Makkar et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data are too sensitive to publish in an anonymous form, as there were only six agencies participating in the study, all of which were located in the state of NSW (Australia). The participating agencies could be identifiable by their scores on the ORACLe and SAGE questionnaires, in particular, their SAGE domain scores for the individual policy documents submitted. The data restrictions are imposed by the University of Western Sydney Human Research Ethics Committee HREC, which approved this study. Readers may email the Sax Institute, if they are interested in accessing the data: communications@saxinstitute.org.au.
Funding: This work was funded by the Centre for Informing Policy in Health with Evidence from Research (CIPHER) an Australian National Health and Medical Research Council Centre for Research Excellence (APP1001436) and administered by the University of Western Sydney. The third author (AW) holds an NHMRC Post-Doctoral Fellowship (ID: 510 391). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
There have been increased calls worldwide for policymakers to make greater use of research when formulating and implementing health policies in order to promote more sustainable and equitable health spending and use of resources, minimise health inequities, and improve health outcomes on a global scale [1–8]. Unfortunately, the use of research in health policymaking is less than optimal [4, 5, 9–16].
Evidence indicates that the suboptimal use of research to inform health policy is due to numerous barriers. Firstly, policymakers argue that research is often not presented in a clear, user-friendly, and summarised format with clear policy implications [5, 16–22]. Secondly, policymakers report deficiencies in skills and confidence in acquiring, appraising, interpreting, synthesising, and applying research to policy [14, 17, 19, 21, 22]. Infrastructural support for these actions (e.g., journal subscriptions; skills training; research support staff such as librarians, knowledge brokers or consultants) has also been reported as insufficient by policymakers [8, 17–20, 23, 24]. Finally, policymakers and researchers have described a lack of collaborative research, despite evidence demonstrating this to be an important facilitator of research dissemination and uptake [20, 25–27].
These barriers could potentially be overcome if organisations and policymakers have a sufficient level of capacity to enable greater use of research in policy. According to the SPIRIT Action Framework [28], capacity encompasses three main aspects: (a) a culture where research use in policymaking is valued by the organisation and staff; (b) sufficient skills and knowledge among staff to access, appraise, and apply research to policy; and (c) tools and systems available to support research use by staff and to establish collaborative relationships with researchers (see Fig 1).
Based on the SPIRIT Action Framework as described in: Redman, S., Turner, T., Davies, H., Williamson, A., Haynes, A., Brennan, S.,… Green, S. (2015). The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med, 136–137, 147–155. doi: 10.1016/j.socscimed.2015.05.009.
The framework emphasises that where there is sufficient capacity, policymakers can initiate six specific research engagement actions that include (i) searching for, and (ii) obtaining research; (iii) appraising its relevance to the policy issue and (iv) appraising its quality or scientific rigour; (v) generating new research and/or data analyses (especially when relevant research is not available); and (vi) interacting and collaborating with researchers. Once relevant research has been accessed and/or generated, it can be used to inform policymaking in four ways. Specifically, research may: (i) directly inform decisions relating to the identified policy issue(s) (instrumental use) [29, 30], (ii) clarify understanding about the policy issue without directly influencing the decision (conceptual use)[13, 31, 32], (iii) justify and/or persuade others to support a predetermined decision (tactical use)[30, 33], or (iv) be used to meet legislative, funding, or organisational requirements (imposed use)[31]. The framework predicts that using research to inform policies will ultimately lead to better health services and outcomes.
Detailed and systematic measurement of organisations’ current research use capacity is essential to identify the tools and systems needed to drive improvement in policymakers’ use of research [34]. Previous research has shown that there are four broad types of capacity-building initiatives that can support evidence-informed policymaking in organisations [35]. These initiatives included (i) availability of staff with expertise to promote and support research use [19, 35]; (ii) mechanisms to establish links with researchers and opinion leaders outside the organisation [6, 7, 18, 19, 21, 22, 35–40]; (iii) technical infrastructure to facilitate research access (e.g., journal and database subscriptions; knowledge management systems for efficient storage and retrieval of research findings) [8, 35, 41–43]; and (iv) training programs to improve staff skills in accessing, appraising and applying research to policy [4, 6, 35, 44, 45] and to build agency executives’ skills and capacity for promoting research use within their organisations [35, 46]. Only a few studies have systematically investigated the availability of these tools, systems, and processes within policy agencies to support research use (e.g., [35, 45, 47]. Most of the research has instead focused on documenting barriers and facilitators to research use beyond organisational factors such as political forces, stakeholder opinions, or the availability of relevant, trustworthy, and timely research [6, 27, 48–51]). Furthermore, studies have yet to examine whether the availability of these tools and systems within agencies is related to policymakers’ actual engagement with, and use of research in policymaking. This information is likely to be useful in identifying organisational tools and systems that have the greatest potential for building health policymakers’ capacity to use research in policymaking.
The aim of the present study was to undertake a systematic investigation of the tools and systems available to support the use of research in health policymaking within Australian agencies located in Sydney, New South Wales. We also investigated whether the availability of these tools and systems was related to the extent to which policymakers engaged with and used research when developing policies.
Materials and methods
Agencies
Six agencies with a role in health policy based in Sydney, New South Wales, Australia participated in the current study: five state government and one national organisation. As part of the recruitment criteria an agency was eligible to participate if (a) a significant proportion of its work was in health policy or program development, and (b) there were at least 20 staff involved in health policy or program development or evaluation. For the purposes of this paper, we refer to such staff as policymakers–individuals employed in policy agencies, who draft or write health policy documents, develop health programs, or make significant contributions to policy decisions relating to health services, programs, and/or resourcing [52]. Three of the participating agencies concentrated on specific areas of health (e.g. cancer), and three worked more broadly across public health and health systems improvement. Of the five state government agencies, four were board-governed statutory organisations that co-reported to the NSW Minstry of Health, and the fifth was a division within the Ministry itself. The national agency reported to a board but was entirely funded by the Federal Department of Health. Thus all were government funded and subject to fluctuations in state and federal budgets. All had been operating for at least three years but were subject to recent or current restructures.
Measures
Organisational tools and systems.
To measure agencies’ tools and systems to support research use, we used the Organisational Research Access, Culture and Learning (ORACLe) measure [34] developed by the Centre for Informing Policy in Health with Evidence from Research (CIPHER) at the Sax Institute, Sydney, Australia. ORACLe consists of two components: (a) a semi-structured interview, and (b) a scoring guide. The interview comprises 23 questions that capture information about the extent to which a range of tools and systems to support research use are available within an organisation, with a focus on initiatives that have been developed in the six months prior to measurement. The questions cover seven key domains of organisational capacity to support research use as displayed in Table 1. Raters use the scoring guide to score each question on the following three-point scale: 1 = the tool or system is not present, 2 = it is present to a limited/moderate extent or 3 = it is present to a large extent. Questions under the same domain are averaged, yielding seven domain scores, each out of 3. These domain scores are entered into an empirically derived algorithm to produce an overall capacity score out of 3 for that organisation (where where 0–0.99 = limited, 1–1.99 = moderate, and 2–3 = extensive overall capacity) [34]. This algorithm gives greater weight to domains that have greater relative importance for building organisational capacity, based on discrete choice modelling of surveys completed by knowledge translation experts. More information on how ORACLe is scored and how the scoring algorithm was derived from a discrete choice consultation with international experts can be found elsewhere [34]. In the present study, SM and AW used the marking guide to score ORACLe interviews from each of the six participating agencies.
Research engagement and use.
The current study also utilises a newly developed measure called SAGE (Staff Assessment of enGagement with Evidence), which assesses ten domains—six research engagement actions and four research use actions—in relation to a specific policy document completed in the last six months (see Table 1 and Fig 1). SAGE consists of a semi-structured interview with the person most heavily involved in producing the document and a scoring checklist. In the interview, policymakers are asked: (i) whether or not research was used to inform the document; (ii) what research engagement actions were undertaken; (iii) how research was used; and (iv) what barriers and facilitators impacted on their use of research (see Table 1). Interviews were audio recorded and professionally transcribed. The interviewer checked, corrected, and de-identified transcripts before they were scored.
The scoring checklist breaks down each of the measured domains into their key subactions. Subactions are the essential features/actions of each domain. For example, a key subaction for Searching for Research is looking through academic literature databases such as Medline. If a particular subaction was performed by the policymaker on the basis of his or her interview, it is ticked off in the scoring checklist. Each subaction has a different point value assigned to it based on its importance in facilitating evidence-informed health policymaking. The degree of importance of each subaction was established through conjoint analysis of surveys completed by over 50 local and international experts in knowledge translation [53, 54]. The points for all ticked subactions are summed to give a score of 0–9 for that particular domain (where 0–2.99 indicates limited, 3–5.99 moderate, and 6–9 extensive efforts to engage with, or use research). Full information on scoring SAGE and the complete tool is available elsewhere [53–56]. The reliability and validity of SAGE has also been demonstrated [55]. In this study, author SM applied the SAGE scoring checklist to score the interview transcripts on the ten measured domains.
Data collection
Data was collected as part of SPIRIT (Supporting Policy In health with Research: an Intervention Trial) [57], a stepped wedge intervention trial designed to improve the capacity of agencies and staff to engage with and apply research in policy work [28, 57]. Prior to the commencement of SPIRIT, the agencies completed a suite of measures including ORACLe and SAGE, in order to evaluate their level of capacity and research use respectively. Information about the agencies and how they were recruited is described elsewhere [57]. For ORACLe, a senior staff member from each of the six agencies (labelled A1 to A6 in this study) was nominated to undertake the ORACLe interview. Senior staff members (executive staff and CEOs) were targeted since they were most likely to possess accurate knowledge of the systems, tools, and structures to support research use within their organisations. Each of the six participating agencies also selected four policy documents which, in their view, best represented the use of research in their work. These documents were used as a proxy for agencies’ wider policy development processes and were the focus of SAGE interviews. The policymaker identified as having contributed most to each document’s development was invited to participate in the SAGE interview. Four interviews were completed for each of the six agencies (24 interviews altogether). Both the ORACLe and SAGE interviews were conducted by a single interviewer, by phone (except when the interviewee requested a face-to-face interview) and ranged from 30–60 min in length.
Data analysis
ORACLe and SAGE scores were calculated for each agency. For ORACLe, each agency received a score out of 3 for each of the seven capacity domains measured by the tool. These domain scores were entered into the algorithm to obtain a score out of 3 indicating the agency’s overall capacity to support research use in policymaking. For SAGE, each of the four policy documents were scored on the 10 domains measured by the tool: six research engagement actions and four research use actions (each scored out of 9), and the scores across the four documents were averaged. To evaluate the relationship between capacity and research engagement/use, associations between ORACLe and SAGE scores were examined qualitatively, since the sample was too small to calculate correlation coefficients.
Results
Organisational tools and systems
Data describing the systems and tools within each agency are displayed in Table 2 and elaborated upon below. Data are organised under the seven domains of systems and tools to support research use in policy as assessed in ORACLe (see Table 1). All data regarding the agency’s systems and tools derived from the reports of a single executive per agency.
Processes for developing policy that mandate or encourage use of research.
In four of the six agencies (A1, A3, A5, and A6), explicit, detailed documented processes for how policies should be developed were available, which often took the form of “a policy on how to write policies”, or “written documents and templates”. A2 also had documented processes on how policies should be developed but they lacked detail. In A1, these documented processes explicitly required consultation of research findings or researchers; whereas in A3, A5, and A6, the requirement to use research was not directly stated. In A4, such documented policy development processes were still in the process of being established.
Programs and systems to assist leaders in supporting the use of research in policy.
No agency had training programs focused on building leaders’ confidence and skills in using research to inform policy. A2 and A4 did offer training to all staff including leaders, but this did not address leadership/managerial issues, and was not “specifically targeted [at] research”. Furthermore, none of the performance management systems for leaders in the agencies studied covered expertise in research use. Finally, in all agencies except A3, systems to enable internal communications from leaders to staff (e.g., email updates, newsletters, blogs, tweets, bulletins) were in place. References to the latest research in these communications occurred on a weekly basis within A1, whereas in the remaining agencies they were either “ad-hoc” or infrequent (e.g., quarterly).
Programs to provide staff with training on how to use research in policy.
A2, A3, and A5 participants reported that their staff were provided with access to training programs to improve research skills. This included external professional development courses (e.g., on using EndNote), “internal workshops on critical appraisal” or using research databases, and “external people coming in [and] doing formal training”. In A3 these programs were offered consistently, whereas within A2 and A5, they were only offered if staff “identified that there’s a need for them” to participate in such training programs. In A2 and A3, participation in training was considered in staff performance management, whereas in A5, it was only considered if it “came up as an issue” for the employee.
Systems and tools and help staff access and use available research.
Within each agency, there were numerous systems to help staff access and obtain available research efficiently. For example, all agencies except A6 disseminated relevant and timely research to staff through various mechanisms such as email, social media, list servers, Google alerts, and journal clubs. In A2-A5 this occurred several times per month, and in A1 less than twice per month. A3 utilised an organised and indexed “library management system” for archiving this research and other knowledge from research. This system provided an efficient and effective means of sharing, editing, storing and retrieving research findings by staff. The remaining agencies (except A6) had centralised repositories for storing and retrieving research, but these were not indexed and therefore not easily searchable. Finally, four agencies (A1, A3, A4 and A5) reported having established systems for commissioning research reviews. These included specific research arms within the agency that conducted formative research (A3), and established relationships with centres that conduct rapid reviews (A1, A4, A5). The remaining agencies used “very ad-hoc methods” for commissioning reviews, which depended on “what the project was about and how much [it costed]”.
Participants reported that agencies provided numerous tools and supports to staff for accessing research. These included subscriptions to databases and relevant research journals, access to physical and/or electronic libraries, licenses for reference management software such as EndNote, and easy access to research expertise in the form of librarians, medical writing teams, research and evaluation teams, epidemiologists, medical staff, and scientists. Finally, A3 also provided staff with a “critical appraisal tool” that provided detailed, documented guidelines on appraising research. Such guidelines weren’t available in the other agencies.
Efforts to generate new research.
All agencies except A1 reported conducting several research projects internally to generate new data to inform programs under development. These were conducted by research divisions (A3 and A5) or policy teams (A2, A4, A5) within the agency often in partnership with research organisations (A4, A5). Three agencies also reported commissioning research externally either via research partner organisations (A1) or external institutes and consultants (A4, A5).
Processes to enable research-informed policy evaluations.
Although respondents from each agency (excluding A4) emphasised that policy and program evaluation was “mandatory” or “an expectation and requirement”, there was an absence of standardised documentation or procedural guidelines that specified how policies should be evaluated, except within A3. In this agency, program evaluations were guided by “an evaluation framework” which acted as “a critical thinking tool… that defined… inputs [and] outputs… [and] how you would measure them”. This framework, however, did not explicitly instruct staff to use research to guide the evaluation. Evaluation processes in A5 varied depending on the program being evaluated or the methods of evaluation staff within different divisions of the agency. These methods were loosely based on “a program logic… [and] documentations [with] some core requirements” and not standardised processes. Furthermore, A5 also commissioned consultants to conduct evaluations who utilised their own evaluation frameworks.
Mechanisms to help strengthen relationships with researchers.
Participants reported that their agencies had multiple mechanisms to strengthen relationships with researchers. Firstly, each agency provided staff with opportunities to represent their organisation at numerous conferences and forums. Second, all agencies reported having established several contractual, and at least one informal, relationship with external research organisations (e.g., universities). Third, at least one staff member from three agencies (A2, A4, A5) was reported to hold an adjunct appointment with a university. Finally, in all agencies except A6, researchers were frequently invited to participate in policy advisory committees. These “external advisors” were reportedly sought out for their expertise and were said to be essential to the policy advisory process such that “we wouldn’t set up such a mechanism without having them at the table”.
Relationship between organisational capacity to support research use and actual research engagement/use for each agency
Fig 2 displays each agency’s scores on the seven ORACLe domains of systems and tools to support research use and the agency’s overall capacity score. Figs 3 and 4 show each agency’s SAGE scores for the six research engagement actions and four types of research use, respectively, as well as average scores for each. The results showed that for four out of the six agencies, ORACLe total scores (reflecting organisational capacity to support research use) were consistent with policymakers’ engagement with, and use of research as measured by SAGE.
Agencies’ SAGE scores for individual research engagement actions and the average score for research engagement actions.
Agencies’ SAGE scores for each research use action and the average score for research use.
A1, A3, and A5 had ORACLe scores reflecting extensive overall capacity to support research use and also displayed a relatively high degree of research engagement and/or use (Fig 2). A1’s high ORACLe scores were attributable to the agency having a documented guideline on writing policies, offering numerous resources to help staff access research (e.g., access to relevant journals, databases, a library and library staff; clear protocols for commissioning research reviews), and numerous mechanisms to strengthen relationships with researchers. In line with this, their SAGE data showed research was used instrumentally (i.e., used to directly inform policy priorities and/or alternatives) to an extensive degree (Fig 4). Policymakers from this agency also reported extensive conceptual and tactical research use.
A3’s overall ORACLe score also revealed extensive capacity to support research use, attributable to having clear frameworks on how to develop policy, ongoing training for staff to improve research skills, and a wide range of resources such as databases, journal subscriptions and guide documents to help staff use existing research (Fig 2). In line with this, their SAGE data suggested A3’s policymakers’ efforts to engage with research were in the moderate to extensive range, particularly with regard to efforts to search for research and the range of research obtained (Fig 3). A3 staff also reported extensive efforts to generate new research and displayed extensive conceptual research use, and moderate to extensive instrumental use (Fig 4).
A5 also received ORACLe scores indicative of extensive overall capacity to support research use (Fig 2). This high overall score was attributable to reportedly extensive documented processes for policy development and evaluation, resources to help staff access existing research (e.g., dissemination of research across the agency, access to research expertise, access to some relevant journals), and clear processes for commissioning research reviews. This agency had also actively undertaken internal and external research projects in the past six months, and reported strong mechanisms to establish relationships with researchers. In line with this, SAGE data revealed that efforts to generate new research and interact with researchers were extensive and that relevance appraisal efforts were in the moderate range (Fig 3). Furthermore, policymakers reported moderately high tactical and imposed research use, and extensive conceptual research use (Fig 4).
A6 received a relatively low overall capacity score on ORACLe, albeit still in the moderate range (Fig 2). This score was reflective of the agency’s reportedly limited to moderate skill-building opportunities for leaders and staff, limited resources to access research (i.e., lack of research dissemination across the agency or a comprehensive library), and unestablished methods to evaluate policies. In SAGE interviews, A6 policymakers reported limited efforts to search for research, and to appraise its quality and relevance (Fig 3).
A2 and A4 displayed inconsistent ORACLe and SAGE profiles. A2 had the highest overall capacity to support research use in policy according to their ORACLE score (Fig 2). This agency reported extensive mechanisms to build leaders’ capacity to support research use (training programs and communications from leaders to staff) and to establish relationships with researchers, as well as training programs for staff that were incorporated into performance management processes. They also reported having undertaken internal and external research projects in the past six months. In terms of actual research use, however, SAGE data showed limited-moderate efforts to search for, and appraise the quality and relevance of research, and A2 obtained a low-moderate range of research (Fig 3). Furthermore, instrumental research use was in the limited-moderate range for A2, and imposed use was limited (Fig 4).
A4, in contrast, scored in the moderate range for capacity to support research use and achieved the lowest ORACLe score amongst our agencies (Fig 2). Although having some resources to access research (e.g., frequent research dissemination; journal and database access, and methods to commission reviews) and multiple mechanisms to establish relationships with researchers, this agency lacked clear processes for developing evidence-based policies, limited tools to support leaders, an absence of staff training programs, and limited documented processes to evaluate policies. In contrast, SAGE data showed that A4 policymakers’ exhibited extensive instrumental and tactical research use, and moderate-extensive conceptual use (Fig 4). Further, A4 policymakers’ efforts to generate research and interact with researchers were also extensive (Fig 3).
Discussion
This is the first study to examine the extent to which Australian health agencies have tools and systems in place to support the use of research in policymaking. Having identified which tools and systems were present in six agencies we then explored their relationship with policymakers’ actual use of research in the development of policy documents. The results showed that, in general, agency executives reported a moderate to high level of organisational capacity to support their staff in using research in policymaking. There were both commonalities and differences between agencies in which tools and systems were available, shedding light on areas for potential improvement. In general, where agencies had extensive tools and systems in place to support research use, policymakers in those agencies displayed moderate to extensive efforts to engage with and/or use research in policymaking. However, two agencies displayed an inversion of research use capacity and actual engagement/use which indicates other mechanisms at work. In the discussion below we discuss the capacity domains that were particularly well developed across agencies, as well as those domains which could be further strengthened. We then discuss the relationship between these capacity domains and policymakers’ research engagement and use in policymaking and the implications for our understanding of the relationship between organisational capacity and research use.
Agencies’ capacity to support research use in policymaking
All agencies reported at least a moderate level of capacity to support research use, with four displaying extensive overall capacity. Most agencies had a range of tools and systems to help staff access and apply existing research to policy, such as access to databases, libraries, reference management software, and research expertise; methods for commissioning research reviews; and frequent organisation-wide dissemination of research. In a previous study, this domain emerged as one of the most important areas of research use capacity [34]. Furthermore, numerous studies have raised the importance of developing such technical infrastructure to access existing research findings [17, 19, 43–45]. In support, among the three agencies that had the most extensive availability of tools and supports to access existing research (A3-A5), policymakers’ average engagement with research was in the moderate to extensive range.
All agencies reported having a number of processes to establish relationships with researchers. Studies show that sustained partnerships between researchers and policymakers encourage greater use of research in policymaking, for example, by informing priorities and recommendations, increasing the intention to use research, improving researchers’ understanding of the policy context, and increasing researchers’ capacity to develop research products suitable for policymakers (e.g., evidence briefs) [20, 58, 59]. Quantitative research has verified these findings, demonstrating significant associations between the intensity of policymaker-researcher linkages and eventual research use in policy [15]. In the present study, policymakers from agencies that reported extensive processes to establish relationships with researchers also displayed moderate to extensive research engagement actions and/or research use when developing policies (except A2).
One of the advantages of establishing relationships with researchers is coproduction of research to inform policy. Evidence suggests that policymakers are more likely to use research if they are jointly involved in the project as integral partners [4, 19–21, 36, 38], and our results are generally consistent with this perspective. Only half the agencies in the present study, however, commissioned an external research institute to conduct research to inform its policies in the past six months. Rather, most conducted internal research, often carried out by research units within the agency or the policy team themselves. This suggests that most agencies had systems for conducting internal research, although mechanisms to facilitate coproduction with external researchers could be developed further, as well as mechanisms to encourage ongoing relationships with researchers beyond the life of individual research projects. Ongoing policymaker-researcher relationships are vital to ensure researchers are aware of pressing policy issues so they can produce relevant and timely research [60]. To facilitate the formation of ongoing relationships, numerous courses have been developed, particularly in the USA and Canada, that aim to increase researchers’ understanding of knowledge translation and exchange (KTE) theory and concepts, improve their skills in developing KTE strategies and embedding these into research projects, enhance their capacity to build relationships with policymakers, and improve their ability to communicate research findings clearly and accessibly, to effectively engage policymakers [61–67].
A number of key areas of research use capacity could potentially be improved across agencies. The first of these are programs targeting agency leaders (domain 2). In our previous study, this domain was rated as the second most important for supporting organisational capacity to use research [68]. Such leadership-targeted programs are essential given that leaders are critical to establishing an evidence-based organisational culture [19, 42, 69], including the promotion of mutual understanding and collaboration between policymakers and researchers [18, 42], and advancing evidence-informed policy initiatives [42, 69]. Without a culture that supports evidence-informed policy it is unlikely that staff will value research, and use it to inform policy development [6, 11, 20].
In terms of additional resources to access existing research findings, studies indicate that policymakers may also benefit from being provided with efficient ways of storing, sharing, and retrieving knowledge from research [6, 19, 24, 45], and standardised frameworks on how to access and apply research to policy [18, 43]. These resources were generally lacking across our agencies, suggesting another potential area for improvement.
Although agencies generally appeared to value research use, the results suggested this could be made more explicit in relation to the policymaking process. For example, although four agencies had documentation on how to develop policy, only one (i.e., A1) had documentation instructing policymakers to consult the latest research. Interestingly, this was the only agency that displayed extensive instrumental, tactical, and conceptual research use, perhaps indicating the importance of having policy frameworks that explicitly encourage staff to seek out a broad range of evidence, including research. Furthermore, although five agencies required evaluation to be built into policy development, only one agency had a documented process for how policies should be evaluated, which was neither based on research nor instructed staff to seek out research to inform the evaluation design. It would be difficult, however, for most agencies to develop a standardised protocol for evaluating the wide variety of policies they develop. A feasible alternative would be for agencies to develop flexible evaluation guidelines that could be adapted to different policies and programs, and which advise staff to seek out relevant research to inform evaluation protocols.
Another area for potential development across most agencies was training programs for staff (domain 3). One agency (A3) provided these programs actively and regularly, two offered such programs on a needs basis, whereas three agencies did not provide any training. Policymakers have raised the importance of opportunities for training to improve their skills in accessing, appraising and applying research to policy [6, 44]. The availability of tools (e.g., database access, knowledge management systems) extends organisational capacity only if staff know how to use them. Indeed, studies have found that provision of training by the organisation is strongly associated with research use [24]. In support, two of the three agencies that provided staff with access to training (A2, A5) also scored in the moderate-extensive range for research engagement actions and use, reinforcing the importance of developing this capacity.
Relationship between organisational capacity and policymakers’ engagement with and use of research
A unique feature of this study was our examination of the relationship between agencies’ research use capacity and policymakers’ actual research use in the development of policy documents. Results indicated that agency capacity, as measured by the tools and systems in place to support research use within the agency, predicted policymakers’ engagement with, and use of research in policy development for four of the six agencies.
A1, A3, and A5 each scored in the extensive range for ORACLe, indicating a high level of capacity to support staff engagement with, and use of, research in policy. These agencies in particular had clear and documented processes to develop evidence-informed policies (and in A3 and A5, there were clear methods to evaluate these policies). They also provided staff with an extensive range of resources to access existing research (e.g., journal subscriptions, mechanisms to commission reviews, and access to research experts and librarians), and had multiple processes to establish relationships with researchers. Consistent with their level of capacity, staff in these agencies exhibited moderate to extensive efforts to engage with research during the policymaking process. In particular, staff in A1 and A3, were likely to undertake thorough searches for literature (e.g., examining academic databases, sources of authoritative grey literature) and find a moderate to extensive range of up-to-date research (e.g., peer-reviewed articles, systematic reviews, policies and data from external agencies). Furthermore, policymakers’ use of this research during policy development in these agencies was moderate to extensive overall. Specifically, conceptual use of research was extensive, particularly within A3; while instrumental and tactical use was extensive within A1, and moderate within A3 and A5.
In contrast, A6 displayed a moderate level of organisational research use capacity, although this agency had numerous domains of capacity in the limited range including limited access to training programs for staff and leaders, limited to moderate tools to access existing research and limited processes to evaluate policies. Their engagement with research was in the moderate range, but their efforts to search for research and appraise its relevance and quality were limited (and lower than the other agencies studied). Instrumental and tactical research use were both in the moderate range, and conceptual use was extensive, suggesting that this agency’s moderate level of capacity more strongly influenced how policymakers engaged with research (e.g., their efforts to search for and appraise research) than how they used it.
Two agencies displayed inconsistencies between their ORACLe profile and their staff’s engagement with and use of research, as measured using SAGE. A2 showed extensive capacity to support research use, however, policymakers’ efforts to search for, appraise, and generate research were either limited or in the limited-moderate range. Their overall research engagement actions and research use scores were also in the limited to moderate range. In contrast, A4 displayed a moderate overall level of capacity. However, SAGE data showed A4 policymakers’ research engagement actions were on average in the moderate-extensive range. Furthermore, their use of research, on average, was in the extensive range.
The lack of relationship between capacity and staff research engagement and use for these two agencies indicates the influence of other factors on policymakers’ research-related behaviour not directly captured in ORACLe. According to the SPIRIT Action Framework (see Fig 1), besides the presence of systems and tools to support research use, an equally important component of capacity is an organisational culture where research use is valued by both staff and the organisation. Implementation research has shown that organisations with cultures that support evidence-based practices (EBPs), encourage innovation, emphasise competence and maintaining up-to-date knowledge, value staff input into decision making, and exhibit greater use and sustainability of new EBPs across the organisation [70–75]. ORACLe does not measure key aspects of culture, such as leadership, behavioural norms and collective expectations among staff regarding how to undertake their work [70–74], or the underlying assumptions and values that influence these behavioural expectations [70–74]. It is possible that these unmeasured components drove the discrepancies that were observed for agencies 2 and 4. For example, research use may not have been valued or expected in the organisational culture of A2, possibly due to leadership characteristics or contextual factors unique to this agency and its work [74]. In contrast, the relatively new executive in A4 may have strongly valued research, but this was not captured in their ORACLe scores because their tools and systems use were still in development and had yet to ‘catch up’ with the agency’s emerging research culture.
These findings suggest that the availability of tools and systems within the organisation is generally predictive of policymakers’ engagement with and use of research in policy development. This may be contingent, however, upon the organisation having an underlying culture where research use in policy is valued.
The results indicate that capacity influences not only instrumental and conceptual uses, but also greater tactical use. Organisations with high scores on capacity generally scored in the moderate to extensive range on tactical research use (particularly A3). The SPIRIT Action framework proposes that research-informed policies may lead to improvements in health services and outcomes. It can be argued that research informs policy only when it is used instrumentally and/or conceptually. For both types of use, research comes before a decision is made, and is used to help solve important policy-related issues, consistent with the problem-solving model of research use proposed by Weiss [31]. Tactical use, on the other hand, comes after a policy decision, where research is used to justify decisions already made. Consequently, research findings may be accessed or reported selectively in order to support these decisions, implying that such tactical use should be discouraged. The reality, however, is that policymaking is highly political [33, 76]. Decisions are often made in response to political pressures or stakeholder interests and consultation, before research evidence has been considered. Research is, therefore, often used tactically as a means of addressing, responding to, or influencing these stakeholders [76, 77]. To the extent that research is used in a non-biased fashion to present the complete facts and most effective strategies to stakeholders, promote discussion and collaboration, and counteract potentially ineffective courses of action, tactical research use may elicit positive outcomes. However, tactical use involving selective and biased searching and reporting of findings to justify positions, represents a misuse of research and should therefore not be encouraged. Policy agencies aiming to promote a culture of research use may benefit from clarifying this distinction to staff, in order to promote unbiased research use in policy formulation.
Strengths and limitations
This study had a number of strengths. Firstly, the data presented in this paper represents the first detailed and systematic investigation of the tools and systems available to support research use in policy within Australian agencies with a role in health policy. Secondly, we used newly developed, validated, and reliable instruments to measure both the capacity of organisations to support research use (i.e., ORACLe [34]) and policymakers’ actual engagement with and use of research in policy development (i.e., SAGE [53–56]). Third, and most important, we qualitatively evaluated the relationship between organisational capacity, and the extent to which policymakers actually engaged with, and used research during the policy development process. These findings provide valuable insights regarding the importance of building organisational capacity, and shed light on which tools and systems could be developed further to enhance organisations’ overall capacity to support research use, and thereby improve how policymakers engage with, and use research in policymaking.
The study also had some limitations. Firstly, our inferences regarding the relationship between culture, capacity, and research use behaviour are speculative since we did not measure (i) organisational culture from the staff perspective, or (ii) the agency’s underlying assumptions and values towards research use. Staff perceptions of culture could be assessed with the validated Organisational Social Context measure [70, 71], which evaluates organisational culture, climate, and staff work attitudes constructs in detail. Another recently instrument recently developed by CIPHER that could also be utilised in future research is SEER (Seeking, Engaging, Evaluating Research), which includes subscales assessing policymakers’ attitudes, and perceptions of the organisation’s attitudes, towards research use in policy. These SEER subscales have demonstrated good test-retest reliability and construct validity [78].
Measurement of underlying assumptions and values regarding research use, is more difficult as these are less consciously accessible. According to Glisson and James [73] this knowledge may be obtained indirectly by examining the behaviour and statements of individuals in the organisation, for example, through in-depth ethnographic studies of policymakers at work. However, conducting this type of research is highly intensive and may be impractical or unwelcome in many policy contexts.
Secondly, our analysis is based primarily on observational data from a small number of agencies based in Sydney, Australia. In order to make more definitive conclusions regarding the relationship between capacity and research use, future studies could recruit a large number of agencies and utilise multilevel modelling to determine whether organisational capacity and culture interacts with staff capacity to predict policymakers’ research engagement actions and research use during the policy development process.
Conclusion
In this study, we have documented and quantified the tools and systems to support research use within six Australian agencies with a role in health policy, and examined their relationship to policymakers’ use of research in policy. The results showed that overall, all agencies possessed at least a moderate level of capacity to support research use. Capacity domains that were well developed included the presence of tools to access existing research (e.g., databases and journal access), mechanisms to develop relationships with researchers, and systems to generate research to inform policy. Areas requiring further development included training programs for leaders and other staff, and processes for conducting research-informed evaluations of policies. For most agencies, the availability of tools and systems predicted better engagement with, and use of research in policymaking. In two agencies, however, this relationship did not emerge, implying that other contextual factors such as organisational culture should be taken into account when attempting to improve policymakers’ engagement with and use of research. The present study provides insights into the importance of organisations building capacity to support the use of research, and highlights particular tools and systems that can contribute to this capacity.
Acknowledgments
We wish to thank all the staff and executives of the agencies that agreed to complete ORACLe and SAGE. We would like to thank Catherine McGrath, Melanie Anderson, and Jennifer Taylor for undertaking the ORACLe and SAGE interviews in each of the policy agencies. We acknowledge all the senior policymakers who provided valuable contributions to refining and enhancing the practical utility of the measures used in this paper. We would finally like to thank Professor Sally Redman and the chief investigators of CIPHER, who conceptualized the development of ORACLe, SAGE and other innovations designed to increase the use of research in policymaking. CIPHER is a Centre for Research Excellence and is a joint project of the Sax Institute; Australasian Cochrane Centre, Monash University; the University of Newcastle; The University of New South Wales; The University of Technology, Sydney; the Research Unit for Research Utilization, University of St Andrews and University of Edinburgh; and the University of Western Sydney.
References
- 1. Lavis JN, Lomas J, Hamid M, Sewenkambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ. 2006;84(8):620–8. pmid:16917649
- 2.
World Health Organization. Changing mindsets: Strategy on health policy and systems research. Geneva: World Health Organization, 2012.
- 3. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implementation Science. 2012;7(1):1–17. pmid:22651257
- 4. El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012;7:2. pmid:22236561
- 5. Trostle J, Bronfman M. & Langer A. How do researchers influence decision makers? Case studies of Mexican policies. Health Policy Plann. 1999;14(2):103–14.
- 6. Hyder AA, Corluka A, Winch PJ, El-Shinnawy A, Ghassany H, Malekafzali H, et al. National policymakers speak out: are researchers giving them what they need? Health Policy Plann. 2011;26:73–82.
- 7. Mirzoev TN, Omar MA, Green AT, Bird PK, Lund C, Ofori-Atta A, et al. Research-policy partnerships—experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia. Health Res Policy Syst. 2012;10:30. pmid:22978604
- 8. Albert M, Fretheim A, Maiga D. Factors influencing the utilization of research findings by health policy-makers in a developing country: the selection of Mali's essential medicines. Health Res Policy Syst. 2007;5:2. pmid:17338810
- 9. Brownson RC, Fielding JE, Maylan CM. Evidence-based public health: A fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175–201. pmid:19296775
- 10. Ham C, Hunter DJ, Robinson R. Evidence based policymaking: research must inform health policy as well as medical care. Brit Med J. 1995;310:71–2. pmid:7833721
- 11. Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS ONE. 2011;6(7):e21704. pmid:21818262
- 12. Weiss C. Knowledge creep and decision accretion. Sci Commun. 1980;1(3):381–404.
- 13. Amara N, Ouimet M, Landry R. New evidence on instrumental, conceptual, and symbolic utilization of university research in government agencies. Sci Commun. 2004;26(1):75–106.
- 14. Campbell DM, Redman S, Jorm L, Cooke M, Zwi AB, Rychetnik L. Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Aust New Zealand Health Policy. 2009;6:21. Epub 2009/08/25. pmid:19698186; PubMed Central PMCID: PMC2739528.
- 15. Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5:41. pmid:20525270
- 16. Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009;20:70–5. pmid:18226519
- 17. Ettelt S, Mays N. Health services research in Europe and its use for informing policy. J Health Serv Res Policy. 2011;16(Suppl 2):48–60.
- 18. Gold M. Pathways to the use of health services research in policy. Health Res Educ Trust. 2009;44(4):1111–36.
- 19. Helmsey-Brown J. Facilitating research utilisation: a cross-sector review of research evidence. The International Journal of Public Sector Management. 2004;17(6):534–52.
- 20. Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006;10:163–80.
- 21. Lavis JN, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E. Towards systematic reviews that inform health care management and policy-making. J Health Serv Res Policy. 2005;10(Suppl 1):S1:35–48.
- 22. Petticrew M, Whitehead M, Macintyre SJ, Graham H, Egan M. Evidence for public health policy on inequalities: 1: The reality according to policymakers. J Epidemiol Community Health. 2004;58(10):811–6. PubMed PMID: WOS:000223830100004. pmid:15365104
- 23. Dobbins M, Ciliska D, Cockerill R, Barnsley J, DiCenso A. A framework for the dissemination and utilization of research for health-care policy and practice. Online J Knowl Synth Nurs. 2002;9:149–60. Epub 2002/11/20. doi: 090007 [pii]. pmid:12439759.
- 24. Meijers JM, Maaike AP, Cummings GG, Wallin L, Estabrooks CA, Halfens RYG. Assessing the relationships between contextual factors and research utilization in nursing: systematic literature review. Journal of Advanced Nursing. 2006;55(5):622–35. pmid:16907795
- 25. El-Jardali F, Lavis JN, Ataya N, Jamal D. Use of health systems and policy research evidence in the health policymaking in eastern Mediterranean countries: views and practices of researchers. Implement Sci. 2012;7. pmid:22236561
- 26. Chagnon F, Poullot L, Malo C, Gervais MJ, Pigeon ME. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5. pmid:20525270
- 27. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14. pmid:24383766
- 28. Redman S, Turner T, Davies H, Williamson A, Haynes A, Brennan S, et al. The SPIRIT Action Framework: A structured approach to selecting and testing strategies to increase the use of research in policy. Soc Sci Med. 2015;136–137:147–55. Epub 2015/05/26. pmid:26004208.
- 29.
Weiss C, Bucuvalas MJ. Social Science Research and Decision-Making. New York: Columbia University Press; 1980.
- 30. Beyer JM, Trice HM. The utilization process: a conceptual framework and synthesis of empirical findings. Administrative Science Quarterly. 1982;27(4):591–622.
- 31. Weiss CH. The many meanings of research utilization. Public Adm Rev. 1979;39(5):426–31.
- 32. Sumner A, Crichton J, Theobald S, Zulu E, Parkhurst J. What shapes research impact on policy? Understanding research uptake in sexual and reproductive health policy processes in resource poor contexts. Health Res Policy Syst. 2011;9(Suppl 1):53.
- 33. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy. PLoS ONE. 2013;8(10):e77404. pmid:24204823
- 34. Makkar SR, Turner T, Williamson A, Louviere J, Redman S, Green S, et al. The development of ORACLe: A measure of an organisation’s capacity to engage in evidence-informed health policy. Health Research Policy and Systems. 2016;14(4):1–18.
- 35. Ellen ME, Lavis JN, Ouimet M, Grimshaw J, Bedard PO. Determining research knowledge infrastructure for healthcare systems: a qualitative study. Implementation Science. 2011;6:60. Epub 2011/06/08. pmid:21645401; PubMed Central PMCID: PMCPmc3123231.
- 36. Kothari A, Birch S, Charles C. "Interaction" and research utilisation in health policies and programs: does it work? Health Policy. 2005;71(1):117–25. Epub 2004/11/27. doi: S0168-8510(04)00181-2 [pii] pmid:15563998.
- 37. Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88(4):444–83. Epub 2010/12/21. pmid:21166865; PubMed Central PMCID: PMCtoc alert.
- 38. Ross S, Lavis J, Rodriguez C, Woodside J, Denis J. Partnership experiences: involving decision-makers in the research process. J Health Serv Res Policy. 2003;8 Suppl 2:26–34.
- 39. Patera N, Wild C. Linking public health research with policy and practice in three European countries. J Public Health. 2013;21(5):473–9.
- 40. Lomas J, Fulop N, Gagnon D, Allen P. On being a good listener: setting priorities for applied health services research. Milbank Q. 2003;81(3):363–88. pmid:12941000
- 41. Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A. Information transfer: what do decision makers want and need from researchers? Implement Sci. 2007;2:20. pmid:17608940
- 42. El-Jardali F, Lavis JN, Moat KA, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst. 2014;12:2. pmid:24438365
- 43. Evans BA, Snooks H, Howson H, Davies M. How hard can it be to include research evidence and evaluation in local health policy implementation? Results from a mixed methods study. Implement Sci. 2013;8:17. pmid:23402558
- 44. Dobbins M, DeCorby K, Twiddy RN. A knowledge transfer strategy for public health decision makers. Worldviews Evid Based Nurs. 2004;1(2):120–8. pmid:17129325
- 45. Ellen ME, Leon G, Bouchard G, Lavis JN, Ouimet M, Grimshaw JM. What supports do health system organizations have in place to facilitate evidence-informed decision-making? A qualitative study. Implement Sci. 2013;8:84. Epub 2013/08/07. pmid:23915278; PubMed Central PMCID: PMCPmc3750753.
- 46. Denis JL, Lomas J, Stipich N. Creating receptor capacity for research in the health system: the Executive Training for Research Application (EXTRA) program in Canada. J Health Serv Res Policy. 2008;13 Suppl 1:1–7. Epub 2008/04/23. pmid:18325161.
- 47. Huckel Schneider C, Campbell D, Milat A, Haynes A. What are the key organisational capabilities that facilitate research use in public health policy. Public Health Research and Practice. in press.
- 48. Hennink M, Stephenson R. Using research to inform health policy: barriers and strategies in developing countries. J Health Commun. 2006;10. pmid:15804906
- 49. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J. How can research organizations more effectively transfer research knowledge to decision makers? Milbank Q. 2003;81. pmid:12841049
- 50. de Goede J, Putters P, van ger Grinten T, van Oers HAM. Knowledge in process? Exploring barriers between epidemiological research and local health policy development. Health Res Policy Syst. 2010;8:26. pmid:20846419
- 51. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-maker’s perceptions of their use of evidence: a systematic review. J Health Serv Res Policy. 2002;7. pmid:12425783
- 52. Haynes A, Turner T, Redman S, Milat AJ, Moore G. Developing definitions for a knowledge exchange intervention in health policy and program agencies: reflections on process and value. Int J Soc Res Meth. 2014;18(2):145–59.
- 53. Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system to score research engagement actions by health decision makers. Health Res Policy Syst. 2015;13:22. Epub 2015/05/01. pmid:25928693; PubMed Central PMCID: PMCPmc4443514.
- 54. Makkar SR, Williamson A, Turner T, Redman S, Louviere J. Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development. Health Res Policy Syst. 2015;13(1):35. Epub 2015/08/05. pmid:26238566; PubMed Central PMCID: PMCPmc4523001.
- 55. Makkar SR, Williamson A, D'Este C, Redman S. The reliability and validity of SAGE: A system to measure and score policymakers’ use and engagement with research in the development of health policy. Implement Sci. Under review.
- 56. Makkar SR, Brennan S, Williamson A, Turner T, Redman S, Green S. The development of SAGE: a tool to evaluate how policymakers’ engage with and use research in health policymaking. Res Evaluat. In press.
- 57. The CIPHER Investigators. Supporting Policy In health with Research: an Intervention Trial (SPIRIT)—protocol for a stepped wedge trial. BMJ Open. 2014;4(7). pmid:24989620
- 58. Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80(1):125–54. Epub 2002/04/06. pmid:11933791.
- 59. Jacobson N, Butterill D, Goering P. Consulting as a strategy for knowledge transfer. Milbank Q. 2005;83(2):299–321. pmid:15960773
- 60. Brownson RC, Royer C, Ewing R, McBride TD. Researchers and policymakers: travelers in parallel universes. Am J Prev Med. 2006;30(2):164–72. Epub 2006/02/07. pmid:16459216.
- 61. Jones K, Armstrong R, Pettman T, Waters E. Knowledge Translation for researchers: developing training to support public health researchers KTE efforts. Journal of public health (Oxford, England). 2015;37(2):364–6. Epub 2015/05/30. pmid:26022812.
- 62. Meissner HI, Glasgow RE, Vinson CA, Chambers D, Brownson RC, Green LW, et al. The U.S. training institute for dissemination and implementation research in health. Implementation Science. 2013;8(1):12. pmid:23347882
- 63. Chambers DA, Proctor EK, Brownson RC, Straus SE. Mapping training needs for dissemination and implementation research: lessons from a synthesis of existing D&I research training programs. Translational behavioral medicine. 2016. Epub 2016/04/01. pmid:27030472.
- 64. Proctor EK, Landsverk J, Baumann AA, Mittman BS, Aarons GA, Brownson RC, et al. The implementation research institute: training mental health implementation researchers in the United States. Implementation Science. 2013;8(1):105. pmid:24007290
- 65. Stamatakis KA, Norton WE, Stirman SW, Melvin C, Brownson RC. Developing the next generation of dissemination and implementation researchers: insights from initial trainees. Implementation Science. 2013;8(1):29. pmid:23497462
- 66. Norton WE. Advancing the science and practice of dissemination and implementation in health: a novel course for public health students and academic researchers. Public health reports (Washington, DC: 1974). 2014;129(6):536–42. Epub 2014/11/05. pmid:25364057; PubMed Central PMCID: PMCPMC4187297.
- 67. Morrato EH, Rabin B, Proctor J, Cicutto LC, Battaglia CT, Lambert-Kerzner A, et al. Bringing it home: expanding the local reach of dissemination and implementation training via a university-based workshop. Implement Sci. 2015;10:94. Epub 2015/07/05. pmid:26141909; PubMed Central PMCID: PMCPMC4490605.
- 68. Makkar S, Gilham F, Williamson A, Bisset K. Usage of an online tool to help policymakers better engage with research: Web CIPHER. Implement Sci. 2015;10(1):1–11. pmid:25903373
- 69. Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: service provider perspectives. Adm Policy Ment Health. 2007;34(4):411–9. Epub 2007/04/06. pmid:17410420.
- 70. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, et al. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Adm Policy Ment Health. 2008;35(1–2):98–113. Epub 2007/12/19. pmid:18085434.
- 71. Glisson C, Schoenwald SK, Kelleher K, Landsverk J, Hoagwood KE, Mayberg S, et al. Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Adm Policy Ment Health. 2008;35(1–2):124–33. Epub 2007/12/18. pmid:18080741.
- 72. Glisson C, Schoenwald SK. The ARC organizational and community intervention strategy for implementing evidence-based children's mental health treatments. Mental health services research. 2005;7(4):243–59. Epub 2005/12/02. pmid:16320107.
- 73. Glisson C, James LR. The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior. 2002;23(6):767–94.
- 74. Hemmelgarn AL, Glisson C, Dukes D. Emergency Room Culture and the Emotional Support Component of Family-Centered Care. Children's Health Care. 2001;30(2):93–110.
- 75. Nelson TD, Steele RG. Predictors of practitioner self-reported use of evidence-based practices: practitioner training, clinical setting, and attitudes toward research. Adm Policy Ment Health. 2007;34(4):319–30. Epub 2007/02/03. pmid:17268858.
- 76. Zardo P, Collie A, Livingstone C. External factors affecting decision-making and use of evidence in an Australian public health policy environment. Soc Sci Med. 2014;108:120–7. pmid:24632115
- 77. Haynes AS, Derrick GE, Redman S, Hall WD, Gillespie JA, Chapman S, et al. Identifying trustworthy experts: how do policymakers find and assess public health researchers worth consulting or collaborating with? Plos ONE. 2012;7(3):e32665. pmid:22403693
- 78. Brennan SE, McKenzie JE, Turner T, Redman S, Makkar SR, Williamson A, et al. Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policy makers’ capacity to engage with and use research. In progress.