https://orcid.org/0000-0002-0264-4059
Figures
Abstract
Background
High-quality implementation evaluations report on intervention fidelity and adaptations made, but a practical process for evaluating implementation strategies is needed. A retrospective method for evaluating implementation strategies is also required as prospective methods can be resource intensive. This study aimed to establish an implementation strategy postmortem method to identify the implementation strategies used, when, and their perceived importance. We used the rural Transitions Nurse Program (TNP) as a case study, a national care coordination intervention implemented at 11 hospitals over three years.
Methods
The postmortem used a retrospective, mixed method, phased approach. Implementation team and front-line staff characterized the implementation strategies used, their timing, frequency, ease of use, and their importance to implementation success. The Expert Recommendations for Implementing Change (ERIC) compilation, the Quality Enhancement Research Initiative phases, and Proctor and colleagues’ guidance were used to operationalize the strategies. Survey data were analyzed descriptively, and qualitative data were analyzed using matrix content analysis.
Results
The postmortem method identified 45 of 73 ERIC strategies introduced, including 41 during pre-implementation, 37 during implementation, and 27 during sustainment. External facilitation, centralized technical assistance, and clinical supervision were ranked as the most important and frequently used strategies. Implementation strategies were more intensively applied in the beginning of the study and tapered over time.
Conclusions
The postmortem method identified that more strategies were used in TNP than planned and identified the most important strategies from the perspective of the implementation team and front-line staff. The findings can inform other implementation studies as well as dissemination of the TNP intervention.
Citation: Gilmartin H, Jones C, Nunnery M, Leonard C, Connelly B, Wills A, et al. (2024) An implementation strategy postmortem method developed in the VA rural Transitions Nurse Program to inform spread and scale-up. PLoS ONE 19(3): e0298552. https://doi.org/10.1371/journal.pone.0298552
Editor: Masoud Behzadifar, Lorestan University of Medical Sciences, ISLAMIC REPUBLIC OF IRAN
Received: May 25, 2023; Accepted: January 25, 2024; Published: March 8, 2024
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: This paper reports on a mixed-methods study and excerpts of appropriate data are shared within the paper. We are unable to provide full de-identified interview transcripts as this was not discussed with participants during the consent process. De-identified, anonymized survey data is available at: https://figshare.com/s/42e7c2efddfc9401cfa4 https://figshare.com/s/e9dff5106988251604ed https://figshare.com/s/cb1b5c842e77582e5a52.
Funding: This material is based upon work supported (or supported in part) by the Department of Veterans Affairs, Veterans Health Administration (VA), VA Office of Rural Health, and Office of Nursing Services. The Office of Research and Development, Health Services Research and Development Service provides support through the Denver/Seattle Center of Innovation for Veteran-Centered and Value Driven Care. Dr. Gilmartin is supported by Career Development Award number 1IK2HX002567-01A1 from the VA Office of Research and Development. Dr. Jones is supported by grant number K08HS024569 from the Agency for Healthcare Research and Quality. The funders had no rule in study design, data collection and analysis, decision to publish, or preparation of the manuscript. All authors receive some salary support from the Department of Veterans Affairs.
Competing interests: The author(s) received no specific funding for this work.
Abbreviations: CI, Confidence Interval; ERIC, Expert Recommendations for Implementing Change; FRAME-IS, Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies; LISTS, Longitudinal Implementation Strategies Tracking System; QUERI, Quality Enhancement Research Initiative; RE-AIM, Reach, Effectiveness, Adoption, Implementation, Maintenance Framework; TNP, rural Transitions Nurse Program; VA, Veterans Health Administration
Introduction
High-quality implementation evaluations commonly report the fidelity of implementation of an evidence-based intervention, and how the intervention was adapted. However, it is far less common to systematically evaluate the extent to which planned implementation strategies were delivered as intended, and how they were adapted over the life-course of a study [1]. There is good reason to suspect that implementation strategies, just like evidence-based interventions, undergo adaptation, to maximize the “fit” in a particular context [2]. Further, different implementation strategies may be used or emphasized during different phases of a study rather than assuming that all strategies are used in the same way from pre-implementation through sustainment phases [3]. Without systematic assessment and documentation of what strategies were used, when, and how, it is difficult to understand why a particular intervention or implementation outcome was achieved and how this could be replicated [1].
Implementation scientists commonly identify specific or a bundle of implementation strategies a priori to support intervention implementation. Ideally, tracking implementation strategy adaptations–like intervention adaptations–would occur over the course of the study. One option is to use the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS) method [4]. FRAME-IS builds on the FRAME method [2] for tracking intervention adaptation. FRAME-IS includes modules for team members and local implementers to document which implementation strategy is modified, the nature, primary goal, rationale, and timing for the modification, participants in the decision-making process, and the spread of the modification. FRAME-IS is designed to capture modifications to strategies identified a priori, but not those added during implementation of the study. Another option involves longitudinal tracking of implementation strategies using a timeline follow-back procedure during the course of a study [5]. The Longitudinal Implementation Strategies Tracking System (LISTS) tracks the use of implementation strategies identified a priori and during a study, including fidelity concerns and adaptations [5].
The FRAME-IS and LISTS methods are a significant contribution to the field that hold potential to promote rigor and reproducibility in implementation research. However, the problem is these methods are resource-intensive, do not capture the perspectives of the staff outside the implementation team, and do not capture potentially valuable information regarding which implementation strategies were perceived as most useful. In other words, while the FRAME-IS and LISTS methods capture how and why the implementation team adapted implementation strategies, they provide limited insight into the extent to which these changes were perceived by staff receiving the implementation strategy. In addition, the FRAME-IS and LISTS methods provide little guidance regarding how to evaluate implementation strategy fidelity and perceived importance once an intervention has moved to the sustainment phase.
Given this, our team developed an implementation strategy postmortem method, using a 3 year, 11-site implementation program as a case study [6, 7]. Our explicit goals were: 1) to identify and specify which implementation strategies were planned a priori and compare them to which strategies were used and when from the perspective of the implementing team and front-line staff (i.e., fidelity); 2) describe the importance, frequency and ease of use of the implementation strategies from these perspectives, and 3) understand the barriers and contributors to successful implementation, and the perceived targets of each strategy from the perspectives of the implementation team and front-line staff. We explicitly did not evaluate the “dose” of implementation strategies delivered, which is an important but distinct concept related to “how much” of a particular implementation strategy was received. Rather, we argue an initial evaluation should focus on which implementation strategies were used, when, by whom and the perceived importance. Once these are evaluated, then it may be appropriate to analyze the “dose” of those strategies used to aid in interpreting study outcomes.
Materials and methods
We used an iterative mixed methods [8] approach to describe and thematically analyze the multiple phases of data collected. We use the Veterans Health Administration (VA) rural Transitions Nurse Program (TNP) as a case study.
The rural Transitions Nurse Program intervention
The transition from VA hospitals to home is a high-risk period for rural Veterans. This group is particularly vulnerable to gaps in communication and fragmented care coordination, resulting in inadequate access to follow-up care and high rates of hospital utilization [9, 10]. To address these gaps, TNP, a nurse-led pre-and post-discharge care coordination intervention, was implemented at 11 urban VA hospitals that treated large numbers of rural Veterans [6]. This national hybrid-effectiveness study enrolled 3,001 Veterans between April 2017 to September 2019. Veterans enrolled in TNP were more likely to see their primary care provider within 14-days of discharge than 6,002 matched controls (Odds Ratio 2.24, 95% CI 2.05–2.45). TNP enrollment was not associated with reduced 30-day emergency department visits or readmissions but was associated with reduced 30-day mortality (Hazard Ratio 0.33, 95% CI 0.21–0.53) [7].
The TNP study design, implementation and evaluation methods have been previously reported [6, 11–16]. Briefly, TNP was implemented with one nurse and physician champion per site. The transitions nurse was hired and trained to identify high-risk Veterans and coordinate care for Veterans across care settings [7]. During pre-implementation, the TNP implementation team conducted site assessments and process mapping of transitions of care activities to learn local context and understand the workflow at each site [11–14]. Evaluation of TNP was guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) Framework [17].
Postmortem implementation strategy method
Specification.
In 2019, the final year of the TNP study, we sought to enhance our understanding of the implementation strategies used, with guidance from the Expert Recommendations for Implementing Change (ERIC) compilation [18] and the Quality Enhancement Research Initiative (QUERI) Roadmap phases (pre-implementation, implementation, sustainability) [3]. Of note, the FRAME-IS [4] and LISTS [5] methods were not published at the time TNP was being implemented. A multi-phase approach was used that started with text analysis of the TNP grant application to identify the implementation strategies proposed a priori (Fig 1). All eleven members of the TNP implementation team were invited and engaged in a single member checking meeting to review the a priori implementation strategies and matched them to one of the 73 ERIC implementation strategies [18]. A follow-up meeting determined if other ERIC strategies were introduced during TNP. The purpose of this phase was to prompt reflection on the strategies originally outlined, identify if any were added as the program was implemented and sustained, and to label and define each strategy (S1 File).
In phase two, the list of TNP implementation strategies were qualitatively mapped to the QUERI implementation roadmap by the TNP clinical leads and principal investigator to identify when each strategy was used (Fig 1) [3]. The TNP pre-implementation phase was defined as the work conducted from the announcement of grant funding to sites hiring their first transitions nurse. The implementation phase was defined as the launch of the transitions nurse, site champion and leadership engagement efforts to 18 months post-implementation. The sustainment phase was defined as the final 18 months of grant funding, when no new sites implemented the intervention. The purpose of phase two was to understand the application and temporality of strategies over the life-course of the 3-year study [3, 19].
Ranking.
In phase three, six TNP transitions nurses and site champions located at six VA medical centers were invited to participate in a survey to indicate the importance, frequency, and ease of use of the TNP implementation strategies identified in phase one. The six sites were purposely selected for they were currently implementing TNP at their sites. The purpose of phase three was to operationalize each strategy and gather perspectives of those who received the implementation strategy. The survey was developed per the specification categories outlined by Proctor et al. [19] and adapted for this project (S2 File). Each implementation strategy was paired with a TNP relevant definition (i.e., centralized technical assistance = In TNP, we developed and used the Denver-based team as a core group to deliver technical assistance focused on implementation issues). Importance was assessed by the question: “How important were the following implementation strategies in the implementation of TNP?” on an ascending five-point Likert scale (not important to very important). Frequency was assessed using the question: “How frequently were the following strategies used to implement TNP?” on an ascending five-point Likert scale (never to 3+times/week). Ease of use was assessed through the following question: “How easy were the following strategies to develop and use to support TNP implementation? (Please consider the time, effort, people, and materials required),” rated on a five-point ascending Likert scale (not easy to very easy). Item means were calculated for each strategy using Excel v16.59. The TNP implementation strategies were ranked by importance, frequency, and ease of use. The magnitude of difference within and between strategies were presented using a heat map data visualization technique designed in Excel, assigning a 5-color gradient to each Likert scale response [20].
In phase four, a convenience sample of eight members of the TNP implementation and clinical site teams participated in phone interviews to understand who delivered the strategies (i.e., the actor), barriers and contributors to successful implementation, and the targets of each strategy (i.e., the action target) [19]. Participants were asked about the top three most important and frequently used implementation strategies from phase three. Additional questions included the frequency of use and key activities that supported each strategy (S3 File). The interviews were recorded, transcribed verbatim, and analyzed in Excel using deductive, thematic matrix analysis [21]. TNP implementation team members not involved in the interviews double coded three transcripts to ensure coding consistency. When coding discrepancies arose, they were discussed and resolved. In accordance with VA institutional review board this study is a designated program evaluation. The postmortem results are presented in accordance with the Standards for Quality Improvement Reporting and Excellence Guidelines [22].
Results
In phase one, four implementation strategies were listed a priori to support TNP implementation: pre-implementation site assessment, intervention training, audit and feedback, and facilitation [15, 16]. These matched the following ERIC-defined strategies: conduct local needs assessment, conduct ongoing training, audit, and provide feedback, and facilitation. The member checking exercise identified an additional 41 implementation strategies, resulting in a total of 45 ERIC strategies used in TNP. Phase two analyses indicated most TNP strategies (n = 22; 49%) were applied across the three QUERI Roadmap phases, while some were applied only during pre-implementation and implementation phases (n = 10; 22%) or only during the pre-implementation phase (n = 8; 18%). Implementation and sustainment phases (N = 4; 9%) or sustainment phase alone (N = 1; 2%) were less common (Table 1).
In phase three, surveys were received from five TNP transitions nurses and one site champion from 6 VA facilities (100% response rate). Facilitation was rated very important (mean: 5) to implementation. An additional 30 strategies were rated as important (mean: 4.0 to 4.9) (e.g., centralize technical assistance, use data experts) and 14 as neutral (3.0–3.9) (e.g., tailor strategies, shadow other experts). No strategies were identified as less or not important. Facilitation was the most frequently used strategy (mean 4.7; 1-3x/week), followed by 12 strategies used 1-2x month (e.g., centralize technical assistance, create a learning collaborative), 31 strategies used 1-2x a year (e.g., remind clinicians, distribute educational material), and one strategy that was never used (i.e., conduct local consensus discussions). None of the strategies were rated as very easy or easy to use or develop.
Fifteen strategies earned neutral responses (e.g., ongoing consultation, develop resource sharing agreements), 29 were less easy (e.g., promote adaptability, build a coalition), and one was deemed not easy to develop or use (i.e., obtain and use patients/consumers and family feedback). Facilitation was identified as the most important and frequently used TNP strategy, followed by centralized technical assistance and provide clinical supervision. All three were acknowledged as less easy to develop and use then other strategies including use of data experts and educational outreach visits. The top 10 most important strategies used, with their frequency and ease of use ratings are presented in Table 2.
In phase four, interviews with six members of the TNP implementation team, one physician champion, and one transitions nurse (n = 8) indicated the TNP clinical leads primarily provided facilitation, centralized technical assistance, and clinical supervision. The TNP data manager also provided centralized technical assistance. Barriers to successful use included limited engagement from physician champions, site staff turnover, lower technical ability of some transitions nurses and personality differences. Contributors to success were the full-time support from the TNP clinical leads, TNP training sessions, the clinical leads communication styles, and on-site clinical observations that occurred in year two of TNP at all sites. The targets of all three strategies were transitions nurses and physician champions. All three strategies were frequently utilized in the beginning of TNP, but use tapered over time.
Phase four results indicated the primary differences between the facilitation, centralized technical assistance, and provide clinical supervision strategies were the activities required to support each strategy. Key facilitation activities included TNP providing “hands on” support to build relationships to improve program effectiveness, logistical support, and the extensive TNP training program [23]. Key activities reported for centralized technical assistance included the creation of the TNP database, the ability for near-real time audit and feedback, hands-on technical support, and the creation of on-line shared folders and a TNP SharePoint site. Key activities reported for the clinical supervision strategy included the availability of the TNP clinical leads during on-site clinical observations and through email, video, and phone.
Discussion
TNP is an evidence-based intervention that positively impacted the health and safety of rural Veterans [7]. The implementation strategy postmortem method was developed to evaluate TNP implementation strategy fidelity and adaptations during the sustainment phase of the program. The postmortem approach identified 45 implementation strategies deployed in TNP, including the four that were identified a priori. This finding demonstrates the complexity of implementing a national care coordination intervention in multiple sites and the interdependence of various implementation strategies [24].
The postmortem method is an iterative, mixed method and multi-stakeholder approach that increased our understanding of the work that supported implementation of TNP across 11 VA medical centers over three years [25]. The initial research question “Which strategies were used” was best answered through qualitative methods (e.g., text analysis). The next question, “When were they used” was best answered using a quantitative approach due to the interest in collecting multiple perspectives. The final question, “Who and what approaches supported implementation” required a qualitative approach (e.g., interviews) to allow for an in-depth examination and contextualization of the previous findings. This approach allowed for meaningful inferences of the qualitative-quantitative-qualitative data [8], though required time and analytic skills. Future users of the postmortem method will benefit from the survey and interview tools provided in this paper, making the retrospective assessment of implementation fidelity and adaptations less cumbersome and time consuming.
Benefits of the multi-phase postmortem approach included insights into which TNP strategies were used, when, and with what purpose. We learned that many more implementation strategies were used during the 3-year study then were outlined in the TNP grant and none of the strategies were rated as very easy or easy to use or develop. We learned the implementation strategies were strategically introduced and discontinued across the QUERI Roadmap stages and the TNP clinical leads and database manager played a critical role in the success of TNP. This helped explain TNPs impact on increasing access to care and reducing mortality for high-risk Veterans [7]. Further, the postmortem findings identified the most important implementation strategies from the perspective of the implementation team and clinical staff.
Interactive assistance, which is a multifaceted strategy provided through facilitation, centralized technical assistance, and clinical supervision, aims to build trust and provide support to clinical teams [18]. This study revealed the specific aspects of interactive assistance and the many players and activities that enabled successful implementation of TNP and the positive outcomes for Veterans [7]. The interactive assistance implementation approach was strengthened by the hiring of full-time clinical leads who received ongoing communication and relationship training, [26] and the hiring of a database manager to create, adapt and deliver technical assistance to sites. The clinical supervision strategy for TNP was delivered at onboarding and annual intervention training, and through the TNP nurse learning community [27]. The postmortem method provided more than evaluation of implementation strategy fidelity and adaptations. The retrospective evaluation provided key data on the importance of having well-trained staff with dedicated time to devote to implementation of the intervention.
The postmortem method revealed variation in the number of implementation strategies used at different phases of the project across TNP sites, with most strategies used in the pre-implementation phase followed by the implementation, and sustainment phases. The emphasis on pre-implementation strategies can be explained by the TNP team’s efforts to build strong relationships with clinical and operational partners, engage stakeholders, diagnose local capabilities, assess barriers to implementation, and identify meaningful measures to inform the evaluation plan at every site to ensure successful implementation [3]. Implementation and sustainment efforts were streamlined once TNP was adapted to local contextual needs. This finding is important as we reflect on the design of implementation studies. Implementation strategies, like interventions, should dynamically change as the study progresses through its natural phases and the context and related barriers, facilitators, and priorities change. Most implementation studies, including TNP, are designed with static implementation strategies which does not align with how research projects and implementation of programs happen in the real world [5, 27].
Similar findings to our study were reported in the context of implementation of a new treatment for Hepatitis C virus in a national sample of VA medical centers [28]. In this 2017 study, researchers found that VA medical centers used, on average, 25 implementation strategies to support adoption of a new Hepatitis C treatment. Interestingly, the number of implementation strategies used was correlated with the number of Veterans who started the treatment at each site. Comparable to our findings, they also reported the ERIC implementation strategies “provide interactive assistance” and “develop stakeholder interrelationships” had the most significant impact on medication treatment. Of interest, when implementation strategies were tracked over time for Hepatitis C treatment starts, the strategies associated with treatment starts during pre-implementation (i.e., “training/educating,” “interactive assistance,” and “building stakeholder interrelationships”) were different than the strategies associated with treatment starts during the implementation phase (i.e., “evaluative and iterative” and “adapting and tailoring”) [29]. These findings match the TNP experience in that multi-faceted implementation strategies, iteratively selected and tailored to program needs, can have meaningful positive outcomes for patients [30].
The postmortem findings should be interpreted with some limitations in mind. The postmortem approach required retrospective identification, ranking, and specification of implementation strategies by staff who had extended relationships with the study investigators. There is a risk the results were influenced by recall and desirability biases. To minimize these biases multiple data sources (e.g., member checking, surveys, interviews) were used [31]. Further, the small sample and inclusion of perspectives from some, but not all TNP clinical sites, limits our understanding of every TNP members perception of the implementation strategies.
Conclusions
Evaluating implementation strategy fidelity and adaptations are important activities to support implementation, sustainment and scale up of evidence-based interventions. The postmortem method described in this paper fills a critical gap. Implementation studies should be required, as part of good scientific practice, to evaluate the implementation strategies they used post-hoc. The method is an accessible and pragmatic approach to gather insights into the most important and frequently used implementation strategies deployed to support complex interventions. Moving forward, implementation scientists and healthcare teams can use the postmortem method and tools to enhance the implementation of evidence-based interventions across various patient safety projects (e.g. healthcare-acquired conditions, pain management) related to improving the health, safety, and equity of healthcare.
Supporting information
S3 File. Interview guide: TNP implementation strategies.
https://doi.org/10.1371/journal.pone.0298552.s003
(DOCX)
Acknowledgments
We would like to thank the Transitions Nurses and site champions that implemented the TNP, members of the Denver/Seattle Center of Innovation implementation and evaluation team and Amy Ladebue for qualitative analytic assistance.
References
- 1. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implementation science. 2007;2(1):1–9. pmid:18053122
- 2. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implementation Science. 2019;14(1):1–10.
- 3. Kilbourne AM, Goodrich DE, Miake-Lye I, Braganza MZ, Bowersox NW. Quality Enhancement Research Initiative Implementation Roadmap: Toward Sustainability of Evidence-based Practices in a Learning Health System. Medical care. 2019;57:S286–S93. pmid:31517801
- 4. Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implementation Science. 2021;16:1–12.
- 5. Smith JD, Merle JL, Webster KA, Cahue S, Penedo FJ, Garcia SF. Tracking dynamic changes in implementation strategies over time within a hybrid type 2 trial of an electronic patient-reported oncology symptom and needs monitoring program. Frontiers in Health Services. 2022;Nov:983217. pmid:36925901
- 6. Leonard C, Lawrence E, McCreight M, Lippmann B, Kelley L, Mayberry A, et al. Implementation and dissemination of a transition of care program for rural veterans: A controlled before and after study. Implementation Science. 2017;12(1):123. pmid:29058640
- 7. Gilmartin HM, Warsavage T, Hines A, Leonard C, Kelley L, Wills A, et al. Effectiveness of the rural Transitions Nurse Program for veterans: a multicenter implementation study. Journal of Hospital Medicine. 2022;17(3):149–57. pmid:35504490
- 8.
Tashakkori A, Johnson RB, Teddlie C. Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. 2 ed. Thousand Oaks, California: SAGE Publications; 2020.
- 9. Burke RE, Kelley L, Gunzburger E, Grunwald G, Gokhale M, Plomondon ME, et al. Improving transitions of care for veterans transferred to tertiary VA medical centers. American Journal of Medical Quality. 2018;33(2):147–53. pmid:28691503
- 10. Garvin LA, Pugatch M, Gurewich D, Pendergast JN, Miller CJ. Interorganizational Care Coordination of Rural Veterans by Veterans Affairs and Community Care Programs: A Systematic Review. Medical care. 2021;59(6 Suppl 3):S259. pmid:33976075
- 11. Gilmartin H, Lawrence E, Leonard C, McCreight M, Kelley L, Lippmann B, et al. Brainwriting Premortem: A Novel Focus Group Method to Engage Stakeholders and Identify Preimplementation Barriers. Journal of Nursing Care Quality. 2018;34(2):94.
- 12. Leonard C, Gilmartin H, McCreight M, Kelley L, Lippmann B, Mayberry A, et al. Operationalizing an implementation framework to disseminate a care coordination program for rural Veterans. Journal of general internal medicine. 2019;34(1):58–66.
- 13. McCreight MS, Gilmartin HM, Leonard CA, Mayberry AL, Kelley LR, Lippmann BK, et al. Practical use of process mapping to guide implementation of a care coordination program for rural Veterans. Journal of general internal medicine. 2019;34(1):67–74. pmid:31098974
- 14. McCreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Translational Behavioral Medicine. 2019;Nov 25(6):1002–11. pmid:31170296
- 15. Rabin BA, McCreight M, Battaglia C, Ayele R, Burke RE, Hess PL, et al. Systematic, Multimethod Assessment of Adaptations Across Four Diverse Health Systems Interventions. Frontiers in public health. 2018;6:102. pmid:29686983
- 16. Connelly B, Ujano-De Motta LL, Leonard C, Mayberry A, Kelley L, Gaskin D, et al. Mapping the reach of a rural Transitions Nurse Program for veterans with geographic information systems. Implementation Science Communications. 2020;1(1):1–7. pmid:32885193
- 17. Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. American journal of public health. 2013;103(6):e38–e46. pmid:23597377
- 18. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change project. Implementation Science. 2015;10(1):21.
- 19. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implementation Science. 2013;8(1):139. pmid:24289295
- 20. Ohannessian R, Bénet T, Argaud L, Guérin C, Guichon C, Piriou V, et al. Heat map for data visualization in infection control epidemiology: an application describing the relationship between hospital-acquired infections, simplified acute physiological score II, and length of stay in adult intensive care units. American Journal of Infection Control. 2017;45(7):746–9. pmid:28549877
- 21. Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qualitative Health Research. 2002;12(6):855–66. pmid:12109729
- 22. Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. American Journal of Critical Care. 2015;24(6):466–73. pmid:26523003
- 23. Kelley L, Broadfoot K, McCreight M, Wills A, Leonard C, Connelly B, et al. Implementation and Evaluation of a Training Curriculum for Experienced Nurses in Care Coordination: The VA Rural Transitions Nurse Training Program. Journal of Nursing Care Quality. 2023;Jul-Sep(3):286–92. pmid:36857291
- 24. Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implementation Science. 2019;14(1):1–13.
- 25. McCarthy MS, Motta U-D, Lexus L, Nunnery MA, Gilmartin H, Kelley L, et al. Understanding adaptations in the Veteran Health Administration’s Transitions Nurse Program: refining methodology and pragmatic implications for scale-up. Implementation Science. 2021;16(1):1–12.
- 26. Gilmartin H, Battaglia C, Warsavage T, Connelly B, Burke RE. Practices to Support Relational Coordination in Care Transitions: Observations from the VA Rural Transitions Nurse Program. Health Care Management Review. 2022;April-Jun 01(2):109–14. pmid:33181554
- 27. Goorts K, Dizon J, Milanese S. The effectiveness of implementation strategies for promoting evidence informed interventions in allied healthcare: a systematic review. BMC Health Serv Res. 2021;21(1):1–11.
- 28. Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Kirchner JE, Proctor EK, et al. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implementation Science. 2017;12(1):60. pmid:28494811
- 29. Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Gonzalez R, Park A, et al. Longitudinal assessment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implementation Science. 2019;14(1):1–12.
- 30. Yakovchenko V, Morgan TR, Chinman MJ, Powell BJ, Gonzalez R, Park A, et al. Mapping the road to elimination: a 5-year evaluation of implementation strategies associated with hepatitis C treatment in the veterans health administration. BMC Health Serv Res. 2021;21(1):1–9.
- 31. Akiba CF, Powell BJ, Pence BW, Nguyen MX, Golin C, Go V. The case for prioritizing implementation strategy fidelity measurement: benefits and challenges. Translational Behavioral Medicine. 2022;12(2):335–42. pmid:34791480