https://orcid.org/0000-0003-2751-3001
Figures
Abstract
Background
The Resilience Assessment Grid (RAG) has attracted increasing interest in recent healthcare discourse as an instrument to understand and measure the resilience performance of socio-technical systems. Despite its growing popularity in healthcare, its applicability and utility remain unclear. This scoping review aims to understand the practical application of RAG method and its outcomes in healthcare.
Method
We followed the Arksey and O’Malley, and the Levac and colleagues’ framework for scoping reviews and the PRISMA-ScR Checklist. We conducted searches of three electronic databases [Medline, Embase and Web of Science] in May 2021. Supplementary searches included Google Scholar, web and citation searches, and hand searches of the nine seminal edited books on Resilience Engineering and Resilient Health Care. All English language, empirical studies of RAG application in the healthcare setting were included. Open Science Framework [Registration-DOI. 10.17605/OSF.IO/GTCZ3].
Results
Twelve studies met the inclusion criteria. Diversities were found across study designs and methodologies. Qualitative designs and literature reviews were most frequently used to develop the RAG and applied it in practice. Eight of the studies had qualitative designs, three studies had mixed-methods designs and one study had a quantitative design. All studies reported that the RAG was very helpful for understanding how frontline healthcare professionals manage the complexity of everyday work. While the studies gained insights from applying the RAG to analyze organizational resilience and identify areas for improvement, it was unclear how suggestions were implemented and how they contributed to quality improvement.
Conclusion
The RAG is a promising tool to manage some of the current and future challenges of the healthcare system. To realise the potential benefits of the RAG, it is important that we move beyond the development phase of the RAG tool and use it to guide implementation and management of quality initiatives.
Citation: Safi M, Thude BR, Brandt F, Clay-Williams R (2022) The application of resilience assessment grid in healthcare: A scoping review. PLoS ONE 17(11): e0277289. https://doi.org/10.1371/journal.pone.0277289
Editor: Yusuke Tsutsumi, Kyoto University / National Hospital Organization Mito Medical Center, JAPAN
Received: February 27, 2022; Accepted: October 24, 2022; Published: November 4, 2022
Copyright: © 2022 Safi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting information files.
Funding: This work was supported by the University Hospital of Southern Denmark as part of a Ph.D. project. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: This work was supported by the University Hospital of Southern Denmark as part of a Ph.D. project. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
The Resilience Assessment Grid (RAG) method developed by Hollnagel [1, 2] provides the conceptual basis for analyzing and supporting the organizational resilience of a complex system. The RAG proposes four abilities [responding, monitoring, learning and anticipating] that jointly enable resilient performance. Resilient performance means that an organization can function as required under expected and unexpected conditions alike such as changes, disturbances and opportunities [2]. Hollnagel suggests that it is not possible to directly measure resilience. However, the four abilities of RAG can help us understand what enables resilience performance in everyday work and conversely what, if it was missing, renders a system brittle [1, 2]. The intention with RAG is to use the four abilities as a set of proxy measures to construct a resilience profile of the system under assessment over time. The RAG provides a generic guideline as well as a set of example questions that need to be tailored for each new system to which it is applied [1, 2]. The set of example questions is provided alongside advice to develop original questions that are based on the underlying construct of the four resilience potentials, but tailored to fit the study setting.
Further, the RAG can be used to collect repeated measures on a single system, to inform continuous improvement and comparison. However, RAG is usually not applied to compare different systems or organizations, and is not considered by Hollnagel to be suitable for that purpose [3].
The RAG is based on resilience engineering (RE) principles [2, 4], which advocate for a system view that recognizes that adoption of a proactive approach is necessary for successful performance and safety in large complex socio-technical systems [5]. Given the challenges regarding safety, effectiveness and value shared by most healthcare systems [3, 4, 5], RE and RAG have garnered considerable interest. The RAG has been applied across several domains in addition to healthcare [6–8] to analyze and support resilience performance; these include aviation [9], air traffic management [10], nuclear power plants [11] and the water sector [12]. In healthcare, the RAG has been primarily applied in emergency care [7, 13] and anesthesia departments [14]. However, the RAG does not prescribe a specific method to develop the questions and rate them, which means that the RAG has been operationalized in different ways in different settings. Despite its growing popularity in healthcare, its applicability and utility remain unclear. There is no comprehensive description of the state-of-the-art application of the RAG in healthcare available to practitioners and academics who wish to be aware of its practical applications and developments since its inception. We, therefore, sought to understand the application of the RAG in healthcare to support resilient performance, including how and why it has (or has not) worked. To develop this knowledge, we undertook a scoping review of the literature on RAG applications in healthcare, focusing on empirical studies.
With a view to assisting practitioners and decision-makers as well as researchers, we set out to clarify:
- What methods are used to adapt the RAG?
- How has the RAG been put into practice and in which settings?
- What outcomes have been attributed to RAG application in healthcare?
Methods
A scoping review methodology was chosen in order to examine the extent, range, and nature of research activity in the chosen field. We elected to use a scoping approach because it would allow for different study designs to be included and their findings synthesized. We followed the methodological stages outlined by Arksey and O’Malley [15] and further refined by Levac et. Al [16]. It entails: 1. Identifying the research question, 2. Identifying relevant studies, 3. Study selection, 4. Charting the data, and 5. Collating, summarizing, and reporting the results. This type of review does not include a quality assessment of the included articles. A protocol for the scoping review was published in the Open Science Framework (OSF)(Registration DOI. 10.17605/OSF.IO/GTCZ3). The reporting of the review is guided by the PRISMA-ScR Checklist [17] (S1 Appendix).
Eligibility criteria
The inclusion criteria was to select studies from the healthcare setting, reporting on empirical research on RAG, published in peer-reviewed journals or books, as well as unpublished reports. These were restricted to English language. No limitations were set regarding publication year; however, the search was conducted in May 2021. Articles that described the RAG in non-healthcare settings, resilience papers not using the RAG, or describing individual or community resilience; conference abstracts with no full texts; and theoretical and review papers were all excluded.
We were not concerned with individual coping skills or resistance to stress and burnout but rather how the RAG method has been used in healthcare to understand how work system function.
Studies that applied the “four abilities of resilience engineering” were excluded, as the specific abilities identified were not necessarily consistent with those that comprised the RAG. We thereby avoided including studies that had a focus on resilience engineering theory, but did not apply the RAG method per se.
Information sources
The scoping review aimed to identify and include a wide range of literature such as original research, book chapters, and reports, since much of the research on RAG is published in scholarly books and technical reports, rather than scientific journals. After the initial pilot testing of search terms, a final search strategy was developed with the assistance of a specialist librarian. The databases systematically searched were Medline, Embase and Web of Science. The search was developed for Medline and Embase and was translated to Web of Science. We included the term “health care or healthcare” in Web of Science since Web of Science also consists of non-healthcare sectors. This led to a balanced precision and sensitivity in the search. For this review, supplementary searches were necessary as well. Supplementary searches were conducted in Google Scholar, edited scholarly books on resilience engineering and a web search at the Resilience Health Care Net website. Additionally, we included the citations of the Resilience Assessment Grid paper [2] by Hollnagel as per advice of the specialist librarian. Table 1 shows the search strategy.
Selection of sources of evidence
Reference details, including titles and abstracts where available, were downloaded into the reference management software Endnote X9 [18]. After removal of the duplicates in Endnote, all references were imported in Covidence [19]—a program that organizes and facilitates the initial screening of titles and abstracts and full-text screening. Two reviewers (RCW, MS), in parallel, independently applied the inclusion and exclusion criteria to title and abstract screening. In the next step, two reviewers (BRT, MS) independently assessed the full-texts of the included citations for final inclusion. A librarian assisted with retrieving the full-texts. In both abstract and full-text screening, the citations where the reviewers did not agree on inclusion were discussed until consensus was reached.
Data charting process and data items
The Microsoft Excel data extraction spreadsheet was specifically designed for this review. Data concerning study characteristics, e.g., authors, year of publication, country where the study was conducted, aim, setting, and the methodology, e.g., design and data as well as information related to the objectives of the review, i.e., how RAG had been applied in healthcare and the outcomes of using RAG were extracted (Table 2). The data extraction form was piloted by one reviewer because of the small number of studies included in the review. The pilot resulted in minor changes to the extraction form. Three reviewers performed data extraction independently. The first reviewer (MS) extracted the data from all the studies (n = 12). The second (BRT) and third reviewer (RCW) divided the studies (50/50) randomly between them and extracted the data independently. Lastly, the extracted data was reviewed by the three reviewers, and disagreements were discussed in the group.
Data processing and analysis
A narrative synthesis [20] was performed for this review. It described the scope of existing research and summarized data using structured narrative and summary tables. Data synthesis was undertaken in the following stages: i) evidence mapping ii) tabular and graphical presentation of the included studies.
The dataset was organized around the characteristics of the studies, content, and outcomes of RAG use in healthcare. The first author completed the analysis and conferred with the other authors regarding any uncertainties.
Results
The systematic search retrieved 2229 citations, and after removal of duplicates, 1365 citations were retained. Overall, 42 references were considered potentially eligible. After the full-text assessment, 12 were included and 31 articles were excluded. Fig 1 illustrates the inclusion and exclusion of citations at each stage of the screening process, using the PRISMA flow diagram.
Characteristics of the studies
Table 2 provides overview of the characteristics of the included studies. The included studies originated from a wide range of countries such as Taiwan, Australia, Canada, United Kingdom, Brazil and United States (see Fig 2). More than 50% of the studies were peer reviewed journal articles. Fig 3 shows the distribution of included reviews published per year (2014–2021) and by document type. Fig 3 shows a small increase in peer reviewed publications from 2018 onwards. The studies were primarily conducted in hospital settings. Only one study had included two primary care locations [21]. The majority of the studies about RAG were conducted in hospital emergency and anesthesia departments [14, 22]. The remainder were single studies in multi-specialty hospitals [23, 24], in intensive care unit [6] and operating theaters [8]. Eight studies [8, 13, 21, 23–27] employed a qualitative design, three studies [6, 7, 22] a mixed-methods design and one study [14] was purely quantitative (see Fig 4). All studies employed a case study design except one study [14] which had cross-sectional design.
The primary purpose of the included studies was how to develop or adapt the RAG in order to assess or understand the resilience performance of the selected study setting or organization. Eleven studies were prospective. The remaining single study [26] used the RAG to retrospectively examine the success and failures of information systems that were challenged by an unexpected event, thunderstorm asthma.
Ways of contextualizing the RAG
In the included studies, qualitative methods and literature review were most commonly used to adapt and develop the RAG tool for the specific study setting (number of the studies). In the study by Alder [7], focus group interviews and individual semi structured interviews were conducted based on the RAG with the nurses in the ICU. The interviews were used to generate the RAG questionnaire referring to specific system activities such as deteriorating patients, allocation of workload, knowing what to do and when, knowing when colleagues need help, and coordinating admissions and discharges. Hunte and Marsden [27] used dialogue workshops consisting of frontline care providers, support staff and leaders to explore and refine the RAG components. The content of the ED specific RAG was derived from these workshops. In the study by Patriarca et al. [22] the generic RAG questions were adapted to the anesthesia domain based on a literature review. Following this, the Analytical Hierarchy Process (AHP) and focus group interviews were used to refine the AHP-RAG. AHP is a method used for organizing and analyzing complex decisions using pairwise comparison. Patriarca et al. [22] used AHP to define the weights of the items. Questions on communication, teamwork, role and responsibilities, expertise, evaluation of process etc. were developed. The study by Falegnam et al. [14] further refined the original AHP-RAG by Patricia et al. to fit the anesthesia domain. This was conducted with the initial literature review and a panel of three subject matter experts (SMEs), who revised the AHP-RAG questionnaire to describe properly the four RAG resilience potential in terms of items related to the anesthesia domain. Likewise, Darrow et al. [21] adapted the original RAG questions to fit the system with the assistance of a literature review and the resilience survey by Lindvall et al. [28]. However, Darrow had included a fifth resilience category, i.e. individual resilience. The questionnaire under each resilience category referred to questions such as resourcefulness, communication, sensemaking and bricolage, team efficacy, safety, psychological safety and problem-solving. The two studies by Chuang et al. [13, 25] are related. In the first study [25] the original RAG questionnaire was first translated into the Chinese language and then edited to fit the corresponding ED system under study by an expert panel. Focus group interviews were used to test and further refine the ED-RAG. In the second study [13], the ED-RAG had some minor changes and was applied in 4 ED hospitals. Bertoni et al. [6] used the RAG framework in combination with Social Network Analysis (SNA) to model interaction between the four resilience abilities in an ICU. The SNA-RAG included questions on social interactions, frequency of the interactions, interruptions, daily rounds, and identifying peers/actors. The remaining studies [8, 23, 24, 26] used the RAG as a framework to develop interview guides and for data analysis to understand resilience.
Additionally, in the studies [7, 13, 14, 22, 25, 27] there was some imbalance in the number of items for each of the four resilience potentials. For example in the study by Alders [7], the potential to respond had 23-items, monitoring had 6-items, learning had 5-items and anticipating had 3-items. In the study by Darrow [21], the distribution of the items for each of the four resilience potentials was not clear.
Ways of applying the RAG
In the included studies, qualitative methods were most commonly used to put the RAG tool into practice. Of these studies, two involved focus group interviews [13, 25], one involved semi-structured interviews [21], one used the RAG in document analysis [26], one study applied the RAG in monthly meetings [27] and three studies [8, 23, 24] used the RAG framework to analyze qualitative data retrieved through interviews and observations. In the qualitative studies, there was a very large sample size variation. The minimum sample size for each setting was minimum 4 and maximum 35. The interviews included a wide range of professional groups, including physicians, nurses, receptionists, and employees with leadership and management roles at various levels. However, the majority of the respondents were frontline healthcare clinicians.
The quantitative studies included four studies [6, 7, 14, 22] that applied the RAG tool in survey format. Three of the studies [6, 7, 22] conducted qualitative interviews to elaborate on the survey results. One of the study [6] used the RAG method in combination with social networking theory to shed light on the ICU actors interactions in relation to the four resilience potentials. The survey participants were physicians, nurse technicians, allied health professionals, psychologists, pharmacists, nutritionists, and speech and occupational therapists. The study by Alders [7] sent a RAG survey to nurses at different levels in an Acute Medical Unit. The study calculated the mean for each item and the overall mean for each resilience potential. The two remaining studies [14, 22] were conducted in the anesthesia department. The first study [22] applied the AHP-RAG survey and involved 12 neuro-anesthesiologist. The second study [14] adopted the original AHP-RAG and applied it in a multi-country survey involving anesthesia professionals to test and validate the AHP-RAG survey. Two [14, 22] out of four studies visualized the results in radar chart or star plot as well. The Radar charts are a way to visualize multivariate data and make visible concentrations of strengths and weaknesses. The radar chart is not itself a measure of resilience [4].
Furthermore, the studies used different methods to define the answer’s score against each question. Most of the studies [6, 7, 13, 14, 21, 22, 25] used a 4-7-point Likert scale. In some of the studies [13, 21, 25], the researchers coded and rated the answers. Hence, the calculation of the average score for each item and for each of the four resilience potentials differed, see Table 2.
Outcomes of resilience assessment grid use in healthcare
All the included studies reported that the RAG was very helpful for understanding how frontline healthcare professionals manage the complexity of everyday work. Following the application of RAG, the studies most commonly reported; increased staff engagement through dialogues and discussions related to the four resilient potentials; identified areas for quality improvement; provided an overview about how well the system responds to expected and unexpected conditions in daily operations; assisted with improving the patient pathway; assisted with improving working condition; and provided a resilient profile of the system or organization related to response, monitor, learn and anticipate(see Table 3.).
Although 12 studies identified areas for improvement, only two studies reported that the improvements had been implemented. In the study by Hunte & Marsden [27] the RAG was used in an emergency department to enhance their collective ability to manage complexity in everyday practice and monitor their work overtime. Following the development of the context specific RAG, the emergency department: i) initiated a monthly interdisciplinary departmental meeting to consider each domain of the RAG; ii) implemented operational metrics to assess their performance, and iii) created an action plan linked to the metrics. Additionally, out of the 12 studies, only the Hunte & Marsden study repeated the RAG. In the study by Hegde et al., a Resilience Mapping Framework [RMF] [23] was developed to illustrate the resilience capabilities and their relationships across different levels of organizational scale for specific cases. Following this, the study applied the RMF to specific issues, such as how to manage patients that have the propensity to become violent and reviewing patients charts before surgery to detect potential harm.
The outputs and benefits identified in the studies were not systematically measured. While the findings showed that the staff identified system level interventions, it was unclear how the suggestions should be implemented and how they added to quality improvement.
Discussion
In this review we aimed to provide an overview and synthesis of how the RAG method has been applied in healthcare. The review found only 12 studies that had applied RAG in the healthcare setting. The RAG method was primarily applied in the hospital setting. The included studies had primarily qualitative and mixed-method designs. While there was diversity across study design and methods, qualitative methods and literature reviews were most frequently used to develop the RAG and to apply it in practice. Only four of the studies [6, 7, 14, 22] used quantitative surveys to assess resilience performance. Three of these four [6, 7, 22] studies also conducted qualitative interviews to further elaborate the quantitative results. Furthermore, some of the studies [8, 23, 24, 26] applied the RAG as a framework for data analysis to understand resilience in relation to the four resilience potentials. Once a RAG is performed, it can be repeated to monitor the continuous improvement of the system or organization over time [2]. While this is the intended purpose of the instrument [1, 2], only one study [27] reported that the RAG was repeated in monthly interdisciplinary meetings to monitor the ED’s performance and work in practice. This may be impacted by financial and other constraints on the project, such as funding and time. A solution to this can be to apply the RAG pre- and post-organizational changes, or to update the survey items and repeat it again.
While the included studies were heterogeneous in terms of data collection, the review showed that the RAG was frequently applied in high-risk areas within healthcare, such as the acute care and the anesthesia domains. The majority of the studies demonstrated how a RAG questionnaire can be developed and used to profile the performance of an organization in terms of the four resilience abilities and identify areas for improvement. Similarly, the RAG tool was applied in other industries such as the oil and gas industry [29], the water sector [12], the Swedish Civil Aviation Administration [9], and process industries [30]. The wide areas of application of the RAG method demonstrates its flexibility and applicability in different types of complex socio-technical systems and at different scales. However, both in the healthcare sector and in other industries there is an implementation gap. It is unclear how the results from RAG contributed to quality improvement initiatives that were put into practice. The suggestions for quality improvement in the studies required further work.
Another concern, that has been raised regarding the RAG method is that it does not prescribe weighting the four abilities. The review confirmed this problem, finding imbalance in the number of items for each of the four resilience potentials in the studies. Hollnagel [2] reiterates the importance of addressing all the abilities to some extent to achieve resilience. Apneseth et al. [29] argues that the relative importance of the four abilities of RAG varies depending on the system in question. While all organizations to some extent need to develop the ability to respond, many organizations also put some effort into the learning ability to improve their responses to activities. However, not all organizations needs to put the same emphasis on monitoring and anticipating ability in order to be resilient [29]. Another option may be to use the analytical hierarchy process framework proposed by Patriarca et al. [22] to study the different weights of the abilities and their relative influence on overall organizational resilience. However, the AHP is a time-consuming and difficult process [22], which may discourage practitioners and managers from using the RAG framework.
Additionally, the review showed that most of the studies used the underlying construct of the RAG to develop the questions pertinent to the 4 resilience potentials, i.e. respond, monitor, learn and anticipate. In the RAG surveys, each of the four resilience potentials included sub-dimensions or themes such as communication, sensemaking, role and responsibility, teamwork, expertise, social interaction, and interruptions. Only three of the studies [13, 25, 27] followed the original RAG questions closely and edited them to fit their system. The study by Darrow [21] included a fifth category, i.e. individual resilience. This indicates the RAG’s flexibility and its increasing potential benefits to cover all aspects of the sociotechnical system. The review also showed that there is no clear method dictating how to define and rate the answer’s score against each question. Hence, some researchers calculated the average score for each item and for each of the four resilience potentials, see Table 2. Aggregating items in this way is a problematic approach, if not accompanied by a statistical analysis (such as a factor analysis) to show that the aggregated items are related.
Resilience in itself is considered to be difficult or impossible to measure. Some other methods and tools such as the Functional Resonance Analyses Method (FRAM) method [31–35], Concepts for Applying Resilience Engineering (CARE) model and the Team Resilience Framework(RMF) [36–38], have also been applied in the healthcare setting to elucidate the complexity of everyday clinical work and to suggest new perspective to improve safety and resilient performance in daily routines. Most of the methods used in healthcare to understand resilient performance are still in their early development and are more theoretical rather than ready-made tools for practical use in organizations [39]. The RAG and FRAM are conceptually clear and established methods [7] that have been used across different healthcare settings. Compared to other tools and methods, the RAG provides healthcare organizations sufficient methodological guide for how to assess, analyze and manage resilient performance at system level [1, 2, 7].
This review showed that the RAG is frequently applied in practice by other researchers to understand the resilient performance of their system. However, the literature shows that the application of the RAG tool has not moved beyond the developing and testing phase, to using the RAG tool to implement and manage quality initiatives in healthcare. The application of the RAG method in healthcare is still evolving, evidenced by the small increase in peer-reviewed publications since 2018. Most of the research on RAG is conducted in developed countries; however, more research is needed to investigate whether the RAG is applicable in countries with different health systems or more limited resources. Future studies should focus on how to use the knowledge derived from application of the RAG to implement changes in the system leading to improved outcomes.
Strengths and limitations
The review followed a comprehensive methodology for scoping reviews as outlined by Arksey and O’Malley [15] and the reporting of the study followed the PRSIMA-ScR [17]. A research librarian was also involved in the review process and assisted with developing the search strategy and retrieving the full texts, which strengthened the quality of our review. To our knowledge, this is the first scoping review on the RAG and its application in the healthcare setting.
Although only English language articles were included in the review, some of these reported on studies where the RAG was applied in languages other than English.
Conclusion
Research on the RAG began approximately a decade ago. It is a young and evolving method and has been applied in different industries, including healthcare, to investigate organizational resilience. The results from the RAG can enhance understanding of everyday operational work, identify challenges to safe and effective patient care, and create a resilience profile of the system or organization related to four resilience potentials (respond, monitor, learn, anticipate). While the reviewed studies gained insights from applying the RAG to analyze organizational resilience, it was unclear in many cases if improvements had been implemented. Despite the fact that the RAG provides a tool for managers to support the resilience performance of a complex socio-technical system, most studies reported that application was limited to a specific unit or department without the strong support of the senior managers, which led to limited organizational reach. To realize the potential benefits of the RAG, it needs to be accessible and manageable when translated to a new setting. More research is needed to investigate how the RAG method can be used to not only identify quality improvements but to implement and manage them in practice. Further, it is important that in the process of developing and applying the RAG, senior managers and other key stakeholders are involved in order to ensure that identified improvements are implemented, and evaluated through repeating the RAG post-intervention.
References
- 1.
Hollnagel E. RAG-the resilience assessment grid. Safety-II in practice: developing the resilience potentials: Routledge; 2017. p.50–77
- 2.
Hollnagel E. Epilogue: RAG–the resilience analysis grid. Resilience engineering in practice: CRC Press; 2017. p. 275–96.
- 3.
Hollnagel E. Safety-II as a management principle: implications for managing and developing an organization; 2020 [https://safetysynthesis.com/onewebmedia/Introduction%20to%20S-II%20Part%202%20%28OCT2020%29.pdf.]. [Accessed 28 of October 2021].
- 4.
Woods DD, Hollnagel E. Prologue: Resilience engineering concepts. Resilience engineering: CRC Press; 2017. p. 1–6.
- 5.
Hollnagel E, Woods DD, Leveson N. Resilience engineering: Concepts and precepts: Ashgate Publishing, Ltd.; 2006.
- 6. Bertoni VB, Saurin TA, Fogliatto FS, Falegnami A, Patriarca R. Monitor, anticipate, respond, and learn: Developing and interpreting a multilayer social network of resilience abilities. Safety Science. 2021;136:105148.
- 7.
Alders MDL. A reflective process for analysing organisational resilience to improve the quality of care: King’s College London; 2019.
- 8.
Mahmoud Z, Churruca K, Ellis LA, Clay-Williams R, Braithwaite J. Using qualitative methods to understand resilience in complex systems. Transforming Healthcare with Qualitative Research: Routledge; 2020. p. 60–70.
- 9.
Ljungberg D, Lundh V. Resilience Engineering within ATM-Development, adaption, and application of the Resilience Analysis Grid [RAG]. 2013.
- 10. Patriarca R, Gravio GD, Costantino F. Resilience engineering to assess risks for the air traffic management system: A new systemic method. International Journal of Reliability and Safety. 2016;10[4]:323–45.
- 11.
Sakuda H, Kitamura M, editors. Resilience Assessment Grid [RAG] for facilitating safety consciousness of nuclear power plant personnel. REA Symposium on Resilience Engineering Embracing Resilience; 2019.
- 12. Rodríguez M, Lawson E, Butler D. A study of the Resilience Analysis Grid method and its applicability to the water sector in England and Wales. Water and Environment Journal. 2020;34[4]:623–33.
- 13. Chuang S, Ou J-C, Ma H-P. Measurement of resilience potentials in emergency departments: Applications of a tailored resilience assessment grid. Safety Science. 2020;121:385–93.
- 14. Falegnami A, Bilotta F, Pugliese F, Costantino F, Di Gravio G, Tronci M, et al. A multicountry comparative survey about organizational resilience in anaesthesia. Journal of Evaluation in Clinical Practice. 2018;24[6]:1347–57. pmid:30334323
- 15. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International journal of social research methodology. 2005;8[1]:19–32.
- 16. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implementation science. 2010;5[1]:1–9. pmid:20854677
- 17. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews [PRISMA-ScR]: checklist and explanation. Annals of internal medicine. 2018;169[7]:467–73. pmid:30178033
- 18. Hupe M. EndNote X9. Journal of Electronic Resources in Medical Libraries. 2019;16[3–4]:117–9.
- 19. Babineau J. Product review: covidence [systematic review software]. Journal of the Canadian Health Libraries Association/Journal de l’Association des bibliothèques de la santé du Canada. 2014;35[2]:68–71.
- 20. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version. 2006;1:b92.
- 21.
Darrow L. Exploring the factors that drive organizational resilience: Lessons from healthcare. 2017. [Available from: Exploring the factors that drive organizational resilience: lessons from healthcare—PDF Free Download [pdffox.com]]. [Accessed 28 of October 2021].
- 22. Patriarca R, Di Gravio G, Costantino F, Falegnami A, Bilotta F. An analytic framework to assess organizational resilience. Safety and health at work. 2018;9[3]:265–76. pmid:30370158
- 23. Hegde S, Hettinger AZ, Fairbanks RJ, Wreathall J, Krevat SA, Bisantz AM. Knowledge elicitation to understand resilience: A method and findings from a health care case study. Journal of Cognitive Engineering and Decision Making. 2020;14[1]:75–95.
- 24.
Hegde S, Wreathall J, Hettinger AZ, Fairbanks RJ, Wears RL, Bisantz AM, editors. Towards the development of a resilience engineering tool to improve patient safety: the RETIPS approach. Proceedings of the Human Factors and Ergonomics Society Annual Meeting; 2014: SAGE Publications Sage CA: Los Angeles, CA.
- 25. Chuang S, Ou J-C, Hollnagel E, Hou S-K. Measurement of resilience potential-development of a resilience assessment grid for emergency departments. Plos one. 2020;15[9]:e0239472. pmid:32956391
- 26.
Clay-Williams R, Braithwaite J. Resilient Health Care: A Determinant Framework for Understanding Variation in Everyday Work and Designing Sustainable Digital Health Systems. Applied Interdisciplinary Theory in Health Informatics. 2019:134–45.
- 27.
Hunte G, Marsden J. Engineering resilience in an urban emergency department. Delivering Resilient Health Care: Routledge; 2018. p. 131–49.
- 28.
Lindvall J, Kecklund L, Arvidsson M. Measuring and visualizing resilience: a railway example. 11th International Symposium on Human Factors in Organisational Design and Management [ODAM]; Tietgensgade 65, 1704 København V, Denmark2014.
- 29.
Apneseth K, Wahl AM, Hollnagel E. Measuring resilience in integrated planning. Oil and Gas, TechnOlOGy and humans: CRC press; 2018. p. 129–44.
- 30. Pęciłło M. The resilience engineering concept in enterprises with and without occupational safety and health management systems. Safety Science. 2016;82:190–8.
- 31.
Hollnagel E. FRAM: the functional resonance analysis method: modelling complex socio-technical systems: CRC Press; 2017.
- 32. Raben DC, Viskum B, Mikkelsen KL, Hounsgaard J, Bogh SB, Hollnagel E. Application of a non-linear model to understand healthcare processes: using the functional resonance analysis method on a case study of the early detection of sepsis. Reliability Engineering & System Safety. 2018;177:1–11.
- 33. Damen NL, de Vos MS, Moesker MJ, Braithwaite J, van Wijngaarden RAdL, Kaplan J, et al. Preoperative anticoagulation management in everyday clinical practice: an international comparative analysis of work-as-done using the functional resonance analysis method. Journal of patient safety. 2021;17[3]:157–65. pmid:29994818
- 34. Clay-Williams R, Hounsgaard J, Hollnagel E. Where the rubber meets the road: using FRAM to align work-as-imagined with work-as-done when implementing clinical guidelines. Implementation Science. 2015;10[1]:1–8. pmid:26319404
- 35. Thude BR, Brink A, Hansen MS, Morsø L. How to Ensure Referral and Uptake for COPD Rehabilitation—Part 1: Disentangling Factors in the Cross-Sectorial Workflow of Patients with COPD to Understand why Most Patients are not Referred to Rehabilitation. International Journal of Integrated Care. 2021;21[1].
- 36. Furniss D, Back J, Blandford A, Hildebrandt M, Broberg H. A resilience markers framework for small teams. Reliab Eng Syst Saf. 2011;96:2–10.
- 37.
Furniss D, Robinson M, Cox A. Exploring resilience strategies in anaesthetists’ work: A case study using interviews and the Resilience Markers Framework [RMF]. Delivering Resilient Health Care: Routledge; 2018. p. 66–79.
- 38.
Furniss D, Back J, Blandford A, editors. Unwritten rules for safety and performance in an oncology day care unit: Testing the resilience markers framework. Proc 4th Resilience Engineering Symposium; 2011.
- 39. Iflaifel M, Lim RH, Ryan K, Crowley C. Resilient health care: a systematic review of conceptualisations, study methods and factors that develop resilience. BMC health services research. 2020;20:1–21. pmid:32303209