Advance care planning (ACP) is reported to improve the quality of outcomes of care among those with life-limiting conditions. However, uptake is low among people living with multiple sclerosis (MS) and little is known about why or how people with MS engage in this process of decision-making.
To develop and refine an initial theory on engagement in ACP for people with MS and to identify ways to improve its uptake for those who desire it.
Realist review following published protocol and reporting following Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES) guidelines. A multi-disciplinary team searched MEDLINE, PsychInfo, CINAHL, Scopus, Web of Science, Embase, Google Scholar in addition to other sources from inception to August 2019. Quantitative or qualitative studies, case reports, and opinion or discussion articles related to ACP and/or end of life discussions in the context of MS were included, as well as one article on physical disability and one on motor neuron disease, that contributed important contextual information. Researchers independently screened abstracts and extracted data from full-text articles. Using abductive and retroductive analysis, each article was examined for evidence to support or refute ‘context, mechanism, and outcome’ (CMO) hypotheses, using the Integrated Behaviour Model to guide theory development. Quality was assessed according to methodological rigour and relevance of evidence. Those studies providing rich descriptions were synthesised using a realist matrix to identify commonalities across CMO configurations.
Of the 4,034 articles identified, 33 articles were included in the synthesis that supported six CMO hypotheses that identified contexts and mechanisms underpinning engagement in ACP for people with MS and included: acceptance of their situation, prior experiences, confidence, empowerment, fear (of being a burden, of death and of dying) and the desire for autonomy. Acceptance of self as a person with a life-limiting illness was imperative as it enabled people with MS to see ACP as pertinent to them. We identified the context of MS—its long, uncertain disease trajectory with periods of stability punctuated by crisis—inhibited triggering of mechanisms. Similarly, the absence of skills and confidence in advanced communication skills among health professionals prevented possibilities for ACP discussions taking place.
Although mechanisms are inhibited by the context of MS, health professionals can facilitate greater uptake of ACP among those people with MS who want it by developing their skills in communication, building trusting relationships, sharing accurate prognostic information and sensitively discussing death and dying.
Citation: Cottrell L, Economos G, Evans C, Silber E, Burman R, Nicholas R, et al. (2020) A realist review of advance care planning for people with multiple sclerosis and their families. PLoS ONE 15(10): e0240815. https://doi.org/10.1371/journal.pone.0240815
Editor: Andrew Soundy, University of Birmingham, UNITED KINGDOM
Received: April 3, 2020; Accepted: October 2, 2020; Published: October 16, 2020
Copyright: © 2020 Cottrell et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files (Appendix A & B, S1 File).
Funding: This review is part of a larger study that was generously funded by the MS Society [Grant code 93]. CE is funded by HEE/NIHR Senior Clinical Lectureship (ICA-SCL-2015-01-001).
Competing interests: The authors have declared that no competing interests exist.
People with multiple sclerosis (MS) face an uncertain future. While some have little disability others can have profound physical and psychological limitations . Approximately 40–70% of people with MS develop cognitive impairment throughout their illness experience, some severe [1–6]. Consequently, people with MS may have reduced decision-making ability, impairing their everyday functioning  that may limit their ability to plan for future circumstances.
Advance care planning (ACP) has been defined as a “process that supports adults at any age or stage of health in understanding and sharing their personal values, life goals, and preferences regarding future care”  and evolved from the advance directive movement for individuals to maintain control and plan their future care in the event of physical or mental incapacity . Whilst advance directives consist of written directions, often within narrow and clearly marked boundaries (such as a Do Not Attempt Resuscitation [DNAR] order), ACP is broader in scope, linked to benefits for patients, health providers, and health systems . The scope of ACP is appealing as it embodies a person and family-centred, holistic approach to facilitate decision-making. Importantly, ACP is not a one-time exercise but is a heterogeneous process of multiple discussions involving different actors across many care settings.
There is a growing interest among people with MS who want to talk about their future with health professionals [1, 11]. However, despite expressed concerns about end of life discussions and choices , few people with MS engage in these important conversations [13–15]. Findings from a 2016 study indicated that amongst neurology patients, people with MS are the least likely to be referred to palliative care services and the most likely to die in hospital . Reasons for this are complex; for example, van Vliet et al. suggest that the nature of the relationship between people with MS and neurologists may be important, as well as the association between palliative care and end of life, which speaks to the uncertainty inherent in MS. This clinical uncertainty is due to the unpredictable trajectory of MS, which includes periods of stability punctuated by crisis, and makes it challenging to identify people with MS who are approaching a point where mental capacity is compromised and/or where life may be limited [16–20]. Moreover, health professionals often fail to initiate these discussions, possibly due to their reluctance to discuss disease progression and death and their difficulty managing their own emotions during difficult conversations [21, 22]. Whereas ACP is well-researched in other patient groups [23–25], it has not been widely examined in relation to people with MS.
Additionally, while Lin et al.  developed a conceptual model of ACP in cancer patients, no theoretical understanding of the contexts or circumstances in which ACP is relevant to people with MS is yet available. Two questions warrant answers when examining ACP for people with MS: (i) “in what circumstances, with whom, how, and why do people with MS (and their families) engage in ACP?” and (ii) “what works for whom, how, and why, during ACP discussions?” This realist synthesis therefore aims to: identify core mechanisms generating engagement in and completion of ACP by people with MS (and their families); identify contextual factors that trigger (or inhibit) core mechanisms, and contribute to the theoretical understanding of the process of ACP in people with MS.
The United Kingdom Medical Research Council’s (MRC) guidance on development and evaluation of complex interventions  and the Methods of Researching End-of-life Care (MORECare) statement  stress that new health care-related interventions are likely to be most effective when they are underpinned by a conceptual framework and a theoretical understanding of the key processes involved in delivering interventions and the contexts in which they are required to operate. This realist review specifically addresses the requirement for theory and conceptual framework development and was developed in April 2019 and published in PROSPERO (registration number: CRD42019142294 https://tinyurl.com/ttdt963). We used a process adapted from the RAMESES publication standards  and adhered to Pawson’s realist methodology . The review drew upon the Integrated Behaviour Model (IBM) to guide theory development. Realist reviews are a theory-driven systematic approach that are particularly suited to helping understand causation; they aim to investigate what works (or fails to work) for whom, in what circumstances, and how, by identifying processes (mechanisms) that lead to desired outcomes in particular contexts [30, 31]. Furthermore, they examine how mechanisms or ‘underlying causal forces or powers’ are triggered in particular contexts and lead to outcomes . This specifically relies on using ‘context-mechanism-output’ configurations (CMOs); these represent testable hypotheses that explain the ways in which the context is able to trigger mechanisms and lead to a variety of outcomes  (Table 1 for glossary of terms).
The phases involved in conducting this realist review involved: (i) determining the scope of the review; (ii) formulating or articulating the key theory (iii) searching for key studies; (iv) selecting the studies and appraising their relevance; (v) extracting, analysing and then synthesising the data.
Scope of the review
We clarified the scope of the review by engaging with key informants (clinical experts in neurology, palliative care and rehabilitation from different professional groups including nursing, medicine, physiotherapy and occupational therapy) and members of our study Patient and Public Involvement (PPI) group (people living with MS and their families). This process is recommended for realist syntheses as it can help to shape the direction of the review by articulating themes and ideas relevant to ACP in MS [29, 32, 33]. During informal interviews, we asked key informants to identify salient contextual factors that encourage or discourage ACP discussions that included, but were not limited to, age of onset, symptoms, and salient concepts such as uncertainty or transitions. Interviews were not recorded, however, notes were taken during these meetings which helped to form the basis of our initial search strategy. PPI members identified the desire to discuss ACP with their health care provider but felt discouraged due to lack of information about their disease trajectory as well as time pressures at medical appointments. Health care providers identified uncertainty in the disease trajectory, an unwillingness to discuss death and dying with MS patients, and a presumption that patients did not want to discuss ACP or that ACP was not important to MS patients. Both the health care providers and PPI informants expressed confusion about how to initiate ACP discussions.
Based on the information received from key informants, we then conducted a scoping exercise to identify strategies that promote successful ACP in people with MS and other illness contexts that may share similar characteristics. We discovered a small body of literature on ACP within MS and. From these sources of evidence we gained a broad understanding of ACP to focus the review and develop key theories to underpin our main literature search.
Articulation of key theories
A distinguishing feature of a realist review is the recognition of theoretical drivers for the intervention (ACP), which are scrutinized using evidence from the review. These drivers are the expectations and justifications for why an intervention might work. We developed our initial programme theory through ‘creative thinking sessions’ involving the key informant consultations, the scoping exercise results, and the use of substantive theory (the Integrated Behaviour Model (IBM), based on Azjen’s Theory of Planned Behaviour [34, 35]). This substantive theory helped us to understand how people with MS and health professionals make decisions about ACP by explaining that behaviour changes are linked to underlying beliefs about the behaviour itself, associated outcomes, and an individual’s ability to perform the behaviour. Motivation represents a key driver for behaviour change and is comprised of three elements: attitudes, perceived norms, and personal agency  which are influenced by context. We used the ‘context-mechanism-outcome’ (CMO) heuristic to explore various configurations that led to the development six CMO hypotheses that reflected our initial programme theory (refer to Table 2).
Literature searches were conducted from inception to 12 August 2019, in MEDLINE, PsychInfo, CINAHL, Scopus, Web of Science, and Embase databases, using terms presented in Table 3. In keeping with realist methodology, our literature search process was multi-pronged and iterative [29, 33].
Additional search strategies included citation tracking on MEDLINE to identify related papers and the citations within identified papers for relevance and additional reference chaining. We also searched grey literature on the following United Kingdom websites: Multiple Sclerosis (MS) Trust, Multiple Sclerosis (MS) Society, National Health Service (NHS), Google and OpenGrey.
Screening methods and inclusion criteria
Articles were screened for eligibility by two researchers (LC and GE) and selected based on predetermined eligibility criteria (Table 4) for their relevance to theory building and our research question [29, 37]. We revisited our criteria throughout the review process and amended them based on iterative literature searches and preliminary data extraction.
To promote consistency, we (LC, GE and JK) developed a bespoke data extraction form and extracted information related to the six CMO hypotheses and information about contexts, mechanisms and outputs to promote consistency in the data extracted. Two researchers (LC and GE) then independently extracted data from all selected papers.
Appraisal of included literature
We supported Pawson and Tilley’s (1997) decision to reject the ‘hierarchy of evidence’  approach to quality appraisal and adopted a system to ensure rigour according to the following principles: (i) faithfulness or correspondence with the programme theories; (ii) trustworthiness or fidelity: the evidence reported was reliable; and; (iii) relevance or application of data to the research aims. The relevance of each document was scrutinized in detail in terms of its ability to contribute to our CMO’s, as well as its credibility or the extent to which its data supported its findings and their clinical utility and applicability [29, 38]. Additionally, for the qualitative studies we used established appraisal techniques , an approach used in other realist reviews , that enabled us to examine to what extent studies presented ‘thick’ or ‘thin’ accounts of the intervention’s components and their respective contexts and mechanisms.
Data analysis and synthesis
The data analysis and synthesis processes were flexible, iterative, and creative. To maintain transparency, LC, GE and JK kept notes from a series of meetings during which they discussed each article and its contribution to the CMOs. Moving between theory and data we used retroduction to explore, compare, and explain observable patterns in the data, seeking their essential conditions, while looking for and exploring non-observable data not captured by our initial programme theories (CMOs). We used abductive reasoning for the non-observable data to create associations and to recontextualise the data, creating new plausible conclusions [29, 41]. For both processes, researchers (LC, GE, and JK) discussed potential explanations, new findings, and strategies to refine and revise our CMOs. We used flipcharts to outline each CMO and its supporting data, while evaluating its relevance to our findings. Relevant key informants were apprised of our preliminary findings and provided feedback. We then mapped our findings to a schematic of the IBM to synthesise our theoretical understanding of ACP. We retained the notes, flip charts, and schematic as an audit trail of decisions made. The final synthesis is an interpretive yet robust collation of the supporting evidence we located for each of our CMOs. To promote transparency, the data are presented in the findings section as direct quotations from the supporting literature .
Our searches yielded 4,034 articles. After removing duplicates, two researchers (LC and GE) scanned titles and/or abstracts for relevance, excluding 3,897 articles. They independently read full texts and excluded a further 64 articles resulting in 42 articles. Based on reviewer’s comments, GE and LC scrutinised the included literature and excluded a further nine articles on neurological diseases other than MS, which did not add new data to our CMOs. This resulted in 33 final articles for inclusion in the review. The data screening processes is depicted in Fig 1.
Of the 33 articles, 17 focused on MS exclusively [20, 42–57] seven on MS in combination with chronic neurologic or non-neurologic illness [17, 58–63]; one on MND  seven on neurological illness (non-specified) [18, 19, 65–69], and one on disability . Nineteen were qualitative or quantitative studies [20, 42–44, 46, 47, 49, 50, 54–56, 59, 61–64, 67, 68, 70]; one was a mixed methods study ; and 13 were case studies, literature reviews or discussion/opinion articles [17, 18, 45, 48, 51–53, 57, 58, 60, 65, 66, 69] of which seven were opinion or expert panel-based articles [17, 18, 48, 51, 53, 58, 66] three were case-studies [45, 57, 65] and three were literature reviews (narrative/systematic) [52, 60, 69] [refer to S1 File for study characteristics].
The source articles reflect an international scope; two had an international authorship [51, 60], seven from USA [43, 44, 46, 48, 66, 68, 70], seven from UK [17, 18, 52, 56–58, 65] seven from Canada [20, 49, 50, 55, 59, 62, 67], four from Germany [16, 47, 54, 63], two from the Netherlands [61, 69], one from Australia , Italy , Peru  and Turkey .
Substantially supported CMO hypotheses
CMO 1: Cumulative losses lead to acceptance of MS as a progressive condition and the creation of a new self-identity where ACP is relevant
For many people with MS this was identified as an important mechanism in both the pre-engagement and engagement phases associated with ACP. By experiencing clinically significant and person-centred losses, including physical functions, roles, paid employment and, in some cases losing touch with friends, people with MS underwent a transition. This markedly disrupted their biography and concept of self to increasingly include MS as more part of their identity. Consequently, they began to see their futures as progressively uncertain and ACP as more relevant to them. This new awareness led to an increased willingness to engage in ACP. This CMO was supported by ten studies; [20, 46–49, 53, 56, 62, 64, 67]. Lowden et al.  explored the process of “coming to a redefined self” (p.e17) in nine participants with relapsing-remitting MS. They discovered coming to accept themselves as a person with MS involved “one’s sense of self-being called into question” (p.e21) associated with multiple losses, particularly previous roles. This redefinition of self was integral to decision-making since participants were not prepared to consider making treatment decisions until this transition occurred .
Similarly, Paterson et al.  observed that decisions about the future were emotionally charged and “implied submitting to the progression of the disease rather than accommodating a temporary setback” (p. 67), a view shared by Lustig et al.  that adjustment to loss was vital in people with MS. Murray et al.  go further to suggest “ACP was easier for patients and caregivers who accepted encroaching death” (p.473) but was also present at other stages of the disease, albeit to a lesser extent.
Acceptance as the mechanism that underpins optimal timing for ACP discussions can be challenging due, in part, to MS’s uncertain trajectory and people with MS often being overwhelmed by the complexities of treatment initiation, balancing treatment side-effects and the need to learn and re-learn how to live with a progressive disease. Despite this, two articles recommended initiating discussions about ACP as early as possible, even at the point of diagnosis [60, 65], citing as justification future impaired communication or cognitive abilities. However, the most common recommendation to initiate discussions was after people with MS experienced key triggers indicating clinically significant disease progression [20, 49, 53, 57, 58]. Lowden suggests the rationale for this is that: “Participants felt ill-equipped to make treatment decisions early in the disease trajectory (p.e19). Those participants who had not yet reached the point of considering MS as part of their identity were therefore unwilling to receive information, education or decision support (p. e22)” . Similarly, Leclerc-Loiselle and Legault described timing in the context of loss and acceptance: “Professionals all considered the adaptation period of people with MS to their new functional reality following an MS crisis was a pivotal moment to introduce a palliative approach. The moment right after an MS exacerbation was a turning point because there has usually been a reflexive process as a result of grieving the loss of previous functional abilities”  (p.267). This reflexive process of acceptance represents the mechanism that underpins potential optimal timing of ACP engagement in discussions in people with MS. Though none of the included studies discussed a definitive time frame, they did suggest that acceptance is a gradual process [20, 62].
CMO 2: Trusting and empathic relationships enable empowerment
Building trust centred on active and reflective listening and validating patient’s concerns and fears are foundational to engagement in ACP. Evidence from 10 studies substantiated this CMO hypothesis [20, 42, 44, 49, 50, 52, 55, 59, 63, 70]. A relationship grounded on trust and empathy was essential when engaging in ACP discussions. This provided a safe space empowering people with MS to share fears and hopes for the future. For example, Thorne et al.  noted “although actual communication practices were important, the orientation of the health care providers to the person with MS was even more fundamental” (p.11). This orientation “was centred on respect and trust and involved validation of the patient’s experience” [p.17]. Some people with MS may approach the health professional-patient relationship with suspicion, based on the inherent complexity and length of the MS diagnostic process, during which they may have spent years feeling undermined and unheard by their health provider [50, 71].
Driedger et al. go further, emphasising the importance of trust and empathy in the relationship between people with MS and their neurologist, remarking “the crucial aspect … was not solely what was being said, but also how one was saying it” (p.10) . Additionally, “empathy, respect, and a willingness to acknowledge the perspectives of people with MS”  were critical to ACP’s success. Likewise, Col et al.  reported that “physicians can encourage or obstruct patient involvement. Patients are more active when interacting with physicians who engage in partnership building and supportive talk that legitimises the patient’s perspective and creates expectations and opportunities for the patient to discuss their needs and concerns” (p.266). Buecken et al.  noted that “physician empathy plays a key role” (p.323) in meeting end of life care-related discussions and is paramount in delivering high-quality care. Furthermore, the study observed trust in a patients’ health professional was imperative in initiating shared decision when considering current and future treatment decisions. Mitchell  remarked: “a trusting relationship with healthcare providers was a prerequisite to ACP discussions” (p.130).
CMO 3: Fear of being a burden to family motivates engagement in ACP
Six studies supported this CMO [20, 43, 65, 66, 69, 70]. The presence of family was important to both the people with MS and the health professional(s). People with MS relied on and welcomed the support and caregiving provided by family, but occasionally their need for caregiving only increased negative thoughts of being a burden [43, 64, 70]. Some people with MS considered ACP a way to alleviate feeling like a burden. Murray et al. observed, “many participants indicated that documented wishes would reduce the decisional burden and help caregivers avoid regret” (p. 475) . Similarly, participants in Mitchell’s study reported that future planning “constituted a personal responsibility to their family members” (p.128) and “expressed concerns about financial and emotional burdens their families may experience” (p.131) . Caregivers in Murray’s (2016) (p.475)  study also described ACP as a “catalyst for communication” to realise open family discussions and ease tension in family negotiations.
For health professionals, family were viewed as invaluable resources whose intimate knowledge of dependants had potential to help initiate timely ACP discussions. Similarly, Seeber stated “involving families in the discussion appears to improve acceptance of decisions for both patient and caregiver” (p.596) .
CMO 4: ACP as a tool for enabling control and autonomy in decision-making
Ten studies supported this CMO hypothesis [17, 18, 48, 56, 57, 61, 65, 67, 70, 72], which is the basis underpinning the original development of concept of ACP. Kalb explained: “people living with MS struggle to maintain a feeling of control in the face of an unpredictable disease course and uncertain future. Effective planning and problem-solving […] help people feel prepared—more in control—regardless of what the future holds” (p.532) . Clarke et al. also acknowledged “some participants used advance planning as a way of extending the zone of personal autonomy and involvement in decision-making beyond the stage when their ability to make decisions or communicate their wishes would be lost” (p.6) . Mitchell reported participants felt engaging in ACP “constituted a responsibility to themselves […] and represented asserting agency over [their] care” (p.129) . Family members in Murray et al.’s study similarly reported that ACP “gave (them) control of life” (p.474) 
CMO 5: Skilled communication inspires confidence and facilitates ACP discussion
Confidence and communication skills were important mechanisms to facilitate ACP completion, reported in 16 source articles [18, 19, 42, 44, 45, 50, 51, 55, 58, 59, 60, 64, 67–70]. Strong evidence supported an inverse relationship between poor communication skills and the absence of ACP discussions. A participant with MS in Thorne et al.’s study stated: “communication with health care providers can shape the entire course of an illness” (p.10) . Similarly, Driedger and colleagues reported, “how things are said are as important as what is being said” (p.10) . Health professional distance and paternalism were perceived as barriers to engaging in clinical decision-making processes . These attitudes, enacted through behaviours such as using simplistic explanations or deliberately obfuscating with highly technical scientific jargon, were perceived as being counterproductive. As Driedger et al. discovered, “a core problem was that neurologists just generally lack a capacity for good communication and compassionate care” (p.10)  possibly a result of health professionals lacking clarity on the situation themselves. Thorne et al. identified withholding information was highly frustrating for people with MS: “I just get the feeling that it’s a very paternalistic sort of attitude […] ‘We don’t need to tell her that” (person with MS) (p.12) .
Beneficial communication included strategies that included legitimising and confirming the person with MS’s experience, considering MS in the context of that person’s life, and assisting them to find the language to describe their illness situation . Col et al. observed that “physicians can encourage or obstruct patient involvement [in future care planning]” (p.266) . Additionally, Oliver et al. observed: “families do appreciate honesty and awareness of deterioration” (p.36) .
However, we identified only limited agreement on which health professional was best placed to have ACP discussions with people with MS. Seeber  stated that the neurologist is “best-poised’ due to their detailed knowledge of neurologic diseases and possible complications” (p.595). Surprisingly, our findings indicate that knowledge was less valued than the ability to establish and maintain what people with MS considered to be a trusting relationship. Our recommendation is therefore that the person best positioned to have discussions with people with MS would be the one with whom they believe to have the strongest and most trusting relationship, a situation that is open to change.
CMO 6: Previous experiences of witnessing death facilitates or hinders engagement in ACP
Seven studies corroborated this CMO [20, 43, 47, 49, 56, 65, 71]. ACP was reported to mitigate the fear of experiencing a distressing or ‘bad’ death [43, 65] and may be a motivating factor for some people with MS. For example, in Chen and Habermann’s study, participants shared views that having witnessed a ‘bad death’ or having experienced a life-threatening illness in a relative acted as a “close call” (p.4) that triggered their own fear of dying and the need to address it through engaging in ACP. Former health professionals were more likely to have witnessed bad deaths, resulting in the acknowledgement that ACP may represent a way to address their concerns [20, 43]. Lowden et al. stated, “previous experiences with illness and treatment influenced participants’ decision making” (p.19) .
In addition to these core mechanisms, we identified two related components: the fear of discussing death and dying and the concept of hope in MS. The fear of discussing death and dying was present in four articles [20, 47, 65, 70] and may be considered to be wide-spread in post-modern Western culture [65, 73]. Related to this, the reviewed literature identified that health professionals were reluctant to initiate ACP discussions because they feared discussions would lead to the loss of hope in the person with MS [19, 20, 49, 64]. Correspondingly, some people with MS were reluctant to engage in ACP because they held out expectations of a cure for their MS . As Clarke et al. observed: “Decisions about future care were deferred to a later date, sometimes in the apparent hope that eventualities that they could plan for may never arise” (p.7) .
We present the first realist review to develop and refine an initial theory to explain engagement in and completion of ACP in people with MS and to guide clinical practice by uncovering in what ways ACP works. Specifically, by testing our six CMOs, we identified mechanisms underpinning the behavioural changes that drive engagement in, and completion of, ACP. They included: acceptance, experience, confidence, empowerment, fear (of being a burden; of death and dying) and the desire for autonomy. In the USA, Levi et al.  explored motivations that led people to engage in ACP. The authors explained that motivation to engage in ACP was contingent on four distinct domains: concern for self; concern for others; expectations about the impact of ACP and anecdotes, stories, and experiences. Their findings share similarities with this review; a number of the CMOs we developed and tested could be similarly categorised. For example, ‘concern for others’ aligns closely with our CMO about burden on family; ‘concern for self’ parallels the desire for autonomy and the fear of suffering a ‘bad death’. The influence of prior experiences on ACP motivation echoes Levi et al.’s category of ‘anecdotes, stories, and experiences’, a finding similarly identified by Shaw et al.  in a study of ACP in long-term care facilities. They observed “both residents and family members indicated that prior experience with […] the advance care planning process had helped in influencing current perspectives and readiness. As a result, both groups described having engaged and feeling ready to engage in the process” (p.745). However, neither Levi et al.  nor Shaw et al.  addressed the mechanisms underlying the motivations for engagement in ACP: they identified what the motivations were but did not identify why these mechanisms work or how to trigger them.
We discovered acceptance of self as a person with a life-limiting illness is the core mechanism underpinning engagement in ACP for people with MS. Acceptance facilitates people with MS to see the relevance of ACP and is a driver for subsequent behaviour change, which is the willingness to engage in ACP discussions. According to the IBM, relevance aligns with attitudes, which are a key facet of behaviour change . Attitudes refer to the individual’s perception of the behaviour; in this case, acceptance of self as a person with progressive illness changes their perception of ACP from irrelevant to relevant. Relevance is critical for ACP. Although Solari et al.  recently reported that as many as 89% of people with severe MS want to discuss ACP, preliminary findings from a study on ACP in MS indicate that while participants were generally positive about ACP, it was no more relevant to them than to anybody, regardless of their MS diagnosis. Concerning their MS specifically, participants were reluctant to engage in ACP because of the uncertainty of their disease trajectory . The differences in interest in ACP may be due to levels of disease severity, as evidence suggests the notion of ACP often becomes relevant to people with MS when disease severity increases [43, 56].
Questions still remain about how and when to make ACP relevant to those living with, and in some instances dying from the complications associated with MS. Careful and sensitive encouragement of people with MS to accept their situation will help, but we as identified, an important challenge encountered with the uptake of ACP is the actual context in which it is enacted. We initially presumed that two related contextual elements—the fear of discussing death and dying and hoping for cure—would influence consideration of ACP. However, we underestimated their importance. Indeed, both elements created contexts that were largely inhospitable to ACP and represent fundamental issues with its uptake. The mechanisms we uncovered, while important, may not sufficiently overcome these issues, but may function instead as ‘trust mechanisms’ . Trust mechanisms work to influence contexts rather than influencing outcomes directly; in this case, mechanisms enable contexts to be more hospitable to notion of ACP. Trust mechanisms alter the shape of traditional CMO configurations, changing from C + M = O to C + M → C → O.
For clinical practice, work must be aimed at reshaping the context, rather than focusing strictly on outcomes associated with behaviour change or ‘completed’ ACPs. Each mechanism we identified can be leveraged to change the landscape for people with MS, to facilitate acceptance of ACP if that is their preference. For example, relationship building may open a trusting platform on which more accurate and realistic prognostic information can be shared with people with MS; this is essential to help them make sense of their illness and reframe their self-image. Recent European Academy of Neurology (EAN) guidelines for palliative care in MS include early discussion of disease progression and future planning with people with severe MS . Realistic hope is also important; our findings indicate health professionals do not discuss ACP for fear of distressing people with MS. However, false hope may undermine trust, whereas honesty, empathy, and truthfulness are valued by people with MS .
Additional to building trusting relationships, advanced communication skills are important for health professionals, many of whom are notably uncomfortable discussing death and dying [19, 67, 68], exacerbating an already difficult conversation. Communication skills training linked to patient-centred outcomes are recommended . This must be accompanied by active listening [44, 55] and the use of accessible language . Andreassen et al.  utilised discourse analysis to explore ACP discussions, using the concept of doctor and patient voices. They discovered that health professionals used a variety of ‘voices’: the ‘doctor voice’ was used to ask specific questions; the ‘educator voice’ to share information and help patients understand their illness and treatment, and the ‘fellow human voice’ to convey empathy. Simply by showing empathy through comments for example “I understand” or “that must be really tough”, health professionals shared a ‘fellow human voice’ encouraging patients to discuss ACP.
Additionally, in an attempt to develop a measurement tool for ACP, several author groups mentioned “prerequisites” [79, 80]. These prerequisites, such as personality factors, cognitive style, coping style, role preferences, risk knowledge, numeracy, risk attribution and tolerance, should also be recognized by healthcare professionals as they are part of the patient’s personal context and might influence the communication pathways to be engaged in this communication relationship.
Overall, this realist synthesis uses a theory-testing method to explore, below the surface, what helps or hinders ACP from taking place and to understand how contextual barriers and facilitators operate. This knowledge is critical to design evidence-informed interventions that have the greatest potential to influence underlying mechanisms that drive behaviour change in different contexts and lead to positive person and family-centred outcomes and experiences.
Strengths and limitations
A strength of this approach is both its explanatory and theoretical nature through which to understand the complex mechanisms underlying pre-engagement and engagement of ACP in people with MS. We also actively incorporated key informants’ and PPI views alongside published literature to refine the review focus to areas considered most pertinent to clinical practice and lived experiences of MS. Moreover, this style of synthesis shifts the focus from specific interventions and services to broader underlying mechanisms or principles. Additionally, our evidence included a diverse range of clinical conditions relevant to people living with MS. Thirteen of the studies were qualitative, a strength being that this permitted salient contexts, mechanisms and outcomes to be understood in detail, particularly where ‘thick’ accounts were evident. Whilst we were able to draw on a relatively wide international literature, we are aware that ACP is still a predominantly Western construct [81, 82]. Cultural homogeneity is therefore a limitation of this review. Caution must therefore be exercised when considering the findings of this review for countries where ACP has yet to be implemented.
Institutional factors are undeniably part of the context of ACP, as health care is embedded in discourses of efficiency , and the overriding dominance of ‘cure-over-care’  in which palliative care, including ACP, can often be disregarded. Our findings do not extend across multiple levels (micro, meso, macro), but are focused principally on the individual. Since we set out to study pre-engagement and engagement phases of ACP, rather than its implementation, we believe focus on the individual is justifiable, although we recognise a multi-level exploration and analysis could provide complementary evidence .
Conclusions and recommendations
We have identified important mechanisms that help to explain how and why ACP works in people with MS. Additionally, based on our findings we suggest strategies to assist health professionals help people with MS to engage in a process where they are able to reframe, perhaps multiple times, their self-image, so that ACP becomes more relevant to them. This research is important, as it is evident that ACP is more than merely an administrative or intellectual task for health professionals to engage in with their patients. Instead, it is a deeply human encounter in which a person reflects on their illness, situation, on-going rehabilitation and management, and impending death and must negotiate a route through at times threatening possibilities.
Consequently, health professionals have a responsibility to train in communications skills  and specifically in engaging in difficult conversations . Evidence suggests that people with MS prefer an active role in decision-making [86, 87], therefore health professionals must work towards empowerment through partnerships. Moreover, they must appreciate that offering patients accurate and honest information about the patient’s prognosis is unlikely to destroy their hope and may paradoxically encourage realistic future planning. Importantly, this can only be fostered if authentic relationships are built with patients and their families; creating environments in which respect, trust, and empathy thrive is fundamental to shape a context that supports engagement in ACP discussions.
Future research must now take place that that informs not only the evaluation of effectiveness of ACP among people living with MS and their families, but also actively engages in the CMOs examined in this review to better understand how this complex intervention can successfully and acceptably be enacted. The ‘gold-standard’ randomised controlled trial design alone does not adequately address the ‘context-specific drivers’ behind implementation outcomes and their relationship to the underlying theory. We therefore suggest researchers consider using a realist approach alongside more conventional designs akin to a ‘hybrid trial’ design that includes multiple methods. Realist evaluation is increasingly applied in the examination of complex healthcare interventions, for example ACP, since it seeks to provide a more in-depth understanding of what works, for whom and in what circumstances [38, 88]. Hybrid trials not only focus on assessing clinical effectiveness, but explore the manner in which an intervention is implemented [89, 90], and it has been argued should also consider context .
Disclaimer: The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, NIHR, the Department of Health or the MS Society.
- 1. Strupp J., Hartwig A.J., and Golla H., Feeling severely affected by multiple sclerosis: What does this mean?. Palliative Medicine, 2012. 26(8): p. 1001–1010. pmid:22042226
- 2. Higginson I.J., Hart S., and Silber E., Symptom prevalence and severity in people severely affected by multiple sclerosis 158–165. Journal of Palliative Care, 2006. 22(3): p. 158–165. pmid:17058754
- 3. Edmonds P., et al., 'Fighting for everything': service experiences of people severely affected by multiple sclerosis. Multiple Sclerosis, 2007. 13(5): p. 660–667. pmid:17548447
- 4. Forbes A., et al., Health problems and health-related quality of life in people with multiple sclerosis 67–78. Clinical Rehabilitation, 2006. 20(1): p. 67–78. pmid:16502752
- 5. Koffman J., et al., Progression, symptoms and psychosocial concerns among those severely affected by multiple sclerosis: a mixed-methods cross-sectional study of Black Caribbean and White British people. PLOS ONE, 2013. 8(10).
- 6. Chiaravalloti N.D. and DeLuca J., Cognitive impairment in multiple sclerosis. Lancet Neurology, 2008. 7: p. 1139–51. pmid:19007738
- 7. Nagy H., et al., The effects of reward and punishment contingencies on decision-making in multiple sclerosis. J Int Neuropsychol Soc, 2006. 12: p. 559–565. pmid:16981608
- 8. Sudore R.L., et al., Defining advance care planning for adults: a consensus definition from a multidisciplinary delphi panel. Journal of Pain and Sypmton Control 2017. 53: p. 821–832.
- 9. Collins L.G., Parks S.M., and Winter L., The state of advance care planning: one decade after SUPPORT. Am J Hosp Palliat Care, 2006. 23(5): p. 378–84. pmid:17060305
- 10. Schickedanz A.D., et al., A clinical framework for improving the advance care planning process: start with patients' self-identified barriers. J Am Geriatr Soc, 2009. 57(1): p. 31–9. pmid:19170789
- 11. Edmonds. P., et al., Loss and change: experiences of people severely affected by multiple sclerosis. Palliative Medicine, 2007. 21: p. 101–110. pmid:17344258
- 12. Köpke S., et al., Patient and caregiver involvement in the formulation of guideline questions: findings from the European Academy of Neurology guideline on palliative care of people with severe multiple sclerosis. European Journal of Neurology, 2019. 26(1): p. 41–50. pmid:30035845
- 13. van Vliet L.M., et al., How integrated are neurology and palliative care services? Results of a multicentre mapping exercise. BMC Neurol, 2016. 16: p. 63. pmid:27165157
- 14. Solari A., et al., EAN guideline on palliative care of people with severe, progressive multiple sclerosis. Eur J Neurol, 2020. 27(8): p. 1510–1529. pmid:32469447
- 15. Koffman J., et al., The Empower Study: Developing Advance Care Planning Resources for People Severely Affected by Multiple Sclerosis, and their Families: Annual Report to MS Society. 2020, Cicely Saunders Institute: King's College London. p. 50.
- 16. Buecken R., et al., Patients feeling severely affected by multiple sclerosis: How do patients want to communicate about end-of-life issues? Patient Education and Counseling, 2012. 88(2): p. 318–324. pmid:22480629
- 17. Campbell C.W., Jones E.J., and Merrills J., Palliative and end-of-life care in advanced Parkinson's disease and multiple sclerosis. Clin Med (Lond), 2010. 10(3): p. 290–2.
- 18. Kent A., Advance care planning in progressive neurological conditions. Nurs Stand, 2015. 29(21): p. 51–9. pmid:25605116
- 19. Vishnevetsky A., et al., Palliative Care: Perceptions, Experiences, and Attitudesin a Peruvian Neurologic Hospital. Journal of Palliative Medicine, 2019. 22(3): p. 250–257. pmid:30526275
- 20. Lowden D., Lee V., and Ritchie J.A., Redefining self: patients' decision making about treatment for multiple sclerosis. J Neurosci Nurs, 2014. 46(4): p. E14–24. pmid:24875289
- 21. Brighton L.J., et al., 'Difficult Conversations': evaluation of multiprofessional training. BMJ Support Palliat Care, 2018. 8(1): p. 45–48. pmid:29118100
- 22. Brighton L.J., et al., Emotional labour in palliative and end-of-life care communication: A qualitative study with generalist palliative care providers. Patient Educ Couns, 2019. 102(3): p. 494–502. pmid:30879492
- 23. Bristowe K., et al., Patient and carer experiences of clinical uncertainty and deterioration, in the face of limited reversibility: a comparative observational study of the AMBER care bundle Palliative Medicine, 2015. pmid:25829443
- 24. Robinson L., et al., A qualitative study: professionals’experiences of advance care planning in dementia and palliative care, a good idea in theory but …’. Palliative Medicine, 2012. 27(5): p. 401–408. pmid:23175508
- 25. Barnes K.A., et al., Advance care planning discussions in advanced cancer: Analysis of dialogues between patients and care planning mediators. Palliative and Supportive Care, 2011. 9: p. 73–79. pmid:21352620
- 26. Lin C.-P., et al., The conceptual models and mechanisms of action that underpin advance care planning for cancer patients: A systematic review of randomised controlled trials. Palliative Medicine, 2018. 33(1): p. 5–23. pmid:30362897
- 27. Craig P., et al., Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ, 2008. 337: p. a1655. pmid:18824488
- 28. Higginson I.J., et al., Evaluating complex interventions in End of Life Care: the MORECare Statement on good practice generated by a synthesis of transparent expert consultations and systematic reviews. BMC Medicine, 2013. 11(1).
- 29. Wong G., et al. Realist Synthesis. RAMESES Training Materials. 2013 [cited 2019 5 December]:[Available from: http://www.ramesesproject.org/media/Reaslist_reviews_training)materials.pdf.
- 30. Pawson R., et al., Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy, 2005. 10 Suppl 1: p. 21–34.
- 31. Wong G., et al., RAMESES II reporting standards for realist evaluations. BMC Medicine, 2016. 14(1): p. 96. pmid:27342217
- 32. Jagosh J., The importance of understanding context: realist methodology for public health and primary care. 2018.
- 33. Brighton L.J., et al., Improving Exercise-Based Interventions for People Living with Both COPD and Frailty: A Realist Review. International journal of chronic obstructive pulmonary disease, 2020. 15: p. 841–855. pmid:32368030
- 34. Ajzen I., From intentions to actions: A theory of planned behavior, in In Action-Control: From Cognition to Behavior, Kuhl J. and Beckman J., Editors. 1985, Springer: Heidelberg. p. 11–39.
- 35. Ajzen I., The theory of planned behavior. Organizational Behavior and Human Decision Processes, 1991. 50(2): p. 179–211.
- 36. Montaño D.E. and Kasprzyk D., Theory of reasoned action, theory of planned behavior, and the integrated behavioral model, in Health behavior and health education: Theory, research, and practice, 4th ed. 2008, Jossey-Bass: San Francisco, CA, US. p. 67–96.
- 37. Schick-Makaroff K., et al., Strategies for incorporating patient-reported outcomes in the care of people with chronic kidney disease (PRO kidney): a protocol for a realist synthesis. Syst Rev, 2019. 8(1): p. 20. pmid:30636637
- 38. Pawson R. and Tilley N., Realistic Evaluation. 1997, London: Sage.
- 39. Dixon-Woods M., et al., Synthesising qualitative and quantitative evidence: A review of possible methods. Journal of Health Services Research and Policy, 2005. 10(1): p. 45–53. pmid:15667704
- 40. O’Campo P., et al., Implementing successful intimate partner violence screening programs in health care settings: Evidence generated from a realist-informed systematic review. Social Science & Medicine, 2011. 72(6): p. 855–866.
- 41. Danermark B., et al., Explaining Society: Critical Realism in the Social Sciences. 2002, London: Routledge.
- 42. Buecken R., et al., Patients feeling severely affected by multiple sclerosis: How do patients want to communicate about end-of-life issues? Patient Education and Counseling, 2012. 88: p. 318–324. pmid:22480629
- 43. Chen H. and Habermann B., Ready or not: planning for health declines in couples with advanced multiple sclerosis. The Journal of neuroscience nursing: journal of the American Association of Neuroscience Nurses, 2013. 45(1): p. 38–43.
- 44. Col N.F., et al., Evaluation of a Novel Preference Assessment Tool for Patients with Multiple Sclerosis. Int J MS Care, 2018. 20(6): p. 260–267. pmid:30568563
- 45. Efendi H., Boz C., and Karabudak R., Evaluating Treatment Decision for Multiple Sclerosis: Real Life and Patient Experiences. Noro psikiyatri arsivi, 2018. 55(Suppl 1): p. S10–S14. pmid:30692848
- 46. Lustig D.C., et al., The Relationship Between Career Thoughts and Adjustment for Individuals With Multiple Sclerosis. Rehabilitation Counseling Bulletin, 2017. 61(2): p. 112–120.
- 47. Golla H., et al., Multiple sclerosis and palliative care—perceptions of severely affected multiple sclerosis patients and their health professionals: a qualitative study. BMC Palliat Care, 2014. 13(1): p. 11. pmid:24641905
- 48. Kalb R., The emotional and psychological impact of multiple sclerosis relapses. J Neurol Sci, 2007. 256 Suppl 1: p. S29–33.
- 49. Leclerc-Loiselle J. and Legault A., Introduction of a palliative approach in the care trajectory among people living with advanced MS: perceptions of home-based health professionals. Int J Palliat Nurs, 2018. 24(6): p. 264–270. pmid:29932829
- 50. Driedger S.M., et al., Caught in a no-win situation: discussions about CCSVI between persons with multiple sclerosis and their neurologists—a qualitative study. BMC Neurol, 2017. 17(1): p. 176. pmid:28882115
- 51. Rieckmann P., et al., Achieving patient engagement in multiple sclerosis: A perspective from the multiple sclerosis in the 21st Century Steering Group. Mult Scler Relat Disord, 2015. 4(3): p. 202–18. pmid:26008937
- 52. Eskyte I., et al., Understanding treatment decisions from the perspective of people with relapsing remitting multiple Sclerosis: A critical interpretive synthesis. Mult Scler Relat Disord, 2019. 27: p. 370–377. pmid:30476873
- 53. Solari A. and Pucci E., There is an urgent need for palliative care specialists in MS—Yes. Mult Scler, 2019. 25(13): p. 1710–1711. pmid:31343953
- 54. Golla H., et al., Patients Feeling Severely Affected by Multiple Sclerosis: Addressing Death and Dying. OMEGA—Journal of Death and Dying, 2015. 74(2): p. 275–291.
- 55. Thorne S., et al., Health care communication issues in multiple sclerosis: an interpretive description. Qual Health Res, 2004. 14(1): p. 5–22. pmid:14725173
- 56. Clarke G., et al., Planning for an uncertain future in progressive neurological disease: a qualitative study of patient and family decision-making with a focus on eating and drinking. BMC Neurol, 2018. 18(1): p. 115. pmid:30115018
- 57. Foley P., et al., Lesley's story: a case report, and discussion of challenges faced in end-of-life care for progressive neurological disease. Pract Neurol, 2012. 12(4): p. 244–8. pmid:22869768
- 58. Barnes J. and Campbell C., Palliative care in multiple sclerosis and motor neurone disease. Br J Hosp Med (Lond), 2010. 71(1): p. 21–5.
- 59. Thorne S.E., et al., The context of health care communication in chronic illness. Patient Educ Couns, 2004. 54(3): p. 299–306. pmid:15324981
- 60. Oliver D.J., et al., A consensus review on the development of palliative care for patients with chronic and progressive neurological disease. Eur J Neurol, 2016. 23(1): p. 30–8. pmid:26423203
- 61. Walter H.A.W., et al., The Role of Palliative Care in Chronic Progressive Neurological Diseases—A Survey Amongst Neurologists in the Netherlands. Frontiers in Neurology, 2019. 9(1157).
- 62. Paterson B., Thorne S., and Russell C., Disease-specific influences on meaning and significance in self-care decision-making in chronic illness. Can J Nurs Res, 2002. 34(3): p. 61–74. pmid:12425011
- 63. Hamann J., et al., Can Psychiatrists and Neurologists Predict Their Patients' Participation Preferences? The Journal of Nervous and Mental Disease, 2010. 198(4): p. 309–311. pmid:20386262
- 64. Murray L., et al., Advance care planning in motor neuron disease: A qualitative study of caregiver perspectives. Palliat Med, 2016. 30(5): p. 471–8. pmid:26847526
- 65. Lennard C., Best interest versus advance decisions to refuse treatment in advance care planning for neurodegenerative illness. Br J Nurs, 2018. 27(21): p. 1261–1267. pmid:30457382
- 66. Boersma I., et al., Palliative care and neurology: time for a paradigm shift. Neurology, 2014. 83(6): p. 561–7. pmid:24991027
- 67. Gofton T.E., et al., Challenges facing palliative neurology practice: A qualitative analysis. J Neurol Sci, 2018. 385: p. 225–231. pmid:29277430
- 68. Taylor B.L., et al., Inpatients with neurologic disease referred for palliative care consultation. Neurology, 2019. 92(17): p. e1975–e1981. pmid:30918095
- 69. Seeber A.A., et al., Discussions about treatment restrictions in chronic neurologic diseases: a structured review. Neurology, 2012. 78(8): p. 590–7. pmid:22351797
- 70. Mitchell S.E., et al., Experiences and Perspectives on Advance Care Planning among Individuals Living with Serious Physical Disabilities. J Palliat Med, 2017. 20(2): p. 127–133. pmid:27809645
- 71. Mitchell S., et al., Achieving beneficial outcomes for children with life-limiting and life-threatening conditions receiving palliative care and their families: A realist review. Palliat Med, 2019: p. 269216319870647.
- 72. Nanton V., et al., The threatened self: Considerations of time, place, and uncertainty in advanced illness. Br J Health Psychol, 2016. 21(2): p. 351–73. pmid:26689299
- 73. Zimmermann C., Acceptance of dying: a discourse analysis of palliative care literature. Soc Sci Med, 2012. 75(1): p. 217–24. pmid:22513246
- 74. Levi B.H., et al., What influences individuals to engage in advance care planning? Am J Hosp Palliat Care, 2010. 27(5): p. 306–12. pmid:20103783
- 75. Shaw M., et al., Characterizing Readiness for Advance Care Planning From the Perspective of Residents, Families, and Clinicians: An Interpretive Descriptive Study in Supportive Living. Gerontologist, 2018. 58(4): p. 739–748. pmid:28329800
- 76. Ajzen I., Timko C., and White J.B., Self-monitoring and the attitude–behavior relation. Journal of Personality and Social Psychology, 1982. 42(3): p. 426–435.
- 77. Selman L.E., et al., The Effect of Communication Skills Training for Generalist Palliative Care Providers on Patient-Reported Outcomes and Clinician Behaviors: A Systematic Review and Meta-analysis. J Pain Symptom Manage, 2017. 54(3): p. 404–416.e5. pmid:28778560
- 78. Andreassen P., et al., Talking about sensitive topics during the advance care planning discussion: A peek into the black box. Palliat Support Care, 2015. 13(6): p. 1669–76. pmid:26033300
- 79. Heesen C., et al., Patient autonomy in multiple sclerosis—possible goals and assessment strategies. J Neurol Sci, 2013. 331(1–2): p. 2–9. pmid:23711752
- 80. Sudore R.L., et al., Measuring Advance Care Planning: Optimizing the Advance Care Planning Engagement Survey. J Pain Symptom Manage, 2017. 53(4): p. 669–681.e8. pmid:28042072
- 81. Lin C.P., et al., What influences patients' decisions regarding palliative care in advance care planning discussions? Perspectives from a qualitative study conducted with advanced cancer patients, families and healthcare professionals. Palliat Med, 2019: p. 269216319866641.
- 82. Lin C.-P., et al., Feasibility and acceptability of a culturally adapted advance care planning intervention for people living with advanced cancer and their families: A mixed methods study. Palliative Medicine, 2020. Palliative Medicine.
- 83. Anderson J.M., Tang S., and Blue C., Health care reform and the paradox of efficiency: “writing in” culture. International Journal of Health Services, 2007. 37 (2): p. 291–320. pmid:17665725
- 84. Glouberman S. and Mintzberg H., Managing the care of health and the cure of disease—Part I: Differentiation. Health Care Manage Rev, 2001. 26(1): p. 56–69; discussion 87–9. pmid:11233354
- 85. Lemmon M.E., et al., Education Research: Difficult conversations in neurology. Neurology, 2018. 90(2): p. 93. pmid:29311368
- 86. Solari A., et al., Conversion to Secondary Progressive Multiple Sclerosis: Patient Awareness and Needs. Results From an Online Survey in Italy and Germany. Front Neurol, 2019. 10: p. 916. pmid:31507519
- 87. Heesen C., et al., Decisional role preferences, risk knowledge and information interests in patients with multiple sclerosis. Mult Scler, 2004. 10(6): p. 643–50. pmid:15584489
- 88. Salter K.L. and Kothari A., Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implementation Science, 2014. 9(1): p. 115.
- 89. Zhu M., et al., Hybrid clinical trials to generate real-world evidence: design considerations from a sponsor's perspective. Contemporary Clinical Trials, 2019: p. 105856. pmid:31669449
- 90. Landes S.J., McBain S.A., and Curran G.M., An introduction to effectiveness-implementation hybrid designs. Psychiatry Research, 2019. 280: p. 112513. pmid:31434011
- 91. Kemp C.G., Wagenaar B.H., and Haroz E.E., Expanding hybrid studies for implementation research: intervention, implementation strategy, and context. Frontiers in Public Health, 2019. 7.
- 92. Papoutsi C., et al., Interventions to improve antimicrobial prescribing of doctors in training (IMPACT): a realist review. Health Serv Deliv Res 2018. 6 (10).