Study design

This process evaluation of an effectiveness-implementation cRCT of DIALOG+ in outpatients with PSD in five Southeast European countries (Bosnia and Herzegovina, Kosovo UN Resolution, North Macedonia, Montenegro and Serbia) adopted a mixed-methods study design and involved data collection and analysis using qualitative and quantitative methods. It was developed according to the UK Medical Research Council’s guidance for process evaluations of complex interventions [8]. The CONSORT diagram of the cRCT is reported elsewhere [6].

Setting

Eleven mental health services (including hospital-based, outpatient and community-based services) participated in the trial across the five countries. The recruitment and geographical location of the participating services, and their standard outpatient mental health care, are described in the IMPULSE cRCT protocol [3].

Intervention and IMPULSE cRCT

DIALOG+ is a psychosocial intervention delivered through an app on a tablet computer. It begins by asking patients to self-assess their satisfaction with eight life areas and three treatment areas, on a 7-point scale, called DIALOG scale, with higher scores indicating higher satisfaction. Next, the mental health professional and patient together review ratings across all areas and patients are encouraged to choose an area to discuss in more detail during which the mental health professional uses a four-step, solution-focused approach (1-Understanding, 2-Looking forward, 3-Exploring options, 4-Agreeing on actions), focused on strengthening and utilising the resources of individuals, rather than on deficits. The DIALOG+ session finishes by agreeing clear actions that the patient, clinicians and/or another person from the patient’s circle should aim to achieve before the next session. The interaction between the clinician and the patient during the DIALOG+ session is characterized by encouragement and social reinforcement, patient-centred communication, and high patient involvement [10].

The IMPULSE cRCT was designed as a two-arm study – DIALOG+ intervention and treatment as usual as a control arm. Recruitment of participants took place between 1^st of February and 1^st of September 2019. Randomisation was conducted at the clinician-level to prevent clinicians having to treat patients in both arms, thereby minimizing contamination between trial arms. Thus, clinicians were randomised 1:1 to deliver either the DIALOG+ intervention or treatment as usual (control). The delivery of the intervention was guided by an implementation strategy that was primarily informed by focus group data from the DIALOG+ pilot study, mapped onto the Theoretical Domains Framework of behaviour change to identify facilitators and barriers to delivering and engaging with the intervention, with suitable behaviour change techniques incorporated to address the identified potential barriers to implementation [6,11,12]. DIALOG+ was implemented by members of the local research teams providing a training programme to clinicians in the intervention arm, which included face-to-face core training before starting to deliver the intervention, top-up training after delivery of the first and fourth DIALOG+ session, with additional supervision available as needed. Psychiatrists, psychologists and mental health nurses were trained to deliver all components of the intervention to patients with PSD. The clinicians were encouraged to deliver the intervention during face-to-face routine clinical meetings, six times during the twelve-month study period. Patients were provided with booklets where they could note the agreed actions. Information about recruitment, randomisation, trial design and outcome measures is reported in the IMPULSE cRCT protocol [3].

Role of context

Intervention acceptability

Intervention fidelity

Intervention fidelity was assessed according to the constructs proposed by Bellg et al. [18] and Carroll et al. [19]: training of clinicians, delivery of intervention, receipt of intervention and enactment of plans agreed during the intervention sessions, and intervention differentiation (extent to which the active intervention differed from the control condition).

Deviations from published protocol

Originally the research team planned to conduct individual face-to-face semi-structured interviews with patients and face-to-face focus groups with clinicians. However, in the context of the COVID-19 pandemic, it was no longer possible to conduct qualitative data collection face-to-face, thus the procedure was updated to remote qualitative data collection. Only semi-structured individual interviews were conducted due to the complexity of conducting focus groups remotely.

Role of context

The analysis followed the procedure described in the process evaluation protocol [9]. Findings from all three sources of data (site visits reports, policy analysis and pre-trial focus group analysis) were coded onto a framework of the attributes and features of context developed by Squires and colleagues [21]. The attributes with their definitions are included in Table 1. After familiarizing themselves with the contextual attributes by Squires et al. [21], two researchers (TP and SM) conducted the coding by categorizing the results from the three data sources and summarizing them into the framework of contextual attributes using Microsoft Excel software. For each contextual attribute the data was classified based on interpretation as a barrier or a facilitator. Triangulation of the findings from the three different data sources was conducted by two researchers (TP and SM) following the triangulation protocol described by Farmer et al. [22]. Frequent team meetings were held during the analysis process to discuss any concerns and coding inconsistencies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Contextual attributes and their definitions (Adapted from Squires et al. [21] to relate to the study’s setting of interest).

https://doi.org/10.1371/journal.pone.0338408.t001

Intervention acceptability

We utilized the Theoretical Framework of Acceptability (TFA) [23] to investigate intervention acceptability. We defined experienced acceptability based on Sekhon et al. [23]: “a multi-faceted construct that reflects the extent to which people delivering or receiving the healthcare intervention consider it to be appropriate, based on experiential cognitive and emotional responses to the intervention”. The seven component constructs of acceptability with their definitions are presented in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. The component constructs of the Theoretical Framework of Acceptability (TFA) and their definitions (Adapted from [23], http://creativecommons.org/licenses/by/4.0/, to relate to the DIALOG+ intervention).

https://doi.org/10.1371/journal.pone.0338408.t002

The end-of-trial interviews were analysed using framework analysis [24], following the steps of the analysis process described in the published protocol [9]. Data was analysed by a data analysis coordinating team including at least one researcher from each of the participating countries (TP, TR, SA, FR, JK, SM, BB, EK). The analysis team began by reading English translations of the collected transcripts (17 transcripts were translated) and meeting to exchange our initial analytical impressions from the data. This was followed by individual ‘open coding’ and development of coding lists for patients and clinicians for each TFA component construct. Several discussions were held to compare individual coding lists and after multiple revisions, consensus was reached on a preliminary analytical framework. It was decided to use the same analytical framework for both clinicians’ and patients’ transcripts since the topic guide questions for both participant groups were largely the same and there was significant overlap between the emerging codes. The analysis team piloted the analytical framework on two translated transcripts and on ten original transcripts from all countries. Further refinements of the analytical framework were made based on the piloting process, such as adding additional distinct categories/codes, merging existing codes and clarifying the scope of the codes and categories. After, researchers organized their transcripts in local languages into the framework codes/categories using the RQDA software. The team met regularly to discuss any uncertainties during this process and further modifications of the analytical framework. The indexing stage was followed by charting which was completed using Microsoft Excel. The chart, organized based on the analytical framework, contained a short summary of the data from each participant relating to the particular code and an illustrative quote in English. Next, two researchers from the team (TP, TR) reviewed half of the chart and interpreted a set of themes in light of the research question of interest. These preliminary themes were presented to the rest of the team to check their credibility and clarity, and were modified accordingly. The remaining part of the chart was analysed by the same two researchers, the themes were further revised and presented to the wider team, after which the interpreted themes were finalized and categorized according to the TFA constructs: ‘Affective attitude’, ‘Burden’, ‘Opportunity-cost’, ‘Ethicality’, ‘Self-efficacy’, clinicians’ ‘Intervention Coherence’ and ‘Effectiveness’.

Intervention fidelity

Intervention fidelity explored training of clinicians, delivery of intervention, receipt of intervention and enactment of plans agreed during the intervention sessions, and intervention differentiation, as proposed by Bellg et al. [18] and Carroll et al. [19].

Quantitative data was analysed using descriptive statistics, such as frequencies, means and standard deviations (SD). Qualitative survey data were analysed by two to three researchers (TP, TR, FR) using conventional content analysis following the methodology described by Hsieh & Shannon [25]. We began by reading through the data, followed by open coding and deciding on preliminary lists of codes and key categories in relation to our research question. During the coding process, new codes were added to the list, and some codes were combined, split or renamed. Regular meetings were held to discuss any uncertainties or proposed changes to the coding lists. The coded data was reviewed several times throughout the analysis process until no further modifications of the coding list were needed. Key categories are reported below and, where appropriate, their frequencies. Qualitative data from the end-of-trial interviews regarding receipt, enactment and usefulness of training materials was analysed using framework analysis following the methodology described by Ritchie & Spencer [24] detailed in the following section.

Ethical considerations

All research activities were conducted according to the ethical standards of the institutional and/or national research committee and consistent with the 1975 Helsinki declaration and its later amendments. All procedures were approved by the relevant ethics committees ahead of the study beginning: United Kingdom (Queen Mary University of London QMREC2204a, 16/10/2018), Bosnia and Herzegovina (Klinicki Centar Univerziteta u Sarajevu—Eticki Komitet 03-02-4216, Eticki komitet JU Psihijatriska bolnica Kantona Sarajevo & JU Zavod za bolesti ovisnosti Kantona Sarajevo 02.8–408/19), Serbia (Eticka komisija Medicinskog fakulteta u Beogradu 2650/XII-20 and Eticka komisija Specijalne bolnice ‘Dr Slavoljub Bakalovic’ Vrsac 01–36/1), Kosovo UN Resolution (Hospital and University Clinical Service of Kosovo —Ethics Committee 2019–85), Republic of North Macedonia (Eticka Komisija za istrazuvanje na luge, Medicinski Fakultet pri UKIM vo Skopje 03–24219), and Montenegro (Javna Zdravstvena Ustanova Klinicki Centar Crne Gore—Eticki komitet 03/01–29304/1, ZU Specijalna Bolnica za Psihijatriju “Dobrota” Kotor—Eticki komitet, Eticki Komitet JZU Dom Zdravlja “DR Nika Labovic” Berane 01–47).

The local researchers provided detailed study information to potential participants and obtained written informed consent before data collection. Researchers explained the study to the participants and provided all relevant information, such as the study’s risks and benefits as well as information about their confidentiality and privacy. Participants continued their routine treatment throughout the duration of the study and were made aware that they could withdraw at any point if they wished to do so. The researchers were instructed to end the research activity and contact a clinician as soon as any participating patient appeared highly distressed or upset.

Study registration

This study was retrospectively registered while recruitment was underway at www.isrctn.com on 29 March 2019 (ISRCTN11913964).

Role of context

We identified 11 out of the 14 attributes of context in the framework developed by Squires and colleagues [21]. Findings from three sources of data (site visits reports, policy analysis and pre-trial focus group analysis) were compared in the triangulation analysis by labelling each of the contextual attributes in terms of Agreement (between data sources), Partial agreement, Silence or Dissonance. Detailed findings from the triangulation step of the analysis are presented in S1 Table.

Six attributes of context that potentially negatively impacted on DIALOG+ acceptability, fidelity and outcomes were identified. The majority relate to the health services and systems: limited access to resources such as time, staff, space and equipment; funding prioritization of quantity of care with lack of funds for specialized therapies and technological equipment; reliance on paper records; dominant hospital-based mental health care treating a large volume of patients with psychiatrists being the main service provider; lack of community mental health centres (CMHCs); and a lack of citizens’ organizations that support patients, their families and health care services to run effectively. One attribute of context was associated with clinicians: moderate expectations of mental health clinicians to possess skills in psychotherapy and technology. Another contextual attribute related to patient characteristics: perception of low cognitive capacity among the patient population with PSD when they are unwell.

Eleven attributes of context were identified as potential facilitators to DIALOG+ acceptability, fidelity and outcomes. Most of them relate to the mental health services and systems, such as availability of necessary resources, some presence of CMHCs, universal health coverage for psychological interventions, presence of electronic medical records, an overall appetite for improving existing outpatient mental health care among key stakeholders (better tracking of patients’ progress, prolonged outpatient meetings, inclusion of nurses in psychosocial intervention delivery, and a shift from a medical model of care towards patient-centred care focused on patients’ quality of life), some existence of citizens’ organizations focused on mental health, established laws for protection of people with mental illness, and a culture that values inclusion of carers in patients’ care. Those associated with clinicians included: experience with psychosocial interventions and technology in their provision of routine care, motivation to reinforce community services and a collaborative network between existing mental health staff. Additionally, we identified patients’ ability to be actively involved in their treatment and use technology as potential contextual facilitators.

The comparison of the three data sources identified considerable ‘silence’ in relation to identified contextual attributes, i.e., each data source provided unique contributions to our understanding of the context where DIALOG+ was implemented during the cRCT. The findings were in partial agreement regarding the number of staff accessible in the mental health facilities for the contextual attribute, access to resources. Additionally, the results from all three data sources were in agreement regarding the type of treatments provided in the participating mental health facilities. These largely included pharmacotherapy and, to a lesser extent, various psychological interventions. Moreover, the findings from the site visits reports and policy analysis were in agreement regarding the maintenance of patient records for the contextual attribute about features of the health system.

Experienced intervention acceptability

Participants.

The total number of patients interviewed was 40; 17% of the total sample of patients in the intervention arm (n = 236). All intervention clinicians who consented to participate were interviewed (n = 35/41, 85%).

Table 3 shows the characteristics of interviewed patients and clinicians, interviewers and methods used to conduct interviews. There was an even split of gender among the patients (male n = 21, 52.5%), whereas the majority of interviewed clinicians were female (n = 28, 80%). Patients were on average 43.20 years old (SD = 10.04, range 22–67), while clinicians had an average age of 46.51 (SD = 8.04, range 26–65). The average years of clinical experience for the clinician participants was 16.69 years (SD = 9.28, range 2–39). The majority of the patient participants (n = 32, 80%) had diagnosis of schizophrenic and related disorders (ICD-10 F20-29), whereas the remaining patients had diagnosis of mood affective disorders (ICD-10 F31). Further details of these characteristics by site are available in S2 Table. Moreover, 97.5% of the patients (n = 39) attended all 6 intervention sessions, and one patient attended 5 sessions. Additionally, 82.5% (n = 33) of the interviews with patients and 77.14% (n = 27) of the interviews with clinicians were conducted by a female interviewer. Patients’ interview duration ranged from 7.56 to 85.26 min, with an average length of 30.02 min (SD = 14.74). Interviews with clinicians were on average 30.97 min (SD = 12.43) long (range 10–70 min).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Characteristics of interviewed participants, interviewers and methods used to conduct interviews.

https://doi.org/10.1371/journal.pone.0338408.t003

Qualitative findings.

Overall, DIALOG+ was found to be acceptable to most interviewed clinicians and patients. Findings are presented per each TFA component construct and summarized in Table 4. S2 Table shows the intervention acceptability findings in more detail, including direct quotes from interviewees.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Experienced acceptability-related themes per component constructs of the Theoretical Framework of Acceptability (TFA).

https://doi.org/10.1371/journal.pone.0338408.t004

1. 1. Affective Attitude

Experienced affective attitude is defined as how the clinicians and patients feel about DIALOG+, after taking part in the intervention. Clinicians and patients expressed very positive feelings about participating in the DIALOG+ sessions. Participating in DIALOG+ sessions felt positive for patients because it met their needs to be heard and empowered. Both patients and clinicians reported experiencing DIALOG+ sessions as enjoyable. Additionally, several participants found DIALOG+ sessions more relaxing in comparison to their usual treatment. However, participants’ accounts also suggested that the intervention acceptability was reduced among some due to its perceived lack of flexibility and monotony. Some of the questions and domains covered by DIALOG+ were perceived as too intimate, and potentially limited the engagement of patients during the sessions.

1. 2. Burden

The TFA component construct ‘Burden’ is concerned with the effort required to participate in the intervention. The intervention was perceived as complex and cognitively demanding for some, whereas for others it was perceived as simple to use and understand. Additionally, clinicians reported feeling disempowered to help with certain patients’ life areas that they perceived as outside of their professional role, such as job search or any other material resources. Experienced feeling of disempowerment could have acted as a burden for clinicians during the intervention. Furthermore, the patients’ booklet was perceived to require too much effort and was viewed as an obligation by some that may have impacted, in turn, their acceptability of the intervention. Overall, there was some cognitive and psychological burden experienced with participating in the DIALOG+ sessions.

1. 3. Opportunity cost

Opportunity cost represents the benefits that had to be given up to engage with DIALOG+ . Some participants reported additional workload associated with participating in the intervention, requiring them to skip their breaks and work longer hours. However, the majority of participants did not report having to forgo any benefits as a result of their participating in DIALOG+, suggesting that the intervention largely showed high acceptability in relation to opportunity-costs.

1. 4. Ethicality

Ethicality is the degree to which DIALOG+ is perceived to have good fit with the clinicians’ and patients’ value systems. DIALOG+ was perceived as similar to clinicians’ regular work and as such also suitable for the existing value system. Many patients, however, indicated that DIALOG+ offers a more detailed approach in comparison with their usual care. This suggests that DIALOG+ intervention is a good fit with the current clinical practice. DIALOG+ was valued by both clinicians and patients because it fit with their view of “best clinical practice”. DIALOG+ was seen to offer care during routine clinical meeting that was beyond medication adjustments, which was perceived as particularly valuable. Overall, participants reported that DIALOG+ was viewed as valuable.

1. 5. Self-Efficacy

This component construct was defined as clinicians’ and patients’ confidence that they can participate in the DIALOG+ sessions. The ability of both clinicians and patients to allow themselves to gradually develop confidence to receive and deliver DIALOG+ emerged from the participants’ accounts. This suggests that support and appropriate training programme can facilitate intervention acceptability. Additionally, it indicates the important role of time for the intervention deliverers and recipients to adapt to new practices, and eventually, adopt them. Participants’ level of confidence in using the intervention depended on their previous familiarity with such therapeutic approaches and between the clinician-patient pair. Participants generally reported feeling confident about participating in DIALOG+ sessions. Clinicians did not perceive any restrictions from their institutions to deliver the intervention, but they perceived that any institutional support received was more passive than active. Overall, clinicians and patients expressed moderate confidence that they can execute the behaviours needed to partake in the intervention.

1. 6. Clinicians’ Intervention Coherence

This component construct is about the extent to which the clinicians understood DIALOG+ and how it works to accomplish its benefits. Clinicians largely expressed a high level of coherence of the intervention’s procedure and its main principles (e.g., patient-centeredness, patient involvement, positive reinforcement, solution-focused and structured communication, a more therapeutic relationship, opportunity for self-reflection and self-expression by patients) through which the intervention effects change, as intended. Clinicians’ view was that the use of tablet facilitated conduct of DIALOG+ meetings and enabled them to economically organize, store, and use data and information relevant to patients. Clinicians also reported that they understood the usefulness of the patients’ booklet during sessions. Accounts about the degree to which patients reported understanding DIALOG+ are included under intervention receipt in the results of the second study objective (subheading: Receipt of Intervention).

1. 7. Perceived effectiveness

Experienced effectiveness is the degree to which DIALOG+ is perceived to have accomplished its main purpose. Participants experienced many of the intended benefits of DIALOG+, as originally envisaged by the trial designers. Clinicians and patients perceived DIALOG+ as highly likely to have improved patients’ quality of life. Participants’ views indicate that poor mental health prevented using the intervention as designed with some patients, which likely limited intervention acceptability. Certain personality characteristics, such as age, proactive attitude, open-mindedness and communicativeness were reported to influence intervention effectiveness. Already established familiarity between the clinician and patient when implementing a new intervention was perceived to act as both an enabler and a barrier to intervention effectiveness. Additionally, some participants did not observe any added value of using DIALOG+ compared to routine care and some felt that the number of intervention sessions delivered during the trial was insufficient to improve patients’ condition.

Overall, the findings from the qualitative study suggest that mental health clinicians and patients with PSD, who were part of the intervention arm of the IMPULSE trial, found DIALOG+ to be moderately to highly acceptable. Using a multi-dimensional framework to investigate acceptability appeared to generate a depth and breadth of informative responses. In particular, findings from the acceptability study point to a number of ways in which acceptability of DIALOG+ could be optimized without compromising the fidelity of the intervention as originally designed. Potential strategies for optimizing DIALOG+ are presented below in the Discussion section.

Intervention fidelity

Fidelity was investigated in terms of five domains: training, delivery, receipt, enactment (from [18]) and differentiation (from [19]). Findings relating to each domain are reported in the sections below. S3 Table presents the intervention fidelity findings in more detail.

Delivery of intervention.

Occurrence, attendance and duration of intervention sessions.

The proportion of DIALOG+ sessions that took place was 87.64% (n = 1,241) of the intended sessions. The majority of patients in the intervention arm (n = 189, 80.08%) attended all 6 sessions during the trial. 15 patients (6.36%) randomized to the intervention arm did not attend any sessions. The mean number of intervention sessions per patient was 5.26 (SD = 1.77). S4 Table contains information that summarizes the occurrence and attendance of all intervention sessions. Each DIALOG+ session was planned to last from 30 to 60 min. The duration of DIALOG+ sessions in the trial ranged from 4 to 90 min, with a mean duration of 28 min (SD = 11.69).

The most common method of delivery of intervention sessions was face-to-face (89.52%, n = 1,111). Few of the fifth and the majority of the sixth sessions were delivered remotely instead of in person due to pandemic restrictions.

After all first DIALOG+ sessions were completed, 32 clinicians (78.05%) reported not experiencing any problems with the delivery of DIALOG+ and participating in the study. The remaining clinicians (n = 9, 21.95%) reported problems related to technical difficulties and communication issues, the majority of which were recorded as successfully resolved.

Life and treatment areas discussed in the four-step approach of the intervention.

In the interests of time efficiency, clinicians were instructed that no more than one domain should be selected for an in-depth discussion during the first DIALOG+ session and no more than three domains in second to sixth session. One domain was selected in 88.24% (n = 195) of the 1^st DIALOG+ sessions as intended, and one to three domains were selected in 97.55% (n = 995) of the 2^nd to 6^th DIALOG+ sessions as planned. The mean number of domains selected per session was 1.8 (SD = 0.89).

The selection frequency of life and treatment areas during the intervention sessions is shown in Fig 1. The most frequently selected area to be discussed in the 4-step approach of the intervention was ‘Mental Health’ (28.93% of all delivered DIALOG+ sessions), whereas the least frequently selected area was ‘Practical Help’ (4.92% of all delivered DIALOG+ sessions).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A bar chart showing the selection frequency of life and treatment areas from the DIALOG scale (part of DIALOG+ intervention) in descending order for percentage of all delivered intervention sessions (n = 1,241).

https://doi.org/10.1371/journal.pone.0338408.g001

Actions agreed at the end of the intervention session.

The mean number of action items per DIALOG+ session was 2.52 (SD = 1.35). Most commonly, one to three action items were agreed during one DIALOG+ session (84.21%). S3 Table contains more detailed information on frequency of action items agreed during the DIALOG+ sessions.

The qualitative data reported as action items were further analysed based on two pre-determined themes: a) person to whom the action is assigned; and b) type of action. When the qualitative data entered as one action was deemed to contain text that referred to more than one action, the analysis was done per each unique action. Reported data which was not considered an action was excluded in the analysis – 101 items were identified in the sample that did not contain an action item. Thus, after data cleaning, 3,126 unique action items set during the intervention sessions were identified. Table 5 contains a summary of the person who was responsible for a unique action and the type of action. A great majority of actions were assigned to the patient (82.15%, n = 2,651). Patient-led actions mostly related to connecting with others and attempts to introduce healthier lifestyle habits (e.g., physical exercises, diets, sleep habits, etc.). It was not possible to determine the responsible person for 6 of the action items.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. The frequency of action items per responsible person and type of action.

https://doi.org/10.1371/journal.pone.0338408.t005

Adherence to the intervention manual.

37 audio recordings from the DIALOG+ sessions were obtained from 17/41 (41.46%) clinicians in the intervention arm. Our goal was to obtain one audio recording from each clinician in the intervention arm. This discrepancy occurred due to too few participants consenting to be audio recorded. Most recordings were of 3^rd DIALOG+ sessions (n = 33), the rest were of 4^th DIALOG+ sessions.

Each recording was scored against the 19 items of the DIALOG+ Adherence Scale [20]. More than one recording was collected from 6 clinicians, in which case an average score was calculated from each of the recordings. The DAS scores for the 17 clinicians in the intervention arm are shown in Table 6. Overall the mean adherence to the DIALOG+ manual was 13.88/19 (73.05%). Seven high-scoring items (mean score ≥ 0.90) were identified, and only one low-scoring item (mean score ≤ 0.25).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Descriptive statistics for DIALOG+ Adherence Scale (DAS) [20] scores for clinicians in the intervention and control arm and the number (%) of clinicians who delivered each of the DAS items.

https://doi.org/10.1371/journal.pone.0338408.t006

Receipt of intervention.

We used Bellg et al.’s [18] definition of receipt to mean the patient side of fidelity, their engagement with and comprehension of the intervention sessions. The majority of patients in the audio recordings were able to use the DIALOG scale, select at least one area for further discussion, discuss at least one area with the 4-step approach and suggest or agree actions to improve their satisfaction (Table 7).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Patient level data on the receipt of the intervention as analysed from audio recordings from intervention sessions.

https://doi.org/10.1371/journal.pone.0338408.t007

Receipt was further explored during the end-of-trial qualitative interviews with patients and clinicians allocated to the intervention arm. Three intervention receipt-related themes were interpreted from the data through framework analysis (Table 8). The qualitative data regarding patients’ receipt of the intervention suggest that they were largely able to recognize the steps involved in the intervention and how it works (e.g., patient-centeredness, patient involvement, therapeutic self-expression and self-reflection, a focus on solutions, comprehensive and structured communication), however patients’ comprehension was perceived as limited among those with cognitive impairments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Intervention receipt-related themes with direct quotes.

https://doi.org/10.1371/journal.pone.0338408.t008

Enactment of agreed actions.

We defined enactment as patients attempting to complete, or completing, the action items agreed upon during the DIALOG+ session, in the target setting [18]. Table 9 shows the descriptive statistics of the enactment data collected from the audio recordings.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Patient level data on the enactment of the agreed actions as analysed from audio recordings from intervention sessions.

https://doi.org/10.1371/journal.pone.0338408.t009

Enactment was further explored during the end-of-trial qualitative interviews with patients and clinicians allocated to the intervention arm. Table 10 presents the enactment-related themes that were interpreted from the data through framework analysis. The results suggest that many patients achieved the intended enactment of the agreed actions, however there were several external and internal factors, such as lack of support, financial resources, motivation, and poor mental health that hindered patients’ enactment during the trial.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Enactment-related themes with direct quotes.

https://doi.org/10.1371/journal.pone.0338408.t010

Occurrence, attendance and duration of control sessions.

The proportion of control sessions that took place was 95.19% (n = 1,325) of the intended sessions. The majority of the patients in the control arm (n = 207, 89.22%) attended all 6 sessions during the trial. Three patients (1.29%) from the control arm did not participate in any trial session. The mean number of sessions per patient was 5.71 (SD = 1.01). Table 11 summarizes the occurrence and attendance of all control sessions. The duration of routine clinical meetings in the participating countries was expected to range from 15–40 min. The duration of control sessions in the trial ranged from 2 to 60 min, with a mean duration of 20.14 min (SD = 11.22). The mean duration of both control and intervention sessions is shown in Fig 2. The mean difference between the duration of intervention and control sessions was 7.86 min (95% CI 6.97, 8.75; p < 0.001).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 11. Control sessions’ occurrence.

https://doi.org/10.1371/journal.pone.0338408.t011

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Mean duration of intervention and control sessions throughout the trial.

https://doi.org/10.1371/journal.pone.0338408.g002

The most common method of delivery of control sessions was face-to-face (91.62%, n = 1,214). Some of the fifth and almost half of the sixth control sessions were provided remotely due to pandemic restrictions.

Adherence to the intervention manual and investigation of potential contamination in control group:

16 audio recordings from control sessions were obtained from 8/40 (20%) clinicians in the control arm, significantly less than our goal of obtaining one audio recording from each clinician in the control arm (n = 40). This sample was smaller because too few participants consented to be audio recorded. The majority of the recordings (n = 13) were of 3^rd control session, the rest were of 4^th (n = 2) and 6^th (n = 1) sessions.

Each recording was scored against the DAS [20]. More than one recording was collected from 2 clinicians, in which case an average score was calculated from each of the recordings. The DAS scores for the 8 clinicians in the control arm are shown in Table 6. The mean total score on the DAS was 3.5/19, compared with 13.88/19 for the recordings from the intervention sessions. There were also notable differences between the mean scores of the DAS subscales between the intervention and control sessions (S3 Table), further supporting the evidence of differentiation.

Summary of process evaluation results

This process evaluation integrates many different data sources and yields an in-depth investigation of the IMPULSE cRCT and the DIALOG+ intervention as delivered and received during the trial. A number of contextual attributes in the participating countries, largely relating to the health services and systems, potentially negatively impacted on DIALOG+ acceptability, fidelity and outcomes. However, more attributes of context with the potential to have facilitated the intervention were identified. Of particular importance was the reported appetite for change of the existing mental health care among key stakeholders.

The experienced acceptability of the intervention was moderate to high. Participants felt positively about the intervention, perceived that a moderate amount of effort was required to take part in the intervention, reported little opportunity-costs associated with participating in the intervention, perceived DIALOG+ as a good fit with their value system, expressed moderate confidence that they can execute the behaviours needed to take part in the intervention, expressed a high level of understanding of the intervention and how it works and perceived DIALOG+ as highly likely to have achieved its purpose of improving patients’ quality of life.

The fidelity of the intervention was high. The intervention training, delivery and attendance at intervention sessions were as planned. The duration of the sessions was slightly lower than envisaged. Regarding the content of the intervention sessions, the number of domains selected for discussion as well as the number of actions set occurred as planned. Most actions were assigned to the patient to complete before the next session, indicating high patient involvement during the sessions. A few problems with intervention delivery that were reported, such as technical issues and communication concerns, were successfully resolved. Additionally, the adherence to the DIALOG+ manual was moderate. Patients’ receipt of the intervention was high, with certain limitations among patients perceived to have cognitive impairments. Patients’ enactment of actions was moderate. Factors that appear to have limited enactment include lack of motivation and poor mental health, as well as lack of access to resources. The delivery, attendance and duration of the control sessions were also as intended and no contamination between intervention and control groups was identified. Thus, intervention differentiation was also high.

In regard to the differences between the DIALOG+ intervention and treatment as usual during routine clinical meetings, DIALOG+ was perceived to foster a more relaxed atmosphere and to offer opportunity for the patients to express themselves more and understand themselves better. They were perceived to enable more comprehensive and structured conversations between the patients and clinicians and thus to establish a stronger clinician-patient relationship. DIALOG+ was seen to encourage more patient involvement, and to ensure a solution-focused approach to care, while reducing the emphasis solely on medication. The intervention also utilized tablet computers during clinical meetings. Routine clinical meetings are usually not led by nurses, which was the case with several clinicians delivering DIALOG+. Additionally, DIALOG+ sessions lasted longer, and were perceived by some to be more monotonous, too intimate and more complex. Importantly, some clinicians did not observe any difference between usual treatment and DIALOG+ . However, upon analysis of audiotapes from control and intervention sessions, it was concluded that treatment as usual was meaningfully distinct from DIALOG+ .

To further improve our understanding about how DIALOG+ works, the integration of the findings was mapped to Omer et al.’s [5] proposed mechanisms of actions of DIALOG+ in Table 12.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 12. Mapping of IMPULSE process evaluation findings to Omer et al.’s [5] mechanisms of actions of DIALOG+ .

https://doi.org/10.1371/journal.pone.0338408.t012

Strengths and limitations

To the best of our knowledge, this is the first process evaluation of a non-pharmacological cRCT with people with PSD in Southeast Europe, and one of few such efforts globally. It is not only a comprehensive exploration of the mental health care in this region, which has been referred as “a blind spot on the global mental health map” [27], but also of implementation and potential sustainability of a non-pharmacological intervention overall. The multi-method approach of this process evaluation and its use of published frameworks to explore the role of context, acceptability of the intervention and the fidelity are major strengths of this study. By integrating the different data sources and data types, this study yields a thorough knowledge which is ‘greater than the sum of the parts’, generated from all five participating LMICs in Southeast Europe. Additionally, all data was collected prior to researchers’ or participants’ awareness of the trial outcomes, which minimized data collection bias. Furthermore, all researchers who participated in the qualitative study received an in-depth training programme in qualitative methods, which reduced the risk of varying levels of quality and depth of information collected across the different interviews. The qualitative data was coded in the original languages and researchers from all participating countries collaborated in the framework analysis process. This minimized the loss of contextual-specific meaning and traits.

This study also has several limitations. The number of audio recordings obtained from the trial sessions was fewer than the anticipated one recording per each clinician, adding a risk of sampling bias. The duration of control sessions and some intervention sessions was collected from clinicians, making it subject to clinicians’ memory and social desirability bias. Moreover, only a subgroup of all trial participants were interviewed at the end of the trial, which may have introduced selection bias.

Strengths and limitations of the IMPULSE cRCT are reported in detail elsewhere [6].

Future implications

The current study findings can inform strategies for further scaling and sustainability of a digital, psychosocial intervention such as DIALOG+. The DIALOG+ intervention may not be well-suited for all PSD patients or during every routine clinical meeting and may need tailoring to different clinical situations. The findings from this study suggest that it could be most suitable for outpatients open to communication, reflection and change. The intervention seems to be more acceptable to younger patients and clinicians. The frequency and intervals between sessions should be flexible, based on the patient’s needs. Further monitoring of intervention delivery and patients’ enactment of agreed actions, as well as availability of continuous supervision for clinicians, could support healthcare systems to track long-term implementation of DIALOG+, and potential lasting effects on patients and clinicians.

Future DIALOG+ training should particularly focus on intervention steps related to the initial DIALOG scale and review of patients’ ratings, as these had the lowest adherence. Additionally, the training should provide more guidance about actions set at the end of the intervention session. To improve the enactment of actions, they should be well-tailored to the individual patient’s needs and abilities. Based on our findings (Tables 6 and 11), we suggest the use of patients’ booklets could be optional for the patient.

The mean difference in length of control and intervention sessions could inform future evaluations of DIALOG+ that should consider a comparison group where the clinical meetings would last similar to the average duration of DIALOG+ sessions to account for the possible “attention effect” – a longer amount of time spent with treating clinician. Future research should also explore evidence-based techniques for addressing the identified contextual barriers in low-resource mental health care settings.