Recruitment and retention strategies in mental health trials – A systematic review

Background Recruitment and retention challenges are very common in mental health randomised trials. Investigators utilise different methods to improve recruitment or retention. However, evidence of the effectiveness and efficiency of these strategies in mental health has not been synthesised. This systematic review is to investigate and assess the effectiveness and cost-effectiveness of different strategies to improve recruitment and retention in mental health randomised trials. Methods and materials MEDLINE, EMBASE, the Cochrane Methodology Register and PsycINFO were searched from beginning of record up to July 2016. Randomised trials involving participants with mental health problems which compared different strategies for recruitment or retention were selected. Two authors independently screened identified studies for eligibility. Results A total of 5,157 citations were identified. Thirteen articles were included, 11 on recruitment and 2 on retention. Three randomised controlled trials compared different recruitment strategies, none of which found statistically significant differences between the interventional recruitment strategies and the routine recruitment methods. Retrospective comparisons of recruitment methods showed that non-web-based advertisement and recruitment by clinical research staff each have advantages in efficiency. Web-based adverts had the lowest cost per person recruited (£13.41 per person recruited). Specialised care referral cost £183.24 per person, non-web-based adverts cost £372.03 per patient and recruitment via primary care cost £407.65 for each patient. Financial incentives, abridged questionnaires and pre-notification had a positive effect on retention rates. Conclusion The recruitment studies included showed differences in strategies, clinical settings, mental health conditions and study design. It is difficult to assess the overall effectiveness of any particular recruitment strategy as some strategies that worked well for a particular population may not work as well for others. Paying attention to the accessibility of information and consent materials may help improve recruitment. More research in this area is needed given its important implications.


Methods and materials
MEDLINE, EMBASE, the Cochrane Methodology Register and PsycINFO were searched from beginning of record up to July 2016. Randomised trials involving participants with mental health problems which compared different strategies for recruitment or retention were selected. Two authors independently screened identified studies for eligibility.

Results
A total of 5,157 citations were identified. Thirteen articles were included, 11 on recruitment and 2 on retention. Three randomised controlled trials compared different recruitment strategies, none of which found statistically significant differences between the interventional recruitment strategies and the routine recruitment methods. Retrospective comparisons of recruitment methods showed that non-web-based advertisement and recruitment by clinical research staff each have advantages in efficiency. Web-based adverts had the lowest cost per person recruited (£13.41 per person recruited). Specialised care referral cost £183. 24 per person, non-web-based adverts cost £372.03 per patient and recruitment via primary care cost £407.65 for each patient. Financial incentives, abridged questionnaires and prenotification had a positive effect on retention rates. PLOS  Retention is another pivotal component to a trial's scientific success as it is key to a trial's validity. Attrition may happen in drug trials because of side effects of the medication. Some patients may experience deterioration of their health during the follow-up period, making them reluctant to continue with treatment or the trial. In trials of complex interventions, such as cognitive behavioural therapy or early supported discharge, the absence of blinding in the control arm means that the participants know that they have not received the intervention, which may reduce engagement with trial follow-up or increase the risk of drop-out. It has been suggested that high drop-out rates are associated with larger sample sizes in antipsychotic trials, more specifically trials with multi-centre design. [10] However in modern trials, the sample size required often necessitates a multi-centre design as a single site would not provide enough participants and may not provide sufficiently generalisable results. This requirement for a multi-centre design might result in retention issues.
Previous systematic reviews by Treweek et al. and Brueton et al. investigated the efficacy of different strategies to improve recruitment and retention to randomised trials. [11,12] However evidence in the mental health trial population remains sparse. The review by Treweek et al. summarised the efficacy of different strategies to improve recruitment into randomised trials, but only included 3 eligible studies in mental health. No mental health studies were included in the systematic review by Brueton et al., which investigated the efficacy of different strategies to improve retention. Treweek et al. found that open trial design, and telephone reminders to people who do not respond to postal invitations may improve recruitment, whereas bespoke participant information materials helped little in recruitment. [13] Offering a small financial incentive for completing follow-up questionnaires appeared to help retain patients in the trials, as suggested by Brueton et al. [12] An increasing number of studies employ the use of a "study within a trial" (SWAT) method to assess the impact of technical or design innovations on a trial's efficiency. [14] To date, most different recruitment strategies are usually employed in an ad hoc manner. Evidence on comparing recruitment strategies retrospectively and observationally can also provide some insight before SWATs are planned.
The aim of this review is to evaluate the evidence base for strategies to improve the recruitment and retention of patients to clinical trials in mental health. A secondary aim was to evaluate the cost-effectiveness of different recruitment and retention strategies, reported as the cost per patient recruited, or cost per patient retained.

Criteria for considering studies for this review
2.1.1 Types of studies. Two review authors independently screened titles and abstracts and any disagreements in selection were resolved through discussion. Studies that used randomised or observational methods to compare different recruitment strategies designed to recruit participants to randomised controlled trials of interventions for mental health problems were considered. Embedded randomised studies of different recruitment strategies were identified, but given the small number of such studies, we also included randomised controlled trials (RCTs) of mental health interventions which reported the effectiveness a range of strategies used in recruitment retrospectively (e.g. without randomising to different recruitment strategies). For retention, randomised trials of different retention strategies that were embedded in a randomised clinical trial (host trial), or within epidemiological studies such as crosssectional surveys were included. A full description of the study protocol is available in S1 File.
2.1.2 Types of data. Studies comparing recruitment or retention that involved adult participants with mental health problems, regardless of gender, ethnicity or geographic location, were included. Of particular interest were trials including patients with serious mental illnesses (SMI), such as schizophrenia, but given the expectation of finding only a small number of studies involving these patients, the criteria were broadened to include common mental health problems such as depression and anxiety. Dementia and other organic mental health conditions were excluded, given the different context in which these trials are likely to be conducted. Studies on substance misuse were also excluded as this group of patients is likely to present different recruitment and retention challenges. Studies which did not report outcomes on recruitment or retention strategies for RCTs, studies in which mental illness was comorbid with other physical medical conditions (e.g. cardiovascular disease), and studies not involving adults (e.g. children or adolescents) were also excluded.

Types of methods.
Strategies aimed at enhancing recruitment and retention included, but were not limited to: For recruitment, the main outcomes of interest were the type of strategies employed in different studies and the number of patients recruited using each individual strategy. We also extracted data on how many potential participants were approached, if available, using each different strategy in each study. For studies comparing different retention strategies, the primary outcome was 'response', defined as the percentage of participants who were successfully engaged in follow-up assessments via each strategy out of the total number of people initially randomised to that strategy.

Secondary outcomes.
We were also interested in the cost of each patient recruited/ retained through a specific strategy (if any mentioned), namely, the cost-effectiveness of each strategy, defined as the mean cost per patient recruited or mean cost per patient retained, respectively.

Search strategy
We designed a search method for identifying published randomised trials that focused on improving recruitment and retention in mental health randomised trials. We did not apply any language restrictions apart from English language abstracts.
The search method was comprised of 4 components, each of which included both free-text terms and subject headings. The Boolean operator OR combined terms related to enhancing recruitment and improving retention. This was then combined using the AND operator with terms related to mental health conditions and randomised controlled trials. A brief search strategy is described as follows: (informed consent OR recruit OR particip) OR (retention OR attrit OR retain) The full search strategies for all of the 4 databases are included in S1 File.

Data extraction and analysis
2.4.1 Data extraction. Two reviewers extracted data from eligible studies. Data extracted for the recruitment trials and their corresponding host trials included: • For host trials: country, disease area, design, sample size, setting, primary outcomes, funding body; • For embedded randomised recruitment trials: strategies to which participants were randomised, number of participants in each arm who were recruited to the host trial; • For studies that compared recruitment strategies retrospectively: strategies used for recruitment, number of participants recruited and approached via each strategy.
For retention trials and their host trials, data extracted included: • For host trials: country, disease area, design, sample size, setting, primary outcomes, follow up period, funding body; • For retention trials: strategies to improve retention; retention rates for each strategy.

Assessment of risk of bias.
We used the Critical Appraisal Checklist for Randomised Controlled Trials developed by the Joanna Briggs Institute (JBI) for the assessment of risk of bias for the eligible studies, and calculated the overall score based on the number of items checked for each assessment. Details of the risk of bias assessments are included in S2 File.

Data analysis.
For randomised comparative studies, we used relative risk to describe the effect of each recruitment strategy. Non-randomised studies were categorised according to similarity of strategies, for instance, by combining optimised consent materials and incentives. We ranked the strategies based on the numbers of patients recruited and identified strategies that recruited most participants in each study. We also calculated the total number of patients randomised through each recruitment strategy for recruitment studies and the number of responses in each retention strategy for retention studies. Cost-effectiveness of strategies where cost data were available was measured by average cost per patient randomised or average cost per response, respectively. Cost information was first converted to the equivalent monetary value in 2016 using relevant inflation rates of the study country, and subsequently into GBP based on average exchange rates between each currency and GBP in 2016. (http:// www.ukforex.co.uk/forex-tools/historical-rate-tools/yearly-average-rates). The average costeffectiveness of each category of recruitment strategy was calculated using a weighted average approach, where the mean costs were weighted by the sample size.
For studies where cost data were not reported, we extracted relevant information on the processes and generated the cost using available reference cost information (from e.g. Personal Social Services Research Unit). Given the considerable uncertainty in this approach due to insufficient information on resources used, we also performed sensitivity analysis under different scenarios. For instance, the cost of recruiting using health care providers, or research staff, depends on the number of hours spent on recruitment. The assumptions made were: parttime (e.g. 3 hours/day) versus full-time (e.g. 7.5 hours/day). The costing mainly employed a bottom-up approach from a UK NHS/personal social services (PSS) perspective. The unit cost of each component mentioned during the recruitment process was multiplied by the recruitment duration, or study duration otherwise, before all the relevant cost components were summed to make the total recruitment cost. The cost-effectiveness of each strategy was obtained using the total cost divided by the total number of participants for each strategy.

Results of the search
We identified 5157 abstracts, titles and other records from electronic databases from inception of records until July 2016. Of these, 116 were identified for full-text screening and 12 were found to be eligible for inclusion. One additional study (Hughes-Morley 2016) identified by one of the reviewers of this article was also included. (Fig 1) Out of the 13 included studies, 11 are studies on recruitment strategies in the mental health clinical trial context, and 2 focused on retention. Five out of 13 were randomised comparative trials of different recruitment or retention strategies, and 8 were observational comparisons of recruitment strategies embedded within a randomised trial. Table 1  Four studies were carried out and funded in the UK, 5 in the US and 2 in Australia. One study involved recruitment to a preventive programme for depression, and one involved a relapse prevention trial in women with a history of post-partum depression. Two of the studies was conducted with people with severe mental illnesses. Five were carried out in a primary care setting. Four involved female participants only. Except for one RCT which was a study of recruitment to a hypothetical trial, the studies involved recruitment to randomised trials involving a range of interventions including mindfulness cognitive behavioural therapy (CBT), health promotion via email, telehealth intervention, exercise, antidepressants, interpersonal therapy and psycho-education. A brief description of the included studies is available in S1 Supplementary.   Recruitment and retention strategies in mental health trials -A systematic review information material was superior to non-optimised information for recruitment of patients with depression in primary care, but this result may have occurred by chance. Finally, offering information on PPI collaboration on the trial was not found to have a positive impact on trial recruitment.

Non-randomised studies.
Krusche et al. suggested that recruiting by adverts and posters showed no less efficacy than recruiting from GP referrals. In contrast, a study using electronic health records to remind GPs to approach potentially eligible patients was more efficient than recruitment by research staff in the clinic waiting room. The latter involved considerably more effort (more than 8000 patients were approached). [19] Le et al. also suggested that being contacted by clinical staff was more successful than being contacted by research staff. Among trials involving people with common mental disorders, GP referrals and contact by clinical staff were the most efficient and successful recruitment strategies and both resulted in an adequate number of patients for the size of a modern trial. Financial incentives are commonly used in commercially funded trials. The study done by DeBar suggested that neither different levels of financial incentives nor different lengths of assessment substantially affected recruitment rates. [23]  In an RCT of mindfulness versus treatment as usual (TAU) in women with depression, anxiety or stress, Woolhouse et al. used both more active (researcher approaching patients in clinic waiting room) and less active (invitations sent to potential participants) strategies. [15] The numbers of patients recruited were similar, despite 2,500 mailshots being sent compared with the researcher approaching 50 patients. A study comparing various forms of online recruitment for a preventive intervention for people with subthreshold depressive symptoms (as assessed by an online questionnaire) found that Google adverts recruited the highest number of participants (755 patients recruited). However, it was indicated that a total of 94,808 potential participants were approached, echoing findings by Krusche et al. suggesting that lower success rates may often be the case in recruitment via online advertisements. [16] 3.2.4 Cost effectiveness of recruitment strategies. The results of the cost-effectiveness of recruitment strategies are reported in Table 3. The strategy with the lowest cost per patient recruited was web-based advertisement (£13.41 per patient), followed by recruiting via specialised care (£183.24 per patient), non-web-based adverts (£372.03 per patient) and recruitment via primary care (£407.65 per patient). The sensitivity analysis considered the variation in cost according to the different strategies used. For instance, the cost of recruiting using health care providers depends on how much time is spent on recruitment. The two assumed levels of time commitment were part-time (3 hours/day) versus full-time (7.5 hours/day). Fig 2 shows the results of the costing and sensitivity analyses, in comparison with the cost-effectiveness reported with original data. As each study reported different information on costing the recruitment, even for similar strategies across different studies, costs obtained from available sources showed considerable variation. Shown below is an example of how recruitment cost was obtained, using a study by Morgan et al. (Table 4). Further details and sensitivity analysis are given in S1 Data. Table 5 summarises two studies identified that compared different strategies to improve postal response in surveys. On joining the trials, participants were randomised to followed up via different methods, and their response rates at follow-up were compared as a proxy for retention rates using the different methods (McLean 2014, Dirmaier 2007). McLean et al. investigated the effects of pre-notification (e.g. notifying participants in advance that they would be asked for information) and envelope 'teaser' (placement of a short message on the survey envelope) on increasing postal response rates in a bulimia nervosa mental health literacy survey. Dirmaier et al. conducted a randomised trial to find out whether small cash incentives and a shortened questionnaire helped increase postal response rates in a mailed follow-up survey one year after inpatient psychotherapeutic treatment for mental health patients. Both studies used a 2×2 factorial design to investigate the impact of strategies on postal response rates. Financial incentives, abridged questionnaire and pre-notification were suggested to be effective to increase postal response rates, but the effects were small.

Discussion
The review identified only 3 eligible randomised comparative studies of alternative recruitment strategies in mental health clinical trials. None showed a statistically significant difference between using standard and optimised patient consent and information materials. Our findings were consistent with those of Treweek et al. The difference approached significance in one trial of recruitment using optimised patient information material compared with original patient material (Man 2015), although the effect was small. The 8 other studies included in the recruitment section of the review consisted of non-randomised, retrospective comparisons of different recruitment strategies. It is difficult to know whether the different strategies were employed in comparable ways in these studies, or for the same duration. Given the small number of randomised comparative studies identified, and the inconclusive results, this review suggests further research in this area may benefit trial recruitment. Two randomised studies comparing different retention strategies in mental health were identified (McLean 2014, Dirmaier 2007). Both involved different ways of maximising response rates to postal assessments. As follow-up assessment in RCTs is often carried out in the form of a questionnaire, the response rate to this type of assessment may be appropriate as a proxy for retention.
Prior to this review, we also piloted a search strategy that encompassed informed consent, recruitment, antipsychotics and randomised trials, attempting to review recruitment strategies in antipsychotic randomised trials. It generated approximately 2,000 records from MEDLINE,  EMBASE, and CMR. However, after screening there was only 1 study (Jeste 2009) which met our criteria. Little attention has been paid to such methodological trials (e.g. using SWATs to increase the evidence base for trial decision-making) that endeavour to tackle some of the most common issues in mental health clinical trials.
The included studies showed substantial differences in strategies used, but also in clinical settings, mental health conditions and study design. We were not able to obtain a pooled estimate of recruitment efficacy of these strategies due to the non-randomised designs used, and the choice of analysis which could be used to assess the relative efficacy of different strategies was limited. It is therefore difficult to estimate the efficacy from beyond an individual level. Also, some included studies did not report numbers of potential participants approached by each strategy, e.g. the denominator for the efficiency measure of recruitment strategies (number recruited divided by number approached), and comparison between numerators should be made with caution, as some strategies have broader reach to the population and some studies required larger sample sizes. There were some interesting insights from the result of some recruitment studies, nevertheless. For instance, although clinical staff and GPs are often thought to be helpful in recruiting patients into randomised trials, here it was shown that they recruited no better than advertisements. The comparisons made were ad hoc, however, and in the absence of randomised controlled experiments, the area needs more rigorous investigation.
In this review, we also considered cost-effectiveness for each strategy based on numbers of participants recruited and cost incurred. It provided some useful information for public funded trials, which often work on a limited budget. However, it is worth noting that the choice of recruitment strategy should consider not solely cost-effectiveness, but also the study design, types of intervention and more importantly, population characteristics. For instance, we found that although using web-based advertisements showed merit in terms of efficacy and cost-effectiveness in recruitment, however the loss to follow-up in the population recruited via this method cannot be ignored. It is essential also to consider whether certain recruitment methods may identify a biased population. We also considered the uncertainty due to the inadequately reported cost information in the included studies, and performed sensitivity analysis of the costs obtained. The lack of a standard and transparent methodological framework for reporting the costs or resource use during recruitment has engendered considerable variations in the analysis and has led to challenges in interpreting the results. For instance, strategies that involved research assistants recruiting in clinic waiting rooms did not specify the total hours spent, therefore it was necessary to make assumptions regarding the numbers of hours spent per day on the recruitment task. Even for studies which employed similar recruitment strategies, reporting on resources used during recruitment varied tremendously, leading to considerable differences in costs obtained. Speich et al. also found in their systematic review that none of the included studies provided empirical resource use and cost data for all aspects of an RCT, and for trials that reported costs of recruitment, even similar recruitment strategies could cost different amounts across studies. Within a given category of recruitment strategies, for instance, the median cost of a mailed invitation was 228 USD, ranging from 15 to 1,116 USD per patient. [28] Recently the ORRCA project (Online Resource for Recruitment research in Clinical triAls, www.orrca.org.uk) has attempted to bring together all the studies on recruitment into randomised trials by creating a searchable database. This initiative may help to inform trialists and recruiters of better ways to recruit patients into trials. [29] Also, Madurasinghe et al. provided guidelines for reporting embedded recruitment trials, for which a checklist based on the Consolidated Standards for Reporting Trials (CONSORT) statement 2010 was developed and several examples were listed. [30] Unlike the existing literature, this review has a focus on recruitment via different channels used as strategies described in the included studies, partly because of the inadequacy of the evidence available for mental health trials. It provides some evidence from a different perspective and makes suggestions regarding possible future research in this area. For instance, SWATs may be designed to compare the efficacy of recruitment by research staff with recruitment by clinical staff. Promoting the guidelines by Madurasinghe et al. will help to improve the quality of reporting for these methodological trials. Furthermore, it is also worth investigating the performance of different recruitment strategies with respect to other aspects of the trial, such as the population characteristics or adherence to the trial intervention, as these features also can determine a trial's precision and efficiency. Some strategies may recruit a biased sample. For instance, using web-based adverts as a recruitment method in mental health trials may inadvertently recruit the "worried well" or those who do not sufficiently resemble real-world patients.
This study has the following limitations. Firstly, we only identified 3 randomised comparative studies of recruitment and two of retention. The rest compared different strategies without randomisation and this may diminish the internal validity of their findings. Secondly, out of 13 identified studies, the majority were in depression-related illnesses. The limited number of studies involving people with diagnoses of severe mental illnesses such as bipolar disorders or schizophrenia, reduces the generalisability of the review. It highlights the need for more research in this area, since there are many challenges to recruitment within this group of people. Moreover, there were no eligible RCTs aimed at improving retention within randomised controlled trials. We included 2 studies which focus on improving postal response rates in follow-up, despite the fact they were not set within a randomised clinical trial. However, since they used a randomised design to assess methods to enhance response rates, we believe they contribute useful information, although clearly more studies are needed to address retention issues in randomised trials and in studies that use face to face assessments rather than postal questionnaires. Lastly, lack of reported information on costs in many of the included studies means there is considerable uncertainty in our findings on cost-effectiveness.

Conclusion
This review discusses the different strategies to improve recruitment and retention in mental health clinical trials. The recruitment studies included showed substantial variation in strategies, clinical settings, mental health conditions and study design. It is difficult to assess the overall efficacy of any particular recruitment strategy as some strategies that worked well for a particular population may not work as well for others. Paying attention to the accessibility of information and consent materials (optimisation) may help improve recruitment. Recruitment by clinical staff and non-web-based adverts showed some efficiency and success in certain circumstances. Pre-notification, abridged questionnaires and financial incentives have small positive effects on retention rates in postal surveys. The limited number of eligible studies identified suggests that more research in this area is needed given its important implications.