Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Acceptability of an open-label wait-listed trial design: Experiences from the PROUD PrEP study

  • Mitzy Gafos ,

    m.gafos@ucl.ac.uk

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Elizabeth Brodnicki,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Monica Desai,

    Affiliation HIV/STI Department, Public Health England, London, United Kingdom

  • Sheena McCormack,

    Affiliations Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom, Directorate of HIV and GU medicine, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom

  • Will Nutland,

    Affiliation Sigma research, London School of Hygiene & Tropical Medicine, London, United Kingdom

  • Sonali Wayal,

    Affiliation Centre for Sexual Health and HIV Research, Mortimer Market Centre, London, United Kingdom

  • Ellen White,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Gemma Wood,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Tristan Barber,

    Affiliation Directorate of HIV and GU medicine, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom

  • Gill Bell,

    Affiliation Sheffield Health, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, United Kingdom

  • Amanda Clarke,

    Affiliation Claude Nichol Centre, Royal Sussex County Hospital, Brighton, United Kingdom

  • David Dolling,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • David Dunn,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Julie Fox,

    Affiliation Guy’s and St Thomas’ NHS Foundation Trust, London, United Kingdom

  • Lewis Haddow,

    Affiliations Centre for Sexual Health and HIV Research, Mortimer Market Centre, London, United Kingdom, Research Department of Infection and Population Health, University College London, London, United Kingdom

  • Charles Lacey,

    Affiliation York Hospitals NHS Foundation Trust, York, United Kingdom

  • Anthony Nardone,

    Affiliation HIV/STI Department, Public Health England, London, United Kingdom

  • Killian Quinn,

    Affiliation King’s College Hospital NHS Foundation Trust, London, United Kingdom

  • Caroline Rae,

    Affiliation Directorate of HIV and GU medicine, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom

  • Iain Reeves,

    Affiliation Department of Sexual Health, Homerton University Hospital NHS Foundation Trust, London, United Kingdom

  • Michael Rayment,

    Affiliation Directorate of HIV and GU medicine, Chelsea and Westminster Hospital NHS Foundation Trust, London, United Kingdom

  • David White,

    Affiliation Heart of England NHS Foundation Trust, Birmingham, United Kingdom

  • Vanessa Apea,

    Affiliation Ambrose King Centre, Barts Health NHS Trust, London, United Kingdom

  • Wilbert Ayap,

    Affiliation St Mary’s Hospital, Imperial College NHS Foundation Trust, London, United Kingdom

  • Claire Dewsnap,

    Affiliation Sheffield Health, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, United Kingdom

  • Yolanda Collaco-Moraes,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Gabriel Schembri,

    Affiliation Manchester Centre for Sexual Health, Central Manchester University Hospitals NHS Foundation Trust, Manchester, United Kingdom

  • Yinka Sowunmi,

    Affiliation Medical Research Council Clinical Trials Unit, Institute of Clinical Trials & Methodology, University College London, London, United Kingdom

  • Rob Horne,

    Affiliation School of Pharmacy, University College London, London, United Kingdom

  •  [ ... ],
  • on behalf of the PROUD Study Team

    A complete list of the PROUD Study Team can be found in the Acknowledgments.

  • [ view all ]
  • [ view less ]

Acceptability of an open-label wait-listed trial design: Experiences from the PROUD PrEP study

  • Mitzy Gafos, 
  • Elizabeth Brodnicki, 
  • Monica Desai, 
  • Sheena McCormack, 
  • Will Nutland, 
  • Sonali Wayal, 
  • Ellen White, 
  • Gemma Wood, 
  • Tristan Barber, 
  • Gill Bell
PLOS
x

Abstract

Background

PROUD participants were randomly assigned to receive pre-exposure prophylaxis (PrEP) immediately or after a deferred period of one-year. We report on the acceptability of this open-label wait-listed trial design.

Methods

Participants completed an acceptability questionnaire, which included categorical study acceptability data and free-text data on most and least liked aspects of the study. We also conducted in-depth interviews (IDI) with a purposely selected sub-sample of participants.

Results

Acceptability questionnaires were completed by 76% (415/544) of participants. After controlling for age, immediate-group participants were almost twice as likely as deferred-group participants to complete the questionnaire (AOR:1.86;95%CI:1.24,2.81). In quantitative data, the majority of participants in both groups found the wait-listed design acceptable when measured by satisfaction of joining the study, intention to remain in the study, and interest in joining a subsequent study. However, three-quarters thought that the chance of being in the deferred-group might put other volunteers off joining the study. In free-text responses, data collection tools were the most frequently reported least liked aspect of the study. A fifth of deferred participants reported ‘being deferred’ as the thing they least liked about the study. However, more deferred participants disliked the data collection tools than the fact that they had to wait a year to access PrEP. Participants in the IDIs had a good understanding of the rationale for the open-label wait-listed study design. Most accepted the design but acknowledged they were, or would have been, disappointed to be randomised to the deferred group. Five of the 25 participants interviewed reported some objection to the wait-listed design.

Conclusion

The quantitative and qualitative findings suggest that in an environment where PrEP was not available, the rationale for the wait-listed trial design was well understood and generally acceptable to most participants in this study.

Introduction

Randomised placebo-controlled trials are considered the ‘gold standard’ when evaluating an investigational product for prevention. Randomised placebo-controlled trials have provided a breakthrough in HIV prevention by demonstrating the benefit of oral and topical pre-exposure prophylaxis (PrEP) [17]. However, results of a few placebo-controlled trials have been hindered by low levels of adherence. This has resulted in trial results either underestimating the level of protection offered by PrEP [1], or showing no benefit despite evaluating efficacious products [810]. Poor adherence in placebo-controlled trials initially undermined confidence in oral products for women and has potentially jeopardised the development of a topical product considered biologically efficacious against both HIV and herpes simplex virus [4, 11].

Measuring product acceptability has been central to PrEP trials, given participants are unlikely to use a product they consider unacceptable [12]. However, it is important to distinguish between product acceptability and product adherence, as it does not necessarily follow that participants are more likely to use a product they consider acceptable [13]. The Necessity-Concerns-Framework (NCF) predicts adherence by measuring an individual’s belief about the necessity of taking medication balanced against their concerns about taking it [14, 15]. This balance of necessity and concern is skewed against adherence in placebo-controlled trials where there is no benefit of taking a placebo, unclear benefit of taking the investigational product, and a heightened focus on potential side effects [16]. Certainly, higher risk participants have been more adherent in open label extension phases of trials than during the placebo-controlled phase [17, 18].

Although randomised placebo-controlled trials are the ‘gold standard’, when there is strong evidence of efficacy, there is growing interest in trial designs that promote instead of hinder adherence [19]. One alternative is the randomised wait-listed control trial design whereby participants either receive the intervention immediately or after a deferred period. Wait-listed trial designs have been used extensively in health research, for example in the field of mental health [20], cognitive function [21], nutrition [22], pain management [23] and organ transplantation [24]. However, open-label wait-listed trial designs have rarely been used in biomedical HIV prevention trials [25].

In 2011, a PrEP e-group, which included community members, healthcare providers, policy-makers and researchers, identified the need to understand the potential public-health benefit and cost effectiveness of PrEP for the National Health Service in England [26]. They agreed that a randomised controlled trial (RCT) was necessary but that a placebo-controlled trial would not be able to measure the net-effect of PrEP, taking account of efficacy, adherence, and any change in sexual behaviour as a result of using PrEP. Consequently, the PROUD study was designed as a randomised open-label wait-listed trial. However, given PrEP was approved for use in the United States of America, there were questions about whether the deferred start would be acceptable to participants. In this paper, we present findings from quantitative and qualitative data on the acceptability of the wait-listed trial design in the context of the acceptability of other study procedures in the PROUD study [27, 28].

Methods

PROUD was a pragmatic open-label wait-listed RCT designed to evaluate the effectiveness of daily oral Truvada (tenofovir disoproxil fumarate and emtricitabine) as HIV pre-exposure prophylaxis (PrEP). The study was only called PROUD, although this was originally derived from the longer title of: PRe-exposure Option for reducing HIV in the UK: an open-label randomisation to immediate or Deferred daily Truvada for HIV negative gay men. PROUD was designed as a pilot study to assess the acceptability of the study design and procedures in advance of a larger scale clinical trial. However, due to higher than anticipated HIV incidence and effectiveness of PrEP, the clinical trial objectives were met during the pilot [29].

Recruitment took place from November 2012 to April 2014 at 13 sexual health clinics in England, eight in London and five outside of London. The study was advertised at the participating sexual health clinics, during community outreach activities, online and via sexual networking applications. Eligibility criteria included being gay or a man having sex with men, being a trans women having sex with men, and being assigned a male gender at birth. Eligible participants also had to be HIV negative, aged 18 or above, to have reported condomless anal sex in the last 90 days and expected to have it again in the next 90 days, and to have previously attended the enrolling clinic. Before enrolment, the study was explained using the participant information sheet (PIS) and participants provided written informed consent. The UK Community Advisory Board (http://www.ukcab.net/) reviewed the protocol, which included recommended changes to the eligibility criteria which supported recruitment, and substantially revised the PIS and informed consent forms, which included improving the readability score from 35 (academic level) to more than 70 (general population level). At enrolment, participants were randomly assigned (1:1) to receive daily Truvada either immediately or wait-listed to receive Truvada after a deferred period of one year. Regular sexual partners were encouraged to enrol together and both partners allocated to the same group to minimise the possibility of drug sharing. The study design, baseline characteristics of the cohort and results have been presented elsewhere [27, 28].

After enrolment, participants were asked to attend the clinic quarterly for HIV and STI screening, with an extra visit one month after starting PrEP to check safety and tolerability (at month 1 for the immediate group and month 13 for the deferred group). Notifications of HIV sero-conversion for participants who did not return for follow-up visits were sourced from electronic clinic records in other PROUD clinics and Public Health England (PHE) database. Participants were asked to self-complete a baseline questionnaire providing demographic data at enrolment (on paper), monthly sexual behaviour and adherence questionnaires (on paper or online), daily sexual behaviour and adherence diaries (on paper or online), and an acceptability questionnaire at the 12 month follow-up visit (on paper). The acceptability questionnaire was developed by the social science advisory group and is available as supporting information file S1 Questionnaire. Paper questionnaires were placed in an envelope, sealed and posted to the central data management team for data entry in MACRO database.

We used the Pearson chi-squared test, or the Fisher exact test where numbers were five or less in any group, to compare responses to the acceptability questions based on trial arm allocation (immediate or deferred), age group (34 and below or 35 and above, split at the median age of the cohort), educational level (degree or below), ethnicity (white/Irish or Black, Asian and minority ethnicity [BAME]), clinic of enrolment (inside or outside London), employment status (full/part-time employed or not) and place of birth (UK or not). Quantitative data were analysed using Stata 14.

In addition to categorical questions about acceptability, we asked participants to write what they most and least liked about the study and coded the free-text responses in Excel using emergent themes. The first author coded the free-text and two co-authors (EB, WN) checked the coding for consistency.

From February 2014 to January 2015, we purposely sampled 25 study participants to take part in semi-structured in-depth interviews (IDIs) which included a discussion topic on study acceptability. Participants were selected based on trial arm allocation (immediate or deferred), changes in their self-reported risk behaviour since enrolment (increased risk or same/decreased risk), and PrEP adherence among participants in the immediate group (high or low). We aimed to interview an equal number of participants in each of the six groups. During the interviews, we asked participants about the acceptability of the study design and procedures. Researchers who were independent of the study clinic team conducted the IDIs. Participants provided written informed consent and the IDIs were conducted in English and were audio-recorded. The recordings were transcribed, coded and analysed using framework analysis in NVivo 10 [30].

The PROUD study protocol was approved by London Bridge Research Ethics Committee, the Medicines and Healthcare products Regulatory Agency and each of the 12 participating Hospital Trusts (see list in acknowledgments). The trial is registered with ISRCTN (number ISRCTN94465371) and ClinicalTrials.gov (NCT02065986). The study protocol, including participant information sheet (PIS) and informed consent form (ICF), and the in-depth interview PIS, ICF and interview guide, are available on the study website.

Results

In order to contextualise the acceptability of the study design, we present this in the context of the overall acceptability of the study. As such, we present the characteristics of the study cohort, acceptability of the study design, acceptability of the study visits, and acceptability of the study procedures. Within each section we present the quantitative data from the acceptability questionnaire, the free text responses from the acceptability questionnaire which are presented as proportions of responses not participants, and qualitative data from the in-depth interviews. To reduce the size of the tables, we make the data by employment status and date of birth available in supporting information file S1 Table as we observed no statistically significant differences in these analyses.

Characteristics of the study cohort

Quantitative data.

Of 544 (immediate [I] = 275, deferred [D] = 269) participants enrolled, 415 (76%) completed an acceptability questionnaire. One hundred and twenty nine participants (I = 50, D = 79) did not complete a questionnaire. Of these 21 (I = 5, D = 16) did not return to the PROUD study clinic after randomisation, 60 (I = 25, D = 35) did not have a visit at or after the scheduled 12 month visit, and 48 (I = 20, D = 28) had a visit at or after the scheduled 12 month visit but did not complete the questionnaire.

A higher proportion of participants in the immediate (82%) compared to deferred (71%) randomisation groups completed the acceptability questionnaire (p = 0.002) (Table 1). Participants who completed the acceptability questionnaire were older than participants who did not, at a median age of 36 (IQR 30–43) compared to 32 (IQR 28–40, p = 0.002). When controlling for age, participants in the immediate group were almost twice as likely to have completed the questionnaire than participants in the deferred group (AOR: 1.86, 95% CI 1.24, 2.81, p = 0.003). There was no evidence of differences between participants who completed the questionnaire based on education level, ethnicity, clinic of enrolment (Table 1), employment status or place of birth (available as supporting information file S1 Table).

thumbnail
Table 1. Comparison of participants who complete an acceptability questionnaire to those who didn't.

https://doi.org/10.1371/journal.pone.0175596.t001

The median time from enrolment to completion of the acceptability questionnaire was 12.13 months (IQR 11.80, 12.80). Of 190 participants in the deferred group who completed the acceptability questionnaire, 125 completed it when first prescribed PrEP but before PrEP initiation, 45 after starting PrEP, 11 before starting, and seven completed it but never started PrEP (missing date of completion for two participants).

Free text data.

Of the 415 (I = 225, D = 190) participants who completed the acceptability questionnaire, 369 (I = 204, D = 165) reported 589 (I = 323, D = 266) free text items that they most liked about the study. Forty-one percent (239/589) of the items reported related to the benefits of PrEP and HIV prevention more broadly, this equates to 45% (146/323) of all issues raised by the immediate group and 35% (93/266) of all issues raised by participants in the deferred group. Fifty-nine percent (350/589) of the items reported related to study acceptability. The key points raised about the acceptability of the study are presented in order of reporting in Table 2 and are discussed below. Prioritisation of issues hardly differed between participants in the immediate and deferred groups, and the narratives within the issues were very similar.

Of the 415 participants who completed the questionnaire, 315 (I = 179, D = 136) provided 360 (I = 199, D = 161) responses when asked what they least liked about the study. Fourteen percent (51/360) of the items reported related to the use of PrEP (I = 32, D = 19). Eighty six percent (309/360) of the items reported related to study acceptability and are presented in Table 3 and discussed below. In this analysis, 25% of participant responses reported there was nothing they disliked about the study, which was almost twice as high in the immediate group (32%) compared to the deferred group (17%).

Qualitative data.

We conducted 25 IDIs with 11 participants in the immediate and 14 in the deferred group. Categorisation based on the risk and adherence matrix used for purposeful sampling is shown in the figure in S1 Fig. The participants were enrolled across seven clinics, two of which were outside of London. Participants were a median age of 37 (range 25 to 55 years), 23 were in employment and 16 were educated to degree level. Twenty-four male (23 gay and one bisexual) and one trans female participated in IDIs. At the time of interview, participants had been enrolled in the study for a median of 12 months (range five to 20 months).

Acceptability of study design

Quantitative data.

Overall, 97% of participants reported being ‘glad they joined the study’, with only one person who was randomised to the immediate group disagreeing with this statement (Table 4). The vast majority (96%) expected to remain enrolled for the duration of the study. Eighty-seven percent were interested in joining another PrEP study after PROUD if available, however this differed by trial arm with 92% in the immediate group and 81% in the deferred group interested in joining another PrEP study (p = 0.002). Despite these high levels of study acceptability, 74% thought that the chance of being in the deferred group might put other volunteers off joining the study.

The majority agreed that the participant information sheet clearly explained the study (98%), although this differed by educational level with 10 participants educated to degree level stating the information sheet did not clearly explain the study or being neutral on this point, compared to none with less than degree level education (p = 0.021). There are no additional data to indicate why they did not think the information sheet was clear. There were no other significant differences based on other demographic factors.

Free text data.

In the free text data, 31% (110/350) of participant responses noted that being able to contribute to HIV prevention research was what they most liked about participating in the study, as this quote illustrates:

“It feels like I'm helping the gay community to fight and eradicate HIV” (deferred).

While no one in the immediate group cited the wait-listed design when asked what they liked least about the study, 20% (27/136) of participants in the deferred group noted ‘being deferred’ as the thing they liked least. This was the second most common reported item disliked after the data collection tools, even though it only accounted for 19% (27/142) of all least liked issues raised by deferred participants:

“I was slightly disappointed to be put in the deferred treatment group but understand the necessity of this” (deferred).

“Having to wait a year to start PrEP during the highest risk period of my life” (deferred).

Qualitative data.

Participants in the IDIs had a good understanding of the rationale for the open-label wait-listed study design in terms of assessing effectiveness, adherence, changes in risk behaviour, acquisition of other STIs, and the cost-effectiveness of an NHS funded PrEP programme.

Seventeen participants (I = 6, D = 11) discussed the acceptability of the wait-listed study design in some detail when asked about it. Most accepted the design with participants in the immediate group stating they would have been disappointed to be randomised to the deferred group but would have accepted it. Similarly, participants in the deferred group stated they were disappointed to be randomised to the deferred group but accepted it. The general sense was that volunteers knew what they were signing up to:

“I was slightly disappointed because I knew I was at risk by my behaviour at the time, I was hoping to get on it (PrEP) for the full 2 years, but obviously it was always made clear right from the start that there was a deferred group and there was an immediate group, so I couldnt complain. It soon came around anyway and it was in time, so (laughter)” (deferred participant who started PrEP before the IDI).

The issue raised in this quote about hoping to access PrEP ‘in time’ before contracting HIV emerged in a hand-full of interviews with deferred participants:

“If I get infected in the first year I would be pretty angry I think…. we know that this exists and we don’t (have it), so why not… I can almost predict that I will run into something (test positive) at the last test…. then I will be quite upset”.

A participant who sero-converted during the deferred period described his experience as follows:

“Knowing how effective Truvada is I think everyone would rather be in the immediate rather than the control, because obviously your chances are better, but it is a 50/50 chance a roll of the dice, I wasn’t particularly put off or anything by being in the deferred”.

Only one participant, who had used PrEP before joining the study, said he had not understood there was a chance of being deferred until he was deferred, but accepted the randomisation:

“The 12 months actually gave me a chance to think about it (PrEP) again, rather than just taking a tablet, but actually do I need it, do I want it. So actually I reassessed my situation, whether I should use PrEP or not use PrEP and how I sexually behave whilst using PrEP or not using PrEP”.

Only five (I = 1, D = 4) participants specifically objected to the wait-listed design. A participant in the immediate group said he probably would not have returned to the clinic if he had been randomised to the deferred group. A participant in the deferred group described how he had planned to accept being randomised to either group, but after spending many hours in the clinic due to delays, was “incandescent with rage” when finally randomised to the deferred group. Instead of returning to the clinic, he enrolled a second time at a different clinic and was randomised to the immediate group (he was one of two co-enrolments excluded from the final analyses). The other three deferred participants were not convinced of the need for the deferred group describing it as a ‘wasted year’ and ‘unfair’. In addition, two other participants interviewed, reported use of PrEP accessed outside of the trial during their deferred period in the trial.

Acceptability of study visits

Quantitative data.

The majority of participants didn’t mind visiting the clinics quarterly (89%), liked having regular HIV (93%) and STI (87%) screening, and felt there were able to access as much support as they needed to reduce the risk of HIV and other STIs (90%) (Table 5). However, 8% of older participants compared to only 4% of younger participants did not like regular STI screening (p = 0.027). Although a similar proportion of BAME and white participants felt that they were able to access as much support as they needed to reduce their risk behaviour (92% v 90%), there were differences in their reporting of a neutral (4% v 10%) or negative response (4% v 1%), (p = 0.029). However, these differences are difficult to interpret with such small numbers and differential representation of BAME participants across clinics.

Pharmacokinetic samples were collected from 59 participants, but all participants were asked about the hypothetical acceptability of drug level monitoring to assess adherence in a future trial. In a series of three questions, 96% of participants thought that drug level monitoring was a good idea, 86% reported they would like to know their drug levels when on PrEP, and only 16% felt that there was no need to conduct drug level testing to assess adherence. These perspectives did not differ by trial allocation or baseline demographics.

Free text data.

In the free text data, a fifth (20%) of participant responses related to the benefits of regular HIV and STI testing in terms of helping them manage their sexual health by providing a structure for regular testing. Twenty-eight percent of participant responses reported the service they received from the study staff as the aspect of the study they most liked. The following descriptors were used in relation to the staff (in alphabetical order):

‘Accessible, amazing, approachable, caring, courteous, efficient, friendly, great, helpful, honest, having integrity, incredible, informative, kind, non-judgement, openly talk about sex, patient, pleasant, polite, professional, reassuring, supportive, trustworthy and understanding’.

Participants particularly appreciated the consistency of care by being able to see the same health professionals regularly and build a relationship with the clinical team, as shown in this response:

I most liked.. “the offer of seeing one designated person every 3 months for a check-up rather than being passed from person to person in a clinic” (deferred)

Other points that participants most liked included the ease of attending study visits and adhering to study procedures (6%), access to support and referral systems (6%), access to additional information about the study, study progress and HIV prevention more generally (5%).

Least liked aspects of the study visits included difficulties of getting appointments and travel time to the clinics (11%), having blood taken and fear of needles (7%), specific procedures (recruitment, enrolment, aims unclear, partner enrolment, recruitment closing) (2%), lack of feedback in terms of an online forum or support phone line (2%), poor treatment by staff (1%), and lack of support (1%).

Qualitative data.

The qualitative data reinforces findings from the quantitative data with the majority of participants reporting that they liked the regular clinic visits and HIV and STI screening. However, three participants did comment on the “minor inconvenience” of quarterly visits, and the difficulty of attending clinics especially during working hours. A few participants would have preferred pre-booked quarterly appointments with visit reminders. Another participant liked the convenience of being able to go to any clinic, not just the study clinic, for HIV and STI screening. A participant in the deferred group had not attended the clinic for six months and felt that, without the motivation of PrEP to attend, it was easy to lose touch with the clinic. He thought that additional email contact with participants during the deferred period would ensure better retention.

As in the free text findings, participants in the IDIs praised the clinic staff without being prompted to talk about them, especially the fact that they were welcoming, respectful and non-judgemental. They also highlighted the benefit of seeing the same staff members regularly in the clinic, as this quote highlights:

“It is really important to me… I don’t know him very well, but I have been able to disclose a few things to him… It means it is not caught up in my head, I am not denying it, I am not avoiding it, I am not parking it in the ether, and I wouldn’t be able to do that if it was a random team member …even though I don’t know him very well the relational aspect is really important”.

Acceptability of study data collection tools

Quantitative data.

Acceptability of the data collection tools differed by individual tool with 82% of participants reporting they didn’t mind completing the monthly sexual behaviour questionnaires, 56% reporting they didn’t mind completing the daily sexual behaviour and adherence diary, and 56% reporting liking completing the questionnaires online (Table 6). However, there was evidence of a difference by trial arm in the proportion of participants who agreed (22% in immediate, 16% in deferred) or were neutral or uncertain (20% in immediate, 30% in deferred) in response to the statement that they disliked completing the daily diary (p = 0.036).

Younger participants disliked the sexual behaviour diary more than older participants (23% v 16%, p = 0.019) and older participants liked completing the questionnaires online more than younger participants (62% v 48%, p = 0.007). Similarly, participants enrolled outside of London liked completing the questionnaires online more than participants enrolled in London (65% v 52%, p = 0.014).

Fifty-eight percent of participants reported they would have preferred to answer the sexual behaviour questions on a mobile device (i.e. smart phone or tablet). This was in comparison to being able to answer the questions on a paper questionnaire or via the on-line database. As above, this differed by age with 73% of younger compared to only 47% of older participants being interested in this option (p<0.001).

In terms of self-completed questionnaires, 88% felt they could report sexual activity honestly, although 47% found it difficult to remember sexual activity over the last 30 days and 28% thought the questionnaires made it difficult to accurately report sexual activity. This differed by age with 54% of younger compared to 41% of older participants finding it difficult to remember their sexual behaviour (p = 0.026).

Free text data.

In the free text responses, the data collection tools were the most frequently reported least liked aspect of the study by participants in both groups. A total of 132 participants (I = 73, D = 59) mentioned at least one data related item that they disliked: 15% of responses related to the monthly self-completed questionnaires, 12% to the daily diary, 9% to data collection generally, and 7% specifically about the online participant website where participants could complete the questionnaires and diary.

Participants said that there were too many forms to complete with the monthly sexual behaviour and adherence questionnaire and daily diary and that it was difficult to remember to complete them. They described the questionnaires and diary as being cumbersome, fiddly, a chore to complete and time consuming. In addition, the questionnaires were described as being too long, frustrating, repetitive and unclear, and the diary as being awkward, laborious, not user friendly, a pain, and tedious, as this quote demonstrates:

“The online diary site is *SUCH* a pain! Having to click into each day separately is very laborious” (deferred, emphasis by participant).

Participant suggestions included having email reminders to complete the questionnaire and diary, to have longer than the 2 week cut off point to complete the last months questionnaire, to be able to view the diary in order to complete the monthly questionnaire, and to be able to block edit days in the diary, repeat data from the previous day or merely complete the diary when you had missed a tablet or had sex.

A number of participants complained that it was difficult to recall sexual behaviour when completing the questionnaire and diary. A few also complained that they were not able to report their behaviour as accurately as they would have liked as the questionnaire and diary did not allow them to report behaviour that was not requested (i.e. group sex, ejaculation), or to elaborate on reasons for specific behaviours, for example:

“Diary doesn't cover all issues, what about non anal sex partners? More than 1 partner a day? Other reasons for not taking pills other than forgot e.g. illness or travel” (immediate)

A number of participants described the online website as difficult to access, unreliable and ‘clunky’, with some arguing that they would have preferred a mobile application. One person raised a concern about data protection issues with the online website.

While 1% of least liked responses related to recalling numbers of sexual partners, 3% of most liked responses related to participants liking the need to recall and record their sexual behaviour as it made them review their behaviour and consider changes:

“It made me feel more aware of my sexual health and has made me less likely to be unsafe than one year ago” (immediate).

Qualitative data.

In the qualitative interviews, there was almost an even split between people who liked and didn’t like completing the questionnaires and diaries. Much of the negative feedback received in the free text responses was repeated in the IDIs including the fact that the data collection tools were duplicative, questions were ambiguous, and the diaries were difficult to complete especially in terms of having to enter data about not having sex and not taking PrEP each day. In addition, participants reported that they rarely completed the diary daily, more likely weekly or monthly either from memory or from personal diaries or even spreadsheets.

As in the free text responses, some participants were frustrated that the questionnaire and diary were restrictive, not allowing participants to add comments or explain entries. In the IDIs, it was clear that this mainly related to participants wanting an opportunity to explain their rationale for risk taking or actions that may be perceived by the categorical approach of data collection to be more risky than the participant felt they actually were.

Out of 12 participants who talked about the accuracy of their self-reported data, half thought they were very accurate and half didn’t, mainly due to forgetting to complete the diary regularly, difficulty recalling their behaviour accurately, and difficulty being as accurate as they wanted to be given the categorical nature of the data collection tools.

Recommendations on how to improve data collection included reminders for both visits and data completion, completing the data on a mobile application, having questionnaires administered instead of self-completed which one participant did not think would increase social desirability bias, and more opportunities for participants to describe their behaviour instead of ticking categorical options.

Discussion

This is the first study to assess the acceptability of using an open-label wait-listed design to evaluate the effectiveness of a new biomedical HIV prevention technology. The first concern when designing the trial was the possibility that randomisation to the deferred group would discourage volunteers from enrolling. Although we enrolled to target, accrual was slower than expected, partly due to limited PrEP awareness [28]. In this paper, we show that three-quarters of participants thought the wait-listed design would discourage other volunteers from joining a similar study. This evidence suggests that the wait-listed design probably contributed to the slower than anticipated rate of recruitment. This is an important consideration in planning future wait-listed trials.

The second concern when designing the trial was that retention would be lower in the deferred-group, potentially compromising the validity of the trial result. We reported previously that participants in the immediate-group attended clinic more regularly than deferred-group participants to collect their PrEP prescriptions, with a mean of 4.2 compared to 3.6 visits in the first year [29]. Without the facility to identify new HIV sero-conversions via the PHE database, this could have affected the outcome of the trial. Indeed, a fifth of deferred participants reported the wait-listed design as an element they disliked about the study. Given completion of the acceptability questionnaire was significantly lower among participants in the deferred than the immediate group, this is likely to underestimate the extent to which participants disliked the wait-listed design. However, it is interesting to note that in free-text data, deferred participants disliked the data collection tools more than the fact that they had to wait a year to access PrEP. In fact, overall, participants had more reservations about the data collection tools than any other aspect of the study design, visits or procedures. The quantitative and qualitative findings suggest that in an environment where PrEP was not available, the rationale for the wait-listed design was well understood and generally acceptable to most participants.

Our findings on the acceptability of study visits provide some lessons for PrEP implementation, as well as future prevention research. In England, MSM are advised to have HIV and STI screening at least annually, and every 3 months if having condomless sex with new or casual partners [31]. At enrolment, participants reported screening for STIs a median of 3 (IQR 2–4) times in the previous 12-months [28] During the study, PROUD participants clearly found that in the context of PrEP provision, quarterly clinic visits and the routine of regular HIV and STI screening were acceptable. These findings suggest that among a cohort of MSM who are engaged with sexual-health services and perceive themselves to be at risk of HIV, regular clinic visits are likely to be acceptable in future PrEP programmes. However, we still know very little about the optimal ways to deliver PrEP to individuals at risk of HIV who are less engaged with sexual-health services or do not perceive their risk to be sufficient to access PrEP. Additionally, in a sub-sample of the study population we used drug level testing to confirm participant’s adherence to daily dosing. Among all participants, the idea of drug level testing was perceived as an acceptable procedure. While this finding is useful for future clinical trials, this type of adherence assessment is unlikely to be deemed necessary in a clinical setting in the UK.

Another lesson for implementation is the importance of seeing the same health professionals regularly in terms of building rapport with the staff. While participants felt they could access as much support as they needed to reduce their risk of HIV and STIs, we did not measure the type of support they wanted in the context of PrEP use. In the baseline cohort, 9.5% of participants reported moderate to severe depression and 44% reported the use of recreational drugs commonly associated with chemsex [28, 32]. In planning a PrEP implementation programme, we need to explore the types of support services that PrEP users require and the best ways to offer these services.

PROUD was originally designed as a pilot study and as such, feedback on the shortcomings of the data collection tools was valuable in preparation for a larger-scale RCT. We asked participants to self-complete sexual-behaviour and adherence questionnaires to reduce the impact of social desirability bias and reduce the clinic workload. Four-fifths of participants didn’t mind completing the questionnaires and over half didn’t mind completing the daily-diary. This is despite feedback that there were too many questionnaires, questions were often unclear, and the online data entry site was difficult to use. This suggests that while participants were critical of the data collection tools, the majority did not object to providing self-completed data and felt they were able to do so honestly. Of note, almost half of participants found it difficult to recall their sexual activity in the last 30 days. The key lessons are that the data collection tools need to be streamlined, for example by using clearer categorizations, cover shorter time-periods or be supported with aides-memoire, and available on multiple devices, including mobile apps, to meet a variety of individual preferences.

Participants were critical about the design of the questionnaires with over a quarter stating they made it difficult to accurately report sexual behaviour, an issue echoed in the free-text responses. While there is a lot of room for improvement of the questionnaires, this issue also relates to the challenge of collecting data on sexual risk behaviour in the current HIV landscape. We know from previous analysis that for participants in this study, it was clear that a ‘risk’ depended not only on condom use, but also PrEP use, a partners HIV-status and viral-load, whether the participant was top or bottom, and whether ejaculate was involved [33]. In the era of PrEP and treatment as prevention (TasP), we need to reconsider the questions we use to measure risk behaviour, taking account of both the need to streamline the data collection tools and realistically capture ‘risk’ from the perspective of participants. As noted by other authors, we need to adapt measures of risk to take account of the expanded HIV prevention options which include the use of ARVs by both positive and negative individuals, and move away from simplified notions of condomless sex being ‘unprotected’, ‘unsafe’ or a ‘risk’ factor for HIV acquisition [34, 35].

Although we were specifically interested in assessing acceptability of the trial design, it was useful to contextualise participant’s perceptions within the broader context of acceptability of study procedures more generally. This analysis benefits from that fact that we assessed acceptability of the study across participants in the immediate and deferred groups. However, one limitation is that a quarter of participants did not complete the acceptability questionnaire and they were almost twice as likely to be in the deferred group. We attempted to arrange in-depth interviews with participants who had stopped attending the clinic or withdrawn from the study, but were either unable to contact them or they were unwilling to attend an in-depth interview. Another limitation is that we relied on the self-completed sexual behaviour data to purposely select participants for the in-depth interviews, and therefore did not sample participants who did not complete any questionnaires. A further limitation is that although trans women were eligible to join the study, our recruitment strategy was not sufficiently trans inclusive and therefore only three trans women joined the study. As such, these findings only really reflect the views of the cis men who joined the study. Our findings highlight the benefit of using mixed method research approaches to assess the acceptability of study designs, procedures and data collection processes in preparation for large-scale clinical trials.

Our findings demonstrate that for PROUD, the open-label wait-listed RCT was an acceptable and feasible trial design. It was the optimal design to estimate the net-effect of PrEP in terms of biological efficacy, product adherence and risk compensation. It also enabled a comparison of behaviour change and rates of STIs between the groups, as well as HIV incidence. Without the deferred control-group, the study results would have been uninterpretable as HIV incidence in the intervention group was comparable to the expected HIV incidence among MSM attending sexual health clinics and yet incidence in the control-group was seven-times higher than expected [29]. The open-label wait-listed RCT design is not a panacea to the challenges that some placebo-controlled biomedical prevention trials have faced in terms of achieving sufficient adherence. However, in this case, an open-label wait-listed RCT was the optimal design to address the research question of interest in PROUD and provided the most valid measure of public health benefit.

Given the challenges of achieving sufficient adherence in recent biomedical HIV prevention trials, evidence from open-label wait-listed RCTs should be considered to support public health interventions, as in PROUD, as well as in support of licensing applications of new drugs where appropriate. The World Health Organisation now recommends PrEP for populations with an HIV incidence of three per 100 person-years, although access remains extremely limited in most countries [36]. Future biomedical HIV prevention trials will need to consider including oral PrEP as either the control group or an additional standard of care. While there may still be circumstances whereby the open-label wait-listed RCT remains ethically acceptable for the evaluation of new biomedical HIV prevention technologies, this trial design should be considered more broadly when the presence of a placebo could impede adherence sufficiently to undermine the power of an RCT.

Supporting information

S1 Fig. Purposive selection of participants for in-depth interviews based on trial arm allocation, risk behaviour and adherence.

https://doi.org/10.1371/journal.pone.0175596.s001

(PDF)

S1 Table. Employment and place of birth data tables.

https://doi.org/10.1371/journal.pone.0175596.s003

(DOCX)

Acknowledgments

We would like to thank all the volunteers who joined the PROUD study for their commitment and support.

We would also like to thank all the staff who worked on the study at the participating Hospital Trusts (in alphabetical order):

  1. Barts Health NHS Trust, London, UK; Vanessa Apea, Drew Clark, Paul Davis, James Hand, Anna Hartley, Machel Hunt, Helena Miras, Rebecca Neale, Jackie O’Connell, Thomas Pasvol, Margaret Portman, Liat Sarner, John Saunders, Louise Terry, Angelina Twumasi, Salina Tsui, Dayan Vijeratnam, Ryan Whyte, Andy Williams
  2. Brighton & Sussex University Hospitals NHS Trust, Brighton, UK; Sian Gately, Gerry Gilleran, Jill Lyons, Chris McCormack, Katy Moore, Cathy Stretton, Stephen Taylor, David White
  3. Central Manchester University Hospitals NHS Foundation Trust, Manchester, UK; James Boateng, Brynn Chappell, Susanna Currie, Carolyn Davies, Dornubari Lebari, Matthew Phillips, Gabriel Schembri, Lisa Southon, Sarah Thorpe, Anna Vas, Chris Ward, Claire Warren, Stephanie Yau
  4. Central & Northwest London NHS Foundation Trust, London, UK; Alejandro Arenas-Pinto, Asma Ashraf, Matthew Bolton, Andrea Cartier, Richard Gilson, Lewis Haddow, Sara McNamara, Ana Milinkovic, June Minton, Dianne Morris, Clare Oakland, Steve O’Farrell, Pierre Pellegrino, Sarah Pett, Abigail Severn, Nina Vora, Carmel Young, Taris Zarko-Flynn
  5. Chelsea and Westminster Healthcare NHS Foundation Trust, London, UK; Simone Antonucci, Tristan Barber, Serge Fedele, Chris Higgs, Kathryn McCormick, Sheena McCormack, Alan McOwan, Alexandra Meijer, Sam Pepper, Jane Rowlands, Gurmit Singh, Alfredo Soler-Carracedo, Sonali Sonecha, Ann Sullivan, David Taylor, Lervina Thomas
  6. Guy’s and St Thomas’ NHS Foundation Trust, London, UK; Margaret-Anne Bevan, Julie Fox, Nina Francia, Eleanor Hamlyn, Lisa Hurley, Helen Iveson, Isabelle Jendrulek, Tammy Murray, Alice Sharp, Andrew Skingsley, Teresa Solano, Chi Kai Tam, Al Teague, Caroline Thomas, Juan-Manuel Tiraboschi
  7. Heart of England NHS Foundation Trust, Birmingham, UK; Sian Gately, Gerry Gilleran, Jill Lyons, Chris McCormack, Katy Moore, Cathy Stretton, Stephen Taylor, David White
  8. Homerton; University Hospital NHS Foundation Trust, London, UK; Frederick Attakora, Marina Bourke, Richard Castles, Rebecca Clark, Anke De-Masi, Veronica Espa, Rumbidzai Hungwe, Mohammed Rashid Khan, Martin Lincoln, Sifiso Mguni, Rhianon Nevin-Dolan, Sambasivarao Pelluri, Iain Reeves
  9. Imperial College Healthcare NHS Foundation Trust, London, UK; Wilbert Ayap, Ling Jun Chen, Adam Croucher, Sarah Fidler, Kristin Kuldanek, Ken Legg, Agathe Leon, Nicola Mackie, Nadia Naous, Killian Quinn, Severine Rey, Judith Zhou
  10. King’s College Hospital NHS Foundation Trust, London, UK; Hannah Alexander, Jake Bayley, Michael Brady, Lucy Campbell, Sophie Candfield, Shema Doshi, Olivia Liddle, Larissa Mulka, Priyanka Saigal, James Stevenson
  11. Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK; Anthony Bains, Gill Bell, Christine Bowman, Terry Cox, Claire Dewsnap, Charlie Hughes, Hannah Loftus, Naomi Sutton, Debbie Talbot, Vince Tucker
  12. York Teaching Hospital and Hull York Medical School, York, UK; Christine Brewer, Richard Evans, Jan Gravely, Charles Lacey, Gary Lamont, Fabiola Martin, Georgina Morris, Sarah Russell-Sharpe, John Wightman

We thank the staff who worked on the study at MRC CTU at UCL: Sarah Banbury, Elizabeth Brodnicki, Christina Chung, Yolanda Collaco-Moraes, David Dolling, David Dunn, Keith Fairbrother, Mitzy Gafos, Adam Gregory, Sajad Khan, Shabana Khan, Sheena McCormack, Brendan Mauger, Yinka Sowunmi, Susan Spencer, Ellen White, Gemma Wood.

We thank the additional members of the Social Science Advisory Group (SSAG) who are not authors on the paper but advised on the development of the acceptability questionnaire and in-depth interview guide at an SSAG workshop on 9-01-2012:

  • Ford Hickson, London School of Hygiene and Tropical Medicine
  • Ingrid Young, University of Glasgow
  • Marsha Rosengarten, Goldsmiths, University of London
  • Elizabeth Glendinning, University College London

We thank the UK Community Advisory Board (CAB) for reviewing the protocol and study materials and recommending changes, and Simon Collins from HIV i-Base for revising the participant information sheet and informed consent forms.

We thank the independent members of the PROUD Governance committees:

  • Trial Steering Committee: Michael Adler (Co-Chair), Gus Cairns (Co-Chair) Daniel Clutterbuck, Rob Cookson, Claire Foreman, Stephen Nicholson, Tariq Sadiq, Matthew Williams
  • Independent Data Monitor: Jack Cuzick
  • Independent Data Monitoring Committee: Simon Collins, Fiona Lampe, Anton Pozniak (Chair)
  • Community Engagement Group: Yusef Azad (NAT), Gus Cairns (NAM), Rob Cookson (LGF), Tom Doyle (Mesmac), Justin Harbottle (THT), Matthew Hodson (GMFA), Cary James (THT), Roger Pebody (NAM), Marion Wadibia (NAZ)

Author Contributions

  1. Conceptualization: MG SM WN SW GB D. Dolling CR MR RH.
  2. Data curation: EW D. Dunn.
  3. Formal analysis: MG.
  4. Funding acquisition: SM.
  5. Investigation: MG SM WN SW TB GB AC JF LH CL KQ CR IR MR DW VA WA CD GS.
  6. Methodology: MG SM WN SW GB D. Dolling CR MR RH.
  7. Project administration: MG EB EW GW.
  8. Software: YS.
  9. Supervision: MG RH.
  10. Writing – original draft: MG.
  11. Writing – review & editing: MG EB MD SM WN SW EW GW TB GB AC D. Dolling D. Dunn JF LH CL AN KQ CR IR MR DW VA WA CD YC GS YS RH.

References

  1. 1. Grant RM, Lama JR, Anderson PL, McMahan V, Liu AY, Vargas L, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. New England Journal of Medicine. 2010;363(27):2587–99. pmid:21091279
  2. 2. Thigpen MC, Kebaabetswe PM, Paxton LA, Smith DK, Rose CE, Segolodi TM, et al. Antiretroviral preexposure prophylaxis for heterosexual HIV transmission in Botswana. New England Journal of Medicine. 2012;367(5):423–34. pmid:22784038
  3. 3. Choopanya K, Martin M, Suntharasamai P, Sangkum U, Mock PA, Leethochawalit M, et al. Antiretroviral prophylaxis for HIV infection in injecting drug users in Bangkok, Thailand (the Bangkok Tenofovir Study): a randomised, double-blind, placebo-controlled phase 3 trial. The Lancet. 2013;381(9883):2083–90.
  4. 4. Karim QA, Karim SSA, Frohlich JA, Grobler AC, Baxter C, Mansoor LE, et al. Effectiveness and safety of tenofovir gel, an antiretroviral microbicide, for the prevention of HIV infection in women. science. 2010;329(5996):1168–74. pmid:20643915
  5. 5. Nel A, Bekker LG, Bukusi E, Hellstrm E, Kotze P, Louw C, et al. Safety, Acceptability and Adherence of Dapivirine Vaginal Ring in a Microbicide Clinical Trial Conducted in Multiple Countries in Sub-Saharan Africa. PloS one. 2016;11(3):e0147743. pmid:26963505
  6. 6. Baeten JM, Palanee-Phillips T, Brown ER, Schwartz K, Soto-Torres LE, Govender V, et al. Use of a Vaginal Ring Containing Dapivirine for HIV-1 Prevention in Women. The New England journal of medicine. 2016.
  7. 7. Baeten JM, Donnell D, Ndase P, Mugo NR, Campbell JD, Wangisi J, et al. Antiretroviral prophylaxis for HIV prevention in heterosexual men and women. New England Journal of Medicine. 2012;367(5):399–410. pmid:22784037
  8. 8. Van Damme L, Corneli A, Ahmed K, Agot K, Lombaard J, Kapiga S, et al. Preexposure prophylaxis for HIV infection among African women. New England Journal of Medicine. 2012;367(5):411–22. pmid:22784040
  9. 9. Marrazzo JM, Ramjee G, Richardson BA, Gomez K, Mgodi N, Nair G, et al. Tenofovir-based preexposure prophylaxis for HIV infection among African women. New England Journal of Medicine. 2015;372(6):509–18. pmid:25651245
  10. 10. Rees H, Delany-Moretlwe S, Baron D, Lombard C, Gray G, Myer L, et al., editors. FACTS 001 phase III trial of pericoital tenofovir 1% gel for HIV prevention in women. Conference on Retroviruses and Opportunistic Infections (CROI); 2015.
  11. 11. Auerbach. Women and PrEP: A Q&A with Dr. Judy Auerbach 2014 [cited 2016 24th January]. Available from: http://betablog.org/women-and-prep-qa-judy-auerbach/.
  12. 12. Morrow KM, Ruiz MS. Assessing microbicide acceptability: a comprehensive and integrated approach. AIDS and behavior. 2008;12(2):272–83. pmid:17592763
  13. 13. Mensch BS, van der Straten A, Katzen LL. Acceptability in microbicide and PrEP trials: current status and a reconceptualization. Current opinion in HIV and AIDS. 2012;7(6):534–41. pmid:23032737
  14. 14. Horne R, Weinman J, Hankins M. The beliefs about medicines questionnaire: the development and evaluation of a new method for assessing the cognitive representation of medication. Psychology and health. 1999;14(1):1–24.
  15. 15. Horne R, Chapman SC, Parham R, Freemantle N, Forbes A, Cooper V. Understanding patients’ adherence-related beliefs about medicines prescribed for long-term conditions: a meta-analytic review of the necessity-concerns framework. PloS one. 2013;8(12):e80633. pmid:24312488
  16. 16. Tolley E, Friedland B, Gafos M, Amico R, Van Damme L, Woodsong C, et al. Socioeconomic and behavioural factors influencing choice, adherence and success of microbicide formulations. In: Neves J, Samento B, editors. Drug delivery and development of anti-HIV microbicides. Singapore: Pan Stanford Publishing, CRC Press.; 2014.
  17. 17. Grant RM, Anderson PL, McMahan V, Liu A, Amico KR, Mehrotra M, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. The Lancet Infectious Diseases. 2014;14(9):820–9. pmid:25065857
  18. 18. Baeten J, Heffron R, Kidoguchi L, Celum C, Mugo N, Bukusi E. Partners Demonstration Project. Presentation delivered at CROI 2015, Seattle, February 23–26. Abstract 24. 2015.
  19. 19. Kuhn L, Stein Z, Susser I. Are randomised controlled trials always required? The lancet HIV. 2015;2(11):e460–1. pmid:26520924
  20. 20. Brown CH, Wyman PA, Guo J, Pena J. Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide. Clinical trials. 2006;3(3):259–71. pmid:16895043
  21. 21. Lam K. School-Based Cognitive Mindfulness Intervention for Internalizing Problems: Pilot Study with Hong Kong Elementary Students. Journal of Child and Family Studies. 2016:1–16.
  22. 22. Batra P, Schlossman N, Balan I, Pruzensky W, Balan A, Brown C, et al. A Randomized Controlled Trial Offering Higher-Compared with Lower-Dairy Second Meals Daily in Preschools in Guinea-Bissau Demonstrates an Attendance-Dependent Increase in Weight Gain for Both Meal Types and an Increase in Mid-Upper Arm Circumference for the Higher-Dairy Meal. The Journal of nutrition. 2016;146(1):124–32. pmid:26609172
  23. 23. Earley J. Two treatment options for patients to assist with management of their chronic pain: Multidisciplinary treatment in a supportive class format & Graded Motor Imagery. 2016.
  24. 24. Orandi BJ, Luo X, Massie AB, Garonzik-Wang JM, Lonze BE, Ahmed R, et al. Survival benefit with kidney transplants from HLA-incompatible live donors. New England Journal of Medicine. 2016;374(10):940–50. pmid:26962729
  25. 25. Jamil MS, Prestage G, Fairley CK, Grulich AE, Smith KS, Chen M, et al. Effect of availability of HIV self-testing on HIV testing frequency in gay and bisexual men at high risk of infection (FORTH): a waiting-list randomised controlled trial. The lancet HIV. 2017.
  26. 26. Fidler S, Fisher M, McCormack S. BHIVA‐BASHH Position Statement on PrEP in the UK: British HIV Association. British Association of Sexual Health and HIV. 2011.
  27. 27. McCormack S, Dunn DT, Desai M, Dolling DI, Gafos M, Gilson R, et al. Pre-exposure prophylaxis to prevent the acquisition of HIV-1 infection (PROUD): effectiveness results from the pilot phase of a pragmatic open-label randomised trial. Lancet. 2015.
  28. 28. Dolling DI, Desai M, McOwan A, Gilson R, Clarke A, Fisher M, et al. An analysis of baseline data from the PROUD study: an open-label randomised trial of pre-exposure prophylaxis. Trials. 2016;17:163. pmid:27013513
  29. 29. McCormack S, Dunn DT, Desai M, Dolling DI, Gafos M, Gilson R, et al. Pre-exposure prophylaxis to prevent the acquisition of HIV-1 infection (PROUD): effectiveness results from the pilot phase of a pragmatic open-label randomised trial. Lancet. 2016;387(10013):53–60. pmid:26364263
  30. 30. Ritchie J, Lewis J, Nicholls CM, Ormston R. Qualitative research practice: A guide for social science students and researchers: Sage; 2013.
  31. 31. Skingsley A YZ, Kirwan P, Croxford S, Chau C, Conti S, Presanis A, Nardone A, Were J, Ogaz D, Furegato M, Hibbert M, Aghaizu A, Murphy G, Tosswill J, Hughes G, Anderson J, Gill ON, Delpech VC and contributors. HIV in the UK–Situation Report 2015: data to end 2014. 2015.
  32. 32. Desai M, White E, Vora E, Gafos M, Apea V, Brady M, et al. Chemsex in the PROUD study. Public Health England conference; London 2016.
  33. 33. Gafos M, Dolling D, Desai M, Wood G, Dunn D, Saunders J, et al. Sexual Behaviour Profile of Gay and Other Men Who Have Sex with Men Enrolled in the PROUD Pre-exposure Prophylaxis Open-label Pilot Study in England. AIDS research and human retroviruses. 2014;30(S1):A22–A.
  34. 34. Holt M. Time to accommodate antiretroviral-based HIV prevention. The lancet HIV. 2016;3(9):e400–1. pmid:27562735
  35. 35. Mansergh G, Koblin BA, Sullivan PS. Challenges for HIV pre-exposure prophylaxis among men who have sex with men in the United States. PLoS medicine. 2012;9(8):e1001286. pmid:22927797
  36. 36. WHO. Guideline on when to start antiretroviral therapy and on pre-exposure prophylaxis for HIV. WHO, 2015.