Project overview

The overall goal of the project was to develop and provide initial psychometric validation for a set of scale measures that may prove inexpensive and easy to administer, with better predictive ability than current self-report adherence measures. Two types of scales were envisioned: 1) Screener measures, administered prior to product use (i.e., during a screening visit) that might predict an individual’s propensity to adhere; and 2) Adherence Monitoring measures that could be administered periodically during participant follow-up to assess participants’ attitudes about and experiences with the product.

We followed a standard scale development process, which included identifying an item and construct pool from the published literature related to adherence measurement and HIV prevention (specifically microbicide) clinical trial research; rounds of cognitive interviewing to improve item framing, comprehensibility and salience; and administration of a survey to facilitate exploratory factor analyses and psychometric evaluation of resulting constructs. After this initial evaluation, we envisioned sharing the resulting scale measures with other clinical researchers, who could prospectively evaluate the scales’ ability to predict adherence among participants in a new HIV prevention clinical trial.

Conceptual framework

We conducted a comprehensive review of the peer-reviewed literature to identify articles describing psychometrically validated scales or other measures related to adherence or compliance. We included measures of adherence in the context of both treatment and prevention to a range of conditions, including HIV, pregnancy prevention, and other disease areas (i.e., blood pressure, diabetes). Measures assessing adherence within clinical trial contexts were of particular interest. In addition, we conducted additional searches on measures assessing motivations for clinical trial participation, as well as measures of social desirability bias.

Based on the literature review, we developed a conceptual framework (Fig 1) depicting potential domains for adherence screening and monitoring tools. As depicted on the left-hand side of the diagram, our screener tool would ideally identify whether a potential trial participant was likely to be adherent, unintentionally non-adherent, or intentionally non-adherent. These adherence categories are based in the Necessities-Concerns Framework [27], a conceptual model postulating that adherence to a range of chronic disease medications is explained by an individual’s perceived need for treatment and their concerns about the potential consequences of taking it [28]. Despite limited application of the framework to prevention, findings from the qualitative studies described earlier are consistent with the framework’s adherence constructs. Predisposing factors which emerged in the qualitative HIV prevention literature and for which we identified potential screening measures include: reasons for trial participation [29–36]; beliefs about clinical trials or the products being tested [37, 38]; and personal attributes such as commitment, self-efficacy, or agency [21, 38–40]. The literature review also identified barriers and facilitators of adherence that might be experienced once a trial participant began using the study product and which could be assessed over time, including: the beneficial or negative impact of trial staff [38, 41, 42]; practical constraints, including travel, cost of participation, and impact on daily life [22, 37, 43, 44]; experience of side effects [44–46]; and impact of stigma [30, 44]. These constructs are presented on the right-hand side of our conceptual framework. Finally, the larger box within which predisposing and experienced factors are located represents the potential for some participants to respond in a socially desirable manner. We reviewed and included measures of social desirability bias in this study [47–50]. In this paper, we describe the outcomes of our survey and initial validation procedures.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Conceptual framework.

https://doi.org/10.1371/journal.pone.0195499.g001

Exploratory dimensional analysis (EDA)

We conducted separate EDAs of screening and monitoring items. Prior to each EDA, we first examined the response distribution, means, standard deviations, skew and kurtosis of all items, eliminating items with very high mean scores (> 5.8 of maximum 6.0). With each set of items, we conducted an exploratory factor analysis (EFA) with oblique (Promax) rotations and a principal component analysis (PCA) with orthogonal (varimax) rotation, retaining items with loadings of .4 or higher on both procedures, and assessing similarities in the resulting factor structure of the two analyses.

We used both the scree test [51] and the parallel analysis (PA) procedure [52, 53] in our initial determination of the underlying dimensional structure of the screening and the adherence monitoring measures. We also examined the internal consistency of each derived factor and used Cronbach’s Coefficient Alpha statistic [54] to help refine an individual scale by removing items with low inter-item correlations to increase reliability. Our final decision as to whether to keep or remove a specific item was based on the conceptual content of each item as it related to a specific factor.

Initial validation

We qualitatively evaluated the fit between our conceptual framework and the resulting constructs. In addition, we made predictions about the direction and strength of correlations between our adherence screener and monitoring constructs, as well as between scale constructs and other sociodemographic variables. We ran Pearson Product Moment Correlations to test our predictions. Finally, we reran our EDA of screener items separately for FTP and TNP samples to assess whether a different factor structure and/or item composition existed in these two groups.

Sociodemographic information

Overall, 60% of survey participants (n = 425) had participated in past HIV prevention clinical trial research; the majority (n = 339: 79.8%) were former participants of ASPIRE or the Ring Study–two trials evaluating the efficacy of an HIV prevention intravaginal ring, while 74 (17.4%) women had used a vaginal gel in the FACTS trial and 12 (2.8%) had used oral pills in the VOICE trial (Table 1). Trial-naïve participants, who were drawn from the same communities, tended to be younger, but similar to the FTP group on other sociodemographic information.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sociodemographic characteristics.

https://doi.org/10.1371/journal.pone.0195499.t001

Screener results

We began with 86 potential screening items and retained 40 items, loading onto six factors (Table 2). The 6-factor solution explained 64% of the variance in the full sample and 57% and 48% in the FTP and TNP samples respectively. Internal reliability of items ranged from acceptable for the Commitment to Research factor (α = .61) to very good for the Partner Disclosure factor (α = .80).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. PCA item loadings for 6-factor solution of screening items, from the full sample and by sub-groups.

https://doi.org/10.1371/journal.pone.0195499.t002

The 6-factor solution mapped well to our conceptual framework (Fig 1), consisting of factors that measured reasons for clinical trial participation (Commitment to Research, Personal & Health Benefits), beliefs about the trial and product (Distrust of Research) and commitment/agency (Trial Incompatibility, Partner Disclosure, Visit Adherence). While the full and subgroup EDAs produced similar underlying factor structures, item composition differed for several factors. For example, in the TNP subgroup, the Distrust factor did not include two items related to the use of experimental products, however, the TNP Distrust factor did include the item “I believe that people who participate in HIV prevention research can experience unpleasant side effects.” This same item did not load on Distrust for the FTP subgroup, but instead, loaded negatively onto the Personal & Health Benefits factor. Additionally, several items indicating potential risk disinhibition (i.e., belief in the therapeutic benefit of an experimental product) loaded onto the Personal & Health Benefits factor for the FTP subgroup, but did not do so for the TNP subgroup. The Trial Incompatibility factor showed the largest differences in item composition by subgroup. Over the full sample, the items suggest participants’ incompatibility or non-alignment with the trial, including being motivated by study reimbursements and having concerns about how others would react to their trial participation. Within the TNP subgroup, most items loading on this factor suggested an external locus of control–that God, luck or others determined their ability to accomplish behaviors. Many of these items did not load onto the Trial Incompatibility factor in the FTP subgroup analysis.

Adherence monitoring results

Adherence monitoring items were adapted during our cognitive interviewing process (not described) to assess attitudes and experiences specific to product use. Because our survey included very few vaginal gel or pill users, we conducted this analysis with FTPs of previous vaginal ring trials only (n = 339). Out of 48 potential adherence monitoring items, we retained 29 items loading on four factors (Table 3), including an “enacted” adherence measure (Ring Removal), several factors related to the presence or absences of practical constraints (Vaginal Ring Doubts and Vaginal Ring Benefits) and Side Effects. The 4-factor solution explained 64% of the variance in items. The resulting scales had moderate to high internal reliability. Although no specific factors emerged that related to Trial Staff or Social Stigma, several items from these domains did load onto the Vaginal Ring Doubts factor.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. EFA item loadings for 4-factor solution of adherence monitoring items, FTP sample from vaginal ring clinical trials only.

https://doi.org/10.1371/journal.pone.0195499.t003

Social desirability

Overall, trial-naïve participants scored lower on the social desirability measure than former trial participants. Participants who scored high (1 SD above the mean) on the SDB scale scored significantly lower on the Distrust in Research and Trial Incompatibility screening scales as well as the Vaginal Ring Doubts and Concern about Side Effects monitoring scales; they also scored significantly higher in the Commitment to Research, Visit Adherence and Vaginal Ring Benefits scales (Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Summary of scale scores by participant type and SDB levels.

https://doi.org/10.1371/journal.pone.0195499.t004

Initial construct validation

After developing the scales, we made a priori predictions about the correlations between screener constructs and several sociodemographic variables, as well as with the adherence follow-up measures (Table 5). The bolded cells in Table 4 show where our predictions matched findings. Few of our screener constructs performed as hypothesized relative to the sociodemographic variables. In many cases, we assumed significant relationships between screening scale scores and demographic information when correlations were not significant. However, our predictions of correlations between screener and adherence monitoring scales were often correct. In particular, participants who scored higher on Distrust in Research–and lower on Visit Adherence–were more likely to have high scores on Vaginal Ring Removal, Vaginal Ring Doubts and Side Effects. Conversely, high scores on Personal & Health Benefits and Visit Adherence were associated with high scores on Vaginal Ring Benefits. The Partner Disclosure scale was negatively correlated with Vaginal Ring Removal, but not associated with other monitoring scales. As predicted, participants with low scores on our SDB measure had higher scores on Distrust in Research, as well as Vaginal Ring Removal, Vaginal Ring Doubts and Side Effects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Predictions of correlations between screener scores, concurrent variables and adherence monitoring scores.

https://doi.org/10.1371/journal.pone.0195499.t005

Correlations between Screening scales and sociodemographic and behavioral variables (as identified in Table 1) shown in non-shaded cells. Correlations between Screening and Monitoring scales shown in shaded cells. Bolded text represents confirmation of hypothesized relationship.

Screening measures

This study produced a set of six scale measures that have the potential to identify individuals at screening who may face challenges with adherence to clinical trial requirements and product use. Two of the screening scales, Commitment to Research and Personal & Health Benefits, represent reasons identified by others as underlying motivations to participate in a range of prevention trials [36, 56–59]. Altruism (a component of the Commitment to Research scale), expressing feelings or actions that show a selfless concern for others, was identified as an independent predictor of adherence in a randomized, placebo controlled trial assessing use of estrogen for the prevention of stroke [60]. It was endorsed by the vast majority of participants in the VaxGen trial as a primary reason for joining the trial [31, 61]; it was also identified in qualitative sub-studies with men-who-have-sex with men (MSM) from U.S. sites of the iPrEX trial [62]. While women participating in the ADAPT open-label pre-exposure prophylaxis (PrEP) study in Cape Town, South Africa were engaged in research to benefit themselves, many also identified “Ubuntu”–contributing to the wellbeing of one’s community–as a strong motivator for study participation [63]. Women, selected to participate in a Fem-PrEP sub-study whose mean drug concentration levels throughout the trial were moderate to high, reported that a mix of personal benefits and altruism motivated their adherence to study product [10]. Indeed, personal benefits, such as access to good healthcare, access to social services or routine testing for HIV and other conditions, were more frequently mentioned as a reason to participate in HIV vaccine research among high-risk drug-using women in the U.S. [64], as well as women participating in a PrEP trial in South Africa [65].

A third set of items produced from our screening items related to Distrust of Research, a construct identified as limiting trial participation among African Americans in cardiovascular trials [66] and HIV prevention trials [36, 67, 68], high-risk women [64] and minorities in the U.S.[68]. In the VOICE-C sub-study, both women and men in Uganda, Zimbabwe and South Africa described how a lack of understanding and trust in research and the product being tested made it difficult for some women to adhere to product use [69]. Similarly, South African women participating in the qualitative sub-study of the ADAPT open-label PrEP study revealed an underlying level of skepticism about PrEP and HIV prevention research more generally [63]. As the authors note, this skepticism did not lead women to reject the research outright, but did lead some to approach study and product use requirements cautiously. Based on their qualitative data, they theorize that participants who hold strong feelings of fear and distrust may still join PrEP trials or open-label studies, particularly if they see high value from other aspects of trial participation, while intentionally avoiding PrEP use and actively discouraging others from using their products.

Additionally, three other potential screening scales assess aspects of the participants’ personal, social and daily contexts that affect their ability to adhere to trial and product use. They include a 15-item Trial Incompatibility scale, a five-item Partner Disclosure scale, and a three-item Visit Adherence scale. In contrast to the Distrust in Research scale, the Trial Incompatibility scale appears to represent attitudes towards health and a life context that is disorganized and therefore doesn’t support the exigencies of clinical trial participation. Although the composition of scale items substantially differs between former trial participants and those who were trial-naïve, they both include items indicating motivation for monetary rather than health reasons for trial participation and that others may influence their health and/or trial-related behaviors. Both also have items suggesting incompatibility with the trial due to disliking risk or the time commitment required by participation. The scale solution produced from the trial-naïve sample also includes several items suggesting that other forces outside of their control (God or fate, for example) influence their behaviors. The Perceptions and Practicalities Approach, a framework developed to explain non-adherence of therapeutic medications, suggests that unintentional non-adherence is driven by lack of capacity and resources which might include perceived difficulty in organizing their life circumstances, lack of social support [28, 70]. The inability to disclose to sexual partners about one’s participation in research and use of study products has been identified by numerous researchers as barrier to product use [13, 71]. The Visit Adherence scale may tap into an individual’s commitment and/or ability to organize their daily lives around health visits, a construct that has been shown to relate to adherence [72].

Monitoring scales

We produced a set of four monitoring scale measures that could be used to tailor counseling and address specific barriers or concerns to product use over trial follow-up. The Vaginal Ring Removal scale is composed of items expressing circumstances in which participants may have removed their ring since the previous follow-up visit. Because participants can express a degree of (dis)agreement to the items, they may be more willing to indicate problems with adherence than if asked direct “yes” or “no” questions. The Vaginal Ring Doubts, Vaginal Ring Benefits and Side Effects scales could provide counselors with some indications of whether ring use is perceived as potentially harmful or beneficial and how well it fits into their daily lives and sexual contexts. Several items on the Vaginal Ring Benefits scale indicated belief in the therapeutic benefit of ring use–a belief that might not be accurate within the context of a randomized, placebo-controlled trial.

Social desirability bias

Ancillary studies from several past HIV prevention trials have found that a non-negligible group of women report high adherence despite having little or no intention of using their study product [7, 13]. However, to date, there do not appear to be any tools available to identify participants who are predisposed to exaggerate their adherence. We adapted and included a measure of social desirability bias, the Marlowe-Crowne scale, to explore, through cognitive interviewing, whether these items translated cross-culturally, and if so, whether the set of items might be used in combination with other scale scores to identify women with potential adherence problems. The Marlowe-Crowne scale has been described as assessing two underlying factors that lead individuals to misrepresent their behaviors–impression management and self-deception [50]. Because the SDB measure was associated in predictable ways with our screening and monitoring measures, it might be a useful tool to identify a subset of participants whose reported trial-related perceptions and behaviors are potentially “too good to be true”. However, before using such a tool to actively screen out potential participants, further research is needed to determine the predictive validity of the measure as it relates to an objective adherence measure. It is notable that SDB scores in our study were higher, on average, among former trial participants than women who had not previously participated in an HIV prevention trial. It is unclear, however, whether trial participation itself conditions individuals to manage the impressions that others have of them, whether those who choose to join trials differ from non-participants on one or both of these underlying factors (i.e., impression management and self-deception), or whether they are generally just more agreeable people.

We believe that the scale approach pursued in this study offers a novel and promising strategy to improving adherence within future HIV prevention clinical trials. Nevertheless, our study has several limitations. First, screening items could perform differently if individuals believe their responses could affect enrollment. Further research is needed to assess the predictive validity of our screening scales, as well as the timing and approach to their administration. We invite collaboration with clinical researchers to integrate these screening and/or monitoring scales into future trials or demonstration studies to determine how well one or more of these scales correlate with objective biomarkers of adherence. Second, the psychometric properties of several scales could be improved. For example, several screening scales, including Commitment to Research, Personal and Health Benefits and Partner Disclosure, are moderately to highly negatively skewed, meaning that a large proportion of participants reported high levels of agreement with items. Without prospective validation, it is difficult to determine whether this is problematic, reducing the scales’ ability to provide meaningful information about participants’ trial-related motivations and partner contexts. On the other hand, perhaps they are adequately sensitive and specific enough to identify the subset of participants whose low levels of commitment, concerns about health impacts, or inability to disclose to a partner would drive intentional non-adherence. Finally, the factor solutions for several of our scales differed for women who had previous trial-experience versus those who were trial naïve. Because most HIV prevention trials will recruit women with a range of trial experience, researchers may want to consider using all items initially, assessing item loading and internal reliability of each scale before analysis.