As HIV infection continues unabated, there is a need for effective interventions targeting at-risk men who have sex with men (MSM). Engaging MSM online where they meet sexual partners is critical for HIV prevention efforts.
A randomized controlled trial (RCT) conducted online among U.S. MSM recruited from several gay sexual networking websites assessed the impact of 2 HIV prevention videos and an HIV prevention webpage compared to a control condition for the study outcomes HIV testing, serostatus disclosure, and unprotected anal intercourse (UAI) at 60-day follow-up. Video conditions were pooled due to reduced power from low retention (53%, n = 1,631). No participant incentives were provided.
Follow-up was completed by 1,631 (53%) of 3,092 eligible men. In the 60 days after the intervention, men in the pooled video condition were significantly more likely than men in the control to report full serostatus disclosure (‘asked and told’) with their last sexual partner (OR 1.32, 95% CI 1.01–1.74). Comparing baseline to follow-up, HIV-negative men in the pooled video (OR 0.70, 95% CI 0.54–0.91) and webpage condition (OR 0.43, 95% CI 0.25–0.72) significantly reduced UAI at follow-up. HIV-positive men in the pooled video condition significantly reduced UAI (OR 0.38, 95% CI 0.20–0.67) and serodiscordant UAI (OR 0.53, 95% CI 0.28–0.96) at follow-up.
Findings from this online RCT of MSM recruited from sexual networking websites suggest that a low cost, brief digital media intervention designed to engage critical thinking can increase HIV disclosure to sexual partners and decrease sexual risk. Effective, brief HIV prevention interventions featuring digital media that are made widely available may serve as a complementary part of an overall behavioral and biomedical strategy for reducing sexual risk by addressing the specific needs and circumstances of the target population, and by changing individual knowledge, motivations, and community norms.
Citation: Hirshfield S, Chiasson MA, Joseph H, Scheinmann R, Johnson WD, Remien RH, et al. (2012) An Online Randomized Controlled Trial Evaluating HIV Prevention Digital Media Interventions for Men Who Have Sex with Men. PLoS ONE 7(10): e46252. doi:10.1371/journal.pone.0046252
Editor: D. William Cameron, University of Ottawa, Canada
Received: December 15, 2011; Accepted: August 31, 2012; Published: October 2, 2012
Copyright: © Hirshfield et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was funded by the Centers for Disease Control and Prevention (CDC), where technical assistance was provided through a federal cooperative agreement in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, and approval of the manuscript. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.
Competing interests: The authors have declared that no competing interests exist.
Only an estimated 19% of the more than 1 million HIV-infected persons in the U.S have an undetectable viral load,  which is likely the result of social and behavioral barriers,  including the lack of awareness of HIV status.  It is therefore not surprising that overall HIV infection rates have been relatively stable in the U.S., though men who have sex with men (MSM) remain disproportionately affected by the transmission of HIV.  New HIV infections in MSM have been attributed in part to increased access to sex partners via online meeting websites,  thereby increasing the potential for transmission of HIV and other sexually transmitted infections (STIs).  HIV prevention studies conducted online have found that men who meet men online report more sex partners, casual partners, ,  and unprotected anal intercourse (UAI) than do men who meet partners offline.  It is therefore critical to deliver behavioral interventions to MSM online to reach and engage men where they meet sex partners , , .
Early in the epidemic, community-wide safer sex messages delivered through print ads and media campaigns were effective at educating the gay community and lowering transmission risk behaviors. ,  Since then, effective HIV prevention interventions geared toward changing high-risk behavior have been developed,  though many of these interventions have been in small-group formats,  which tend to be costly (i.e., labor-intensive, require many trained professionals), difficult to implement and sustain, and lack large-scale reproducibility. ,  Thus, online behavioral interventions, particularly low intensity methods, hold much promise for the future of HIV prevention as part of a multifaceted approach to risk reduction, given their relatively low cost to implement and potential to reach a wide audience of at-risk persons efficiently. , .
Technology-based HIV behavioral interventions are increasingly incorporating digital media, ranging from brief, untailored video interventions to complex computer-tailored multimedia interventions that target individual behaviors. ,  As an intervention tool, digital media can be delivered in a variety of settings, such as in schools or clinics,  and can be utilized electronically via text messaging,  handheld computers or Smartphones,  or online. , , ,  Videos are a prime example of using digital media to engage learners, and they have been used as an HIV prevention tool since the 1980s.  Further, online video-based interventions are an appealing and effective way to deliver HIV prevention content to MSM. , ,  They can be easily replicated after development, require minimal staffing, have broad geographic reach, and have the potential to change both community norms and individual behavior at low cost. .
Although online research studies tend to report higher attrition than offline research as there are fewer social constraints,  a growing number of validity studies indicate higher reporting of sexual risk and substance-using behaviors with computer-based surveys compared to mail, phone, and in-person surveys. , , ,  In addition, several large-scale studies comparing online to mail survey modes have found that online surveys have lower overall response rates but yield higher item response rates on both open- and close-ended questions, suggesting higher data quality. , , , .
We report findings from a theoretically grounded online randomized controlled trial (RCT), evaluating the use of HIV prevention videos and a prevention website to deliver risk reduction content to HIV-negative, HIV-positive and untested MSM. The two HIV prevention videos used in this trial were designed to reflect issues specific to MSM, while incorporating social learning theory, situated cognition, and developmental learning theory into the dialogue, storyline, and realism of the characters , ; principles of these theories are reflected in the videos' use of realistic stories, recognizable character types, and conflict to promote critical thinking.  For this online trial, we hypothesized that between baseline and 60-day follow-up, there would be increases in HIV disclosure and HIV testing, and decreases in UAI among those randomized to an HIV prevention video or a prevention webpage, compared to those in the control condition.
The protocol for this trial and supporting CONSORT checklist are available as supporting information; see Checklist S1 and Protocol S1. Survey outcome rates and methodological terminology are based on the reporting standards of the American Association for Public Opinion Research (AAPOR). .
This online 5-arm RCT, with a 1∶1 allocation ratio, compared the impact of three HIV prevention video conditions, an HIV prevention webpage, and a no-content control on three primary outcomes (HIV testing, HIV disclosure and UAI) at 60-day follow-up. The study design was based on an online pre-post test video-based intervention pilot conducted in 2005.  Since the retention rate for the pilot study was 54% with a 90-day follow-up period, investigators in the current study opted for a shorter, 60-day follow-up period to potentially increase the overall retention rate.
The primary objective of this 5-arm RCT was to assess the feasibility and efficacy of implementing a large-scale single-session online intervention, using HIV prevention videos or an HIV prevention webpage versus a no-content control, among sexually active U.S. MSM, aged 18 and over, who were recruited from four gay-oriented sexual networking websites.
Eligibility criteria for the intervention were programmed into the online baseline survey. Participants had to: 1) identify as a man; 2) be age 18 or older; 3) reside in the U.S.; 4) provide an email address; 5) report oral or anal sex with a current male partner (new or not), and oral, anal, or vaginal sex with at least one new partner (male or female) in the previous 60 days; 6) and have the ability to read and respond in English. Men who completed the baseline survey but were ineligible for the intervention were automatically transferred to the exit page that contained links to health-related websites and hotlines.
In April 2008, a banner ad was placed on four gay-oriented sexual networking websites for U.S. men. The demographic characteristics of the banner ad sampling frame are undefined as we do not know who was exposed to ad views. After several weeks of slow recruitment, one of the websites agreed to send emails through its internal system to all of its U.S. members (i.e., a list-based sampling frame of emails). Members of this site are automatically assigned an email address upon becoming a member, thus ensuring a valid user email address. The email sent to members contained a study banner, and all email recipients were considered potentially eligible for study inclusion.  Participants were recruited online between April and June 2008. A total of 609,960 emails were sent nationwide, with a 99.6% absorption rate (i.e., successful delivery of emails, indicating a high-quality sampling frame). The absorption rate was calculated by dividing the number of delivered emails (607,777) by all email invitations sent.  No incentives were offered to study participants.
The institutional review boards at Public Health Solutions (a nonprofit organization in New York City) and the Centers for Disease Control and Prevention (CDC) approved all study procedures. A waiver of documentation of written consent was obtained, given the internet-based research approach. Men who clicked on the study banner ad provided informed consent online by reading the consent form and clicking agreement to participate in the baseline survey. Following completion of the baseline survey, participants who met the inclusion criteria for the RCT were provided a second consent form inviting them to participate in additional study activities.
The five study conditions included a: 1) dramatic video; 2) documentary video; 3) both dramatic and documentary videos; 4) prevention webpage; and 5) control (i.e., received no intervention content). Both the dramatic and documentary HIV prevention videos (http://hivbigdeal.org/) were designed to promote critical thinking about HIV disclosure, HIV testing, and condom use. The videos were based on social learning theories and strategies , ,  that informed the instructional design and delivery of the online intervention along three important design dimensions: (1) the medium selected (i.e., video); , , ,  (2) the degree of realism in the content; ,  and (3) the finer-grained structure of the content, such as conflict between the characters to promote critical thinking. , , , , , .
Both HIV prevention videos were designed to tell the same story, through drama, on the one hand, and through documentary, on the other. Both videos provided positive and negative modeling examples of HIV disclosure to sex partners, which emphasized critical thinking and decision-making. Goals of the modeling examples conveyed by the prevention videos included increasing HIV disclosure awareness, serving as a refresher for risk reduction, and increasing a sense of responsibility to protect one’s sexual partners. ,  The Morning After is a 9-minute dramatic video addressing sexual risk reduction and features 3 gay male friends, one of whom thinks he had unprotected sex with an HIV-positive man while intoxicated and seeks advice from friends. Talking About HIV is a 5-minute documentary video addressing sexual risk reduction through testimonials of HIV-positive men and was created with footage from the feature-length documentary, “Meth.” .
Participants randomized to both videos were provided the videos in random order. Men randomized to the prevention webpage were provided a CDC webpage that featured information about HIV among MSM, with links to prevention information and resources. Participants assigned to the control condition were only provided with links to HIV prevention resources following completion of the behavioral survey. All participants could view their assigned online content, though once they closed their web browser they could not view the content again.
Participants were emailed 60 days post-baseline to complete the follow-up survey, which paralleled the baseline survey. A hyperlink was embedded in the email, and when clicked on, automatically transferred the participant to a second consent and follow-up survey. For those who did not respond to the follow-up email, we waited one week before sending a first reminder email, another week before sending the second reminder, and a third week before sending the final reminder (for an additional 21–30 days). The 60-day follow-up period was slightly extended to September 2008 (after receiving IRB approval) in order to send an additional email reminder to all intervention participants who did not complete the 60-day follow-up survey; this resulted in an additional 194 responses (12%).
We hypothesized that between baseline and 60-day follow-up, there would be increases in HIV disclosure and HIV testing, and decreases in UAI among those randomized to an HIV prevention video or a prevention webpage, compared to those in the control condition. The three primary outcome measures were increased HIV testing, HIV disclosure, and decreased UAI with sex partners at 60-day follow-up. At baseline, HIV testing variables included ever testing for HIV and month/year of last test and the result. At follow-up, HIV testing was measured as receiving an HIV test during the 60-day follow-up period. HIV disclosure was defined as partial (i.e., asking or telling) or full (i.e., both asking and telling) with a partner in a sexual encounter. Anal intercourse was defined as ‘any’ insertive and/or receptive sex (yes/no). UAI was defined as ‘any’ unprotected (without a condom) insertive and/or receptive sex (yes/no). Men were asked about the three most recent sexual partners in the past 60 days beginning with the most recent. Due to a software programming error, baseline HIV disclosure data captured information for only the last partner reported in the past 60 days. To address this limitation, we compared primary outcome data between the last partner reported at baseline and the corresponding last partner reported during follow-up.
Based on the prevalence of behaviors in our previous studies, we calculated true proportions and sample sizes using chi-square tests for this 5-group design. We estimated that by enrolling approximately 600 men per group and retaining 70% at 60 days follow-up, we would have 80% power at a 5% alpha level to detect behavioral change differences between 10–15% for the dichotomous primary outcomes (increased HIV disclosure, testing, and decreased UAI) between the video conditions and control condition at 60-day follow-up. As an example of the actual range of power that we had to detect a difference between the control and video conditions, the power ranged from 57% to 82% across the individual video conditions for asking a sex partner’s HIV status at follow-up. After combining the video conditions, we had 90% power to detect a difference in HIV disclosure at follow-up.
Those who consented were randomized into one of the five study conditions using a computerized randomization program, which included a blocking scheme to balance randomization across the study conditions using a non-deterministic algorithm. ,  The decisecond of the participant’s “click” determined study condition assignment, which continued throughout the balancing process, resulting in a study sample that was balanced within a 1% range.
Once participants consented and were randomized, they were kept in their original assignment condition and sent a link to the 60-day follow-up survey (i.e., intention to treat [ITT]). Thus, we used an ITT approach for primary outcome analyses. Since randomization was not stratified, for Tables 1 and 2, we took a conservative approach and tested for baseline differences across conditions. Chi-square tests assessed group-level comparisons of the dichotomous primary outcome measures at 60 days post-intervention. Follow-up sample characteristics were assessed using bivariate, rather than multivariate, logistic regression, as several characteristics were highly correlated. Due to reduced power from low retention (53%), the 3 video conditions were pooled for all outcome analyses. Statistical analyses were conducted using SPSS 20 for Windows. .
Analyses in Tables 3 and 4 account for losses and exclusions, which include being lost to follow-up (n = 1,461), dropping out during the follow-up survey (n = 142), and reporting no sexual activity during follow-up (and thus being automatically skipped out of the sexual encounter sections, but not the HIV testing section) (n = 104) (Figure 1). In Table 3, for group-level comparisons of primary outcomes at 60-day follow-up, the control condition was compared to the pooled video condition and prevention webpage condition separately. In Table 4, within-person changes for primary outcomes from baseline to 60-day follow-up were compared using McNemar’s test for paired data in the pooled video, webpage, and control conditions. Finally, we conducted sensitivity analysis and assessed follow-up non-response bias to examine the impact of attrition.
One of the four sexual networking websites agreed to send emails to all of its U.S. members (n = 609,960). A total of 23,213 (3.8%) men clicked on the study recruitment email hyperlink or banner ad that took them to the study landing page (Figure 1). Of those, 9,539 broke off from the landing page immediately, and 13,674 consented to participate in the baseline survey. Among men who consented, 11,798 (86%) completed the baseline survey and 1,876 (14%) had partial baseline surveys. Overall, the email recruitment response rate (AAPOR RR1) was 1.9% for completed cases and (AAPOR RR2) 2.2% with partials included. The number of banner ad impressions men were exposed to was not available from the websites, therefore we could not calculate a click-through-rate for men recruited through banner ads.
Among the consenting 11,798, the following were excluded: 157 residing outside of the US; 25 female, 17 female-to-male transgender, and 4 male-to-female transgender; 311 duplicate cases were detected and excluded. In addition, 5,449 men were ineligible for study enrollment, and 2,743 men were eligible but refused participation. A total of 3,092 participants were eligible and randomized into the study as follows: 609 in the control arm, 609 in the prevention webpage arm, 625 in the dramatic video arm, 633 in the documentary video arm, and 616 in the dramatic/documentary video arm (Figure 1). The drop-out rate for each study arm is provided in Figure 1.
Tables 1 and 2 describe the demographic and behavioral characteristics of the 3,092 randomized participants at baseline. Overall, respondents were predominantly white with high income and education, and were from every U.S. state. The median age was 39 (range 18 to 81). In the last sexual encounter within the past 60 days, 44% asked their partner’s HIV status, 53% told a partner their HIV status, and approximately one-third reported full disclosure (both asking and telling). No differences were found across study conditions at baseline for the primary outcome behaviors, except for a higher prevalence of asking/telling among men in the webpage condition versus the control. This was considered due to chance as participants were unaware of study hypotheses. Regarding past 60-day sexual behavior with partners in up to the last 3 sexual encounters, 80% reported anal intercourse and 58% reported UAI. Overall, 17% self-reported being HIV-positive. Among HIV-negative men, 18% reported getting an HIV test within the 60 days prior to baseline and 69% were tested in the past year (data not shown). In the past 60 days, most participants reported at least 3 sexual partners (72%), followed by 2 (17%), and 1 (10%); 13 men (1%) reported only multiple-partner encounters and therefore did not answer this section.
Among randomized respondents, 32% (n = 976) were recruited through banner ads and 68% (n = 2,116) through an e-mail member list. Most (93%) respondents recruited via banner ads were from the same sexual networking site that sent the study e-mail. Compared to men recruited through emails, men recruited via banner ads were significantly more likely to complete 60-day follow-up (58% vs. 50%, OR 1.39, p<.001). No demographic or primary outcome differences were found between the two recruitment methods, except for HIV status; HIV-positive men were significantly more likely to have been recruited through a banner ad than by email (20% vs. 16%, OR 1.31, p<.01) (data not shown).
Follow-up Retention Sample Characteristics
Compared to men who did not complete 60-day follow-up, men who did complete follow-up were significantly older (t = 7.27, p<.001), and more likely to be white (versus black OR 0.59, p<.01; Hispanic OR 0.71, p<.01; mixed/other race OR 0.60, p = .001), college-educated (OR 1.67, p<.001), HIV-positive (versus negative or untested, OR 1.35, p<.01), and earn more than $50,000 per year (OR 1.33, p<.001). Men who completed follow-up were also significantly more likely to have non-main partners at baseline (OR 1.59, p<.01) and to report more than 100 lifetime anal sex partners (OR 1.34, p<.01) than men who did not complete follow-up.
Figure 1 lists the reasons and rates of drop-out per condition. The median duration between completion of baseline intervention activities and the 60-day post-baseline assessment was 61 days.
Primary differences across treatment conditions at follow-up.
At follow-up (Table 3), men in the pooled video condition were significantly more likely than men in the control condition to report both partial HIV serostatus disclosure (“asking”, OR 1.51) and full disclosure (“asking and telling”, OR 1.32) in their sexual encounters in the past 60 days. Among men with non-main partners, those in the pooled video condition were significantly more likely than men in the control condition to ask (OR 1.49); between the two conditions, asking and telling approached significance (OR 1.29). No disclosure differences were seen for men with main partners. Among HIV-negative and untested men who completed follow-up (n = 1,116), 21% reported getting an HIV test; however there were no differences across study conditions. Eight men who self-reported never testing at baseline were tested within the 60-day follow-up period and reported a negative test result. No difference in sexual behavior was observed between conditions, with 68% reporting anal sex (data not shown) and 49% reporting UAI during follow-up. No differences were found between the prevention webpage and control condition for the primary outcomes.
Behavior change from baseline to follow-up.
With McNemar’s test (Table 4), we examined within-person behavior changes using paired data from baseline to 60-day follow-up by study condition (pooled videos, webpage, and control) for the primary outcomes: HIV disclosure, UAI, and HIV testing. Significant decreases in UAI were seen from baseline to follow-up within the digital media conditions (OR 0.61, OR 0.42). HIV-negative men in both the pooled video and webpage conditions reported significant reductions in UAI from baseline to follow-up (OR 0.70, OR 0.43), while HIV-positive participants only in the pooled video condition reported a significant decrease in UAI (OR 0.38). However, HIV-positive men in the pooled video condition also reported a significant reduction in UAI with HIV-negative and unknown status non-main partners (OR 0.53) from baseline to follow-up.
Contrary to expectations, men in both the webpage and control conditions had significant reductions in asking their partner’s HIV status at follow-up compared to baseline (OR 0.61, OR 0.60). For full disclosure, men in the pooled video condition were significantly less likely to ask and tell, overall (OR 0.76), and among HIV-positive participants (OR 0.35), at follow-up compared to baseline. Regarding HIV testing from baseline to follow-up, no changes were seen in any of the conditions.
Comparing men who did and did not complete follow-up, we assessed the potential impact of attrition by conducting sensitivity analyses. We assessed HIV disclosure at follow-up with the last sexual partner for parsimony and made two different assumptions about HIV disclosure among those lost to follow-up. First, we assumed that all men lost to follow-up were disclosers and found that asking a sex partner’s HIV status would be significantly higher in the pooled video condition than in the control (79% vs. 74%, OR 1.38, 95% CI 1.10–1.72). Second, we assumed that all those lost to follow-up were non-disclosers and found that there would also be a significant difference, albeit smaller, in asking between conditions (26% vs. 22%, OR 1.25, 95% CI 1.00–1.57).
Follow-up non-response bias.
To assess whether men who did not complete follow-up differed significantly on the primary outcome estimates from those who did complete follow-up, we tested for non-response bias. The full sample baseline primary outcome estimates for past 60-day behaviors were 44% for asking HIV status, 53% for telling HIV status, 58% for any UAI, and 18% for HIV testing. Using the baseline primary outcome estimates, we calculated follow-up non-response bias by subtracting the mean difference between men who responded to follow-up and the baseline sample, divided by the baseline sample estimate. Men who did complete follow-up had lower baseline mean estimates than men who did not complete follow-up for the baseline primary outcomes asking HIV status (41% vs. 47%), telling HIV status (52% vs. 53%), reporting any UAI (57% vs. 58%), and HIV testing (16% vs. 17%). We found significant non-response bias for asking HIV status, though the magnitude was small and was biased towards the null. Thus, men who completed the follow-up survey were significantly less likely to have asked their last sexual partner’s HIV status than men who did not complete the follow-up survey (41% vs. 47%, p = .001). No differences were found between men who did and did not complete follow-up for the other primary outcomes.
Findings from this online RCT of MSM recruited from sexual networking websites suggest that a low cost, brief digital media intervention conducted completely online can increase HIV disclosure to sexual partners and decrease sexual risk. Theoretically grounded HIV prevention videos, designed to engage critical thinking, and an HIV prevention website were evaluated for their impact on HIV disclosure, UAI, and HIV testing. MSM participating in the online trial were predominantly white with high income and education and were from every U.S. state. As reporting a new sex partner at baseline was one eligibility criterion for inclusion into the online trial, men participating in our study reported considerable HIV transmission risk at enrollment. On average, men reported 2 anal sex partners in the past 60 days, with more than half reporting UAI with non-main partners. Men who completed 60-day follow-up differed from men who did not complete follow-up by demographic and behavioral characteristics; most notably, men completing follow-up were significantly more likely to be HIV-positive, report non-main partners at baseline, and report more lifetime anal sex partners than men not completing follow-up.
At 60-day follow-up, modest group-level effects were seen for HIV disclosure. Men in the pooled video condition were significantly more likely than men in the control condition to report full (“asking and telling”) and partial (“asking”) HIV disclosure with sexual partners. Among men who had non-main partners during follow-up, partial HIV disclosure was significantly higher in the video condition than control condition. Significant changes in sexual behavior and HIV testing were not seen at the group level.
Within-Person Behavior Change
Contrary to expectations, some men were significantly less likely to disclose their HIV status to sex partners from baseline to 60-day follow-up. In contrast, within-person effects were found for reduced UAI at follow-up compared to baseline, with men reporting significant reductions in UAI in both the pooled video (OR 0.61) and webpage conditions (OR 0.42). The reduction in UAI among men in the pooled video condition is similar to a previously published one-group, pre-post pilot intervention – using one of the same videos used in the current trial – among MSM recruited from one of the same gay-oriented sexual networking websites that assessed within-person reduction of UAI (OR 0.55).  This finding suggests that watching a video about two gay men negotiating HIV disclosure and sexual risk can promote critical thinking and lead to reduced sexual risk.
From baseline to 60-day follow-up, HIV-negative men reported significant reductions in UAI in both the pooled video and webpage condition. HIV-positive men in the pooled video condition also reported significant reductions in UAI, but the most striking finding for HIV-positive men in the video condition was their significant reduction in UAI with HIV-negative or unknown status partners at follow-up compared to baseline (OR 0.53). It appears that that this low intensity digital media intervention may have resonated most with sexually active HIV-positive MSM, who may not be reached by traditional offline prevention messages.
While HIV disclosure during follow-up was significantly higher across several sex partners (i.e., video versus control condition) it was significantly lower when examined by participants’ baseline-to-follow-up corresponding partner. UAI was also significantly lower with corresponding partners from baseline to follow-up, which suggests that, rather than having a potentially awkward discussion about sex and serostatus, some men may have opted not to have unprotected sex. Disclosure to sexual partners and its relationship to sexual risk is complex and varies by partner type and venue , ,  but is a critical component of safer sexual behavior, particularly among those with HIV. .
Historically, online research has had lower retention rates than offline research as there are fewer social constraints compared to in-person interviewing.  In the current study that was conducted in 2008, online retention at 60 days post-intervention was 53%, which is low but comparable to other online prevention interventions that were also conducted several years ago. , ,  Another factor potentially impacting retention in this study was the lack of incentives, which likely contributed to lower retention rates.  Online delivery of the study provided an efficient means to reach a large sample size relatively quickly, but given the size of the sample, the study was unable to provide incentives for participation. Offline studies have also found pronounced differences in retention rates among those offering and not offering incentives. , .
Within a given study, non-response bias can occur when respondents differ from non-respondents, though non-response, in and of itself, does not mean that the data are inherently biased.  Observed differences between responders and non-responders do not necessarily indicate response bias, unless the differences are related to the study outcomes.  Across the primary outcomes, only one variable, asking a sex partner’s HIV status, was found to be consistently higher at baseline among men who did not complete follow-up than men who did. In sensitivity analysis, men in the video condition reported significantly higher asking of their partner’s HIV status than men in the control condition, demonstrating a robust finding. In follow-up non-response bias analyses we found a small but significant difference in asking a sex partner’s HIV status, with non-responders reporting higher disclosure, indicating that non-response bias was present but biased towards the null. Thus, men who completed the online follow-up survey were less likely to disclose at baseline and may have had a greater need for an online risk reduction intervention. These findings may not be generalizable to all men who access gay-oriented sexual networking websites or who may be exposed to study emails or banner ads.
Even within the context of these limitations, in this predominantly white sample of MSM recruited from one of the largest U.S. sexual networking websites for gay men, there was a modest effect from this online digital media intervention, with an overall increase in HIV disclosure and reduced sexual risk in both HIV-positive and HIV-negative men. The reduction in serodiscordant UAI among HIV-positive participants suggests that the videos resonated most with HIV-positive men, who may have been more amenable to change in an online intervention setting,  as a relatively high proportion enrolled into the study, were significantly more likely to complete follow-up than HIV-negative and untested men, and significantly reduced sexual risk behavior.
The development of online HIV prevention interventions is a burgeoning field. But in the overall landscape of HIV prevention, how do technology-based interventions fit in? There has been a recent push in HIV prevention for “combination prevention.”  This multifaceted approach combines biomedical prevention tools and behavioral science, with the goal of targeting and engaging specific at-risk populations and communities to reduce HIV transmission risk behaviors with a sustained impact. With the recent FDA approvals of an over-the-counter in-home HIV oral rapid test,  and the use of pre-exposure prophylaxis (PrEP) to reduce the risk of sexually-acquired HIV,  Internet-based HIV prevention interventions including digital media are uniquely positioned to provide a critical platform to promote new HIV prevention technologies among a broad audience of at-risk MSM.
The use of theoretically grounded dramatic or documentary video designed to engage critical thinking, in combination with biomedical science, is a largely unexplored area. Brief, widely available, effective digital media interventions like the HIV prevention videos and webpage described here may serve as a complementary part of an overall strategy for increasing HIV disclosure and reducing sexual risk. Even if modestly effective, online social marketing campaigns have the potential to expose a much broader audience to brief, digital media prevention interventions. Delivering HIV prevention content to MSM online where they seek sexual partners is an efficient way to reach large samples of geographically dispersed at-risk men and has the potential for a public health impact through addressing the specific needs and circumstances of the target population, and by changing individual knowledge, motivations, and community norms.
We would like to thank Drs. Samantha Garbers and Peter Messeri for their assistance on statistical matters.
Conceived and designed the experiments: SH MAC HJ RS WDJ RHR FSS RE ADM. Performed the experiments: SH MAC RS RE GY. Analyzed the data: SH MAC HJ RS WDJ GY ADM. Wrote the paper: SH MAC HJ RS WDJ RHR FSS RE GY ADM.
- 1. Gardner EM, McLees MP, Steiner JF, Del Rio C, Burman WJ (2011) The spectrum of engagement in HIV care and its relevance to test-and-treat strategies for prevention of HIV infection. Clin Infect Dis 52: 793–800. doi: 10.1093/cid/ciq243
- 2. Lange JM (2011) “Test and treat”: is it enough? Clin Infect Dis 52: 801–802.
- 3. Centers for Disease Control and Prevention (2011) HIV surveillance–United States, 1981–2008. MMWR Morb Mortal Wkly Rep 60: 689–693.
- 4. Prejean J, Song R, Hernandez A, Ziebell R, Green T, et al. (2011) Estimated HIV Incidence in the United States, 2006–2009. PLoS One 6: e17502. doi: 10.1371/journal.pone.0017502
- 5. Wolitski R (2005) The emergence of barebacking among gay men in the United States: a public health perspective. J Gay Lesbian Psychother 9: 13–38. doi: 10.1080/19359705.2005.9962410
- 6. McFarlane M, Bull S, Rietmeijer C (2000) The Internet as a newly emerging risk environment for sexually transmitted diseases. JAMA 284: 443–446. doi: 10.1001/jama.284.4.443
- 7. Rosser BR, Miner MH, Bockting WO, Ross MW, Konstan J, et al. (2009) HIV risk and the internet: results of the Men's INTernet Sex (MINTS) Study. AIDS Behav 13: 746–756. doi: 10.1007/s10461-008-9399-8
- 8. Horvath KJ, Rosser BR, Remafedi G (2008) Sexual risk taking among young internet-using men who have sex with men. Am J Public Health 98: 1059–1067. doi: 10.2105/ajph.2007.111070
- 9. Hooper S, Rosser BR, Horvath KJ, Oakes JM, Danilenko G (2008) An online needs assessment of a virtual community: what men who use the internet to seek sex with men want in Internet-based HIV prevention. AIDS Behav 12: 867–875. doi: 10.1007/s10461-008-9373-5
- 10. Chiasson MA, Parsons JT, Tesoriero JM, Carballo-Dieguez A, Hirshfield S, et al. (2006) HIV behavioral research online. J Urban Health 83: 73–85. doi: 10.1007/s11524-005-9008-3
- 11. Pequegnat W, Rosser BR, Bowen AM, Bull SS, DiClemente RJ, et al. (2007) Conducting Internet-based HIV/STD prevention survey research: considerations in design and evaluation. AIDS Behav 11: 505–521. doi: 10.1007/s10461-006-9172-9
- 12. Rosser BR, Oakes JM, Horvath KJ, Konstan JA, Danilenko GP, et al. (2009) HIV sexual risk behavior by men who use the Internet to seek sex with men: results of the Men's INTernet Sex Study-II (MINTS-II). AIDS Behav 13: 488–498. doi: 10.1007/s10461-009-9524-3
- 13. Plant A, Montoya JA, Rotblatt H, Kerndt PR, Mall KL, et al. (2010) Stop the sores: the making and evaluation of a successful social marketing campaign. Health Promot Pract 11: 23–33. doi: 10.1177/1524839907309376
- 14. Katzman J, Gulati H, Higa DH, Welch Q, Wood RW (2007) A “Community Manifesto” for gay and bisexual men: an appeal to control HIV/STDs. J Public Health Manag Pract 13: 244–251. doi: 10.1097/01.phh.0000267682.77787.3a
- 15. Kotler P, Roberto N, Lee N (2002) Social Marketing: Improving the Quality of Life. Thousand Oaks: Sage.
- 16. Valdiserri R, Ogden L, McCray E (2003) Accomplishments in HIV prevention science: Implications for stemming the epidemic. Nat Med 9: 881–889. doi: 10.1038/nm0703-881
- 17. Noar SM (2011) Computer technology-based interventions in HIV prevention: state of the evidence and future directions for research. AIDS Care 23: 525–533. doi: 10.1080/09540121.2010.516349
- 18. Johnson WD, Diaz RM, Flanders WD, Goodman M, Hill AN, et al.. (2008) Behavioral interventions to reduce risk for sexual transmission of HIV among men who have sex with men. Cochrane Database Syst Rev: CD001230.
- 19. Kiene SM, Barta WD (2006) A Brief Individualized Computer-Delivered Sexual Risk Reduction Intervention Increases HIV/AIDS Preventive Behavior. J Adolesc Health 39: 404–410. doi: 10.1016/j.jadohealth.2005.12.029
- 20. Guse K, Levine D, Martins S, Lira A, Gaarde J, et al.. (2012) Interventions Using New Digital Media to Improve Adolescent Sexual Health: A Systematic Review. J Adolesc Health May 7 [Epub ahead of print].
- 21. Noar SM, Black HG, Pierce LB (2009) Efficacy of computer technology-based HIV prevention interventions: a meta-analysis. AIDS 23: 107–115. doi: 10.1097/qad.0b013e32831c5500
- 22. Warner L, Klausner JD, Rietmeijer CA, Malotte CK, O'Donnell L, et al. (2008) Effect of a brief video intervention on incident infection among patients attending sexually transmitted disease clinics. PLoS Med 5: e135. doi: 10.1371/journal.pmed.0050135
- 23. Cole-Lewis H, Kershaw T (2010) Text messaging as a tool for behavior change in disease prevention and management. Epidemiol Rev 32: 56–69. doi: 10.1093/epirev/mxq004
- 24. Jones F, Fox S (2009) Generations online in 2009. Philadelphia: Pew Internet Am Life Proj.
- 25. Carpenter KM, Stoner SA, Mikko AN, Dhanak LP, Parsons JT (2010) Efficacy of a web-based intervention to reduce sexual risk in men who have sex with men. AIDS Behav 14: 549–557. doi: 10.1007/s10461-009-9578-2
- 26. Rosser BR, Oakes JM, Konstan J, Hooper S, Horvath KJ, et al. (2010) Reducing HIV risk behavior of men who have sex with men through persuasive computing: results of the Men's INTernet Study-II. AIDS 24: 2099–2107. doi: 10.1097/qad.0b013e32833c4ac7
- 27. Blas MM, Alva IE, Carcamo CP, Cabello R, Goodreau SM, et al. (2010) Effect of an online video-based intervention to increase HIV testing in men who have sex with men in Peru. PLoS One 5: e10448. doi: 10.1371/journal.pone.0010448
- 28. Chiasson M, Shuchat Shaw F, Humberstone M, Hirshfield S, Hartel D (2009) Increased HIV disclosure three months after an online video intervention for men who have sex with men (MSM). AIDS Care 21: 1081–1089. doi: 10.1080/09540120902730013
- 29. Solomon MZ, DeJong W (1986) Recent sexually transmitted disease prevention efforts and their implications for AIDS health education. Health Educ Q 13: 301–316. doi: 10.1177/109019818601300403
- 30. Chiasson MA, Hirshfield S, Rietmeijer CA (2010) HIV Prevention and Care in the Digital Age. J Acquir Immune Defic Syndr 55: S94–S97. doi: 10.1097/qai.0b013e3181fcb878
- 31. Cohen D, Wu S, Farley T (2004) Comparing the cost-effectiveness of HIV prevention interventions. J Acquir Immune Defic Syndr 37: 1404–1414. doi: 10.1097/01.qai.0000123271.76723.96
- 32. Birnbaum M (2004) Human research and data collection via the internet. Annu Rev Psychol 55: 803–832. doi: 10.1146/annurev.psych.55.090902.141601
- 33. Link MW, Mokdad AH (2005) Alternative Modes for Health Surveillance Surveys: An Experiment with Web, Mail, and Telephone. Epidemiol 16: 701–704. doi: 10.1097/01.ede.0000172138.67080.7f
- 34. Perlis TE, Des Jarlais D, Friedman S, Arasteh K, Turner C (2004) Audio-computerized self-interviewing versus face-to-face interviewing for research data collection at drug abuse treatment programs. Addiction 99: 885–896. doi: 10.1111/j.1360-0443.2004.00740.x
- 35. Elford J, Bolding G, Davis M, Sherr L, Hart G (2004) Web-based behavioural surveillance among men who have sex with men: a comparison of online and offline samples in London, UK. J Acquir Immune Defic Syndr 35: 421–426. doi: 10.1097/00126334-200404010-00012
- 36. Newman J, Des Jarlais D, Turner C, Gribble J, Cooley P, et al. (2002) The differential effects of face-to-face and computer interview modes. Am J Public Health 92: 294–297. doi: 10.2105/ajph.92.2.294
- 37. Shin E, Johnson TP, Rao K (2012) Survey Mode Effects on Data Quality: Comparison of Web and Mail Modes in a U.S. National Panel Survey. Soc Sci Comput Rev 30: 212–228. doi: 10.1177/0894439311404508
- 38. Bech M, Kristensen MB (2009) Differential response rates in postal and web-based surveys among older respondents. Surv Res Methods 3: 1–6.
- 39. Denscombe M (2009) Item non-response rates:A comparison of online and paper questionnaires. Int J Soc Res Methodol 12: 281–291. doi: 10.1080/13645570802054706
- 40. Kwak N, Radler B (2002) A comparison between mail and web surveys: Response pattern, respondent profile, and data quality. J Off Stat 18: 257–273.
- 41. Bandura A (1986) Social foundations of thought and action: A social cognitive theory. Englewood Cliffs: Prentice Hall.
- 42. Schank R, Berman T (2002) The pervasive role of stories in knowledge and action. In: Green M, Strange JJ, Brock TC, editors. Narrative impact: Social and cognitive foundations. Mahway: Lawrence Erlbaum Associates. 287–313.
- 43. Schank R (1990) Tell me a story: Narrative and intelligence. Evanston: Northwestern University Press.
- 44. The American Association for Public Opinion Research (2011) Standard Definitions: Final dispositions of case codes and outcome rates for surveys. 7th ed. Deerfield: AAPOR.
- 45. Manfreda KL, Berzelak N, Vehovar V (2011) Nonresponse in web surveys. Int Ency Stat Sci 14: 984–987.
- 46. Brown S, Collins A, Duguid P (1989) Situated cognition and the culture of learning. Educ Res 18: 32–42. doi: 10.3102/0013189x018001032
- 47. Liebert R, Sprafkin J (1988) The early window: Effects of television on children and youth. Elmsford: Pergamon Press.
- 48. Collins A (1996) Design issues for learning environments. In: Vosniadou S, De Corte E, Glaser R, Mandl H, editors. International perspectives on the design of technology-supported learning environments. Mahwah: Lawrence Erlbaum Associates.
- 49. Bransford J, Sherwood R, Hasselbring T, Kinzer C, Williams S (1990) Anchored instruction: Why we need it and how technology can help. In: Nix D, Spiro R, editors. Cognition, educaton, and multimedia: Exploring ideas in high technology. Hillsdale: Lawrence Erlbaum Associates. 115–141.
- 50. Jonassen D, Howland J, Moore J, Marra R (2003) Learning to solve problems with technology: A constructivist perspective. Columbus: Merrill/Prentice-Hall.
- 51. Jonassen D, Hernandez-Serrano J (2002) Case-based reasoning and instructional design: Using stories to support problem solving. Educ Technol Res Dev 50: 65–77. doi: 10.1007/bf02504994
- 52. Wetzel CD, Radtke PH, Stern HW (1994) Instructional effectiveness of video media. Mahwah: Lawrence Erlbaum Associates.
- 53. Bandura A (1977) Social learning theory. Englewood Cliffs: Prentice-Hall.
- 54. Piaget J (1969) The mechanisms of perception. London: Rutledge & Kegan Paul.
- 55. Festinger L (1957) A theory of cognitive dissonance. Evanston: Row Peterson.
- 56. Schank R (1982) Dynamic memory: A theory of reminding and learning in computers and people. New York: Cambridge University Press.
- 57. Serovich JM, Mosack KE (2003) Reasons for HIV disclosure or nondisclosure to casual sexual partners. AIDS Educ Prev 15: 70–80. doi: 10.1521/aeap.18.104.22.16846
- 58. Wolitski RJ, Flores SA, O'Leary A, Bimbi DS, Gomez CA (2007) Beliefs about personal and partner responsibility among HIV-seropositive men who have sex with men: measurement and association with transmission risk behavior. AIDS Behav 11: 676–686. doi: 10.1007/s10461-006-9183-6
- 59. Ahlberg T (2006) Meth. Documentary.
- 60. Friedman L, Furberg C, DeMets D (1996) Fundamentals of Clinical Trials. St. Louis: Mosby-Year Book, Inc.
- 61. Hromkovič J (2005) Design and analysis of randomized algorithms: introduction to design paradigms. New York: Springer.
- 62. IBM (2012) IBM SPSS Statistics 21. Armonk.
- 63. Simoni JM, Pantalone DW (2004) Secrets and safety in the age of AIDS: does HIV disclosure lead to safer sex? Top HIV Med 12: 109–118.
- 64. Grov C (2012) HIV Risk and Substance Use in Men Who Have Sex with Men Surveyed in Bathhouses, Bars/Clubs, and on Craigslist.org: Venue of Recruitment Matters. AIDS Behav 16: 807–817. doi: 10.1007/s10461-011-9999-6
- 65. Grov C, Hirshfield S, Remien RH, Humberstone M, Chiasson MA (2011) Exploring the Venue's Role in Risky Sexual Behavior Among Gay and Bisexual Men: An Event-Level Analysis from a National Online Survey in the U.S. Arch Sex Behav Oct 20. [Epub ahead of print].
- 66. Serovich JM, Craft SM, McDowell TL, Grafsky EL, Andrist D (2009) Methods of promoting safer sex behaviors utilized by men who have sex with male casual sex partners. AIDS Educ Prev 21: 1–13. doi: 10.1521/aeap.2009.21.1.1
- 67. Bull SS, Vallejos D, Levine D, Ortiz C (2008) Improving recruitment and retention for an online randomized controlled trial: experience from the Youthnet study. AIDS Care 20: 887–893. doi: 10.1080/09540120701771697
- 68. Bull SS, Lloyd L, Rietmeijer C, McFarlane M (2004) Recruitment and retention of an online sample for an HIV prevention intervention targeting men who have sex with men: the Smart Sex Quest Project. AIDS Care 16: 931–943. doi: 10.1080/09540120412331292507
- 69. Goritz AS (2006) Incentives in Web Studies: Methodological Issues and a Review. Int J Internet Sci 1: 58–70.
- 70. Church AH (1993) Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opin Q 57: 62–79. doi: 10.1086/269355
- 71. Singer E, Van Hoewyk J, Gebler N, Raghunathan T, McGonagle K (1999) The Effect of Incentives on Response Rates in Interviewer-Mediated Surveys. J Off Stat 15: 217–230.
- 72. Groves RM, Peytcheva E (2008) The Impact of Nonresponse Rates on Nonresponse Bias. Public Opin Q 72: 167–189. doi: 10.1093/poq/nfn011
- 73. Rogelberg SG, Luong A (1998) Nonresponse to Mailed Surveys: A Review and Guide. Curr Dir Psychol Sci 7: 60–65. doi: 10.1111/1467-8721.ep13175675
- 74. Wolitski RJ, Gomez CA, Parsons JT (2005) Effects of a peer-led behavioral intervention to reduce HIV transmission and promote serostatus disclosure among HIV-seropositive gay and bisexual men. AIDS 19 Suppl 1S99–109. doi: 10.1097/01.aids.0000167356.94664.59
- 75. Hankins CA, de Zalduondo BO (2010) Combination prevention: a deeper understanding of effective HIV prevention. AIDS 24 Suppl 4S70–80. doi: 10.1097/01.aids.0000390709.04255.fd
- 76. U.S. Department of Health and Human Services (2012) July 3, 2012 Approval Letter, OraQuick In-Home HIV Test. In: Office of Blood Research and Review, editor. Silver Spring: Food and Drug Aministration.
- 77. U.S. Department of Health and Human Services (2012) Truvada for PrEP Fact Sheet: Ensuring Safe and Proper Use. Silver Spring: Food and Drug Administration.