Historical and current stigmatizing and discriminatory experiences drive sexual and gender minority (SGM) people away from health care and clinical research. Being medically underserved, they face numerous disparities that make them vulnerable to poor health outcomes. Effective methods to engage and recruit SGM people into clinical research studies are needed.
To promote health equity and understand SGM health needs, we sought to design an online, national, longitudinal cohort study entitled The PRIDE (Population Research in Identity and Disparities for Equality) Study that enabled SGM people to safely participate, provide demographic and health data, and generate SGM health-related research ideas.
We developed an iPhone mobile application (“app”) to engage and recruit SGM people to The PRIDE Study–Phase 1. Participants completed demographic and health surveys and joined in asynchronous discussions about SGM health-related topics important to them for future study.
The PRIDE Study–Phase 1 consented 18,099 participants. Of them, 16,394 provided data. More than 98% identified as a sexual minority, and more than 15% identified as a gender minority. The sample was diverse in terms of sexual orientation, gender identity, age, race, ethnicity, geographic location, education, and individual income. Participants completed 24,022 surveys, provided 3,544 health topics important to them, and cast 60,522 votes indicating their opinion of a particular health topic.
We developed an iPhone app that recruited SGM adults and collected demographic and health data for a new national online cohort study. Digital engagement features empowered participants to become committed stakeholders in the research development process. We believe this is the first time that a mobile app has been used to specifically engage and recruit large numbers of an underrepresented population for clinical research. Similar approaches may be successful, convenient, and cost-effective at engaging and recruiting other vulnerable populations into clinical research studies.
Citation: Lunn MR, Capriotti MR, Flentje A, Bibbins-Domingo K, Pletcher MJ, Triano AJ, et al. (2019) Using mobile technology to engage sexual and gender minorities in clinical research. PLoS ONE 14(5): e0216282. https://doi.org/10.1371/journal.pone.0216282
Editor: Mellissa H. Withers, USC Keck School of Medicine, Institute for Global Health, UNITED STATES
Received: July 27, 2018; Accepted: April 17, 2019; Published: May 2, 2019
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: The PRIDE Study iPhone app source code is available at github.com/delcurrie/pride-study. Underlying data cannot be made publicly available for the following reasons: members of the lesbian, gay, bisexual, transgender, and queer (LGBTQ) communities have experienced significant stigma and discrimination from society including the medical and investigational communities. As such, we are ethically bound to upholding the principle of non-maleficence; we promise our participants to not let any data (including deidentified) fall into the hands of people who may use it to publish stigmatizing results about the LGBTQ communities. As such, we have developed an Ancillary Study process in which investigators interested in using our data submit a brief application which is reviewed by both a Research Advisory Committee (composed of scientists) and Participant Advisory Committee (composed of participants) to affirm appropriate data use. Details about the Ancillary Study process are available at www.pridestudy.org/collaborate or by contacting us at email@example.com or 855-421-9991 (toll-free).
Funding: Research reported in this article was partially funded through a Patient-Centered Outcomes Research Institute (PCORI, www.pcori.org) Award (PPRN-1501-26848) to M.R.L. The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of PCORI, its Board of Governors or Methodology Committee. M.R.L. was partially supported by a Ruth L. Kirschstein NRSA Institutional Training Grant (T32DK007219) from the National Institute of Diabetes and Digestive and Kidney Diseases (www.niddk.nih.gov). M.C. received funding through a Clinical Research Training Fellowship from the American Academy of Neurology and Tourette Association of America as well as grants from The Health Trust and the Center for California Studies at Sacramento State University. J.O.M. was partially supported by the Veterans Affairs Women’s Health Clinical Research Fellowship (https://www.va.gov/oaa/specialfellows/programs/SF_WomensVets.asp?p=20 and partially by K12DK111028 from the National Institute of Diabetes and Digestive and Kidney Diseases (www.niddk.nih.gov). A.F. was partially supported by K23DA039800 from the National Institute on Drug Abuse (www.nida.nih.gov). THREAD Research provided support in the form of salaries for authors (AJT, CS, JF), but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of all authors are articulated in the ‘author contributions’ section.
Competing interests: CS and JF are current employees of THREAD Research. AJT is a former employee of THREAD Research. All other authors have no financial or professional conflicts of interest. This project (including all of the authors) did not receive any financial support from Apple, Incorporated. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Understanding the health needs of vulnerable populations–and the inequities that contribute to those needs–is critical to improving health and wellbeing. Engaging vulnerable populations, however, is often challenging because these populations are underserved in clinical settings, understudied in research, and experience structural and societal barriers to wellness including institutionalized discrimination and stigmatized identities. Unethical conduct from investigational communities (e.g., Tuskegee syphilis experiments on African-Americans, harmful conversion “therapies” to change sexual orientations and gender identities, limited initial federal response to the HIV epidemic) has broken the trust of many populations and deterred them from interacting with the health care system. Some populations–such as sexual and gender minority (SGM) people–are particularly difficult to identify or find because their demographic information (e.g., sexual orientation, gender identity) is not captured by health care systems , reported in clinical research [2, 3], or included in federal population surveys (e.g., Census, American Community Survey) which are used to generate probability-based sampling frames for larger national health studies . These absent data make SGM people invisible [5, 6].
Digital engagement for clinical research may be an effective strategy with vulnerable populations for five reasons. First, digital interactions (i.e., with a website or a mobile app) can be free of stigmatizing or discriminatory interpersonal interactions (e.g., homophobia, transphobia) [7–10]. Second, the ubiquity of Internet access via numerous devices (e.g., computers, tablets, smartphones) enables a broad reach into geographic areas away from traditional research centers [11, 12]. Third, customization enables unique experiences composed only of study components applicable to a specific participant (e.g., tailored survey language and questions based on demographics or reported behaviors). This also permits study participants to consent to sharing specific aspects of their participation and designating who can receive those data . Fourth, the participant remains in full control of their participation and can easily determine the specific study components they would like to perform without the pressure of an in-person interaction during which they may feel obligated to maintain participation or embarrassed to refuse specific study components. Finally, digital approaches are nimble and can be rapidly modified as needs dictate including due to societal changes and in response to participant feedback. Despite these advantages, digital engagement strategies do have pitfalls and may be unsuccessful for several reasons. First, they do not allow meaningful in-person interpersonal interactions and relationship-building, which may help foster engagement and retention in research studies. Second, verifying participant identity is difficult and expensive in a digital-only setting. Third, digital interactions require less participant effort than in-person engagement; while too much effort will deter participation, too little effort may result in non-committed participants who are subject to drop-out and loss to follow-up. Finally, while the vast majority has Internet access, Internet access may be limited for those older than 65 years, with less annual income, with less education, and in rural areas .
The near-ubiquity of Internet access (89% of US adults  and increasing smartphone ownership (77% of US adults ) have opened new avenues to influence and study health in participants’ natural environments using tools (i.e., smartphone, Internet access) already in participants’ possession. Health-related mobile applications (“apps” ) are commonly available for behavior tracking (e.g., tracking diet, weight, exercise), often in conjunction with a connected consumer device (e.g., physical activity tracker, scale). Additional health-related apps provide coaching (e.g., weight loss, smoking cessation) or facilitate interactions with the health care system by connecting to patient portals in electronic health records. While significantly fewer apps exist for the express purpose of conducting health-related research, the recent advent of the open-source software development frameworks ResearchKit (for iOS, researchkit.org) and ResearchStack (for Android, researchstack.org) enabled rapid app development and fostered researcher-software developer partnerships to study a variety of conditions from heart disease  to asthma [18, 19] and from melanoma  to Parkinson disease . This paradigm shift in clinical research simultaneously pushed novel methods to facilitate scalable participant-administered consent processes .
Sexual and gender minority (SGM) people (i.e., lesbian, gay, bisexual, transgender, and queer communities and people whose sexual orientation, gender identity/expression, or reproductive development varies from traditional, societal, cultural, or physiological norms ) are an underserved, vulnerable, and understudied population . To promote SGM health equity and understand the health needs of SGM people, we designed an online, national, longitudinal cohort study entitled The PRIDE (Population Research in Identity and Disparities for Equality) Study (pridestudy.org). Because SGM people have been subject to stigma and discrimination in society and by the medical and investigational communities [25, 26], we used a digital approach to promote safe, comfortable participation by SGM people. We desired to partner with SGM community members to learn about them and their health concerns and to actively engage them in designing The PRIDE Study.
Here, we report the development of a novel ResearchKit-based iPhone app for The PRIDE Study–Phase 1. We used the app for three main goals: (i) to recruit a large sample of diverse SGM people , (ii) to collect demographic and health data, and (iii) to engage SGM people in the generation of SGM health-related research ideas. This paper focuses on the development of our iPhone app, the successful recruitment of diverse SGM people, and their engagement with study activities (surveys, community forum). (Health survey data and participant-proposed SGM health research topics will be reported elsewhere.) We believe this is the first time that a mobile app has been used to specifically recruit and engage large numbers of an underrepresented and understudied population for clinical research.
The PRIDE Study is a national, online, longitudinal cohort study of the physical, mental, and social health of SGM people. The conception, development, implementation, and success of The PRIDE Study–Phase 1 in engaging and recruiting a diverse sample of SGM people are discussed here. The PRIDE Study–Phase 1 launched on June 25, 2015 to engage SGM communities in proposing and discussing health-related topics for research in The PRIDE Study and to create a snapshot of the cohort’s health. To do this, we designed and deployed an Apple iPhone app with the following goals: (i) recruiting a large diverse sample of SGM adults, (ii) gathering participant-level demographic and health data, and (iii) engaging the SGM communities in research question generation and prioritization. Data and lessons learned from Phase 1informed Phase 2, which launched via a web-based portal (pridestudy.org) accessible from any Internet-connected computer, tablet, or smartphone on May 2, 2017.
Mobile app design and development
Apple announced the release of ResearchKit (researchkit.org), a framework to more easily develop iPhone apps for medical research, on March 9, 2015. Apple chose to facilitate the development of an iPhone app for The PRIDE Study as one of six initial apps released on the platform.
Because we did not have in-house software development expertise, we partnered with a technology design firm (THREAD Research) for app design and development. We were responsible for study concept, study design, and ensuring that app design and functionality met our study’s needs. THREAD Research was responsible for gathering and implementing the business and functional requirements that we desired in our iPhone app, designing a visually-appealing and engaging user interface/experience (UI/UX), coding the app employing the ResearchKit framework, and setting up the cloud-based backend (i.e., compute and database instances, storage) for proper data collection and storage. We had full access to the study data and conducted all analyses.
App development occurred rapidly over four weeks using the scrum software development framework. The development team included one scrum master, one iOS developer, one back-end developer, one product designer, one digital producer, one quality assurance specialist, and two executives (Vice President of Digital, Chief Operating Officer). We had frequent (i.e., daily, every other day) “sync sessions” with the THREAD Research team to decide key design elements and ensure all questions relating to current and upcoming development were answered. Rigorous load testing, testing the app on different iPhone versions, software development quality assurance (QA) checks, bug remediation, user acceptance testing, and live QA review were performed by THREAD Research under our supervision.
Data management and security
All data transmissions from app users to the cloud computing services (Amazon Web Services, AWS; Seattle, WA) were encrypted using 256-bit Secure Sockets Layer (SSL) cryptographic protocols. A secure load balancer forced encrypted connections and distributed the workload across several compute instances as needed. All app account data (except Community Forum data) were stored in AWS using services provided by Aptible (New York, NY). In this infrastructure, the database was within a private subnet (not accessible from the Internet) inside an AWS Virtual Private Cloud. All data were encrypted in transit and at rest. Identifying information (first name, last name, e-mail address, date-of-birth) was stored in a database separate from survey data. All community forum data were stored in a MongoDB (NoSQL) database running on Heroku (San Francisco, CA). Survey data (linked to the participant by participant ID number) were stored in Qualtrics Research Suite (Salt Lake City, UT) to facilitate easy reporting and analysis.
Regulatory and compliance
Final versions were submitted to the Apple App Store for review under the University of California, San Francisco (UCSF)’s developer account. The first version of the app was available on June 25, 2015. The UCSF Institutional Review Board (IRB) approved this study (Study Numbers: 15–16697 and 16–18773).
PRIDEnet: Community engagement network
Prior to the development and launch of the app, a preliminary market survey was conducted by Salesforce to understand potential participant motivations for participating in an electronic health study of SGM people. We subsequently developed PRIDEnet, a participant-powered research network and community engagement mechanism designed to catalyze SGM health research by building relationships, recognizing multifaceted identities, creating equity among SGM communities, and maintaining transparency in our work. PRIDEnet (pridestudy.org/pridenet) is currently composed of a national participant advisory committee, ambassadors, and a national community partner consortium of approximately 41 SGM-focused non-profit health clinics, community centers, and advocacy/professional organizations to engage participants in research (including The PRIDE Study), publicize study launch broadly, and disseminate study results back to SGM community members. The community partner consortium members are active, known organizations within their local communities and serve as trusted partners to their constituencies to help ensure that SGM voices from across the country are influencing The PRIDE Study.
Publicity and recruitment
Eligibility screening and enrollment
Interested participants who downloaded and installed the app from the Apple App Store were presented with a welcome screen that provided study and consent information (Fig 2A). For those wishing to join the study, Boolean eligibility screening questions were presented (Fig 2B). People were eligible to participate in The PRIDE Study–Phase 1 if they (i) were at least 18 years of age, (ii) living in the United States, (iii) were a sexual and/or gender minority, (iv) were comfortable reading and writing in English, and (v) had access to an Apple iPhone. Eligible participants were subsequently directed through a series of screens (analogous to an in-person consent discussion) explaining study activities, time commitment, data security, potential benefits, etc. (selected examples in Fig 3A–3C). Participants were subsequently required to successfully pass a comprehension quiz about study participation before affirming their desire to participate and providing their name, date-of-birth, e-mail address, ZIP code, and digital signature. Enrollment was completed when participants verified their e-mail addresses by visiting a unique link sent by e-mail. Participants were also able to authorize sharing of Apple HealthKit data (i.e., height, weight, steps walked, flights climbed, distance traveled).
(A) Welcome screen. (B) Eligibility screening questions.
(A-C) Example screens to guide participants step-by-step through informed consent.
In addition to enrollment procedures, the app contained a community discussion forum, surveys, and a real-time cohort data dashboard (all discussed in detail below). In response to participant feedback, a new version of the app was released on September 11, 2015 that included design changes to better represent gender identity data from the transgender community, to add push notification functionality to notify participants of new study activities, and to provide bug fixes.
To engage study participants in a discussion forum about SGM health-related topics, we developed a community module (using Telescope–telescopeapp.org–as this functionality did not exist in ResearchKit) (Fig 4A). Participants selected a screen name, which was not linked to their PRIDE Study iPhone app account or any survey responses. Participants were invited to “post a topic [they] would like researchers to study about the LGBTQ community” (Fig 4B) and classify the topic by health (mental/emotional, physical, social), identity (asexual, bisexual, gay, lesbian, queer, transgender men, transgender women), and age (children and teenagers– 0–18 years-old; young adults– 18–40 years-old; middle-aged adults– 41–65 years-old; senior adults– 65+ years-old). Participants could reply to topics posted by others to start a discussion thread and could vote each topic up (favorable) or down (unfavorable) to indicate their opinion of the topic. Participants could “follow” a conversation to be notified of new discussion and could “flag” conversations as inappropriate for investigation by study staff.
We developed five surveys: (i) Demographics, (ii) Physical Health, (iii) Mental Health, (iv) Social Health, and (v) Improving The PRIDE Study. The Demographics survey was released on June 25, 2015; all other surveys were released one year later on July 25, 2016. With the exception of items assessing sexual orientation and gender identity, existing survey items from federal studies (e.g., National Health Interview Survey) and validated scales were used when available. For sexual orientation and gender identity items, we combined items created by several groups [29–31] in order to have more comprehensive answer choices (Obedin-Maliver et al., under review). Surveys were piloted and subsequently modified to improve understanding, usability, and burden.
Participant recruitment and demographics
During this phase, The PRIDE Study iPhone app was downloaded 35,168 times with 18,099 participants completing consent and 16,394 participants providing data (i.e., completing any survey). Of the 16,385 participants providing demographic data, more than 98% identified as a sexual minority (i.e., identified as anything but exclusively heterosexual), and more than 15% identified as a gender minority (i.e., gender identity that is not congruent with sex assigned at birth). The median age was 27.6 years (IQR, 22.6–36.5). Complete sociodemographics are available in Table 1. Participants reported significant digital connectedness with 95.9% using social media at least once per week. Participants each had an average of 3.15 social media accounts (standard deviation, 1.70). Participants self-reported numerous “other” sexual orientations and gender identities not captured by our provided answer choices; we are reporting these details elsewhere (manuscript in preparation). Participants were able to view real-time cohort statistics of sexual orientation, gender identity, race/ethnicity, and relationship status via a participant dashboard (Fig 5A–5C).
(A) Overall enrolled participant count and distribution of participants with similar demographics. (B) Distribution of participants’ sexual orientations and gender identities. (C) Distribution of participants’ racial/ethnic identities and relationship statuses.
Participant engagement and experience
Nearly all consented participants (n = 16,385; 99.95%) completed the Demographics survey while only a relatively small minority completed the Physical Health (n = 2,075; 12.7%), Mental Health (n = 1,833; 11.2%), Social Health (n = 1,803; 11.0%), and Improving The PRIDE Study (n = 1,926; 11.8%) surveys. The median completion time for each survey was less than ten minutes: 7 minutes 20 seconds for Physical Health, 5 minutes 11 seconds for Mental Health, and 8 minutes 55 seconds for Social Health, and 2 minutes 25 seconds for Improving The PRIDE Study. Participants reported that they would be willing to complete an average of 7.89 (standard deviation 4.25) 30-minute surveys annually. Participants were asked “what will keep you coming back and completing surveys annually for The PRIDE Study?” and most frequently selected: updates about study results (59.7%), feedback about how my participation is helping the study (51.7%), and incentives (e.g., gift cards, giveaways; 31.7%). More than one-quarter (25.6%) selected “I don’t need anything in return. I will keep participating to improve LGBTQ health.” Physical activity information (collected by iPhone sensors) was shared by 5270 (32.2%) participants.
In the community forum module, 3,544 health topics were provided by 3,089 participants (average 1.14 health topics posted per user, range 1–8). There were 5,063 responses from 3,174 participants (average 1.6 responses per user, range 1–38) posted about these health topics. There were 56,341 up votes and 4,181 down votes cast by participants to indicate their opinion of a particular topic. Fewer than ten comments required moderation by the study staff for inappropriate/offensive language including anti-SGM language. The “Top Five” discussion topics (algorithm based on number of views and number of up votes) were presented in the live participant-facing dashboard, along with number of topics posted daily, and a breakdown by health, identity, and age classifications. Participant-proposed SGM health research topics and health survey data will be presented elsewhere.
App development and study costs
Our estimated costs were approximately $605,000 including ~$300,000 for researcher/staff salaries (two fellows at $75,000 per person per year for two years), ~$250,000 in app development costs (provided pro bono by THREAD Research), ~$45,000 in cloud computing expenses (provided pro bono by Aptible and THREAD Research), and ~$10,000 in advertising. We did not separate participant recruitment from other study costs; our costs included app design and development, cloud computing and database services, recruitment of 18,099 participants, completion of 24,022 surveys, and robust community forum engagement with 3,544 posted health topics, more than 5,000 comments, and more than 60,500 votes cast by participants. With 18,099 consented participants, this equates to $33.43 per participant inclusive of all study infrastructure and activities.
The PRIDE Study–Phase 1 utilized an innovative iPhone app that successfully recruited a diverse national sample of nearly 20,000 SGM people who completed over 24,000 surveys and provided more than 3,500 health topics for future study in The PRIDE Study–Phase 2. This is believed to be the first time that an app has specifically engaged and recruited large numbers of an underrepresented and understudied population for clinical research . It proved to be a cost-effective way to engage participants in providing and prioritizing health-related research questions/concerns and completing surveys. Similar approaches may be effective in other populations that are also underserved, understudied, and vulnerable to poor health outcomes. With stigmatizing, discriminatory, or dangerous experiences in society, health care, or investigational communities, potential participant interactions with the health care system and traditional research enterprise may be limited out of fear. The online approach, however, avoids in-person interactions and may thereby make research study participation safer and more comfortable by enabling high quality control for omitting stigmatizing content that may arise during interpersonal interactions [7–10].
With the development of mHealth technologies including consumer health devices (e.g., activity trackers)  and increasing Internet-connected device ownership , clinical research is changing. Clinical research is moving away from the traditional in-person model that recruits participants to come to medical centers for research or recruits patients into research studies through their interactions with the health care system. Such studies are generally inconvenient (due to travel, parking, scheduling, etc.), and particularly so for participants with limited mobility, who live far from academic medical centers, and who are unable to take time off from work. Despite significant development costs, digital research is appealing to researchers due to the ability to reach a large geographic region, more facile recruitment (in part due to less invasive/intense study designs), and decreasing per-participant costs as additional participants enroll. These advanced technologies enable clinical research study recruitment and participation (i.e., data collection) on a more frequent basis from various sources (e.g., surveys, consumer health devices, smartphone location and sensor data) in participants’ natural environments (e.g., home, workplace) without the burden created by in-person visits. These benefits are balanced with significant concerns, especially in today’s sociopolitical climate, about data security and privacy. Methods (and appropriate messaging to participants) to increase data security (e.g., two-factor authentication, encrypting databases at rest and in transit, limiting administrator access to a certain IP address range, etc.) and ensure participant privacy (e.g., establishing methods to deidentify data, data use agreements to ensure data is released to researcher committed to improving SGM health) are critical to building trustworthiness.
Like The PRIDE Study, several digital cohort studies have successfully recruited high numbers of participants in recent years. Stanford’s MyHeart Counts Study (a ResearchKit-powered iPhone app) recruited nearly 50,000 participants, collected survey-based participant-reported data including laboratory values, and utilized the iPhone sensors to conduct 6-minute walk tests . The Health eHeart Study (health-eheartstudy.org) is a web-based longitudinal cohort study based at the University of California, San Francisco that has recruited over 180,000 consented participants . Other studies have utilized mobile technologies to study disease in new ways: image-based analysis of moles to study melanoma , geofencing to assess hospitalizations , and geolocation in combination with ecological momentary assessments to assess asthma triggers .
Community engagement improves trust and willingness to participate in research by engaging stakeholders in creating and maintaining patient/participant-centered approaches . Technological advances have expanded community engagement techniques by moving traditional in-person experiences to digital spaces. Community engagement strategies, such as online focus groups or crowdsourcing of research questions (as we did here), have enabled researchers to harness the power of large numbers in generating data, analyzing data, or generating novel research questions [37, 38]. In longitudinal digital studies where participants do not form in-person relationships with study staff, community engagement techniques to foster ownership of and belonging in research studies may be particularly important to retain active participation. Finally, returning research results directly to participants demonstrates the immediate outputs of individual participation by returning the data (and interpretation of those data) for a specific population. While ethical, legal, and regulatory issues abound, the 2018 revisions to the Federal Policy for the Protection of Human Subjects (“the Common Rule”) signal this evolution to a more participatory biomedical research model .
Our digital approach has many strengths. First, a large, diverse, national sample of SGM people was successfully recruited with participants from many age, racial, ethnic, educational, and socioeconomic groups. Second, the average cost-per-participant was significantly less than typical observational clinical research studies. Pharmaceutical clinical studies, for example, ranged in cost from $1.4 to $52.9 million depending on study procedures, therapeutic area, and phase . Third, participation was low burden, accessible to anyone with access to an iPhone, and occurred in participants’ natural environments without requiring clinical research physical or human resources. Fourth, participants exhibited high engagement by completing surveys and readily providing their current health concerns and commenting and voting on concerns posted by others. Fifth, research data were returned to participants via a real-time dashboard. This also permits researchers to easily track the cohort composition and alter recruitment efforts as needed. Sixth, passive collection of activity-based data was successful using Apple’s HealthKit (developer.apple.com/healthkit/) code framework.
Our work has several limitations. First, we recruited a convenience (non-probability) sample. This limitation is inherent in studies involving communities with limited population-level data (e.g., SGM people) to generate probability sampling frames. Sexual orientation and gender identity have not been previously collected and will not be collected by the US Census Bureau for the 2020 Census or American Community Survey , which are used as the sampling frames for other large scale studies . Comparing our sample to U.S. Census data is not possible. Despite this limitation, important within-sample comparisons remain possible. Second, our sample was not as racially/ethnically diverse as we would have liked. This may be due to the demographics of Apple iPhone ownership, online advertising that did not target specific populations, the audience demographics of our earned media coverage, etc. Specific community engagement attention to diverse subcommunities may recruit and retain a more heterogeneous sample. Third, our app-based approach without an in-person component limited the data types that can be collected or verified, and it limited participation to iPhone owners. Fourth, the low response rate to some surveys was likely due to a long lag between initial launch and deployment of the physical health, mental health, and social health surveys; a shorter data collection period for these surveys; and limited notifications to participants about the new surveys. Continued engagement with surveys or other activities may be an important aspect for continued participation in digital longitudinal cohort studies. Fifth, we did not verify participant identity. As such, our sample could be contaminated by duplicate participants and non-SGM-identified people (e.g., those who might be curious about the study or press). As no monetary incentive was offered for participation, the impetus for multiple accounts created by a single user was low. Finally, we did not collect app engagement metrics (e.g., daily app opens, daily active users, daily sessions, churn rate, etc.) to measure user engagement with our iPhone app.
We encountered and learned from several technical challenges. First, incomplete bug and quality assurance testing resulted in the date-of-birth of many participants not being stored in the database upon enrollment. We were able to obtain a significant fraction of these missing data via an updated app and by asking participant date-of-birth in other surveys. Second, the high volume of responses in the community forum resulted in slow loading. This, along with limited functionality to sort and filter data, frustrated participants and limited their ability to participate fully. We suspect that our engagement would be even higher with a higher performance, user experience-optimized community forum. Third, at the time of our ResearchKit app release, the early ResearchKit framework had several limitations including cumbersome coding for survey branching, the inability to push surveys live directly to participants (downloading an app update is required), and the inability to link results of one survey to another. (The ResearchKit framework has since been expanded, and ResearchStack (researchstack.org)–a similar software development framework for Android–has been developed.)
The approach that we described here may be adapted for other communities/populations. Partnering with engaged community members as meaningful, valued stakeholders in the study design and app development processes can help vet decisions while simultaneously building a group of trusted partners. Adaption to another population requires updating the language, imagery, and study procedures (e.g., survey questions) to resonate with the communities of interest; inclusive, culturally competent/humble studies enable potential participants to see themselves as welcomed participants. Depending on the historical or current transgressions, additional work may be needed to combat mistrust that many populations have of the medical or investigational communities. Engaging community members throughout the entire process creates a study that was truly designed with the community, rather than for the community. Digital research platforms–like the new NIH-funded Eureka Research Platform (info.eurekaplatform.org)–enable researchers to create mobile health research studies customized to the specific study without the need to set-up expensive computing infrastructure.
We chose to not pursue further development of our iPhone app for several reasons. First, the high cost of iPhone devices (and the lack of pre-paid iPhones) limited the diversity of the SGM communities that we could reach. Second, as noted above, deploying additional surveys could not be pushed live directly to participants in the initial ResearchKit release (i.e., participants had to download an app update to get new surveys). Third, including Android users would have required us to conduct parallel app design and development of an Android app. All app updates would also need to be done on both platforms. Non-iPhone/Android smartphone users would still be unable to participate.
We decided to retire our iPhone app and develop a web-based research portal accessible from any Internet-connected device (i.e., computers, tablets, all smartphones) to allow for broader accessibility of diverse populations including those who do not own a smartphone and to enable immediate deployment of new surveys to participants without requiring additional steps by the participant. This digital research platform features automated eligibility screening, consenting, and enrollment along with state-of-the-art data security and numerous integrations with services that permit facile survey design and administration, cohort segmentation, and e-mail- and text message-based notifications. Enrolled participants complete health profiles, annual health questionnaires, quarterly hospitalization assessments, and topic-specific cross-sectional ancillary studies (e.g., breast cancer screening study distributed only to those assigned female sex at birth). In addition to participant-generated health data, mailing addresses are automatically geocoded to provide additional details about participant’s environments (e.g., Census tract), and participants can link a variety of mHealth devices including activity trackers, smart scales, and blood pressure cuffs. We are reporting details about our custom web-based research portal elsewhere (manuscript under review).
In conclusion, we developed an iPhone app that enabled the recruitment of an underserved and understudied population into a national online cohort study of SGM adults. Existing software frameworks facilitated app development and the collection of data from a large, diverse national sample at lower cost than traditional cohort studies. Digital engagement features (e.g., Community Forum) empowered participants to become engaged stakeholders in the research development process and provided them with real-time data via visually appealing dashboards to help maintain engagement. Mobile app-based approaches may be successful, convenient, and cost-effective at engaging and recruiting other vulnerable populations into clinical research studies.
We thank Carolyn Hunt, MPA and Micah Lubensky, PhD for their community engagement and program management expertise, respectively. We thank the members of the PRIDEnet Community Partner Consortium, the PRIDEnet Participant Advisory Committee, and The PRIDE Study participants for their passion, dedication, and time to improving SGM health. We thank Salesforce (salesforce.com) for pro bono website design, Maku (makucreative.com) for pro bono video production, THREAD Research (threadresearch.com) for pro bono mobile app development and Heroku cloud services, and Aptible (aptible.com) for pro bono Amazon Web Services cloud computing and database services.
- 1. Grasso C, McDowell MJ, Goldhammer H, Keuroghlian AS. Planning and implementing sexual orientation and gender identity data collection in electronic health records. J Am Med Inform Assoc. 2019;26(1):66–70. Epub 2018/11/18. pmid:30445621.
- 2. Heck NC, Mirabito LA, LeMaire K, Livingston NA, Flentje A. Omitted data in randomized controlled trials for anxiety and depression: A systematic review of the inclusion of sexual orientation and gender identity. Journal of consulting and clinical psychology. 2017;85(1):72–6. Epub 2016/11/16. PubMed Central PMCID: PMCPMC5161712. pmid:27845517
- 3. Flentje A, Bacca CL, Cochran BN. Missing data in substance abuse research? Researchers' reporting practices of sexual orientation and gender identity. Drug Alcohol Depend. 2015;147:280–4. Epub 2014/12/17. PubMed Central PMCID: PMCPMC4297716. pmid:25496705
- 4. CDC/National Center for Health Statistics. About the National Health Interview Survey 2019 [cited 2019 April 11]. Available from: https://www.cdc.gov/nchs/nhis/about_nhis.htm.
- 5. Thompson JH. Director's Blog: Planned Subjects for the 2020 Census and the American Community Survey: U.S. Census Bureau, U.S. Department of Commerce; 2017 [cited 2018 July 5]. Available from: https://www.census.gov/newsroom/blogs/director/2017/03/planned_subjects_2020.html.
- 6. Cahill S, Singal R, Grasso C, King D, Mayer K, Baker K, et al. Do ask, do tell: high levels of acceptability by patients of routine collection of sexual orientation and gender identity data in four diverse American community health centers. PloS one. 2014;9(9):e107104. Epub 2014/09/10. PubMed Central PMCID: PMCPMC4157837. pmid:25198577
- 7. Brooks H, Llewellyn CD, Nadarzynski T, Pelloso FC, De Souza Guilherme F, Pollard A, et al. Sexual orientation disclosure in health care: a systematic review. Br J Gen Pract. 2018;68(668):e187–e96. Epub 2018/01/31. PubMed Central PMCID: PMCPMC5819984. pmid:29378698
- 8. Mattocks KM, Sullivan JC, Bertrand C, Kinney RL, Sherman MD, Gustason C. Perceived Stigma, Discrimination, and Disclosure of Sexual Orientation Among a Sample of Lesbian Veterans Receiving Care in the Department of Veterans Affairs. LGBT Health. 2015;2(2):147–53. Epub 2016/01/21. pmid:26790121.
- 9. Harper GW, Serrano PA, Bruce D, Bauermeister JA. The Internet's Multiple Roles in Facilitating the Sexual Orientation Identity Development of Gay and Bisexual Male Adolescents. Am J Mens Health. 2016;10(5):359–76. Epub 2015/01/15. PubMed Central PMCID: PMCPMC4824670. pmid:25585861
- 10. Valkenburg PM, Sumter SR, Peter J. Gender differences in online and offline self-disclosure in pre-adolescence and adolescence. Br J Dev Psychol. 2011;29(Pt 2):253–69. Epub 2011/01/05. pmid:21199497.
- 11. Pew Research Center. Demographics of Internet and Home Broadband Usage in the United States 2018 [cited 2018 July 5]. Available from: http://www.pewinternet.org/fact-sheet/internet-broadband/.
- 12. Pew Research Center. Demographics of Mobile Device Ownership and Adoption in the United States: Pew Research Center; 2018 [cited 2018 July 5]. Available from: http://www.pewinternet.org/fact-sheet/mobile/.
- 13. Kim H, Bell E, Kim J, Sitapati A, Ramsdell J, Farcas C, et al. iCONCUR: informed consent for clinical data and bio-sample use for research. J Am Med Inform Assoc. 2017;24(2):380–7. Epub 2016/09/04. PubMed Central PMCID: PMCPMC5391727. pmid:27589942
- 14. Pew Research Center. Internet/Broadband Fact Sheet 2018 [cited 2018 November 14]. Available from: http://www.pewinternet.org/fact-sheet/internet-broadband/.
- 15. Pew Research Center. Mobile Fact Sheet [cited 2018 November 14]. Available from: http://www.pewinternet.org/fact-sheet/mobile/.
- 16. Lewis TL, Boissaud-Cooke MA, Aungst TD, Eysenbach G. Consensus on use of the term "App" versus "Application" for reporting of mHealth research. Journal of medical Internet research. 2014;16(7):e174; discussion e. Epub 2014/07/18. PubMed Central PMCID: PMCPMC4129112. pmid:25033233
- 17. McConnell MV, Shcherbina A, Pavlovic A, Homburger JR, Goldfeder RL, Waggot D, et al. Feasibility of Obtaining Measures of Lifestyle From a Smartphone App: The MyHeart Counts Cardiovascular Health Study. JAMA Cardiol. 2017;2(1):67–76. Epub 2016/12/16. PubMed pmid:27973671.
- 18. Chan YY, Wang P, Rogers L, Tignor N, Zweig M, Hershman SG, et al. The Asthma Mobile Health Study, a large-scale clinical observational study using ResearchKit. Nat Biotechnol. 2017;35(4):354–62. Epub 2017/03/14. PubMed Central PMCID: PMCPMC5559298. pmid:28288104
- 19. Chan YY, Bot BM, Zweig M, Tignor N, Ma W, Suver C, et al. The asthma mobile health study, smartphone data collected using ResearchKit. Sci Data. 2018;5:180096. Epub 2018/05/23. PubMed Central PMCID: PMCPMC5963336. pmid:29786695
- 20. Webster DE, Suver C, Doerr M, Mounts E, Domenico L, Petrie T, et al. The Mole Mapper Study, mobile phone skin imaging and melanoma risk data collected using ResearchKit. Sci Data. 2017;4:170005. Epub 2017/02/15. PubMed Central PMCID: PMCPMC5308198. pmid:28195576
- 21. Bot BM, Suver C, Neto EC, Kellen M, Klein A, Bare C, et al. The mPower study, Parkinson disease mobile data collected using ResearchKit. Sci Data. 2016;3:160011. Epub 2016/03/05. PubMed Central PMCID: PMCPMC4776701. pmid:26938265
- 22. Moore S, Tasse AM, Thorogood A, Winship I, Zawati M, Doerr M. Consent Processes for Mobile App Mediated Research: Systematic Review. JMIR Mhealth Uhealth. 2017;5(8):e126. Epub 2017/09/01. PubMed Central PMCID: PMCPMC5597795. pmid:28855147
- 23. Division of Program Coordination Planning and Strategic Initiatives—National Institiutes of Health. Sexual and Gender Minority Research Office 2018. Available from: https://dpcpsi.nih.gov/sgmro.
- 24. Institute of Medicine. The Health of Lesbian, Gay, Bisexual, and Transgender People: Building a Foundation for Better Understanding. Washington, DC: 2011.
- 25. Lambda Legal. When Health Care Isn’t Caring: Lambda Legal’s Survey of Discrimination Against LGBT People and People with HIV. New York, NY: Lambda Legal, 2010.
- 26. James SE, Herman JL, Rankin S, Keisling M, Mottet L, Anafi M. The Report of the 2015 U.S. Transgender Survey. Washington, DC: National Center for Transgender Equality, 2016.
- 27. Pérez-Stable EJ. Sexual and Gender Minorities Formally Designated as a Health Disparity Population for Research Purposes: National Institute on Minority Health and Health Disparities, National Institutes of Health; 2016. Available from: https://www.nimhd.nih.gov/about/directors-corner/message.html.
- 28. Beedle M, Devos M, Sharon Y, Schwaber K, Sutherland J. SCRUM: An Extension Pattern Lan- guage for Hyperproductive Software Development. In: Harrison N, Foote B, Rohnert H, editors. Pattern Languages of Program Design 4. Reading, MA: Addison-Wesley; 2000. p. 637–51.
- 29. GenIUSS Group. Best practices for asking questions to identify transgender and other gender minority respondents on population-based surveys. Los Angles: Williams Institute, 2014.
- 30. SMART Collaboration. Best Practices for Asking Questions about Sexual Orientation on Surveys. Los Angles: Williams Institute, 2009.
- 31. Cahill S, Makadon HJ. Sexual Orientation and Gender Identity Data Collection Update: U.S. Government Takes Steps to Promote Sexual Orientation and Gender Identity Data Collection Through Meaningful Use Guidelines. LGBT Health. 2014;1(3):157–60. Epub 2014/09/01. pmid:26789707.
- 32. Gandhi M, Wang T. The Future of Biosensing Wearables. San Francisco: Rock Health, 2014.
- 33. Pew Research Center. Mobile Fact Sheet: Pew Research Center; 2018 [cited 2018 April 1]. Available from: http://www.pewinternet.org/fact-sheet/mobile/.
- 34. Guo X, Vittinghoff E, Olgin JE, Marcus GM, Pletcher MJ. Volunteer Participation in the Health eHeart Study: A Comparison with the US Population. Sci Rep. 2017;7(1):1956. Epub 2017/05/18. PubMed Central PMCID: PMCPMC5434039. pmid:28512303
- 35. Nguyen KT, Olgin JE, Pletcher MJ, Ng M, Kaye L, Moturu S, et al. Smartphone-Based Geofencing to Ascertain Hospitalizations. Circ Cardiovasc Qual Outcomes. 2017;10(3). Epub 2017/03/23. PubMed Central PMCID: PMCPMC5363280. pmid:28325751
- 36. Joosten YA, Israel TL, Williams NA, Boone LR, Schlundt DG, Mouton CP, et al. Community Engagement Studios: A Structured Approach to Obtaining Meaningful Input From Stakeholders to Inform Research. Academic medicine: journal of the Association of American Medical Colleges. 2015;90(12):1646–50. Epub 2015/06/25. PubMed Central PMCID: PMCPMC4654264. pmid:26107879
- 37. Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, Becker LB, et al. Crowdsourcing—harnessing the masses to advance health and medicine, a systematic review. Journal of general internal medicine. 2014;29(1):187–203. Epub 2013/07/12. PubMed Central PMCID: PMCPMC3889976. pmid:23843021
- 38. Swan M. Crowdsourced health research studies: an important emerging complement to clinical trials in the public health research ecosystem. Journal of medical Internet research. 2012;14(2):e46. Epub 2012/03/09. PubMed Central PMCID: PMCPMC3376509. pmid:22397809
- 39. US Department of Health and Human Services. Revised Common Rule Washington, DC: Office for Human Research Protections; 2018 [cited 2018 April 1]. Available from: https://www.hhs.gov/ohrp/regulations-and-policy/regulations/finalized-revisions-common-rule/index.html.
- 40. Sertkaya A, Wong HH, Jessup A, Beleche T. Key cost drivers of pharmaceutical clinical trials in the United States. Clin Trials. 2016;13(2):117–26. Epub 2016/02/26. pmid:26908540.