https://orcid.org/0000-0003-0230-473X
Figures
Abstract
Background
Asian-Americans are one of the most understudied racial/ethnic minority populations. To increase representation of Asian subgroups, researchers have traditionally relied on data collection at community venues and events. However, the COVID-19 pandemic has created serious challenges for in-person data collection. In this case study, we describe multi-modal strategies for online recruitment of U.S. Vietnamese parents, compare response rates and participant characteristics among strategies, and discuss lessons learned.
Methods
We recruited 408 participants from community-based organizations (CBOs) (n = 68), Facebook groups (n = 97), listservs (n = 4), personal network (n = 42), and snowball sampling (n = 197). Using chi-square tests and one-way analyses of variance, we compared participants recruited through different strategies regarding sociodemographic characteristics, acculturation-related characteristics, and mobile health usage.
Results
The overall response rate was 71.8% (range: 51.5% for Vietnamese CBOs to 86.6% for Facebook groups). Significant differences exist for all sociodemographic and almost all acculturation-related characteristics among recruitment strategies. Notably, CBO-recruited participants were the oldest, had lived in the U.S. for the longest duration, and had the lowest Vietnamese language ability. We found some similarities between Facebook-recruited participants and those referred by Facebook-recruited participants. Mobile health usage was high and did not vary based on recruitment strategies. Challenges included encountering fraudulent responses (e.g., non-Vietnamese). Perceived benefits and trust appeared to facilitate recruitment.
Conclusions
Facebook and snowball sampling may be feasible strategies to recruit U.S. Vietnamese. Findings suggest the potential for mobile-based research implementation. Perceived benefits and trust could encourage participation and may be related to cultural ties. Attention should be paid to recruitment with CBOs and handling fraudulent responses.
Citation: Vu M, Huynh VN, Bednarczyk RA, Escoffery C, Ta D, Nguyen TT, et al. (2021) Experience and lessons learned from multi-modal internet-based recruitment of U.S. Vietnamese into research. PLoS ONE 16(8): e0256074. https://doi.org/10.1371/journal.pone.0256074
Editor: Marcel Pikhart, University of Hradec Kralove: Univerzita Hradec Kralove, CZECH REPUBLIC
Received: April 28, 2021; Accepted: July 31, 2021; Published: August 13, 2021
Copyright: © 2021 Vu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data that support the findings of this study are available upon reasonable request and with the approval of Emory University Institutional Review Board (IRB 00111688). The data are not publicly available as it contains information that could compromise the privacy of research participants. In particular, given that participants are Vietnamese parents of adolescents (a small, specific population), and that the dataset contains their zip code, making the data available can risk the possibility of participants being identified. Please see more information here: http://www.irb.emory.edu/documents/phi_identifiers.pdf Please contact the lead author, MV, as well as the Institutional Review Board at Emory University, with any request for data access (milkie.vu@emory.edu and IRB@emory.edu).
Funding: This work is supported by the American Psychological Foundation 2019 Visionary Grant and the American Association for Cancer Education 2019 Grant in Research, Education, Advocacy, and Direct Service (READS), the Grants-in-Aid program from the Society for the Psychological Study of Social Issues, the Professional Development Support Fund at Emory University, and the Healthcare Innovation Program Student-Initiated Project Grant at the Georgia Clinical & Translational Science Alliance (CTSA). The Georgia CTSA is supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002378. Our data collection receives support from the Center for AIDS Research at Emory University (P30AI050409). Ms. Vu is supported by the US National Cancer Institute (5F31CA243220-02), a 2020-2021 PEO Scholar Award, and the 2020-2021 Student Fellowship in Patient Engagement from the Society of Public Health Education. Dr. Berg is supported by the US National Cancer Institute (R01CA215155-01A1; R01CA179422-01; R01CA239178-01A1), the US Fogarty International Center/National Institutes of Health (1R01TW010664-01), and the US National Institute on Environmental Health Science/Fogarty International Center (D43ES030927-01). Dr. Bednarczyk is supported in part by the US National Cancer Institute (1R37CA234119-01). Open-access publication support was made possible in part by the Research Reimbursement Award from grant 3P30CA076292 (Geographic Management of Cancer Health Disparities Program (GMaP) Region 2) funded by the National Cancer Institute.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Asians are the fastest growing racial/ethnic groups in the U.S. [1], with a population of more than 18 million as of 2019 (6% of the total population) [2], which is projected to grow to 41 million (9% of the total population) by 2050 [3]. Despite their growing numbers, Asian-Americans remain one of the most understudied racial/ethnic minority groups [4–7]. For example, studies including Asian-American participants totaled 0.01% of MEDLINE/PubMed articles published between 1966 and 2000 [4] and 0.17% of the NIH-funded clinical research budget between 1992 and 2018 [5]. Moreover, when Asian-American participants are included, research typically aggregates Asian subgroups into one single category [6–8] instead of providing data for separate subgroups. This trend masks important subgroup differences, as Asian-Americans are extraordinarily heterogeneous, comprising of people from over 30 countries and speaking over 100 languages/dialects, with diverse socioeconomic status, religious and cultural backgrounds, immigration histories, and patterns of health services utilization [8–11].
While in recent years, more large-scale or population-based datasets and registries (e.g., the National Health and Nutrition Examination Survey or the National Health Interview Survey) have incorporated questions assessing separate Asian subgroups in their survey, due to small sample sizes, research using these datasets has rarely reported disaggregated subgroup outcomes [6, 7]. Moreover and critically, with a few notable exceptions (e.g., the National Latino and Asian American Study, the California Health Interview Survey), large-scale or population-based surveys often are not available in Asian languages, thus excluding the participation of those with lower English proficiency [6, 7]. According to the 2019 American Community Survey, 74% and 31% of the Asian population in the U.S. reported that they spoke a language other than English and that they spoke English less than "very well," respectively [2]. The literature has demonstrated several differences between Asians with and without lower English proficiency. In particular, compared to Asians with higher English proficiency, Asians with lower English proficiency have higher psychological distress [12, 13], higher unmet healthcare needs [14], poorer quality of life and health [15, 16], more limited access to care [14] and are less likely to adhere to screening guidelines [17] or receive and use health services [13, 18].
To overcome these issues, many researchers interested in Asian-American health have relied on the use of community-engaged research [19, 20]. Typically, researchers partner with community-based organizations (CBOs) serving Asian populations to collect data from clients using CBO services or attending community venues and events [7, 21–25]. This approach has allowed for the administration of surveys in diverse languages and the collection of adequate sample sizes from different Asian subgroups, thereby providing important representation and understanding of Asian-American health behaviors and outcomes.
The ongoing COVID-19 pandemic in the U.S., however, represents a serious challenge to research employing in-person modes of data collection. A growing emphasis is being placed on transitioning to online data collection [26, 27]. To successfully execute these methods in Asian communities, an understanding of the success and challenges of different recruitment strategies is critical. Yet, only a handful of published studies to date have described the process of online recruitment and data collection in Asian-Americans populations in detail [28–35].
Existing literature found that, in general, Asian participants recruited from Internet sources were more likely to be younger, U.S.-born, fluent in English, and more educated than Asian participants recruited from non-Internet sources [28, 29]. Researchers also highlighted several issues in the online recruitment of Asian participants, notably difficulties in establishing trust and transparency between researchers and participants as well as participants’ concerns about the security of information transmitted through the Internet [30]. In addition, challenges in creating rapport with "gatekeepers" (e.g., webmasters, Facebook group moderators, directors of Asian community groups) were also a barrier to recruitment [31, 32]. On the other hand, having a culturally matched research team (e.g., same ethnicity or strong cultural ties) [31, 33] and clear communication and outline of research procedures [31] were facilitators to recruitment.
The primary goal of this paper is to contribute to the body of literature on strategies for online data collection efforts in Asian communities by presenting a case study of a research project with U.S. Vietnamese (i.e., those living in the U.S. and identifying as Vietnamese). U.S. Vietnamese represent the fourth largest Asian-American subgroup, with a population of more than 1.8 million as of 2019 [2]. According to the 2019 American Community Survey, U.S. Vietnamese have lower English proficiency, median household income, and education and are more likely to live in poverty compared to other Asians [2]. These disadvantages reflect potential barriers to health services utilization and health research participation, thus highlighting the need to focus on this specific subgroup (rather than Asians as an aggregate).
Indeed, a body of literature has documented lower utilization of various health services among U.S. Vietnamese when compared to other major Asian subgroups. For example, compared to Chinese and Korean populations in the U.S., fewer U.S. Vietnamese have a personal doctor as a main healthcare provider [36], have ever been tested for Hepatitis B [37], or have ever had colorectal screening [38]. Moreover, a higher proportion of U.S. Vietnamese women have never had a Pap smear compared to Chinese and Cambodian women [39]. Additionally, fewer U.S. Vietnamese women on average have ever sought mental health services compared to Chinese women or Filipino women in the U.S [40].
The data presented in this paper are from a study investigating multilevel factors influencing U.S. Vietnamese parents’ HPV vaccine initiation for their adolescent children. U.S. Vietnamese have higher cervical cancer incidence rates than other racial/ethnic groups [41–44]. A solution to reduce cervical cancer burden in this population is HPV vaccination. Unfortunately, our understanding of HPV vaccination among U.S. Vietnamese is limited, partly because large national probability surveys on HPV vaccine uptake in the U.S. typically do not supply separate statistics for U.S. Vietnamese. For example, the National Immunization Survey–Teen and the Behavioral Risk Factor Surveillance System, both of which provide estimates on HPV vaccine uptake, aggregate "Vietnamese" under the category of "Asians" and do not provide disaggregated Asian subgroup data in public-use datasets [45–47]. The Health Information National Trends Survey (HINTS), which examines knowledge of the HPV vaccine, reports separate data for Vietnamese in some cycles but not all; the sample sizes for Vietnamese in the public-use datasets are also relatively small (~1%) which can hinder meaningful statistical modeling [48]. Importantly, none of these surveys includes Vietnamese language versions of their questionnaires, which can be a major obstacle to research participation for U.S. Vietnamese with low English proficiency. In the context of research with HPV vaccination, this issue is particularly problematic, given that U.S. Vietnamese women with lower English proficiency also had lower HPV vaccine uptake [49]. Consequently, excluding those with low English proficiency could bias estimates of HPV vaccine uptake and lead to an incomplete understanding of barriers underlying HPV vaccination in this population.
In this study, we describe multiple strategies for online recruitment of U.S. Vietnamese parents and compare response rates among strategies. We also compare several characteristics (e.g., sociodemographic, acculturation, and mobile health usage) of participants drawn from different strategies. Moreover, we discuss lessons learned from our experiences conducting online recruitment, including handling fraudulent responses and how perceived benefits and trust can encourage participation.
Materials and methods
Eligibility criteria and research procedure
Eligibility criteria included: 1) self-identified as Vietnamese; 2) having lived in the U.S. for at least 12 months; 3) able to read either Vietnamese or English; and 4) having at least one child aged 9 to 18 and living in the same household with the child. Only one parent per household was allowed to participate. Participants first completed an online eligibility screener (administered via SurveyGizmo). Those eligible were directed to a webpage with a study description and a consent form. Consenting individuals indicated consent via a link and were directed to the online survey (via SurveyGizmo). The survey took approximately 60 minutes to complete. We limited duplicate responses by permitting only one response per IP address. Survey participants were compensated with a $30 Amazon gift card. We also selected a subset of 32 participants from those who completed the survey to participate in semi-structured interviews to further explore findings from the survey. We obtained electronic/online consent prior to the interview for each participant. Interviewed participants were compensated with a $60 Amazon gift card. Participants could choose to do the survey and interview in either Vietnamese or English. The Institutional Review Board at Emory University approved of this study.
Recruitment procedures: Survey
Survey recruitment took place virtually from April to December 2020. We utilized several venues for survey recruitment: community-based organizations, Facebook groups, listservs, the personal network of the first author, and snowball sampling [50]. Convenience sampling was used for all recruitment methods.
Community-based organizations.
We contacted CBOs serving Asians and/or Vietnamese populations in the U.S., introduced the research study and eligibility criteria, and asked the CBOs to send information about our research to their members. Additionally, we contacted Vietnamese Students Associations (VSA) at colleges and universities and requested VSA students to have their parents contact us if their parents met the eligibility criteria and were interested.
Fig 1 details the recruitment process with CBOs.
In total, we contacted 115 Vietnamese-serving CBOs, 91 Asian-serving CBOs, and 114 VSAs. Among those, 40 Vietnamese-serving CBOs, 20 Asian-serving CBOs, and 13 VSAs agreed to disseminate information about the study to their members. Six Vietnamese-serving CBOs and one Asian-serving CBO declined to participate in the project; reasons for nonparticipation included lack of members in the targeted demographics (n = 2), no email addresses of members on file (n = 1), insufficient staff capacity due to COVID-19 (n = 2), and insufficient technological knowledge among members to navigate online surveys (n = 2). In total, 51 participants from 19 Vietnamese-serving CBOs, nine participants from four Asian-serving CBOs, and eight participants from five VSAs completed the surveys.
Facebook groups.
Through a search on Facebook with terms such as "Vietnamese in the U.S.", we identified Facebook groups that operated in Vietnamese and focused on topics that might be of interests to U.S. Vietnamese parents of adolescents (e.g., general discussion of life in the U.S., parenting advice for those with children in the U.S., immigration and visa applications, etc.). We posted information about the research study and eligibility criteria in 12 Facebook groups, which ranged in membership from 50 to approximately 58,000. Ninety-seven participants from these 12 Facebook groups completed the surveys.
Listservs.
We posted information about the research study and eligibility criteria to two listservs, one focusing on Vietnamese Studies and the other on Vietnamese-related events in California. Four participants (two from each listserv) contacted the study team and completed the surveys.
Personal network.
The first author is a U.S. Vietnamese, fully fluent in Vietnamese and English, and has several years of experience partnering with Vietnamese and Asian-serving CBOs in different U.S. regions and conducting research on Vietnamese health. She disseminated information about the research study and eligibility criteria to her personal network via social media and emails. She requested those who were eligible and interested to contact the study team. She also asked others in her personal network to refer eligible and interested acquaintances to the study team. With this method, 42 participants completed the surveys.
Snowball sampling.
Using snowball sampling, we asked participants who completed the surveys to refer their eligible and interested acquaintances to the study team. A total of 197 participants completed the surveys through snowball sampling recruitment, in which two were referred from previous participants recruited through Vietnamese-serving CBOs, two referred from previous participants recruited through VSAs, 77 referred from those recruited through Facebook groups, and 116 referred from those recruited through the first author’s personal network.
Survey measurements.
The survey was available in Vietnamese and English. For the surveys, we used the Brislin’s back-translation method [51], an iterative translation process involving an independent translation of survey questions into Vietnamese and back-translation into English by two different translators, and then reviewed by the first author (who is fully fluent in both languages) and by approximately 10 Vietnamese native speakers to ensure comprehensibility.
For sociodemographic characteristics, we assessed a participant’s age, sex, highest education level, combined household income, and child’s country of birth.
For acculturation-related characteristics, we assessed a participant’s percentage of life in the U.S. and their ability to understand medical information in English. In addition, we asked about participants’ zip codes and used data from the 2019 American Community Survey [52] to construct two variables capturing zip code-level percentage of Asians and percentage of Vietnamese.
We also used the Asian American Multidimensional Acculturation Scale (15 items for each culture) [53] to separately assess cultural identity (6 items), cultural knowledge (3 items), language (4 items), and food consumption (2 items) for Vietnamese culture and American culture. Examples of questions included: "How much do you interact and associate with [Vietnamese people]/[typical American people]?"; "How much do you actually practice the traditions and keep the holidays of [Vietnamese culture]/[mainstream American culture]?"; "How well do you speak [Vietnamese]/[English]?"; and "How often do you actually eat [Vietnamese food]/[the food of mainstream American culture]?" Each item was scored on a 6-point Likert scale (0 –Not very much to 5 –Very much). A higher subscale score, derived as an average across subscale items, indicated higher cultural identity, cultural knowledge, language ability, or food consumption.
For mobile health usage, we used items from the Health Information National Trends Survey 5 Cycle 2 [54] to assess participants’ daily use of a home computer or mobile device to access the Internet, past 12-month use of electronic devices to look for health-related information for themselves or their children, and past 12-month use of emails or the Internet to communicate with a doctor. Given the critical role of digital technologies in facilitating data collection during the COVID-19 pandemic, these data can indicate whether access to the Internet (and, consequently, ability to participate in online data collection) differs between participants from different recruitment methods and potentially explain differences in response rates. In addition, they can also provide information on the possible receptiveness of participants to research and interventions leveraging mobile health technologies.
Recruitment procedure: Interviews
Between November 2020 and February 2021, we invited a subset of participants who had already completed the survey to participate in in-depth, semi-structured qualitative interviews. We purposively sampled participants and stratified the sample of interviewees by their adolescent child’s sex (female versus male) and HPV vaccination status. For selected potential interviewees, we sent each of them an email with information about the interviews and a consent form. Depending on participants’ preferences, interviews were conducted in either Vietnamese or English and via telephone or the Zoom platform. Of 38 invited participants, 32 agreed to participate in the interviews (84% response rate). Among the 32 interviewed participants, 7 (21.9%) were recruited through CBOs, 1 (3.1%) through listservs, 10 (31.3%) through Facebook groups, 1 (3.1%) through personal network, 8 (25.0%) referred from those recruited through Facebook groups, and 5 (15.6%) referred from those recruited through the first author’s personal network.
Interview questions.
Interview questions explored participants’ sources of information about the HPV vaccine. While these questions were not designed specifically to inquire participants’ thoughts about the research project, several participants brought up the research project as a resource that motivated them to learn more about the HPV vaccine.
Data analysis
We used the American Association for Public Opinion Research’s (AAPOR) response rate calculator version 4.1 to determine response rates [55]. Chi-square tests and one-way analyses of variance were used to compare differences in sociodemographic characteristics, acculturation-related characteristics, and mobile health usage among recruitment methods. To further understand significant differences between recruitment methods, the Tukey test was used for post-hoc analyses of differences in pairs of means, while a Tukey-type procedure was used for post-hoc analyses of differences in pairs of proportions [56]. Analyses were conducted in SAS 9.4 and Stata 15.1 and alpha levels were set at 0.05. Qualitative interview transcripts were uploaded to MAXQDA 2020. We used a hybrid approach of qualitative thematic analysis, which incorporated both 1) a deductive a priori template of codes and themes included in the interview guide and relevant literature and 2) a data-driven inductive approach that identified new emergent themes [57]. Two coders (MV and DT) coded all transcripts independently. Coding results were compared and any discrepancies were resolved through discussion.
Results
Response rates
Table 1 summarizes the number of completed surveys as well as response rates with different venue.
The overall response rate was 71.8% (range: 51.5% for Vietnamese CBOs to 86.6% for Facebook groups). Fig 2 shows the recruitment flowchart for the entire sample.
Fig 3 shows the percentages of participants in the survey sample from different recruitment methods.
Sample characteristics
Sociodemographic characteristics.
We found differences in all sociodemographic characteristics among recruitment methods (Table 2).
On average, participants recruited through CBOs were the oldest (45 years). Post-hoc analyses show that, compared to those recruited through CBOs, a significantly higher proportion of those recruited through Facebook groups reported that they had a Bachelor’s degree or higher (66.2% versus 87.6%) and that their child was born outside of the U.S. (22.1% versus 70.1%). The distribution of education appeared similar between those recruited through Facebook groups, snowball sampling via Facebook groups, and the first author’s personal network.
Acculturation-related characteristics.
Except for a Vietnamese acculturation subscale (food consumption), we found significant differences in all acculturation-related characteristics among recruitment methods (Table 3).
On average, Facebook groups-recruited participants and those referred by them had lived in the U.S. for the shortest durations, had the highest scores on the Vietnamese cultural knowledge and Vietnamese language subscales, and had the lowest scores on the American cultural identity, American cultural knowledge, and English language subscales. On average, personal network-recruited participants and those referred by them had the highest scores on the English language subscale and lived in zip codes with the lowest percentages of Vietnamese. Post-hoc analyses show that, compared to all other groups, participants recruited through CBOs had lived in the U.S. for the longest percentage of their lifetime (55.94) and had the lowest scores on the Vietnamese language subscale (3.76). Compared to those recruited through CBOs, those recruited through Facebook groups lived in zip codes with significantly lower percentages of Vietnamese (7.37 versus 3.24) and Asians (19.92 versus 12.70).
Mobile health usage.
We did not find any differences among recruitment methods regarding mobile health usage (Table 3). Overall, 77% reported daily use of a home computer to access the Internet and 92% reported daily use of a mobile device to access the Internet. Additionally, 89% and 88% of participants reported past 12-month use of a computer, smartphone, or other electronic means to look for health information for themselves and their children, respectively. A majority (60%) also reported past 12-month use of email or the Internet to communicate with doctors.
Lessons learned in conducting online recruitment
Handling fraudulent responses in online recruitment.
At the beginning of data collection, we created public, non-unique links to the screener that participants could fill out to determine their eligibility. Participants would be automatically redirected to the main survey if they met all the eligibility criteria. However, we encountered problematic responses after these links were publicly shared on Facebook. A closer examination of the initial responses and a review of the literature regarding fraudsters in online survey recruitment [58] led us to conclude that a majority of the initial responses were fraudulent.
These potentially fraudulent responses often followed these patterns: The provided email address generally indicated that both the participant’s first and last name were not Vietnamese (e.g., Robert Anderson). Occasionally, the IP address was outside of the U.S. despite the survey response indicating that they lived in the U.S. Responses from the same participant about their sociodemographic characteristics (e.g., age, zip code) in the screener and in the main survey were considerably different. In addition, we detected illogical patterns of responses (e.g., providing a year of immigration to the U.S. that was earlier than the year participant was born or much earlier than the end of the Vietnam War; responding that they had lived in Vietnam for the majority of their lives but indicated little fluency in Vietnamese).
Prior to launching the study, based on research team members’ previous experiences conducting Internet-based surveys, we had configured the survey to only accept one response per IP address. All the potentially fraudulent responses had different, unique IP addresses. After detecting these responses, we then implemented the reCAPTCHA system embedded in SurveyGizmo/Alchemer and asked participants to check a box that said, "I am not a robot." We still, however, received problematic responses after implementing the reCAPTCHA system.
To ensure reliable data quality, we then decided to not continue with using public, non-unique links to the screener. Instead, we asked CBOs to disseminate information about our study and have eligible members contact us via emails. Similarly, with recruitment through Facebook groups, listservs, and snowball sampling, we asked participants to contact us directly. Study team members who were fluent in English and Vietnamese monitored the mailboxes. For participants who contacted us, we first verified where they learned of the study (e.g., certain CBOs, Facebook groups, or previous participants). We also reiterated the eligibility criteria. For those who confirmed where they learned of the study and that they met the criteria, we privately emailed them unique, one-time use links to the screeners.
Perceived benefits and trust as facilitators of research participation.
Several participants remarked that they found our research project very meaningful. Specifically, many participants had limited knowledge of the HPV vaccine before completing the survey and reported that the project made them more aware of the benefits of the vaccine. For example, one interviewee stated: "I got my son vaccinated because I participated in your project and looked into the vaccine. Your research was a valuable source for me. Before that, I’d heard my friends talking about getting this vaccine to prevent cervical cancer, but it was all vague for me… So, your project came just in time." Another echoed the same sentiment: "Before filling out your survey, I didn’t know that boys should get the HPV vaccine too." In addition, a participant said: "This project is useful for me as it has helped me pay more attention to cervical cancer. I have heard about cervical cancer, but I have not paid much attention to the vaccine".
Snowball sampling appears to be a successful recruitment method. Many participants who found the project meaningful were enthusiastic in referring their eligible acquaintances, and several of those who were referred mentioned that they felt motivated to participate because their acquaintances had completed the survey. The information about the project that was included in introductory emails and messages also helped generate trust. The first author had included her name, credentials, past experiences with research in Vietnamese health, study funders, IRB approval, and contact information, in addition to study information (e.g., purpose, eligibility criteria, procedure, and compensation). Some participants noted that having such comprehensive information helped them trust that the responses they were providing would be used for research and health programs for Vietnamese communities and not for other purposes (e.g., for-profit purposes).
Discussion
In this paper, we described the results of different online recruitment strategies of U.S. Vietnamese for a health research project as well as lessons drawn from conducting this project. To our knowledge, this paper is the first to present a variety of online recruitment strategies for U.S. Vietnamese and compare characteristics of participants among methods. While the overall response rate was high (72%), response rates varied among different recruitment methods. In addition, we found significant between-group differences in all sociodemographic characteristics and almost all acculturation-related characteristics. Mobile health use, however, appeared similar among recruitment methods. A considerable challenge during the recruitment process was encountering fraudulent responses. Moreover, participants’ perceived benefits and trust encouraged their research participation.
In general, response rates from CBOs were the lowest (e.g., 52% for Vietnamese-serving CBOs). While we made efforts to contact 320 CBOs (including VSAs) for recruitment, we received completed responses from participants from only 28 of those (i.e. less than 10%). A previous study noted similar difficulties in online recruitment of Korean-Americans through communities and groups catering to this population [32]. Of 422 Korean-American communities and groups contacted, only 72 agreed to announce the study through their websites and email lists, and only 13 participants were recruited into the study [32]. Research using online recruitment of Asian populations, especially those planning for a moderate sample size, should contact a high number of CBOs given these low observed response rates. Establishing relationships with gatekeepers (e.g., webmasters of online groups, directors of CBOs) may also be essential to facilitating recruitment [31]. Two studies conducted during the COVID-19 pandemic (one with residents in rural areas in Mozambique [59] and one with mothers who had experienced domestic violence in Canada [60]) have also found that engagement with CBOs and stakeholders continued to be crucial for participant recruitment, even when researchers moved to remote data collection. At the same time, the pandemic likely created great disruptions to the activities of CBOs and exacerbated healthcare and social needs from community members [61], all of which would put a strain on the operations of CBOs. This situation may explain the low response rates from CBOs in our study and researchers should be mindful of the limited time and effort that CBOs may be able to offer for research recruitment during a time of crisis.
In contrast to recruitment with CBOs, we received completed responses from participants from all Facebook groups where we advertised the study, with a high overall response rate (87%). Moreover, response rates for snowball sampling recruitment from either Facebook or the first author’s personal network were also high (76% and 73%, respectively). These results suggest the high feasibility of recruitment through Facebook groups or snowball sampling for U.S. Vietnamese populations during the COVID-19 pandemic.
Participants recruited through CBOs had lived in the U.S. for the longest duration and had the lowest Vietnamese language ability. When reviewing websites of Vietnamese-serving CBOs during recruitment, we noted that many were established between 1976 and mid-1990s. This period followed the Vietnam War, when hundreds of thousands of Vietnamese refugees left for the U.S.; it was also prior to the bilateral normalization of U.S.–Vietnam relations [62, 63]. Participants recruited through CBOs may reflect the demographic of the groups that emigrated out of Vietnam in the earlier waves. Compared to those recruited through Facebook groups, those recruited through CBOs also lived in areas with higher concentration of Vietnamese and Asians, which are likely also places where CBOs are based. Moreover, given that Facebook groups that we selected operated in Vietnamese and were geared towards advice for Vietnamese living in the U.S., it is understandable that participants recruited through Facebook groups had lived in the U.S. for shorter durations, had higher Vietnamese language ability and lower English language ability, and had lower scores on American acculturation subscales.
Researchers looking to replicate these recruitment methods for future research studies in Vietnamese populations should be mindful of subgroup variability in terms of sociodemographic characteristics and acculturation-related characteristics, even within a cultural population (e.g., U.S. Vietnamese). For health promotion efforts in particular, these differences point to the need to consider tailored approaches to cultural appropriateness [64]. Knowing a person’s educational background, level of Vietnamese cultural identity, degree of Vietnamese cultural knowledge, and ability to speak Vietnamese or English will facilitate crafting effective and resonating health messages [64]. For example, in our sample, the majority of Facebook groups-recruited participants and those referred by them likely would require messages and programs in Vietnamese that leverage Vietnamese cultural information. In contrast, a considerable number of those recruited through CBOs and the first author’s personal network may prefer information in Vietnamese that incorporates Vietnamese culture but may also be open to receiving information in English given their higher English proficiency. Moreover, findings from our study highlight the complexity of the relationships between participants’ Vietnamese and American acculturation degrees, and consequently, the importance of including multidimensional acculturation-related measurements [65, 66]. For example, lower Vietnamese acculturation does not necessarily mean higher American acculturation (and vice-versa), as on average, those recruited through the first author’s personal network had the highest scores for both Vietnamese and American cultural identity subscales.
To our knowledge, no research has examined mobile health usage among U.S. Vietnamese. Our study found no differences in mobile health usage among recruitment methods. Overall, more participants reported daily use of a mobile device than daily use of a computer to access the Internet, which has implications for the delivery mediums of research and interventions. In addition, nearly 9 out of 10 of participants reported past 12-month use of a computer, smartphone, or electronic means to look for health information for themselves and their children. This finding points to the potential for implementing mobile-based interventions to address health issues in U.S. Vietnamese.
We encountered fraudulent responses in the beginning, which we resolved by switching to manually verifying participants via emails first and then sending them unique screener links. While we believe this process helped ensure reliable data quality, it was also time-consuming and may have discouraged some participants (e.g., due to the need to send emails before they could participate in the study). Previous research with non-Asian populations has raised issues with fraudulent responses in online recruitment [67–69]. A growing body of literature is focusing on strategies to implement quality checks and identify fraudulent or automated responses in online survey research [58, 70–74]. These threats may be mitigated through methods such as assessing response coherence and consistency, including open-ended questions, and adding attention and logic checks [58, 70–74]. Aside from questionnaire data, researchers could also examine other information provided (e.g., emails and phone numbers) and look for similar/duplicate items or validate participants via phone prior to accepting the data [58]. Having cultural knowledge of the targeted population was useful for our fraud detection, as we noticed patterns of non-Vietnamese names in email addresses and illogical responses.
Previous studies with Asian communities in the U.S. (including both online and non-online recruitment) have highlighted the value of cultural concordance in facilitating recruitment. For example, Im and colleagues described how they were able to contact Chinese-American communities and groups only after working with a Chinese-American research assistant, but not research assistants of other Asian subgroups [31]. Mukherjea and colleagues also noted that respondents in their study had preferences for interviewers from the same South Asian regions, and that effective "cultural research brokers" are those that not only visibly and linguistically represented the community but also possessed credentials (e.g., advanced degrees) held in high regard by community members [21]. Other studies have also reported a preference for culturally matched research personnel in Hawaiian and Filipino populations [75, 76]. In our study, aside from being U.S. Vietnamese and fluent in the language, the first author had shared her credentials and past experiences with research in Vietnamese health with participants, which increased some participants’ trust in the project. In addition, a few also perceived gaining more knowledge about the HPV vaccine as an indirect benefit from study participation, which made them enthusiastic about referring others to the study. Recent systematic reviews have also documented knowledge gain and wanting to help others as facilitators of participation in research [77, 78].
While, overall, our project had relative success in recruiting participants remotely during the COVID-19 pandemic, we note that remote data collection that relies on digital technologies is not always appropriate in every context and situation [79–81]. Researchers should consider how the pandemic may have affected the participants’ well-being and their ability to participate in research [81]. In addition, while our participants did not voice concerns about technological barriers, multiple studies have demonstrated inequities in access to and ability to use mobile devices (e.g., smartphones, tablets, or laptops) for research participation along with challenges related to poor connectivity. Specifically, such issues have been documented in studies conducted during the COVID-19 pandemic with older adults in the U.S. and Italy [82, 83], youth volunteers in Sudan [84], and adults in Malawi and the United Kingdom [81, 84].
Limitations
Our results should be interpreted in light of a few limitations. We did not collect data on sociodemographic characteristics, acculturation, or mobile health usage of those who refused participation; therefore, we could not detect whether there were significant differences between those who chose to participate and those who did not. Additionally, we did not collect data on reasons why individuals refused participation.
While recruiting participants from existing social networks of research personnel has been used in several past studies with minority populations [69, 85, 86], including Vietnamese [87], given the prevalence of homophily [88] in social networks (e.g., “tendency for friendships to form between those who are alike in some designated respect”) [89], it is possible that personal network-recruited participants would share several similarities to the first author. For example, homophily may explain why personal network-recruited participants reported on average the highest level of English language proficiency. Future studies using similar methods should take into account the characteristics of the recruiters when determining the generalizability of research findings. Likewise, snowball sampling can lead to an overrepresentation of participants who share similarities or have larger social networks, consequently creating a sample that is unbalanced in several characteristics [50, 86, 90]. Future research can consider employing respondent-driven sampling, which uses the chain-referral sample to make estimates about the social networks, and then uses the information about the social networks to derive population proportions [90, 91]. Moreover, due to limited resources, for recruitment using Facebook, we posted about the study in relevant Facebook groups instead of using targeted Facebook advertising. While our approach had the advantage of bearing no direct costs, only members of the chosen Facebook groups had access to information about the study. The use of targeted Facebook advertising could have increased our reach to users who were not members of these groups.
Finally, the lack of sampling frames or population estimates for U.S. Vietnamese parents of adolescents prevents us from comparing estimates of our sample and those from the population. Our sample of parents of adolescents, on average, has higher socioeconomic status (e.g., income and education) compared to the 2019 American Community Survey data for all Vietnamese in the U.S. [52].
Conclusions
This paper presents a case study of multi-modal internet-based recruitment of U.S. Vietnamese into research. We found differences in sociodemographic and acculturation-related characteristics of participants drawn through different sources. Participants reported high mobile health usage among recruitment methods. Recruitment through Facebook groups and snowball sampling yielded high response rates, suggesting that these are feasible strategies to recruit U.S. Vietnamese. Perceived benefits and trust appear to be facilitators to study recruitment. Special attention should be paid to recruitment with CBOs and handling fraudulent responses. Our findings can inform future multi-modal data collection efforts for research with U.S. Vietnamese, Asian communities in the U.S., and other understudied populations in an international context.
References
- 1.
Pew Research Center. Key facts about Asian Americans, a diverse and growing population. 2017 [cited 22 Mar 2020]. Available: https://www.pewresearch.org/fact-tank/2017/09/08/key-facts-about-asian-americans/
- 2.
United States Census Bureau. 2019 American Community Survey 1-Year Estimates Total Population. 2019 [cited 27 Sep 2020]. Available: http://data.census.gov
- 3.
Passell JS, Cohn D. U.S. Population Projections: 2005–2050. In: Pew Research Center Hispanic Trends [Internet]. 2008 [cited 2 Jan 2021]. Available: https://www.pewresearch.org/hispanic/2008/02/11/us-population-projections-2005-2050/
- 4. Ghosh C. Healthy People 2010 and Asian Americans/Pacific Islanders: Defining a Baseline of Information. Am J Public Health. 2003;93: 2093–2098. pmid:14652340
- 5. Ðoàn LN, Takata Y, Sakuma K-LK, Irvin VL. Trends in Clinical Research Including Asian American, Native Hawaiian, and Pacific Islander Participants Funded by the US National Institutes of Health, 1992 to 2018. JAMA Netw Open. 2019;2: e197432. pmid:31339543
- 6. Holland AT, Palaniappan LP. Problems With the Collection and Interpretation of Asian-American Health Data: Omission, Aggregation, and Extrapolation. Ann Epidemiol. 2012;22: 397–405. pmid:22625997
- 7. Islam NS, Khan S, Kwon S, Jang D, Ro M, Trinh-Shevrin C. Methodological Issues in the Collection, Analysis, and Reporting of Granular Data in Asian American Populations: Historical Challenges and Potential Solutions. J Health Care Poor Underserved. 2010;21: 1354–1381. pmid:21099084
- 8. Srinivasan S, Guillermo T. Toward improved health: disaggregating Asian American and Native Hawaiian/Pacific Islander data. Am J Public Health. 2000;90: 1731–4. pmid:11076241
- 9. Thompson CA, Gomez SL, Hastings KG, Kapphahn K, Yu P, Shariff-Marco S, et al. The Burden of Cancer in Asian Americans: A Report of National Mortality Trends by Asian Ethnicity. Cancer Epidemiol Biomarkers Prev. 2016;25: 1371–1382. pmid:27694108
- 10.
Hamilton AS, Wu AH, Stram DO. Resources and Methods for Studying Cancer Among Asian Americans. In: Wu AH, Stram DO, editors. Cancer Epidemiology Among Asian Americans. Springer International Publishing; 2016. pp. 1–18. https://doi.org/10.1007/978-3-319-41118-7
- 11. Vu M, Berg CJ, Escoffery C, Jang HM, Nguyen TT, Travis L, et al. A systematic review of practice-, provider-, and patient-level determinants impacting Asian-Americans’ human papillomavirus vaccine intention and uptake. Vaccine. 2020;38: 6388–6401. pmid:32829979
- 12. Zhang W, Hong S, Takeuchi DT, Mossakowski KN. Limited English proficiency and psychological distress among Latinos and Asian Americans. Soc Sci Med. 2012;75: 1006–1014. pmid:22717362
- 13. Kim G, Worley CB, Allen RS, Vinson L, Crowther MR, Parmelee P, et al. Vulnerability of Older Latino and Asian Immigrants with Limited English Proficiency. J Am Geriatr Soc. 2011;59: 1246–1252. pmid:21718269
- 14. Jang Y, Kim MT. Limited English Proficiency and Health Service Use in Asian Americans. J Immigr Minor Heal. 2019;21: 264–270. pmid:29797103
- 15. Jang Y, Yoon H, Park NS, Chiriboga DA. Health Vulnerability of Immigrants with Limited English Proficiency: A Study of Older Korean Americans. J Am Geriatr Soc. 2016;64: 1498–1502. pmid:27305524
- 16. Gee GC, Ponce N. Associations Between Racial Discrimination, Limited English Proficiency, and Health-Related Quality of Life Among 6 Asian Ethnic Groups in California. Am J Public Health. 2010;100: 888–895. pmid:20299644
- 17. Sentell T, Braun KL, Davis J, Davis T. Colorectal Cancer Screening: Low Health Literacy and Limited English Proficiency Among Asians and Whites in California. J Health Commun. 2013;18: 242–255. pmid:24093359
- 18. Sentell T, Shumway M, Snowden L. Access to Mental Health Treatment by English Language Proficiency and Race/Ethnicity. J Gen Intern Med. 2007;22: 289–293. pmid:17957413
- 19. Mikesell L, Bromley E, Khodyakov D. Ethical Community-Engaged Research: A Literature Review. Am J Public Health. 2013;103: e7–e14. pmid:24134352
- 20. Eder M “Mickey,” Evans E, Funes M, Hong H, Reuter K, Ahmed S, et al. Defining and Measuring Community Engagement and Community-Engaged Research: Clinical and Translational Science Institutional Practices. Prog Community Heal Partnerships Res Educ Action. 2018;12: 145–156. pmid:30270224
- 21. Mukherjea A, Ivey SL, Shariff-Marco S, Kapoor N, Allen L. Overcoming Challenges in Recruitment of South Asians for Health Disparities Research in the USA. J Racial Ethn Heal Disparities. 2018;5: 195–208. pmid:28364371
- 22. Nguyen G, Hsu L, Kue KN, Nguyen T, Yuen EJ. Partnering to Collect Health Services and Public Health Data in Hard-to-Reach Communities: A Community-Based Participatory Research Approach for Collecting Community Health Data. Prog Community Heal Partnerships Res Educ Action. 2010;4: 115–119. pmid:20543486
- 23. Ma GX, Toubbeh JI, Su X, Edwards RL. ATECAR: An Asian American Community-Based Participatory Research Model on Tobacco and Cancer Control. Health Promot Pract. 2004;5: 382–394. pmid:15358911
- 24. Katigbak C, Foley M, Robert L, Hutchinson MK. Experiences and Lessons Learned in Using Community-Based Participatory Research to Recruit Asian American Immigrant Research Participants. J Nurs Scholarsh. 2016;48: 210–218. pmid:26836035
- 25. Trinh-Shevrin C, Kwon SC, Park R, Nadkarni SK, Islam NS. Moving the Dial to Advance Population Health Equity in New York City Asian American Populations. Am J Public Health. 2015;105: e16–e25. pmid:25905858
- 26.
Samuels F. Tips for collecting primary data in a Covid-19 era (Overseas Development Institute). 2020 [cited 2 Nov 2021]. Available: https://www.odi.org/publications/16977-primary-data-collection-covid-19-era
- 27.
American Psychological Association. Adapting Your Research Methods in Response to COVID-19. 2020. Available: https://www.apa.org/members/content/research-methods-covid-19-slides.pdf
- 28. Park H, Sha MM. Do Different Recruitment Methods Reach Different Asian Demographics? Surv Pract. 2015;8.
- 29. Wong CK, Horn-Ross PL, Gee GC, Shariff-Marco S, Quach T, Allen L, et al. Strategies for recruiting representative samples of Asian Americans for a population-based case–control study. J Epidemiol Community Health. 2016;70: 974–982. pmid:27053683
- 30. Valdez RS, Guterbock TM, Thompson MJ, Reilly JD, Menefee HK, Bennici MS, et al. Beyond Traditional Advertisements: Leveraging Facebook’s Social Structures for Research Recruitment. J Med Internet Res. 2014;16: e243. pmid:25348050
- 31. Im E-O, Kim S, Xu S, Lee C, Hamajima Y, Inohara A, et al. Issues in Recruiting and Retaining Asian American Breast Cancer Survivors in a Technology-Based Intervention Study. Cancer Nurs. 2020;43: E22–E29. pmid:30346330
- 32. Chee W, Ji X, Kim S, Park S, Zhang J, Chee E, et al. Recruitment and Retention of Asian Americans in Web-Based Physical Activity Promotion Programs. CIN Comput Informatics, Nurs. 2019;37: 455–462. pmid:31518339
- 33. Park H, Sha MM. Evaluating the Efficiency of Methods to Recruit Asian Research Participants. J Off Stat. 2014;30: 335–354.
- 34. McDonnell DD, Lee H-J, Kazinets G, Moskowitz JM. Online Recruitment of Targeted Populations: Lessons Learned from a Smoking Cessation Study among Korean Americans. Soc Mar Q. 2010;16: 2–22.
- 35. Barreto MA, Frasure-Yokley L, Vargas ED, Wong J. Best practices in collecting online data with Asian, Black, Latino, and White respondents: evidence from the 2016 Collaborative Multiracial Post-election Survey. Polit Groups, Identities. 2018;6: 171–180.
- 36. Tran H, Do V, Baccaglini L. Health Care Access, Utilization, and Management in Adult Chinese, Koreans, and Vietnamese with Cardiovascular Disease and Hypertension. J Racial Ethn Heal Disparities. 2016;3: 340–348. pmid:27271075
- 37. Ma GX, Tan Y, Wang MQ, Yuan Y, Chae WG. Hepatitis B Screening Compliance and Non-compliance among Chinese, Koreans, Vietnamese and Cambodians. Clin Med Gastroenterol. 2010;3: CGast.S3732. pmid:31327917
- 38. Ma GX, Wang MQ, Toubbeh J, Tan Y, Shive S, Wu D. Factors Associated with Colorectal Cancer Screening Among Cambodians, Vietnamese, Koreans and Chinese Living in the United States. N Am J Med Sci (Boston). 2012;5: 1–8. pmid:23243486
- 39. Ma GX, Toubbeh JI, Wang MQ, Shive SE, Cooper L, Pham A. Factors Associated With Cervical Cancer Screening Compliance and Noncompliance among Chinese, Korean, Vietnamese, and Cambodian Women. J Natl Med Assoc. 2009;101: 541–551. pmid:19585922
- 40. Appel HB, Huang B, Ai AL, Lin CJ. Physical, Behavioral, and Mental Health Issues in Asian American Women: Results from the National Latino Asian American Study. J Women’s Heal. 2011;20: 1703–1711. pmid:21777144
- 41. Parker SL, Davis KJ, Wingo PA, Ries LA, Heath CW. Cancer statistics by race and ethnicity. CA Cancer J Clin. 1998;48: 31–48. pmid:9449932
- 42.
Miller BA, Kolonel LN, Bernstein L, Young JL, Swanson GM, West D, et al. Racial/Ethnic Patterns of Cancer in the United States 1988–1992. NIH Pub. N. Miller BA, editor. Bethesda, MD: National Cancer Institute; 1996.
- 43.
Surveillance Epidemiology and End Results Program. Cancer Stat Facts: Cervical Cancer. Available: https://seer.cancer.gov/statfacts/html/cervix.html
- 44. Jin H, Pinheiro PS, Xu J, Amei A. Cancer incidence among Asian American populations in the United States, 2009–2011. Int J Cancer. 2016;138: 2136–2145. pmid:26661680
- 45.
NORC at the University of Chicago. National Immunization Survey—Teen. A Codebook for the 2019 Public-Use Data File. 2020. Available: https://www.cdc.gov/vaccines/imz-managers/nis/downloads/NIS-TEEN-PUF19-CODEBOOK.pdf
- 46. Elam-Evans LD, Yankey D, Singleton JA, Sterrett N, Markowitz LE, Williams CL, et al. National, Regional, State, and Selected Local Area Vaccination Coverage Among Adolescents Aged 13–17 Years—United States, 2019. MMWR Morb Mortal Wkly Rep. 2020;69: 1109–1116. pmid:32817598
- 47.
Centers for Disease Control and Prevention. LLCP 2019 Codebook Report. Overall version data weighted with _LLCPWT. Behavioral Risk Factor Surveillance System. July 31, 2020. 2020 [cited 6 Jul 2021]. Available: https://www.cdc.gov/brfss/annual_data/2019/pdf/codebook19_llcp-v2-508.HTML
- 48.
National Cancer Institute. Public Use Dataset (Health Information National Trends Survey). [cited 6 Jul 2021]. Available: https://hints.cancer.gov/data/download-data.aspx
- 49. Yi JK, Anderson KO, Le YC, Escobar-Chaves SL, Reyes-Gibby CC. English proficiency, knowledge, and receipt of HPV vaccine in Vietnamese-American Women. J Community Health. 2013;38: 805–811. pmid:23526096
- 50. Sadler GR, Lee H-C, Lim RS-H, Fullerton J. Research Article: Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy. Nurs Health Sci. 2010;12: 369–374. pmid:20727089
- 51. Brislin RW. Back-Translation for Cross-Cultural Research. J Cross Cult Psychol. 1970;1: 185–216.
- 52.
United States Census Bureau. 2019 American Community Survey 5-Year Estimates Data Profiles. 2020. Available: http://data.census.gov
- 53. Gim Chung RH, Kim BSK, Abreu JM. Asian American Multidimensional Acculturation Scale: Development, Factor Analysis, Reliability, and Validity. Cult Divers Ethn Minor Psychol. 2004;10: 66–80. pmid:14992631
- 54.
National Cancer Institute. Health Information National Trends Survey 5 Cycle 2 Instrument. 2018. Available: https://hints.cancer.gov/docs/Instruments/HINTS5_Cycle2_Annotated_Instrument_English.pdf
- 55.
American Association for Public Opinion Research. American Association for Public Opinion Research—Response Rate Calculator 4.1. 2020 [cited 23 Dec 2020]. Available: https://www.aapor.org/Education-Resources/For-Researchers/Poll-Survey-FAQ/Response-Rates-An-Overview.aspx
- 56.
Elliott AC, Reisch JS. Implementing a Multiple Comparison Test for Proportions in a 2xc Crosstabulation in SAS. SAS Users Group International 31. San Francisco, CA; 2006. Available: https://support.sas.com/resources/papers/proceedings/proceedings/sugi31/204-31.pdf
- 57. Fereday J, Muir-Cochrane E. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. Int J Qual Methods. 2006;5: 80–92.
- 58. Teitcher JEF, Bockting WO, Bauermeister JA, Hoefer CJ, Miner MH, Klitzman RL. Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and Tradeoffs. J Law, Med Ethics. 2015;43: 116–133. pmid:25846043
- 59. Magaço A, Munguambe K, Nhacolo A, Ambrósio C, Nhacolo F, Cossa S, et al. Challenges and needs for social behavioural research and community engagement activities during the COVID-19 pandemic in rural Mozambique. Glob Public Health. 2021;16: 153–157. pmid:33125306
- 60. Archer-Kuhn B, Beltrano NR, Hughes J, Saini M, Tam D. Recruitment in response to a pandemic: pivoting a community-based recruitment strategy to facebook for hard-to-reach populations during COVID-19. Int J Soc Res Methodol. 2021; 1–12.
- 61.
Health Management Associates. Community Based Organizations and COVID-19. [cited 30 Jun 2021]. Available: https://www.healthmanagement.com/services/covid-19-resources-support/community-based-organization-needs/
- 62.
Rkasnuam H, Batalova J. Vietnamese Immigrants in the United States in 2012. In: Migration Policy Institute [Internet]. 2014 [cited 27 Feb 2021]. Available: https://www.migrationpolicy.org/article/vietnamese-immigrants-united-states-2012
- 63.
United States Department of State. U.S. Relations with Vietnam. 2020 [cited 27 Feb 2021]. Available: https://www.state.gov/u-s-relations-with-vietnam/
- 64. Kreuter MW, Lukwago SN, Bucholtz DC, Clark EM, Sanders-Thompson V. Achieving Cultural Appropriateness in Health Promotion Programs: Targeted and Tailored Approaches. Heal Educ Behav. 2003;30: 133–146. pmid:12693519
- 65.
Berry JW. Theories and Models of Acculturation. Schwartz SJ, Unger J, editors. Oxford University Press; 2017. https://doi.org/10.1093/oxfordhb/9780190215217.013.2
- 66. Sam DL, Berry JW. Acculturation: When Individuals and Groups of Different Cultural Backgrounds Meet. Perspect Psychol Sci. 2010;5: 472–481. pmid:26162193
- 67. Quach S, Pereira JA, Russell ML, Wormsbecker AE, Ramsay H, Crowe L, et al. The Good, Bad, and Ugly of Online Recruitment of Parents for Health-Related Focus Groups: Lessons Learned. J Med Internet Res. 2013;15: e250. pmid:24231040
- 68. Ballard AM, Cardwell T, Young AM. Fraud Detection Protocol for Web-Based Research Among Men Who Have Sex With Men: Development and Descriptive Evaluation. JMIR Public Heal Surveill. 2019;5: e12344. pmid:30714944
- 69. Redmond N, Harker L, Bamps Y, Flemming SSC, Perryman JP, Thompson NJ, et al. Implementation of a Web-Based Organ Donation Educational Intervention: Development and Use of a Refined Process Evaluation Model. J Med Internet Res. 2017;19: e396. pmid:29191799
- 70. Kennedy R, Clifford S, Burleigh T, Waggoner PD, Jewell R, Winter NJG. The shape of and solutions to the MTurk quality crisis. Polit Sci Res Methods. 2020;8: 614–629.
- 71. Buchanan EM, Scofield JE. Methods to detect low quality data and its implication for psychological research. Behav Res Methods. 2018;50: 2586–2596. pmid:29542063
- 72. Dupuis M, Meier E, Cuneo F. Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices. Behav Res Methods. 2019;51: 2228–2237. pmid:30091086
- 73. Berinsky AJ, Margolis MF, Sances MW. Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys. Am J Pol Sci. 2014;58: 739–753.
- 74. Simone M. How to Battle the Bots Wrecking Your Online Study. Behavioral Scientist. Nov 2019. Available: https://behavioralscientist.org/how-to-battle-the-bots-wrecking-your-online-study/
- 75. Berg JA. GAINING ACCESS TO UNDERRESEARCHED POPULATIONS IN WOMEN’S HEALTH RESEARCH. Health Care Women Int. 1999;20: 237–243. pmid:10633622
- 76. Gollin LX, Harrigan RC, Calderón JL, Perez J, Easa D. Improving Hawaiian and Filipino involvement in clinical research opportunities: qualitative findings from Hawai’i. Ethn Dis. 2005;15: S5-111–9. Available: http://www.ncbi.nlm.nih.gov/pubmed/16312944 pmid:16312944
- 77. van der Zande ISE, van der Graaf R, Hooft L, van Delden JJM. Facilitators and barriers to pregnant women’s participation in research: A systematic review. Women and Birth. 2018;31: 350–361. pmid:29373261
- 78. Sheridan R, Martin-Kerry J, Hudson J, Parker A, Bower P, Knapp P. Why do patients take part in research? An overview of systematic reviews of psychosocial barriers and facilitators. Trials. 2020;21: 259. pmid:32164790
- 79. Grantz KH, Meredith HR, Cummings DAT, Metcalf CJE, Grenfell BT, Giles JR, et al. The use of mobile phone data to inform analysis of COVID-19 pandemic epidemiology. Nat Commun. 2020;11: 4961. pmid:32999287
- 80. Douedari Y, Alhaffar M, Duclos D, Al-Twaish M, Jabbour S, Howard N. ‘We need someone to deliver our voices’: reflections from conducting remote qualitative research in Syria. Confl Health. 2021;15: 28. pmid:33865454
- 81. Rahman SA, Tuckerman L, Vorley T, Gherhes C. Resilient Research in the Field: Insights and Lessons From Adapting Qualitative Research Projects During the COVID-19 Pandemic. Int J Qual Methods. 2021;20: 160940692110161.
- 82. Brody AA, Convery KA, Kline DM, Fink RM, Fischer SM. Transitioning to Remote Recruitment and Intervention: A Tale of Two Palliative Care Research Studies Enrolling Underserved Populations during COVID-19. J Pain Symptom Manage. 2021. pmid:34161811
- 83. Melis G, Sala E, Zaccaria D. Remote recruiting and video-interviewing older people: a research note on a qualitative case study carried out in the first Covid-19 Red Zone in Europe. Int J Soc Res Methodol. 2021; 1–7.
- 84. Hensen B, Mackworth-Young CRS, Simwinga M, Abdelmagid N, Banda J, Mavodza C, et al. Remote data collection for public health research in a COVID-19 era: ethical implications, challenges and opportunities. Health Policy Plan. 2021;36: 360–368. pmid:33881138
- 85. Rodriguez MD, Rodriguez J, Davis M. Recruitment of First-Generation Latinos in a Rural Community: The Essential Nature of Personal Contact. Fam Process. 2006;45: 87–100. pmid:16615255
- 86. Browne K. Snowball sampling: using social networks to research non‐heterosexual women. Int J Soc Res Methodol. 2005;8: 47–60.
- 87. Tung W-C, Nguyen DHT, Tran DN. Applying the transtheoretical model to cervical cancer screening in Vietnamese-American women. Int Nurs Rev. 2008;55: 73–80. pmid:18275539
- 88. Block P, Grund T. Multidimensional homophily in friendship networks. Netw Sci. 2014;2: 189–212. pmid:25525503
- 89.
Lazarsfeld P, Merton RK. Friendship as a Social Process: A Substantive and Methodological Analysis. In: Berger M., Abel T., Charles H, editors. Freedom and Control in Modern Society. New York: Van Nostrand; 1954. pp. 18–66.
- 90. Salganik MJ, Heckathorn DD. 5. Sampling and Estimation in Hidden Populations Using Respondent-Driven Sampling. Sociol Methodol. 2004;34: 193–240.
- 91. Heckathorn DD. Comment: Snowball versus Respondent-Driven Sampling. Sociol Methodol. 2011;41: 355–366. pmid:22228916