Rapport and trust.

Intuitively, it is not surprising that many sources providing practical interview advice emphasize the importance of building rapport between the interviewer and interviewee. Rapport has been described as the “key ingredient of successful qualitative interviewing” [8]. Previous work has hypothesized that, in establishing a friendly and comforting relationship, an interviewee may be more motivated to participate and to provide accurate information during an interview [9]. However, researchers have also argued that high levels of rapport may motivate the interviewee to alter their responses to align with what they think the interviewer wants to hear or are generally more socially acceptable. This problem, also known as social desirability bias, is particularly challenging in interviews on sensitive topics where an interviewee’s true behavior may not align socially defined norms [10].

Evaluating and measuring rapport as an operational concept has been criticized for its ambiguity and lack of structure [11]. Yet despite its murky underpinnings, rapport is widely recognized and discussed among interview researchers. Practical advice for building rapport includes negotiating clear expectations, dressing to an appropriate level of formality, mirroring terminology used by participants, and avoiding judgmental response [1]. Previous studies have measured rapport using a variety of methods. Hill and Hall (1963) evaluated rapport using post-interview surveys in which interviewer reported how ill-at-ease they felt during the interview and the degree to which the respondent was favorable to the interview [8]. Rapport has also been measured using eye contact [12], attentiveness [12, 13], interviewee motivation [1], and empathetic moments [14]. Due to the multiple interview conditions evaluated in this study, most conducted without being able to see the interviewee, we opted for a self-reported measure based on Hill and Hall’s questionnaire.

Honesty.

In human subjects research, whether qualitative or quantitative, researchers face the fundamental challenge of accurately capturing how participants behave and think in the real world. This can be particularly challenging when using self-reported measures or interview responses, as the validity of the data collected is reliant on the honesty of the participant’s responses. However, due to the lack of ground truth information, a participant’s honesty is difficult to assess directly. This has been of particular interest to researchers who focus on sensitive behaviors like sexual activity and drug use. Fortunately, previous work in this area has found that self-reported measures of honesty are noisy but often correlated with actual behavior [15–17]. Therefore, we included a self-reported honesty measure in this study.

Self-disclosure and intimacy.

While self-disclosure has many definitions, we prefer that of Moon (2000) which simply defines it as “any personal information that a person communicates to another” [18]. Conceptually, self-disclosure is closely related to the intimacy of a conversation. To measure this in an interview, self-disclosure is often separated into breadth, the quantity of different topics of information shared, and depth, the amount of personal detail shared in each topic [19]. Measures of self-disclosure have included counting the number of topics volunteered by participants [19], using a survey instrument [20], asking participants to self-rate their level of disclosure, asking conversation partners to rate each other, and using independent coders [20, 21]. In this study we employ two of these methods, one that leverages a previously validated instrument and another that applied qualitative coding to interview transcripts.

Interviewer effects.

Racism and race [22, 23], gender, age, and other identity factors can affect interview data and the interviewer-interviewee relationship [24, 25]. Several studies speculate that online modes with fewer visual cues (e.g. email) can minimize these effects. This is supported by evidence that interviewer effects are smaller in telephone interviews than face-to-face ones [3]. While we aimed to capture interview affects across mode, we also wanted to minimize this effect within mode. Therefore, our three interviewers were all of similar age and were of the same race and gender.

Sensitive content.

In their review of previous studies involving surveys on sensitive topics, Tourangeau and Yan (2007) found that the inclusion of sensitive topics reduced the response rate in some surveys [26]. Sensitive topics have introduced additional challenges for research teams including intrusiveness, the threat of disclosure, potential consequences of disclosure such as legal sanctions, and social desirability bias [6]. Nandi and Platt (2017), in their study comparing face-to-face and telephone questionnaires, did not observe any mode-related differences except in response to a sensitive question on political identity [27]. This suggests that sensitive questions may heighten differences observed across mode. Since the interview questions in this study focused on sensitive topics, the research team took several steps to protect interviewees and interviewers. It was made clear to the interviewees in the beginning of the interview that they could skip any questions they felt uncomfortable with and at any point during or after the interview they could rescind their participation in the study. The study protocol included a process to follow in cases where interviewees had strong negative reactions to the interview and ways to escalate any problems encountered during or after. The protocol also included contingency plans for cases where the interviewer felt uncomfortable or unsafe including the ability to end the interview immediately at their discretion.

Social desirability bias.

Social desirability bias is a phenomenon where interviewees alter or withhold their response because they are aware of the possible social implications of their intended answer. Social desirability bias has been interpreted by researchers as something that is inherent in the participant, possibly a part of their personality, or inherent to the topic of the interview question, as some behaviors and preferences are socially less accepted [28]. While the factor is difficult to measure directly, it is closely related to rapport, honesty, and self-disclosure. In our research design, we attempted to limit the opportunity for social desirability bias by randomizing interviewers across mode and collecting self-reported measures of honesty and rapport to validate our results.

Perceived anonymity.

Often, interactions conducted in an online setting provide users with greater opportunities for anonymity. Previous work has hypothesized that this factor creates opportunities for greater self-disclosure than in face-to-face conversation. Suler (2004) hypothesized that there exists an online disinhibition effect that allows individuals to express different aspects of themselves online that they would not in person [29]. Supporting this theory through empirical evidence, Joinson (2001) concluded that when people feel less identifiable, they tend to reveal more information about themselves and are less influenced by social desirability bias [30]. However, this effect may be context specific as Hollenbaugh and Evereett (2013) found an opposite relationship between anonymity and disclosure in their study of personal blogs online [31].

To measure perceived anonymity, Hite et al. (2014) developed and validated a five-question, context-independent measurement instrument to capture how users perceived their own anonymity [32]. In this study we leverage this instrument to assess how anonymous users felt across conditions.

Anonymity.

In all but one condition we discouraged, but did not prevent, participants from sharing identifying information for the study. This included encouraging the use of the anonymous email provided to participants by Prolific Academic for communication, discouraging users from entering their name when prompted by the video platform, and encouraging them to use pseudonyms during the interview. We instructed interviewers to avoid using the participant’s names when addressing them in the interview. However, in the Non-anonymous Chat condition, we omitted all instructions discouraging interviewees from sharing their own identifying information. In addition, we instructed non-anonymous chat interviewers to use and repeat the participant’s name if it was provided in order to emphasize to the participant that they were not anonymous.

Scheduling.

All interviewers maintained a minimum of 20 hours/week available for interviewing. Eligible interviewees were invited to self-select an interview time slot from those available using the Calendly scheduling platform. To control for possible selection effects resulting from scheduling, we required all participants assigned to the Email and Scheduled Survey conditions to schedule the start of their interview.

Interview logistics.

We chose online platforms to use based on interviewee usability, privacy options, and a feature set that matched our condition requirements. For video, audio, and chat conditions, we used Zoom cloud meetings. For email conditions, participants were encouraged to use the anonymous email address and platform provided by Prolific Academic, but they could also choose to use a personal email address and platform of their choice. Survey conditions were hosted on Qualtrics.

Interview calendar invites were automatically sent via email immediately after sign-up and a reminder email was sent 24 hours before and 30 minutes before the interview start time. Reminders included instructions on how to access and test the assigned platform. In pilot testing, we found that interviews usually lasted around 30 minutes. This included email, where we asked pilot testers to track the cumulative time they spent reading and answering study emails. We asked both interviewers and interviewees to allot at least 45 minutes per interview to allow ample time to complete the interview. Participants who completed an interview were compensated $10 USD.

Interviewers.

Unlike many studies, our interviewers were considered research participants due to the fact that we analyzed their behavior in addition to interviewee behavior. For recruitment, we opened a screening survey for paid interviewers and distributed it across the departments at our university. Given that interviewer identity and presentation is a known variable in interview experience (See Interviewer Effects), we hoped to find a set of interviewers with similar demographics and levels of experience. All applicants were students at our university. From our pool, we selected three interviewers with similar traits: all three were White, women, aged 20–25 years, fluent English-speakers, willing to broach sensitive topics, with a similar gender presentation and hairstyles, and who had no previous experience with research interviewing. All interviewers received the same paid five hour training. The training curriculum included information on research ethics, safety, confidentiality, followup probes, and hands-on practice. Given the sensitive topic matter, interviewers were encouraged to stop an interview at any point if they felt uncomfortable, without loss of compensation. All interviewers completed a consent form and took the required human subjects training approved by the Institutional Review Board at Carnegie Mellon University.

During the study, each interviewer met weekly with a member of the research team to review progress, check compliance with the research protocol, debrief emotional concerns, and troubleshoot technical issues. Interviewers were compensated $60 USD for training, $12 USD per check-in meeting, $12 USD for each completed interview, and a 7% bonus on all earnings if they completed the entire study. One interviewer chose to leave the study early due to time constraints, so interviewees were reassigned to the remaining interviewers at random. As a result, Interviewer 1 and Interviewer 2 completed 42% and 36% of all interviews respectively, while Interviewer 3 completed only 22% of the interviews.

While interviewer effects are difficult to measure directly, we looked for differences in interviews conducted by different interviewers using several indirect measures. This included the interviewee’s self-reported rapport rating with the interviewer, the interviewee’s self-reported honesty, interview word count, qualitative code count, technical difficulties and completion rates. No substantial differences were found between interviewers across any of these metrics, indicating that interviewer effects were likely not a source of differences.

Deception.

To mitigate confirmation bias, the research design included mild deception for both the interviewees and interviewers. This use of deception was approved by the Institutional Review Board at Carnegie Mellon University. We informed interviewees that the study broadly focused on improving online interviews. We informed interviewers that the study focused broadly on improving interviewer training. Real-world interviews require a research question to guide the arc of the interview, so we also informed interviewers, transcribers, and coders that the study specifically focused on how research participants narrativize sensitive memories, and encouraged interviewers to ask followup questions that elicited anecdotes and details. If the interviewees inquired about the study purpose, the interviewer was instructed to give a broad answer about improving online interviews, and direct further questions to the research team.

We debriefed interviewees in writing on the last page of their followup survey that they completed after their interview. This followup included information about how they would receive their compensation. We debriefed interviewers, transcribers, and coders in person when they concluded their portion of the study.

Interviewee recruitment.

Interviewees were recruited using the Prolific Academic digital research platform. Participants were recruited from a pool of individuals who were 18 years of age or older and resided in the United States. A 5-minute screening survey was used to determine eligibility for the study and individuals who completed the survey were compensated $1 USD. A full version of the screening survey can be found in S4 Protocol. The screening survey included questions related to demographics, work habits, willingness to complete five different types of online interviews, and reliable access to the technology needed to participate in the interviews. Participants were deemed eligible if they were willing to complete all five modes of interviewing, were willing to be recorded, and had regular access to a private, quiet space.

Of 1,240 people screened, 27% were eligible to participate. The most common reason for ineligibility was being unwilling to participate in one or more of different interview modes (usually Video or Audio) and/or being unwilling to be recorded (see Recruitment and Logistics). Eligible participants were batched and randomly assigned an interviewer and interview mode. Interviewers sent a uniform interview invitation to all participants. If a participant did not schedule an interview within two weeks of the invitation, they were dropped from the active pool and replaced with a new randomly-assigned interviewee candidate.

We anticipated that different modes would yield different completion rates. Following reliability guidance from Simmons et al. (2011), we aimed to complete at least 20 interviews in each mode [37]. For scheduling and monetary reasons, we capped each condition at 25 participants. After having difficulty completing enough Video and Audio interviews, we extended the duration of our study for an additional month. Due to monetary and scheduling constraints, we chose to end interviews after this time period, yielding 18–26 interviews per condition. See Table 1 for the complete list of interviews conducted per mode.

Post-interview survey.

After each completed interview, both the interviewee and interviewer were required to immediately complete a post-interview survey. For interviewers, this survey inquired about technical difficulties, interruptions, their discomfort, and their perception of the interviewee’s comfort. It required less than three minutes to complete. A full version of the interviewer survey can be found in S5 Protocol. For interviewees, the post-interview survey focused on perceptions of the interview experience, the online platform used for the interview, and their opinions on their own disclosures. This survey took less than 15 minutes to complete and was slightly tailored by mode. For example, rating the audio quality was not relevant for all modes. Table 2 summarizes the interviewee followup survey, and a full version can be found in S4 Protocol.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of interviewee post-interview survey.

https://doi.org/10.1371/journal.pone.0263876.t002

Interview transcription.

In order to compare the interview content on relatively even footing, interviews were all transcribed and reformatted uniformly. For audio and video interviews, Zoom’s automatic transcription service was used to generate the initial text transcript. Then, a four-person team validated the transcription and removed personally identifiable information. In order to maintain possible non-textual communication, transcribers were asked to capture any changes in tone of voice, meaningful pauses, and body language used (e.g., laughs or haltingly). For the chat, email, and survey conditions, transcribers consolidated emails, exported timestamps, tagged the responses by question, and broke up long responses into shorter utterances. Other text-based communication vectors like emojis or spelling choices were retained in chat, email, and survey conditions. For all modes, the text transcripts were consolidated into a standard format. Coders were instructed to separately flag portions of the interview that were related to the initial question and those that were followup questions. Each transcript was reviewed by a second transcriber to mitigate errors. Transcribers discussed any differences in interpretation and resolved them.

Qualitative coding.

Previous studies evaluating qualitative interviews have used the number of “topics,” generally defined, to measure the breadth of information disclosed during an interview [19, 21, 39]. This method is based on early work in offline communication and Social Penetration Theory [40]. However, the definition of topics in these studies depended on the context of the research study. Given the broad set of interview questions in our study, we decided to employ the common practice of qualitative coding, using emergent coding techniques, to define the set of topic categories [41]. To this end, a subset of questions were coded as if conducting an actual qualitative study. Four qualitative coders, two of whom were also on the transcription team, were split into pairs to code separate interview questions. All coders received the same introductory training on qualitative content coding, with an overview from Lazar et al’s (2017) book chapter on “Analyzing Qualitative Data” and specific advice from Erlingsson and Brysiewicz’s (2017) “A hands-on guide to doing content analysis” [42, 43]. Each question was coded using a main research question along with the overarching theme of how participant’s narrativized their response. For example, when coding the interview question pertaining to the participant’s views on death, coders operated under the motivating research question “what are Americans’ feelings and attitudes about death?” The detailed coding handbook is available in S2 Protocol.

For each question that was evaluated using this method, a codebook was generated using the following process. First, each coder sampled approximately 250 text utterances and highlighted what Erlingsson and Brysiewicz refer to as “meaning units”: a lower level abstraction of an interviewee’s response [43]. Second, the two-person team met to reconcile and consolidate their work into “condensed meaning units.” At this step, the original “meaning units” were shortened while retaining the response’s core meaning and a common interpretation was agreed upon between coders. Third, the pair examined and grouped those units into codes, and then codes into higher-level categories. To avoid large codebooks, the research team instructed the coders to aim for a codebook that had 5–35 codes. Fourth, the coders drew another sample of 100–200 text utterances and individually applied the initial draft of the codebook to the new text. At this point, coders were instructed to note any points of ambiguity, utterances that did not fit well into existing codes, and points of particular interest. Fifth, the team met to compare their codes, deliberate on points of disagreement, and adjust the codebook accordingly. These steps were repeated as many times as needed, modifying the codebook until both coders were in agreement or they thought that disagreements stemmed from subjectivity of the reader rather than ambiguity in the codebook. In practice, the pairs developed three to five different drafts of the codebook per question. In the final step, each coder individually re-coded the entire response set with the final codebook (≈150 interviews and ≈1500 text utterances). The pair met a final time to reconcile final codes. We emphasized to coders that differences in codes were not necessarily errors, they could also be the product of reasonable differences in interpretation or experience. As a result, the coders could agree to include both their codes. The coding teams did not use specific qualitative data analysis software. Instead, shared spreadsheet templates were used to generate the codebooks and code the responses.

To practice, each two-person coding team was assigned one question set to get hands-on experience with the coding process. These codebooks and codes were not used in our analysis. After the initial practice round, coding team 1 coded the questions related to guilt, best friends, and death. Coding team 2 coded pride and sex. Rather than do a full qualitative analysis of themes, we stopped our qualitative coding at the level of codes and categories. Our intention was to create data that could both summarize an interview as a qualitative researcher genuinely might and allow for a meaningful quantitative and qualitative comparison across interview modes. After all transcripts were coded, we divided the texts by mode in order to analyze possible differences in data equivalence.

Partial interviews.

Several interviews were only partially completed during the course of the study due to several factors. In five interviews (four chat, one email) the interviewer skipped a question. These interviews were considered complete and were included in the analysis done at the question level for those that were not skipped. These interviews were excluded from any analysis done on the content of the interview as a whole. For example, when computing average word count at the question level, we included interviews missing questions, but excluded them for word counts at the interview level (see Word Count). Notably, missing questions were distributed evenly across interviewers, but occurred only in text-based modes. This suggests that text modes were more prone to interviewer error. In three separate cases (two email, one chat) an interview was started but abandoned by the interviewee. These interviews were considered incomplete and were excluded from all aspects of the analysis. As an additional note, in one email interview the participant did not provide responses to several follow-up questions. This interview was also considered complete and included in all aspects of the analysis.

Missing survey data.

There were also several gaps in the screening and post-interview surveys. Participants were never required to answer any survey questions except to provide their ID number, so some questions were skipped or missed. Four participants completed an interview, but did not complete their post-interview survey (one audio, two email, and one survey). They were all compensated despite failing to complete the survey. Interviewers did not perfectly complete the task of filling out post-interview survey reports (see Technical Issues).

Missing qualitative data.

Despite implementing double-coding and double-transcribing, occasional errors were introduced when analyzing interview content. Most of these were discovered and addressed during the coding process. However, after the coding team completed their work, they retired from the project team, and were not available to correct errors discovered much later. As a result, in a few interviews the occasional utterance was left without a corresponding code and in one case an entire audio transcript was not qualitatively coded.

Participant demographics.

The screening pool of 1,240 participants spanned a wide age range with the youngest participants falling in the 18–24 category and the oldest in the 65–74 category. Overall our sample skewed younger, with over one-third of the interviewees in the 25–34 age range. Around 60% of our sample had never been married, likely a result of our skew in age. Of those screened, men were slightly over-represented, with a breakdown of 51% men, 46% women, 1% non-binary, genderfluid, or genderqueer (non-binary+) and 1% unknown. However, this trend reversed for completed interviews where 52% of our sample were women and 45% were men. Table 4 summarizes the demographic breakdown of the screening pool, those who were invited to an interview, and those who completed their interview.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Participant demographics.

https://doi.org/10.1371/journal.pone.0263876.t004

In the screening survey, participants were presented with two freeform text boxes to allow participants to self-report race, ethnicity, and any cultural group they felt was significant to them. Answers ranged from “Mostly midwestern caucasian american, with some asian american and malaysian aspects thrown in” to “Black, cosmic” to “White, Burning Man.” Full responses were shared with interviewers to provide context about their interviewees. To provide a summary and assess the distribution of our sample, we adapted the US Census Bureau guidelines to standardize these responses [49]. Our estimates of race and ethnicity roughly track the 2010 census data. However, our sample over-represents White participants and under-represents black, African American, and Latino participants.

Our exploratory data analysis indicated that, after random assignment, the gender and race of participants were distributed relatively evenly across modes. However, on average, interviews conducted with black participants were shorter in terms of word count (μ = 585 σ = 347) and yielded slightly fewer qualitative codes (μ = 14.8 σ = 4.2) than Latino (word count: μ = 974 σ = 754, codes: μ = 17.4 σ = 6.1) or white (word count: μ = 870 σ = 762, codes: μ = 18.1 σ = 7.4) participants, but were generally within a standard deviation of each other. Since the number of black and Latino participants were relatively small (1–2 participants of each minority group per condition) we were unable to do any meaningful sub-group analysis across mode. As a result, we cannot say with any certainty whether mode would be more or less of an effect for these groups. This raises some question of external validity that indicate our findings should be generalized cautiously, particularly for studies focusing on minority communities (see Accessibility for additional discussion of external validity).

Age was less evenly distributed which raised concern of internal validity. For example, we examined whether age might affect completion rates in different interview modes. We hypothesized that older participants would be more likely to complete email or audio interviews where younger participants would be more likely to complete chat interviews. However, we did not observe an effect and, in general, we found no evidence to suggest that demographics had an impact on scheduling rates, completion rates, or completion by mode.

Technology and environment access.

Access to the necessary technology and environment to participate in an online interview is a limiting factor when recruiting participants and is a more restrictive factor for audio and video interviews. Of our screening pool, 99% reported having access to a computer with a keyboard, as opposed to a touch screen device like a smartphone or tablet. A keyboard was required for eligibility in this study because it is very useful for text-based interviews, though not a prerequisite. However, only 70% of those screened had access to a webcam. This precluded participation in video interviews conducted using a computer, however these participants might have a tablet or smartphone with a camera suitable for video interviews.

In addition to technology, Jenner and Myers (2018) noted that a private space was a more salient factor than mode for their participants [7]. Of our respondents, 80% had access to a quiet, private room. More than 90% reported that they usually completed Prolific tasks at home, followed by 8% at an office space, and a final 2% mix of specific locations such as coffee shops. However, in interviews, several participants were concerned about being overheard in their space. In one case, an Audio participant was concerned about responding to a question about sex. While laughing, they responded “oh my god. can I answer this?…I’m not alone in the house.” The Audio participant eventually opted instead to write their response in the chat box. In another case, a Video participant got up to close the door, saying “I have a 16-year-old son who might walk by, so… [laughs]”. In follow-up surveys, nine participants (6% of 125) reported that their conversation was easily overheard and six (4%) reported that they were interrupted.

While we did not collect socioeconomic data directly, the high level of access to technology and private space, combined with a sample that is disproportionately white, indicates that our sample likely overrepresents individuals with a higher socioeconomic background. Since participants’ race was distributed relatively equally across mode and access to suitable technology and private space was a prerequisite for eligibility, we conclude that this skew does not affect the internal validity of the study or bias our findings related to interview mode. However, this does challenge the external validity of our results. While our study draws on participants from a large online panel of respondents, it is likely that there are other populations for which particular modes may have more of an effect. This is indicative of a broader issue of representation in online studies that researchers should take into consideration when deciding to conduct their research online. Since participants in most online panels are digital laborers, it’s likely their access to technology does not represent the access of the broader population, especially those of lower income and members of marginalized communities. At the same time, other online research studies using large panels of digital workers are likely to encounter similar bias. Therefore, researchers using similar recruitment methods can likely apply our findings directly. However, those working with participants of lower income, those with less access to online resources, or members of marginalized groups should use our results cautiously.

Willingness to participate differs by mode.

In the screening survey, participants were asked: “What types of online interview are you willing to complete? Select all platforms that apply.” From the participant’s responses we found large differences in willingness to participate across modes. As displayed in Fig 1A, participants were less willing to complete audio and video interviews compared to survey, email, or instant chat. While access to the requisite technology and to a private environment factor into this, those factors alone do not explain the strong preferences of participants. Of respondents with webcam and computer access, only 49% were willing to do a video interview. It is likely that the additional logistical burden and lower level of anonymity also contribute to the trend. In addition, participants might also have been less willing to participate in audio or video interviews because the platforms are not conducive to multi-tasking and require a participant’s full attention. Overall, unwillingness to participate in a video was the largest disqualifying factor in our pool.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Differences in recruitment and logistics across mode.

(A) Differences in willingness to participate across interview mode (n = 1240). (B) Differences in the scheduling rate, comparing those who scheduled an interview (n = 200) to those who were invited (n = 310), across mode. (C) Differences in completion rate, comparing those who scheduled an interview (n = 154) to those who were invited (n = 310), across mode.

https://doi.org/10.1371/journal.pone.0263876.g001

Respondents were also asked about their willingness to be recorded as a part of the interview. Since “in most qualitative traditions it is strongly preferable, if not essential, to have a full record of each interview,” this is an important factor for qualitative research teams to consider when recruiting [1, p. 44]. Specifically participants were asked “are you willing to conduct an interview where the video is recorded?” Only 45% responded affirmatively. However, this sentiment appears to be driven largely by the video mode itself rather than the recording. Of those who were willing to do a video interview, 95% were willing to be recorded. However, it is possible that respondents assumed that all interviews would be recorded when indicating their willingness to participate in interview modes; we did not specifically ask about the counterfactual of participating in a non-recorded video interview.

Scheduling rates differ by mode.

Although all participants who were eligible to participate in the study indicated they were willing to participate in any type of interview, we observed substantial differences in the rate that participant’s scheduled interviews across mode. As shown in Fig 1B, 310 participants received an email inviting them to schedule an interview. Of those, 200 (65%) initiated the interview process. We found a statistically significant difference (F_6,303 = 2.657, p <.05) in the scheduling rate across mode. The post-hoc Tukey pairwise comparisons showed that the video and email conditions differed significantly at p < 0.05, with a 20% drop in engagement from email to video. The other 20 comparison combinations were not significantly different. A summary of the results can be found in S1 Table.

Note that the participants received their interview invitation via email, thus the increased scheduling rate for the email condition may be in part the product of platform familiarity or self-selection (i.e., people that checked their email are more likely to use email). While not statistically significant, the drop in engagement from the Survey condition to the Scheduled Survey condition illustrates the cost of scheduling. With all other factors held constant, asking a participant to schedule a time to take their survey, as opposed to taking it immediately, appears to have caused a drop in engagement.

Completion rates differ by mode.

Similar to scheduling rates, we also observed statistically significant differences in the rate that participant’s completed interviews across mode. Overall, the completion rates aligned with the overall screening pool’s willingness to complete an interview by mode. As shown in Fig 1C, participants completed surveys and email interviews at higher rates than the chat conditions. Audio and video interviews were completed at the lowest rate. An ANOVA test (F_6,303 = 4.46, p <.001) and subsequent Tukey revealed statistically significant differences in survey versus audio, video, chat, and non-anonymous chat, as well as in email versus video. There was only one instance where an interviewer missed a scheduled interview. However, the single occurrence indicates that interviewer error was not a substantial factor in completion rates.

The chat and non-anonymous chat conditions had nearly identical scheduling and completion rates. The invitation emails and scheduling site did have small differences between these conditions to emphasize the lack of anonymity in the non-anonymous condition. This suggests that emphasizing anonymity had little effect on recruiting for this population.

Estimated cost.

While there are many considerations that a researcher must weigh when designing a study, cost is often one of the most important driving factors. Based on our experience running this study, we found that the cost of conducting an interview study varies widely depending on mode. These estimates, summarized in Table 5, are based on conducting 25 interviews using our interview script. To provide context, the median duration of a video interview, the mode that most closely mimics an in-person interview, was just over 17 minutes. In our our study, we compensated interviewers and participants $10 equally across mode, regardless of the duration of the interview. However, in designing an interview study researchers are likely to set compensation according the the expected duration of an interview. Therefore, in this estimate we assume participants are compensated $1 for completing the screening survey and both interviewers and interviewees are compensated at a $12 per hour rate upon completion of an interview. These estimates were calculated using five factors that we found varied widely across mode. The first was the screening eligibility rate: the percent of eligible participants (18+ with access to a computer) that are willing to participate in a given type of interview. This affects the number of participants that must be screened and compensated for completing screening. The second was the completion rate: the percent of eligible participants that go on to complete an interview. This also affects the number of participants that must be screened. The third factor was the no-show rate: The proportion of participants who do not show up at their scheduled interview to those that do. This incurs a cost assuming that the interviewer is compensated for the lost time that they had blocked off for the interview even though it did not occur. For some studies this might not be applicable. The fourth variable was the duration of the interview which influences the compensation paid both the interviewer and interviewee. With the exception of email interviews, we used the median interview duration for scheduled, anonymous interview conditions. Email interviews, which are inherently asynchronous, could not be measured in the same fashion. For this mode, the research team timed a mock interview that reproduced an interview transcript of median length. The fifth and final factor was the time required to process transcribed data, in the case of audio and video modes, and to process all of the transcripts (e.g. compile, clean, format, and anonymize).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Estimated costs to complete interviews by mode.

https://doi.org/10.1371/journal.pone.0263876.t005

We should note that our estimates do not account for the cost of recruiting participants to take the screening survey or fees for using a participant pool or crowd working service. In addition, they do not include other potential costs such as transcription, qualitative data coding, fees to use a survey platform, or the researchers’ time.

With a total cost of $143, a survey is the least expensive way to collect data. Since there is no interviewer to compensate and, screening aside, no cost to a participant who does not show up or complete the survey this is not surprising. However, it is important to remember that this mode does not provide researchers with the ability to follow up immediately on a participant’s response with follow-up questions (researchers would have to contact participants later to ask follow-up questions). On the other end of the spectrum, we find that chat-based interviews are the most expensive costing a total of $652. While chat interview do not require transcription, the high cost is driven by a much longer interview duration compared to other modes. For long interviews, this may make chat a prohibitively expensive option. However, unlike other synchronous modes (audio and video) which may have a shorter total duration, chat interviews afford the participant and the interviewer with the flexibility to multitask while completing the interview. This may allow for slightly lower hourly payments, but more importantly it may help with scheduling constraints and contribute to a higher completion rate and lower no-show rate than in audio and video.

Since audio and view interviews incur additional costs related to transcription, we found that the total cost of audio and video interview, at $566 and $638 respectively, to be comparable to chat interviews. While our study used an automated transcription service included in our institution’s Zoom subscription, our research assistants spent an additional 25 minutes verifying and correcting each automated audio and video transcript. This additional cost makes audio and video interviews almost as expensive as chat interviews despite being almost three times shorter to complete. Between audio and video, we find that audio interview are slightly less expensive. The difference is driven primarily by the difficulty in recruiting participants who are willing to be in a recorded video interview. As a result, researchers must weigh the added logistical burden of video interviews with the possible benefits of a richer medium.

Email, an infrequently used method for conducting interviews, provides an interesting option that researchers should give further consideration. In addition to its asynchronous nature we found that, surveys aside, email interviews had the highest eligibility rate, highest completion rate, and lowest no-show rate. As a result, email interviews provide a relatively cheap alternative to other interview modes at a total of $199. This is particularly advantageous option for researchers under substantial resource constraints.

Effects of the Covid-19 pandemic.

The interviews in this study were conducted in the summer of 2019, before the COVID-19 pandemic began. The pandemic rapidly increased the adoption of online communication technologies, but also complicated the relationships users had with the platforms. Zoom, the platform used in this study, saw an enormous increase in users during the pandemic. However, the constant use of online communication platforms gave rise to “Zoom fatigue,” a specific exhaustion caused by repeated video calls [50].

The implications of this phenomenon in relation to this study was not clear so in September 2020 we re-ran our screening survey with 100 new participants on Prolific Academic. We hypothesized that that the willingness to participate in modes like audio and video interview might have increased due to familiarity and access to the requisite technology and to a private space for online communication may have increased with people working from home. The new batch of results showed a slight increase in webcam access (+6%) but a decrease in access to private space (-3.9%). This indicates that working from home may have brought around changes in online communication technology, but the more people working in a shared living space during the pandemic decreased access to private space. Overall, we observed the willingness to participate in online studies dropped 3–7% depending on the mode. We performed eight two-sample t-tests comparing metrics between the survey results prior to the pandemic to those after. This included testing access to private space, webcam access, and willingness to participate in video, audio, chat, survey, email, and video interviews. None of the tests yielded statistically significant results. Full results tables are available in S16 Table.

Technical issues were minimal.

After each interview and attempted interview, we surveyed the interviewer about the experience. Interviewers did not complete this protocol task perfectly and it was not required for either survey condition. As a result we collected 137 interviewer post-interview surveys, covering 129 unique participants, 106 of whom successfully completed their interview.

Of the 137 post-interview interviewer surveys we collected, only four reported that technical issues prevented an interview from proceeding. In two cases the participants used an anonymous email address provided by Prolific, but withdrew after reporting trouble with Prolific’s email forwarding. In one case, a participant was unable to find the instructions or link for joining the video interview. In the final case, a chat interview was abandoned halfway through due to participant confusion with the platform.

There were five instances of dropped connections, but all five interviews were completed nonetheless. Audio quality issues were more frequent, with eight separate occurrences (21%) during the 37 attempted audio and video reports collected. In half of these cases interviewers reported that the interview was affected only “A little” and in the in the other half the interview was affected “A moderate amount.” For example, after one video interview, interviewer 2 reported “moderate” issues with the audio quality, saying it “was not very clear at times so it was difficult to understand everything he said. He also had a hard time understanding my questions.” In several of these instances, the transcription team was unable to transcribe parts of the interviewee’s response as a result of the quality of the recording. However, in none of these cases did the interviewer report that the interview was affected “A lot” or “A great deal”. All of the affected interviews were completed in their entirety.

Of those who completed their interview and post-interview survey, interviewees who used the audio, video, and chat platform reported less familiarity with the technology used. Fig 2 shows the distribution of interviewee responses when asked how familiar they felt with the online platform used in their interview.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Interviewee’s familiarity with online interview platform.

Interviewees rated their level of familiarity with their mode’s platform and were asked to agree or disagree with the following statement: “I felt familiar with [platform], used for the interview.” While users were less familiar with the audio, video, and chat platform, participants did not report differences in ease of use across modes.

https://doi.org/10.1371/journal.pone.0263876.g002

Despite the relative lack of familiarity with the video, audio, and chat platform (Zoom), in general participants overwhelmingly thought the platforms were easy to use across modes. This indicates that neither technical problems nor usability of online communication tools were a substantial barrier in our study. However, our sample may be more familiar with online platforms than the general population.

Interviews had reasonable rapport.

Across the non-survey conditions, the mean rapport, based on a 4-question subset of the Arizona Clinical Interview Rating Scale, was μ = 15.8 out of 20 (σ = 2.7) [12]. We examined the rating and individual items by mode, but observed no systematic differences for those 103 interviews. When we included the survey conditions, unsurprisingly, there were large differences in the rapport metrics. For example, a majority of survey participants reported that the survey was unsupportive.

Interviewees felt ill at ease regardless of mode.

In the post-interview surveys, both the interviewer and the interviewee were asked two questions from Hill and Hall (1963) [8]: “How often did you feel ill at ease during the interview?” and “How often did the [interviewer/respondent] seem ill at ease?” As might be expected from an interview on sensitive topics, many interviewees reported having felt ill at ease at least “a number of times” during their interview. However, very few felt uncomfortable during “almost all” of it. In breaking this down by mode, as shown in Fig 3, we observed no clear patterns in the distribution of responses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Interviewee’s self-report of feeling ill at ease.

In the post-interview survey interviewees were asked to respond to: “How often did you feel ill at ease during your interview?” In general, interviewees felt uncomfortable at least some of the time which is understandable given the sensitive nature of the questions asked.

https://doi.org/10.1371/journal.pone.0263876.g003

Despite feeling ill at ease, most interviewees still indicated that the interview was a positive experience. One audio participant summed up the interview saying, “It was interesting, slightly uncomfortable, but still fun.”

Interviewers may overestimate Interviewee comfort.

In asking both interviewers and interviewees about their rates of feeling ill at ease, we were interested in whether interviewers were able to reasonably pick up on interviewee discomfort. Anecdotally, Interviewer 3 noted in one post-interview survey that they had difficulty establishing rapport in a chat interview: “They were very neutral in their answers, so it was hard to discern their emotions over chat.” Fig 4 shows how the interviewer rated the interviewee’s comfort level compared to how the interviewee rated themselves. In the jitter plot, the survey responses were converted to a number between one and four (one: Never, four: Almost all). Overall, interviewers were more likely to overestimate than underestimate interviewee comfort.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of perceived and actual interviewee comfort.

Comparing interviewer (x) and interviewee ratings (y) of how ill at ease the interviewee appeared or felt reveals some overestimation by the interviewer. However, no patterns related to mode or interviewer emerge. There is a small amount of noise added to the points to allow for a visible scatter. Points are color-coded by interview mode.

https://doi.org/10.1371/journal.pone.0263876.g004

To examine whether the mismatch in perception was more or less likely by mode, we calculated the distance between ratings, and plotted them by mode. We excluded surveys in this case because the interviewer had no insight into the interviewee’s experience. As the Fig 4 shows, we observed no differences across mode. In addition, the overestimation had no ties to specific interviewers and was observed equally among them.

Chat modes could help or hurt rapport.

In post-interview surveys, interviewees who commented on mode and rapport often did so for chat conditions. This could be because of novelty effects (as chat isn’t a usual mode for interviews) or because interviewees had strong feelings about the mode. Anecdotally, chat seemed to be polarizing. Some interviewees felt it was easier to share, while others felt disconnected. One interviewee in a chat condition said, “It felt like I was talking to an AI,” while another in a non-anonymous chat said “The chat format worked well. I was surprisingly comfortable typing about these personal topics.” Both interviewers and interviewees voluntarily observed that it was hard to gauge engagement in chat. One interviewee said, “The interviewer seemed a bit distracted or took long to respond at times.” While in a check-in meeting, Interviewer 1 laughed that she “never had people with attitude on audio or video, but there was lots more on chats, because they have a screen to hide behind.”

In a much more serious situation, Interviewer 2 felt very uncomfortable in one chat interview when a participant expressed suicidal thoughts. The interviewer recalled that the participant “said he hates living life…being a chat, I wasn’t really sure how to respond to that. I think I just said, ‘thanks for sharing.’” In cases like this where emotionally distressing issues arise, we had explicitly instructed interviewers that while they could offer assistance, they should thank the participant for their honesty and “gently disengage” from the topic since they did not have professional training as a therapist. While the interviewer followed those instructions, the lack of rapport and other audio and visual cues in the text-based interview likely affected the interaction and led to the interviewer disengaging less gently than in an optimal situation. As per the study protocol, the interviewer reported this issue to the research team and it was escalated to the principle investigator. The interviewer met with a member of the research team and discussed the interview, how they felt afterwards, and if they needed any additional support. The interviewer reported that they were emotionally okay and were fine to continue conducting interviews.

Motivation not affected by mode.

To gauge participant motivation, we asked interviewers to rate how motivated the interviewee appeared during the interview and asked interviewees to self-report. This was done using a 5-point Likert scale based on similar measures in Hill and Hall (1636) [8]. Overall, both the interviewees (μ = 1.79, σ = 0.77) and interviewers (μ = 2.04, σ = 1.08) felt that the interviewee were relatively motivated during the interview. On average, this fell roughly between “very motivated” and “motivated” or “motivated” and “neither motivated nor unmotivated” respectively. In examining the level of motivation across mode, we did not observe any substantial differences.

Honesty does not differ by mode.

In the post-interview survey we were interested in honesty as it relates to openness and truth-to-self, rather than honesty related to some objective truth. Interviewees self-reported withholding information in 16% of interviews (25 of 152) and altering the details of their responses 7% (11 of 152). However, it is not entirely clear if the latter was a result of interviewees wanting to hide information or a result of our encouragement for interviewees to protect the identity of themselves or others by using pseudonyms. In addition to these two questions, interviewees were asked to respond on a 5-point Likert scale to the statement “During the interview, I was not as honest as I could have been.” As shown in Fig 5, interviewees generally disagreed with the statement and rated themselves as honest.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Interviewee’s self-reported honesty.

Interviewee’s were asked to rate themselves based on the following statement: “During the interview, I was not as honest as I could have been.” No statistically significant differences were found across mode.

https://doi.org/10.1371/journal.pone.0263876.g005

After converting the Likert to a numerical scale, running an ANOVA test (F_6,145 = 0.25, p >.5) showed no significant differences in self-reported honesty across mode.

Little evidence of social desirability bias.

While not one of our outcome measures, we wanted to ensure that social desirability bias was not substantially different across modes such that it might affect our other outcome variables. Using the BIDR-16 scale, we found an average score of 3.7 (σ = 0.8) on a 7-point scale. The distributions of BIDR-16 scores were incredibly consistent across mode. This suggests that none of the modes uniquely invited socially desirable responding bias.

Self-disclosure index does not differ by mode.

Our first assessment of participant self-disclosure used the Sensitive Information Disclosure (SID) scale that was included as a part of the interviewee followup survey. In their development of the 12-question scale, Pickard et al (2018) noted that the instrument “can be used to derive insights previously masked in similar studies comparing disclosures of sensitive information across interview modes” [20]. Within our sample we observed a normal distribution of scores with a mean of 2.81 (σ = 0.48) out of 5, where 1 represented a low level and 5 a high level of self-disclosure. However across modes, we did not observe any statistically significant differences (F_6,145 = 1.79, p >.1). The distribution of scores by mode is illustrated in Fig 6.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Interview self-reported self-disclosure.

Interviewee scores on the Sensitive Information Disclosure (SID) index by mode. We observe no meaningful differences across mode.

https://doi.org/10.1371/journal.pone.0263876.g006

Disclosure depth was difficult to define.

The second self-disclosure measure we evaluated was disclosure depth. However, we encountered difficulties in defining an appropriate method to apply this metric to our results. While others have measured self-disclosure by asking third-parties to rate disclosure sensitivity or depth, the scales we encountered were not sensitive enough to detect differences for our interview with inherently sensitive topics [51]. We initially hoped to define our own scale for disclosure depth, and piloted several attempts, but were never satisfied with the reliability or interpretability of the emerging scales. For example, co-author Bernagozzi, a member of the coding team, wrote a coding reflection that discussed the difficulty of relying on hesitation as one indicator of limited disclosure, an idea repeatedly proposed:

sometimes, the [interviewee] changes their mind or switches gears in the middle of what they are saying. Often, I attribute this to them “thinking out loud” as they are answering the questions, but…it could also indicate that they are holding something back…Other folks will decide mid-answer that another response would fit the question better and they will switch tracks, but I don’t necessarily think they are being dishonest.

Therefore, we looked to Jenner and Myers (2018) [7], who analyzed their video and in-person interview transcripts across mode with special attention to identifying instances of exceptional or limited disclosures. Members of the interviewer, coding, and research teams all hypothesized that the question related to sex had the greatest variability in disclosure. For example, in an early post-interview survey, Interviewer 3 volunteered the observation that in her interviews so far, “…it seems like the sexual fantasy question is something they tend to pass over with their answers” or “they responded quickly, honestly, and with detailed answers.” One survey participant noted in their final open response, “I really don’t want to talk about sexual fantasies with anyone I’m not in a relationship with. Way, way too personal, even anonymously.”

Similarly to Meyers and Jenner, as the coding and research teams explored the sexual fantasy question each team independently found that it was more fruitful to identify cases of limited and exceptional disclosure rather than classify or rate general cases. In their codebook development, the coding team organically developed codes related to disclosure level. For example, the elaborate code identified responses with “an elaborate answer, including complete/detailed fantasy scene” while the hesitation code applied when the respondent “shows surprise or is unsure how to answer when hearing the question.” After the second codebook draft, we asked the coding team to synthesize some of their codes related to limited disclosure, and they defined the code soft pass for when “the interviewee answers but does not directly or fully answer the research question.” The research team took the final codebook and noted each code definition that involved limited (soft pass) or exceptional disclosure (incest, infidelity, non-consent, and trauma). These codes served as our markers of disclosure depth.

As expected, occurrences of limited and exceptional disclosure were very sparse in our corpus. In total, we found 29 exceptional disclosures and 10 limited disclosures. Given the low frequency, we were not comfortable doing a quantitative comparison of disclosure depth. We qualitatively examined the distribution and relevant transcripts, noting only that there was no evidence of limited disclosure in the survey conditions. We concluded that while it’s possible that surveys prevent limited disclosure, it’s more likely that indicators of limited disclosure, like non-verbal cues and pauses, do not present themselves in surveys. This conclusion is anecdotally supported by one survey respondent who, unprompted, decided to re-take the entire survey weeks later in order to disclose deeper detail (see Personal Information Sharing).

Disclosure breadth did not differ by mode.

Previous work has operationalized disclosure breadth by counting the discrete, relevant topics introduced by the respondent [19]. We reasoned that our coding process organically identifies equivalent topics through the creation of code categories. For example, the categories for the codebook on death included Emotions, Focus of emotions, Eras of life, Post-Loss actions, Spiritual views, Life experience, and Distance to death. Treating code categories as our unit of measure, we found that the count and distribution of categories did not differ by mode.

Individuals reported that mode impacted their disclosure.

Overall, the quantitative metrics we used to examine self-disclosure did not indicate any systematic differences across mode. However, at the end of interviews or in post-interview surveys, individual participants did sometimes volunteer their opinions on mode. Anecdotally, participants seemed more likely to volunteer a mode-related opinion after chat interviews (see Chat and Rapport). One non-anonymous chat participant wrote: “Given the format (text message), I tried to be as open as I could.” In addition, after a non-anonymous chat interview, Interviewer 3 wrote:

The interviewee told me that it was much easier for them to open up about these difficult topics over chat to a complete stranger than to talk to someone in person. I thought this was really interesting, and I didn’t know how that level of anonymity affected what/how much we choose to share.

In contrast, one email participant explicitly described that they withheld or altered their response during the interview, but felt comfortable discussing them in a post-interview survey.

I wasn’t dishonest but I was super uncomfortable about the sexual fantasy question. I have a lot of insecurities about my body so I’m just not really open about sex because of that. And I don’t like talking about it. I also wasn’t as open about my depression as I should have been—I didn’t mention the time I was laid off from work and didn’t leave my apartment once in five months. Dunno why I’m now comfortable enough to admit this stuff. Maybe it’s easier when I don’t feel like I’m having a conversation with someone.

On one hand, we observe evidence that text-based communication may allow for greater disclosure, but at the same time the presence of an interviewer, even in as abstract a form as email, can have an affect on both the interviewee’s level of comfort and their disclosure.

Participants provided more personal information than requested.

All conditions except for the non-anonymous chat mode discouraged participants from sharing personally-identifying information and gave them instructions on how to prevent doing so (see Anonymity). However, there were cases where participants still provided us with personal information despite the instructions. In our scheduling sign-up form and interview platform, we auto-filled any personal information fields with reminders such as “Do not provide your name.” Nonetheless, some participants chose to provide their name or use a personal email address rather than the anonymous one provided by Prolific. We logged 191 scheduling forms, including individual reschedules, noting whether the name fields appeared to be personally-identifying or likely pseudonymous. Of the 27 scheduling forms for the non-anonymous chat condition, 26 participants provided an identifying name and one provided a name that was possibly pseudonymous. In contrast, only 17 of 164 participants (10%) in the remaining (anonymous) modes chose to overwrite the default values with an identifying name. For audio and video modes, we checked whether the participants chose to provide an identifying name when prompted for a username by the Zoom platform. Of the 36 interviews, 30% shared a likely identifying name.

We do not fully understand why participants chose to provide personally-identifying information despite explicit instructions not to do so. It is possible that participants were confused by instructions, lacked tools to protect their identity if they desired, or felt ambivalent. One email participant did provide insights in a post-interview survey, reporting:

I thought the process for answering the survey through Prolific looked complicated, and I worried that I wouldn’t get notifications for Prolific messages and would then not answer fast enough, so I didn’t try that route and just set up my email address to be used. However, my email address contained identifying features, and I now kind of regret using it.

Another email participant responded:

I want to point out that I accidentally shared my first and last name with my interviewer when responding to her the first time, which is the reason for my answers to the privacy questions in this survey. That was my fault, but I don’t believe that my personal identification will result in any privacy concerns.

Individuals were conscious of anonymity.

While the mode did not appear to have substantial effects on the perceived anonymity, interviewers and interviewees both indicated that anonymity was a factor during their interviews. Interviewer 2 noted in a post-interview survey that after answering all the questions, a non-anonymous chat participant “was concerned about how his identity would be kept confidential.” In addition, a survey participant said, “I was honest in describing everything due to the fact that the survey is anonymous,” and a scheduled survey participant said clearly, “I have nothing to hide when I am anonymous.” Furthermore, after being asked the sensitive question pertaining to guilt, one video participant paused to confirm that the researchers were not collecting any personal information about them.

While researchers often collect identifying information from otherwise anonymous participants for logistical reasons, for example to facilitate scheduling and compensation, sometimes it is beneficial to shield participant identity even from researchers. For example, one scheduled survey participant said:

There were some questions, in which I could have gone more in-depth with, but because I know several people who go to [the university conducting the study] (and of which, one of whom is the subject of the questions of what I feel most guilty about), I felt as though I’d be discovered by them somehow, so I didn’t feel comfortable in elaborating as much as I could have on some topics.

In one unusual case, a survey participant completed their interview and post-interview survey with quality answers and received their compensation. Curiously, five weeks later, the participant re-took both the survey and the post-interview survey. The participant was able to retake the surveys because our survey mechanisms were set up to minimize the use of trackers and thus did not automatically block retakes. The participant identified themselves by providing their anonymous Prolific ID in the survey, the primary identifier that was asked of all participants and was used to match between screening, interview, and post-interview tasks. In their second submission of the post-interview survey, the participant stated that they had purposefully decided to fill out the surveys a second time. This was the only instance of a participant completing an interview multiple times. The responses in both were similar except for a few instances related to the question about what the participant felt most guilty about. In the first iteration, the participant provided a general response about the betrayal of trust and in the post-interview survey said, “I am not comfortable sharing the worst thing I’ve ever done in my life.” However in the second survey, the participant disclosed a much more serious situation in which they expressed guilt over a past event involving sexual violence without disclosing any details. It seems the participant had been thinking about the interview response and, of their own volition and without notifying the research team, decided to re-take the survey interview in order to include this deeper disclosure. While it is difficult to interpret this story in the context of this study, it’s interesting that the survey condition afforded the participant (1) a degree of anonymity and (2) the ability to take the full initiative in updating their response without needing to interact with anyone to do so. In the re-take the participant closed by saying, “I sure hope this is anonymous,” illustrating that that the participant took a risk with disclosure despite having privacy concerns. As with other sensitive issues that arose during the course of this study, this case was immediately escalated to the principal investigator. The research team discussed this situation at length and considered what actions might be taken as a result. Ultimately, given the anonymity of the participant and the lack of any specific details surrounding the event, the team decided not to escalate further.

Word counts were higher for verbal modes.

Word count is a structural feature of interview data, but previous work has applied it to content-related constructs like self-disclosure depth or engagement [19]. Our own interviewers and coders also implied that they associated response length with content quality. For example, Interviewer 1 was concerned that in one chat interview “the answers were short and [the participant] not too willing.” The coding team who worked on the question pertaining to sex even defined a code called short for responses that “give a very short answer.” This further indicates that the team thought length had thematic implications worth investigating.

To calculate a word count for the main interview questions, we excluded non-interview related conversations at the start or close of interviews and included any non-verbal annotations that the transcribers noted. The mean across all completed interviews was 312 words (σ = 318.4) for the followups removed subset and 889 words (σ = 629.0) words for the followups included subset. In particular, the video and audio conditions had the most words, approaching twice as much as surveys and chats.

We ran ANOVA tests on both subsets and found statistically significant differences across mode in both. Fig 7A shows the distribution of word count across mode for the followups removed subset. After the ANOVA indicated statistically significant differences (F_6,142 = 10.01, p <.001), we evaluated the pairwise comparisons using a Tukey multiple comparison. This assessment showed that the audio and video conditions have a statistically significantly higher word count than chat, scheduled survey, and survey conditions at p <.05 level. Video was also higher than non-anonymous chat and email (p <.01). Detailed test results can be found in S7 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Interviewee word count by mode.

(A) Interviewee word count, excluding responses to followup questions, by mode. (B) Interviewee word count, including followup questions, by mode. Excludes survey conditions where there were no followups.

https://doi.org/10.1371/journal.pone.0263876.g007

Fig 7B shows the word count distribution for the followups included subset across mode. There were similar statistically significant differences found in the ANOVA test (F_4,94 = 10.61, p <.001) and the Tukey multiple comparisons. We found video had a significantly higher word count than email, non-anonymous chat, and chat at the p <.001 level. Audio also averaged significantly higher word counts than both chat (p <.001) and non-anonymous chat (p <.05). Detailed results are included in S7 Table.

Regardless of whether we included followup questions, audio and video participants had more to say than those in other modes. However, this may be a direct result of being able speak faster than one can type. Of the text-based modes, email had higher variability in word count. This suggests that the reduction of time-pressure inherent in the asynchronous email mode affords respondents the ability to write more than in other text-based modes.

Code counts do not differ by mode.

In addition to word counts, we examined thematic differences across qualitative codes. According to the SAGE Handbook of Qualitative Data Analysis, the “generation of themes via coding… and categorization… is arguably the most common analytic approach taken by qualitative researchers using interviews” [41]. While not a representation of interpretive value, these codes serve as a proxy for the the breadth of content and themes covered during the course of an interview. The frequency and distribution of thematic codes, and differences in such across mode, provide richer insight into the quality of the data collected than word count alone. In this study, the coding was done by framing each qualitative code as one “meaning unit” of thematic analysis. We dis-aggregated these codes in order to compare unique code counts by mode, counting each code at most once in each interviewee’s response to a particular question. While simplistic, we use code count as a rough approximation of thematic analysis potential.

Fig 8A shows the code count distributions for the followups removed subset of transcripts across mode. Each condition averaged between 10 and 18 codes per interview. ANOVA tests revealed no statistically significant differences in code counts across mode (F_6,140 = 1.582, p >.05).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Qualitative code count by mode.

(A) Qualitative code count, excluding codes derived from followup questions, by mode. (B) Qualitative code count, including codes derived from followup questions, by mode. Excludes survey conditions where there were no followups.

https://doi.org/10.1371/journal.pone.0263876.g008

Fig 8B shows the code count distributions for the followups included subset of transcripts. With the followup question content included, code count averages increased to between 18 and 22 codes per interview. Just as in the other subset, there were no statistically significant differences across mode (F_4,94 = 2.369, p >.05).

Code distributions differ by followup questions rather than mode.

Stemming from this analysis, we conducted an additional exploratory qualitative analysis of the code distributions by question and mode. Fig 9 shows an example of a frequency table generated for the question on death. The figure highlights the relative proportion of interviews that had at least one code in a given category. We were curious whether certain codes were more or less likely to be present in certain modes. In the case of the death question, we found that survey participants did not detail their life events or volunteer a focus of emotion as often as other modes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Sample of qualitative code frequency.

Percentage of interviews (in each mode) that had a least one code applied from the given category. These code categories are from the Death question. Example interpretation: 55% of Survey interviews were tagged with at least one code related to distance.

https://doi.org/10.1371/journal.pone.0263876.g009

Similar observations were made for other questions as well. In general, we observed that certain types of codes were less likely to appear in survey modes. However, we concluded that this effect was likely due more to the lack of followup questions than to some other quality inherent to the mode. Among modes that did afford followup questions, our exploratory analysis didn’t suggest any consistent trends in code distribution. We hypothesize that if our surveys did use followup questions (e.g. using a dynamic survey method or bot interviewers [19]), we would see no differences between surveys and other modes.

Rare codes occur more in verbal modes.

Returning to the discussion of thematic content, the aspiration of most qualitative interviewing is not to gather as many codes as possible, but rather to represent a diversity of experience. We reasoned that rare codes were thus also of interest in data equivalence. As a unit of analysis, rare codes can be interpreted as such: If an interview has a rare code, it indicates that the interview successfully elicited a unique, individual experience. Modes with more rare codes may be better at eliciting diverse individual experiences.

To define rare codes, we calculated the frequency of each code across all interviews and modes, and examined the distribution of frequencies. We defined rarity in two different ways. First, a code was labelled as rare if its frequency was 1 standard deviation below the mean (frequency < 3). Second, a code was labelled as rare if its frequency fell into the bottom quartile of the distribution (frequency < 7). We examined both definitions of rare over both the followups removed and followups included subsets of data. The full statistical results are available in S10 Table. Using the first definition of rarity, we did not observe a statistically significant difference across mode (followups removed: F_6,140 = 1.618, p >.05, followups included: F_4,94 = 0.924, p >.05)

Under the second definition of rarity, we found statistically significant differences in the mean number of rare codes per interview across mode. The distribution of rare codes across interviews is shown in Fig 10. The differences in mean for the followups included subset were substantial (F_4,94 = 4.437, p <.01), while the differences for the followups removed subset were borderline significant (F_6,140 = 2.097, p ≈.05). A Tukey comparison test for the followups included subset confirmed that rare codes were statistically significantly higher in the audio condition than in the chat or email conditions (both at p <.01, for details see S12 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Distribution of rare codes by mode.

Number of rare codes per interview per mode. A codes is rare if its frequency falls in the first quartile of the overall code distribution. Audio interviews were statistically significantly higher than Email and Chat interviews.

https://doi.org/10.1371/journal.pone.0263876.g010

However, the absolute difference in rare codes is small. Audio interviews had an average of two rare codes per interview, while the other modes averaged one. We noticed a possible outlier in the set of audio interview that had six rare codes. This may have inflated the mean and overestimated the magnitude of the differences. To confirm it was not a spurious outlier, we removed that point and re-ran the ANOVA. The test confirmed that the relevant values were still significant. Observing that the video interviews also varied more than other modes, we weakly hypothesize that verbal interviews may help elicit more unique, individual experiences. A larger sample size of interviews would provide more evidence to this claim.