Cognitive interviewing

Cognitive interviewing can be defined as ‘the administration of draft survey questions while collecting additional verbal information about the survey responses, aimed at evaluating the quality of the response or helping determine whether the question is generating the information that its author intends.’ [8] This research methodology identifies and remedies mismatch between respondent interpretation of survey questions and researcher intent. The process of cognitive interviewing generates insight into the linguistic quality of translations [9–11] and the cultural and context-specific resonance of words and phrases in questions [12]. As a result, the implementation of cognitive interviewing extends beyond determining whether respondents interpret questions as intended to examining the constructs captured by the survey questions and whether those constructs are understood consistently across culturally diverse respondents [13].

This study followed the methodological approach detailed in Scott, Ummer & LeFevre [14]. During cognitive interviews, our researchers asked survey questions exactly as written, as if administering a quantitative survey, and attempted to elicit a response based on the quantitative response options available. Then, the researchers asked a series of qualitative probes to assess how the respondents interpreted the questions, why they gave the responses provided, and whether their interpretation of the question matched the research intention. Testing each question took between one and five minutes: less than a minute to administer the quantitative question and then several minutes to discuss the question with the respondent in order to assess the question’s comprehensibility before moving on to the next question [14]. Extensive debriefs, central to the cognitive interviewing methodology, were held at the end of each interview day. During these debriefs, the research team discussed each question and the responses elicited from research participants. When misunderstandings were identified the researchers revised the questions. The revised questions were taken to the field for further testing with new respondents. After three rounds of testing and revision, the survey questions were finalized for use in quantitative large-scale surveys [14].

Developing survey questions for testing and cognitive interview guides

After reviewing and mapping questions on digital access and use from major surveys, we identified priority questions for cognitive testing. We also added new questions proposed by our team to fill measurement gaps for a total of 118 questions to be tested (these questions, both the initial version tested and the final version developed, in Hindi and English, are available in S1 Appendix). These questions sought to measure key outcomes of interest across the domains of (a) phone ownership, access, and use; (b) differences in how men and women use the phone and the internet, (c) norms and attitudes about phone and internet use, and (d) concerns and harms related to the phone and internet use.

During a week-long intensive workshop, our research team undertook a rigourous translation process. Each question was read and discussed at length in English until all researchers understood its intended meaning (i.e., the construct it was seeking to assess), then multiple Hindi translations were developed, compared, and discussed. The researchers drew from their experience working in rural Uttar Pradesh to propose words and phrases that they thought would be most comprehensible and straightforward for the respondent population. Back translations were developed and examined. Further discussion and adjustments to the Hindi translation resulted in a final draft version of all 118 questions.

These 118 questions were divided across three cognitive interview guides, i.e., approximately 40 questions per guide, so that each cognitive interview took no more than an hour and a half. Each quantitative survey question to be tested was followed by probes to assess understanding, such as “Why did you say [response provided]?”; “Could you tell me in your own words what this question is asking?”; “What does [keyword] mean to you?” and “Is there any other word for [keyword] that people generally use here?”. Researchers also asked tailored probes (also called adaptive follow up questions) based on the respondent’s specific replies to uncover individual insights, which would be overlooked by fixed protocols and survey questions [14,15]. For instance, if a respondent earlier stated that she did not know what the internet was, but later said she learned a new recipe by watching YouTube, the researcher asked a follow up question to assess her understanding of the word “internet” and to explore what alternative words or examples are available to better convey the concept of “internet.”

Study location and participant sampling

We selected adults with fewer years of formal education than average, living in more remote geographic regions of Uttar Pradesh, whom we approached under the guidance of local community leaders. It was assumed that educated urban residents would be more likely to find the draft questions comprehensible. Testing the questions with adults who had limited formal education in rural areas would reveal the weaknesses of the questions, thereby enabling revisions that improve comprehensibility for the entire target population.

We cognitively tested the questions in rural areas of Badaun district in the west and Jaunpur in the east of the state. In both districts, Hindi is the primary language of communication and is the language used for major surveys (such as the Indian National Family Health Surveys). Also, we selected respondents who had used a mobile phone at least once in the last two weeks. Since many survey questions focused on respondents’ phone use, someone who had not used a phone at all over the last two weeks would skip most questions, rendering the interview ineffective for testing the comprehensibility of those questions. As we proceeded through the study, we also increased focus on smartphone users because several of the survey questions that we wanted to test were linked to internet use and would otherwise be skipped for those with basic phones.

Guided by the concept of thematic saturation [16], we interviewed a total of 101 different respondents across three rounds (Fig 1) between 01/04/2023 and 27/04/2023. The questions were revised for each subsequent round to produce version 2, version 3 and then version 4, which was the final version and was not tested. We required fewer respondents per round as the questions became increasingly comprehensible [14]. We also found that male respondents found the questions easier to understand, therefore additional female respondents were sampled to ensure the questions were comprehensible for all.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Iterative sampling approach for cognitive interviews in Uttar Pradesh, India.

https://doi.org/10.1371/journal.pone.0354874.g001

Ethics statement

Ethical considerations are essential in research to protect participants’ rights, welfare, and dignity, while maintaining scientific integrity, accountability, and societal trust [17]. Accordingly, this study adhered to established ethical standards and received ethical approval from the Sigma Institutional Review Board (IRB) (IRB Number: 10123/IRB/22–23) in India and the Johns Hopkins Bloomberg School of Public Health IRB (23938) in the United States. Only interested and eligible respondents were approached by the qualitative researchers, and interviews were set up at a time most suitable for the respondents. Additional information regarding the ethical, cultural, and scientific considerations specific to inclusivity in global research is included in the Supporting Information (S1 File).

Data collection

The cognitive interviews were conducted by five female (ZT, AS, MS, PK & MSH) and two male (PM & CPV) qualitative researchers. All had master’s level social science education. Their experience as qualitative researchers ranged from five to 25 years; selecting experienced qualitative researchers was considered particularly important for cognitive interviewing, as effective tailored probing relies on researchers’ interviewing experience and skills, ensuring methodological rigor, ethical sensitivity, and credible interpretation during interviews [14]. The researchers were from the study region (Uttar Pradesh) or had worked there previously. A logistics manager initiated the process of recruitment by approaching the village head (sarpanch), explaining the study, and taking help from the sarpanch or a community health worker in identifying potential respondents.

Interviews were conducted at respondent’s homes in teams of two where one researcher was responsible for leading the interaction and the other was accountable for notetaking and supporting the interaction. The study information and informed consent was read to each potential participant and then summarized to ensure comprehension. All respondents provided informed oral consent to participate in the study and have the conversation recorded, apart from one woman who refused for the conversation to be audio recorded. Interviews averaged 52 minutes in duration.

Data analysis

Analysis began during fieldwork at our extensive team debrief sessions [14]. For each day of data collection (every eight to 12 interviews), the team engaged in a full day of debriefing to discuss emergent cognitive mismatches and identify potential revisions. During these sessions, language and translations were refined, question order was revised to improve flow, response options were amended, alternative questions were developed, and questions that did not resonate with the respondents’ context were eliminated. While debrief sessions were aimed at refining the tool until cognitive interviews demonstrated better comprehension of the survey questions amongst respondents, these sessions also helped inductively gauge a preliminary understanding of the types of recurring cognitive mismatches on the basis of which revisions to the tool were also made. After data collection, audio recordings of the 70 most substantive interviews were translated in English with key Hindi terms retained in the transcripts. The different types of cognitive mismatches identified through team discussions and debrief sessions were used to code interviews on a qualitative analysis software (Dedoose). The remaining 31 interviews that were not transcribed were analyzed through thorough reading of the extensive notes maintained in the debrief sheet.

Findings

Respondents were rural men (n = 44) and women (n = 57), who ranged in age from 18 to 47 years (average 30 years). Respondent characteristics are presented below (Table 1). Only 20% of the men in our sample and 18% of the women had nine or more years of education, which is lower than the Uttar Pradesh rural average of 81% of men and 62% of women [18]. All the men and 81% of the women in our sample reported owning their own phones, while the remaining 11% used someone else’s phone or shared a household phone. This phone ownership status was higher than the Uttar Pradesh average: NFHS-5 [19] found that 94% of rural households possessed at least one phone (no data specific to men) and 42% of rural women had their own phones. 84% of our male respondents and 56% of our female respondents owned smartphones. While there is no comparable data on smartphone ownership, our sample likely had higher smartphone ownership than the Uttar Pradesh average: a representative 2022 survey of households with school-going children in Uttar Pradesh found that 68% of households had a smartphone [19].

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Sample population (N = 101).

https://doi.org/10.1371/journal.pone.0354874.t001

Nearly half the respondents belonged to the “Other Backwards Caste” category (a somewhat marginalized group), almost 40% were in the highly marginalized Scheduled Caste/Scheduled Tribe category and the remaining 13% were in the “General” category (not marginalized). About a third of the respondents reported their annual household income when asked, which ranged from Rs 35,000 (USD$427) to Rs 11,00,000 (USD$13,414), with an average of Rs 1,82,000 (USD $2219) across families ranging in size from 4 to 13 people. Although a number of dialects of Hindi are also spoken across Uttar Pradesh, as well as the distinct languages Urdu and Bhojpuri, all potential respondents whom our researchers approached, and all respondents included in our study were Hindi speakers and completely comfortable conversing in Hindi.

Drawing on previous studies [12] concerning the typology of survey question failures, we identified seven question issues that affected comprehension (Table 2). Each of the seven issues from Table 2 are elaborated upon in the sub-sections below.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Typology of problems identified in the digital survey questions.

https://doi.org/10.1371/journal.pone.0354874.t002

1. 1. Inappropriate terminology

Although the researchers sought to select commonly used Hindi words for the first draft of the questions, many words selected were unfamiliar to our study population, especially those with lower education levels. One interviewee (F27), when asked a question about being exposed to “inappropriate or offensive things” [anuchit ya aaptijanak cheeze] on the internet, summarized this issue by replying: “Please speak in Hindi. What does the question mean?” In fact, the questions were entirely in Hindi but used more formal Hindi words that were not familiar to respondents. In most cases, we were able to identify a more appropriate word or phrasing (Table 3) that is commonly used in everyday language to improve comprehension among respondents of differing educational backgrounds.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Vocabulary issues and potential resolutions.

https://doi.org/10.1371/journal.pone.0354874.t003

While many words were completely unfamiliar to respondents, others were understood but interpreted in a manner consistent with their context but misaligned with the researchers’ intent. For instance, one question asked whether respondents had used the phone for work or business [kaam dhandhe]. While the question intention was focused on using phones for income-generating activity, some female respondents understood the question to be asking about domestic work.

Researcher: So, as I asked, since yesterday you have used a mobile for work or business. What does “kaam- dhandha” [business] mean to you?

Respondent: “Kaam-dhandha” means cooking food, washing utensils. All the household chores. (F06)

Furthermore, female respondents reported never speaking to or having friends when we used the term dost [friend]. Upon probing, we learned that dost was gendered as a male friend in this region. The term for female friend, saheli, allowed women to consider whether they spoke to female friends.

1. 2. Long and overly complex questions

Many questions were too long and complex, causing respondents to lose track of the question’s intent. For instance, we tested the following question about SIM use, drawn from a global survey: “You said earlier that you don’t own or have the main use of a mobile phone. However, do you own or have the main use of any SIM cards (i.e., mobile phone numbers) that you use in other people’s mobile phones at least once a month?” Most respondents were not able to meaningfully retain and process the five clauses (owning a SIM or having the main use of a SIM; a SIM is a mobile phone number, using one’s SIM in another person’s mobile, doing so at least once a month). In several cases, respondents lost the intent of complex questions completely, replying “I don’t know.” In some cases, however, respondents retained and addressed only part of questions, which did not align with question’s intent (Table 4).

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Examples of overly long and complex questions that led to partial retention.

https://doi.org/10.1371/journal.pone.0354874.t004

1. 3. Low resonance of digital concepts

Many concepts related to digital services were unfamiliar to respondents. In some cases, as with ‘the internet’ and ‘apps’, respondents were engaging with these technological tools but did not link their practices with the broader concept. For instance, majority of the respondents had not heard of the internet (or ‘net,’ a more common but not universal term in the area) but were in fact using WhatsApp regularly. Respondents did not know the term ‘app,’ but knew how to press the red icon on the phone to open YouTube. We were able to resolve this issue by introducing an explainer box that informed respondents that using specific digital services and programs was using the internet and apps. The explainer box was successful when it mentioned specific well-known products or companies such as YouTube, WhatsApp, Google, Meesho, Moj, Jio Cinema, Flipkart, and PayTM.

Other digital concepts, such as privacy policies, personal data, and hacking, were simply not resonant among the respondent population. Each of these terms brought unique challenges. We tested survey questions that sought to assess whether respondents read and understood privacy policies. However, the term privacy policy had no accessible direct Hindi translation. Attempts to describe a privacy policy involved long, complex sentences that were incomprehensible to some respondents. Among the respondents who followed, our attempted explanation mapped onto a different but related concept: the steps required to create user accounts rather than privacy policies.

Researcher [reading the draft question]: Many times, when we download a new app or go to a new website, before using it for the first time, we need to read some kind of notice [privacy policy] and give our permission. Have you ever seen something like this?

Respondent: Yes, while making an ID they ask for name and OTP [one-time-password]. (M20)

We tested questions that sought to assess whether respondents knew about data protection and companies collecting their personal data. To do so, we first established whether respondents understood ‘personal data’. The best translation was niji jaankaari [personal information]. However, this term has interpretations unrelated to data, such as family secrets or day to day practices in the home.

Researcher [reading the draft question]: Have you ever heard of personal information?

Respondent: Yes. Personal information means household, food, cooking, what has happened to whom in the family, who is going where, all that. Children’s studies. (F49)

Asking about online companies such as Facebook and WhatsApp collecting personal information added several layers of complexity. In addition to ambiguity around the concept of personal information, respondents understood companies to be physical factories rather than providers of online services, and conflated collection of personal data with how public or private the online space was.

Researcher: According to you, do companies or various apps like Facebook and WhatsApp collect your personal information?

Respondent: Facebook is not your personal life. That is known to all. When you are using Facebook, another person who is using Facebook, that person will know your number and identity. That person will know that this Facebook account is run by this person. That is not personal. But WhatsApp is my personal app. (F40)

We tested a question that sought to assess whether respondents’ phones or accounts had ever been hacked. The English word “hack” has entered global usage but had no relevance to our respondent population. Instead, it was instead consistently misheard as “hang,” another English term used in Hindi to mean a system freeze or slowdown.

Respondent: My phone was hacked but it got recovered.

Respondent: Like touch screen is not working or phone is not working.[…] if mobile’s storage is full then it gets hacked. (F23)

Attempting to describe the concept of hacking as someone accessing or controlling your phone from a geographically distant location did not resonate. Only one respondent (M38) understood this concept. For this and other digital concepts (privacy policy, personal data), we used explainer boxes, but they were met with limited success.

1. 4. Difficulties conveying permission and supervision

Questions about permission and supervision sought to gauge agency and surveillance over the use of digital devices. We asked respondents whether they needed permission to use the phone or were supervised when using the phone for different activities, and whether they thought these conditions should apply to women, men, young unmarried women, and young unmarried men in general. What we observed was a conceptual blurring between the idea of social control inherent to the English words permission and supervision, and the more benign concepts of notification (“I am taking the phone, I want to talk, tell me where the phone is kept.” F03) and assistance (“Educated people will use it on their own, others will ask” F08). The trickiness associated with these concepts was also context-driven where, despite owning a phone, both men and women reported sharing their phones with other family members when required, and it was thus common courtesy to ask for permission when using someone else’s phone or to inform others about using a shared phone.

Researcher: Do you think women should take someone’s permission (anumati lena) to use a mobile phone?

Respondent: Yes, they should take permission because if they don’t know how to use it, then they can ask. (F06)

Researcher: Ok then tell me, do you think women should use the internet on a mobile phone without having anyone watch or supervise them (bina kisi ki dekh rekh ke)?

Respondent: You can do as per your wish, but when someone will supervise you, then you can understand things easily. (F04)

After testing the questions, we observed that the control aspect of taking permission could only be assessed in relation to the respondent’s phone ownership status. Specifically, if a phone owner is required to seek permission before using their own phone, that would indicate control, as would having to always ask permission whenever one wants to use a shared phone. The control aspect of supervision continued to be tricky particularly because the actual Hindi word for supervision, nigrani, was not understood by respondents, the term nazar rakhna (keep an eye on) approximated the meaning better than dekh rekh (watch over).

1. 5. Problematic question construction and response options

While preamble text that simply helped respondents’ transition to a new section was not problematic, preamble text that provided important guidance to respondents was not retained. Relatedly, questions built around a “stem” and then several sub-components were unsuccessful because respondents did not retain the stem once they moved into the sub-components.

For questions which were worded similarly but had differences in key clauses, some respondents found it difficult to distinguish the questions and considered them to be the same question being repeated. For example, questions about communicating with strangers and communicating with men outside the home were understood to be asking the same thing, despite the research intent to differentiate the gender-agnostic “strangers” from the gender-specific “men” (Table 5).

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. Example of similar questions that respondents considered to be identical.

https://doi.org/10.1371/journal.pone.0354874.t005

1. 6. Self-practice bias

We sought to assess respondents’ perceptions of the attitudes held by their families and communities. Typically, our questions included a statement followed by a question such as “what do your family members think about this?” or “what do people in your community think about this?” We observed two major challenges; first, when questions were asked in a quantitative format the first time, respondents often reported their own practice or opinions. Either they did not understand that the question was seeking family/community members’ opinions or they conceptualized their experience as indicative of their family members’ views. Second, when probed further by researchers, some respondents still found it challenging to report others’ opinions and attitudes. Many respondents would either say that they didn’t know what their family members thought or that they were unwilling to speak on their behalf. Some had never had conversations about views on phone use and so were unable to arrive at an estimation of others’ opinions and attitudes.

Researcher: Women who use the internet on a mobile phone can earn more money. What do your family members think about this?

Respondent: I don’t earn money. Rather, I buy all the necessary things. (F15)

No, I don’t know so much about people. […] It does not happen in my family. (F42)

Researcher: Does this happen when women post photos or videos of themselves or others on Facebook, WhatsApp, YouTube or Instagram?

Respondent: I don’t do that madam. (F44)

Respondents indicated that their families’ views on these issues could be discerned only in relation to what was actually happening. That is, if women in the home were taking specific actions--using the phone to learn new things, saving time and effort through using the phone, helping their child children with schoolwork through using the phone--then the family was supportive. If women were not using the phone for family oriented and productive work, then it was difficult to know families’ perspectives on this potential action. Respondents also found it very unclear what “community” was supposed to mean, with some thinking of their caste community and others noting that they could not possibly know what everyone in the community thought. The best data was captured from reported personal practice rather than questions on opinions about others. However, questions can ask respondents what they think and what their families think to capture the difference between the two cognitive domains.

1. 7. Unclear time frames and recall expectations

We identified three issues with time frames and recall. First, most respondents struggled to report activity within pre-set categories. When asked whether they had done an activity within the last one month or within the last three months, respondents did not conceptualize frequent use as falling within these umbrella categories. Phone use was a frequent activity for many of the respondents, which made it difficult to report frequency of phone use behavior over a longer time period. For example, when asked whether they used a SIM card in their phone within the last one month, respondents who did so every day were unsure how to reply. A few respondents interpreted the question to be asking whether they had engaged in the particular activity one month ago or three months ago, rather than within one or three months. Some respondents also reported that thinking back over the course of a month or three months was too long a recall period. We found success with questions that asked whether the respondent had ever performed an activity, and, if yes, when they last did the activity.

The second challenge was with regard to aggregating and estimating time spent on activities. Respondents were unable to articulate what an ‘average’, ’typical’, ‘overall’ time period in a day was for a mobile phone-related activity. Instead, they sometimes narrated incidents related to the activity. Distilling multiple separate activities into a concrete set of hours and minutes was challenging for respondents.

If I get to use it, I will. If I don’t, if I’m busy, I can’t. If the phone is at home, I use it. And if I don’t, it’s not necessary. […] If I get a call it stays in my hands. If I am working or am busy, I keep it on a shelf or in the room or kitchen. If I go to someone’s house, I carry my phone with me. (F14)

I don’t use it every day. Only whenever one gets time. (F11)

It depends on the time of the day, and how long I talk. (F08)

When this question was edited to include the day or time of day the activity was carried out – for instance, ‘yesterday’, ‘whole day’, ‘mostly in the mornings’ ‘mostly in the evenings’ etc., it improved comprehension among respondents.

The third challenge was that the Hindi word for ‘week’ [saptah or hafta] was not consistently understood by respondents. A respondent said a week is made up of eight days, while another said two weeks constitute a month. Days were understood so rather than asking about a week, we found that asking about seven days was better understood. We also noted that respondents discussed time in relation to events, such as harvests and religious festivals. However, anchoring time-related recall questions in a survey would be difficult.

Challenges and limitations

Our sample over-represented phone owners and smartphone owners, particularly female smartphone owners. This sampling decision enabled us to collect feedback on a wider range of questions (rather than having to skip some questions due to irrelevance). However, it also skewed our sample towards women who are privileged in their access to phones and therefore, are likely more familiar with technical terminology related to phone use. To address this limitation, researchers conducting cognitive interviews would ask respondents whether certain terms were widely used in the village or whether alternative terms would be more commonly used. The researchers also discussed key issues (such as data privacy, whether phone use should be supervised, and whether people should ask permission before using the phone) with a wide range of respondents, not just smartphone owners.

We deliberately excluded more educated and urban people, reasoning that revising questions to be comprehensible to a less-educated rural population would also produce questions that were easily understood by more-educated urban respondents. However, this exclusion means we lacked insights from those with higher education and who live in urban areas. We are thus cautious about dropping questions on complex digital topics, such as personal data and hacking, since people outside our sampling population may have greater awareness of and opinions on these topics.

We noted that cognitive interviews are taxing for respondents. Unlike conversational interviews, they flow less naturally and ask respondents to explain thought processes and unfamiliar words or ideas, sometimes leading to frustration or anxiety about being tested. We mitigated these relational challenges by building rapport, clarifying our goals, expressing empathy when questions were confusing or repetitive, and emphasizing the value of their input.

Privacy during the interviews was another challenge. The respondents’ houses, where we conducted the interviews, often had limited private space and were occupied by family members who were curious about the research. Younger female respondents were often checked on throughout the interview by mothers-in-law, elder sisters-in-law or husbands. While we aimed for privacy, we did not insist on it to avoid compromising women’s safety. Instead, we explained the interview’s purpose to family members and politely requested space. Sensitive questions were postponed or skipped when privacy could not be ensured.

Finally, some of our questions on technology-facilitated harms or gender norms risked introducing new fears or problematic ideas. Asking whether women should be supervised or raising topics like hacking sometimes planted concerns that respondents had not previously considered. A few respondents said that our questions were introducing new reasons to worry or even to limit women’s phone use. While our research team was able to counter these views during a post-interview conversation with the respondents, such framing risks influencing public opinion if deployed at scale and must be carefully managed. Despite these challenges and limitations, cognitive interviews provided critical insights into question comprehension among rural Hindi-speaking respondents, enabling targeted refinements that enhance instrument accessibility and validity.