The crosswise model: A promising alternative to conventional RRT

Nonrandomized response techniques [8, 9], such as the crosswise model (CWM), represent recent advancements of the RRT. Questions in nonrandomized response format do not require an external randomization device and employ simpler instructions, supposedly making them easier to administer for the experimenter and easier to understand for the respondents. In the CWM, respondents are presented with two statements–a sensitive statement A (e.g. I have used cocaine) with unknown prevalence and a non-sensitive statement B (e.g. I was born in November or December) serving as a randomization device. Respondents are instructed to give a joint answer to these questions indicating whether “both statements are true or both statements are false” or whether “exactly one of the statements (irrespective of which one) is true”. Fig 1 shows the CWM as a tree diagram. Significantly higher and thus presumably more valid prevalence estimates have been obtained via the CWM as compared to DQ for sensitive attributes such as xenophobia [10, 11], plagiarism [12], tax evasion [13, 14], distrust in the Trust Game [14], crossing the street on a red light in plain view of children [15], the use of anabolic steroids among bodybuilders [16], intention to vote for the far-right German party Alternative for Germany [17], and prejudice against female leaders [18]. Moreover, in one strong validation study, the CWM accurately estimated the prevalence of experimentally induced cheating behavior, while DQ led to a severe underestimation [19]. Furthermore, the CWM is easier to understand than other RRT models and is perceived as significantly more confidential than DQ [20].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Tree diagram of the crosswise model.

The parameter π represents the unknown prevalence of the sensitive attribute, and the parameter p represents the known randomization probability.

https://doi.org/10.1371/journal.pone.0235403.g001

Cautionary evidence of false positives in the CWM

However, the results of two recent studies by Höglinger and Diekmann [21] and Höglinger and Jann [22] indicate that the CWM may sometimes produce false positives, that is, some non-carriers of the sensitive attribute are falsely classified as carriers. Höglinger and Diekmann [21] asked respondents whether they had ever received a donated organ and whether they had ever suffered from Chagas disease, both of which are attributes with a prevalence close to zero. As expected, DQ provided estimates that did not significantly differ from zero. In the CWM condition, however, the prevalence estimates for the two zero-prevalence items–and thus false positive rates–were 8% and 5%, respectively. In an additional individual-level validation, the authors asked about a somewhat sensitive control attribute (i.e. whether respondents had completed the German general university entrance qualification). Again, DQ provided a prevalence estimate close to zero; for the CWM, a false positive rate of 7% was observed. Remarkably, the CWM also produced a substantial number of false negatives, that is, some carriers of the sensitive attribute were falsely classified as non-carriers. As the false positives and false negatives cancelled each other out on the aggregate level, the overall prevalence estimates accurately reflected the known prevalence. However, the interpretability of this individual-level validation is limited because the relevant question was presented as a practice question in the CWM but not in the DQ condition, and because the prevalence estimates were compared with an external criterion that had been collected up to five years earlier and in a different response format. Finally, the authors found that the rate of false positives was moderated by the choice of the unrelated questions used for randomization. This finding implies that researchers using indirect questioning techniques must make a well-informed decision about which unrelated question to use.

In the second study, Höglinger and Jann [22] conducted individual-level validations via an online experiment in which participants had to play one of two dice games: In the prediction game, they had to predict the outcome of a die roll in private and were then asked to roll the die. Afterwards, to determine whether they qualified for a payout, respondents were asked to indicate whether they had rolled the predicted outcome. Since the predictions were made in private, cheating was observable only on the group level; an individual-level validation could only be computed by making two strong assumptions. First, it had to be assumed that all respondents whose predictions were correct actually claimed the payout; second, the false positive rate among respondents whose predictions were correct and who claimed the payout had to be assumed to be equal to the false positive rate among respondents whose predictions were incorrect and who did not claim the payout. In the roll-a-six game, participants had to roll a die and were then asked to indicate whether they had rolled a six, in which case they would receive a financial reward. In this second game, the outcomes were tracked, making cheating directly observable on the individual level. After each of the two dice games, participants had to answer a sensitive question about whether they had cheated in the respective game. On the aggregate level, the CWM estimates of cheating were significantly higher than the DQ estimates for both games, thus satisfying the “more-is-better” criterion. However, in both individual-level validations, the CWM produced more than 10% false positives, whereas the false positive rate in the DQ condition did not significantly differ from zero.

At this point, it is not yet understood whether false positives only occur under certain circumstances, or whether they pose a general threat to the validity of the CWM and of indirect questioning techniques as a whole. Höglinger and Diekmann [21] exploratively examined potential causes and correlates of false positives, but did not find a consistent pattern. Respondents who sped through the CWM instructions and may therefore not have understood them properly produced descriptively, but not significantly, more false positives. However, the reverse pattern emerged when only the sensitive questions were examined: here, speeders tended to produce fewer false positives. The authors hypothesized that the problem of false positives might be less severe in “better designed C[W]M implementations” (p. 5). Consequently, identifying conditions under which respondents show high levels of understanding and trust in the method could help to improve CWM implementation. Trust and understanding are necessary prerequisites for RRTs to yield valid results [23], but are often not achieved [20, 23–28]. Although the comprehensibility of the CWM, operationalized in terms of correct responses to scenario-based questions testing understanding of the model, was shown to be comparatively higher than the comprehensibility of other indirect questioning techniques, more than 16% incorrect responses were still observed [20]. Accordingly, Hoffmann et al. [20] suggested employing detailed instructions and comprehension checks to ensure respondents’ understanding of and trust in indirect questioning techniques. Building upon these recommendations, the present study sought to investigate whether the validity of results obtained via the CWM can be improved by providing respondents with more detailed instructions.

The present study

We sought to obtain a deeper understanding of the conditions under which false positives and false negatives occur in CWM surveys, and how they affect measurement validity. To this end, we conducted a strong validation based on a known external criterion by employing the anagram paradigm introduced by Hoffmann et al. [19]. This paradigm induces cheating to generate a sensitive attribute with known prevalence in the sample. It allowed us to compare all prevalence estimates with a known true value, and to conduct separate analyses of false negatives among carriers and false positives among non-carriers of the sensitive attribute. Based on the results of Höglinger and Diekmann [21] and Höglinger and Jann [22], we hypothesized that prevalence estimates based on self-reports would suffer from both false positives and false negatives. Moreover, we expected false positives to occur more frequently in the CWM condition compared to the DQ condition [cf. 21, 22]. In contrast, we expected false negatives to occur more frequently in the DQ condition compared to the CWM condition due to the influence of socially desirable responding [cf. 10, 12, 13].

Most importantly, the current study sought to identify potential means of reducing false positives and false negatives in order to maximize the validity of prevalence estimates obtained via indirect questioning techniques such as the CWM. We therefore tested the assumption that an insufficient understanding of and trust in the method are major causes of false positives in the CWM. To this end, we experimentally manipulated the amount of information respondents received in the CWM instructions. Specifically, we compared two groups, one of which received detailed instructions combined with several questions assessing comprehension (CWM detailed), and the other of which received only brief instructions and no comprehension questions (CWM brief). We expected that false positives were less likely when respondents had a better understanding of the CWM (CWM detailed) than when they had only a superficial understanding of the method (CWM brief). Regarding the rate of false negatives, we did not have a directed hypothesis. On the one hand, it might seem reasonable to expect that a better understanding of the method reduces false negatives resulting from misunderstandings; on the other hand, a better understanding may also help respondents to present themselves as non-carriers, which in turn could increase false negatives.

Since comprehension of CWM instructions has been shown to be positively associated with education [20], and lower-educated respondents have been found to disobey RRT instructions more often [29], we additionally compared the false positive and false negative rates between a higher-educated (at least 12 years of education, the German Abitur) and a lower-educated subgroup (at most 10 years of education, the German Realschule). We expected a higher false positive rate among lower educated than among highly educated respondents.

Participants

Respondents were recruited by a commercial German online panel provider. To avoid a lack of understanding of the instructions due to language difficulties, a necessary prerequisite for participation was that respondents were German native speakers. Moreover, to avoid confounding education with age, we restricted the age range of respondents to 30 to 40 years. This homogeneity with respect to age helped maximize the statistical power for testing our main hypotheses because it reduced the variance in education that would have been present in a more age-diverse sample due to a general trend towards higher educational attainment among younger cohorts in Germany [30].

The survey was carried out in accordance with the revised Declaration of Helsinki [31] and the ethical guidelines of the German Society for Psychology [32]. In Germany, there is no binding obligation that research projects can only be carried out after approval by an ethics committee. Participation in the present study could not have any negative consequences for the respondents, and anonymity was ensured at all times. The respondents participated voluntarily and after informed consent was obtained. There was no risk that participation could cause any physical or mental damage or discomfort to participants beyond their normal everyday experiences. Therefore, ethics committee approval was not required according to the “Ethical Research Principles and Test Methods in the Social and Economic Sciences” formulated by the Ethics Research Working Group of the German Data Forum [33] and the “Ethical Recommendations of the German Psychological Society” [34].

Sample size was determined on the basis of a priori power considerations indicating that to ensure sufficient statistical power (1-ß ≥ .80), a sample of more than 1500 participants was required. We decided to allocate twice as many respondents to the CWM conditions than to the DQ condition to compensate for the lower efficiency of the CWM that is a consequence of the randomization procedure [7, 35].

The initial sample consisted of 3060 respondents, with an equal distribution regarding education (higher-educated: at least 12 years of education, the German Abitur; lower-educated: at most 10 years of education, the German Realschule) and gender (male vs female). Due to incomplete data, 347 respondents had to be excluded from the analysis (11.34% of the initial sample). This dropout was nonselective in terms of cheating on the anagram task, χ²(1, N = 2934) = 2.75, p = .098, Cramer’s V = .03. Dropout rates were slightly lower among higher-educated respondents (8.25%) compared with lower-educated respondents (13.13%). χ²(1, N = 3040) = 18.87, p < .001, Cramer’s V = .08. However, this effect was small and thus considered negligible. Respondents in the CWM detailed condition were more likely to drop out (19.20%) than respondents in the other conditions (CWM brief: 3.00%; DQ: 3.67%), χ²(2, N = 3002) = 211.75, p < .001, Cramer’s V = .27.

The final sample consisted of 2713 respondents (50.31% female) with a mean age of M = 34.73 years (SD = 3.15). Half of the respondents (49.98%) were lower-educated, while the other half were higher-educated (50.02%). Overall, 972 respondents (35.83%) were assigned to the CWM detailed condition, 1164 (42.90%) to the CWM brief condition, and 577 (21.27%) to the DQ condition. Respondents in the three conditions did not differ with regard to education, χ²(2) = 0.92, p = .632, Cramer’s V = .02.

Measures

Procedure

Respondents filled in an online questionnaire that began with a short introduction, followed by demographic questions asking about their gender, age, native language and highest school-leaving qualification. They were then given the instructions for the anagram task and had the opportunity to practice the task with two example anagrams. Next, respondents were informed that the actual task would now start and that if they could solve all three anagrams, they could take part in a lottery for 100€, 50€ and 30€. After the anagram task, they were given the opportunity to cheat on reporting the number of solved anagrams as described above. Subsequently, they were queried with regard to their cheating behavior in the anagram task in either the DQ, the CWM detailed, or the CWM brief format (between-subjects). After the sensitive question, the respondents were asked to evaluate the questioning technique, were debriefed and were then given the opportunity to participate in the lottery. In order to avoid discriminating against honest respondents, all respondents were given the opportunity to participate in the lottery regardless of whether they had answered honestly or dishonestly.

Statistical analyses

For parameter estimation and comparison, we formulated multinomial processing tree (MPT) models [37, 38] following the procedure outlined in previous studies [35, 39, 40]. The parameter π represents the prevalence of the sensitive attribute (cheating on the anagram task) and the parameter p represents the known prevalence of the non-sensitive attribute used for randomization (birth month, p = .158 according to official birth statistics [36]). Maximum likelihood estimates were obtained using the expectation maximization algorithm [41, 42] implemented in the software multiTree [43], version 0.46. Parameter comparisons and restrictions were assessed via differences in the asymptotically χ²-distributed log-likelihood statistic G² between an unrestricted baseline model and a restricted alternative model (e.g. π_{CWM_detailed} = π_DQ or π_{CWM_detailed} = .00).

To more thoroughly investigate the validity of the obtained estimates, we transferred the approach of analyzing false positives and false negatives detailed in Höglinger and Jann [22] to the multinomial framework. To this end, we first split the sample into two parts: respondents who claimed to have solved all three anagrams in the anagram task were categorized as cheaters, while respondents who reported having solved fewer than three anagrams were considered honest respondents. This categorization is justified by the fact that solving all three anagrams has been shown to be virtually impossible in a previous study [19]. We then formulated separate multinomial processing trees for cheaters and honest respondents, and within these subsamples, for the DQ and CWM conditions. Hence, the false positive rate was estimated as the proportion of carriers of the sensitive attribute (π) within the sub-sample of honest respondents for the respective questioning technique, and the false negative rate was estimated as the proportion of non-carriers (1- π) in the sub-sample of cheaters.

Parameter estimates, false positives and false negatives

Our analyses revealed significantly higher prevalence estimates in both CWM conditions (detailed: = 25.48%, SE = 2.21%; brief: = 30.78%, SE = 2.07%) as compared to the DQ condition ( = 11.79%, SE = 1.34%); CWM detailed vs. DQ: Δ = 13.69%, ΔG²(1) = 27.94, p < .001; CWM brief vs. DQ: Δ = 18.99%, ΔG²(1) = 56.74, p < .001; CWM detailed vs. brief: Δ = 5.30%, ΔG²(1) = 3.06, p = .080. The known prevalence of the sensitive attribute (DQ: 58.93%, CWM brief: 56.70%, CWM detailed: 59.26%) did not differ across conditions, χ²(2) = 1.63, p = .442, Cramer’s V = .03, and was underestimated by all questioning techniques; CWM detailed vs. known prevalence: Δ = 33.78%, ΔG²(1) = 210.75, p < .001; CWM brief vs. known prevalence: Δ = 25.92%, ΔG²(1) = 147.47, p < .001; DQ vs. known prevalence: Δ = 47.14%, ΔG²(1) = 558.69, p < .001. Thus, the CWM met a weak (“more is better”), but not a strong validation criterion, as it still substantially underestimated the known prevalence. Moreover, we detected substantial rates of false positives in all experimental groups (see Table 1 and S1 Appendix), with the highest rates in both CWM conditions (CWM detailed: 13.08%, CWM brief: 14.32%, DQ: 2.53%; this corresponds to a specificity of CWM detailed: 86.92%, CWM brief: 85.68%, DQ: 97.47%). In all conditions, the false positive rates were significantly higher than 0%, CWM detailed vs. 0%: Δ = 13.08%, ΔG²(1) = 20.99, p < .001, CWM brief vs. 0%: Δ = 14.32%, ΔG²(1) = 31.70, p < .001, DQ vs. 0%: Δ = 2.53%, ΔG²(1) = 165.09, p < .001. The false positive rates in both CWM conditions did not significantly differ from each other, CWM detailed vs. CWM brief: Δ = 1.24%, ΔG²(1) = 0.08, p = .771, but they were significantly higher than the false positive rate in the DQ condition, CWM detailed vs. DQ: Δ = 10.55%, ΔG²(1) = 10.82, p = .001, CWM brief vs. DQ: Δ = 11.79%, ΔG²(1) = 15.86, p < .001. Comparatively high rates of false negatives were observed in all conditions (see Table 1 and S2 Appendix). While the highest rate was found in the DQ condition (81.77%; sensitivity: 18.23%), the false negative was also substantial in both CWM conditions (detailed: 65.99%, sensitivity: 34.01%; brief: 56.65%; sensitivity: 43.35%). In all conditions, the false negative rates were significantly higher than 0%, CWM detailed vs. 0%: Δ = 65.99%, ΔG²(1) = 601.97, p < .001, CWM brief vs. 0%: Δ = 56.65%, ΔG²(1) = 522.21, p < .001, DQ vs. 0%: Δ = 81.77%, ΔG²(1) = 9918.94, p < .001. The false negative rates in all conditions differed significantly from each other, CWM detailed vs. CWM brief: Δ = 9.34%, ΔG²(1) = 5.15, p = .023, CWM detailed vs. DQ: Δ = 15.78%, ΔG²(1) = 18.47, p = .001, CWM brief vs. DQ: Δ = 25.12%, ΔG²(1) = 47.71, p = .001.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. False positives and false negatives in the total sample and split by randomness of responses, perceived comprehensibility, perceived confidentiality and perceived clarity of the questioning technique (standard errors in parentheses).

https://doi.org/10.1371/journal.pone.0235403.t001

Effects of education.

A split by level of education (high vs. low) revealed that false positives were particularly prevalent among lower-educated respondents (see Fig 2). In both CWM conditions, false positive rates were significantly lower for higher-educated respondents than for lower-educated respondents, CWM brief: Δ = 11.91%, ΔG²(1) = 4.33, p = .038; CWM detailed: Δ = 18.16%, ΔG²(1) = 8.13, p = .004. In the DQ condition, this tendency was not significant, Δ = 3.53%, ΔG²(1) = 3.23, p = .072. False negative rates (see Fig 1) were not affected by level of education, DQ: Δ = 2.36%, ΔG²(1) = 0.32, p < .574; CWM brief: Δ = 6.98%, ΔG²(1) = 1.51, p = .220; CWM detailed: Δ = 3.90%, ΔG²(1) = 0.43, p = .511.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. False positives and false negatives as a function of level of education and condition.

DQ = direct questioning, CWM brief = crosswise model with brief instructions, CWM detailed = crosswise model with detailed instructions and comprehension questions.

https://doi.org/10.1371/journal.pone.0235403.g002

Effects of self-reported response behavior and perception of the questioning technique.

Table 2 reports descriptive statistics for self-ratings of randomness of responses, perceived comprehensibility, perceived confidentiality, and perceived clarity of the questioning techniques. All of these variables were significantly intercorrelated (see Table 3); a Cronbach’s alpha of .70 indicated that they measured a homogeneous construct. ANOVAs and Bonferroni-corrected post-hoc tests indicated that the CWM detailed condition was evaluated as less understandable, less confidential and less clear than the CWM brief condition, which in turn was evaluated as worse than the DQ condition on all of these variables. Moreover, respondents in the CWM detailed condition indicated significantly more random responses than respondents in the CWM brief or DQ conditions (see Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics and results of ANOVAs for self-reported response behaviors and perceptions of the questioning technique split by condition.

https://doi.org/10.1371/journal.pone.0235403.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Correlations between self-reported response behaviors and perceptions of the questioning technique.

https://doi.org/10.1371/journal.pone.0235403.t003

Spearman rank correlations revealed that respondents who performed better on the comprehension questions (1 = ‘failed at least one comprehension question in the third attempt’, 2 = ‘comprehension questions solved in second or third attempt’, 3 = ‘comprehension questions solved in first attempt’), indicated lower rates of random responses (r_s = -.31, p < .001, n = 972) as well as higher perceived comprehensibility (r_s = .30, p < .001, n = 972), confidentiality (r_s = .21, p < .001, n = 972) and subjective clarity of the questioning technique (r_s = .35, p < .001, n = 972).

To determine whether respondents’ self-assessment of the randomness of their responses was associated with the validity of the results obtained, we identified respondents who had indicated that they strongly disagreed with the statement “I simply ticked anything” (80.9% of the sample). These respondents were classified as having provided “non-random responses”, while all other respondents were considered to have provided “random responses”. An exploratory split by this moderator variable revealed that false positive rates were substantially lower among respondents who indicated having provided non-random responses; this pattern was observed in both the CWM detailed and the DQ conditions, but not in the CWM brief condition. However, this decrease in false positive rates came at the expense of an increase in false negative rates in both the CWM detailed and the DQ condition. Similar results were observed for exploratory median splits of perceived comprehensibility, perceived confidentiality and perceived clarity of the questioning technique: Higher values on these variables were associated with reduced false positives, but also increased false negatives in the CWM detailed and the DQ conditions. However, these tendencies were only significant in the DQ condition, and only for splits with reference to perceived comprehensibility and perceived clarity. In a multinomial modeling framework based on binary trees, continuous variables such as perceived comprehensibility or perceived confidentiality cannot be included directly. To transform these variables in a binary format, we therefore applied median splits. For detailed statistics on these analyses, see Table 1 and S1 and S2 Appendices.

Completion times

A Kruskal-Wallis test showed that completion times for the experimental section of the questionnaire differed significantly across the three experimental conditions, χ²(2) = 2232.53, p < .001. Dunn-Bonferroni corrected post-hoc tests revealed that the detailed CWM instructions were associated with higher completion times (median: 380 seconds) than the brief CWM instructions (median: 43 seconds), and the brief instructions with higher completion times than the DQ instructions (median: 9 seconds), DQ vs. CWM brief: z = 20.14, p < .001; DQ vs CWM detailed: z = 45.61, p < .001; CWM brief vs. CWM detailed: z = 31.56, p < .001.

Limitations and future research directions

While the current study will hopefully contribute to a better understanding of the conditions under which false positives and false negatives occur in applications of the CWM, some questions cannot be answered on the basis of our data and should therefore be addressed in future research.

First, it would be interesting to gain a deeper understanding of the cognitive processes involved in the formation of false positives and false negatives in the CWM. As corroborated by our data, false positives are most likely a product of inadvertent instruction non-adherence. It seems rather unlikely that non-carriers try to make themselves appear to be carriers of the sensitive attribute by deliberately choosing an answer that does not correspond to their actual status. False negatives, however, could be a mixture of carriers inadvertently providing a false response due to instruction miscomprehension, and carriers deliberately choosing the answer that minimizes the probability of them being identified as a carrier. While a particular advantage of the CWM is that none of the answer options clearly excludes the possibility of carrying the sensitive attribute, one of the answer options is still associated with a lower risk of being identified as a carrier, depending on the randomization probability p. Given our data, we cannot answer whether and to what extent carriers pursued this strategy. Hence, future research should address this question via methods such as personal interviews and open-ended questions about how respondents arrived at their specific answers.

Second, our data cannot uncover the processes responsible for the large share of inaccurate responses provided by lower-educated respondents. While false positives could be reduced among higher-educated respondents when detailed instructions and comprehension questions were included, the false positive rate among lower-educated respondents was not affected by such measures. Future research projects should therefore continue to optimize conditions until both higher- and lower-educated respondents are willing and able to provide accurate responses. This is of particular importance when the attribute under investigation is moderated by respondents’ level of education (e.g., negative attitudes towards foreigners [29]), because differential comprehension levels might lead to erroneous conclusions.

Third, the present study highlights that respondents’ thorough comprehension of indirect questions is a necessary prerequisite for obtaining valid results. For this reason, the exact implementation of the questioning technique seems crucial. However, implementation details are often unknown due to insufficient documentation, and a considerable amount of research focuses exclusively on the development of new statistical models and ignores questions of feasibility and implementation. We therefore recommend that future research focus more on the procedural implementation and comprehensibility of indirect questioning techniques. In addition, we encourage researchers to contribute to the improvement of tools that capture respondents’ understanding of indirect questions, such as the comprehension checks employed in the present study. It would be desirable to design these measures in a way that ensures a thorough comprehension of indirect questioning techniques even among lower-educated samples.

Fourth, it remains unclear why the CWM was perceived as less confidential overall than DQ. This finding contrasts with the objective confidentiality guaranteed by the CWM and also contradicts a recent study in which the CWM’s subjective privacy protection was rated significantly higher than the protection provided by DQ [20]. Possible reasons might include the perceived high complexity of the CWM instructions as well as the between-subjects design of the current study, which could have prevented the respondents from establishing common reference frames [cf. 53]. Moreover, it is unclear why the CWM detailed format, which was supposed to enhance understanding, was perceived as less comprehensible than the CWM brief format, and why respondents were less sure of what to do in the CWM detailed than in the CWM brief condition. It seems likely that the comprehension questions in the CWM detailed condition raised respondents’ awareness of the complexity of the CWM instructions, leading them to subjectively perceive the questioning format as rather complicated, whereas respondents in the CWM brief condition received no feedback on their understanding of the instructions and thus might not have realized that they did not understand the procedure properly. Once again, the between-subjects design of the current study may have also prevented respondents from establishing common reference frames [cf. 53].

Finally, the harmful influence of false negatives was substantially more pronounced in the present study than the influence of false positives. Overall, this led to an underestimation of the known prevalence. Future research should thus also try to identify conditions under which false negatives can be avoided. To this end, we recommend that studies employing a strong validation approach, comprehensive instructions and comprehension checks also be conducted for other indirect questioning techniques (e.g. cheating detection models). Such studies should ideally compare the validity of different models across sensitive attributes with varying prevalence in order to explore the potential influence of the population value on the validity of the prevalence estimates obtained.

Practical implications

In light of the current results, the CWM can be recommended for application if–and only if–the sample under investigation is highly educated, and detailed instructions and comprehension questions are used. As the present results also show, however, the desirable positive effect of detailed instructions and comprehension questions on the validity of the prevalence estimates obtained comes at the expense of higher dropout rates and higher completion times. Moreover, our results call into question the application of the CWM in its current format among lower-educated samples. In order to obtain valid answers among lower-educated respondents, more research seems needed to find ways of improving such respondents’ instruction comprehension. The present results also highlight the importance of including measures of instruction comprehension as well as specific aspects of respondents’ subjective experience (such as perceived confidentiality and randomness of responses) in surveys of sensitive personal attributes. Moreover, the present study underscores the importance of strong validations, since only individual-level data allow for the detection of false positives and false negatives, and thus for a comprehensive assessment of a method’s validity [6, 22].