Survey instrument

Pre-registered sample size and power analysis

The sample of 2,000 was randomly divided into 3 groups of 675, 663, and 662 individuals each. In phone surveys, the KIIS may only conduct randomization during the survey, so the exact size of each group could not be predetermined.

The power analysis was conducted in the pre-registered report using Soper’s [27, 28] software. For the desired statistical power level of 0.8, the probability level of 0.05, and for 10 predictors (treatment, baseline attitudes towards Russia, ideology, plus a battery of demographic indicators), the required sample size per group should be between 333 and 549 respondents to discern the anticipated effect size of between 0.0305 and 0.0503 respectively (the effect size range is based on previous study by Chung and Woo [17]).

To discern a very small effect size in t-tests (Cohen’s d = 0.1), we would need 1,238 respondents per treatment to discern such an effect, i.e., it would be impossible given our group sizes. However, if Cohen’s d = 0.2, then our sample size would be sufficient, as only 310 respondents per treatment would be required to uncover such an effect.

Data inclusion criteria, and regional subsamples

No pilot data were collected for this study. The survey was fielded by KIIS in Ukraine in late May-June 2020 as a phone interview with a random sample of 2,000 individuals. The survey was described as a study on public opinion and included participants in the study if they were over 18 years of age. The KIIS obtained a verbal confirmation of informed consent from respondents. All translation procedures were conducted by the KIIS and verified by the authors.

Using software, the KIIS generated random mobile telephone numbers. After removing non-existing phone numbers, 2,000 phone numbers were randomly selected and contacted. The rate of mobile phone ownership in Ukraine is 96% among adults; furthermore, only 7% of respondents reported that they regularly use a landline phone, and only 1% of respondents reported no access to a mobile phone. These statistics are based on survey results obtained by the KIIS team face-to-face in February 2020.

During data collection, each respondent was randomly assigned into one of three groups, which resulted in 675 respondents assigned to the Eastern Slavic identity treatment, 663 to the national identity treatment, and 662 to the control (in phone surveys, the KIIS conducts randomization during the survey, so the exact size of each group differs slightly).

The KIIS randomized the language in which the enumerators initiated the interview: half of the respondents were greeted in Ukrainian and the other half—in Russian. After hearing the participants’ initial response, the enumerators switched to the language the respondents used. 57.1% of respondents continued the interview in Ukrainian and 42.9% continued the interview in Russian.

Тhe KIIS reweighed the sample based on four attributes (macroregions, type of settlement, age, and gender) in accordance with 2019 data collected by the Central Election Commission of Ukraine and the State Statistics Service of Ukraine. These weights were used in the regression analyses.

There are four well-defined macroregions in Ukraine: West (Chernivetska, Ivano-Frankivska, Khmelnytska, Lvivska, Rivnenska, Ternopilska, Volynska, and Zakarpatska oblasts), Center (Kyiv City as well as Kyyivska, Vinnytska, Zhytomyrska, Kirovohradska, Poltavska, Sumska, Cherkasska, Chernihivska oblasts), South (Dnipropetrovska, Zaporizka, Mykolayivska, Odeska, and Khersonska oblasts), and East (Kharkivska, Donetska, and Luhanska oblasts) [29]. Based on the country’s history and linguistic composition, most residents in the Western and Central regions of Ukraine tend to espouse an “anti-Soviet” identity ([30], p. 151) and to support Ukraine’s integration into the EU and NATO [29]. By contrast, most residents in the Southern and Eastern regions tend to be “Sovietophiles” ([30], p. 151) and tend to oppose Ukraine’s potential joining of the EU and NATO [29].

Besides using macroregions in weights, we return to regional differences to examine subsample-level effects of affirmation (section “Understanding the results: Regional heterogeneity in Ukraine” of the paper).

Terminating data collection and data exclusion criteria

The data collection was terminated after 2,000 complete survey units were acquired. At the data collection stage, the KIIS only considered whether respondents completed all survey items.

‘Careless or insufficient effort (C/IE)’ responses in surveys are often screened out based on extreme response times and ‘long-string analysis,’ which examines the longest string of identical responses [31–33]. While the former criterion is not applicable in a live phone survey, we screen out the longest strings of identical responses, using the baseline rule of thumb of a conservative cut score as half the scale ([31], p.16). Since most scales used in our survey have 5 response options, individuals who deliver a string of the same response to three or four consecutive Likert items should be removed. The consecutive Likert items appear in the dependent variables of our survey: four outcome questions that measure trust toward Russia all have the same answer options. 230 respondents gave identical answers to three consecutive outcome questions, with 162 of those respondents answering identically to all four questions.

Below we present the most conservative version of analysis where we removed all 230 respondents from the sample as insufficient effort responses. Additional analyses in the S1 Appendix replicate all tests excluding 162 respondents who gave four identical consecutive responses to outcome questions and also not excluding anyone. The results do not change.

Positive controls

The survey includes two manipulation checks to measure whether the treatments operated as intended. First, after each treatment, respondents were asked whether the task on values asked subjects to think about things Eastern Slavs (in treatment 1) or Ukrainians (in treatment 2) value about themselves or do not value about themselves. In the control group, the check asked whether the task on jellybeans made them think about flavors they think will be tasty or not tasty. These are tasks that have been used in psychological experiments testing identity affirmation [25].

Below we describe the proportions of those who passed or failed the manipulation checks, while removing careless and insufficient effort (C/IE) responses defined as those who reported three consecutive identical values in the outcome measure. The S1 Appendix also presents the results when removing those who reported four consecutive identical values. Table 1 presents the frequencies of manipulation check passing, failure, and no response using both types of C/IE definitions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Breakdown of answers when careless and insufficient effort (C/IE) responses are defined as three consecutive identical values in the outcome measures.

https://doi.org/10.1371/journal.pone.0270266.t001

Table 1 demonstrates that out of 598 individuals assigned to the overarching identity affirmation treatment (total number of assigned respondents was 675, but 77 subjects were removed as C/IE responses because they provided three identical consecutive values in outcome measures; all other responses are defined as ‘quality responses’), 45% of the quality responses (i.e., those deemed as sufficient effort responses) passed the manipulation check. 22% failed the manipulation check, reporting that the exercise made them think about values that Eastern Slavs do not value about themselves, 20% answered “Hard to say,” 1% said they did not want to answer the question, and the remaining 12% of respondents did not want to engage with the manipulation check at all, keeping quiet.

We observe similar rates among the 582 respondents who received the national identity affirmation treatment (out of 663 subjects assigned to this category, 81 were removed as C/IE responses): 46% of the quality responses passed the manipulation check, 26% failed, 17% answered “Hard to say,” 2% refused to answer, and 9% did not engage with the manipulation check question.

Among the 590 interviewees in the control group (out of 662 subjects assigned to control, 72 were removed as C/IE responses), we observe different rates of response distribution: 53% passed the check, 5% failed, 11% answered “Hard to say,” 1% refused to answer, and 30% did not engage with the manipulation check question.

To summarize, in the two treatment groups, 22–26% of respondents who provided quality responses reported that the treatment made them think about values unimportant to Eastern Slavs and Ukrainians respectively, i.e., they failed the manipulation check. These high failure rates suggest that about a quarter of treated respondents did not react to the treatment exercise in the anticipated way. Additionally, we observe high rates of “hard to say” answers, refusal to answer, and non-engagement with the manipulation check question (combined 28–42% out of quality responses). We revisit this unexpected outcome as well as high rates of various nonresponse in the discussion section by exploring the possible impact of regional heterogeneity in Ukraine.

The second positive control included in the survey assigned half the sample to self-report their attachment to the Ukrainian and Eastern Slavic identity before treatments were administered, while assigning the other half to do so following the treatments. We calculate the difference between ‘before’ and ‘after’ groups within treatment 1, within treatment 2, and within the control group and then compare if the resultant differences are statistically different between treated and control observations by calculating difference-in-differences (DiDs). The results are reported in section 3 of the S1 Appendix. These quantities are not statistically discernible from zero, which means that our confidence that the treatments manipulated one’s attachment to these identities does not increase, however we note that the preregistered design did not calculate power analyses to conduct this positive control check, i.e., we do not have sufficient power to uncover a small effect size when we split the samples into control and treated units). Nonetheless, the DiD analysis indicates that receiving the national identity affirmation (hereafter NIA) treatment moves respondents in the expected direction; the results are not as consistent for the overarching identity affirmation (hereafter OIA).

Resultant samples and descriptive statistics

Given the heterogeneity of responses to the manipulation check question, there are multiple ways to define which observations are considered “treated.” First, the most restrictive conceptualization of “treated” includes only those respondents who passed the manipulation check. Second, a less restrictive conceptualization of “treated” includes those respondents who passed the manipulation check or answered ‘hard to say.’ Third, the least restrictive definition of ‘treated’ only considers whether a respondent was assigned to a given treatment, regardless of their answer to the manipulation check. Additionally, ‘careless/insufficient effort’ (C/IE) responses may be defined as three or four identical consecutive responses. This yields a total of six samples. The S1 Appendix provides detailed breakdowns for each of these definitions.

The main analysis (included in the paper) reports the results for the subjects who either passed or answered “hard to say” to the MC question (the second definition), and defines C/IE responses as three identical consecutive responses. This sample includes 387, 367, and 376 respondents in the OIA, NIA, and control groups respectively. Table 2 summarizes the descriptive statistics for this sample that we use throughout the main analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics.

https://doi.org/10.1371/journal.pone.0270266.t002

Based on the pre-registered power analysis (see section ‘Pre-registered sample size and power analysis’), this sample size allows us to uncover a medium effect size in t-tests but not a small effect size.

The S1 Appendix includes all other results. Our results largely do not change based on the definition of “treated.”

Given high rates of failure to pass the manipulation check and high tendency of non-engagement with the manipulation check question, we also include a section—not planned in the preregistered design—that discusses the impact of Ukraine’s regional heterogeneity on these rates.

Using demographic data to ensure balance on observables

Using sociodemographic data, we compare the means of the treated and control units on observable attributes such as age, gender, education, income, location, and other collected control measures. As the preceding section summarizes, we use three definitions of treatment and two definitions of C/IE responses which produces six samples relative to those in the control group. The main analysis (reported in the paper) uses only one sample of 1,130 respondents (definition 2 of treated and three consecutive identical responses excluded as C/IE). For this sample, all 13 controls are balanced between the treated and control groups. Respondents with more positive views of Russia and older respondents were more likely to pass the MC after the NIA treatment or to say “don’t know.” We make sure to include the imbalanced controls in all regression models.

The S1 Appendix presents the analysis of whether covariates are balanced for all six samples. Section “Randomization” of the S1 Appendix concludes that Age, Education, Income, Retired, and Voted none are imbalanced under certain definitions.

Additionally, the S1 Appendix also analyzes whether excluded participants differed from included participants in any demographics or primary variables. Sections “Terminating data collection and data exclusion criteria” and “Resultant samples and descriptive statistics” explain the criteria for excluding observations. The results indicate that respondents who were excluded because they gave C/IE answers were quite different from those who gave quality answers: the C/IE observations tend to be men of higher income, those residing in Center-West, less likely to identify Russian as their native language, with a lower baseline affinity toward Russia, and exhibiting lower trust toward Russia. By contrast, those excluded because they failed to pass the manipulation check or refused to engage with treatments did not differ from the rest of the sample in many ways. First, more educated individuals with higher incomes, residing in larger cities were more likely to fail/refuse the MC after the OIA treatment. Second, those residing in the Southeast were more likely to fail/refuse the MC after the NIA treatment. Finally, older, retired individuals with lower incomes exhibiting lower trust toward Russia were more likely to fail/refuse the MC after the control exercise.

Comparing trust levels by treatment group

The preregistered design states that visualizations of the differences-in-means between treated and control subjects will be done in addition to presenting the results of t-tests. Since the differences-in-means (as described below) are not discernible from 0, we skip the visualization part of the design.

As section “Resultant samples and descriptive statistics” indicates, the strictest conceptualization of treatment does not allow us enough power to obtain differences-in-means. Table 3 presents some of the results of t-test analyses using definition 2 of treatment and defining insufficient effort (C/IE) responses as three identical consecutive values in the outcome measures. In addition, section 5 of the S1 Appendix presents all remaining results of t-tests conducted using definition 2 of treatment–which defines insufficient effort (C/IE) responses as four identical consecutive values in the outcome measures, as well as using all observations assigned to each treatment under both definitions of C/IE responses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Differences in trust between treated and control units.

https://doi.org/10.1371/journal.pone.0270266.t003

Table 3 shows that there are no statistically discernible differences in trust between the subjects treated with the Eastern Slavic identity affirmation and control group (top three rows) as well as between the subjects treated with the Ukrainian national identity affirmation and control group (bottom three rows). Three measures of trust are employed: Trust toward Russian Government (index of two survey items), Trust toward Russian People (index of two survey items), and Overall Trust (index of four survey items). All measures were standardized to have a mean of 0 and a standard deviation of 0.79–0.92. In all tests, the differences are substantively negligible and do not rise to the level of statistical significance. These results are inconsistent with hypotheses 1 and 2. The section on regional heterogeneity of Ukraine explains these results further.

Individual-level determinants of trust

Table 4 records the impact of overarching identity affirmation (OIA, Models 1–3) and of national identity affirmation (NIA, Models 4–6) on Ukrainians’ trust toward the Russian government (a standardized index of two survey items, Models 1, 4), trust toward Russian people (a standardized index of two survey items, Models 2, 5), and overall trust toward Russia (a standardized index of all four survey items, Models 3, 6). The analyses shown conceptualize treatment as passing the manipulation check or answering “hard to say” after the affirmation/control exercise, and C/IE responses are defined as three consecutive identical values in outcomes. In addition, section 6 of the S1 Appendix presents all remaining regression results conducted using both definitions of treatment as well as using all observations assigned to each treatment under both definitions of C/IE responses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. The impact of Eastern Slavic identity affirmation and Ukrainian national identity affirmation on trust toward Russian government, Russian people, and overall trust.

https://doi.org/10.1371/journal.pone.0270266.t004

Table 4‘s results may be summarized as: both affirmation treatments have a substantively negligible effect on trust that is not statistically discernible from zero. These results are inconsistent with hypotheses 1 and 2. Therefore, consistent with our analyses of group differences-in-means, we further corroborate the conclusion that neither identity affirmation approach impacted trust when we consider the full sample. We observe the same result in all other samples, as shown in the S1 Appendix.

While the experimental treatments have no impact, baseline attitudes toward Russia measured on a thermometer scale as well as one’s identification as ethnic Russian have strong positive correlations with how much a respondent trusts Russia (the impact of these two measures is statistically discernible in Models 1–6). Furthermore, one’s usage of Russian language during the interview is also positively associated with trust; this effect is statistically discernible in Models 1, 4–6. Residing in the Southeast region of Ukraine is also positively associated with trusting Russia; this effect is statistically meaningful in Models 2, 3, and 5. These results suggest that one’s ethnolinguistic and regional identity as well as preexisting level of affinity toward Russia explain why some respondents trusted Russia more.

We note that the preregistered design specified the usage of ordered logistic regressions, but since we combined items into continuous standardized measures of trust, OLS is a more appropriate tool. The S1 Appendix includes ordered logistic regressions using raw 5-category outcome survey items as robustness analyses; all results are consistent with those reported here, i.e., in the full sample neither identity affirmation treatment impacts trust toward Russia.

To summarize, we have followed the preregistered design to uncover no differences in trust between individuals whose Eastern Slavic identity or Ukrainian national identity was affirmed and those who received the control exercise; we conclude that in the full sample, the affirmation treatments produced no impact on the extent to which Ukrainians trust Russia, which is inconsistent with our expectations. The next section was unplanned in the preregistered design; however, we include it as exploratory analysis to probe why we see no differences in the full sample.

Understanding the results: Regional heterogeneity in Ukraine

Regional disparity in the impacts of group identity affirmations.

This section presents exploratory analyses of the effects of Eastern Slavic identity affirmation and of Ukrainian national identity affirmation on trust in regional subsamples of Ukrainians; this section was not preregistered, and we offer it to suggest a way to understand our results.

As described in the section on control measures, Ukraine has distinct macroregions: West, Center, South, and East. Different trajectories of historical development between Center-West on the one hand versus Southeast on the other hand have produced different ethno-linguistic composition (Southeast has more Russian speakers) as well as different political preferences (more than half of Center-West respondents supported the EU and NATO integration in 2015, while less than half did so in the Southeast) [29].

Due to these patterns, we obtain differences-in-means in trust between treated and control subjects in the two subsamples of West-Center vs. Southeast. These results are summarized in Table 5. Section 8 of the S1 Appendix also presents the results for samples in which careless or insufficient effort (C/IE) responses are defined as four identical consecutive values in the outcome measures and defining treatment as all assigned subjects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Regional differences in trust between treated and control units.

https://doi.org/10.1371/journal.pone.0270266.t005

The impact of the overarching identity affirmation (OIA) produces no effect on trust toward Russia in both Center-West and Southeast regions.

Table 5 also reports the impacts of NIA on trust measures in regional subsamples. In contrast to the expectation, the NIA treatment decreases trust by 0.09–0.12 units (the effects are significant in one-tail test, p = 0.09) toward government and overall trust in the Center-West region and increases trust toward government and overall trust by 0.22–0.30 units (p = 0.02) in the Southeast. The NIA treatment makes no difference on trust toward people.

Table 5 reveals that the control group’s baseline levels of trust toward Russia in West-Center are 0.28 units lower than control group’s trust in the Southeast. Furthermore, if we compare the reported levels of trust among treated subjects between regions, we observe that trust toward Russia is higher among treated subjects in the Southeast by 0.44 to 0.72 units. This means that exposure to the Ukrainian national identity affirmation generates the expected effect of increasing trust toward Russian government and overall trust among individuals who have a priori pro-Russian baseline views. Among individuals who have anti-Russian baseline views, the NIA treatment decreases trust toward government and overall trust (in a one-tail test). To sum up, the negligible substantive effects of the NIA treatment on trust observed in the full sample (Table 6) appear to be the result of disparate effects that NIA has in West-Center vs. Southeast.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Regional differences in failure/refusal to engage with treatments.

https://doi.org/10.1371/journal.pone.0270266.t006

Summary of findings

This project asks what type of identity affirmation is more likely to increase the extent to which Ukrainians trust Russians. We draw on two competing theoretical approaches to overcoming distrust—a common, supranational identity affirmation vs. national identity affirmation—to answer this question. We have followed the preregistered design [35] to uncover no differences in trust between individuals whose Eastern Slavic identity or Ukrainian national identity was affirmed and those who received the control exercise. We conclude that in the full sample, the affirmation treatments produced no impact on the extent to which Ukrainians trust Russians, which is inconsistent with our expectations.

However, our additional analyses suggest that regional heterogeneity of Ukraine is likely responsible for the null results of the NIA treatment in the full sample. These findings imply that the effects of identity affirmation on attitudes toward adversarial outgroups may depend on prior sentiments. Center-West vs. Southeast macroregions differ substantially in their levels of baseline affinity and trust toward Russia. We find that the national identity affirmation (NIA) treatment boosts Trust toward Russian Government and Overall Trust in the pro-Russian Southeast. By contrast, the NIA treatment decreases trust toward government and overall trust (one-tail test) in the more anti-Russian Center-West region. These are exploratory results, which is why we encourage future research to identify further scope conditions under which the NIA treatment does and does not increase prosocial attitudes toward adversaries.