We present a lab-in-the-field experiment and surveys of marginalised Roma children in Slovakia to examine whether reminding Roma of their ethnicity reduces their performance in a cognitive task. Research on social identity and stereotypes has documented that when individuals feel that their social group is negatively stereotyped in a domain, their performance declines, which can reinforce discrimination. In an effort to break the cycle of negative stereotypes, we remind Roma subjects of either Roma or non-Roma role models. We find that the activation of a Roma’s ethnicity reduces cognitive performance. In contrast, Roma exposed to Roma role models outperform those reminded of their ethnicity and of non-Roma role models. We then attempt to understand the channels through which social identity and role models affect performance. We show that priming the identity of a Roma has a direct effect on confidence, decreasing performance.
Citation: Vecci J, Želinský T (2019) Behavioural challenges of minorities: Social identity and role models. PLoS ONE 14(7): e0220010. https://doi.org/10.1371/journal.pone.0220010
Editor: Isabel Novo-Cortí, Universidade da Coruna, SPAIN
Received: April 12, 2019; Accepted: July 6, 2019; Published: July 26, 2019
Copyright: © 2019 Vecci, Želinský. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: TZ acknowledges support from the Slovak Research and Development Agency under the Grant No. APVV-0125-12. Funder’s URL: https://www.apvv.sk/?lang=en The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Social identity commonly refers to an individual’s own perception of self, based on his or her membership in a group such as ethnicity, race, or gender . A person’s social identity provides a set of rules that govern group behaviour, as such social identity can help to explain behavioural differences across groups [2–7]. An important aspect of one’s social identity are its stereotypes; the physical, mental and psychological characteristics attributed to a typical member of a given social group. Once a set of characteristics are used to describe a social group, those characteristics influence the behaviour of people who are associated with the group . Stereotypes predominately highlight the largest differences between groups as such many stereotypes are exaggerated . For instance, a negative stereotype can generate negative perceptions about one’s self, leading people to perform worse than their abilities would suggest. This is commonly known as stereotype threat .
Stereotype threat is especially prevalent in populations that suffer from discrimination. The perceived negative characteristics of these groups in a particular domain tends to influence the group members’ self-beliefs . Negative stereotypes have been found to explain race-based performance differences on academic tests [8, 11, 12]. Asian American women were found to perform better on a math test when their ethnic identity is activated, but worse when their gender identity is activated . A similar stereotype threat was also found when comparing a task titled “sport intelligence” in one treatment and “natural athletic ability” in the second treatment on a sample of black and white college students. Black participants performed significantly worse in the “sports intelligence” diagnostic and, in contrast, white participants performed worse in the “natural athletic ability” diagnostic . Gender stereotypes were also investigated and a study  assessed the impact of reinforcing stereotypical information suggesting that women are less competent players on women’s digital gaming performance. The authors found that the stereotype threat condition had a negative effect on performance and affective responses, whereas no differences were observed when comparing the stereotype neutral and stereotype boost conditions. Further, in India, when the stereotypes associated with low caste subjects, who have historically been discriminated against, is made salient, the group’s performance declined compared to that of non-discriminated-against high caste subjects [2, 15].
While most of the literature studying stereotype threat focuses on adults, there is a smaller but growing literature using children. This literature shows that children are aware of gender and ethnic categories at a very young age , although evidence regarding the identification of their own ethnicity is rather ambiguous . Nevertheless, children as young as 6-7 have been shown to believe that (other) children from high socioeconomic status (SES) are better at school than children from low SES . Similar conclusions were obtained by a recent study  investigating the perceptions of stereotype threat by 7-11 years old children. The majority of studies involving a child sample focus on stereotype threat related to gender, in particular, females’ gap in maths abilities. Nonetheless, there is also a stereotype according to which boys are academically inferior to girls, which also was supported by experimental evidence . In a recent meta-analytic study  the effects of stereotype threat on performance of girls in maths, science and spatial skills tests were studied. The authors identified signs for the presence of publication bias which may significantly distort the literature on the effects of stereotype threat among school-age girls. This is in accordance with the suspicion of a previously published study  that used implicit and explicit methods to activate stereotype threat, but found no impact on school-age girls’ mathematics performance. The study thus suggests that stereotype threat either occurs all the time or only in very specific circumstances.
The evidence for stereotype threat related to children’s ethnicity and race has also been supported by numerous studies in contexts such as: the contribution of diagnostic testing to the achievement gap in African American students ; effects of stereotype threat on African American, American Indian, Latinos, Asian American, and European American children’s connection to school ; or the negative effects of stereotype threat on the school success of minority adolescents from Turkish and Moroccan backgrounds in Belgium .
Negative stereotypes can be particularly harmful as they can reinforce discrimination across groups. A negative stereotype increases negative thoughts about oneself and can reinforce a negative identity. In turn, this can directly affect confidence, aspirations and levels of effort expended by the individuals or groups, ultimately affecting their learning and performance, reinforcing negative attitudes towards the group, and terminally discrimination . A key component of this cycle is that negative stereotypes not only influence how a group member perceives him/her self, but also how others perceive the social group. For example, research has found that negatively stereotyped names can reduce an employer’s effort to inspect resumes , and influence wages and employment opportunities . See, e.g. [29–31] for further discussion on statistical discrimination and related literature. Thus, understanding the factors that influence group stereotypes and determining how stereotype threat can be reduced is important in countering discrimination directed towards social groups.
In this paper, we examine the effect of role models on social identity and, in particular, on social group stereotypes of Roma children. This is a particularly important in contexts, where, for instance, social identity and skin colour stereotypes affect the aspirations and performance of students , as previous research shows that numerous Roma students do not aspire to secondary education, which is also attributed to the lack of role models . We begin by investigating whether reminding members of a negatively stereotyped ethnic group of their ethnic identity affects their academic performance. Negative stereotypes are often numerous and may be continuously activated throughout an individual’s day. In an effort to break the cycle that negative stereotypes may perpetuate, we examine the activation of ethnic and non-ethnic role models on academic performance. Reminding people of their role models, individuals who are perceived as worthy of emulation , may remind subjects of a positive aspect of their social identity, offsetting negative self perceptions, improving performance, and thereby lessening the effects of negative stereotypes . Research on social identity confirms that similarity between self and others increases the likelihood of social comparison . By comparing ethnic role models to non-ethnic role models, we can also identify whether role models who are similar to participants in terms of ethnicity and perhaps upbringing have a different effect on subjects’ behaviour than more distant role models students may not personally relate to. Lastly, we attempt to understand the channel through which negative stereotypes and role models affect performance. We hypothesise that priming a subject’s ethnic identity may have a direct effect on confidence, resulting in a decrease in performance. Therefore, we test whether confidence changes when a subject’s ethnicity is made salient and whether being reminded of role models can offset any reduction in confidence.
Other common interventions aimed at offsetting the negative effects of stereotype threats implemented by researchers include, for instance, mentoring programmes. A field experiment  targeting female, minority, and low-income adolescents was shown to have significant positive impacts on math standardized test scores of all three investigated groups (in comparison to control conditions), thus reducing the stereotype threat consequences. Creating an environment of perceived equal treatment of students of different ethnicities has also been shown to buffer negative effects of discrimination and stereotype threat .
We employ an artefactual field experiment with 394 Roma children in Eastern Slovakia. Roma are the largest minority in Europe and, according to the EU, they suffer from widespread discrimination . In a recent report, Roma were found to be the minority discriminated against second most often, with 26% of Roma reporting experiencing discrimination, preceded by North Africans (31%), and followed by Sub-Saharan Africans (24%), Turks (20%), South Asians and Asians (10%), recent immigrants (10%) and Russians (6%) . This translates into Roma’s belief that negative attitudes exist towards them in education, housing, and employment . This context provides a unique opportunity to study the effects of social identity and whether reminding Roma of their role models can help to mitigate the effects of negative stereotypes on academic performance.
Similar to [4, 8, 39–41], we use a pre-experiment background questionnaire to make Roma ethnicity salient. Participants then take part in a simple math task to gauge cognitive performance. In two additional treatments using the background questionnaire, participants are reminded of Roma or non-Roma role models. We find that the activation of a Roma’s ethnicity reduces cognitive performance compared to the control of no ethnic or role model information. In contrast, students reminded of Roma role models outperform students reminded of their ethnicity and non-Roma role models, but they perform similar to the control group. We thus find that the reduction in performance when children are reminded of their identity can at least partially be attributed to a reduction in confidence. Our findings suggest that reminders of Roma role models can potentially mitigate this decrease in confidence.
The effect of role models on student outcomes has been investigated in a number of empirical studies. This research suggests that role models can significantly influence outcomes, as they have the potential to motivate individuals and serve as a source of inspiration . In this regard, numerous studies estimate the racial interactions between students and teachers [43–46]. They find a positive effect of similar race teachers on various student outcomes. Another study  uses survey data to examine whether minority students benefit from taking courses with a same-minority lecturer. The authors show that the performance gap in terms of class drop-out rates and grade performance between white and minority students was smaller when the minority students were taught by minority instructors. Bettinger & Long  investigate the impacts of policies designed to increase female representation on college faculties. Their research shows that female instructors positively influence course selection and major choice in some disciplines. Recently, Lusher, Campbell & Carrell  use administrative data from a large, diverse university in California, their results show a positive and significant increase in course grades when students are assigned teaching assistants of a similar race/ethnicity. However, ethnic “matching” in their setting may be too simplistic, as it does not necessarily take into account minority students’ perceptions of such teachers, and students may not recognise them as role models . Another study  used a representative survey data of middle school students in China and they show that female teachers alter girls’ beliefs about commonly held gender stereotypes, increase their motivation to learn, and ultimately, raise girls’ test scores and improves their mental status and social acclimation relative to those of boys. In contrast to commonly reported results, Puhani  uses a German administrative panel and shows only a very weak effect of teacher gender on either recommended or chosen school type for either boys or girls. He also finds no effect of elementary school teacher gender on either subsequent changes in school type or grade repetition. In part, such results can be explained by omitted variables bias, as pointed by . While  examines whether Barack Obama had a positive effect on the achievements of African Americans, he finds little difference in a range of outcomes including crime rates and labour force participation between African Americans and whites. Further, role models have been shown to have an effect on adolescent consumer purchase intentions and purchase behaviour , as well as on adolescents’ attitudes toward violence and violent behaviour . Related is also the stream of literature on leaders, who have the potential to act as role models and they can influence people’s behaviour in social domains like charitable giving, tax evasion, corporate culture, and corruption .
There is a large psychology and growing economic literature on social identity, and, separately, on role models. Our work differs from this research in four important ways. First, while there is a large body of work identifying the importance of social identity and similarly a large body of literature on understanding the impact of role models, the literature bridging these two fields is relatively rare. In this respect, the papers most similar to ours are the novel works [56, 57]. Study  investigated the effect of role models when role models are perceived to be doubtful (i.e., express doubt about their abilities). The authors show that doubtful in-group role models hurt females, but were beneficial for males, while out-group role models had no differential effect on performance. Further, increasing the similarity of highly competent in-group role models reduces the stereotype threat experienced by females . These studies investigate the effects of in-group and out-group role models in a gender-stereotyped setting, while this paper assesses the effects of in-group/out-group role models on buffering the academic performance of negatively stereotyped Roma, i.e. stereotype threat based on ethnicity rather than gender. The two settings differ considerably. Clearly, in most societies women and men interact regularly, whereas, segregated ethnic groups such as the Roma (among others) often limit their interaction with the majority population . Further, unlike the current literature, we identify the participants’ actual role models, and then prime those role models, rather than the current procedure in the literature which involves researchers selecting people who may act as role models, such as teachers, and then testing if a role model effect exists. Unlike the current literature we also incentivise the experiment.
Second, our experimental approach allows us to explicitly observe and identify the effects of role models separately from other social and environmental factors. This is particularly difficult in empirical studies because many factors that influence student and role model interactions tend to be unobserved. For example, without an experiment it is difficult to isolate whether effects are due to students or teachers behaving differently. Third, we add to the literature by investigating confidence as a possible channel through which negative stereotypes and importantly role models affect behaviour. By influencing belief in one’s social group, role models may directly affect confidence. Credible empirical evidence on the effect of role models on self-confidence is rare, due to the difficulties of measuring and collecting data on confidence. Improved self-confidence has been found to impact: motivation ; firm outcomes ; labour market outcomes ; wage rates ; and the persistence in intergenerational income and educational inequality . Finally, in this paper we study the behaviour of an understudied but important group—the Roma community in Europe. Roma are the largest minority in Europe, and experience far lower living standards than other groups. Approximately, 20% never finish a single grade of primary school, less than 30% of European Roma are in paid work, and 87% live below the national poverty line, according to the European Union Agency for Fundamental Human Rights . To overcome deprivation and bolster inclusion, the EU has spent over 7 billion euro since 2000 on inclusion and anti-poverty programs aimed at Roma . We investigate the effectiveness of an inexpensive program that has the potential to bolster inclusion and increase academic performance.
Ethics approval was obtained from the Dean of the Faculty of Economics, Technical University of Košice, as the University does not have its own Institutional Review Board. Participation in the research was conditional upon parents’ signed consent on behalf of their children. Data was collected and analysed anonymously.
Upon commencement of the experiment, subjects responded to a “background questionnaire” that varied by experimental treatment [see S1 Appendix]. The experiment consisted of a control in which neither ethnicity nor role models were made salient, and three treatments: 1. Roma ethnicity salient (Roma salient); 2. Roma role model; 3. non-Roma role model. Following [4, 8, 39–41], we use a background questionnaire to make identity and role models salient, because this creates saliency without explicitly reminding the subjects of their ethnicity, thus reducing the probability of an experimenter demand effect.
In the control, neither stereotype or role models were made salient. The background questionnaire consisted of 5 nondescript background questions, such as the subject’s favourite food, favourite colour etc. The questions were unrelated to identity or role models. In the Roma ethnicity salient treatment, the background questionnaire included the same five simple questions asking subjects about their favourite food and colour, followed by three questions related to their ethnicity. These were: their self-declared ethnicity, language spoken most frequently at home and whether their grandparents spoke any language other than Roma. It is important to note that Roma children are often exposed to the non-Roma majority throughout societal interactions, in particular at school, where the majority of teachers are non-Roma. Thus, it is entirely plausible that their identity may be salient in many daily activities, making the Roma salient treatment a plausible representation of reality. Further, in the background questionnaire we use the term “nationality” instead of the term “ethnicity”, which would be difficult for children to understand. With the reference to The Cambridge Dictionary , one of the meanings of nationality is “a group of people of the same race, religion, traditions”. In addition, the Slovak language strictly distinguishes between nationality in the context of ethnicity [národnosť, which is the literal translation of the term nationality] and nationality in the context of citizenship [občianstvo].
The background questionnaire in the Roma role model and non-Roma role model treatments did not directly include questions on ethnicity, but consisted of six simple questions regarding famous Roma/non-Roma role models. In this study, we define role models as nonparental adults who adolescents look up to and want to emulate. To gather information on the people Roma children look up to, we surveyed and informally interviewed 50 Roma adults and children, as well as teachers and community social workers two weeks prior to the experiment. Each interviewee was asked to list the people they most look up to, or that they think Roma look up to. A list was then compiled of the most popular Roma and non-Roma. In order to cover a broad group of role models, we selected popular male and female role models from a range of occupations. The following people were selected as Roma role models: Vladimír Oláh, a poet who established a Roma cultural association in Slovakia; Dr Ján Cibuľa, an activist who was nominated for a Nobel Peace Prize and who was a past president of the International Roma Union; and four popular Roma Slovak or Czech singers/performers. These were: Silvia Šarköziová, the lead female vocalist in the Gypsy Devils band; Igor Kmeťo, the front man of the popular band Kmeťoband; female vocalist Věra Bílá; and the members of a male Slovak band, Gipsy Kajkos. The participants were asked questions such as the maiden name of Věra Bílá, the number of band members in Gipsy Kajkos, the name of Silvia Šarköziová’s band, and the association established by Vladimír Oláh. The survey did not explicitly reference the role models’ ethnicity, however, with reference to shared group membership [67, 68], we believe that this treatment also primed ethnicity. As such comparison between the Roma role model treatment and Roma salient treatment will reflect the effect of Roma role models. We selected this design as we believe it is representative of reality. It would be very difficult to separate the two effects—Roma identity salience and Roma role models as one cannot make a Roma role model salient without also making Roma identity salient. In both treatments (Roma salient; Roma role models), our aim was thus not to remind the subjects of their ethnicity directly.
To ensure that the role models were comparable across treatments, we selected the four most popular non-Roma musicians and two non-Roma role models from other fields. Similarly, we ensured there were two female non-Roma role models. The non-Roma role models consisted of two football players, Cristiano Ronaldo and Marek Hamšík, from Portugal and Slovakia respectively, and four singers/performers; Justin Bieber (Canada), Shakira (Columbia), Helena Vondráčková (Czechia) and Karel Gott (Czechia)—the latter two having been popular in Slovakia (and formerly Czechoslovakia) for decades. Questions included the soccer clubs of the players, the age of Justin Bieber and whether the singer Karel Gott has been awarded “The Golden Nightingale” prize more than 20 times.
To ensure the icons selected are actually role models of the subjects, as part of the post experiment survey we asked whether subjects admire the people mentioned in the background questionnaire. In the Roma role model treatment, 88% of subjects and 94% in the non-Roma role model treatment report looking up to the people/groups mentioned. This is consistent with the existing literature according to which entertainers/athletes are amongst the most frequently chosen role models by children  and suggests that the experiment does focus on people the Roma children look up to. This paper differs from a large part of the literature selecting people such as teachers as role models and then testing if a role model effect exists .
After participating in the background questionnaire, subjects took part in a numeric task. Participants were given 65 strings of numbers, each string containing between 10 and 20 digits [S2 Appendix]. Subjects were given 3 minutes to count the zeros in each string. For each correctly solved puzzle, the children received 5c. We selected this task as it required mathematical ability, yet it was simple enough for children who have not completed primary school to understand. Roma children are often stereotyped as having low academic ability, and their numeracy skills are perceived to be lower than those of the average population (see the next section). Further, the task was labelled as a “numeric maths task” to emphasise the task’s mathematics nature. The instructions included an example practice string of digits that subjects completed before they began the actual task.
After understanding of the instructions had been checked, but prior to the commencement of the numeric task, a measure of confidence was elicited. We examine confidence as a possible channel through which identity and role models affect behaviour. Similar to [71, 72], subjects were asked to predict their own performance in the task. More specifically, we follow  and ask about confidence before the numeric task, to avoid actual behaviour influencing a subject’s guess. Subjects were asked to estimate the number of numeric puzzles they expected to complete in 1 minute. Then, in the post experiment survey they were asked to elicit relative confidence: If you were to compare your performance with everyone else in this community, how would you rate your performance in comparison to other people? Possible answers were: “among the best, better than average, same as average, worse than average and among the worst”. The latter question was asked in the post experiment survey in order to avoid priming ethnicity.
Setting and municipality selection
Roma children are the subject of our study. We specifically selected early adolescent children, as researchers have found that as children enter adolescence, they increasingly focus their attention on nonparental adults to identify models they want to emulate [73, 74]. We study Roma, as they are the largest minority in Europe and, according to the European Committee of Social Rights , they suffer from pervasive historic discrimination, which has further risen during the economic crisis. In EU Member States, 85% of Italians and 66% of French hold unfavourable views of Roma. In Greece, Britain and Poland, about half hold a negative view of Roma, and 40% hold this view in Germany and Spain . In the context of education, around 60% of non-Roma Slovak students reported an objection when asked to share a desk with a Roma student, and almost 50% report bad or very bad experiences with Roma people .
Roma minority are often stereotyped (by majority populations) as wild, noisy, dirty, and passively adapting to situations . In particular, the prevailing stereotypes about Roma include low education and lowered intelligence skills, their negative attitudes to work, misuse of the social system and dependence on social benefits, high fertility rate, inability and unwillingness to adapt to the standards of the majority society . According to a recent study based on a social constructivist theoretical framework, Slovak Roma were perceived by the majority population as “irresponsible dependants and deviants not valuing education and incapable of making wise decisions about their lives” [80, p. 71]. The negative stereotypes about Roma related to their cognitive abilities are often based on their educational outcomes.
In Europe, approximately 20% of adult Roma have attained higher than primary education , Roma children are five times less likely to attend compulsory primary education compared to the majority populations and 87% of the Slovak Roma population aged 18-24 leave school without completing secondary education and early school leaving rates of Roma range between 72–98% . Furthermore, between 80 and 95% of Roma speaking students sampled in an international PISA survey did not acquire basic cognitive skills and competencies . Besides, in Slovakia, around 91% of Roma people are at risk of poverty, with only 21% in paid work, and similarly, 87% of Roma in the EU are at risk of poverty, and only 35% of men and 21% of women are in paid work .
Such stereotypes may be internalized by Slovak adolescents, who are the least tolerant towards Roma—39% of respondents consider Roma to be “inferior people”, while 5.9% of students studied considered Africans, and 6.5% considered Asians to be “inferior people” . Negative attitudes towards Roma are not only expressed by students, but Roma students are also negatively perceived by teachers. According to a study on ethnical attitudes of teachers in Slovakia , surveyed teachers held tremendously negative attitudes towards Roma (mean score of 6.3 on the 7-point Bogardus social distance scale, where category “7” represents “Would exclude [Roma] from entry into my country”). Similar attitudes were expressed by the general Slovak population, of which two thirds held an aversion towards Roma . The negative stereotypes about Roma have further been fostered by media, which often present Roma as criminals or asocial people . These tendencies are confirmed in our post experiment survey, which asks subjects “How often do you hear your classmates saying bad things about Roma?”, 41% report every day, 13% of respondents report at least once a week, while 22% report at least once a month. This suggests perceived negative perceptions towards Roma children exist within these communities.
The experiment was conducted in November 2015, in 15 municipalities in Eastern Slovakia, with Roma children living in segregated communities (on the outskirts of a municipality). It is estimated that Eastern Slovakia accounts for 85% of the total Roma population living on the outskirts of municipalities in Slovakia.
Using the 2013 Atlas of Roma Communities in Slovakia , we randomly selected 15 municipalities in four districts in Eastern Slovakia. The Atlas is a census of the Roma community initiated by the Slovak government and UNDP, aimed at monitoring the living conditions of the Roma population. Information was collected on all communities in Slovakia with 30 or more Roma people. Column 1 in S1 Table presents the average characteristics of the sample communities, while column 2 presents average characteristics for Eastern Slovakia municipalities (with Roma communities on the edge or outskirts). The results show that there is no statistically significant difference between the sample municipalities and other Eastern Slovakia municipalities in terms of characteristics including percentage of households with access to water, public sewer systems, household electricity/gas, and distance to kindergarten/primary school and physician. Distance to the closest general practitioner and pediatrician show a small statistical difference. All selected municipalities are not further than 30 km from the regional capitals, the cities of Košice and Prešov (see S1 Fig). To aid in comparing community characteristics, in S2 Table we report the average characteristics of our sample municipalities (including Roma and non-Roma) taken from the 2011 census and compare them to Eastern Slovakia and Slovakia. As expected, our sample municipalities contains a much higher rate of Roma than municipalities in either Eastern Slovakia or Slovakia in general.
Participant recruitment and procedure
To recruit participants for the experiment, community representatives were contacted by telephone and/or email, and asked for permission to conduct research in their community centre. In each municipality, one local social community worker was then hired to invite and recruit individual households. The social workers were either employed with an NGO (ETP Slovakia—www.etp.sk) working on development programmes in the Roma communities, or were employed by municipalities. Social community workers were responsible for visiting Roma families and providing counselling services to them on a daily basis. Because of this relationship, a high level of trust towards them exists. Each social worker was trained in how to select and invite participants, were never informed about the topic of the research. The social community workers were directed to randomly select households from different parts of the municipality the day before the actual experiment. Community workers approached households, informing them that they could participate in research the following day with the possibility of earning some money, and that a session would last 60-90 minutes. Within each household unit, we invited both parents (or only the mother/father in single parent households) and two randomly selected children above the age of 8. There were 19 such cases in which more than two children from the same household participated. Upon agreeing to participate, a contact number for each household was recorded. On the morning of the experiment, participants were informed of the time of their session, some of which were advised they would serve as alternates. Only children who were able to read, write and understand basic math were eligible to participate.
Each session included four children from at least 2 households. Between four and six sessions took place in each municipality, cumulating in between 15–47 participants per municipality and 394 subjects in total. Usually four to six sessions took place in each municipality. In cases in which more suitable rooms were available, the number of sessions was higher. In the case of two municipalities, the experiments took place twice (with a three-week delay), though people from different communities were selected both times. We do not believe that information spillovers were a significant issue due to the task being explained to participants as a simple maths quiz, reducing the likelihood of participants identifying the research questions being investigated.
Experiments took place in the municipal community centres. Upon arrival, the participants were screened for eligibility and the consent form read aloud, and a hard copy was then signed by the parents. Adults then moved to a separate room, where they participated in a separate experiment. Upon entering the room, each child was placed at a desk and assigned a unique ID in order to ensure their anonymity. Each participant received their own envelope with answer sheets with their unique participant ID. Data was entered and checked by two research assistants. Three pilot rounds were conducted prior to the experiment (pilot data is not included in the sample).
S3 Table lists the demographics of the average subject in our experiment. Column 1 presents the full sample means, columns 2—5 report the averages by treatment where “C” refers to the control, “T1” the Roma salient treatment, “T2” the Roma role model treatment, and “T3” is the non-Roma role model treatment. Participants in the sample are 13 years old on average and are currently attending regular primary school (66.5%). Within participating households, 70% of the children’s parents are married and 63% have at least one parent that completed primary school. Almost 46% of households have a parent who is employed (full/part-time job or activity in small community services). The mean household income of our subjects is 409 euro per month.
In this section, we provide details of the experimental results. In order to understand the effect of a subject’s ethnicity on performance, we compare the control to the Roma salient treatment. To examine if subjects reminded of role models have better outcomes, we compare the control to the role model treatments. We compare the role model treatments to the ethnicity salient treatment to identify if reminding children of role models with the same (different) ethnicity reduces negative self perception.
Table 1 displays the average number of numeric puzzles solved, by treatment. Column 4 reports the differences in means and the associated level of statistical significance. In the Roma identity revealed treatment, Roma children solve 8.4% fewer numeric puzzles than when identity is not primed (control). This result is consistent with the negative stereotype threat identified in the literature and suggests that Roma children hold a negative stereotype about their ability in this math task.
Can role models offset this stereotype threat? Children in the Roma role model treatment solve 9.6% more puzzles than in the ethnicity salient treatment. Making Roma role models salient reduces the negative self-perception of Roma children that appears when subjects are reminded of their ethnicity. However, this role model effect only appears when the role model is from the Roma society, and not when a role model is non-Roma. Subjects reminded of non-Roma role models solve 2.6 fewer puzzles than subjects in the Roma role model treatment. We also find statistical difference between priming Roma role models and non-Roma role models.
Fig 1 presents the distribution of the number of math puzzles solved, by treatment. Using a Kolmogorov-Smirnov test, the null hypothesis of equality of distributions between the Roma salient and the control is rejected (p–value = 0.012). The mass of the distribution of the math puzzles solved by those in the Roma salient treatment lies to the left of the distribution when identity is not revealed. We also find that the distribution of the Roma role model treatment lies to the right of that of the Roma salient treatment (K-S test, p = 0.060). The distribution of maths puzzles solved by subjects in the Roma role model treatment is similar to those in the control treatment (K-S test, p = 0.275). Finally, we find that the number of math puzzles solved by those in the non-Roma role model treatment lies to the left of those in the Roma role model treatment (K-S test, p = 0.016).
Note: The figure presents comparison of numeric puzzles solved by control and all treatments (kernel density estimations).
The differences in absolute performance among control and treatments are displayed in Fig 2 suggesting statistically significant differences between the control and Roma salient treatment (t-test, p = 0.097); Roma salient and Roma role models treatment (t-test, p = 0.035); and Roma role models and Non-Roma role models treatments (t-test, p = 0.046).
Notes: Mean number of numeric puzzles solved by control and treatments. Error bars represent SE estimates. Significance level indications: ***p < 0.01, **p < 0.05, *p < 0.10.
As a robustness test we examine treatment differences using regressions. Results are reported in Table 2. Column 1 reports results without controls, in the second column we include subjects, household and parental controls, and in the third column we add municipality dummies. The results are consistent with the non-parametric results across all three models. These results use the full sample, despite a small proportion of subjects who stated that they did not consider the Roma and non-Roma role models to be actual role models. Excluding these subjects may bias the result. When we re-estimate our main results in Table 1 excluding those who do not believe the role models are actually role models, we find the number of maths puzzles solved by children in the Roma role model treatment increases slightly to 28.11 while the non-Roma role model group decreases slightly to 25.09. The statistical significance of the results are quantitatively unchanged. Similar results are found when we re-estimate the Table 2.
In summary, we find that the negative stereotypes related to Roma identity can have a significant negative effect on achievement, however, this negative stereotype threat can be reduced by reminding children of role models associated with their in-group identity, but not with an out-group identity. However, reminders of role models do not lead to better performance relative to the control.
It is plausible that the number of puzzles solved is related to whether an individual correctly answers the survey questions. This may be problematic if correct answers differ across treatments. To test for this, we compare the number of puzzles solved and responses to the survey questions in the Roma Role model and Non-Roma Role model treatment. If subjects receive a boost from answering survey questions correctly, we would expect subjects in the Roma Role models treatment to answer more questions correctly relative to the Non-Roma Role model treatment. However, we find little robust evidence of this. In fact, in non-Roma role models treatment, subjects score 3.68 survey questions correctly relative to 3.06 in the Roma Role Models treatment (t-test p-value: 0.097; M-W test p-value: 0.106), suggesting that behaviour in the numeric task is not a result of how individuals respond in the survey.
The decline in academic performance of Roma children when reminded of their ethnic identity raises an important question—What are the channels through which negative stereotypes affect performance and can the influence of role models impact these channels? One hypothesis is that priming a subject’s identity may have a direct influence on confidence. Role models may improve confidence if participants are primed positively by providing them with an example of someone similar to them who has been successful.
Table 3 (Panel A, columns 1-3 and Panel B, columns 2-3) and Fig 3 present our first measure of confidence—participants’ ex-ante guess of the number of numeric puzzles they expected to solve in one minute. The results demonstrate that subjects’ ex-ante expectation of performance is lowest when they are reminded of their ethnicity. We find that children reminded of their Roma ethnicity expect to perform worse in the task compared to when their ethnicity is not made salient. In turn, subjects in the Roma role model treatment expect to perform better than those in the Roma salient treatment. We also find that the expectation of performance associated with the activation of a Roma role model is similar to that in the non-Roma role model and the control treatments.
Notes: The figure presents comparison of absolute confidence measure by control and all treatments. Error bars represent SE estimates. Significance level indications: *** p < 0.01, ** p < 0.05, * p < 0.10.
Intuitively, we would further expect the absolute confidence level to be positively correlated with subjects’ performance level. The correlation is, however, extremely weak, although positive. We believe a weaker correlation could exist for multiple reasons: 1) If subjects are either very overconfident or underconfident; 2) Since all subjects have never played this numerical task, they have little experience about their predicted performance. We conjecture that since we compare confidence across treatments, these two effects should be mitigated. 3) It is not uncommon in the literature to find low level of correlation between the confidence and outcome .
We report our second measure of confidence in column 4 of Panel A and column 4 of Panel B in Table 3. We create a dummy variable equal to 1 if a subject rated their performance as at least better than average compared to other people within their community. Subjects were asked to compare their performance with everyone else in their community. We find that children reminded of Roma role models are more likely to believe they will perform better than others in their community compared to children in the Roma salient treatment. There is little difference across the other treatments. Both measures are at least suggestive that Roma role models have a positive effect on confidence.
The results of this study indicate that reminding Roma children of their ethnicity reduces performance. This result is consistent with the theory of stereotype threat and the previous empirical studies in different cultural settings [2, 8, 10–13, 15, 23–25]. When a negative stereotype is associated with their social group, reminding children of their identity can be detrimental to their academic performance. We find that reminding children of a known role models from their own social group can improve achievement, as opposed to reminding children of role models from different ethnic groups. Our results, based on vicarious role models are hence in line with the results of studies employing role models such as teachers [43–47, 49], as well as the so-called “Obama effect” stream of literature [10, 70].
The desirability of a policy that emphasizes Roma role models is complicated by our finding that reminding children of Roma role models has a similar effect on performance compared to when nothing is highlighted, as in the control treatment. This suggests that not emphasising a child’s ethnicity may also be an important policy option. It is important to note that Roma are often in contact with and exposed to non-Roma (who are the majority); for example, the vast majority of teachers in Slovak schools are non-Roma. According to theories of context dependence , the settings in which decisions are made influence behaviour. In this context, the environment Roma are most often exposed to in schools and throughout societal interactions may be more reflective of the Roma salient treatment rather than the control, in which they are not reminded of their ethnicity. The fact that Roma children are in schools exposed to environment where their Roma ethnicity is made salient is also supported by the post-experimental survey data, according to which more than a half of participants reported that their classmates said bad things about Roma every day or at least once a week. Similar unfavourable perceptions of Roma were also reported by surveys conducted among Slovak students [77, 84] or teachers . This would imply the effect of Roma role models on academic performance maybe underestimated when comparisons are made to the control.
We also find that reminding Roma children of non-Roma role models may actually decrease performance in the maths task. It may be possible that role models with little resemblance to oneself may increase the belief that success is unattainable, which may lead to self-deflation decreasing performance. This finding is consistent with literature on shared social categorical self  and upward social comparison  and similar effects were observed in a study of females exposed to strong/poor-performing female/male peer . When females were exposed to strong-performing female (in-group), stereotype threat effects were eliminated, whereas the exposure of females to strong-performing males (out-group) resulted in an exacerbation of stereotype threat effects. In the light of this study, in our setting non-Roma role models can be thought of as strong-performing out-group models. To understand this mechanism would require further research.
Finally, we investigate confidence as a possible channel through which stereotypes and role models affect behaviour. We find that confidence decreases when subjects are reminded of their ethnicity relative to the Roma role model treatment and the control. This result suggests that the effects of negative stereotypes and role models on achievement at least partially operate through changed confidence. Because confidence plays an important role in behaviour, the effect of negative stereotypes can indirectly impact the outcomes of children in other areas, such as achieving aspirations and employment decisions, which may continue to affect their decisions later in life.
Social identity is an important and growing field of research in economics, in part because identity has a significant influence on behaviour. Its effect on behaviour can be welfare reducing when social groups are negatively stereotyped, such as the Roma group studied here. Roma are among the most socially excluded groups in Europe, facing significant rates of poverty and material deprivation, low levels of education and high levels of long-term unemployment [64, 82]. In order to overcome Roma deprivation and spark Roma inclusion, it is estimated that the EU and local governments spent 3 billion Euro on activities targeting Roma during the 2000-2006 programming period and a further 4.7 billion Euro between 2007-2013 on inclusion and anti-poverty governmental programs and initiatives . Role models may influence social identity and improve confidence and inspire young people to work harder to achieve goals. In the long-term, recognized role models can help to reduce discrimination and provide a cost effective tool to aid inclusion. This paper breaks new ground by experimentally examining whether different role models positively affect self-perception and reduce the effects of a negative stereotype, a small step in breaking the cycle of the negative perception of Europeans towards Roma.
S1 Table. Randomization at the community/municipal level.
S2 Table. Selected Census 2011 municipality characteristics.
S3 Table. Demographic characteristics of subjects.
We are grateful for comments and suggestions from Julie Chytilová, Lata Gangadharan, Phillip Grossman, Vilas Gobin, Andreas Leibrandt, Pushkar Maitra, Dillip Mookerjee, Paulo Santos, Julia Talbot-Jones and Kevin Wu. This paper benefited from comments by participants at the University of Gothenburg, Australian and New Zealand Workshop on Experimental Economics, Monash University, the Joint Annual Meeting of the Slovak Economic Association and the Austrian Economic Association, The Economic Science Association Annual Meetings and the Southern Economic Association Conference. We thank Slávka Mačáková of ETP Slovakia and her colleagues for assistance with organizing pilots and providing contacts in communities.
- 1. Tajfel H. Human groups and social categories. Cambridge University Press; 1973.
- 2. Hoff K, Pandey P. Discrimination, social identity, and durable inequalities. American Economic Review. 2006;96(2):206–211.
- 3. Chen Y, Li SX. Group identity and social preferences. American Economic Review. 2009;99(1):33–50.
- 4. Benjamin D, Choi J, Strickland A. Social identity and preferences. American Economic Review. 2010;100(4):1913–1928. pmid:20871741
- 5. Akerlof GA, Kranton RE. Economics and identity. Quarterly Journal of Economics. 2000;115(3):715–753.
- 6. Akerlof GA, Kranton RE. Identity and schooling: Some lessons for the economics of education. Journal of Economic Literature. 2002;40(4):1167–1201.
- 7. Morita H, Servátka M. Group identity and relation-specific investment: An experimental investigation. European Economic Review. 2013;58:95–109.
- 8. Shih M, Todd P, Nalini A. Stereotype susceptibility: Identity salience and shifts in quantitative performance. Psychological Science. 1999;10(1):80–83.
- 9. Bordalo P, Coffman K, Gennaioli N, Shleifer A. Stereotypes. Quarterly Journal of Economics. 2016;131(4):1753–1794.
- 10. DellaVigna S. The Obama effect on economic outcomes: Evidence from event studies. University of California, Berkeley; 2010.
- 11. Steele CM. A threat in the air: How stereotypes shape intellectual identity and performance. American Psychologist. 1997;52(6):613–629. pmid:9174398
- 12. Steele C, Aronson J. Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology. 1995;69(5):797–811. pmid:7473032
- 13. Stone J, Lynch CI, Sjomeling M, Darley JM. Stereotype threat effects on Black and White athletic performance. Journal of Personality and Social Psychology. 1999;77(6):1213–1227.
- 14. Vermeulen L, Castellar EN, Janssen D, Calvi L, Van Looy J. Playing under threat. Examining stereotype threat in female game players. Computers in Human Behavior. 2016;57:377–387.
- 15. Hoff K, Pandey P. Making up people: The effect of identity on performance in a modernizing society. Journal of Development Economics. 2014;106(1):118–131.
- 16. Ambady N, Shih M, Kim A, Pittinsky TL. Stereotype susceptibility in children: Effects of identity activation on quantitative performance. Psychological Science. 2001;12(5):385–390. pmid:11554671
- 17. Morland JK, Hwang CH. Racial/ethnic identity of preschool children: Comparing Taiwan, Hong Kong, and the United States. Journal of Cross-Cultural Psychology. 1981;12(4):409–424.
- 18. Désert M, Préaux M, Jund R. So young and already victims of stereotype threat: Socio-economic status and performance of 6 to 9 years old children on Raven’s progressive matrices. European Journal of Psychology of Education. 2009;24(2):207–218.
- 19. Wegmann KM. “His skin doesn’t match what he wants to do”: Children’s perceptions of stereotype threat. American Journal of Orthopsychiatry. 2017;87(6):615–625. pmid:28277688
- 20. Hartley BL, Sutton RM. A stereotype threat account of boys’ academic underachievement. Child Development. 2013;84(5):1716–1733. pmid:23402479
- 21. Flore PC, Wicherts JM. Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis. Journal of School Psychology. 2015;53(1):25–44. pmid:25636259
- 22. Ganley CM, Mingle LA, Ryan AM, Ryan K, Vasilyeva M, Perry M. An examination of stereotype threat effects on girls’ mathematics performance. Developmental Psychology. 2013;49(10):1886–1897. pmid:23356523
- 23. Wasserberg MJ. Stereotype threat effects on African American children in an urban elementary school. The Journal of Experimental Education. 2014;82(4):502–517.
- 24. Mello ZR, Mallett RK, Andretta JR, Worrell FC. Stereotype threat and school belonging in adolescents from diverse racial/ethnic backgrounds. The Journal of At-risk Issues. 2012;17(1):9–14.
- 25. Baysu G, Celeste L, Brown R, Verschueren K, Phalet K. Minority adolescents in ethnically diverse schools: Perceptions of equal treatment buffer threat effects. Child Development. 2010;87(5):1352–1366.
- 26. Loury G. The anatomy of racial inequality. Harvard University Press; 2002.
- 27. Bartoš V, Bauer M, Chytilová J, Matějka F. Attention discrimination: Theory and field experiments with monitoring information acquisition. American Economic Review. 2016;106(6):1437–1475.
- 28. Bertrand M, Mullainathan S. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review. 2004;94(4):991–1013.
- 29. Guryan J, Charles K. Taste-based or statistical discrimination: The economics of discrimination returns to its roots. Economic Journal. 2013;123(572):433–468.
- 30. List JA, Rasul I. Field experiments in labor economics. In: Ashenfelter O, Card D, editors. Handbook of Labor Economics. North Holland; 2011. p. 103–228.
- 31. Altonji JG, Blank RM. Race and gender in the labor market. In: Ashenfelter OC, Card D, editors. Handbook of labor economics. Elsevier; 1999. p. 3143–3259.
- 32. Campos-Vazquez RM, Medina Cortina EM. Social identity and stereotypes by skin color. Aspirations and performance in young Mexicans. Trimestre Economico. 2018;85(337):53–79.
- 33. Messing V. Good practices addressing school integration of Roma/Gypsy children in Hungary. Intercultural Education. 2008;19(5):461–473.
- 34. Pleiss MK, Feldhusen JF. Mentors, role models, and heroes in the lives of gifted children. Educational Psychologist. 1995;30(3):159–169.
- 35. Good C, Aronson J, Inzlicht M. Improving adolescents’ standardized test performance: An intervention to reduce the effects of stereotype threat. Applied Developmental Psychology. 2003;24(6):645–662.
- 36. FRA. European Union minorities and discrimination survey (EU-MIDIS): Main results report; 2009. Vienna, Austria: European Union Agency for Fundamental Rights.
- 37. FRA. Second European Union minorities and discrimination survey: Main results; 2017. Vienna, Austria: European Union Agency for Fundamental Rights.
- 38. European Commission. Report on the implementation of the EU framework for national Roma integration strategies: Communication from the Commission to the European Parliament, the Council, the European Economic and Social Commitee and the Committee of the Regions; 2014. Brussels, Belgium: European Commission—Directorate-General for Justice.
- 39. Cadsby CB, Servátka M, Song F. How competitive are female professionals? A tale of identity conflict. Journal of Economic Behavior and Organization. 2013;92:284–303.
- 40. Chen Y, Li SX, Liu TX, Shih M. Which hat to wear? Impact of natural identities on coordination and cooperation. Games and Economic Behavior. 2014;84:58–86.
- 41. Cohn A, Maréchal MA, Noll T. Bad boys: How criminal identity salience affects rule violation. The Review of Economic Studies. 2015;82(4):1289–1308.
- 42. Hurd NM, Zimmerman MA, Reischl TM. Role model behavior and youth violence: A study of positive and negative effects. The Journal of Early Adolescence. 2011;31(2):323–354.
- 43. Dee T. Teachers, race and student achievement in a randomized experiment. The Review of Economics and Statistics. 2004;86(1):195–210.
- 44. Dee T. A Teacher like me: Does race, ethnicity, or gender Matter? American Economic Review. 2005;95(2):158–165.
- 45. Dee T. Teachers and the gender gaps in student achievement. Journal of Human Resources. 2007;42(3):528–554.
- 46. Ehrenberg RG, Goldhaber DD, Brewer DJ. Do teachers’ race, gender and ethnicity matter? Evidence from the NELS 1988. Industrial and Labor Relations Review. 1995;48(3):547–561.
- 47. Fairlie RW, Hoffmann F, Oreopoulos P. A community college instructor like me: Race and ethnicity interactions in the classroom. American Economic Review. 2014;104(8):2567–2591.
- 48. Bettinger EP, Long BT. Do faculty serve as role models? The impact of instructor gender on female students. American Economic Review. 2005;95(2):152–157.
- 49. Lusher L, Campbell D, Carrell S. TAs like me: Racial interactions between graduate teaching assistants and undergraduates. Journal of Public Economics. 2018;159:203–224.
- 50. Maylor U. ‘They do not relate to Black people like us’: Black teachers as role models for Black pupils. Journal of Education Policy. 2009;24(1):1–21.
- 51. Gong J, Lu Y, Song H. The effect of teacher gender on students’ academic and noncognitive outcomes. Journal of Labor Economics. 2018;36(3):743–778.
- 52. Puhani PA. Do boys benefit from male teachers in elementary school? Evidence from administrative panel data. Labour Economics. 2018;51:340–354.
- 53. Sansone D. Why does teacher gender matter? Economics of Education Review. 2017;61:9–18.
- 54. Martin CA, Bush AJ. Do role models influence teenagers’ purchase intentions and behavior? Journal of Consumer Marketing. 2000;17(5):441–454.
- 55. Gächter S, Renner E. Leaders as role models and ‘belief managers’ in social dilemmas. Journal of Economic Behavior & Organization. 2018;154:321–334.
- 56. Marx DM, Monroe AH, Cole CE, Gilbert PN. No doubt about it: When doubtful role models undermine men’s and women’s math performance under threat. The Journal of Social Psychology. 2013;153(5):542–559. pmid:24003582
- 57. Marx D, Sei-Jin K. Superstars “like” me: The effect of role model similarity on performance under threat. European Journal of Social Psychology. 2012;42(7):807–812.
- 58. Berlin J. Assimilated individuals and segregated communities: A Comparative study of housing and living related well-being of Finnish Roma and housed Gypsies and Travellers in England; 2015. University of Eastern Finland, Kuopio.
- 59. Bénabou R, Tirole J. Self Confidence and Personal Motivation. Quarterly Journal of Economics. 2002;117(3):871–915.
- 60. Camerer C, Dan L. Overconfidence and excess entry: An experimental approach. American Economic Review. 1999;89(1):306–318.
- 61. Kőszegi B. Ego utility, overconfidence, and task choice. Journal of the European Economic Association. 2006;4(4):673–707.
- 62. Fang H, Moscarini G. Morale hazard. Journal of Monetary Economics. 2005;52(4):749–777.
- 63. Filippin A, Paccagnella M. Family background, self-confidence and economic outcomes. Economics of Education Review. 2012;31(5):824–834.
- 64. FRA. Poverty and employment: The situation of Roma in 11 EU member states (Roma survey—Data in focus); 2014. Vienna, Austria: European Union Agency for Fundamental Rights.
- 65. European Commission. The European Social Fund and Roma. Brussels, Belgium: European Commission; 2010.
- 66. Cambridge Dictionary | Free English Dictionary, Translations and Thesaurus;. Available from: http://dictionary.cambridge.org/.
- 67. Blanton H, Crocker J, Miller DT. The effects of in-group versus out-group social comparison on self-esteem in the context of a negative stereotype. Journal of Experimental Social Psychology. 2000;36(5):519–530.
- 68. Morgenroth T, Ryan MK, Peters K. The motivational theory of role modeling: How role models influence role aspirants’ goals. Review of General Psychology. 2015;19(4):465–483.
- 69. Anderson KJ, Cavallaro D. Parents or pop culture? Children’s heroes and role models. Childhood Education. 2002;78(3):161–168.
- 70. Marx DM, Ko SJ, Friedman RA. The “Obama Effect”: How a salient role model reduces race-based performance differences. Journal of Experimental Social Psychology. 2009;45(4):953–956.
- 71. Niederle M, Vesterlund L. Do Women Shy Away From Competition? Do Men Compete Too Much? The Quarterly Journal of Economics. 2007;122(3):1067–1101.
- 72. Dasgupta U, Gangadharan L, Maitra P, Mani S, Subramanian S. Choosing to be trained: Do behavioral traits matter? Journal of Economic Behavior & Organization. 2015;110:145–159.
- 73. Erikson E. Identity: Youth and crisis. New York: Norton; 1968.
- 74. Scales PC, Gibbons JL. Extended family members and unrelated adults in the lives of young adolescents: A research agenda. The Journal of Early Adolescence. 1996;16(4):365–389.
- 75. European Committee of Social Rights. European Roma Rights Centre (ERRC) v. Bulgaria: Report to the Committee of Ministries. Strasbourg, France: Council of Europe, European Committee of Social Rights; 2009.
- 76. PRC. A fragile rebound for EU image on eve of European Parliament Elections; 2014. Washington, DC, USA: Pew Research Center.
- 77. Sloviková M. Tolerant behaviour model of youth in family and school; 2012. Bratislava, Slovakia: Institute of Information and Prognoses of Education.
- 78. Villano P, Fontanella L, Fontanella S, Donato MD. Stereotyping Roma people in Italy: IRT models for ambivalent prejudice measurement. International Journal of Intercultural Relations. 2017;57:30–41.
- 79. Podolinská T. Roma in Slovakia—silent and invisible minority (Social networking and pastoral pentecostal discourse as a case of giving voice and positive visibility). Slovak Ethnology. 2017;65(2):135–157.
- 80. Miškolci J, Kováčová L, Kubánová M. Trying to include but supporting exclusion instead? Constructing the Roma in Slovak educational policies. European Education. 2017;49(1):71–88.
- 81. FRA. Second European Union minorities and discrimination survey: Selected findings; 2018. Vienna, Austria: European Union Agency for Fundamental Rights.
- 82. FRA. Education: The situation of Roma in 11 EU member states (Roma survey—data in focus); 2014. Vienna, Austria: European Union Agency for Fundamental Rights.
- 83. Bloem S, Brueggemann C. Student performance and inequality in Central and South Eastern Europe cross-country comparison and case study on Romani-speaking students in Slovakia; 2016. Budapest, Hungary: Roma Education Fund.
- 84. Bieliková M. Expressions of intolerance, violence and extremism by young people aged 13 to 18; 2010. Bratislava: Iuventa.
- 85. Rosinský R. Ethnical attitudes of teachers, students, and pupils (with the focus on Roma ethnic group). Constantine the Philosopher University in Nitra; 2009.
- 86. Gajdošová M. How do you perceive extremism?; 2013. Bratislava: IUVENTA.
- 87. Poláková E, Spálová L. Discriminatory tendencies of media presentation of Roma and features of extremism in social networks communication. In: Far-right extremism and youth in Slovakia. Brno: Tribun EU; 2013. p. 92–126.
- 88. Mušinka A, Škobla D, Hurrle J, Matlovičová K, Kling J. Atlas of Roma communities in Slovakia 2013. Bratislava, Slovakia: UNDP Europe and the CIS, Bratislava Regional Centre; 2014.
- 89. Tversky A, Kahneman D. Extensional versus intuitive reasoning: The conjunction fallacy in probabilistic reasoning. Psychological Review. 1983;90(4):293–315.
- 90. Turner JC, Oakes PJ, Haslam SA, McGarty C. Self and collective: Cognition and social context. Personality and Social Psychology Bulletin. 1994;20(5):454–463.
- 91. Collins RL. For better or worse: The impact of upward social comparison on self-evaluations. Psychological Bulletin. 1996;119(1):51–69.
- 92. Elizaga RA, Markman KD. Peers and performance: How in-group and out-group comparisons moderate stereotype threat effects. Current Psychology. 2008;27(4):290–300.