Text-based causal inference

While there are several socio-economic and legal studies on text classification based on Natural Language Processing (NLP) and also some NLP-based analyses on language development, literature on text-based causal inference is scarce. The social science literature on text classification ranges from studies on differences in the linguistic style of posts in different online communities [4] and of comments on #MeToo articles in different news outlets [5] to studies that develop classifiers for political speeches in order to predict the speaker’s ideology [6] or identify their sentiment towards the topic discussed in the speech [7]. In the legal domain, [8] developed a document classifier for judicial opinions from U.S. circuit courts that classifies the opinions according to the predicted ideological direction (conservative vs. liberal) of the decision. In addition, there are several studies on classifying legal documents by topic (see, e.g., [9–11]).

The evolution of language over time has been analyzed using quantifiers for the context in which words are used (see, e.g., [12–14]) as well as based on indicators for the sentiment of that context (see, e.g., [3, 15]). [16] analyze how new members in a medical forum gradually adapt their language to the forum’s linguistic standards during their first year of forum participation.

Literature on text-based causal inference has emerged only in recent years. Some studies integrate NLP elements into causal inference for confounding adjustment (see [17] for a review). [18], e.g., develop a framework for estimating treatment effects which combines text-based matching and confounding adjustment based on text. They estimate how a scholar’s gender affects the number of citations of their publications, while controlling for and matching based on the content of publications. Other studies that rely on text-based confounding adjustment include those by [19, 20]. [21, 22] propose a text matching approach based on distance metrics rather than text classification. [23] integrate text classifiers into causal inference in order to tackle problems with missing data and measurement error.

Other studies use text as treatment or outcome. [24] analyze how a judge’s score on an indicator of gender-stereotyped language affects their decisions on women’s rights’ issues. [25] assess how wording in tweets affects the number of re-tweets. To do so, they apply different text vectorization approaches and quantify wording by means of sentiment, lexical distinctiveness and readability indicators, in order to identify words that in-/decrease re-tweet propensity. Similarly, [26–30] evaluate how to estimate causal effects of wording, semantics and lexical choices in texts.

[31] develop a sample splitting framework for estimating treatment effects with text as outcome, building on the classification of outcome texts based on a model trained in the training sample. [32] assesses how troll activity that promotes a pro-government agenda on Russian social media affects the evolution of online discussions using a regression discontinuity approach. To do so, he models the development of conversations on social media as changes in the mixture of topics with topics identified through NLP-based classification. Other examples of studies with texts as outcome include a study by [33] on how Reddit’s 2015 anti-harassment policy affected the usage of hate speech, a paper by [34] examining the influence of social norms on the rise of hate speech following terrorist attacks, as well as an analysis by [35] on the effect of tagging articles as not written in a “neutral point of view” on the development of lexical patterns in the labeled articles.

The present study contributes to this expanding field of text-based causal inference by integrating different text quantification approaches with causal inference methods. It develops a novel indicator drawing from psychology literature on the relationship between grammatical structure and perceptions of responsibility, while also employing established text quantifiers such as sentiment analysis and word2vec. Additionally, it proposes an approach to use text vectorization for quantifying language development within a text corpus for later integration into causal inference. In essence, it provides a diverse set of approaches to quantify text that can then be incorporated as outcomes in causal analyses.

The #MeToo movement and its societal, cultural and political impact

After starting as an online campaign against sexual harassment, #MeToo soon evolved into a movement that led to extensive and sustained media coverage of the prevalence of sexual violence in society. It prompted discussions about abuse of power, rape myths and the importance of supporting victims of sexual assault.

The phrase “Me Too” was initially coined by social justice activist Tarana Burke who began using this phrase in 2006 to campaign for the empowerment of sexual violence victims, particularly among women of color. It gained widespread attention in late 2017 after sexual misconduct allegations against Harvey Weinstein were exposed in a New York Times exposé by [36] published on October 5th, 2017. On October 15th, the actress Alyssa Milano tweeted a post encouraging victims of sexual harassment and assault to come forward by using the hashtag #MeToo. On Twitter, the hashtag was tweeted about 300,000 times on the day after Milano’s post, reached a peak of 750,000 tweets within 24h and was used on average more than 55,000 times per day during the year following the initial tweet [37]. On Facebook, the #MeToo conversation peaked at 4.7 million participating users within 24 hours, who engaged with over 12 million posts, comments, and reactions [38].

The discussion on social media grew into a movement that led to protest marches in the U.S. and around the globe, resulted in extensive and sustained media coverage of the issues of sexual violence and abuse of power, and shaped the public discourse in the months following October 2017. [39] estimate that in the first 8 months after the movement’s emergence, the number of google searches on sexual harassment and assault exceeded the expected amount by some 86%. According to the Women’s Media Center, the number of articles on sexual assault and harassment in a sample of 14 leading U.S. newspapers was more than double the pre-#MeToo average in November 2017 and still exceeded the pre-#MeToo average by 30% some 10 months after the onset of the movement [40]. [41] named the “Silence Breakers”—victims of sexual harassment or assault who came forward and thereby started the global dialogue on sexual violence—as its 2017 “Person of the Year”.

About 65% of social media users report having regularly encountered at least some content related to sexual harassment or assault on social media platforms in the months following the start of the #MeToo movement, with little difference across demographic groups [37]. The observation that a large share of society has been confronted with the issue of sexual violence is also reflected in Google search data. Following [39, 42], I look at how often the terms “sexual assault” and “sexual harassment” were searched for on Google in the U.S. during the study period. Fig 1 shows that public interest in these topics has never been higher than at the start of the #MeToo movement, with a similarly pointed peak of searches on Google News. Following [42], Fig 1 further shows that the #MeToo movement, rather than the Weinstein scandal, generated the rise in searches, as scandals involving men with similar or even higher profile than Harvey Weinstein did not attract nearly as much public attention as the #MeToo movement.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Google searches for “sexual assault” & “sexual harassment”.

Searches on Google and Google News in relation to the highest point in the period January 2015 to November 2020. The vertical lines indicate when the sexual harassment/assault allegations against stand-up comedian Bill Cosby, former U.S. President Donald Trump and Fox News CEO Roger Ailes became public.

https://doi.org/10.1371/journal.pone.0302827.g001

The #MeToo movement has led to changes in attitudes and responses to sexual violence, but also provoked some disillusionment and backlash. While several surveys indicate that both men and women have observed others to avoid sexually harassing behavior in response to #MeToo in public or private settings [43–46], others suggest that employers anticipate or observe increased caution among men in their interactions with female colleagues, potentially hindering women’s professional advancement (see, e.g., [47–51]). Furthermore, several articles argue that #MeToo (almost) exclusively benefited affluent white women, while women from lower socioeconomic backgrounds and women of color struggle to gain public attention and support (see, e.g., [52–54]), a criticism also supported by several studies (see e.g. [55, 56]). However, different studies [42, 57, 58] suggest that the under-representation of victims from certain demographic groups does not necessarily translate into a lesser impact of the movement on those groups.

The societal, cultural and political impact of the #MeToo movement has been analyzed extensively with mixed findings about the movement’s influence. There are studies showing that the #MeToo movement did not significantly increase self-reported interest in political participation [59], that the portrayal of male entrepreneurs in Swedish media changed only marginally after the onset of the movement [60], that the #MeToo movement caused a significant increase in the propensity of female dating platform users in South Korea to decline dating requests [61] and that it induced a change in gender norms in Swedish-language tweets [62]. A study by [63] reveals that support for the #MeToo movement is associated with a stronger belief in the sexual misconduct allegations against Donald Trump among Democrats but not among Republicans. [64] examined how the movement affected women’s perception of safety by comparing the development of men’s and women’s perceptions of safety in subway stations with a DiD approach, finding a significant decline in women’s perceptions of safety after #MeToo.

In the study most relevant to the present one, [42] assess the effect of the #MeToo movement on the propensity to report a sexual offense to the police. They apply a triple-difference approach over time, across 31 OECD countries, and between sexual and non-sexual offenses. For countries where the #MeToo movement attracted a great deal of attention, they find that it caused a significant 10% increase in the number of reported sexual offenses during the first six months after the movement began. A DiD analysis comparing the development of reported sexual and non-sexual crimes in the U.S. reveals a similar long-term effect for the 15 months following Alyssa Milano’s tweet. The study further shows that the movment did not affect the reporting of all types of sexual assault equally: reports of rape and fondling increased, while reports of statutory rape and sodomy remained unaffected. Finally, the study assesses the impact of the movement on sexual offense arrests, finding that the increase in reported sex offenses in the U.S did not bring about a similar surge in the number of arrests. Rather, the authors estimate that the movement raised the number of arrests by only about 6%, which they attribute to the fact that the movement had a stronger effect on cases with a low likelihood of arrest due to delayed reporting, less severe offenses, or insufficient evidence. The study does not provide any conclusions about the effect of the movement on convictions.

Victim blaming

Besides raising awareness about the prevalence of sexual harassment and assault in society, the #MeToo movement has also fueled discussions about the acceptance of rape myths. For this reason, this study assesses the judicial opinions specifically with respect to the reinforcement of rape myths and victim blaming.

The term “rape myths” refers to stereotypical and inaccurate beliefs about sexual assaults that are prevalent in society. These myths are frequently used to redirect blame from the perpetrator to the victim and to downplay the seriousness of the incident. Rape myths may not always be explicitly stated; they can also be reinforced through rhetorical techniques or by highlighting particular details when discussing sexual assault. For example, studies show that sexual violence reports that emphasize external circumstances and the victim’s behavior can increase the likelihood of readers to accept rape myths and view the victim as partly responsible for the assault (see, e.g., [65–67]).

Victim blaming and reinforcement of rape myths can be observed in a variety of contexts, albeit with varying degrees of explicitness. While often more bluntly expressed in social media and other informal contexts (see, e.g., [68, 69] for studies on identification of victim-blaming language in informal contexts), victim blaming and reinforcement of rape myths are also present in traditional media (see, e.g., [66, 70, 71]) and within the justice system. In court, there are general victim rights and sexual assault-specific regulations, such as rape shield laws, that to some extent prohibit explicit victim blaming, stigmatization, and stereotyping. Yet, several studies, based on testimonies from various actors, such as victims, barristers, sex crime investigators and independent observers, suggest that victim blaming and rape myths are still prevalent in courts (see, e.g., [72–75]). Although it is often the defense that introduces rape myths for strategic purposes, some judges do not always intervene and may even endorse such arguments [72, 76]. In addition, many judges tend to employ terminology of affection and consensual sex in cases where the perpetrator is familiar to the victim [76].

The U.S. justice system

The legal documents examined in this paper are judicial opinions from 51 U.S. state and federal appellate courts. Appellate courts review legal cases that have already been heard in a lower court (trial court) after a party appeals the trial court’s decision. In civil cases, both parties have the right to appeal the trial court’s decision, while in most states, only the defendant can appeal in criminal cases. To appeal a decision, the appealing party (the appellant) must file a brief, i.e., a written argument setting forth the facts and arguing why the trial court’s decision was erroneous, to which the other party (the appellee) must respond with an appellee’s brief.

The appellate court does not usually admit new evidence or witnesses; it may rule solely on the basis of the written briefs or after hearing oral arguments. The appellate court often issues what is called a judicial opinion, i.e., a written decision outlining the court’s reasoning; it is usually written by a single judge and reviewed by the other judges on the panel. If a judge disagrees, they can issue a dissenting opinion. Judges who disagree with the reasoning but agree with the result can issue a concurring opinion. Sometimes, judges issue an unsigned opinion called a per curiam opinion [77].

The body of judicial opinions

The body of judicial opinions examined in this paper includes opinions from 51 courts. Most judicial opinions of state and federal appellate courts have a similar structure. They usually begin with a summary of the trial court’s ruling, followed by the arguments of defense and prosecution that were presented and admitted before the trial court, as well as the appellate brief filed by the defense. They typically conclude with reasoning and the decision of the appellate judges. Thus, the opinions reflect the atmosphere during the trial court hearing and the appeal proceedings, as well as the attitude of the defense and the judges towards the victim.

To single out the opinions on sexual offenses and those on other crimes against persons from the full body of precedential opinions, I take advantage of the fact that the opinions have a similar structure and usually begin with an introduction specifying the offense(s) for which the offender was convicted in trial court. The opinions can therefore be reliably classified based on term search in the introduction. To distinguish opinions on sexual-violence related cases and those on other cases of interpersonal crimes, I rely on regular-expression-based (regex-based) search of the legal terms for such crimes in the introduction of the respective opinion (for more information on the legal terms used, see S1 Appendix).

As the headers of the opinions differ between courts, their divisions and even the judges who author the opinions, the introductions cannot be reliably identified based on regex rules. Therefore, the first third of each opinion but no more than 5,000 characters are defined as the introduction. The introductions of all opinions available at CourtListener are searched for matches to legal terms related to sexual violence; the remaining opinions are then searched for matches to other interpersonal-violence related legal terms. While certainly not adequate to accurately classifying opinions into crime categories, the identification procedure described above ensures that opinions are selected into the sample based on the same rules throughout the observation period. Focusing on the introduction of opinions prevents erroneously classifying opinions to one of the two groups when an opinion mentions crimes from a party’s past.

Using the regex procedure described above, I can identify 43,088 opinions on crimes against persons published between January 2015 and November 2020, including 15,307 on sexual offenses and 27,781 on non-sexual offenses against persons (performance assessment in S1 Appendix). The number of opinions per court is provided in S1 Table.

Victim-Blaming indicators

To quantify the extent of victim blaming in judicial opinions, three indicators are constructed based solely on sentences in which the victim or perpetrator is named. These indicators aim at capturing the extent to which opinions contain wording that implicitly shifts some blame from the perpetrator onto the victim, where such wording may come from the defense, the prosecution, or the judges involved in the case. The first indicator represents a novel approach based on findings from the field of psychology and will be the primary focus here. The remaining two indices assess the context in which the victim and perpetrator are discussed, where the first one measures the sentiment of the context, while the second one more comprehensively assesses the linguistic environment in which they are referenced.

In the appellate court opinions, the victim is often referred to by their name or by the term “victim”. The appealing party, i.e., the person found guilty in trial court, is usually called the “appellant” or by their name, but may also be referred to as the “petitioner”. As the victims’ and the appellants’ names are not clearly identifiable, only sentences containing the words “victim”, “appellant” and “petitioner” as well as inflected forms of these words are considered in the construction of the indicators (As the gender of victim and offender cannot be unmistakably identified, personal pronouns cannot be used to construct the indicators). If the names of victim and offender were identifiable in all texts, replacing the occurrences of the names with the corresponding roles of the individuals (victim/offender) and then applying the approaches below would certainly add substantial value to this study. Similarly, the approaches outlined below would be particularly interesting for text corpora where the names of the victim and offender are known.

Text vectorization

For analyzing the linguistic development in the judicial opinions, the opinions are vectorized by means of the Bag of Words (BoW) and the term frequency—inverse document frequency (tf-idf) algorithms, both of which are frequently used for text classification.

Both algorithms convert text documents into fixed-length vectors, where the vector length is determined by the number of unique words in the entire corpus of text documents. The vector elements in the BoW representation of a document correspond to the relative frequency of each word. For a tf-idf representation, these vector elements are multiplied by the logarithmically scaled inverse proportion of documents in the corpus that contain the respective word. This way, words that are characteristic of one or a set of document(s) are given a higher weight, while vector elements are comparably small if the corresponding word occurs in (almost) every document.

The BoW approach can be refined by excluding common words like “court”, “trial”, or “judge” that appear in almost all opinions and offer little insight into the evolution of court language. Additionally to the BoW representation of opinions based on the entire vocabulary, I also construct a second set of BoW vectors including only words that occur in less than 95% of all documents. This alternate BoW approach will in the following be referred to as reduced-corpus BoW.

Concerning tf-idf, applying this approach to the corpus of judicial opinions without additional text preprocessing would pose a problem: names of persons, places and organizations that only occur in one or a few opinion(s) receive disproportionately high weights, despite their lack of relevance to the document’s content. To circumvent this, I apply the Stanford PoS Tagger to identify and exclude all proper names before applying the tf-idf algorithm (see S1 Appendix for more information on the Stanford PoS Tagger).

Difference-in-Differences approach

I apply a DiD approach both with two and multiple time periods in order to identify the Average Treatment Effect on the Treated (ATET) (see, e.g., [82–84]). The sample consists of all judicial opinions on cases that can be classified as “crimes against persons”. Sexual offense opinions form the treatment group, i.e., the set of opinions that are affected by the #MeToo movement, opinions on other crimes against persons serve as control group. One advantage of choosing the sample of “crimes against persons” is that all crimes against persons involve a victim and a perpetrator, which is important for detecting victim-blaming language. Further, the comparison of sexual offenses and other crimes against persons is also common in the literature on victim blaming and rape-myth acceptance (see, e.g., [42, 85, 86]).

In order to identify the ATET of the #MeToo movement using a DiD approach, certain assumptions must hold. For simplicity, I will discuss these assumptions for the simple case of two time periods, but the discussion is easily transferable to the multiple time period approach.

Event study approach

I complement the DiD analysis with an event study approach, in which I assess the development of the text quantifiers before and after the #MeToo movement in a panel setting. In doing so, I attempt to address the problem that the set of opinions on non-sexual offenses may represent an imperfect control group. In the event study approach, I examine the judge-specific developments in the text quantifiers and victim blaming language indicators before and after the onset of the #MeToo movement. The purpose of this event study application is not to identify the causal effect of the movement, but rather to observe whether there was a shift in the overall development of language in sex offense cases and the use of victim-blaming language from before to after the movement.

I identify 1,382 judges, who, on average, publish roughly 11 sexual offense case opinions during the observation period, with the median number of opinions being 4, i.e., there are a few judges who authored a large amount of opinions (up to 184 opinions) while many others only published a handful during the study period (for more detail on the identification of the authoring judges, see S1 Appendix). For each judge in the sample, I calculate the six-month average of each text quantifier and victim blaming indictator to obtain a panel data structure with one or no observation per individual and time period. To avoid losing too many observations, I keep all judges who published at least one opinion before the onset of the #MeToo movement and at least one a year or more after the start of the movement. Then, I apply a Fixed Effects (FE) approach while weighting the observations by the number of opinions they were calculated from. Through weighting the observations, I account for the fact that the judges differ greatly in how many sex offense opinions they publish per six-month period, which makes some judges much more important for the development of language in court than others.

Although some information is lost by averaging the observations per judge and 6-month period, the panel approach might eliminate some of the noise typical of text analyses by increasing the amount of text per observation. On the other hand, however, it requires me to exclude several observations from the sample (3,843 opinions authored by 785 judges) because I do not have observations from either before or after #MeToo for these judges. By excluding judges who, for whatever reason, do not frequently publish precedential opinions on sex offenses, important information may be lost.

While the DiD approach captures language changes resulting from shifts in linguistic conduct of court actors as well as compositional shifts in the judiciary, this panel approach does not capture the latter. To some extent, comparing the estimates from these two approaches can help discern whether the observed evolution in language is primarily attributable to changes in the linguistic conduct of indiviudal judges or rather to compositional shifts in the judiciary. There is a growing literature on the drivers of attitudinal, cultural and literary change in society. Most studies conclude that such change is driven to a greater extent by generational turnover than by shifts in the behavior of individuals (see, e.g., [90–92]), and that intra-personal changes are more pronounced among younger people (see, e.g., [91]). The strong influence of cohort change on cultural shifts would suggest higher DiD than panel event study effect estimates.

Assessing effect heterogeneity

Finally, I also assess whether there is evidence of effect heterogeneity with respect to a judge’s gender or political affiliation, as well as with regard to the political orientation of the state in which a court is located. This is because different studies on victim blaming and rape myths acceptance show that females are less likely to accept rape myths and shift the blame for an assault upon the victim (e.g., [93–96]). [97] finds that this is particularly true for female Democrats.

Most studies in which participants were confronted with a sexual assault scenario find that men were more likely than women to blame the victim and show signs of rape myth acceptance, while other studies find no significant effect of gender on the likelihood of victim blaming (see [98–100] for reviews). Other research shows that study participants with politically conservative views are more likely to (partially) blame the victim for a sexual assault [101, 102]. For the judicial context, [97] finds that female Democratic judges are less likely to use rape myths than male judges (regardless of political affiliation), while her results show no significant difference between female Republican judges and male judges.

In addition to the differences in victim blaming and rape myth acceptance noted above, there is also evidence that the perception of and reaction to the #MeToo movement differ across genders and political camps. [59] find in a poll that Democrats are more likely than Republicans to say they were aware of the movement and mobilized by it. An analysis of the members of Congress’ communications on their public Facebook pages reveals that in the wake of the #MeToo movement, far more female than male members addressed the issue of sexual violence in their posts, with this pattern evident in both political parties [37]. In light of the findings by [91] that intra-personal changes are more pronounced among younger people, it might also be interesting to assess effect heterogeneity with regard to the age of judges.

For the scope of this analysis, I will however concentrate on whether the judges’ language in court is affected differently by the #MeToo movement depending on their gender and political affiliation. The research cited above suggests that female and/or politically liberal judges were more receptive to the #MeToo movement and more willing to change their behavior. Then again, these judges may have been more cautious in their choice of words prior to the movement and may have already avoided language that implied victim blaming, leaving them little room for change toward language that attributed less blame to the victim.

The analysis of treatment effect heterogeneity by judge characteristics is restricted to courts for which the names of the authoring judge is available on CourtListener. Information on the judges’ gender and political affiliation is obtained from ballotpedia, an online encyclopedia on American politics and elections. Ballotpedia only provides the political affiliation of some judges. To determine the political affiliation of the other judges, I draw on the party for which the judge ran in the judicial election or the political affiliation of the politician who appointed the judge, depending on the process used to select judges. For about 21% of the judges no political affiliation can be determined. To assess effect heterogeneity with respect to a state’s political orientation, I categorize those states as predominantly Democratic (Republican) that were won by the Democratic (Republican) party in at least three of the four 2008–20 presidential elections. Opinions from swing states and courts at the supra-state level are excluded for this analysis.

Victim blaming indicators

Fig 2 shows the development of the victim blaming semantics and sentiment indicators in opinions on sexual offenses and on other crimes against persons. Neither plot suggests any substantial differences between the development of these two indicators in the treatment and the control group. Further, the DiD estimates provided in Table 1 do not indicate a significant impact of the #MeToo movement on either of the two indicators. The DiD estimates for the semantics indicator suggest a slight, but not statistically significant, #MeToo-induced decline in the number of victim mentions as a grammatical subject and hence a decline in victim blaming. The estimates for the sentiment indicator are also not statistically significant and even indicate, contrary to the research hypothesis, a decrease in the use of words with negative connotations in the context of mentions of the perpetrator. Likewise, the event study approach (see Fig 3) shows no substantial changes in either of the two victim blaming indicators.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Development of victim blaming indicators in treatment & control group.

Kernel-smoothed plot of victim blaming indicators, with the solid line representing sexual offense opinions and the dotted line representing the control group. Curves were smoothed using the default settings of the sm.regression function in R. The semantics indicator measures mentions of the victim as a subject as a percentage of total mentions, and the sentiment indicator measures the negativity of the context words of offender mentions, ranging from 0 to 100, with the words being more negatively connotated the higher the score.

https://doi.org/10.1371/journal.pone.0302827.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Event study estimates for evolution of victim blaming indicators.

Event study estimates for the evolution of the victim blaming indicators in sexual offense opinions. The gray lines indicate the 90 percent confidence interval around the estimates.

https://doi.org/10.1371/journal.pone.0302827.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. DiD estimates of effect heterogeneity for victim blaming indicators.

https://doi.org/10.1371/journal.pone.0302827.t001

While assessing the impact of the #MeToo movement on the victim blaming indicators does not yield significant results, the plot on the development of the semantics estimator is nevertheless interesting. It shows that the use of the victim as grammatical subject is generally substantially higher in opinions on sexual assault cases than in cases on other crimes against persons. Given that several studies indicate higher prevalence of victim blaming in sexual assault cases than in cases on other crimes against persons (see Section) and that the indicator is constructed based on scientific findings (see Section), it seems reasonable to explore whether this indicator may be useful for measuring the extent of victim blaming in judicial opinions—as long as the purpose is to classify opinions or quantify the status quo rather than measure changes over time. The sentiment indicator, on the other hand, seems to vary little both over time and crime types. It generally takes very low values, which may be because judges deliberately avoid sentiment-charged language. It therefore does not seem appropriate for use in the context of court opinions and may be better suited for contexts with more colloquial language such as social media.

The results of the robustness checks for the DiD with victim blaming indicators are provided in S3 Table. Given that no significant effects on either indicator can be identified using the DiD approach, the fact that the placebo test does not indicate a violation of the parallel trend assumption is not particularly informative. For the sentiment indicator, both the IPW estimates as well as the DiD estimates based on sodomy and rape cases only indicate a significant decrease in negatively connoted words in the context of perpetrator mentions. In the case of the semantics indicator, both estimates have different signs and are not statistically significant.

The effect heterogeneity estimates obtained using the DiD and event study approach, respectively, can be found in S5 Table, S1 and S2 Figs. For the semantics indicator, the estimates suggest a larger decline in mentions of the victim as subject among female and Democratic judges as well as in predominately Democratic states, although again the differences are not statistically significant. The DiD estimates for the sentiment indicator imply a greater decline in the use of words with negative connotations among females and no difference in the effect of #MeToo on this indicator between Democrats and Republicans. The event study approach, on the other hand, suggests that Democrats have increased their use of negatively connoted words when compared to the development of this indicator among Republicans.

Finally, Fig 4 displays the development of the Word2Vec representation of the words “victim” and “appellant” in both the control and treatment group. Contrary to the research hypothesis, the context in which victim and offender are mentioned does not evolve faster for opinions on sexual offenses than for those on other crimes against persons. Again, this approach may be better suited to contexts with more flexible and rapidly evolving language.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Development of the word2vec representation of the word “victim” and “appellant”.

Development of the word2vec representation of the word “victim” (left) and “appellant” (right), with the solid line representing sexual offense opinions and the dotted line representing the control group.

https://doi.org/10.1371/journal.pone.0302827.g004

Text vectorization

Fig 5 illustrates the evolution of the BoW (the reduced sample BoW quantifier evolves similarly to the BoW quantifier in both groups, which is why it is not shown here). and the tf-idf text quantifiers in opinions on sexual offenses and on other crimes against persons. Both charts suggest that language in opinions on sexual offenses evolves more rapidly than that in opinions on other crimes against persons between the onset of the movement and 2019. However, the graphs also indicate that there may be problems with the parallel trend assumption. Moreover, the BoW graph shows a narrowing of the distance between the opinion vectors and their H1–2015 average in 2016 and 2017, as well as in 2020, which could be due to changes in the composition of courts, but also to other unobservable factors, which in turn would be critical for identifying the causal effect (other smoothing methods and smaller bandwidths yield similar curves. Thus, it does not seem to be an (over-)smoothing issue).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Distance of text vectors from their H1 2015 average over time.

Kernel-smoothed plot of the distance of text vectors from their H1 2015 average, with the solid line representing sexual offense opinions and the dotted line representing the control group. The distance of opinions to the H1 2015 average is expressed relative to the median distance of all opinions to the H1 2015 average, in percent.

https://doi.org/10.1371/journal.pone.0302827.g005

The DiD estimates for the text vectorization-based opinion quantifiers can be found in Table 2. The results point at a slight #MeToo-induced change in courtroom language, which however materializes with a substantial time lag. The estimates suggest that the language in sexual offense opinions deviates more quickly from the 2015 average than in opinions on other cases of crimes against persons. The DiD estimate for the second half of 2020 is statistically insignificant for all three text quantifiers and numerically small for the two BoW quantifiers, indicating that the language change in sexual offense opinions is not likely due to the COVID-19 crisis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. DiD estimates for text vectorization-based opinion quantifier.

https://doi.org/10.1371/journal.pone.0302827.t002

The event study estimates in Fig 6 show a decrease in the distance to the H1–2015 average in 2016 that is similar to, though less pronounced than, that observed in Fig 5. Further, the event study estimates do not indicate a stronger deviation of language from the H1–2015 average in the years after #MeToo than in the years before #MeToo, suggesting that the effect estimated with the DiD approach may be attributable to changes in case composition or personnel rather than changes in judges’ attitudes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Event study estimates for evolution of text vectorization-based opinion quantifiers.

Event study estimates for the evolution of the text vectorization-based opinion quantifiers. The gray lines indicate the 90 percent confidence interval around the estimates. The distance of opinions to the H1 2015 average is expressed relative to the median distance of all opinions to the H1 2015 average, in percent.

https://doi.org/10.1371/journal.pone.0302827.g006

The results of the robustness checks are presented in S4 Table. While the placebo tests do not reveal a significant violation of the common trend assumption, they do not allow me to rule out such a violation, especially because the estimated effects of the placebo treatment on the (reduced sample) BoW quantifiers are positive, just like the observed effect in the main DiD analysis. I therefore also estimate the effect of placebo treatments at other points in time during the pre-#MeToo period, all of which turned out to be statistically insignificant, with some of them having a positive and others a negative sign. The fact that the IPW-based DiD and the reduced sample DiD estimate positive effects for all quantifiers, some of which are statistically significant, support the finding that the #MeToo movement has caused a slight increase in language development in sexual assault opinions, with the IPW results ruling out that the observed effect in the main analysis is due to changes in the composition of courts, while the latter rules out that it is due to an increase of reports of sexual offenses. When considering these results in conjunction with the event study results, one possible explanation for the observed slight linguistic change in sexual offense opinions is the change in judicial appointments toward judges with more progressive views on sex offenses. However, given the numerically small effect estimates, the noise in these quantifiers, and the fact that the effect does not appear until two years after the movement began, it is difficult to attribute the observed effect to the #MeToo movement.

A look at the effect heterogeneity estimates in S6 Table, S1 and S2 Figs does not give a clear picture. While the DiD estimates for the (reduced sample) BoW suggest a faster evolution of language in opinions written by females and Democrats, the event study plots point to a similar evolution of language in opinions written by female and male or Democratic and Republican judges, respectively. For the tf-idf approach, the DiD estimates suggest that the language of female and Democratic judges evolves less rapidly than that of their counterparts.