Introduction

Just as Britain and America have been described as two nations separated by a common language, different academic disciplines often use the same words with divergent meanings [1]. Interdisciplinary research thus poses unique challenges for information retrieval (IR). Word sense disambiguation [2, 3], differing publication practices across disciplines [4–6] and disjoint authorship networks [7] pose special challenges to information retrieval for interdisciplinary work. When the dimension of time is added, terminological shifts [8, 9], changing citation standards [10–13], and shifting modes of scholarly communication [4, 5, 14, 15] all amplify the challenges for IR to serve the need of interdisciplinary scholars.

Widespread digitization of monographs and journals by HathiTrust [16, 17] and Google Books [18, 19] enable new longitudinal studies of change in language and discourse [8, 9, 12, 20–22], an approach known as “distant reading” [23]. These data-driven distant readings contrast with “close readings”, in which short passages and particular details are emphasized for scholarly interpretation. Newly digitized materials, which enable distant reading, differ from born-digital scholarly editions in three key ways: First, the reliance on optical character recognition (OCR) over scanned page images introduces noise into the plain-text representations of the text. Second, the unstructured text does not contain any markup that may differentiate page header and footer information, section headings, or bibliographic information from the main text. Finally, metadata is often automatically extracted and lacks the provenance information important to many humanities scholars. Researchers seeking to marry these “distant readings” to more traditional “close readings” are impacted by these factors [24].

Our goal is to develop computational methods for scholarly analysis of large-scale digital collections that are robust across both the technological inconsistency of the digitized materials and the variations of meaning and practice among fields and across time. A further goal of our approach is that these methods should inform interdisciplinary research by suggesting novel interpretations and hypotheses. The methods should support scholars who wish to drill down from high level overviews of the available materials to specific pages and sentences that are relevant for understanding the various responses of scholars and scientists to contentious issues within their fields.

In this paper, we provide a case study that focuses on meeting these challenges within the interdisciplinary field of History and Philosophy of Science (HPS). HPS must not only bridge the humanities and the sciences, but also the temporal divide between historically-significant materials and the present [25–28]. We show how faceted search using a combination of traditional classification systems and mixed-membership models can go beyond keyword search to inform resource discovery, hypothesis formulation, and argument extraction in our test domain, delivering methods that can be generalized to other domains.

Using a novel approach of drill-down topic modeling—simultaneously reducing both the size of the corpus and the unit of analysis—we demonstrate how a set of 1,315 fulltext volumes obtained by a keyword search from the HathiTrust digital library is progressively reduced to six focal volumes that did not appear in the top ten results in the initial HathiTrust search. Topic modeling of these volumes at various levels, from whole book down to individual sentences, provides the contexts for word-sense disambiguation, is relatively robust in the face of OCR errors, and ultimately supports a system for semi-automatic identification of argument structure. We show how visualizations designed for macroanalysis of disciplinary scientific journals can be extended to highlight interdisciplinarity in arguments from book data [29]. This guides researchers to passages important for the kind of “close reading” that lies at the heart of scholarly work in the humanities, supporting and augmenting the interpretative work that helps us understand the intellectual and societal contexts in which scientific writings are produced and received.

While the extension of computational methods such as these to various questions in the humanities may eventually provide ways to test specific hypotheses, the main focus of such research is likely to remain exploratory and interpretative, in keeping with the humanities themselves [24, 30]. This approach nevertheless shares something with the sciences: it is experimental to the extent that it opens up a space of investigation within which quantitatively defined parameters can be systematically varied and results compared. Such exploratory experimentation is common not just in the social sciences, but also in the natural sciences [31, 32].

Our study consisted of six stages. (1) We used a keyword search of the HathiTrust collection to generate an initial corpus and we used probabilistic topic models on these volumes. (2) We exploited the mixed-membership property of the topic models to identify the multiple contexts of the selected volumes and reduce the original search space even further. (3) Because topic models define the notion of a document flexibly, we drilled down further by constructing page-level topic models of the reduced set of volumes selected at the previous stage. (4) We used the page-level results to rank books and select pages from them for closer analysis, demonstrating an approach to semi-automatic argument extraction which showcases the interpretive results of our search process. (5) We exploited the close reading of arguments for exploratory investigation of drilling down even further, to sentence-level topic modeling within a single volume. (6) We used scientific mapping to locate relevant volumes [33]. Because current science maps represent journal data, and data overlays are created based on journal names, we needed to construct a classification crosswalk from the UCSD Map of Science to the Library of Congress Classifications of these journals, finally allowing us to project books onto the science map.

We assessed success in our case study in three ways: (1) by the effectiveness of the process in leading non-experts to drill down to highly-relevant content in a very large collection of books; (2) by the ability of this process to spotlight a somewhat forgotten woman scientist who is important to the history of psychology; (3) by the capacity of the process to lead domain experts to a surprising discovery about the breadth of species discussed in these historical materials, thus enriching the historical context for current discussions of intelligence in microscopic organisms [34, 35]. Our assessments are qualitative rather than quantitative in nature, but they are appropriate given current limitations in quantitative assessments of the quality of topic models [36–38].

Materials

Methods

Methods overview

We followed a six-stage process, summarized in Fig 1. Each step is described in more detail further below. We introduce them briefly here:

1. LDA Topic modeling of a subset of volumes from the HathiTrust Digital Library selected by a keyword search, treating each volume as the unit document for the LDA process.
2. Querying the model to further reduce the original set of documents to a more relevant subset for our HPS objectives.
3. Drill-down LDA topic modeling on the smaller set treating individual pages as the unit documents, using this page-level model to select pages for further analysis.
4. Mapping of arguments on the selected pages by manual analysis, supported by the enhanced Online Visualisation of Arguments (OVA+) tool [58].
5. LDA topic modeling of single books, treating each sentence as document unit.
6. Mapping identified volumes onto UCSD Map of Science via a crosswalk from Library of Congress classification data to the journals used to construct the basemap.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Corpus analysis sequence.

Schematic rendering of the six-step process that sequentially drills down from macroscopic “distant reading” to microscopic “close reading” before zooming back out to the macroscopic scale at the final step. The approximate orders of magnitude of the datasets either side of each processing step are shown below the icons as powers of 10 of book/fulltext-sized units, and grey bars representing the data are scaled logarithmically.

https://doi.org/10.1371/journal.pone.0184188.g001

Results: A case study

In this section we describe the application of these methods to a case study in the History & Philosophy of Science (HPS), specifically in the history of comparative psychology. When we began the study the HathiTrust digital library provided access to the full texts of just over 300,000 public domain works. The keyword-based search for items of interest reduced this set to a corpus of 1,315 volumes published between 1800 and 1962, which we designate as our HT1315 corpus. (Publication dates after 1928 correspond to items in the public domain, such as government reports and university course bulletins.) A list of titles and HT handles is provided in the supplemental materials. Because the HT collection has changed over time, this exact set of results cannot be recreated by doing the same keyword search at hathitrust.org (see http://bit.ly/1LBbqnS). Currently there are over 5.5 million public domain works in the collection (see https://www.hathitrust.org/visualizations_dates_pd). The same query conducted in August 2015 yielded 3,027 full-text results.

Table 1 shows the top topics when the k = 60 topic model is queried using the single word ‘anthropomorphism’. The topic model checking problem [37]—i.e., how to assess the quality of the model’s topics—remains an important open problem in topic modeling. Nevertheless, most of the topics in the model can be quickly summarized. Inspection of this list indicates that ‘anthropomorphism’ relates most strongly to a theological topic (38), a biological topic (16), a philosophical topic (51), an anthropological topic (58), and a child development topic (12). The topic model thus serves to disambiguate the different senses of ‘anthropomorphism’, especially between contexts where the discussion is about anthropomorphized deities (38) and contexts where it is about nonhuman animals (16), with the second topic being the most obvious attractor for researchers interested in comparative psychology. The second-to-last topic (1) is targeted on bibliographic citations, and is dominated by bibliographic abbreviations and some common German and French words that were not in the English language stop list used during initial corpus preparation. Although from one perspective this may seem like a ‘junk’ topic, this topic is nonetheless very useful to a scholar seeking citations buried in the unstructured pages in the corpus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Topics ranked by similarity to ‘anthropomorphism’ in the HT1315 corpus.

Topic 16 (highlighted with bold text) is highly relevant to the inquiry.

https://doi.org/10.1371/journal.pone.0184188.t001

Table 2 shows the top topics returned by querying the k = 60 model of HT1315 using ‘anthropomorphism’, ‘animal’, and ‘psychology’ to construct the query. This new query reveals two relevant topics (numbers 26 and 10) that were not returned using ‘anthropomorphism’ alone. The top ten documents found by querying the model using these two topics in combination with the previously noted topic 16 is shown in Table 3. By selecting from the continuation of this list up to a threshold of 1.25 on the aggregated distance measure, we reduced the number of volumes of interest from 1,315 to 86, constituting the HT86 corpus.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Topics ranked by similarity to ‘anthropomorphism’, ‘animal’, and ‘psychology’ in the HT1315 corpus.

Topics 26, 16, and 10 (highlighted with bold text) were used to derive the HT86 corpus, as they were most relevant to the inquiry.

https://doi.org/10.1371/journal.pone.0184188.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Book titles ranked by proximity of the full texts to topics 10, 16, and 26 in the k = 60 model of the HT1315 corpus.

https://doi.org/10.1371/journal.pone.0184188.t003

The result of querying the k = 60 page-level model of the HT86 corpus with the single query word ‘anthropomorphism’ is shown in Table 4. (Topic numbers are arbitrary and do not correlate across the HT86 and HT1315 models.) Although a theological topic (18) is again at the top of the list, it is clear that biological and psychological topics have become more prevalent in the HT86 model. Even within Topic 18, ‘evolution’ and ‘science’ are now among the ten highest probability words indicating that the topic is closer to a “religion and science” topic than the more general religion Topic 38 from the HT1315 model (Table 1), and reflecting the narrower range of books in the HT86 subset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Topics ranked by similarity to ‘anthropomorphism’ in the HT86 corpus, as modeled at the page level.

https://doi.org/10.1371/journal.pone.0184188.t004

Using ‘anthropomorphism’, ‘animal’ and ‘psychology’ in combination to query the k = 60 HT86 model, topic 1 is the highest ranked topic (Table 5). In comparison to the earlier topics 10 and 16 from the HT1315 results in Table 2, this topic has more terms relevant to psychology (i.e., stimulus, experience, instinct, reaction), suggesting that for the purposes of locating specific pages in HT86 collection that are relevant to our HPS interests, topic 1 provides the best starting point. Table 6 shows the first rows of a list of 800 highest ranked pages from HT86 using topic 1 as the query.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Topics ranked by similarity to ‘anthropomorphism’, ‘animal’, and ‘psychology’ in the HT86 corpus.

https://doi.org/10.1371/journal.pone.0184188.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Pages ranked by similarity to Topic 1.

https://doi.org/10.1371/journal.pone.0184188.t006

None of the six volumes from HT86 collection which had the most pages in the top 800 highest-ranked pages had appeared among the top 10 keyword search results in the original Solr search in the HathiTrust collection. These volumes formed the HT6 collection:

1. The Animal Mind: A Textbook of Comparative Psychology, 1908 (first edition), by Margaret Floy Washburn, psychologist. Washburn’s textbook was foundational for comparative psychology and she is notable as the second woman to be president of the American Psychological Association.
2. Comparative studies in the psychology of ants and of higher animals, 1905, a monograph by Erich Wasmann, an entomologist who only partly accepted evolution within species, rejecting common descent, speciation via natural selection, and human evolution.
3. The Principles of Heredity, 1906, a scientific monograph by G. Archdall Reid, a physician who argued against the Lamarckian idea of inheritance of acquired characteristics.
4. General Biology, 1910, a text book by James G. Needham, entomologist and limnologist.
5. The Nature and Development of Animal Intelligence, 1888, a compilation of articles by Wesley Mills, physiologist, physician and veterinarian.
6. Progress of Science in the Century, 1908, a book on the history of science for general readers by J. Arthur Thomson, naturalist.

These books were written by two Americans (Washburn and Needham), two Scots (Reid and Thomson), a Canadian (Mills), and an Austrian (Wasmann). They provide a broad array of perspectives on animal intelligence and psychology, from specialist monographs to textbooks to general-audience nonfiction, spanning both pro-Darwinian and anti-Darwinian viewpoints.

Using the k = 60 model of the HT6 collection to identify sections of each book with highest proportion of Topic 1, we selected 108 pages from the six HT6 volumes for further analysis (Table 7). From these we generated 43 argument maps using AIF annotated documents, providing a visual representation of the structure of each argument (e.g., Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Pages for which OVA+ argument maps were created, showing total number of pages analyzed and numbers of arguments identified on each of the passes described in the main text.

https://doi.org/10.1371/journal.pone.0184188.t007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. An argument map derived from The Animal Mind, represented in OVA+.

https://doi.org/10.1371/journal.pone.0184188.g002

We performed two types of argument analysis: Pass 1 aimed to summarize the arguments presented in each volume. Pass A aimed to sequence the arguments presented in each volume. All argument maps can be found at http://bit.ly/1bwJwF9. A full description of the study, including analysis of the arguments can be found in [64], and is summarized in [66].

As a proof of concept, these arguments show the utility of new techniques for faceted search enabling access from a library of over 300,000 books to volume-level analysis of a subset of 1,315 books all the way down to page-level analyses of 108 pages for the purpose of identifying, encoding, modeling, and visualizing arguments. These argument diagrams function as a type of close reading, common in the humanities, where this approach is related to a range of work in human-computer argument modeling [62, 67] and argument mapping Kirschner2012. This approach also draws on a rich tradition of philosophical literature (reviewed in [68]).

The query sentence (Q) that we chose from Argument 15 of the Washburn book is shown below with the first half dozen results (and their similarity scores). It is important to remember that LDA topic modeling is a “bag of words” approach; i.e., it uses only an unordered list of words in each document. It has no information about word order, punctuation, or other formatting in the text, and some of the most common words are not included. The full sentences are shown here only to aid the reader.

1. Q: Every statement that another being possesses psychic qualities is a conclusion from analogy, not a certainty; it is a matter of faith. (1.0000)

1. If any consciousness accompanies it, then the nearest human analogy to such consciousness is to be found in organic sensations, and these, as has just been said, must necessarily be in the human mind wholly different in quality from anything to be found in an animal whose structure is as simple as the Amoeba’s. (0.8413)
2. Fancy, for example, one of us entering a room in the dark and groping about among the furniture. (0.8239)
3. This, of course, does not refer to the power to judge distance. (0.8235)
4. Again, a bodily structure entirely unlike our own must create a background of organic sensation which renders the whole mental life of an animal foreign and unfamiliar to us. (0.8224)
5. She disposes of the psychic learning by experience theory of Nagel by saying that the only experience upon which the animal could reject the filter paper must be experience that it is not good for food. (0.8198)
6. We speak, for example, of an “angry” wasp (0.7924)

Sentence 1 is obviously related in meaning to the query sentence: the sentences overlap in some words, and directly express related ideas. But the relevance of the other examples is less direct. Sentence 6 provides a nice illustration of anthropomorphic attribution with no word overlap whatsoever. The inclusion of sentences 2 and 3 is, more puzzling. However, in the context of where these sentences appear in Washburn’s book, the relationship become plainer. Sentence 2 comes in the context of the discussion of what it might be like to be an amoeba. It is thus related to sentence 1, and it is used by Washburn to make the point that our experience in the dark, which still involves visual imagination and memories of what we touch, must be “wholly different in quality” (per sentence 1) from what an amoeba might experience. Sentence 3 occurs in a footnote on page 238, and it is worth quoting the footnote in full:

Porter observed that the distance at which spiders of the genera Argiope and Epeira could apparently see objects was increased six or eight times if the spider was previously disturbed by shaking her web (612). This, of course, does not refer to the power to judge distance.

[Italics in original.]

Here, then, we see the author cautioning the reader not to jump to a high-level interpretation of the spider behavior. The spiders may perceive objects at various distances but they don’t judge it. The term ‘judge’ here is philosophically interesting, as it suggests an influence of Immanuel Kant on framing the debate. While Kant’s name does not appear in Washburn’s book, the term ‘judgment’ is important to Kant’s theory of cognition, and fundamental to the cognitive divide he posits between humans and animals. We emphasize that this is just a speculative suggestion about Washburn’s influences, but it does show how the topic modeling process can bring certain interpretive possibilities to the fore, moving the digital humanities another step closer to the goal of generating new insight into human intellectual activity.

Finally, we found Library of Congress classification records for 776 out of 1,315 books and we used the LCCO classification crosswalk that we constructed to locate these books on the UCSD Map of Science, as shown in Fig 3. (In the interactive online version, nodes can be selected, showing which volumes are mapped and providing the title and links to various external sources of metadata.) The map confirms that the initial keyword-based selection from the HathiTrust retrieved books that are generally positioned below the “equator” of the map, with particular concentrations in the life sciences and humanities, as was to be expected. The map provides additional visual confirmation that the further selections via topic modeling to the HT86 and HT6 corpora managed to target books in appropriate areas of interest.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. UCSD map of science with overlay of HathiTrust search results.

This image shows topical coverage of humanities and life science data. The basemap of science shows each sub-discipline denoted by a circle colored or shaded according to the 13 core disciplines. Links indicate journal co-citations from the basemap. The 776 volumes of HT1315 with LCCN metadata are shown on the map as circles. Volumes also in HT86 are shown with thicker circles, and those in HT6 are shown in the thickest circles. An online, interactive version can be explored at http://inpho.cogs.indiana.edu/scimap/scits.

https://doi.org/10.1371/journal.pone.0184188.g003

Ultimately, the map overlay provides a grand overview and a potential guide to specific books that were topic modeled, although without further guidance from the topic models, the map does not fully meet the desired objective of linking a high-level overview to more detailed textual analysis.

General discussion

The notion of “distant reading” [23] has captured the imagination of many in the digital humanities. But the proper interpretation of large-scale quantitative models itself depends on having a feel for the texts, similar to Barbara McClintock’s stress on having a “feeling for the organism” [69] or Richard Feynman on the importance for nascent physicists of developing “a ‘feel’ for the subject” beyond rote knowledge of the basic laws [70]. The interpretation of data and models, whether in science or the humanities, is itself (as yet, and despite a few small successes in fields such as medical diagnosis) a task at which humans vastly outperform machines. For this reason, the digital humanities remain a fundamentally hermeneutic enterprise [30], and one in which distant readings and close readings must be tightly linked if anything is to make sense.

In this paper we have motivated, introduced, and exemplified a multi-level computational process for connecting macro-analyses of massive amounts of documents to micro-level close reading and careful interpretation of specific passages within those documents. Thus we have demonstrated how existing computational methods can be combined in novel ways to go from a high-level representation of many documents to the discovery and analysis of specific arguments contained within documents.

We have also shown how to zoom out to a macro-level overview of the search results. We presented a novel classification crosswalk between the Library of Congress Classification Outline (LCCO) and the UCSD Map of Science, which was constructed using only journal data, to extend the data to books. Because of the mismatch between the book data and the journal metadata, the crosswalk is not perfect, and the method of averaging locations places many books in uninterpretable regions of the map. Nevertheless, the visualization provides some useful information about the effectiveness of a simple keyword search in locating items of interest within a collection of hundreds of thousands of books.

That our method succeeded in discovering texts relevant to a highly specific interdisciplinary inquiry shows its robustness to inconsistent and incomplete data. The HathiTrust Digital Library had OCR errors in 2.4% of volumes as of May 2010 [71]. While the quality of the HathiTrust has increased in the intervening years, it is still a pervasive issue in digital archives [72].

Multi-level topic modeling combined with appropriate measures of distance can efficiently locate materials that are germane to a specific research project, going from more than a thousand books, to fewer than a hundred using book-level topic models, and further narrowing this set down to a small number of pages within a handful of books using page-level topic models. The similarity measures used to span the word-topic and topic-document distributions are mediated by the topics in the model, and because every topic assigns a probability to every word in the corpus, this approach is highly adept at finding implicit relationships among the documents. Typical applications of topic modeling used elsewhere, such as graphing the rise and fall of topics through time, may show large-scale trends, but do not directly facilitate the interplay between distant reading and close reading that leads to deeper understanding. By connecting abstract, machine-discovered topics to specific arguments within the text, we have shown how topic modeling can bridge this gap.

Could the similar outcomes have been accomplished with alternative methods such as Latent Semantic Analysis (LSA; [73]) or other Natural Language Processing (NLP) methods? We have investigated LSA as a discovery tool in the HT1315 and HT86 corpora. For users conditioned by online search engines, the word-centric search paradigm of LSA is more familiar than the topic-centric retrieval methods introduced here. Nonetheless, once users become aware of the potential for topic models, they provide more information about retrieved documents. For example, Washburn’s The Animal Mind is significant for its mixture of topics. (The topic models of the HT1315 collection can be explored using Washburn’s book as an entry point to the InPhO Topic Explorer page here: http://bit.ly/2qQZCJm.) Indeed the most prominent topic in the book (Topic 42: light, eye, two, eyes, visual, fig, vision, red, distance, movement) points to the importance of perceptual systems in her discussion of anthropomorphic attributions, and it is noteworthy that her dismissal of amoeba consciousness is grounded in what she argues is the lack of crossmodal associations (between, e.g., touch and vision) that are part of the content of human perceptual experiences—viz. the sentence previously discussed concerning why our human experience in the dark must be “wholly different in quality” from what an amoeba might experience.

Conclusions

Using a six-step process we progressively reduced 300,000 public domain volumes from the HathiTrust Digital Library to the 1,315 books in the HT1315 corpus, to the roughly 32,000 pages in the HT86 collection, to the over 17,000 sentences of the HT6 collection, to smaller set of the 108 pages selected for close reading and argument markup. This reduction allowed us to filter out discussions of anthropomorphic deities and zero in on key elements of late 19th and early 20th Century arguments about anthropomorphizing nonhuman organisms.

The application to the history of comparative psychology described in our case study was successful in at least three ways. First, team members unfamiliar with the domain of comparative psychology were effectively guided towards highly relevant material in an unreadably large set of books. Second, team members were introduced to the work of Margaret Washburn, a pioneer of scientific comparative psychology, who wrote an important textbook of the early 20th Century [65]—a book that went through four editions in as many decades, but has been largely forgotten since then (although see [74] for a recent tribute). Third, close reading of the arguments in the HT6 corpus revealed the surprising taxonomic range of these arguments, to include consideration even of consciousness in amoebae. This was surprising even to the domain experts on the team. This discovery raises new questions about the context for contemporary discussions of slime mold intelligence [34, 35] and opens up new avenues for research and analysis. By putting words into context, topic modeling enabled researchers to zero in on passages worthy of detailed analysis and further humanistic interpretation.

It took Charles Darwin 23 years to read a number of books comparable in size to HT1315, as documented in his Reading Notebooks. At the unlikely rate of one book a day, it would take nearly four years to read this set of books in its entirety. Even allowing for the fact that one fifth of the volumes retrieved in our were course catalogs, eliminating those would nonetheless leave a daunting, if not quite Olympian, reading task. As the majority of the volumes selected by keyword search were not directly relevant to the research project, the payoff made possible by more sophisticated computational analysis of the full texts was critical for the present task of finding arguments about anthropomorphizing animals.

Although we do not claim that our methods provide the only possible approach, we are unaware of any other approach providing the kind of multilevel macro-to-micro approach needed to assist digital humanities researchers who wish to leverage computational methods against large data collections to support both distant reading and more traditional close reading analyses in the humanities. Nevertheless, these methods suffer from the problem already noted by Blei [37] and others that objective methods for evaluating the quality of topics are lacking. We regard this as a function of the complexity of human semantic understanding—the holy grail of strong Artificial Intelligence. And while studies such as ours raise more questions than they answer, they point the way to even deeper understanding of large text corpora and more useful tools to scholars.

Availability of data and materials

Acknowledgments

This work was funded by the National Endowment for Humanities (NEH) Office of Digital Humanities (ODH) Digging Into Data Challenge (“Digging by Debating”; PIs Allen, Börner, Ravenscroft, Reed, and Bourget; award no. HJ-50092-12). The authors thank the Indiana University Cognitive Science Program for continued supplemental research funding, and especially for research fellowships for Jaimie Murdock and Robert Rose. We also thank the HathiTrust Research Center (HTRC) for their support of research activities and generous access to materials.