I would like to thank the authors for once again replying. I’m afraid that we are no further along in our understanding of the problems with the study. In their reply to my most recent comment, Schoonvelde et al. make a note about a “speechwriter problem” which they discuss elsewhere. But I didn’t raise the problem of speechwriter bias. Instead, I asked whether the speeches were performed exactly as they are written in the corpus. If they were not, then the study by Schoonvelde et al. does not research politicians’ “speech complexity” (as is claimed in the article), but rather how political speeches are written. The Schoonvelde et al. study then makes overarching claims about how liberal and conservative politicians speak. But as Milroy & Milroy (1991: 60) point out, there are “very considerable differences that exist between modes of sentence construction in speech and writing” and that “devices available to speakers for organizing their linguistic presentation are quite different from those available to writers” (p. 142; italics theirs). In that way Schoonvelde et al. confuse “communication” and “speech”. This confusion has a bearing on the authors’ whole dataset too. Crystal (2003: 181) says that “[the differences in speech and writing] are much greater than people usually think. The contrast is greatest when written texts are compared with informal conversation; but even in fairly formal and prepared speech setting, such as a teacher addressing a class, the structure of the language that is spoken bears very little similarity to that found in writing.”

Schoonvelde et al. claim to be studying political speeches and politicians’ speech, but they are really studying one restricted form of communication: the punctuation habits of political speechwriters. The authors claim in their reply to my comment that “the written speeches may be different from the actual speeches.” I will have to assume that by “actual speeches,” they mean spontaneous speeches. But their claim that the written (i.e. pre-planned) speeches and spontaneous speeches “may be different” shows that the authors do not understand the differences between speech and writing. It’s not that the written and spontaneous speeches may be different: they are different. Milroy & Milroy are again useful here to understand this difference. They note (1991: 64) that “although spoken language is diverse in its forms and functions, the norms of written grammar, spelling and vocabulary are much more uniform” and they emphasize that “the forms and functions of spoken language are very largely different from those of writing.” Crystal (2003: 181) agrees with this notion and says “the differences of structure and use between spoken and written language are inevitable, because they are the product of radically different kinds of communicative situation.” This uniformity goes beyond just spelling and vocabulary and into the level of sentence-construction (Milroy & Milroy 1991: 67). What this means is that written and spoken forms of English are fundamentally different, both in how “words” are represented and how “sentences” are (or can be) formed.

In the paragraph marked “Punctuation”, Schoonvelde et al. claim to disagree with me that “there is no place in science for beliefs”. I would encourage the authors to reread my comment so that they can notice it does not include anything about the place in science for beliefs. However, they are correct that hypotheses and assumptions are kinds of scientific beliefs. Hypotheses and assumptions, however, come before an analysis. When hypotheses are confirmed by evidence, they are called facts. It is therefore good that Schoonvelde et al. raised the idea of hypotheses in science. Their claim that they “think it is highly unlikely that transcribers’ punctuation policies differ between liberal and conservative speakers” is a hypothesis, not a fact that is backed up by research or evidence.

To answer the larger point made in the authors’ paragraph marked “Punctuation”, however, it is important to again state that the authors have misunderstood my criticism. I did not point to “punctuation errors”, which of course will be washed out when there is a large enough amount of data. I did, however, point to the fact that transcribers can place periods or commas in front of conjunctions and still follow the norms of standard English. But their choice of which punctuation mark to use has the potential to greatly affect the results of the Schoonvelde et al. study because the measurement they use (the Flesch-Kincaid test) so crucially depends on punctuation. In addition to this, we know from linguistic research that “formal (and planned) speech events are in fact characterized by an absence of reliance upon immediate context for their interpretation, and by conjunctions such as because, therefore, since which express explicitly temporal and causal relationships between clauses” (Milroy & Milroy 1991: 147). In addition, Crystal (2003: 181) says that “writing avoids words where the meaning relies on the situation (deictic expressions, such as this one, over there).” This means that 2/3 of the dataset in Schoonvelde et al. (the part of the data which the authors claim is made up of pre-written speeches) will include more opportunities for periods to be placed in front of conjunctions (as well as fewer opportunities to use one-syllable deictic words in English). In order to be sure that the conservative or liberal political speechwriters do not affect the results of the study by using periods instead of commas (or no punctuation marks) before conjunctions (and vice versa), the authors should have probably looked at how punctuation was used before conjunctions. Note that I am talking here about the pre-written speeches in the data, i.e. the data that was not written by transcribers. In their response to my criticism above, the authors have mistaken which part of their dataset is more likely to contain this problem, as they refer to how they “think it is highly unlikely that transcribers’ punctuation policies differ between liberal and conservative speakers.” The 1/3 of the dataset that is made up of transcribed speeches will also have this problem, but probably to a lesser extent than the prewritten speeches because of the fundamental differences between spontaneous spoken language and pre-planned written language (Milroy & Milroy 1991: 142; Crystal 2003: 181; Rowe & Devine 2006: 293).

In their most recent reply to me, Schoonvelde et al. claim that they “are more interested in the words that politicians choose, and less so in how these words are received”. This comment shows another lack of understanding of the data. Schoonvelde et al. are clearly studying how the words are received because at least 1/3 of the data (according to the authors) was transcribed by someone other than the speaker, who was the person that “chose” the words. While we cannot talk about spoken language having sentences, the transcribers of this section of the data have clearly made choices about where to place the punctuation. In other words, their reception of where each “sentence” ends has motivated their placement of punctuation and their placement of punctuation has a crucial bearing on the data. So again the question is about the transcribers’ actions: did they place periods in between clauses at a higher rate for conservative or liberal politicians? The speakers did not use any punctuation, so the essential aspect of the dataset falls on the transcribers. This means that what Schoonvelde et al. are studying (with this subsection of their dataset) is not whether conservative or liberal politicians use complex language, nor really which words the politicians choose, but whether the transcribers make the language seem more complex or simple by their use of punctuation. As Schiffrin (1994: 25) notes, intonation is what breaks speech up into chunks, but the breaks in intonation do not fit with punctuation and they do not “always correspond to syntactic boundaries,” which are marked by punctuation in writing. Instead, the use of punctuation in transcriptions of speech “forces us to think of such chunks as sentences, rather than as providing an accurate representation of how speakers themselves produce language” (Schiffrin 1994: 25). And this is why the F-K test is not sufficient for studying complexity in language. The liberal politicians may use more complex language, but the F-K test can’t tell us that.

The authors do not address my criticism of their use of a test for written language on spoken language. As I pointed out in my previous comment, Carter et al. (2001: 243) says that “spoken discourse should not be judged using the rules of written English: terms such as ‘word’, ‘sentence’ and ‘paragraph’ above, all come from the study of writing. The written form is not an appropriate medium for oral language”. Crystal (2003: 181) illustrates how writing and speech differ by saying “units of discourse [in writing], such as sentences and paragraphs, are clearly identified through layout and punctuation. By contrast, the spontaneity and rapidity of speech minimizes the chance of complex preplanning, and promotes features that assist speakers to ‘think standing up’ – looser construction, repetition, rephrasing, filler phrases and the use of intonation and pause to divide utterances into manageable chunks.” These linguistic scholars are not alone. In this thread, I have already listed Brazil (1995), Bright (n.d.), Carter (2001), Linell (2005), Liberman (2014, 2016), and Milroy & Milroy (1991). There is also van Kemenade and Los (2013: 218), who say that “one of the hallmarks of oral versus written styles is the way clauses are connected. The development of a written style tends to involve a tighter syntactic organization: instead of the loosely organized string of main clauses (“parataxis”) characteristic of oral styles, written styles tend to have complex sentences, with embedding (“hypotaxis”) of subclauses that function as subjects, objects, or adverbials of a higher clause.” To this we could add English and Marr (2015: 141), who show that prepared writing uses “frequent clause embedding” rather than chaining clauses in an additive way, which is what spoken language does. They also point out “another feature which characterizes writtenness, particularly in formal writing […], is nominalization. […] It is this that leads to the lexical density that Halliday (1989) refers to and partly to the organizational complexity. Nominalization avoids the need for lots of simple clauses and fewer verbs and promotes instead the embeddedness and indirectness of clause complexity.” I encouraged Schoonvelde et al. to find linguistic studies which agree with their methodology, but they have yet to cite any. Linguists, however, are in agreement, which means that if Schoonvelde et al.’s methodology of studying spoken language with a written language test is permissible, then decades of linguistic research is wrong.

Another one of my criticisms which Schoonvelde et al. did not address was about how they took a test which was developed for English and used it to study other languages. Roach (1998: 151) has noted that comparing different languages involves a “serious problem” because “some languages (e.g. German, Hungarian) have some very long words, while others (e.g. Chinese) have very few words of more than one or two syllables.” But Schoonvelde et al. used the F-K test, which was developed for one language (English), and applied it to other languages (Spanish, Danish, Dutch and Swedish). There is no discussion about why this should be permitted beyond the claim that the F-K test correlated highly with some other German readability tests, which were themselves adapted from the F-K test.

The authors are welcome to reply to this comment. The fundamental problems that I brought up in my original comment remain. But I hope that the authors do indeed take what they have learned from this exchange into their future work.

Joe McVeigh

References
Crystal, David. 2003. The Cambridge Encyclopedia of Language (2nd ed.). Cambridge: CUP.

English, Fiona and Tim Marr. 2015. Why do Linguistics? Reflective Linguistics and the Study of Language. London: Bloomsbury.

Milroy, James and Lesley Milroy. 1991 (2nd ed.). Authority in Language: Investigating Language Prescription & Standardization. London: Routledge.

Roach, Peter. 1998. “Some languages are spoken more quickly than others.” In Language Myths, ed. by Laurie Bauer and Peter Trudgill, 150-158.

Rowe, Bruce and Diane P. Devine. 2006. A Concise Introduction to Linguistics. London: Pearson.

Schiffrin, Deborah. 1994. Approaches to Discourse. Oxford: Blackwell.

van Kemenade, Ans and Bettelou Los. 2013. “Using historical texts”. In Research Methods in Linguistics ed. by Robert J Podesva and Devyani Sharma, 216-232.