PCSY-D-24-00128

Human languages trade off complexity against efficiency

PLOS Complex Systems

Dear Dr. Koplenig,

Thank you for submitting your manuscript to PLOS Complex Systems. After careful consideration, we feel that it has merit but does not fully meet PLOS Complex Systems's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript within 60 days Dec 09 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at complexsystems@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pcsy/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

* A rebuttal letter that responds to each point raised by the editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

* A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

* An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

We look forward to receiving your revised manuscript.

Kind regards,

Giulio Rossetti

Academic Editor

PLOS Complex Systems

Journal Requirements:

Additional Editor Comments (if provided):

Reviewers found the proposed contribution interesting for PLOS Complex Systems but not yet ready for publication.

Concerns were raised about the novelty, clarity of certain sections (especially the introduction), and some methodological choices.

Additionally, the reviewers emphasized the need for the authors to revise the study's framing to better support their conclusions.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Does this manuscript meet PLOS Complex Systems’s publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: No

Reviewer #3: Yes

--------------------

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

--------------------

3. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

--------------------

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Complex Systems does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

--------------------

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors produce an interesting entropy-based framework for comparing language complexity and its interplay with token-based efficiency. They find that there is an interplay across topics and languages so that complexity cannot be considered equal across languages. The results are presented in a coherent way and several quantitative findings are contrasted in view of relevant literature, including reference 118, which is quite important in the field.

The manuscript adheres to the standards of the journal venue and I think that, with some minor rewritings in some part and more caution on not exchanging language structure per se against the mirrored reflection of language captured by LMs, this manuscript will be a nice addition to PLOS Complexity.

---- I hope my comments can help the authors:

- The Authors should explain more what are the differences between the current work and their past dataset release in reference 35.

- The introduction of the equi-complexity hypothesis is a bit sudden. Shouldn't we expect for LLMs to mirror human patterns also in terms of grammatical complexity of their generated language? Why should there be deviations despite reinforcement learning? I understand loss should be present because context or attention might be limited when training models but this aspect should be better underlined. This is a delicate point and, with some rewriting or addition, it should be better presented to the readership who might be familiar with some aspects of LMs' training.

- The Intro should be split in subsections highlighting more the review and the scoping components.

- Page 18 - The discussion for the entropy estimates is clean and precise. However, I am not sure how spaces and punctuation are managed. From the text it looks like they are also compressed into symbols or letters. However, spaces and punctuation might have different entropy rates relative to stylistic choices of the author. In the same language, punctuation can provide information about the style of individuals, see the work by Darmon and colleagues 2021 (https://arxiv.org/abs/1901.00519). My confusion might be originating from the original definition of \\k as a "text", thus including sequences of words.

- If spaces were erased, this would also be problematic. Forbidden sequences of phonemes/graphemes could appear if adjacent words were put together without spaces. Please provide more explanations in this part.

-Why was the Pearson correlation preferred over the Kullback-Leibler divergence?

- I think the message at line 685 should be frontloaded and replace the quasi-complexity hypothesis. Efficiency and complexity should be better defined and identified in the Introduction.

- Pages: 31-32. Authors should also be careful: They are finding what is captured of language complexity/efficiency by the mirrors of LMs. Authors make some claims directly on language structure, instead, whereas the entropy analysis clear has limitations in capturing features like context or overall narrative flow (a distribution of sequences cannot be ergodic if it depends on narrative structure or topics, right?).

- Page 42 - Maybe add some references when mentioning the works of Transformers? Also, transformers like BERT or GPT might work in very different ways in terms of extracting syntactic relationships from texts, it would be better to be more specific.

- The Discussion starting at Page 47 on complexity is highly relevant and authors should be praised for mentioning specific examples. It should be noted also that, once trained, LMs can attribute meaning to individual concepts in ways way different from how humans do. An example are the semantic frames produced by high-schoolers and GPT-4 simulated schoolers for specific concepts in Abramski et al. (2023) - https://doi.org/10.3390/bdcc7030124. GPT's semantic frames contained way more and more specific jargon compared to human responses.

Reviewer #2: In the present manuscript several different types of language models (ranging in sophistication from n-gram models to transformer models) are trained on a large corpus of texts from many different languages found in the world. The languages examined are found to have different measures of complexity, but the rankings are consistent across the different types of language models. The authors argue that their results challenge claims that all languages are equally complex. Although this was a well-written and interesting paper to read, there are several issues that significantly undermine the present work.

1) The claim that all languages are equally complex applies to spoken and signed languages, which are readily acquired from the ambient environment by hearing/deaf children without formal instruction. Writing is not language; it is a technology that has been invented multiple times (with several different approaches—i.e., alphabetic vs non-alphabetic systems) around the world. Further, reading and writing are skills that require excruciating amounts of time and effort to acquire and master through formal instruction. I am not aware of any instances in which a person learned to read or write in the same way they learned to speak/sign their first language (i.e., without formal instruction). Therefore, the present analysis of text—while interesting—does not directly address the issue of languages being equally complex, in part because there are many other aspects of language (some of which are discussed below) that occur in spoken/signed languages that do not appear in written text. Because these other linguistic features are not included in the calculus for complexity used in the present study, the estimates of complexity that are derived for each language are incomplete and therefore not valid. There are also aspects of various writing systems that do not appear to have been considered, further reducing the reader’s confidence in the estimates of complexity that are derived for each language.

2) In lines 385-386 the authors state that they extracted characters, words, and sub-word units to use for their analysis. By characters I assume the authors mean letters. What does not seem to be included in the analysis is a measure of the orthographic depth of the languages. By orthographic depth I mean how directly letters map on to phonemes (the sounds used to make words) in a given language (Frost, Katz, & Bentin, 1987). In languages with a shallow orthography like Spanish there is a (nearly) one-to-one mapping between letters and sounds; the /f/ sound is always made by (and only by) the letter f. However, in languages with a deep (or opaque) orthography like English or French there is more variability in the mapping between letters and sounds. Consider for example the /f/ sound in the English words: fig, phone, cuff, and cough (and then contrast the sound made by the -gh- letters in cough with the sound not made by those letters in the word dough, and the sound made by those letters in the word ghost). It is not clear how orthographic depth is factored in to the present measure of complexity, if it is at all. Are languages like Mandarin which use a “character” (which may contain few or many line-strokes) to represent a whole syllable considered more or less complex than the same syllable spelled out with several letters in an alphabetic writing system?

Further regarding “characters,” was lexical tone coded somehow as a character for the tone languages in the corpus? Mandarin, a tone language spoken by a large number of people in the world, does not represent lexical tone in the written form. However, Hmong (one of the languages that appears to be included in the corpus) does use a letter at the end of words to represent the lexical tone that should be used (the only instance of encoding tone in the orthography that I am aware of).

Furthermore regarding “characters,” how were “characters” that are not typically included in the written form of a word coded? For example, in Hebrew and Arabic (one of the languages that appears to be included in the corpus) the vowels of a word are not written—only the 3 letter consonant roots are written with the reader left to infer the vowels from the context. Do such “invisible” characters make the language more or less complex?

Regarding the sub-word units, do these correspond to syllables, morphemes or some other meaningful linguistic unit? Or are these simply n-grams which may or may not capture the large amounts of meaning carried in morphemes, for example?

Regarding the word units, how exactly was “word” defined (I saw no definition of this in the manuscript)? There is, to my knowledge, no definition of “word” that applies cross-linguistically (Haspelmath, 2011). For example, in Yucatec Mayan (one of the languages that appears to be included in the corpus), a “word” may contain several prefixes, suffixes, and infixes creating a single (long) “word” that might be better described in English as a sentence that contains several independent words (e.g., chilikbalen = “I'm in a lying down position.”).

3) In lines 468-470 the authors refer to “levels of writing script.” I did not see this defined anywhere. To what does this phrase refer? Is this distinguishing among alphabetic and non-alphabetic orthographies, depth of orthography, or something else?

4) Using only entropy to define “complexity” seems incomplete. Including a measure of training time (as in Table 2) might add more nuance to what is meant by complexity. Children around the world hit the same linguistic developmental milestones at roughly the same time (which may say something about the equi-complexity of languages…). However, if one of the present language models needs more/less time to learn one language versus another language that might further hint at the complexity of a given language (e.g., perhaps one language makes extensive use of long-distance dependencies, but another does not). To be clear, such an observation would still not constitute evidence for/against the equi-complexity of language debate because (as described above) the present work examined the technology of writing, not language (and missed a number of relevant measures related to orthography as well).

In short, I found the work interesting, but it might need to be conceptually reframed because it is not a convincing piece of evidence in the equi-complexity of languages debate, and therefore does not meet one of the publication criteria for PLOS Complex Systems (i.e., Substantial evidence for its conclusions).

References

Frost, R., Katz, L., & Bentin, S. (1987). Strategies for visual word recognition and orthographical depth: A multilingual comparison. Journal of Experimental Psychology: Human Perception and Performance, 13(1), 104–115.

Haspelmath, Martin. (2011). The indeterminacy of word segmentation and the nature of morphology and syntax. Folia Linguistica, 45(1), 31–80. https://doi.org/10.1515/flin.2011.002

Reviewer #3: The paper presents an investigation into text complexity across various languages, employing concepts from information theory as the core analytical framework. It uses both parallel and comparable corpora from different genres and analyzes them through a range of language models, from simple sequential models to more advanced transformer-based architectures, to derive entropy values. The key findings reveal a consistent hierarchy of text complexity across languages, regardless of the model used, and highlight a trade-off between complexity (measured via entropy rates) and efficiency (measured by document length). The authors propose that more complex languages achieve greater efficiency by compressing symbols more effectively and that this could be shaped by social demographics factors, particularly speaker population size.

Strengths:

- The analysis is rigorous, framed within a strong, well-structured framework that applies entropy to explore text complexity. The methodologically sound approach strengthens the reliability of the findings, especially in a cross-linguistic context.

- The authors provide an in-depth analysis of the results, particularly their discussion of the complexity-efficiency trade-off within the context of social environments. Their insights into how speaker population size might influence this trade-off add an additional layer of depth to the research, contributing meaningfully to the understanding of a highly debated notion as linguistic complexity.

Weaknesses:

The paper’s contribution in terms of novelty is limited, as much of the research builds on previously reported work, particularly the study cited as [35]. While the authors acknowledge this, it restricts the paper’s potential to push the field forward with new findings or approaches.

The Introduction is dense and somewhat confusing, as it mixes the methodological approach with core research questions. It begins with a discussion on the effectiveness of language models for linguistic research in a cross-linguistic setting, then shifts to the importance of parallel corpora, and only later touches on the main objective of the study. Rearranging this section — starting with the research objective, followed by the methodological tools — would make the paper more cohesive, guiding the reader more effectively and setting a clearer foundation for the study.

The research questions are not clearly articulated, making it difficult to interpret the analysis of results fully. For example, the discussion section focuses on the similarity in outcomes across different language models, but this is not introduced early enough to give proper context. If this is a central issue, it should be explicitly stated in the Introduction, along with the rationale for choosing different language models based on their architectural structure and the expectations for their performance. Clarifying these points earlier would improve the paper’s flow and help readers better understand the significance of the findings.

Minor:

PPM should be defined when first mentioned.

--------------------

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

--------------------

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Human languages trade off complexity against efficiency

PCSY-D-24-00128R1

Dear Dr. Koplenig,

We are pleased to inform you that your manuscript 'Human languages trade off complexity against efficiency' has been provisionally accepted for publication in PLOS Complex Systems.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow-up email from a member of our team. 

Please note that your manuscript will not be scheduled for publication until you have made the required changes, so a swift response is appreciated.

IMPORTANT: The editorial review process is now complete. PLOS will only permit corrections to spelling, formatting or significant scientific errors from this point onwards. Requests for major changes, or any which affect the scientific understanding of your work, will cause delays to the publication date of your manuscript.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they'll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact complexsystems@plos.org.

Thank you again for supporting Open Access publishing; we are looking forward to publishing your work in PLOS Complex Systems.

Best regards,

Luca Maria Aiello

Section Editor

PLOS Complex Systems

Hocine Cherifi

Editor-in-Chief

PLOS Complex Systems

***********************************************************

I am pleased to inform you that, following the final review, your paper has been accepted for publication.

Reviewer Comments (if any, and for reference):

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Does this manuscript meet PLOS Complex Systems's publication criteria? Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe methodologically and ethically rigorous research with conclusions that are appropriately drawn based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available (please refer to the Data Availability Statement at the start of the manuscript PDF file)?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception. The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS Complex Systems does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors addressed all my points.

Reviewer #2: (No Response)

Reviewer #3: The revised manuscript represents a significant improvement in both structure and content. I am pleased to see that the authors have effectively addressed the key points raised during the first round of reviews. I particularly appreciate the inclusion of the Limitations section, which thoughtfully acknowledges the constraints of the study, as well as the additional analysis conducted to enhance the paper’s depth. In my opinion, the manuscript is now suitable for publication in its current form.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

Do you want your identity to be public for this peer review? If you choose “no”, your identity will remain anonymous but your review may still be made public.

For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********