PONE-D-22-13798Evolutionary-like dynamics of writingPLOS ONE

Dear Dr. Lo Sardo,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The manuscript has been revised by two experts in the field. Both  Reviewers appreciated your work in terms of the potential contribution at the empirical and theoretical level. However, they both  found some critical issues that prevented me to consider this version of the manuscript ready for publication. I agree with them that the measures you introduced need to be better described and discussed, also with respect to other measures present in the literature; in addition, the study needs to be more appropriately situated at the theoretical level. Finally, but very importantly, I recommend to make data on which analyses are based available, clarify what type of consent has been asked to participants and whether the study has been approved by an ethical Committee .

At this stage, I am inviting you to submit a revised version of the paper for consideration.  In doing so, please carefully address each of the reviewers' comments and outline the changes you have made in cover letter to me.

Please submit your revised manuscript by Sep 02 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Francesca Peressotti, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“The following institutions provided financial and organizational support for the organization of the SEW workshops: Sony CSL Paris, Inserm, Sapienza University of Rome, Complexity Science Hub Vienna (Prof. Stefan Thurner), The Albert Einstein Center for Fundamental Physics in Bern (Prof. Gilberto Colangelo), the Institut des Syst`emes Complexes in Paris (Prof. David Chavalarias).”

We note that you have provided additional information within the Acknowledgements Section that is not currently declared in your Funding Statement. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“SONY CSL Paris funded the work of Donald Ruggiero Lo Sardo, Pietro Gravino and Vittorio Loreto

La Sapienza university of Rome funded the work of Donald Ruggiero Lo Sardo and Vittorio Loreto

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

3.  Thank you for stating the following in your Competing Interests section: 

“No”

Please complete your Competing Interests on the online submission form to state any Competing Interests. If you have no competing interests, please state "The authors have declared that no competing interests exist.", as detailed online in our guide for authors at http://journals.plos.org/plosone/s/submit-now

 This information should be included in your cover letter; we will change the online submission form on your behalf.

4. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability.

Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories. Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions. Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: N/A

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: My name is Mark Torrance, I think authors should know who is reviewing their work. My expertise is in the educational and cognitive psychology of text production. I have quite extensive experience of interpreting real-time data from text composition but no direct experience of developing the kinds of revision-analysis methods reported in this paper.

The authors of this paper provide three metrics for capturing how a text grows and develops as a writer or writer moves from first word through production and editing, to a final text. These measures draw on versioning methods that compare editing distance and so are semantically, linguistically, and chronometrically agnostic. They illustrate the potential of these metrics in an analysis of a relatively large sample of writing process data from academic writers producing demanding texts.

I am pleased to see a paper with this focus submitted to a multidisciplinary, general interest journal. I see value in the metrics that they report - I think they make a useful and potentially important contribution - although I would need to invest more time than I currently have to work though in detail what they provide. I also agree with the authors that this general approach - exploring in meaningful ways variation across texts in the extent to which they are manipulated / modified during their production - helps develop out understanding of the mental (and possibly social) processing that underlies human text generation.

I think, therefore, that it should be considered for publication. My concern, however, is that the authors currently make unnecessarily inflated claims about the scope of their work. It is not necessary (or correct) to represent it as offering a radical new research paradigm for studying writing. It is also not necessary to couch their work in buzz words like "evolutionary". What they are doing is (a) offering what are (I think) robust methods for capturing one aspect of writing process that, combined with other approaches, will help to build cognitive theory around text production, and (b) introducing the possibility of rigorous science of text production to a new audience. These are both valuable goals. But their paper will only achieve them if (a) they adequately situate their work in existing science and (b) the provide readers with a stronger feel for the constructs that they are measuring. So less hyperbole and more basic science.

I therefore want to make just three broad points. I'm hoping that the authors will be give the opportunity to revise their paper in light of these, and will be very happy to comment more specifically on the detail of their proposed metrics. (Please excuse typos - I'm rushing to meet the Journal's tight deadline.)

1. Theorising in the introduction is imprecise and insufficiently rooted in existing theory and research. There is too much grand gesture here, I think, and too little incremental science. I hope the following two examples illustrate what I mean. "There is ample evidence that the writing process influences the quality of the final text and that mastering self-regulating strategies leads to higher coherence and, in general, better texts [3–12]. " The first part of this claim is trivially true. Text only results from the processes used to create it. There is not other way in which it appears on the page. The authors might here be thinking just about the particular strategic activity that a writer engages in - the ways in which they choose to regulate their own behaviour as they produce text affects text quality independently of their content knowledge and linguistic skills. If so, that's what they need to say. However, that's an empirical claim, and there's counter evidence. The list of papers referenced here contains a whole range of research, including papers on the effects of first grade instruction - a literature which does not come down strongly on the side of students needing explicit writing strategies (I would and have argued). It also includes one of my papers where our main finding is actually no relationship between process and text quality.

"As such, many have argued that writing is a complex task. The writer has a mental model of her/his work that must adapt since the whole is affected by variations of the constituent parts, such as individual sentences or even words [13]. We postulate that these adaptations trigger shifts from one writing phase to another. During these shifts in the cognitive task, the model of the text can undergo creative reconstruction. The identification and analysis of these events, as well as the study of other patterns of the process, can be crucial to improving writing strategies." Several claims here that need to be properly worked through and situations in existing research and theory. (1) "The writer has a mental model of their work." I am not sure what is meant by "work" here. This could be one or both a representation of the macrostructure (or surface form?) of the text produced so far, or a message-level representation of what they aim to communicate, or what what they believe that they have communicated so far. There are other possibilities. It's also not clear that these representations actually exist or play an important role in written production. They may do, but the evidence isn't there, to my knowledge. See, for example, https://doi.org/10.1017/S0140525X04000056 - and I have argued that similar processes are likely to occur in writing. It is possible that the macrostructure that makes text cohere into a message is emergent rather than something that maps directly onto any mental model, even one that has changed during writing. (2) "that must adapt since the whole is affected by variations of the whole is affected by individual sentences…or even words" Again, I'm not sure what this means. Words and sentences do not just appear. They are output from the writer's mental representation (at whatever level, assuming this exists) and so should necessarily cohere with existing representations. If the writer was a reader then this claim would make perfect sense. There may be a sense in which it is true for writers, but this is a theoretical claim. If it's important, then it needs unpacking and supporting. If it's not then if should be dropped. (3) " We postulate that these adaptations trigger shifts from one writing phase to another. During these shifts in the cognitive task, the model of the text can undergo creative reconstruction." I'm not sure what the authors are thinking of here. Something like you output a particular sentence that has a meaning that surprises you (for some reason). This then means that you decide to revise your text in light of new ideas? Are there other possibilities? If not, then just say this. If there are, then elaborate. Then, why does the shift in the task result in reconstruction (I'm assuming of mental representation of message? Isn't it the other way round - you change your ideas and so see the need to revise your text. And so on. My point here is definitely not that the authors should present a full blown theory of mental processing underlying text production, or that this is necessary, or that the authors intended meaning is wrong. They do, however, need to avoid vague hand-waving. If they are going to make a priori claims about mental processing underlying text production these need to be explained clearly and precisely - avoiding imprecise use of jargon - rooted in existing theory and research, and as far as possible evidenced.

2. The authors should carefully situate the new methods that they described in previous efforts to extract similar information from writers text-generation processes. "Here for the first time" in the following - "Here, for the first time, we uncover those processes by recording, through a suitably dedicated software, all edits and actions in the making of a text, from the early attempts to the finishing touches." - is not justified, I think (or at minimum the authors need to strongly evidence this claim). To be clear, I think the methods that they describe are of potential value, and I would be excited to see them reported in a general-interest journal. However, they do not represent that paradigm shift that the authors are claiming. In particular, they are not an alternative to approaches involving think aloud (although it's a long time since I've seen papers using these methods), experimental manipulations, and keystroke logging. These are currently represented as alternative approaches to a single research goal. In fact, they serve quite different purposes, and are not in any sense mutually exclusive. This is not an appropriate framing for their new methods. What the authors want to demonstrate is how texts evolve over time, particularly in contexts (relatively rare but important) where the text is subject to extensive revision. This is an absolutely worthwhile goal. It is, though, something that has been addressed before. I've not used these methods in my own research, but if I was going to I would search forward from Eklundh, K. S., & Kollberg, P. (2003). Emerging discourse structure: computer-assisted episode analysis as a window to global revision in university students’ writing. Journal of Pragmatics, 35(6), 869–891), which is implemented in Inputlog (cited in the current paper), Also graph-based methods (see https://doi.org/10.1075/z.194.09leb; https://www.ggxlog.net/). Another example is the choice of revision unit problem that they authors address from lines 133 onwards. Their solution sounds good, but the problem has been addressed in previous research (I think) - see papers here https://research.tue.nl/en/persons/ma-rianne-conijn/publications/ . I think a review of this literature needs to be the main focus of the authors' introduction. I am definitely not claiming that the authors' cover the same ground as these, but their focus needs to be on how the methods the they describe move beyond these.

3. The measures that the authors report are interesting and potentially valuable. On the basis of what they report I do not get a strong feel for construct validity. This is for two reasons. First, the authors are vague about the construct that they aim to measure. It feels at times that the measure is driving the construct and not the other way round. For example I am not clear what it means for a writing process to be "structured" as in "The Complexity Score aims to quantify how structured the writing process is." (line 134). The authors illustrate this with "only one sentence was edited repeatedly, the process is highly organized. If all sentences were edited a comparable number of times, the process is indistinguishable from a random one" (line 162). I can see how these two cases are different, and that much more revision occurs in the latter case, but I'm not sure why the second one is less structured or organised. The second reason is that, even as a reader who is used to abstract representations of texts and writing processes, I find it very difficult to relate indices and plots to my understanding of how texts might change or how writers might engage in cycles of revision. I don't feel that I have a strong enough understanding of the measures that the authors propose to make recommendation for how this gap might be bridged, but something is needed. Again, to be clear, I am not at all criticizing the authors for seeking to abstract features of the ways that a writer or writers interact with and develops their texts and specific indices. I just need a stronger feel for what is being measured.

Reviewer #2: Summary

The authors propose a new set of measures to describe text writing processes in the manuscript. The measures are: a) the Complexity Score, which accounts for the complexity encountered by the writer in building the text, measured as the distribution of changes along text versions; b) the Exploration Coefficient, that is the degree of exploration of ideas, measured as the edit distance of a version from the initial and the final versions (the shortest path); c) the Twist Ratio, as the turning points in the flow of thoughts of the writer (exposition and exploration pattern).

These measures were explored in a sample of texts produced by young researchers who attended a workshop on scientific writing. Conclusions stress how these measures can be useful for modeling the writing process.

I found the paper's topic interesting and the research itself appears to be conducted carefully. I applaud the authors for attempting to develop metrics able to quantify such complex behaviors during text production.

I have some advice for improving the exposition which I have found good, but somehow difficult to follow mostly because complex metrics are described in the methodological section without a clear introduction on their meanings and aims. I report below my observations according to the manuscript sections.

Abstract

In the final sentence, I suggest substituting the word "learning" because the study does not treat learning in the writing process.

Introduction

The introduction is overall well-structured and easily readable by a large audience. It makes a good, even if synthetic, overview of the state-of-the-art methods for analyzing text writing processes, from natural language processing to cognitive sciences works.

My impression is that references are scarce. For example, in line 26 there is a list of fields that studied text analysis but only one reference. The following paragraph on cognitive sciences presents only one reference for each method, even if the scientific works are many. Here, I would use a different term instead of "pedagogy" (line 34). Studies are generally concentrated on learning progress during the education path or in analyzing the writing level in a specific grade. "Pedagogy" is more related to "educational facts", while most of the cited studies are in the field of developmental and educational psychology or general psychology and linguistics.

In the same paragraph (lines 47-48), it is not so clear to me why the authors compare keystroke logging to technics such as functional magnetic resonance, GRS, EEG. These technics are common to cognitive psychology but are quite different from each other. I would omit that sentence.

The last paragraphs of the introduction present the study and the outline of the following sections.

I would suggest focusing more clearly on the aims of the work (analyzing complexity, exploration, and exploitation in text generation) and briefly introducing the three original metrics developed.

It seems that the introduction lacks presenting the intents of the study or, in other words, why those metrics are chosen to analyze the writing process. A more detailed explanation of how each metric is devoted to analyzing a specific behavior would make it easier to read the methodological section. Only in Results the reader has a clearer idea.

Finally, in line 62 "we also present a new dataset composed of …". I suggest rephrasing given that the authors are not presenting a dataset. As I understand, the data will be not public for privacy reasons. This poses some doubts about the absence of an ethical statement and the collection of personal information.

Minor points: line 85 and line 88, substitute "section" with "subsection". Line 83, introduce ED as an acronym for Edit Distance.

Materials and Methods

Experimental Setup. Table 1 presents the number of participants in the workshop. However, the authors declare that "the analysis includes only data for participants who's (whose?) draft change at least 10 times". How many participants were discarded? Please, report the final sample size.

Measuring Complexity. Line 146, I suggest introducing the scales you tested. The reader understands only later that are characters, words, and sentences.

Measuring Exploration. Please, define "shortest sequences" and shortest path".

The last sentence of the subsection is not clear and poses doubts on the choice to use characters and not sentences as for the complexity measure. Please define "the structure of individual change".

Results

Overall, the introductory paragraphs of each subsection are discursive and useful to understand the aims of the metrics. It sounds unusual to read in the results section such useful information that would be precious also to understand the methodological section. I would suggest anticipating this information in the introduction of the study.

Exploration. Lines 282-284. Are the minor edits counted in the analysis of exploration? Could this be a limit of the metric?

Discussion

The discussion is well written and gives a good overview of the findings and the potentialities of the metrics.

I was wondering if the heterogeneity of the complexity score (discussed in line 364) would be due also to differences in L2 levels. Less proficient L2 writers probably necessitate more time in revising and restructuring sentences, for example.

Overall, the final discussion is easy-to-read and open to interesting possible applications of the metrics.

Conclusions

Line 445, "discuss university" seems an improper insertion.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Mark Torrance

Reviewer #2: Yes: Tania Cerni

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-22-13798R1How do authors navigate the complexity of the writing

process?PLOS ONE

Dear Dr. Lo Sardo,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

Dear Authors,

I received the comments of two Reviewers. Luckily, both Reviewers were the same who reviewed your paper in the first round. They found that you valuably revised the manuscript, and they clearly see some improvements. However, they both think that this version of the manuscript is not acceptable for publication. In particular, and I agree with them, the study must be situated in the theoretical framework. Pertinent literature should be reviewed, highlighting the innovative aspect of the method described, and its contribution should be explicitly outlined and discussed with respect to previous studies. The details of the methods must be clearly laid out and exemplified. I do not want to discourage resubmission but, since the main issues are almost the same raised in the previous round, I recommend the authors to resubmit their manuscript only if they could fully address them.

With many regards,

Francesca Peressotti

==============================

Please submit your revised manuscript by Nov 12 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Francesca Peressotti, Ph.D

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: My name is Mark Torrance and I was Reviewer 1 of the previous version of this paper. I appreciate the opportunity to review a revised version, and I remain convinced that there is value in the analytic methods that the authors present.

I need to be honest about my limitations, both in time and in competence. I am a writing researcher / cognitive psychologist / psycholinguist. I have used editing ed editing distance in my own research, understand versioning software reasonably well, and so forth. But I'm struggling. I think there is probably real value in the methods that are described but I don’t have the time to work through (or skill to quickly work through) them in detail and assess the authors' claims. I make three points below about areas that I feel need further attention. I recognise and appreciate that work that the authors have put into revising their paper, but in these areas I do not feel that they have gone far enough. However I don’t think my review really does justice to the meat of the paper.

1. The authors continue to make over-inflated claims for the method that they present. If this was the first paper to ever explore text composition processes, then this might possibly be appropriate. However that is not the case. The title and the first four sentences of the abstract illustrate this. An appropriate title would be "A new method for describing the changes that writers make to their texts during composition" (or a better version of this). The title that authors offer is "How do authors navigate the complexity of the writing process?" I am not sure what this really means. Something like "How do writers produce complex texts?" perhaps? But if so, that's not a question that could possibly be answered in a single study (or lifetime!). Note that this is not a criticism of (just) the title and the start of the abstract. It's a repeat of my request to recast the whole paper so that it is clear what the method that the authors presents offers to researchers interested in establishing patterns in how writers' texts develop. This requires carefully specifying the problem that needs solving, looking in some detail at existing methods, and then presenting the new method in such a way as to make clear what it is adding to what already exists. As I said above, I am convinced that their method does make a contribution, and my understanding is that PlosOne is very happy to present basic, incremental science (i.e. there is no need to inflate claims).

2. Unless the authors have strong supporting evidence then I think they should drop all claims about underlying cognitive processes. Their method is useful (I think) in describing a set of phenomena. It does not provide direct insight into the cognitive processes by which these occur. For example "This result is consistent with the idea that a sentence is the smallest complete unit of thought." (line 171). I don't know what a "complete unit of thought" might be - as a cognitive psychologist it's not a construct that I'm familiar with. There is the idea of "planning scope" in literature on spoken and written production, which might be the sort of thing that the authors are thinking of. However, there's a quite well developed literature here that suggests this is less than a clause, and certainly less than an orthographic sentence (string ending with a sentence-terminating punctuation) which I think is the unit that the authors are talking about here. Similarly here "We are focusing on the organization of the author’s attention across the text. The distribution of edits in the text space represents the distribution of the author’s effort. We use the effort as a proxy for the author’s attention." Both attention and effort have specific (and varied) meanings in cognitive theory. Exactly what is meant here is not clear. It either needs to be unpacked and evidenced or (better) dropped. Again, this is illustrative. I think the authors need to go through their paper and remove all unsupported assumptions about underlying process. The one exception to this is where the authors discuss findings from their own data. Here I think some conjecture about process is valuable, though it needs to be strongly caveated.

3. I am still struggling to understand the methods that the authors describe. This is partly because I feel that I need to spend a lot more time with them than I have available for this review. But partly because I think there are gaps in the authors explanations. For example - lines 152 to 172 - first paragraph of "measuring complexity". I understand how you can establish editing distance at the character level. However, if your unit of analysis is a sentence, then are two sentences that are different by one character and that are different by multiple words then treated as similarly different. The fundamental problem I'm struggling with is that whereas characters are themselves invariate (an "s" is always an "s"), words and sentences aren't. Authors don't just move whole sentences around. I'm clearly missing something here. But I don't think I can find it in the author's existing explanation. I need a worked example, or more detailed explanation to understand.

Reviewer #2: I wish to thank the authors for considering my suggestions. I would also reiterate my positive opinion about their work, especially considering the metrics they developed. Some aspects in the methodological section (such as participants and ethical issues) are solved.

However, in my opinion, they did not substantially improve the introduction of the manuscript as theoretical aspects are not well contextualized. As I stated in my previous comments, the introduction is not exhaustive, and the aim of the study and of the metrics is not easily deducible without going into the methodological section. The methodological section should be devoted to understanding the technical details of the metrics, not their theoretical functions which should be better described in the introduction. This is only partially done. I see in this new version a synthetic picture of the framework and of the study aims. Nevertheless, I suggest rethinking how to introduce the metrics keeping in mind the large audience that the manuscript could reach thanks to this journal.

In this vein what I recommend is to rethink the theoretical framework. As I previously wrote, I have the impression that references are scarce. Lack of references is not only (and not always) a matter of few citations, but it is also (and more importantly) a matter of anchoring the present study into a theoretical framework. For instance, text planning, editing and revising processes are previously studied. However, the literature review presented here does not say how the previous studies addressed and measured these processes.

A targeted example is: (line 22) "We postulate that these adaptations trigger shifts from one writing phase to another, such as from sentence generation to revising or even planning. The identification and analysis of these events, as well as the study of other patterns of the process, can be crucial to improving writing strategies". How did the cited literature in the next paragraphs (in natural language processes and in cognitive science) address (or did not) this "identification and analysis"? And why do the present metrics add new insights? Answering these questions would help in identifying why and in which framework the specific methods (the metrics) are developed, and so the aims and the merits of the work.

I take the liberty to report a sentence of reviewer #1 that probably better explain in a few words my impression: "It feels at times that the measure is driving the construct and not the other way round".

Some details I would like to underline:

Line 51-61. The authors re-formulate the sentence about keystroke logging and other techniques such as EEG, and fMRI. Now the reason why they mention these techniques is clearer. However, (1) such techniques are just mentioned without any citation of the studies they refer to. It also lacks to explain to us the contribution of these studies to text production analysis. Furthermore, the sentence: "Keystroke logging is not conceived to analyze text evolution as a whole or where the author focuses her/his effort." Is this really true? Can you justify the sentence? What do you mean by "text evolution as a whole"? I suppose that keystroke logging (also when powered with physiological or neural measures) is devoted to understanding deeply the cognitive effort of the writer. Knowing the studies that the authors are referring to would be helpful to disentangle these doubts.

Line 82-90. The added sentence helps in contextualizing the measures. I think that the authors should reformulate the expression "writing cloud" as they have not mentioned what is a writing cloud yet. Alternatively, they should define it.

To conclude, I sincerely thank the authors for their efforts in disentangling some passages of their manuscript. I encourage them to further review their work by reformulating the introduction keeping in mind the novelty of their contribution and the specificity of their field of study, but without forgiving to contextualize their work also in the framework of previous literature about theoretical and methodological aspects of text production.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Mark Torrance

Reviewer #2: Yes: Tania Cerni

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-22-13798R2Exploitation and exploration in text evolution. Quantifying

planning and translation flows during writing.PLOS ONE

Dear Dr. Lo Sardo,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

ACADEMIC EDITOR:

Dear authors,

As you can see from the attached comments, both Reviewers consider that your manuscript has really improved as, pending some minor revisions, it is valuable for publication in PlosOne. I particularly appreciated how the result section has been organized, with accurate description of each measure before the corresponding results. Please, solve the minor points raised by Reviewer 1 who asked for few additional methodological clarifications. I am looking forward to receiving your revised manuscript.  

Best regards,

Francesca Peressotti

Please submit your revised manuscript by Apr 02 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Francesca Peressotti, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: (No Response)

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: My name is Mark Torrance, and I reviewed two previous versions of this paper.

First I want to say again that I think the methods described in this paper have the potential to make a important and significant contribution to our understanding of how writers produce extended texts. This is a difficult multi-dimensional problem. The approach is conceptually different from anything I've seen before and I think the academic community needs to hear about it. I therefore would very much like to see this paper published.

Second, I congratulate the authors on their new framing of their methods. As someone very close to, and critical of, the field, there is detail here that I'd disagree with, but nothing that needs revision. I think this is a very competent summary, particularly for a non-specialist audience. I also appreciate the way that this new framing is worked into the rationale for their measures, and interpretation of findings.

I have made a few specific comments below, written as I read through the text. I hope they make sense.

My major remaining concern is still that there are bits of the method that I don't understand. Certainly not at the level at which I could imagine reproducing the analysis, but for some measures (twist ratio in particular), I don't feel I have enough of a grasp of how these were computed to properly understand what is being measured. For a paper that has as a primary focus introducing new ways of characterising how texts develop, I don't think readers should have to take on trust that measures measure what the authors say they measures (and clearly this also isn't the authors' intention). So I am still asking for some clearer explanation in places. Some more detail in my notes below.

Partly with this in mind, I think the authors should consider separating out description of measures - with associated illustrative plots (Figures 1a, 2a, 3) and their reporting of results. In the results section, in addition to what the authors already provide, I would like to see a couple of other things: Some analyses that show the relationship among measures (aggregating within writer/text). Just scatterplots with regression lines would be fine here. So are texts with a high exploration coefficient have higher or lower twist ratios, and so forth. And also relationship between aggregate measures and (a) total writing time (I realise that this is closely related to version index - and you've already done this for the exploration measure) , and (b) final text length (in words).

Specific comments

Abstract: "More than 60 junior researchers" - give exact N.

"A substantial body of theoretical and experimental work across psychology, linguistics, and education has developed well-accepted models of the process of writing. " So this is an over-correction and has swung too far in the other direction. There is a reasonably substantial body of work in this area, and there are some well-cited (though often not well-supported) theories. But understanding of processes underlying written production remains weak, and lags a long way behind, for example, understanding of how people read. I think all you need to say here is just that there is an existing and growing body of psycholinguistic and educational research.

"Here, we propose a new method that can efficiently detect patterns indicative of writing sub-cycles in much smaller datasets than existing NLP methods, presenting a much smaller overhead than previous work such as S-notation [15] given that it does not require keystroke logging to reconstruct the activity of the author." I'm not clear here what overheads the authors are referring to. Collecting keystrokes and then reconstructing all of the author's revisions is not computationally too complex (or at least is a problem that's been solved) and datasets are not huge. Versioning is an alternative approach, and there might be some practical advantages. However, I don't see this as a major contribution of the authors method. Also, given that any version could be reconstructed from a keystroke log, it would presumably be possible to apply them to keystroke data (after some post-processing). My point here is just that the methods that the authors describe contribute something conceptually valuable (and more important) than the fact that text-states were captured by versioning. Versioning isn't a main selling point (or perhaps not a selling point at all).

There is a very short introduction to the key concepts of complexity, exploration, and the twist ratio at the very end of the introduction. I suggest another paragraph, at the start of the analysis / results section of the paper (p6) that unpacks these a bit more and provides orientation to the three sections that follow.

Complexity / Figure 1. I don't know whether this is because this text has been revised, or whether I've finally given it enough attention, but I think I understand the approach here, and what Figure 1 shows. However I'm am still puzzled by something. I was confused by how you handled inserted sentences without changing the index of all subsequent sentences (and therefore seeing them all as having been revised). But then I noticed "taking care to add place-holder elements for the sentences that will be inserted later in the draft so that unmodified sentences will correspond to the same positional index as the draft evolves". Good. But I don't understand how this could be achieved. Or how you handle sentence deletion. So I'm puzzled again about how this works.

Twist Ratio. The explanatory panel in Figure 3, combined with the text, gives me a good understanding, I think, of how you are measuring translation flow, and I think this is a valuable idea. Beyond that I get very lost, however, in particular I don't understand lines 402 to 416. I understand twist ratio measures as angle. But I don't understand "we must decide how many dimensions to use" - dimensions in what space? I'm clearly missing something here, which suggests the need for some more careful unpacking and explanation.

Minor point, but this "If the writer follows this “path of least resistance” very directly, this is an indicator that their original plan for the text went largely unchanged during the process of writing." isn't necessarily the case. Students often write in a very linear way, with very little revision, but this could equally be because they are simply following a train of thought - allowing previous sentences to cue what they say next- and are not particularly motivated to then restructure what they've written. This is the sort of behaviour you would expect if, for example, you asked a student to write a 5 paragraph essay on a familiar topic, in a low-stakes context (or when writing in an exam). In fact, strongly linear writing, with little revision, is, I think ) less likely when writers have a clear plan. Because in this case the text they produce interacts with (is evaluated in light of) their intention. Plans set out communicative intentions for the text but (obviously) not the surface form of the text that is to be produced to fulfill these. Working out these intentions in the final text typically involves trial and error. Hence strong plans tend to result in more revision.

Minor presentational point: I suggest removing contours from Figure 2b and 4c - they don't add anything useful and makes the plot look like it's showing more than it does.

Draft vs. version: The authors use "draft", "version" and "draft version" interchangeably, I think. For me "draft " suggest a version that is identified by the author as significant (ready to show to a co-author, perhaps). Version is a better term for the text that's captured every 1 minute / 3 minutes .

I don't find breakdown by workshop session useful in density plots. I'm not really sure what I should do with this information. Similarly, I think the authors could probably just say "across three workshop sessions (Ns = )" or similar in the methods rather than giving more detail.

Reviewer #2: I commend the authors on the work done in revising the manuscript. I appreciated that they have addressed my major concerns and I am satisfied with the revision. The explanation of their aims and measures is much clearer. Terminology is better specified, and the flow of the manuscript is improved.

I report an oversight: the sentence in lines 85-87 is incomplete ("In analogy with the study of strategies in evolutionary game theory we also use the term exploration to refer to planning phases, where the author tests different options for the draft, an").

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Tania Cerni

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Exploitation and exploration in text evolution. Quantifying

planning and translation flows during writing.

PONE-D-22-13798R3

Dear Dr. Lo Sardo,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Francesca Peressotti, Ph.D

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments: