PONE-D-24-30453Using natural language processing to understand loneliness in English older long-term care users from free text case notesPLOS ONE

Dear Dr. Rickman,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I have read the reviewer's comments, and I believe they are valid and will improve the manuscript.

I have a question regarding the performance of traditional machine learning algorithms, particularly logistic regression. If you use scikit-learn, I wonder if the parameter class_weight is set to "balanced" instead of the default.

Please submit your revised manuscript by Nov 28 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office atplosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Mario Graff-Guerrero, Ph.D.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. Thank you for stating the following financial disclosure:

“This paper is based on independent research funded through the NIHR Policy Research Unit in Adult Social Care, reference PR-PRU-1217-21101. Funding was also received from the UK National Institute of Health and Care Research (NIHR) Applied Research Collaboration (ARC) North Thames under grant number NIHR200163. An additional grant was received from the NHS Digital Social Care Pathfinders initiative under the contract 8717. The views expressed are those of the authors and not necessarily those of the NIHR, ARC, NHS, or the Department of Health and Social Care.”

Please state what role the funders took in the study. If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“The authors extend their heartfelt gratitude to Uche Osuagwu for his tireless dedication

to managing data extraction and quality, ensuring the data met the highest standards

for academic research.

We are deeply appreciative of William Wood and the Intelligence Solutions for London

team for their vital contributions to Information Governance.

Our sincere thanks go to Hannah Kendrick for her extraordinary generosity in

dedicating her time and effort to establish Inter-Rater Reliability of human annotators.

We are also immensely grateful to Explosion AI for providing us with a free research

license for their proprietary annotation software, Prodigy.

This paper is based on independent research funded through the NIHR Policy Research

Unit in Adult Social Care, reference PR-PRU-1217-21101. Funding was also received

from the UK National Institute of Health and Care Research (NIHR) Applied Research

Collaboration (ARC) North Thames under grant number NIHR200163. An additional

grant was received from the NHS Digital Social Care Pathfinders initiative under the

contract 8717. The views expressed are those of the authors and not necessarily those of

the NIHR, ARC, NHS, or the Department of Health and Social Care.”

We note that you have provided funding information that is currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“This paper is based on independent research funded through the NIHR Policy Research Unit in Adult Social Care, reference PR-PRU-1217-21101. Funding was also received from the UK National Institute of Health and Care Research (NIHR) Applied Research Collaboration (ARC) North Thames under grant number NIHR200163. An additional grant was received from the NHS Digital Social Care Pathfinders initiative under the contract 8717. The views expressed are those of the authors and not necessarily those of the NIHR, ARC, NHS, or the Department of Health and Social Care.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

5. For studies involving third-party data, we encourage authors to share any data specific to their analyses that they can legally distribute. PLOS recognizes, however, that authors may be using third-party data they do not have the rights to share. When third-party data cannot be publicly shared, authors must provide all information necessary for interested researchers to apply to gain access to the data. (https://journals.plos.org/plosone/s/data-availability#loc-acceptable-data-access-restrictions)

For any third-party data that the authors cannot legally distribute, they should include the following information in their Data Availability Statement upon submission:

a) A description of the data set and the third-party source

b) If applicable, verification of permission to use the data set

c) Confirmation of whether the authors received any special privileges in accessing the data that other researchers would not have

d) All necessary contact information others would need to apply to gain access to the data

6. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent. If consent was waived for your study, please include this information in your statement as well.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1:Yes

Reviewer #2:Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1:Yes

Reviewer #2:Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1:No

Reviewer #2:No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1:Yes

Reviewer #2:Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1:This manuscript presents the development of NLP methods to extract information on loneliness/social isolation from clinical notes of older adults in a London borough. Additionally, it correlates these findings with established social isolation scales such as CES, SNI, UCLA, and other data. The study addresses a relevant issue, especially in the context of loneliness during and after the COVID-19 pandemic. However, the reviewer has several major concerns that need to be addressed.

Major Comments:

1. The manuscript covers a broad range of topics, from NLP method development to comparisons with multiple outcomes and activities. The current title does not reflect the full scope of the work. The authors may want to consider splitting the study into separate publications—one focused on NLP methodology and another on the comparative analysis of loneliness/social isolation with various outcomes.

2. The manuscript reports that 3,046 patients have 1.1 million notes, which seems unusually high. It would be helpful to provide more details on the types of notes, their categories, and the nature of the visits associated with them.

3. The authors mention correlating loneliness/social isolation with 'living alone,' but it is unclear how this was identified from the clinical notes. More details on the methodology used to determine this aspect would be beneficial.

4. The manuscript lacks a clear definition of 'loneliness' and 'social isolation.' It is important to note that these terms are not synonymous. One of the recent work (https://arxiv.org/pdf/2403.17199) shows distinctions between these concepts. Including definitions and illustrative examples from the notes would improve the manuscript clarity.

4. The following key studies on social isolation and adverse social support should be cited:

https://arxiv.org/pdf/2403.17199

https://www.nature.com/articles/s41746-023-00970-0

5. The manuscript lacks comparison with existing studies on loneliness and social isolation in clinical settings. It would also be useful to see baseline methods using rule-based approaches for comparison with the NLP methods.

6. The manuscript does not provide sufficient information on how CES, SNI, and UCLA data were collected. Clarifying this will be crucial for understanding the validity of the correlations made in the study.

7. Table 8 presents interesting observations, but the manuscript does not explain how the underlying data were collected.

Minor Comments:

1. Use "1.1 million" instead of "1.1m".

2. Table 8 is referred to far earlier in the text than it is presented.

3. Consider merging Tables 5 and 6 to streamline the presentation of the data.

Reviewer #2:The paper applies NLP techniques to create an index for loneliness or social isolation. Their best model is based on Roberta, and it scored 0.92 using F1. The paper then develops a series of tests to compare the model to survey data.

Since the importance of the study is evident, I have some comments that could benefit the final manuscript.

Comments

It will help the readability if you include some aspects of the appendices in the text, for example, how to pseudonymize the data or the labeling process. Also include the rationale for some of the decisions like replacing masks with "real text".

Explain the content of your data in the "Data" section. Later in the text, you mention it includes gender, ethnicity, age, mobility, demographic, etc. You can describe your data here.

Line 279 "In order to establish that our best-performing model was not picking up demographic characteristics, or simply those cases where there are more notes, we also included these in the model" Here, the model refers to the logistic regression model right? Then, it is not clear how adding some characteristics to the LR model establishes that the Roberta model was not pickin' up those.

The manuscript could benefit from a significant reordering of the text. The section "Model Evaluation" can be explained in the Results section together with the tables from the experiments. You could also include there the discussion of the findings. This could simplify the reading and the flow of the paper. For example, the sentences on line 404: "This difference may not explain the significant association in ELSA between unpaid care and loneliness or isolation" is referring to table 5 (I guess) which appeared 4 pages before.

Figure 4 is unreadable.

line 327 "The χ2 results indicate that living alone and shopping are the only indicators associated in ELSA with both measures of loneliness and social isolation" So if we are comparing CES and UCLA columns with the Both columns, what about Contincence and Meal prep?

In apendix, the sections Classification rules and Lonely or socially isolated are empty.

In the appendix, page 5: "Notes from the test set which contained these terms were then manually classified by human annotators." How were the notes in the train set annotated?

Minor Issues

Introduction

line 2 public expenditure [was] on long-term care was 1.98%

The impact of loneliness and social isolation

line 62 first appearence of term ELSA without a definition (which is on line 65)

Materials and methods

Data

line 94 In Fig 1 we show how [the] the assessment

Data access and time frame

line 97 Write the meaning of NHS

Descriptive statistics

Methods overview

Model development

Model Evaluation

line 241 Explain what are the waves 6-9 of ELSA.

line 263 individuals in the [the]administrative

Equation 1 The order of the variables and their explanation in the text don't match.

Results

line 338 Define GVIF.

Table 6 The columns ELSA and Administrative Records are in reversed order according to Table 5.

Table 6 What is the difference between CES-D and CES?

Table 6: The legend for the * and . marks is wrong.

Discussion

line 390 it may be the case [that]that the individuals

Limitations

line 490 The paragraph appeared on line 475.

line 505 Table 7.[ .]

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1:No

Reviewer #2:No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS atfigures@plos.org. Please note that Supporting Information files do not need this step.

PONE-D-24-30453R1Understanding patterns of loneliness in older long-term care users using natural language processing with free text case notesPLOS ONE

Dear Dr. Rickman,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.The manuscript is going in the right direction, one of the reviewers accept the current version, and the otherhas some concerns and madesuggestions to improve the manuscript, please read them carefully. In my opinion, the comments contribute to improving the paper.

Please submit your revised manuscript by Jan 20 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office atplosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Mario Graff-Guerrero, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1:All comments have been addressed

Reviewer #2:All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1:Partly

Reviewer #2:Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1:Yes

Reviewer #2:Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1:No

Reviewer #2:No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1:Yes

Reviewer #2:Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1:The authors developed NLP systems to identify social isolation and loneliness. They compared the NLP-derived output with other collected data, such as ELSA, SNI, and UCLA loneliness scores. Below are my comments:

1. The abstract and manuscript can be significantly condensed without losing essential details.

2. Recent literature uses "SDOH" as a short form for Social Determinants of Health. The authors are encouraged to adopt similar abbreviations for consistency.

3. All figure and table captions are too brief. These should be expanded for clarity. Also, in Table 1, replace "N notes" with "# Notes" and "N words" with "# Words."

4. The numbers in Table 2 do not align with the 10,083 training sentences and 3,573 test sentences. If these are not identical, the authors should clarify how they differ and provide a detailed explanation.

5. The manuscript mentions a feed-forward neural network alongside RoBERTa and DistilRoBERTa. These are distinct architectures and should not be conflated.

6. The equations for precision, recall, and F-score can be moved to the supplementary materials to streamline the manuscript.

7. To improve clarity, the construct validity results could be presented in a table format within the results section.

8. It would enhance readability if all figures were included within the main text rather than at the end of the publication.

9. The discussion section should be divided into multiple subsections aligned with the results for better organization and coherence.

10. The authors should address why some categories exhibit high accuracy but have zero precision, recall, and F-scores.

11. Including counts of loneliness/social isolation versus no loneliness/no social isolation in both the training and test datasets would be valuable for understanding data balance.

12. It would be helpful if the authors could provide a detailed review response letter outlining the changes made in the text.

Reviewer #2:All my comments were attended to. I have no further recommendations. Since your data is anonymized, you could make it public.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1:No

Reviewer #2:No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS atfigures@plos.org. Please note that Supporting Information files do not need this step.

Understanding patterns of loneliness in older long-term care users using natural language processing with free text case notes

PONE-D-24-30453R2

Dear Dr. Rickman,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager®and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Mario Graff-Guerrero, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

I enjoyed reading the manuscript. Congratulations on the excellent research!

Reviewers' comments: