Predicting a diagnosis of ankylosing spondylitis using primary care health records–A machine learning approach

Jonathan Kennedy; Natasha Kennedy; Roxanne Cooksey; Ernest Choy; Stefan Siebert; Muhammad Rahman; Sinead Brophy

doi:10.1371/journal.pone.0279076

Peer Review History

Original SubmissionNovember 23, 2020
16 Aug 2021 Decision Letter - Alfredo Vellido, Editor PONE-D-20-36921 Predicting a diagnosis of ankylosing spondylitis using primary care health records – a machine learning approach. PLOS ONE Dear Dr. Kennedy, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Sep 30 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Alfredo Vellido Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section. 3. Thank you for stating the following in the Acknowledgments Section of your manuscript: "This study was funded by UCB. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The SAIL databank was used and all data used in this study are available through the SAIL application process, https://www.saildatabank.com/application-process. This work was supported by Health Data Research UK , which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and the Wellcome Trust." We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: This work was supported by Health Data Research UK , which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and the Wellcome Trust." Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 4. Thank you for stating the following in the Competing Interests section: "JK, SB reports grants from UCB Pharma, during the conduct of the study; and involved in (but not directly funded by) grants from Biogen, Sanofi and Novartis. RC reports grants from Pfizer, during the conduct of the study; . EC reports grants from UCB, during the conduct of the study; grants and personal fees from Pfizer, grants and personal fees from UCB, grants from BioCancer, grants from Biogen, grants and personal fees from Novartis, grants and personal fees from Roche, personal fees from Amgen, personal fees from Chugai Pharma, personal fees from Eli Lilly, grants and personal fees from Sanofi, personal fees from Abbvie, personal fees from Janssen, personal fees from Gilead, personal fees from Bristol Myer Squibbs, outside the submitted work; In addition, Dr. Choy has a patent null pending. SS reports grants and personal fees from AbbVie, grants and personal fees from UCB, grants and personal fees from Novartis, grants and personal fees from Janssen, grants and personal fees from Pfizer, grants from Bristol Myers Squibb, from Boehringer-Ingelheim, outside the submitted work." Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests). If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf. 5. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide. 6. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ. 7. Please upload a new copy of Figure 1 and figure supplemental 1 & 2. as the detail is not clear. Please follow the link for more information: https://blogs.plos.org/plos/2019/06/looking-good-tips-for-creating-your-plos-figures-graphics/" https://blogs.plos.org/plos/2019/06/looking-good-tips-for-creating-your-plos-figures-graphics/. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes Reviewer #2: Partly ******** 2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes Reviewer #2: Yes ****** 3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #1: Yes Reviewer #2: Yes ****** 4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #1: Yes Reviewer #2: Yes ****** 5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #1: 1. Introduction section can be extended to add the issues in the context of the existing work 2. Literature review techniques have to be strengthened by including the issues in the current system and how the author proposes to overcome the same. 3. What is the motivation of the proposed work? 4. Research gaps, objectives of the proposed work should be clearly justified. 5. The authors should consider more recent research done in the field of their study (especially in the years 2018 and 2020 onwards). 6. The paper needs to provide significant experimental details to correctly assess its contribution: What is the validation procedure used? 7. Kindly provide several references to substantiate the claim made in the abstract (that is, provide references to other groups who do or have done research in this area). 8. An error and statistical analysis of data should be performed. 9. The conclusion should state scope for future work. 10. Discuss the future plans with respect to the research state of progress and its limitations. 11. Kindly refer the below paper: 1. Rajput, D.S., Basha, S.M., Xin, Q. et al. Providing diagnosis on diabetes using cloud computing environment to the people living in rural areas of India. J Ambient Intell Human Comput (2021). https://doi.org/10.1007/s12652-021-03154-4 Reviewer #2: * METHODOLOGY * Novel machine learning methods can potentially identify -> Are any of the ML methods employed truly novel or designed for the purposes of the experiment? If not, remove the word "novel". In this study, 10% (n=543/5751) of patients were used to develop the model and the remaining patients with a diagnosis between the ages of 15 to 35 years (n=1559/5751, 27% of total) were used to examine the generalizability of the model. This means 63% of the data was not used in the analysis as these patients were diagnosed with AS before the age of 15 or after the age of 35. -> The 5751 seems to be from the identified male patients. What about the female? Or am I misunderstanding, and this should be made clearer? This means 63% of the data was not used in the analysis as these patients were diagnosed with AS before the age of 15 or after the age of 35. -> The majority was not used. NICE was mentioned as the reasoning behind this decision, but you should offer a further discussion as to the benefits vs. detriments of effectively discarding this much data. Additionally, codes are aggregated so that concepts can be included in the features (e.g. a “pain” code flag may aggregate all the specific pain codes (pain in left thumb and pain in lumbar spine) into one combined “pain present” code -> Unless I am misunderstanding, how is it beneficial to aggregate all the pain responses into 1 feature? I'd think it would make more sense to have a "pain in AS-associated areas" and "pain in other areas". (I think you actually do this as elaborated later, but then this sentence needs to be better worded.) * MODEL * 79300 tests were run for all 793 AS patients and 100 controls each for the male and female cohorts -> Shouldn't this be 555 (380+175), or were you also using the data set aside for generalizability? If you were, please explain why, because this would seem somewhat problematic. The model selected is the one with the highest average F value across the 100 runs. -> Why select only 1 model? May there be a benefit to using the previous 100 training models together as a random forest? This list of significant variables was then transformed utilising principal component analysis to better understand the structure and relationship of the features to each other. -> More information on PCA is needed in the manuscript body, e.g. how many principle components were kept after the transform and why was that number selected? Arbitrary? * RESULTS AND ANALYSIS * Table 1 -> Validation PPV and F-score are very low (0.15% and 0.30% respectively)? Table 4 -> Validation PPV and F-score are very low (0.25% and 0.50% respectively)? If we make the assumption that a “perfect” model would have 90% sensitivity and specificity as higher 19 percentages than this would likely be attributed to overfitting of the model. -> One could counter overfitting e.g. by making better use of a validation set during hyperparameter tuning/optimization. (But this is a small point that you needn't necessarily do anything with, I'm just saying >90% should be achievable without overfitting.) This implies that the female model is more accurate than the male model at identifying patients with AS. -> Could it have something to do with the fact more principle components seemed to be used? Please explain. It must be recognised that with low prevalence of AS in the general population, the positive predictive value in general and primary care clinical settings will always be very low. -> I don't think this is necessarily true -- a model with extremely high sensitivity and specificity should be able to achieve a relatively high PPV. However, I acknowledge there are some limitations of your specific study -- and perhaps the nature of your model -- that inhibit your model's ability to get a high PPV. Perhaps this can be better clarified in the manuscript body (but it's not detrimental if it cannot). The results of laboratory tests were not used in this model as they were too complex to interpret in the time available -> Do you mean the project development time or the 3 years of time before diagnosis or something else? This should be specified. Thus, 100 Chi-squared tests were run for each of the 3 years preceding the diagnosis date (700 chi-squared tests) -> You should better explain why 700 tests were required (the reasoning behind the number). There is some discussion mixed into the results, e.g. "This may imply that there is an increased reluctance to consider a diagnosis of AS or to perform spinal or pelvic x-rays for young female patients compared to their male counterparts" or "suggesting that women have a more complicated path to being diagnosed with AS, with multiple tests, diagnoses, and referrals, compared to men with AS" or "however, women are more likely to be given the test earlier than men. This is likely due to AS symptoms in females being suspected as a result of other conditions, such as RA" or "There was a smaller female cohort for analysis, which could have meant the study was underpowered to detect uveitis as a predictor in women." Please have a separate Discussion section for these and like comments. * WHAT YOU NEED YET INCLUDE * What software package did you use to implement your models? What were your hardware specs? What's the runtime of the model(s) on the system? What is the filesize of the models? Needs more background and cited works, especially about other machine/deep-learning related AS-diagnostic works regardless of their inputs (e.g. images, features derived from images). Include at least: -> http://dx.doi.org/10.1186/s42358-020-00126-8 -> https://doi.org/10.1016/j.compmedimag.2020.101718 -> https://doi.org/10.1007/s10067-019-04553-x Please provide an ROC curve for your models and report AUC. If possible, provide some statistical analysis on the ROC curve (e.g. bootstrapped confidence intervals). * POSSIBLE DISCUSSION POINTS * This is likely due to the general population cohort having incomplete coverage of data compared to the other cohorts. -> For generalizability, what is the likelihood that any given patient with back pain has in their record all the utilized data/stats? What could be done about it if they are missing some of the stats? * POSSIBLE RESEARCH EXPANSION * What about taking only the most common stats (that would be in the record of most patients) and possibly a shorter history time frame and re-running the experiment? How might results differ? It would be interesting to see and possibly include, but if you feel it is too out of scope consider it for future work. * GRAMMAR AND WORD CHOICE * AS is an important inflammatory musculoskeletal condition with effective therapy available, especially if identified early -> Word choice: "important"? Important how so? A better word could be used here to convey the meeting you want, which admittedly was somewhat vague to me. especially for women with AS who have not well studied or recognised. -> Grammar and meaning: "especially for women with AS, who traditionally have had later diagnoses possibly due to other factors; this area requires further research." an unspecific back pain code present between age 15 and 20 AND a pelvic pain codes present between age 25 and 30. -> Grammar: between ages X and Y (also elsewhere); a pelvic pain code or pelvic pain codes. who hasan ESR test -> Grammar: has an The model was the applied to the unused individuals -> Grammar: The model was applied to... * OTHER * Maybe combine the supplementary figures and write XXX individuals as XXX/YYY individuals, explaining that XXX denotes male and YYY denotes female. The next stage would be to take the profiles of the decision trees forward so that we can identify people who fit the profile but are not diagnosed, and seek additional data about these patients from both medical records (e.g. results of laboratory tests such as elevated CRP) and to incorporate information directly from patients via an online patient reported measures questionnaire to capture more granularity and current symptoms (e.g. duration of pain, stiffness and pain worse in morning, pain improved with movement, buttock pain, hot burning joints, family history of SpA) -> Sounds reasonable! In the discussion, I really liked how you characterized the male and female "warning signs" and proposed actionable steps toward diagnosis in clinical practice. I enjoyed reading the article and hope you use my suggestions to make it the best it can be! Have a good day. ****** 6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #1: No Reviewer #2: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0279076.r001
Revision 1
30 Jan 2022 Author Response Dear Editor, The following is a response to the comments from the reviewers. Reviewer #1: ResponseI feel I have responded to the reviewers points in the manuscript. Reviewer #2: * METHODOLOGY * Novel machine learning methods can potentially identify -> Are any of the ML methods employed truly novel or designed for the purposes of the experiment? If not, remove the word "novel". Response- The machine learning approaches are have been utilised in other industries (manufacturing, etc.) so are novel (new for this area) for this area. However, I have decided to remove this word to avoid confusion. In this study, 10% (n=543/5751) of patients were used to develop the model and the remaining patients with a diagnosis between the ages of 15 to 35 years (n=1559/5751, 27% of total) were used to examine the generalizability of the model. This means 63% of the data was not used in the analysis as these patients were diagnosed with AS before the age of 15 or after the age of 35. -> The 5751 seems to be from the identified male patients. What about the female? Or am I misunderstanding, and this should be made clearer? Response- That section does refer to male patients and I have edited the section to try and clarify the intention for clarity as suggested. This means 63% of the data was not used in the analysis as these patients were diagnosed with AS before the age of 15 or after the age of 35. -> The majority was not used. NICE was mentioned as the reasoning behind this decision, but you should offer a further discussion as to the benefits vs. detriments of effectively discarding this much data. Response- Added to document. Additionally, codes are aggregated so that concepts can be included in the features (e.g. a “pain” code flag may aggregate all the specific pain codes (pain in left thumb and pain in lumbar spine) into one combined “pain present” code -> Unless I am misunderstanding, how is it beneficial to aggregate all the pain responses into 1 feature? I'd think it would make more sense to have a "pain in AS-associated areas" and "pain in other areas". (I think you actually do this as elaborated later, but then this sentence needs to be better worded.) Response- Codes existed in their original state and in aggregated states. So, a ‘pain in back’ code would occur on its own and as part of a ‘pain’ code list. Additionally, the pain code list had a overall code list and was split into locations, e.g. a code for pain in lumbar spine would exist on its own, in a overall pain group and in a back pain group. The clinicians wanted to split the pain codes by locations rather than AS specific as male and female [patients can experience different locations of pain. * MODEL * 79300 tests were run for all 793 AS patients and 100 controls each for the male and female cohorts -> Shouldn't this be 555 (380+175), or were you also using the data set aside for generalizability? If you were, please explain why, because this would seem somewhat problematic. Response-Your observation is correct. Adjusted in document. The model selected is the one with the highest average F value across the 100 runs. -> Why select only 1 model? May there be a benefit to using the previous 100 training models together as a random forest? Response- It was decided to not employ a random forest approach as the output needed to be as understandable as possible to clinicians and patients to improve trust in the results. This list of significant variables was then transformed utilising principal component analysis to better understand the structure and relationship of the features to each other. -> More information on PCA is needed in the manuscript body, e.g. how many principle components were kept after the transform and why was that number selected? Arbitrary? Response- The principle component were all kept and presented to the model, allowing the best significant variables to be selected. * RESULTS AND ANALYSIS * Table 1 -> Validation PPV and F-score are very low (0.15% and 0.30% respectively)? Table 4 -> Validation PPV and F-score are very low (0.25% and 0.50% respectively)? If we make the assumption that a “perfect” model would have 90% sensitivity and specificity as higher 19 percentages than this would likely be attributed to overfitting of the model. -> One could counter overfitting e.g. by making better use of a validation set during hyperparameter tuning/optimization. (But this is a small point that you needn't necessarily do anything with, I'm just saying >90% should be achievable without overfitting.) Response- Thank you for the comment I will keep it in mind for future work. This implies that the female model is more accurate than the male model at identifying patients with AS. -> Could it have something to do with the fact more principle components seemed to be used? Please explain. Response- Added an additional comment. It must be recognised that with low prevalence of AS in the general population, the positive predictive value in general and primary care clinical settings will always be very low. -> I don't think this is necessarily true -- a model with extremely high sensitivity and specificity should be able to achieve a relatively high PPV. However, I acknowledge there are some limitations of your specific study -- and perhaps the nature of your model -- that inhibit your model's ability to get a high PPV. Perhaps this can be better clarified in the manuscript body (but it's not detrimental if it cannot). The results of laboratory tests were not used in this model as they were too complex to interpret in the time available -> Do you mean the project development time or the 3 years of time before diagnosis or something else? This should be specified. Response- This refers to the routine data available. Since the project completed, SAIL has got access to lab results from the hospitals in Wales. However, a large amount of work is required to get this data in a suitable format for this kind of analysis. Thus, 100 Chi-squared tests were run for each of the 3 years preceding the diagnosis date (700 chi-squared tests) -> You should better explain why 700 tests were required (the reasoning behind the number). Response- Tidied up in text. There is some discussion mixed into the results, e.g. "This may imply that there is an increased reluctance to consider a diagnosis of AS or to perform spinal or pelvic x-rays for young female patients compared to their male counterparts" or "suggesting that women have a more complicated path to being diagnosed with AS, with multiple tests, diagnoses, and referrals, compared to men with AS" or "however, women are more likely to be given the test earlier than men. This is likely due to AS symptoms in females being suspected as a result of other conditions, such as RA" or "There was a smaller female cohort for analysis, which could have meant the study was underpowered to detect uveitis as a predictor in women." Please have a separate Discussion section for these and like comments. * WHAT YOU NEED YET INCLUDE * What software package did you use to implement your models? What were your hardware specs? What's the runtime of the model(s) on the system? What is the filesize of the models? Response- The dataset was made employing SQL with data in SAIL. The model was run in R studio and had a runtime of < 24 hours for the complete run. Needs more background and cited works, especially about other machine/deep-learning related AS-diagnostic works regardless of their inputs (e.g. images, features derived from images). Include at least: -> http://dx.doi.org/10.1186/s42358-020-00126-8 -> https://doi.org/10.1016/j.compmedimag.2020.101718 -> https://doi.org/10.1007/s10067-019-04553-x Please provide an ROC curve for your models and report AUC. If possible, provide some statistical analysis on the ROC curve (e.g. bootstrapped confidence intervals). Response- Added to document. * POSSIBLE DISCUSSION POINTS * This is likely due to the general population cohort having incomplete coverage of data compared to the other cohorts. -> For generalizability, what is the likelihood that any given patient with back pain has in their record all the utilized data/stats? What could be done about it if they are missing some of the stats? Response- Many of these individuals were not selected due to failing to meet sufficient criteria for the main analysis. This means that many of these individuals may have incomplete history and relevant information may be missing from there timeline. In future it would be better to perform a validation on data from a different routine data source. * POSSIBLE RESEARCH EXPANSION * What about taking only the most common stats (that would be in the record of most patients) and possibly a shorter history time frame and re-running the experiment? How might results differ? It would be interesting to see and possibly include, but if you feel it is too out of scope consider it for future work. Response- Having a shorter minimum history would increase the population size. * GRAMMAR AND WORD CHOICE * AS is an important inflammatory musculoskeletal condition with effective therapy available, especially if identified early -> Word choice: "important"? Important how so? A better word could be used here to convey the meeting you want, which admittedly was somewhat vague to me. Response- The word “important” has been removed to avoid confusion. especially for women with AS who have not well studied or recognised. -> Grammar and meaning: "especially for women with AS, who traditionally have had later diagnoses possibly due to other factors; this area requires further research." Response- Changed in document. an unspecific back pain code present between age 15 and 20 AND a pelvic pain codes present between age 25 and 30. -> Grammar: between ages X and Y (also elsewhere); a pelvic pain code or pelvic pain codes. Response- Changed in document. who hasan ESR test -> Grammar: has an Response- The word “hasan” has been changed to “has an”. The model was the applied to the unused individuals -> Grammar: The model was applied to... Response- Changed in document. * OTHER * Maybe combine the supplementary figures and write XXX individuals as XXX/YYY individuals, explaining that XXX denotes male and YYY denotes female. Response- I think that even though this would make the supplementary figures more streamlined, it might make the figures more confusing to read. The next stage would be to take the profiles of the decision trees forward so that we can identify people who fit the profile but are not diagnosed, and seek additional data about these patients from both medical records (e.g. results of laboratory tests such as elevated CRP) and to incorporate information directly from patients via an online patient reported measures questionnaire to capture more granularity and current symptoms (e.g. duration of pain, stiffness and pain worse in morning, pain improved with movement, buttock pain, hot burning joints, family history of SpA) -> Sounds reasonable! In the discussion, I really liked how you characterized the male and female "warning signs" and proposed actionable steps toward diagnosis in clinical practice. I enjoyed reading the article and hope you use my suggestions to make it the best it can be! Have a good day. Response- Thank you very much for your comments and suggestions. I feel they have made the document more clear. Yours Faithfully, Dr Jonathan Ian Kennedy. Attachments Attachment Submitted filename: Response to Reviewers.docx https://doi.org/10.1371/journal.pone.0279076.r002
28 Feb 2022 Decision Letter - Alfredo Vellido, Editor PONE-D-20-36921R1Predicting a diagnosis of ankylosing spondylitis using primary care health records – a machine learning approach.PLOS ONE Dear Dr. Kennedy, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Apr 14 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Alfredo Vellido Academic Editor PLOS ONE Journal Requirements: Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. Additional Editor Comments (if provided): Dear authors, The revised version of your paper has improved on the original one, but you still need to provide a response to the fair further requests from reviewer #2. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: No ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: I've looked through the changes and the manuscript has certainly been improved. I can agree with most of the authors' modifications and/or refutations of my suggestions. Nevertheless, there are still a few things I would like clarified or included: 1. I still do not think it is clear why the Validation PPV and F-score are very low, compared to the train and test sets, in Tables 1 and 4. A more detailed description of how you used the validation set (e.g. vs the test set during model development/evaluation) may help. 2. Thanks for the information on model runtime (<24 hours) and development platform (R Studio). Please include these in the manuscript body. Furthermore, please include the system hardware specifications, and try to give a (more) specific runtime for the complete prediction/processing of 1 patient record through the model. 3. I would like to see in the manuscript a (bootstrapped) ROC curve over the final test data; please make and provide one. Since you are using R, the pROC package may be useful (https://www.rdocumentation.org/packages/pROC/versions/1.18.0). I have provided some sample (psuedo)code below to assist with this, where IN_DATA_TRUE and IN_DATA_PREDICT are vectors containing binary class info (e.g. 1 for AS and 0 for non-AS). 4. "This implies that the female model is more accurate than the male model at identifying patients with AS." -> In partial regard to this, you stated, "The principle component were all kept and presented to the model, allowing the best significant variables to be selected," which is a fair response. Could you provide some brief commentary on why you think the feature engineering process created more principle components for females than males, and to what degree (you believe) the number of components has on the final model accuracy? Thanks for submitting your revised article, it was nice to read and exciting to see how it has progressed! --- Sample pROC Code --- library(pROC) roc_obj <- roc(IN_DATA_TRUE, IN_DATA_PREDICT) auc <- auc(roc_obj) aucci <- ci.auc(roc_obj, boot.n=10000) # get 95% confidence interval thresh <- ci.thresholds(roc_obj, boot.n=10000) plot.new() plot.roc(roc_obj, xlab="FPR", ylab="TPR", legacy=TRUE) plot(thresh) dev.off() ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: No [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0279076.r003
Revision 2
21 Jun 2022 Author Response Thank you very much Reviewer 2 for recognising the improvements made between the two versions of this paper, we really appreciate it. Thank you for your further suggestions, we have endeavoured to make the changes or explanations required. 1. I still do not think it is clear why the Validation PPV and F-score are very low, compared to the train and test sets, in Tables 1 and 4. A more detailed description of how you used the validation set (e.g. vs the test set during model development/evaluation) may help. The lower values observed in the validation dataset, in comparison to the training/testing datasets, can be explained in the following ways. • The validation dataset includes records that were deemed not suitable for the main analysis, so those records might not have suitable data coverage and be of lower quality, resulting in incorrect classification. • The models were developed on a 1:1 (case to control) basis and when presented with a less favourable ratio it became harder to achieve good PPV values as there is a much higher ratio of not cases in the general population compared to the training dataset. This is likely why many other models do not work well in the real world. However, the authors feel that the approach undertaken achieved some good hypotheses and results which can better aid the identification of AS. 2. Thanks for the information on model runtime (<24 hours) and development platform (R Studio). Please include these in the manuscript body. Furthermore, please include the system hardware specifications, and try to give a (more) specific runtime for the complete prediction/processing of 1 patient record through the model. Thank you for this feedback, the following section has been added to the paper (Materials and methods section: Dataset). The dataset construction and analysis were undertaken in Eclipse and R Studio in the SAIL remote desktop server; the processor employed was an AMD EPYC 7543 32-Core Processor 2.79 GHz (2 processors) with 16.0 GB of RAM and running Windows 10. The run time for construction of the dataset and running the analysis was < 24 hours, with most of the time expended generating the dataset for the analysis to be undertaken. The processing of one patient took 32 seconds, with much of the time taken in R studio communicating with the SQL server. 3. I would like to see in the manuscript a (bootstrapped) ROC curve over the final test data; please make and provide one. Since you are using R, the pROC package may be useful (https://www.rdocumentation.org/packages/pROC/versions/1.18.0). I have provided some sample (psuedo)code below to assist with this, where IN_DATA_TRUE and IN_DATA_PREDICT are vectors containing binary class info (e.g. 1 for AS and 0 for non-AS). This section requires graphs to answer which are included in the attachment. 4. "This implies that the female model is more accurate than the male model at identifying patients with AS." -> In partial regard to this, you stated, "The principle component were all kept and presented to the model, allowing the best significant variables to be selected," which is a fair response. Could you provide some brief commentary on why you think the feature engineering process created more principle components for females than males, and to what degree (you believe) the number of components has on the final model accuracy? There are more cases of AS present in males than females. AS usually develops in young adults, and male/female have a different healthcare usage at this age. Young men tend to have a much lower usage than females so will likely have a simpler profile in their healthcare record than females. An additional reason might be that the symptoms expressed in a male would be identified as AS quicker, compared to females who might investigate other conditions before AS (e.g. RA, etc.). Attachments Attachment Submitted filename: Response to Reviewers 2.docx https://doi.org/10.1371/journal.pone.0279076.r004
3 Aug 2022 Decision Letter - Alfredo Vellido, Editor PONE-D-20-36921R2 Predicting a diagnosis of ankylosing spondylitis using primary care health records – a machine learning approach. PLOS ONE Dear Dr. Kennedy, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please submit your revised manuscript by Sep 17 2022 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript: A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols. We look forward to receiving your revised manuscript. Kind regards, Alfredo Vellido Academic Editor PLOS ONE Journal Requirements: Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: (No Response) ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: All of my questions have been answered satisfactorily. Thank you very much! The ROC curve and the corresponding ROC AUC must be reported in the manuscript body as part of the results. Furthermore, I believe the explanation about the validation set and the rationale for why there are more principle components for females should be incorporated into the manuscript as part of a discussion; this would make the manuscript stronger and give readers a better understanding of the results and implications. Thereafter, this manuscript should be fit for publication. ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step. https://doi.org/10.1371/journal.pone.0279076.r005
Revision 3
10 Oct 2022 Author Response Response to Reviewers The ROC curves have been incorporated into the results section. The discussion section has been expanded to discuss why the male and female cohorts generate different amounts of principle components, as suggested by the reviewers. The authors would like to thank that comment submitted by the reviewers for recognising the improvements made between the versions of this paper, we really appreciate it. Thank you for your further suggestions, we have endeavoured to make the changes or explanations required. Attachments Attachment Submitted filename: Response to Reviewers 20221010.docx https://doi.org/10.1371/journal.pone.0279076.r006
1 Dec 2022 Decision Letter - Alfredo Vellido, Editor Predicting a diagnosis of ankylosing spondylitis using primary care health records – a machine learning approach. PONE-D-20-36921R3 Dear Dr. Kennedy, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org. If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Alfredo Vellido Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation. Reviewer #2: All comments have been addressed ******** 2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #2: Yes ****** 3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #2: Yes ****** 4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified. Reviewer #2: Yes ****** 5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here. Reviewer #2: Yes ****** 6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters) Reviewer #2: I reviewed the part of the document containing the revision markings (pages 69 to 97 of the full PDF) and my concerns have all largely been addressed. I would just advise a proofread before publication (e.g. "an approach" rather than "a approach" where the original read "a new approach"). Additionally, "Limitations" should probably be its own (sub)section, rather than just "Limitations:" at the start of a paragraph. Congratulations! I hope you all have a nice celebration and enjoy a well-deserved drink. :) ****** 7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy. Reviewer #2: No ******** https://doi.org/10.1371/journal.pone.0279076.r007
Formally Accepted
24 Mar 2023 Acceptance Letter - Alfredo Vellido, Editor PONE-D-20-36921R3 Predicting a diagnosis of ankylosing spondylitis using primary care health records – a machine learning approach. Dear Dr. Kennedy: I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org. If we can help with anything else, please email us at plosone@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Alfredo Vellido Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0279076.r008

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .