Extracting user profile via large language models and ontologies

Pegah Safari; Mehrnoush Shamsfard

doi:10.1371/journal.pone.0329934

Peer Review History

Original SubmissionApril 9, 2025
26 Jun 2025 Decision Letter - Ying Shen, Editor -->PONE-D-25-19130-->-->Extracting User Profile via Large Language Models and Ontologies-->-->PLOS ONE Dear Dr. Shamsfard, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please add in required details mentioned in reviewer comments. Please submit your revised manuscript by Aug 10 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. Please include the following items when submitting your revised manuscript:--> A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. -->If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols . We look forward to receiving your revised manuscript. Kind regards, Ying Shen, Ph.D. Academic Editor PLOS ONE Journal Requirements: -->1. When submitting your revision, we need you to address these additional requirements.-->--> -->-->Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at -->-->https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and -->-->https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf-->--> -->-->2. Thank you for uploading your study's underlying data set. Unfortunately, the repository you have noted in your Data Availability statement does not qualify as an acceptable data repository according to PLOS's standards.-->--> -->-->At this time, please upload the minimal data set necessary to replicate your study's findings to a stable, public repository (such as figshare or Dryad) and provide us with the relevant URLs, DOIs, or accession numbers that may be used to access these data. For a list of recommended repositories and additional information on PLOS standards for data deposition, please see https://journals.plos.org/plosone/s/recommended-repositories.--> 3. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions -->Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. --> Reviewer #1: Yes Reviewer #2: Yes ******** -->2. Has the statistical analysis been performed appropriately and rigorously? --> Reviewer #1: I Don't Know Reviewer #2: Yes ****** -->3. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.--> Reviewer #1: Yes Reviewer #2: Yes ****** -->4. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.--> Reviewer #1: Yes Reviewer #2: Yes ****** -->5. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)--> Reviewer #1: The journal submission "Extracting User Profile via Large Language Models and Ontologies" presents an information extraction work where the authors infer personal information of Persian speakers from their conversations with chatbots. The authors propose a multi-step approach involving prompting, slot-filling and ontology-mapping to find the user profile. Experiments show that the proposed approach performs much better than baseline zero-shot and few-shot prompting methods. The authors should clarify the distinction between single-intent and multi-intent modes. An utterance could convey multiple intents and each intent belongs to a finite set of intents. Therefore, intent detection becomes a multi-label classification task and the classifier should model k separate probabilities, where k = 13. However, the model architecture uses a single sigmoid or softmax layer. I recommend the authors should clarify their model description. While the authors do not need to perform this experiment, I would much appreciate their thoughts on why they did not fully write their prompts in Persian. Currently, only the demonstrations are in Persian but the instructions are in English. I would recommend the authors change the subsection headers of the discussion section by removing the word "ablation". The authors are qualitatively evaluating the results of their models' outputs. This is different from ablation studies which includes evaluating the performance when we suppress some subcomponent in our model. The authors could also try minimizing their discussion sections and only providing the principal insights; the discussion content is too detailed and specific, and probably would not generalize to other datasets. In lines 143-145, the authors say they use separate evaluation and test sets. I think what the authors are referring to as the evaluation set is actually the development or validation set which is used to pick the best model parameters. I recommend using the more conventional name: development or validation set. The authors should also mention the statistical tests they performed to compare performances of different methods. This work makes an important contribution towards modeling in low-resource languages. I believe some minor revisions, as recommended above, could greatly strengthen the manuscript. Reviewer #2: This paper proposes a multi-step method for extracting user profiles from Polish dialogues. Specifically, the approach employs a multi-task learning model combining slot filling and intent detection to extract phrases that reveal user attributes. In parallel, a large language model (LLM) is used in a few-shot in-context learning setup to infer more complex user attributes from the same utterances. Finally, the outputs of both the slot filler and the LLM are integrated into a designed ontology to construct the final user profiles. This approach enables the transformation of unstructured information in free-text into structured concepts, and allows for the detection of potential semantic inconsistencies. The authors further validate the superiority of their method through comparative experiments against strong baselines such as GPT-4o and LLaMA-3-70B. Overall, the paper is well-written and the research is solid. However, I have the following comments and suggestions for improvement: 1. Lines 73–79*: The use of variables x, y, and h* is confusing, as xᵢ, yᵢ, and hᵢ already represent tokens, their labels, and their vector representations, respectively. Please consider using different notations to avoid ambiguity. 2. Section 3.2 – Slot Filling: I recommend including a figure that illustrates the slot filling process described in this section, along with concrete input-output examples to help readers better understand the model workflow. 3. Line 95: Please clarify how the positive and negative samples were constructed. Specifically, for how many target concepts were these samples created? 4. Line 96: Which LLM is used in this process? 5. Line 100 – Step 2: Ontology-based Information Extraction: The construction of the ontology should be described here rather than being deferred to Section 3.4. Additionally, it would be helpful to include a table or diagram illustrating the hierarchical structure of concepts in the ontology. 6. Line 164: The authors mention that "in our prompt, we directly ask about each of our nine desired personal information." Please enumerate what these nine items are. 7. Section 4 – Experimental Results: I suggest including a case study to illustrate the outputs generated by different models. Accompanying content analysis would be beneficial to highlight the strengths and limitations of each model. ******** -->6. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .--> Reviewer #1: Yes: Sabyasachee Baruah Reviewer #2: No ******** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step.--> https://doi.org/10.1371/journal.pone.0329934.r001
Revision 1
10 Jul 2025 Author Response We sincerely thanks the editor and the reviewers for the time and effort spent for reviewing our manuscript We appreciate the constructive comments and suggestions which helped us improve the clarity and the quality of our manuscript. the point-by-point response to each of the reviewers' comments is as follows: Reviewer #1: 1. The authors should clarify the distinction between single-intent and multi-intent modes. An utterance could convey multiple intents and each intent belongs to a finite set of intents. Therefore, intent detection becomes a multi-label classification task and the classifier should model k separate probabilities, where k = 13. However, the model architecture uses a single sigmoid or softmax layer. I recommend the authors should clarify their model description. Answer: Thanks for your observation. After pooling hidden vectors, we apply a fully connected layer as our intent classifier with k=13 outputs. On top of these outputs which represent intent logits, we then apply two separate activations: softmax for single-intent and sigmoid for multi-intent classification. So, we added more clarifying descriptions to the end of the ‘Multitask Learning with Intent Detection’ section (in lines 85–93). 2. While the authors do not need to perform this experiment, I would much appreciate their thoughts on why they did not fully write their prompts in Persian. Currently, only the demonstrations are in Persian but the instructions are in English. Answer: Thank you for your question. We intentionally used English for prompting the LLMs while keeping the context and input utterances in Persian. This approach minimizes the influence of the prompt language on model performance, ensuring that only the input language (Persian) affects the outcome. Prior studies [1, 2] have also shown that prompting multilingual LLMs in English is generally more effective than using languages like Persian. Additionally, translating well-established terms such as "intent" into Persian can introduce ambiguity or translation errors for the model. Using English prompts also improves the comparability of our work, allowing future researchers to reuse the same prompt format by substituting only the input utterances in their target language. We added this description to the article in section 3.3 (in lines 228-232). 3. I would recommend the authors change the subsection headers of the discussion section by removing the word "ablation". The authors are qualitatively evaluating the results of their models' outputs. This is different from ablation studies which includes evaluating the performance when we suppress some subcomponent in our model. The authors could also try minimizing their discussion sections and only providing the principal insights; the discussion content is too detailed and specific, and probably would not generalize to other datasets. Answer: As recommended, we have replaced the term ‘ablation’ (which is more related to section 5) with ‘analysis’ in the subsection titles. Additionally, we revised the discussion sections to present more concise and focused content by highlighting the key points and removing any overly specific details. 4. In lines 143-145, the authors say they use separate evaluation and test sets. I think what the authors are referring to as the evaluation set is actually the development or validation set which is used to pick the best model parameters. I recommend using the more conventional name: development or validation set. Answer: Thanks. Since it may be misinterpreted, we replaced two occurrences of ‘evaluation’ with ‘validation’ in section 3.1. 5. The authors should also mention the statistical tests they performed to compare performances of different methods. Answer: Thank you for your comment. We performed paired t-tests over the five folds to assess the statistical significance of our method against LLMs which results confirm the superiority of our method over both Llama-3-70B (t(4) = 39.10, p < 0.001) and GPT-4o (t(4) = 29.50, p < 0.001). We added these statistical test results in the section 4.1 (in lines 242-244). Reviewer #2: 1. Lines 73–79: The use of variables x, y, and h is confusing, as xᵢ, yᵢ, and hᵢ already represent tokens, their labels, and their vector representations, respectively. Please consider using different notations to avoid ambiguity. Answer: Thanks for your comment. To avoid ambiguity, we use ‘X’ to represent the input sentence, and ‘Y’ and ‘H’ to clearly distinguish between the label vector components (‘yᵢ’) and the hidden state representations (‘hᵢ’), respectively. 2. Section 3.2 – Slot Filling: I recommend including a figure that illustrates the slot filling process described in this section, along with concrete input-output examples to help readers better understand the model workflow. Answer: Thanks for your suggestion. To illustrate the overall flow of input utterance and the system output, we include Figure 5 at the end of section 2 (‘our approach’) that demonstrate suitable example. 3. Line 95: Please clarify how the positive and negative samples were constructed. Specifically, for how many target concepts were these samples created? Answer: These examples are heuristically constructed with a complex sentence structure and inclusion of idioms. We build these examples for two main concepts in the ontology alongside their 25 sub-concepts related to family roles and educational statuses. In order to make this process clearer, we add more description in lines 101-106. 4. Line 96: Which LLM is used in this process? Answer: The LLM model utilized in the process is ‘Llama-3-70B’. To add more clarification, we explicitly mentioned the model’s name in line 109. 5. Line 100 – Step 2: Ontology-based Information Extraction: The construction of the ontology should be described here rather than being deferred to Section 3.4. Additionally, it would be helpful to include a table or diagram illustrating the hierarchical structure of concepts in the ontology. Answer: For better alignment with the presentation of our method, as you recommended, we have moved the description of the ontology structure from Section 3.4 to the section of ‘Step 2: Ontology-based Information Extraction’. Also, following your helpful suggestion, we have included Figure 4 to illustrate a simplified view of the concepts along with a subset of their relationships. It is noteworthy that the ontology is more complicated than its depiction in Figure 4. 6. Line 164: The authors mention that "in our prompt, we directly ask about each of our nine desired personal information." Please enumerate what these nine items are. Answer: These nine items include name, age, gender, marital status, occupation, hobby, residence, number of children, and number of siblings. To improve clarity, we explicitly listed these nine categories of personal information in this section (in lines 220-221). 7. Section 4 – Experimental Results: I suggest including a case study to illustrate the outputs generated by different models. Accompanying content analysis would be beneficial to highlight the strengths and limitations of each model. Answer: Thanks for your comment. In Section 6 (‘Discussion’), particularly in the first two subsections of 6.1, we already have provided a detailed analysis of the results from our proposed approach and the LLM outputs for profile extraction and conflict detection. Also, we have examined the outputs thoroughly and have highlighted the strengths and limitations of the models. However, as you suggested and to further enhance understanding of the models’ outputs, we included Table 13 as the case study at the end of section 6.1. Comment references: 1. Jin Y, Chandra M, Verma G, Hu Y, De Choudhury M, Kumar S. Better to ask in English: Cross-lingual evaluation of large language models for healthcare queries. In: Proceedings of the ACM Web Conference 2024; 2024. p. 2627–2638. 2. Myung J, Lee N, Zhou Y, Jin J, Putri R, Antypas D, et al. Blend: A benchmark for LLMs on everyday knowledge in diverse cultures and languages. Advances in Neural Information Processing Systems. 2024; 37:78104–78146. Also, in response to the journal's requirements mentioned in the letter, we reviewed our paper to ensure compliance with the journal template and submitted our data to the Zenodo repository (with DOI: 10.5281/zenodo.15839326). Attachments Attachment Submitted filename: Reviewer_answer_letter.pdf https://doi.org/10.1371/journal.pone.0329934.r002
24 Jul 2025 Decision Letter - Ying Shen, Editor Extracting User Profile via Large Language Models and Ontologies PONE-D-25-19130R1 Dear Mehrnoush Shamsfard, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. For questions related to billing, please contact billing support . If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Ying Shen, Ph.D. Academic Editor PLOS ONE Additional Editor Comments (optional): Reviewers' comments: Reviewer's Responses to Questions -->Comments to the Author 1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.--> Reviewer #2: (No Response) ******** -->2. Is the manuscript technically sound, and do the data support the conclusions? The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. --> Reviewer #2: (No Response) ****** -->3. Has the statistical analysis been performed appropriately and rigorously? --> Reviewer #2: (No Response) ****** -->4. Have the authors made all data underlying the findings in their manuscript fully available? The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.--> Reviewer #2: (No Response) ****** -->5. Is the manuscript presented in an intelligible fashion and written in standard English? PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.--> Reviewer #2: (No Response) ****** -->6. Review Comments to the Author Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)--> Reviewer #2: (No Response) ****** -->7. PLOS authors have the option to publish the peer review history of their article (what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .--> Reviewer #2: No ******** https://doi.org/10.1371/journal.pone.0329934.r003
Formally Accepted
Acceptance Letter - Ying Shen, Editor PONE-D-25-19130R1 PLOS ONE Dear Dr. Shamsfard, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. You will receive an invoice from PLOS for your publication fee after your manuscript has reached the completed accept phase. If you receive an email requesting payment before acceptance or for any other service, this may be a phishing scheme. Learn how to identify phishing emails and protect your accounts at https://explore.plos.org/phishing. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Ying Shen Academic Editor PLOS ONE https://doi.org/10.1371/journal.pone.0329934.r004

Open letter on the publication of peer review reports

PLOS recognizes the benefits of transparency in the peer review process. Therefore, we enable the publication of all of the content of peer review and author responses alongside final, published articles. Reviewers remain anonymous, unless they choose to reveal their names.

We encourage other journals to join us in this initiative. We hope that our action inspires the community, including researchers, research funders, and research institutions, to recognize the benefits of published peer review reports for all parts of the research system.

Learn more at ASAPbio .