https://orcid.org/0000-0003-0509-4686
Figures
Abstract
Large language models (LLMs) have shown promise in interpreting clinical free-text like provider notes. There is limited evidence on tabular electronic health record (EHR) tasks. Our objective was to evaluate the accuracy of LLMs on structured EHR administrative tasks using direct prompting, chain-of-thought (CoT) reasoning, and tool-enabled code generation. We evaluated nine LLMs randomly sampling from a real-world sampled dataset of 50,000 emergency department (ED) visits. Tasks were tested across 25 combinations of table sizes (5–25 rows and columns). Models were prompted directly or with CoT reasoning to return numerical answers. In the tool setting, models generated Python code, which was executed to retrieve answers. Accuracy was defined as the proportion of model outputs matching validated references. We also assessed JSON format compliance. Across 32,950 model queries, performance varied by model, task type, and prompting strategy. Direct prompting produced uniformly low accuracies. CoT prompting moderately improved performance, particularly for logical filtering, but results degraded significantly as table size increased. The tool-based strategy substantially improved accuracy. Smaller models and distilled reasoning variants had more frequent formatting and execution errors. In conclusion, for structured EHR tabular data extraction, direct and CoT prompting strategies resulted in limited accuracy and poor scalability, particularly as table size increased. Tool-based prompting, where models generated and executed Python code, achieved higher accuracy and valid output formatting. Structured data tasks in clinical workflows may require hybrid approaches that combine LLMs with code execution to ensure accuracy and consistency.
Author summary
Hospitals depend on electronic health records (EHRs) not only for patient care, but also for everyday administrative tasks such as counting visits, tracking admissions, and identifying groups of patients who meet specific criteria. Large language models (LLMs) are increasingly promoted as tools that could help staff answer these questions using plain language instead of database queries or custom scripts. We tested whether this is currently realistic by evaluating nine LLMs on structured EHR tasks using real emergency department data. The models were asked to count records or filter patients based on one or more conditions, either by answering directly, by using step-by-step reasoning, or by generating Python code to perform the task. Direct answers were often inaccurate, and step-by-step prompting improved performance only modestly. Accuracy also dropped as tables became larger. The most reliable results came from a hybrid approach in which the model generated code and the code was executed to retrieve the answer. These findings suggest that current LLMs are not yet dependable as standalone clinical administrators for structured data tasks, and that safer healthcare use will likely require pairing LLMs with conventional computational tools.
Citation: Klang E, Sorin V, Korfiatis P, Sawant AS, Freeman R, Charney AW, et al. (2026) Large language models are poor clinical administrators: An evaluation of structured queries in real-world electronic health records. PLOS Digit Health 5(5): e0001326. https://doi.org/10.1371/journal.pdig.0001326
Editor: Yuzhe Yang, University of California Los Angeles, UNITED STATES OF AMERICA
Received: May 2, 2025; Accepted: March 9, 2026; Published: May 7, 2026
Copyright: © 2026 Klang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The data cannot be shared publicly due to patient privacy and institutional data protection policies. Access to the data may be granted to qualified researchers upon reasonable request and subject to approval by the Mount Sinai Institutional Review Board, which can be contacted at irb@mssm.edu, for those who meet the criteria for access to confidential information.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Electronic health records (EHRs) are essential repositories of patient data used for clinical care, hospital operations, and biomedical research [1,2]. Within healthcare settings, administrators routinely perform tasks such as patient census counts, daily patient visits summaries, equipment inventory tracking, and resource utilization monitoring. These tasks are critical for operational planning, quality assurance, billing, and efficient resource allocation [3,4]. However, generating these reports often relies on data analysts. They use structured query language (SQL) or custom scripting to extract relevant data [5]. This reliance introduces operational delays and limits staff’s ability to access and interpret information directly from EHR systems when needed [6].
Large language models (LLMs) offer a promising pathway to simplify access to EHR data through their ability to interpret user queries in natural language [7,8]. LLMs have previously demonstrated effectiveness across a variety of clinical applications, including clinical decision support, synthetic patient data generation, clinical question-answering, and empathetic patient communication [9–14]. Additionally, they have been explored for workflow-specific tasks, such as shift change summarization, emergency department documentation, and the creation of administrative dashboards [15,16].
Despite these advances, the use of LLMs for structured clinical administrative queries, such as summarizing patient populations during shift changes, generating daily resource utilization reports, or compiling lists of upcoming patient appointments, remains underexplored [17]. These structured tasks require precise logical reasoning, consistent numerical computation, and accurate data filtering, capabilities that many LLMs have yet to reliably demonstrate [18].
Prior research mainly focused on using LLMs to extract structured information from clinical free text, such as pathology or radiology reports [19–21]. Few studies have directly compared multiple LLMs on standardized administrative tasks [22]. To address this gap, we conducted a head-to-head evaluation of nine LLMs, utilizing direct prompting, chain-of-thought (CoT), and a tool-enabled approach (i.e., LLMs generating executable code for data queries) to better understand their out-of-the-box performance on structured EHR administrative tasks. Our goal was to systematically quantify model accuracy, identify critical areas of failure, and provide clear guidance for leveraging LLMs effectively and safely in healthcare administrative workflows.
Methods
Study design and setting
We evaluated nine LLMs on two tabular tasks, counting and logical filtering, in a real-world EHR environment. We also tested a tool-based approach in which LLMs generated Python code to perform the same filtering tasks, allowing us to run the code against a known reference and compare outputs. Further study details can be found in the TRIPOD-LLM checklist [23] (S1 Checklist).
Data source and IRB statement
The Mount Sinai Hospital Institutional Review Board approved the study. We extracted 407,080 emergency department (ED) visits from 2024 via the Mount Sinai Health System’s data warehouse, which encompasses seven hospitals in New York City. From these notes we randomly selected 50,000. Each note included 93 complete entries (See S1 List). Records with missing data were excluded.
LLM models and prompting
We evaluated each model in two ways: an “out-of-the-box” approach and a tool approach (Table 1). The out-of-the-box approach included direct and CoT prompting. We also tested distilled models that emulate CoT internally. In the out-of-the-box approach, we directly provided tables in CSV format and prompted the models to return counts or apply logical filters. In the tool approach, the models were instructed to generate executable Python code. We then executed the code and compared the outputs against the same tables to test accuracy.
“Out-of-the-box” counting and logical filtering.
In the counting task, models were instructed to return the number of rows that met a single condition (S1–S2 Prompts for direct and CoT versions). In the logical filtering task, models were instructed to return the number of rows that met multiple conditions, including one ro more exclusion criteria (S3–S4 Prompts).
Five models were tested with both direct (“simple”) prompts and CoT prompts. The remaining four models, namely DeepSeek reasoning versions, incorporate reasoning internally and were tested using direct prompts only. For both tasks, the input table was included in the prompt in CSV format. The output from each was compared to a ground-truth reference computed via Python.
Tool-enabled approach.
For the tool approach, each model was prompted to produce executable Python code to perform the logical filtering task (S5 Prompt). We used a Python function to parse and execute each LLM-generated code snippet in an isolated environment, compiling the code and running it under controlled globals and locals. We compared the output against the same reference DataFrame to verify the results. The complete list of all experiments and the number of API calls per model is detailed in Table 2.
Dataset construction
We created tables by sampling from the 50,000-row dataset, creating 25 combinations (5, 10, 15, 20, or 25 rows × 5, 10, 15, 20, or 25 columns). For each combination we created 50 unique tables through random sampling. This process was repeated for all nine models, resulting in 25 sets of 50 samples per model. This process resulted in overall 32,950 discrete model queries across various table dimensions, tasks, and prompt combinations. An overview of the study workflow is illustrated in Fig 1.
Schematic depicting the evaluation pipeline for large language models (LLMs) on structured administrative tasks using electronic health record (EHR) data. Input tables varied in volume (row count) and complexity (column count). Nine LLMs were tested using either direct or chain-of-thought (CoT) prompting (“out-of-the-box”) or a tool-enabled strategy that generated executable code. Tasks included counting and logical filtering. Model outputs were evaluated against reference answers for accuracy and formatting compliance.
Sample size and power
A one-way ANOVA power analysis with five groups (α = 0.05, power = 0.80) and a hypothesized 10% difference as a large effect size (Cohen’s f = 0.5) indicated the need for 50 replicates per group (via the FTestAnovaPower function in statsmodels). We therefore repeated each row-column configuration 50 times for each model.
Statistical analysis
The main outcome was accuracy, defined as the proportion of counts or filters matching a validated reference. When Python code was generated, its output was also checked against the same reference. Secondary outcomes included JSON formatting validity and code syntax correctness. We used descriptive statistics to compare performance across row-column dimensions, prompt types, and model classes. Confidence intervals were derived using the Wilson score interval for binomial proportions. Between-group differences in accuracy (proportion correct) were tested using two-sided two-proportion z-tests with pooled standard errors (α = 0.05).
Computational infrastructure
All GPT-based queries were run through Mount Sinai Hospital’s Azure tenant, a HIPAA-compliant environment. Experiments with open-source models were conducted on a local cluster with four H100 GPUs. Default hyperparameters were used for each model. Data processing and analyses were done with Python 3.9.18, using torch, transformers, NumPy, and pandas.
Results
Cohort characteristics
Of the ED visits extracted, most patients were triaged as urgent (emergency severity index [ESI] 3, 58.8%), followed by emergent (ESI 2, 19.7%), and less urgent (ESI 4, 17.5%) (S1 Table). Most patients arrived by personal means (66.2%), or by emergency transport (30.3%) (S2 Table). The majority were discharged following their visit (75.2%), and 16.1% were admitted (S3 Table).
JSON compliance and output quality
Models exhibited substantial variability in their ability to produce valid JSON output across tasks and prompt strategies (Table 3). Llama-3.1-8B demonstrated the highest failure rates, with JSON formatting errors in 96.0% of direct count tasks, 99.6% of direct filter tasks, and 84.0% of tool-based filtering tasks. Due to these consistently high failure rates, the model was excluded from further quantitative analysis. Llama-3.3-70B performed well in both count and filter tasks using direct and CoT prompting (0.0–0.08% failure), but surprisingly failed to return structured JSON in 98.0% of tool-style filtering tasks, indicating difficulty with code-wrapped outputs despite strong overall reasoning capabilities. In contrast, GPT-4o and the Qwen-2.5 models (7B and 72B) exhibited near-perfect JSON compliance across all tasks and prompt strategies, with failure rates of 0.0–0.4%, including under tool-enabled prompting. These models consistently returned well-structured outputs, regardless of the complexity of the task. The DeepSeek-distilled models showed moderate formatting reliability in direct prompting (≤0.5% failure) but consistently struggled in tool settings, where failure rates ranged from 34.0% to 50.0%. These issues stemmed largely from malformed JSON structures or failure to wrap generated code in the correct output schema.
Effectiveness and limitations of prompting strategies
Model performance was positively correlated with size (parameter count), with larger models generally outperforming smaller and distilled variants (Table 4). However, performance varied substantially based on prompting strategies. While CoT generally improved accuracy compared to direct prompting, its effectiveness was notably task-dependent and model-specific.
GPT-4o accuracy improved from 68.6% with direct prompting to 77.6% (p < 0.001) using CoT in counting tasks, and from 49.4% to 77.4% (p < 0.001) in logical filtering tasks. Qwen-2.5-72B similarly benefitted from CoT prompting, showing significant gains from 47.0% to 72.3% (p < 0.001) in counting and 35.2% to 73.1% (p < 0.001) in filtering tasks. However, despite these gains, both direct and CoT prompting still left substantial room for error, especially as dataset size increased, indicating limitations in scalability and computational reasoning capability even among larger models (Figs 2-4).
Mean accuracy of GPT4o, Llama-3.3-70B, Qwen-2.5-72B, and Qwen-2.5-7B models using direct prompting. (A) Mean accuracy across dataset sizes (number of rows); (B) Mean accuracy across complexity levels (number of columns); (C) Mean accuracy across dataset sizes (number of rows) using chain-of-thought (CoT) prompting; (D) Mean accuracy across complexity levels (number of columns) using CoT prompting.
Mean accuracy of GPT4o, Llama-3.3-70B, Qwen-2.5-72B, and Qwen-2.5-7B models using direct prompting. (A) Mean accuracy across dataset sizes (number of rows); (B) Mean accuracy across complexity levels (number of columns); (C) Mean accuracy across dataset sizes (number of rows) using chain-of-thought (CoT) prompting; (D) Mean accuracy across complexity levels (number of columns) using CoT prompting.
Failures of chain-of-thought and direct prompting
Analysis of failures revealed distinct patterns associated with direct and CoT prompting strategies. Direct prompting frequently resulted in lower baseline performance across all models, particularly in complex filtering tasks (Fig 3, panels A-B). Models often failed to capture multi-step logic or consistently apply exclusion criteria correctly. CoT prompting was designed to mitigate such issues by providing explicit intermediate reasoning steps. Nonetheless, CoT strategies were only partially successful, especially when the number of records (rows) increased significantly. As shown in Fig 3 (panels C-D), model accuracy notably degraded when moving from smaller (5–10 rows) to larger datasets (20–25 rows). Even GPT-4o, the highest performing model, exhibited accuracy decreases from ~95% at minimal dataset sizes to below 60% at larger dataset sizes under CoT conditions.
Additionally, CoT prompting was less effective in scenarios requiring significant numerical aggregation or sustained memory of intermediate results across multiple records. This performance degradation was particularly acute for smaller models, such as Qwen-2.5-7B, where accuracy rapidly declined from approximately 80% to below 20% with increasing dataset size (Fig 2).
Impact of dataset size vs. complexity
Distinct effects of dataset size (number of records) versus complexity (number of data fields) were observed (Figs 2-4). Increasing dataset size had a markedly negative impact across all models and task types, reflecting limitations in maintaining accurate numeric aggregation and logical consistency across larger volumes of data. In contrast, increasing data complexity by adding more fields (columns) had a subtler effect, slightly decreasing performance without causing the same magnitude of decline observed with increased dataset size. This suggests that numeric reasoning and aggregation present greater computational challenges than multi-condition logic for current LLMs.
Performance of tool-enabled strategy for filtering task
The tool strategy, in which models were prompted to generate executable Python code to perform the filtering task, demonstrated substantial variation in performance across models. GPT-4o and Qwen-2.5 models exhibited excellent accuracy, achieving nearly perfect performance with minimal JSON compliance errors (2% and 0%, respectively; Table 3, Fig 5). In contrast, distilled DeepSeek variants struggled significantly under this approach, demonstrating markedly lower accuracy accompanied by high JSON compliance failure rates ranging from 34% to 50%. The primary source of these errors involved syntactic mistakes or incorrect JSON formatting that rendered code non-executable, underscoring a critical limitation in their capacity to reliably operate as independent agents for structured data extraction tasks.
Mean accuracy DeepSeek-R1-Distill-Llama-70B, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Qwen-7B models using direct prompting. (A) Mean accuracy across dataset sizes (number of rows) for the Counting Task among distilled models; (B) Mean accuracy across complexity levels (number of columns) for the Counting Task; (C) Mean accuracy across dataset sizes (number of rows) for the Filtering Task; (D) Mean accuracy across complexity levels (number of columns) for the Filtering Task.
Mean accuracy (+/- standard error) for models Qwen-2.5-72B, Qwen-2.5-7B, GPT4o, DeepSeek-R1-Distill-Llama-70B, DeepSeek-R1-Distill-Qwen-7B, and DeepSeek-R1-Distill-Llama-8B.
Comparative evaluation of direct, chain-of-thought, and tool strategies
When comparing across the three strategies (direct, CoT, and tool-based) for the filtering task, the tool-based approach generally showed superior performance, particularly among the largest and most advanced models (GPT-4o, Qwen-2.5-72B). Direct prompting consistently produced the lowest accuracies, especially with increasing dataset complexity and size (Fig 3), while the CoT strategy provided modest but significant improvements for tasks requiring complex logical reasoning or multi-step data filtering (Fig 3C and 3D). Reasoning models, designed to internally emulate chain-of-thought without explicit prompting, failed to match the performance of models explicitly guided through CoT or tool-enabled strategies.
Discussion
This study demonstrates that current LLMs, when used directly for structured administrative queries in EHR, consistently exhibit substantial limitations, even with small datasets (e.g., ≤ 25 rows). Accuracy and format compliance varied significantly across models, tasks, and prompting strategies. Notably, even the highest-performing models such as GPT-4o struggled to achieve acceptable accuracy levels when relying on direct or CoT prompts alone, particularly as query complexity or dataset size increased.
LLMs have been previously reported to struggle with tabular data. Previous efforts for LLMs’ tabular data analysis in healthcare mainly focused on data extraction, including benchmarks such as EHRSQL [24,25] and MIMICSQL [26,27]. Our study systematically analyzes how nine different models analyze tabular data under stress conditions with increasing data complexity, and under various prompting condition, a gap that was yet to be addressed in the literature. Some commercial tools, such as Google’s Healthcare Data Search, now incorporate LLMs for structured data retrieval [28]. Yet, there has been little evidence on how LLMs perform when applied directly to structured healthcare data, particularly with larger datasets, complex logic, or strict formatting requirements.
In our study, direct prompting strategies resulted in uniformly low accuracies across all models for both counting and filtering tasks, highlighting the insufficiency of straightforward natural language instructions for structured data extraction tasks. CoT prompting yielded moderate improvements, particularly in scenarios involving logical filtering tasks; however, these gains were inconsistent and insufficient for practical, reliable clinical use. Distilled reasoning models, intended to internally emulate CoT processes, similarly underperformed, indicating that implicit reasoning alone does not guarantee performance gains.
The tool-enabled approach, where LLMs generated executable Python code to perform logical filtering, emerged as the only viable strategy, markedly improving accuracy and reliability in advanced models (GPT-4o and Qwen-2.5-72B). This finding aligns with results from Shi et al. publication on EHRAgent [29], which showed that code-generating agents outperform both direct and CoT-based approaches. This highlights that current LLMs can effectively interpret the intent behind structured queries, but still struggle with reliable, basic data pulls without help in this form. Across all tasks, format compliance (valid JSON output) was problematic, especially among smaller or distilled models, severely limiting their practical integration into automated clinical workflows. The widespread inability of models to consistently provide structured and syntactically correct outputs, even with explicit formatting instructions, underscores a major deployment barrier.
Our findings indicate that without explicit computational delegation (i.e., a tool-based strategy), current LLMs are unsuitable for standalone use even on minimally complex administrative tasks in clinical settings. Their frequent errors, inability to express uncertainty, and formatting inconsistencies introduce substantial risks for clinical and administrative operations, particularly in high-stakes scenarios involving billing, compliance, or resource allocation [30,31].
This study has its limitations. Models did not directly access databases or receive iterative feedback, potentially restricting generalizability and requiring further operational efforts. Additionally, we did not assess runtime performance or cost-efficiency, factors crucial for practical deployment. Future studies could potentially address these gaps through fine-tuning models specifically for structured EHR tasks, implementing confidence or verification systems, and evaluating hybrid workflows in real-world clinical contexts.
In conclusion, despite their potential in natural language interpretation, current LLMs require integration with explicit computational tools (such as code-generation approaches) to reliably execute even basic structured queries against healthcare data. Hybrid frameworks combining LLMs with traditional database or coding strategies, like the Model Context Protocol (MCP) [32], are currently required to achieve practical utility, accuracy, and safety in clinical administration tasks.
Supporting information
S1 List. EdEncounterFact Table Columns Names.
https://doi.org/10.1371/journal.pdig.0001326.s002
(DOCX)
S1 Table. Emergency Severity Index (ESI) Acuity Level Distribution.
https://doi.org/10.1371/journal.pdig.0001326.s003
(DOCX)
S2 Table. Patient Arrival Method Distribution.
https://doi.org/10.1371/journal.pdig.0001326.s004
(DOCX)
S3 Table. Emergency Department Disposition Distribution.
https://doi.org/10.1371/journal.pdig.0001326.s005
(DOCX)
S2 Prompt. Counting Task (Chain-of-Thought Prompt).
https://doi.org/10.1371/journal.pdig.0001326.s007
(DOCX)
S3 Prompt. Logical Filtering Task (Direct Prompt).
https://doi.org/10.1371/journal.pdig.0001326.s008
(DOCX)
S4 Prompt. Logical Filtering Task (Chain-of-Thought Prompt).
https://doi.org/10.1371/journal.pdig.0001326.s009
(DOCX)
S5 Prompt. Logical Filter Task (Agentic Strategy).
https://doi.org/10.1371/journal.pdig.0001326.s010
(DOCX)
References
- 1. Uslu A, Stausberg J. Value of the Electronic Medical Record for Hospital Care: Update From the Literature. J Med Internet Res. 2021;23(12):e26323. pmid:34941544
- 2. Coorevits P, Sundgren M, Klein GO, Bahr A, Claerhout B, Daniel C, et al. Electronic health records: new opportunities for clinical research. J Intern Med. 2013;274(6):547–60. pmid:23952476
- 3. Bhati D, Deogade MS, Kanyal D. Improving patient outcomes through effective hospital administration: a comprehensive review. Cureus. 2023;15(10):e47731.
- 4. Akinleye DD, McNutt L-A, Lazariu V, McLaughlin CC. Correlation between hospital finances and quality and safety of patient care. PLoS One. 2019;14(8):e0219124. pmid:31419227
- 5. Aguirre RR, Suarez O, Fuentes M, Sanchez-Gonzalez MA. Electronic Health Record Implementation: A Review of Resources and Tools. Cureus. 2019;11(9):e5649. pmid:31700751
- 6. Richman BD, Kaplan RS, Kohli J, Purcell D, Shah M, Bonfrer I, et al. Billing and insurance–related administrative costs: a cross-national analysis. Health Affairs. 2022;41(8):1098–106.
- 7. Van Veen D, Van Uden C, Blankemeier L, Delbrouck J-B, Aali A, Bluethgen C, et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat Med. 2024;30(4):1134–42. pmid:38413730
- 8. Vrdoljak J, Boban Z, Vilović M, Kumrić M, Božić J. A Review of Large Language Models in Medical Education, Clinical Decision Support, and Healthcare Administration. Healthcare (Basel). 2025;13(6):603. pmid:40150453
- 9. Goodman RS, Patrinely JR, Stone CA Jr, Zimmerman E, Donald RR, Chang SS, et al. Accuracy and Reliability of Chatbot Responses to Physician Questions. JAMA Netw Open. 2023;6(10):e2336483. pmid:37782499
- 10.
Agarwal U, Tanmay K, Khandelwal A, Choudhury M. Ethical reasoning and moral value alignment of LLMs depend on the language we prompt them in. arXiv preprint arXiv:240418460. 2024.
- 11. Sorin V, Brin D, Barash Y, Konen E, Charney A, Nadkarni G. Large Language Models and Empathy: Systematic Review. J Med Internet Res. 2024;26:e52597.
- 12. Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023;183(6):589–96. pmid:37115527
- 13. Chen S, Guevara M, Moningi S, Hoebers F, Elhalawani H, Kann BH, et al. The effect of using a large language model to respond to patient messages. Lancet Digit Health. 2024;6(6):e379–81. pmid:38664108
- 14. Wiest IC, Ferber D, Zhu J, van Treeck M, Meyer SK, Juglan R, et al. From Text to Tables: A Local Privacy Preserving Large Language Model for Structured Information Retrieval from Medical Documents. medRxiv. 2023;2023.12.07.23299648.
- 15. Hartman V, Zhang X, Poddar R, McCarty M, Fortenko A, Sholle E. Developing and evaluating large language model–generated emergency medicine handoff notes. JAMA Network Open. 2024;7(12):e2448723.
- 16. Glicksberg BS, Timsina P, Patel D, Sawant A, Vaid A, Raut G, et al. Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room. J Am Med Inform Assoc. 2024;31(9):1921–8. pmid:38771093
- 17.
Gebreab SA, Salah K, Jayaraman R, Rehman MHu, Ellaham S. LLM-based framework for administrative task automation in healthcare. In: 2024.
- 18. Soroush A, Glicksberg BS, Zimlichman E, Barash Y, Freeman R, Charney AW. Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying. NEJM AI. 2024;1(5):AIdbp2300040.
- 19. Huang J, Yang DM, Rong R, Nezafati K, Treager C, Chi Z, et al. A critical assessment of using ChatGPT for extracting structured data from clinical notes. NPJ Digit Med. 2024;7(1):106. pmid:38693429
- 20. Wiest IC, Ferber D, Zhu J, van Treeck M, Meyer SK, Juglan R, et al. Privacy-preserving large language models for structured medical information retrieval. NPJ Digit Med. 2024;7(1):257. pmid:39304709
- 21.
Belem CG, Pezeskhpour P, Iso H, Maekawa S, Bhutani N, Hruschka E. From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization. arXiv preprint arXiv:241013961. 2024.
- 22. Chen N, Shou L, Gong M, Pei J, You C, Chang J, et al. Bridge the Gap between Language models and Tabular Understanding. arXiv. 2023;abs/2302.09302.
- 23. Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60–9. pmid:39779929
- 24. Lee G, Hwang H, Bae S, Kwon Y, Shin W, Yang S, et al. Ehrsql: A practical text-to-sql benchmark for electronic health records. Adv Neural Inform Process Syst. 2022;35:15589–601.
- 25.
Lee G, Kweon S, Bae S, Choi E. Overview of the ehrsql 2024 shared task on reliable text-to-sql modeling on electronic health records. arXiv preprint arXiv:240506673. 2024.
- 26.
Wang P, Shi T, Reddy CK. Text-to-SQL Generation for Question Answering on Electronic Medical Records. In: Proceedings of The Web Conference 2020. 2020. p. 350–61.
- 27.
Lovon J, Mouysset M, Oleiwan J, Moreno JG, Damase-Michel C, Tamine L. Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval. arXiv preprint arXiv:250109384. 2025.
- 28. Get search results for healthcare data. https://cloud.google.com/generative-ai-app-builder/docs/search-hc-data
- 29.
Shi W, Xu R, Zhuang Y, Yu Y, Zhang J, Wu H, et al. EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Records. In: Proc Conf Empir Methods Nat Lang Process. 2024. p. 22315–39.
- 30. Tripathi S, Sukumaran R, Cook TS. Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care. J Am Med Inform Assoc. 2024;31(6):1436–40. pmid:38273739
- 31. Omar M, Soffer S, Agbareia R, Bragazzi NL, Apakama DU, Horowitz CR, et al. Sociodemographic biases in medical decision making by large language models. Nat Med. 2025;31(6):1873–81. pmid:40195448
- 32. Al Attrach R, Moreira P, Fani R, Umeton R, Celi LA. Conversational LLMs Simplify Secure Clinical Data Access, Understanding, and Analysis. arXiv. 2025;2507.01053.