https://orcid.org/0009-0001-4839-5182
Figures
Abstract
Objective
To evaluate the reliability and diagnostic performance of ChatGPT-o3 in conducting Risk of Bias (RoB) assessments of randomized clinical trials (RCTs) using the Cochrane RoB 2.0 tool.
Materials and methods
This methodological validation study analyzed 50 RCTs sampled from 50 published meta-analyses. Each trial was independently assessed by the original systematic review authors (OSRAs), our masked human panel, and ChatGPT-o3. Structured prompts based on RoB 2.0 guidelines were used to elicit ChatGPT-o3 assessments. Agreement was evaluated using weighted Cohen’s kappa and Gwet’s AC2. Diagnostic performance was measured by sensitivity, specificity, and balanced accuracy, with human ratings as the reference.
Results
ChatGPT-o3 classified 34% of trials as high risk, compared with 22% by our panel, and 12% by the OSRAs. Agreement was modest (median κ: 0.33 with our panel; 0.14 with OSRAs). Overall Gwet’s AC2 was 0.30. For detecting high-risk trials, ChatGPT-o3 achieved a sensitivity of 0.46, specificity of 0.69, and balanced accuracy of 0.57. For low-risk trials, its sensitivity was 0.47, specificity was 0.86, and balanced accuracy was 0.66.
Citation: Gandhi S, Shokravi A, Chelliahpillai Y, Balas M (2026) Evaluating large language model performance in Risk of Bias assessments: A cross-sectional validation study. PLoS One 21(7): e0353155. https://doi.org/10.1371/journal.pone.0353155
Editor: Satyaki Roy, The University of Alabama in Huntsville, UNITED STATES OF AMERICA
Received: September 28, 2025; Accepted: June 18, 2026; Published: July 9, 2026
Copyright: © 2026 Gandhi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting information files.
Funding: The author(s) received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Accurate Risk of Bias (RoB) assessments in randomized clinical trials (RCTs) are essential for producing high-quality systematic reviews, which directly inform clinical guidelines and healthcare decisions [1–3]. Although widely accepted, tools like the Cochrane RoB 2 tool remain labor-intensive and subject to variability between reviewers, posing challenges as the volume of biomedical literature grows rapidly [4,5].
Large language models (LLMs), including ChatGPT, have emerged as potential tools to automate critical components of evidence synthesis, such as RoB assessment [6–8]. Previous evaluations utilizing simpler, checklist-based instruments have reported substantial agreement between LLMs and human reviewers. For instance, Lai et al. (2024) demonstrated that LLMs achieved accuracy rates exceeding 85% when assessing bias using a modified version of the original Cochrane tool (the CLARITY instrument), which relies on explicit text extraction [9]. Similarly, Xia et al. (2025) reported high performance when applying LLMs to cohort studies using the Newcastle-Ottawa Scale, suggesting that AI models perform well with explicit criteria [10]. However, the transition to complex, domain-based tools like RoB 2.0 has challenged AI reasoning capabilities. Šuster et al. (2024) found that zero-shot prompting of LLMs on RoB 2.0 yielded poor performance, while Huang et al. (2025) noted that accuracy for domain-level judgments dropped to between 57% and 70% [11,12] This struggle extends to non-randomized studies as well; Hasan et al. recently reported only moderate agreement (61%) between GPT-4 and humans using the ROBINS-I tool [13]. Ultimately, early research suggests that these models can achieve good agreement with expert assessments, indicating their promise for enhancing efficiency and consistency in systematic reviews [9,12,14,15]. However, existing evidence is limited by small, narrowly selected datasets and reliance on older model versions [9,16]. There is also a need to further assess these LLMs with the latest RoB tools.
The present study evaluates the performance of a recent, advanced LLM (ChatGPT-o3) for RoB assessments across a broad and diverse sample of 50 randomly selected RCTs from various medical specialties [17]. We aimed to quantify the agreement between LLM-generated assessments and blinded human expert ratings, providing essential evidence regarding the potential of LLMs as reliable adjunctive tools for systematic review methodologies.
By clarifying the capabilities and limitations of state-of-the-art LLMs in conducting RoB evaluations, this study seeks to inform future directions in automated evidence synthesis. These advancements could streamline the systematic review process and help ensure faster, more reproducible, and methodologically rigorous integration of clinical evidence into practice.
Methods
Study design and search strategy
This methodological validation study aimed to quantify the reliability and accuracy of an advanced LLM (ChatGPT-o3; OpenAI, San Francisco, CA, USA) in performing RoB assessments for RCTs [18]. The reporting of this study follows the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis-Large Language Models (TRIPOD-LLM) guidelines [19]. We conducted a systematic search on PubMed using the query: “clinical trial*” AND “meta analysis” AND “Risk of Bias 2”. This specific search strategy was used to identify studies that employed the revised Cochrane RoB 2.0 tool, as it is the most widely adopted instrument for assessing RoB in RCTs [20] A formal study protocol was not pre-registered for this methodological validation. The search yielded meta-analyses published between 2020 and 2025. From the resulting records, we randomly selected 50 meta-analyses to ensure a diverse sample for interrater agreement analysis. Each meta-analysis was manually screened to ensure that complete RoB assessments were included. Subsequently, from each meta-analysis, we randomly selected one RCT using a random number generator based on the total number of RCTs assessed in that review, resulting in a sample of 50 RCTs across diverse medical subspecialties (SMethods 1 in S1 File). The publication dates of the 50 randomly selected RCTs ranged from 1998 to 2025. Because no human participants or data were involved, institutional review board approval was not required. The study adhered to the tenets of the Declaration of Helsinki.
Data collection and RoB assessment
For each included RCT, we extracted the RoB ratings reported by the original systematic review authors (OSRAs). Separately, three human reviewers (S.G., A.S., and Y.C.), masked to the OSRA’s assessments, independently conducted their own RoB assessments. To ensure a comprehensive evaluation, particularly for selective reporting (Domain 5), our human panel retrieved trial registry entries (e.g., ClinicalTrials.gov) whenever available. Similarly, ChatGPT-o3 assessments were conducted with web-browsing capabilities enabled, allowing the model to theoretically search for and cross-reference external registry data if referenced within the manuscript. Discrepancies between human reviewers were resolved by consensus discussion, or by a fourth author (M.B.), if necessary. Finally, ChatGPT-o3 was used to independently generate RoB assessments for each RCT while being masked to the ratings from our panel and the OSRAs. The ChatGPT-o3 assessments were generated using a structured RoB prompt that was developed by integrating the Cochrane RoB 2.0 guidelines with a detailed template from Lai et al. (2024) (SMethods 2 in S1 File) [9,21]. We accessed the model via the standard web interface (chatgpt.com) between April 20, 2025, and May 20, 2025, utilizing the “ChatGPT-o3” model version. As the web interface does not permit manipulation of generation parameters (e.g., temperature, top_p), default settings were applied. To ensure reproducibility and prevent context accumulation (data leakage between trials), each assessment was performed in a new, independent chat session (single-shot approach). For each trial, the complete PDF of the article was uploaded along with the prompt, allowing the model to parse the full text and tables directly. This prompt explicitly instructs the model to base all judgments strictly on reported information without extrapolation or speculation, following the signaling questions and branching logic found in the RoB 2.0 cribsheet [22].
Statistical analysis
RoB ratings were compared across three sources: the OSRAs, our manual masked assessment, and the assessments generated by ChatGPT-o3. All ratings were made using the Cochrane RoB 2.0 tool across five individual domains: (1) bias arising from the randomization process, (2) bias due to deviations from intended interventions, (3) bias due to missing outcome data, (4) bias in measurement of the outcome, and (5) bias in selection of the reported result; as well as an overall judgment. Each domain was rated on an ordinal scale with three categories: Low risk, Some concerns, or High risk.
Pairwise interrater agreement was assessed using linearly weighted Cohen κ statistics. Cohen’s κ values were interpreted using standard thresholds: < 0.00 (poor), 0.00–0.20 (slight), 0.21–0.40 (fair), 0.41–0.60 (moderate), 0.61–0.80 (substantial), and 0.81–1.00 (almost perfect) agreement [23]. Agreement was calculated within our panel and between our panel, OSRAs, and ChatGPT-o3 for each domain as well as the overall judgement. 95% confidence intervals (CIs) were estimated using percentile bootstrap methods with 1000 resamples. To assess global agreement among all three raters, Gwet’s AC2 was computed. This statistic was chosen over Fleiss’ κ due to its greater stability under conditions of uneven category distribution (the prevalence paradox). In RoB assessments, where a majority of studies may fall into a single category (e.g., “Low Risk”), Kappa often drastically underestimates agreement; Gwet’s AC2 corrects for this by adjusting for chance agreement based on the marginal probabilities of the ratings [24,25]. We also calculated exact percent agreement between each rater pair across domains to provide an intuitive summary of rating concordance.
To examine the direction and magnitude of disagreement, ordinal scores were assigned to ratings (0 = Low risk, 1 = Some concerns, 2 = High risk). Mean signed differences were calculated for each domain and rater pair. Symmetry of disagreement patterns was tested using Bowker’s test for paired ordinal data. To further characterize disagreement severity, we also quantified “extreme disagreements” between raters, defined as instances where one rater assigned a “High risk” and another a “Low risk” rating for the same study within the same domain.
To evaluate diagnostic performance, ratings were dichotomized into High Risk versus Not High Risk. Sensitivity, specificity, and balanced accuracy of ChatGPT-o3 were calculated using our masked human assessment as the reference standard. We selected our panel as the reference because their ratings were generated specifically for this study under blinded conditions, effectively mitigating the potential optimism bias or conflicts of interest that may influence original systematic review authors. However, we acknowledge that RoB assessments possess inherent subjectivity; therefore, this reference represents a rigorous expert consensus rather than an indisputable objective ground truth. A secondary analysis dichotomized ratings as Low Risk versus Not Low Risk. Pairwise McNemar’s tests were also conducted to assess differences in binary classification rates between raters.
All analyses and data visualizations were conducted using R version 4.4.1 (R Foundation for Statistical Computing, Vienna, Austria). To control for family-wise error rates, all reported p-values were Bonferroni-adjusted for the 18 distinct hypotheses tested (3 rater pairs × 6 domains), with a two-sided significance threshold of 0.05.
Results
A total of 50 RCTs underwent independent RoB assessment using the Cochrane RoB 2.0 tool across five individual domains and an overall RoB judgment. ChatGPT-o3, our masked human panel, and OSRAs independently provided ratings, yielding 300 complete assessments for analysis (Table 1). All full LLM rating responses can be found in SMethods 3 in S1 File.
Distribution of high-risk judgments and extreme disagreement patterns
The distribution of RoB ratings varied notably across raters (Fig 1). ChatGPT-o3 classified 34% of trials (17 of 50) as high risk, compared with 22% (11 of 50) by our panel and 12% (6 of 50) by the OSRAs. ChatGPT-o3 consistently assigned high-risk ratings more frequently than our panel across most domains, with the largest differences observed in domain 1 (18% vs 6%), domain 3 (18% vs 10%), domain 4 (12% vs 8%), and domain 5 (6% vs 2%). However, none of these differences were significant (all p > 0.05). We also assessed extreme disagreements, defined as one rater assigning a “High risk” and another a “Low risk” rating for the same study. Between ChatGPT-o3 and our panel, these occurred in 4 cases (8%), split evenly in both directions. In contrast, extreme discrepancies between ChatGPT-o3 and OSRAs were more unidirectional, with ChatGPT-o3 rating 7 trials (14%) as “High risk” that OSRAs had judged “Low risk”; the reverse did not occur. Between OSRAs and our panel, extreme discrepancies were likewise unidirectional, with our panel rating 3 trials (6%) as “High risk” that OSRAs had judged “Low risk”; the reverse did not occur.
Stacked bar plots showing the distribution of Risk of Bias 2.0 ratings assigned by ChatGPT-o3 (LLM), our panel, and the original systematic review authors (OSRAs) across each risk of bias domain (Domain 1 to Domain 5) and overall. Ratings are categorized as low risk (green), some concerns (yellow), or high risk (red).
Interrater agreement
Pairwise agreement, assessed using linear-weighted Cohen κ, demonstrated limited consistency overall (Table 2). We conducted 18 pairwise comparisons by evaluating agreement between each pair of raters (ChatGPT-o3, our masked panel, and the OSRAs) across the five individual RoB domains and the overall rating. Of these 18 comparisons, 10 (56%) reflected fair or moderate agreement, while 6 (33%) showed slight and 1 (6%) poor agreement, highlighting substantial variability in concordance. Median κ values across all domains were highest between ChatGPT-o3 and our panel (0.33), followed by the OSRAs and our panel (0.28), and lowest between ChatGPT-o3 and OSRAs (0.14). For the overall RoB judgment, Fleiss’s κ across all three raters was 0.16. The strongest individual agreement occurred between the OSRAs and our panel in domain 1 (κ = 0.63; 95% CI, 0.31–0.83), while the weakest occurred between ChatGPT-o3 and the OSRAs in domain 2 (κ = –0.10; 95% CI, –0.25 to 0.08), suggesting near-random agreement in that domain.
Exact percent agreement also varied widely, ranging from 36% for overall RoB ratings between ChatGPT-o3 and OSRAs to 82% for domains 1 and 3 between OSRAs and our panel. For the overall RoB judgment specifically, agreement was 54% between OSRAs and our panel, 46% between ChatGPT-o3 and our panel, and 36% between ChatGPT-o3 and OSRAs. A domain-level heatmap of exact agreement is provided in SFigure 1 in S1 File. To better understand internal consistency within our reference group, we conducted an additional analysis of interrater reliability among members of our masked panel. Agreement varied across domains, with percent agreement ranging from 60% to 90%, and linear-weighted κ values ranging from 0.21 (fair agreement) to 0.74 (substantial agreement). For the overall RoB judgment, percent agreement was 70%, with a linear-weighted κ of 0.56, reflecting moderate internal agreement. Full results are provided in STable 1 in S1 File.
Global agreement across all three raters, as assessed using Gwet’s AC2, was higher for individual domains (range, 0.64–0.79) but lower for the overall rating (AC2 = 0.30) (STable 2 in S1 File). Pairwise AC2 values followed a similar pattern: overall agreement was 0.33 (ChatGPT-o3 vs our panel), 0.15 (ChatGPT-o3 vs OSRA), and 0.46 (OSRA vs our panel), with domain-level pairwise AC2 values ranging from 0.56 to 0.87 (STable 3 in S1 File). In the within-panel analysis, AC2 values were higher across domains (range, 0.66–0.93), and agreement remained strong overall at 0.71. (STable 4 in S1 File).
Directionality of rating differences
Analysis of disagreement patterns revealed systematic differences in rating tendencies. To quantify directional differences, categorical RoB judgments were converted to ordinal scores (0 for low risk, 1 for some concerns, and 2 for high risk). ChatGPT-o3 consistently applied more severe ratings than our panel, with a positive mean difference of 0.18 for overall RoB assessments (p > 0.05, STable 5 in S1 File). In contrast, OSRAs assigned lower-risk ratings relative to our panel (mean difference, –0.36, p < 0.01), with an even larger discrepancy observed between OSRAs and ChatGPT-o3 (mean difference, –0.54, p < 0.001). Bowker symmetry tests identified significant asymmetry in domain 1 ratings between OSRAs and our panel (p < 0.05; STable 6 in S1 File), whereas all other comparisons across domains and rater pairs were not statistically significant (p > 0.05).
Diagnostic accuracy relative to panel ratings
Using our masked panel ratings as the reference standard, ChatGPT-o3 demonstrated moderate sensitivity (0.46) and specificity (0.69) for identifying high-risk studies, resulting in a balanced accuracy of 0.57 (Table 3). The OSRAs had the same sensitivity (0.46) but substantially higher specificity (0.97), yielding a higher balanced accuracy of 0.71. When identifying low-risk studies (Table 4), ChatGPT-o3 achieved a balanced accuracy of 0.66 (sensitivity, 0.47; specificity, 0.86), while OSRAs demonstrated similar overall performance (balanced accuracy, 0.67), with higher sensitivity (0.80) but lower specificity (0.54).
Discussion
This study evaluated the reliability and diagnostic performance of ChatGPT-o3 in assessing RoB in RCTs using the RoB 2.0 tool. While the model demonstrated promising consistency and potential as a screening tool, systematic deviations from human reviewers limit its readiness for standalone use. Our findings diverge from earlier studies, such as Lai et al. (2024), who reported high accuracy (>80%) using a modified version of the simpler Cochrane RoB 1.0 tool [9]. In contrast, evaluations utilizing the complex, domain-based RoB 2.0 tool have yielded significantly lower performance, highlighting a reasoning gap in current LLMs. For instance, Šuster et al. (2024) reported F1 scores as low as 0.10–0.20 for zero-shot RoB 2.0 predictions, concluding that the task complexity currently exceeds off-the-shelf model capabilities [11]. Similarly, Huang et al. (2025) found that while LLMs could identify signaling questions with reasonable accuracy, their ability to derive correct domain-level judgments dropped to 57.5%, which is a figure strikingly similar to our balanced accuracy of 0.57 [12]. This difficulty in synthesizing methodological nuances is further corroborated by Hasan et al., who reported only 61% agreement for the similarly complex ROBINS-I tool [13]. Collectively, these data confirm that while LLMs excel at information extraction, they struggle with the algorithmic logic required for advanced bias assessment tools. ChatGPT-o3 consistently assigned higher RoB ratings than both our masked panel and the OSRAs, likely reflecting a more conservative or literal interpretation of signaling criteria. This tendency persisted even though the model had web-browsing capabilities enabled. This strongly suggests that our prompting strategy may have been the primary driver of these conservative ratings: by explicitly instructing the model to assess strictly based on reported information without extrapolation, we likely inhibited it from utilizing its browsing tool to verify external protocols. Consequently, the model penalized incomplete reporting, such as assigning “Some concerns” or “High risk” where human reviewers (who accessed registries) confirmed low risk. This stricter approach may help identify subtle or underreported sources of bias, reducing false negatives, but it risks overclassifying studies as high risk. Consequently, this could lower the certainty of evidence in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) assessments and negatively impact clinical recommendations. In contrast, OSRAs tended toward more lenient ratings, possibly reflecting implicit incentives to classify studies as lower risk, thereby increasing overall certainty and strength of conclusions. This underscores the value of independent reviews to counterbalance bias.
The divergence between ChatGPT-o3 and human raters underscores the potential value of LLMs as adjunct reviewers. By identifying inconsistencies and offsetting overly permissive human ratings, LLMs can support authors or journal editors. Specifically, it may be possible to have a “human-in-the-loop” workflow where the LLM functions as an adjunctive, conservative screener to flag potential risks. While the model’s moderate sensitivity (0.46) indicates it should not be relied upon to catch every high-risk trial independently, its tendency toward strict criteria results in a high volume of flagged studies. In this capacity, reviewers must accept a higher rate of false positives that human experts can then adjudicate. This approach leverages the model’s rigid adherence to signaling questions to highlight potential methodological flaws, thereby allowing human reviewers to focus their cognitive effort on verifying flagged domains rather than performing entirely de novo extraction. A hybrid model combining automated screening with expert oversight could offer a transparent and scalable approach, particularly in high-volume evidence synthesis contexts.
Overall interrater reliability, as measured by Gwet’s AC2, was modest, reflecting both the inherent subjectivity of RoB assessments and limitations in LLMs replicating nuanced judgment. Notably, agreement within our own panel varied substantially across domains, highlighting that discrepancies between ChatGPT-o3 and human raters may stem more from structural features of the RoB tool rather than individual reviewer performance. Agreement was notably weakest for Domain 2 (deviations from intended interventions), which yielded near-random concordance. This finding is consistent with landmark validation studies of RoB 2.0; for instance, Minozzi et al. (2020) reported an interrater reliability of only 0.04 for this domain among experienced reviewers [26]. This specific complexity arises because RoB 2.0 requires reviewers to distinguish between the effect of assignment to intervention and the effect of adhering to intervention, a process that necessitates nuanced interpretation of protocol deviations and their potential impact on the trial results. Beyond this individual domain, our overall results align with broader research documenting that RoB 2.0 consistently exhibits low to moderate interrater reliability (κ often < 0.50) even among trained methodologists [4,26,27]. In the Minozzi et al. study, human agreement for the overall RoB 2.0 judgment was as low as 0.16, which is categorized as slight agreement [23]. Such inherent subjectivity complicates the designation of any single human rating as an objective gold standard. Consequently, we utilized our blinded panel as the reference not because it represents a definitive truth, but because it offers a consensus-based benchmark independent of the potential incentives or outdated methods that may influence OSRAs. Indeed, low agreement between SR authors and our panel was somewhat comparable to internal panel variability, underscoring the need for improved guidance, training, or consensus processes to enhance consistency across reviewers.
ChatGPT-o3 exhibited moderate sensitivity (0.46) and specificity (0.69) in identifying high-risk trials, indicating some capacity to flag problematic studies, though it may miss many genuinely high-risk cases. For low-risk trials, specificity was higher (0.86), with similar sensitivity (0.47), reflecting a cautious approach that avoids false positives but may overlook genuinely low-risk studies. These results suggest a practical role for LLMs in triage workflows. For example, they could help deprioritize clearly low-risk studies for manual review, thereby improving overall efficiency. However, prospective validation is needed to confirm accuracy and maintain integrity.
Several limitations warrant consideration. First, our search strategy employed specific quoted terms; while necessary to identify studies utilizing the relatively recent RoB 2.0 framework, this strict approach may have excluded relevant meta-analyses that utilized alternative terminology or abbreviations (e.g., “RoB2” or “Risk of Bias tool 2”). Consequently, our sample may represent a subset of trials with more standardized reporting, potentially narrowing the diversity of reporting styles assessed. Second, we acknowledge the potential for data leakage. Since the meta-analyses in our sample were published prior to the model’s training cutoff, the LLM may have been exposed to the OSRA ratings during training. However, the low agreement observed between ChatGPT-o3 and the OSRAs (κ = 0.14) suggests the model was not simply regurgitating memorized training data but rather generating de novo judgments. Third, while newer iterations such as ChatGPT-5.2 have since been released, ChatGPT-o3 represented the state-of-the-art at the time of study design. Additionally, the evaluation involved only one LLM and one prompting strategy; other models or prompts, particularly fine-tuned on methodological content, may yield different outcomes. Furthermore, larger datasets would enhance precision and enable subgroup analyses based on trial characteristics or reporting quality. Time saved, a critical practical consideration, was not assessed. Consequently, we cannot calculate the net benefit of the LLM; without efficiency data, it is unclear if the time required to verify the model’s frequent “High Risk” flags outweighs the time taken to conduct a manual review from scratch. Additionally, although web browsing was enabled, we did not quantify how frequently the model successfully retrieved external protocols compared to human reviewers, which limits the transparency of the model’s information-gathering process. It appears the model may have prioritized our strict textual constraints over its ability to browse external data; however, without access to the model’s internal browsing logs, we cannot definitively determine when external searches were attempted or failed. Future research should perform sensitivity analyses using varied prompting strategies, such as specifically testing “optimistic” prompts that explicitly command the model to search for and ingest registry data, to determine if this yields better alignment with human raters. Furthermore, because this study focused solely on English-language RCTs, generalizability to non-English contexts remains uncertain. Finally, our study utilized a generalist approach, assessing trials across 50 different medical subspecialties. Unlike human experts who often possess deep domain-specific knowledge, which allows them to detect subtle field-specific biases (e.g., blinding difficulties in surgical trials versus pharmacological trials), the LLM applied a generalized logic. This lack of domain-specific context may have contributed to the lower agreement observed, suggesting that future implementations might require fine-tuning on specialty-specific literature.
Future research should investigate the performance of diverse LLMs across various tasks and contexts, including comparisons between general-purpose and domain-specific models. Prospective evaluations within real-world workflows will clarify practical utility concerning efficiency, reproducibility, and reviewer consistency. Additionally, exploring language-specific performance, international generalizability, and assessing LLM utility with other RoB tools beyond RCTs is essential. Alternative output formats such as confidence scores or probabilistic judgments may also enhance nuance and more accurately reflect uncertainty. Collectively, these efforts will support the responsible and scalable integration of LLMs into evidence synthesis.
Ultimately, ChatGPT-o3 showed potential as an adjunctive tool for RoB assessment in RCTs, consistently applying stricter criteria than both our masked panel and SR authors. While not yet ready for standalone use, the model may be useful in screening workflows or as a secondary reviewer. Continued development and validation will be essential to support the responsible integration of LLMs into SR practices.
Supporting information
S1 File. Supporting information.
This file contains all supplementary methods, figures and tables.
https://doi.org/10.1371/journal.pone.0353155.s001
(DOCX)
S2. File. TRIPOD-LLM Checklist.
Completed TRIPOD-LLM checklist for this study.
https://doi.org/10.1371/journal.pone.0353155.s002
(DOCX)
Acknowledgments
Patients and Public Involvement Disclosure: No patients or members of the public were involved at any point in this study.
References
- 1. Drucker AM, Fleming P, Chan AW. Research techniques made simple: assessing risk of bias in systematic reviews. J Invest Dermatol. 2016;136(11):e109-14.
- 2. Viswanathan M, Patnode CD, Berkman ND, Bass EB, Chang S, Hartling L, et al. Recommendations for assessing the risk of bias in systematic reviews of health-care interventions. J Clin Epidemiol. 2018;97:26–34. pmid:29248724
- 3. De Cassai A, Boscolo A, Zarantonello F, Pettenuzzo T, Sella N, Geraldini F, et al. Enhancing study quality assessment: an in-depth review of risk of bias tools for meta-analysis-a comprehensive guide for anesthesiologists. J Anesth Analg Crit Care. 2023;3(1):44. pmid:37932825
- 4. Nejadghaderi SA, Balibegloo M, Rezaei N. The Cochrane risk of bias assessment tool 2 (RoB 2) versus the original RoB: a perspective on the pros and cons. Health Sci Rep. 2024;7(6):e2165. pmid:38835932
- 5. Tomlinson E, Cooper C, Davenport C, Rutjes AWS, Leeflang M, Mallett S, et al. Common challenges and suggestions for risk of bias tool development: a systematic review of methodological studies. J Clin Epidemiol. 2024;171:111370. pmid:38670243
- 6. Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT-4’s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages. Res Synth Methods. 2024;15(4):616–26. pmid:38484744
- 7. Luo X, Chen F, Zhu D, Wang L, Wang Z, Liu H. Potential roles of large language models in the production of systematic reviews and meta-analyses. J Med Internet Res. 2024;26:e56780.
- 8. Lieberum J-L, Toews M, Metzendorf M-I, Heilmeyer F, Siemens W, Haverkamp C, et al. Large language models for conducting systematic reviews: on the rise, but not yet ready for use-a scoping review. J Clin Epidemiol. 2025;181:111746. pmid:40021099
- 9. Lai H, Ge L, Sun M, Pan B, Huang J, Hou L, et al. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Netw Open. 2024;7(5):e2412687. pmid:38776081
- 10. Xia D, Lai H, Zhao W, Huang J, Liu J, Ye Z, et al. Assessing risk of bias of cohort studies with large language models. Res Synth Methods. 2025;16(6):990–1004. pmid:41626981
- 11. Šuster S, Baldwin T, Verspoor K. Zero- and few-shot prompting of generative large language models provides weak assessment of risk of bias in clinical trials. Res Synth Methods. 2024;15(6):988–1000. pmid:39176994
- 12. Huang J, Lai H, Zhao W, Xia D, Bai C, Sun M, et al. Large language model–assisted risk-of-bias assessment in randomized controlled trials using the revised risk-of-bias tool: evaluation study. J Med Internet Res. 2025;27:e70450. pmid:40554779
- 13. Hasan B, Saadi S, Rajjoub NS, Hegazi M, Al-Kordi M, Fleti F, et al. Integrating large language models in systematic reviews: a framework and case study using ROBINS-I for risk of bias assessment. BMJ Evid Based Med. 2024;29(6):394–8. pmid:38383136
- 14. Lai H, Liu J, Bai C, Liu H, Pan B, Luo X, et al. Language models for data extraction and risk of bias assessment in complementary medicine. NPJ Digit Med. 2025;8(1):74. pmid:39890970
- 15. Di Pumpo M, Riccardi MT, De Vita V, Damiani G. Evaluation of a large language model (ChatGPT) versus human researchers in assessing risk-of-bias and community engagement levels: a systematic review use-case analysis. Eur J Public Health. 2025.
- 16. Kuitunen I, Nyrhi L, De Luca D. ChatGPT-4o in risk-of-bias assessments in neonatology: a validity analysis. Neonatology. 2025;122(3):360–5. pmid:39999815
- 17.
Introducing OpenAI o3 and o4-mini [Internet]. [cited 2025 Jul 1]. Available from: https://openai.com/index/introducing-o3-and-o4-mini/
- 18.
ChatGPT [Internet]. [cited 2025 Jun 29]. Available from: https://chatgpt.com
- 19. Gallifant J, Afshar M, Ameen S, Aphinyanaphongs Y, Chen S, Cacciamani G, et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60–9. pmid:39779929
- 20. Farrah K, Young K, Tunis MC, Zhao L. Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Syst Rev. 2019;8(1):280.
- 21.
RoB 2: a revised Cochrane risk-of-bias tool for randomized trials | Cochrane Bias [Internet]. [cited 2025 Jul 1]. Available from: https://methods.cochrane.org/bias/resources/rob-2-revised-cochrane-risk-bias-tool-randomized-trials
- 22.
Risk of bias tools - Current version of RoB 2 [Internet]. [cited 2025 Jul 1]. Available from: https://www.riskofbias.info/welcome/rob-2-0-tool/current-version-of-rob-2
- 23. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. pmid:843571
- 24. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276–82. pmid:23092060
- 25. Quarfoot D, Levine RA. How robust are multirater interrater reliability indices to changes in frequency distribution? Am Stat. 2016;70(4):373–84.
- 26. Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. J Clin Epidemiol. 2020;126:37–44. pmid:32562833
- 27. Minozzi S, Dwan K, Borrelli F, Filippini G. Reliability of the revised Cochrane risk-of-bias tool for randomised trials (RoB2) improved with the use of implementation instruction. J Clin Epidemiol. 2022;141:99–105. pmid:34537386