Introduction

Peer review serves as the primary quality control mechanism for scientific publications, yet studies consistently demonstrate substantial limitations in detecting methodological errors and reporting deficiencies [1]. The increasing strain on peer review processes, driven by rapid growth in manuscript submissions, has affected evaluation rigor across publishers [2]. When investigators intentionally introduced nine significant errors into randomized controlled trial manuscripts, peer reviewers detected an average of only three errors, with nearly 25% of reviewers identifying one error or fewer [3]. These systematic limitations extend to harm data reporting, where inconsistent analysis in randomized controlled trials fails to provide adequate information for clinical decision-making [4].

The Consolidated Standards of Reporting Trials (CONSORT) provides an evidence-based, minimum set of recommendations for reporting the methodology and results of randomized controlled trials to ensure clarity and transparency. First published in 1996 and subsequently updated in 2001, 2010, and 2025, CONSORT guidelines facilitate critical appraisal and enhance research reproducibility [5]. Despite widespread endorsement in high-impact medical journals, compliance remains suboptimal. Analysis of 463 abstracts from five leading journals found overall adherence of 67%, with individual journal rates ranging from 55% to 78% [6]. Similar shortfalls persist over time; a review of heart failure trials reported compliance between 60% and 70% across two decades [7].

Large language models (LLMs) have increasingly been utilized in scholarly processes, including literature reviews, manuscript preparation, and systematic error detection [8]. Advanced models, such as GPT-4, detect approximately 53% of intentionally inserted errors, approaching the performance of human peer reviewers [9]. When comparing LLM feedback to human reviewer comments on 3,096 manuscripts from Nature family journals, the overlap between GPT-4 and individual human reviewers (31%) closely matched inter-human reviewer agreement (29%) [10]. The practical significance of these capabilities was demonstrated when an AI model identified, within seconds, a ten-fold mathematical error in published flame-retardant research, a miscalculation that human reviewers had missed. This detection prompted the Black Spatula Project, an open-source initiative employing LLMs to identify overlooked errors in scientific literature [11]. LLMs have also demonstrated utility in automated paper screening for clinical reviews, extending their application to systematic literature evaluation [12].

How consistently different LLMs score the same trial against a structured reporting guideline such as CONSORT remains unexplored. If different models applied to the same manuscript produce systematically different compliance scores, the choice of model becomes a hidden determinant of any editorial or screening decision that relies on such scoring. This investigation compared three leading LLM platforms as raters of CONSORT compliance on a fixed set of 20 published randomized controlled trials, treating the trials as a common stimulus set and the models as the objects of study.

Methods

Results

Quality threshold analysis

Using a 90% compliance threshold as a quality benchmark, ChatGPT-4o identified 5 articles (25%), while Gemini 2.5 Flash and Claude Sonnet 4.6 each identified no articles (0%) as meeting this standard. At least 75% of papers failed to meet the 90% threshold across all three models. These findings indicate substantial variation in how the models scored CONSORT reporting even among peer-reviewed publications (Fig 1).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. CONSORT compliance assessment by LLM platform.

Mean CONSORT 2010 compliance rates for 20 randomized controlled trials evaluated by three LLM platforms. Error bars represent 95% CIs. The dashed line at 90% indicates the quality threshold benchmark. N = 20 articles per platform. Compliance was assessed using a three-level scoring system (0, 0.5, 1.0 point) across 37 CONSORT 2010 subpoints.

https://doi.org/10.1371/journal.pone.0358873.g001

Discussion

This investigation demonstrates that LLMs vary substantially in how they assess CONSORT compliance in published randomized controlled trials. The three models produced a consistent ordering: ChatGPT-4o scored compliance highest (80.8% mean), Gemini 2.5 Flash intermediate (64.4%), and Claude Sonnet 4.6 lowest (54.7%), and all pairwise differences were statistically significant. Even the highest-scoring model found that only 25% of papers met the 90% compliance threshold expected for high-quality clinical journals.

These findings occur within the broader context of well-documented peer review limitations [1]. Previous investigations demonstrate that peer reviewers detect only 3 of 9 intentionally inserted significant errors in randomized controlled trial manuscripts, with 25% of reviewers identifying only one error and 16% failing to detect any errors at all [3]. The increasing strain on peer review processes, driven by rapid growth in manuscript submissions, has reduced the time available for thorough evaluation of research quality and methodology [2]. Different peer review procedures demonstrate varying abilities to flag problematic publications, with systematic limitations across traditional editorial processes [14]. This investigation suggests that LLMs may serve as valuable complementary tools to address these systematic limitations in conventional editorial processes.

LLMs have demonstrated the ability to detect errors missed by peer review, such as the ten-fold dosage miscalculation in brominated diethyl ether research that prompted the Black Spatula Project, a grassroots initiative using LLMs to identify overlooked academic errors [11]. They have also been shown to detect commonly missed problems, including absent protocols, ambiguous ethics statements, and inappropriate citation practices [15].

Across all three models, at least 75% of papers failed to meet a 90% CONSORT compliance threshold. Inadequate CONSORT compliance compromises the ability of clinicians and researchers to appraise study methodology critically, assess risk of bias, and apply findings to patient care [16]. The potential for missed reporting errors to affect real-world clinical decisions underscores the importance of systematic quality assessment tools. Clinical research errors that escape detection can have substantial consequences. High-profile retractions for statistical problems illustrate this risk, including the 2023 retraction of a highly cited medication adherence study for misleading statistical reporting [17]. Among all reasons why a medical article may be deemed unfit for publication, the most common are plagiarism and data fabrication [18], but broader applications for error detection remain underutilized.

The design of this study identifies the magnitude of the differences between models but not their cause. Several explanations are plausible. Model architecture and the composition of training data differ across the three platforms, and models trained on different distributions of biomedical text may weight the same reporting elements differently [8]. The models also appear to apply different thresholds for what counts as adequate reporting: the divergence concentrated in judgment-dependent subpoints, such as why a trial was stopped, the description of intervention similarity under blinding, and losses after randomization, while structurally explicit subpoints, such as background and objectives, produced near-complete agreement. That pattern is consistent with the models differing in strictness when a criterion is partially addressed rather than differing in their ability to locate information in the text. Differences in how each model interpreted the shared prompt, and in the alignment and instruction-tuning procedures applied to each, may contribute as well. These explanations are plausible rather than established; the present design cannot separate them, and testing them would require systematic variation of prompt wording, model configuration, and criterion definitions. Because no human gold-standard assessment was available, the differences also cannot be attributed to one model being more accurate than another; they establish only that the models diverge systematically when scoring the same trials against the same criteria.

The relationship between LLM scoring and expert human assessment remains the central open question. When comparing LLM feedback to human reviewer comments on 3,096 papers from Nature family journals, the overlap between LLM and human assessments (31%) closely matched inter-human reviewer agreement (29%), with 82.4% of users finding GPT-4 feedback more helpful than some human reviewers [10]. However, a cross-sectional comparison of four LLMs against human reviewers across 22 manuscripts submitted to a surgical journal found that although the models were highly consistent across repeated evaluations (ICC = 0.88), they recommended rejection far less often than human reviewers, who rejected 68.2% of manuscripts compared with LLM rejection rates of 0 to 9.1% (p < 0.001) [19]. Agreement between the models and human recommendations in that study was moderate (weighted kappa 0.38 to 0.46), a range close to the item-level agreement observed among models in the present study.

Human CONSORT assessment is itself imperfect. When two trained reviewers independently scored published trials against the CONSORT checklist, they agreed completely on explicit items such as inclusion criteria, exclusion criteria, and the point estimate, but reached only moderate agreement on judgment-dependent items, with kappa values of 0.53 for allocation concealment and 0.54 for deviation from protocol [20]. That is the same pattern observed between models here. The relevant standard is therefore not perfect concordance but whether model-to-human agreement approaches the agreement observed between trained human assessors. The present study cannot address that question, because no human ratings of these 20 trials were obtained. The item-level agreement values reported here therefore describe only how interchangeable the models are with one another. Whether any of the three tracks expert human judgment, and which one tracks it most closely, requires a study in which the same trials are scored by both trained human CONSORT assessors and the models.

LLMs demonstrate particular strengths in linguistic editing, summarization, and systematic guideline evaluation, but may show limitations with statistics-heavy or nuanced medical content [8]. Previous research indicates that while LLMs can identify various types of errors comparable to human reviewers, they may exhibit weaknesses in evaluating complex statistical analyses and mathematical content [9]. Some studies have raised concerns about LLM limitations, including the generation of confident-sounding hallucinations with little relationship to actual paper content [21]. With certain prompting techniques or retrieval-augmented generation, newer LLM iterations can achieve lower hallucination rates on specific tasks compared to earlier models, though these improvements are often limited and do not eliminate the underlying issue [22].

These results extend previous investigations, demonstrating that GPT-4 detected 53% of intentionally inserted errors, comparable to the performance of human peer reviewers [9]. The current investigation focuses on systematic guideline compliance scoring rather than error detection. It shows that LLMs applying the same CONSORT criteria to the same trials produce systematically different compliance scores [23]. This application is distinct from previous work on linguistic editing, summarization, and basic error detection, and it addresses a gap in understanding how LLMs evaluate the quality of clinical trial reporting. Studies have demonstrated ChatGPT’s capability in identifying methodological flaws and providing insightful feedback on theoretical frameworks, with critical analyses aligning with human reviewers [24]. Recent work has also demonstrated LLM effectiveness in automated paper screening for clinical reviews, extending their utility beyond individual manuscript assessment to systematic literature evaluation [12].

Several limitations affect the interpretation of these findings. First, the trials were drawn from a single specialty, immunology, and from a narrow publication window, so the findings may not extend to other medical fields, other study designs, or other reporting contexts. The models may diverge to a different degree, or in a different order, when scoring trials whose reporting conventions differ from those in immunology. A single-specialty stimulus set nonetheless enabled a controlled comparison in which every model scored identical material, isolating between-model differences from variation in subject matter. Second, each model generated a single evaluation per article, so run-to-run variability within a model was not captured; the design nonetheless isolates differences between models under identical inputs. The partial scoring system (0/0.5/1.0) introduced subjective interpretation; conversely, it reflects real-world evaluation where criteria are often partially met. Third, no gold-standard human expert assessment was conducted, so this study characterizes how the models differ from one another rather than their absolute accuracy; establishing accuracy would require a validated human reference standard. Fourth, the stimulus set of 20 trials is small, and estimates derived from it are correspondingly imprecise. The sample supports comparison of the models against one another, but it does not support generalization of any single model’s absolute compliance score to the wider published literature. The set was nonetheless scored identically by all three models, and the large observed effect size indicates it was sufficient to detect the between-model differences under study. Fifth, LLMs may exhibit limitations in statistical content [8], although CONSORT predominantly assesses reporting transparency. Finally, CONSORT compliance represents only one quality dimension; however, this focused scope provides clear evidence for systematic differences between models in structured guideline evaluation.

Future research should prioritize human-AI agreement studies in which the same trials are scored independently by trained human CONSORT assessors and by multiple LLMs, reporting model-to-human agreement alongside human-to-human agreement so that model performance is judged against the reproducibility that human assessment actually achieves. Such studies would establish accuracy against a validated reference standard rather than the relative comparison reported here. Work should also focus on developing standardized LLM evaluation protocols for manuscript assessment and expanding evaluation to additional reporting guidelines such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and the Standards for Reporting of Diagnostic Accuracy Studies (STARD). Repeating the evaluation across multiple runs per model would quantify within-model variability alongside the between-model differences observed here. Investigation of prompt engineering techniques to optimize LLM performance and reduce inter-model variability represents another research priority. Traditional peer review has documented limitations in identifying research misconduct [14]. Whether LLMs can detect other forms of misconduct, including statistical fraud and citation manipulation, therefore warrants investigation. The continued evolution of LLM capabilities suggests expanding potential applications in scientific quality assessment [25].

Conclusions

Three LLMs scoring the same 20 randomized controlled trials against the same 37 CONSORT subpoints produced significantly different compliance scores. Mean compliance ranged from 54.7% to 80.8%, with a consistent ordering of ChatGPT-4o highest, Gemini 2.5 Flash intermediate, and Claude Sonnet 4.6 lowest. Item-level agreement between models was only moderate, with no pairwise weighted kappa reaching the 0.61 substantial-agreement threshold. This degree of inter-model variability indicates that the choice of model materially affects the assessment, and that standardized evaluation protocols and validation against expert human review are needed before LLM-assisted compliance screening is adopted in editorial processes.

Supporting information