Methodological components and implementation

The overview of our method called MetaMind is described in Fig 1, and involved chaining together several modules including information retrieval, data curation, and comparative effect estimation. This study aimed to develop and evaluated an AI-driven framework for structured clinical evidence synthesis, applying advanced retrieval and extraction methodologies to comparative efficacy analysis in moderate-to-severe inflammatory bowel disease (IBD). Specifically, the approach was designed to extract and analyze clinical trial data for UC and CD using promptable PICO metrics to ensure targeted retrieval of relevant studies. To achieve this, Promptriever was employed to search and retrieve high-impact clinical trials from PubMed, focusing on studies assessing biologic and small-molecule therapies compared to placebo or active comparators. In this work, we pre-selected the relevant studies for downstream analysis for ease of comparability and validation with the manual Bayesian Evidence Generation. The MoA framework was then applied to extract key clinical endpoints, including baseline and final remission rates, sample sizes, and confidence intervals. This structured extraction, which was FLASK (Fine-grained Language model evaluation based on Alignment Skill Sets) evaluated, enabled a comprehensive evaluation of treatment efficacy across heterogeneous study designs, facilitating a scalable and potentially generalizable methodology for automated evidence synthesis in IBD research.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Workflow Schematic.

https://doi.org/10.1371/journal.pone.0342895.g001

To operationalize PICO-aligned retrieval and extraction, we used structured, model-ready prompts tailored to each stage of the pipeline:

Retrieval via Promptriever was guided by user-defined PICO elements embedded into a query-instruction format. For example, a prompt submitted to the model might read:

“Query: ulcerative colitis AND placebo AND (mirikizumab OR upadacitinib OR filgotinib OR ustekinumab OR etrasimod OR tofacitinib). A relevant document would describe clinical trials where patients were tested against placebo and the document includes efficacy results for the named treatments.”

This query format supports retrieval based on the PICO framework—specifying patient group, intervention, comparator, and outcome. Extraction via MoA system used standardized prompts across models like Meta-LLaMA-3, Mistral, and Qwen2 to extract structured clinical trial data. A representative example prompt is:

“Extract baseline and final clinical remission values for each treatment group, including confidence intervals (CIs) or standard deviations (SDs) where available. Report weekly remission rates at Weeks 4, 8, and 12. Include sample sizes per group. If any values are missing, clearly indicate and suggest plausible methods for estimation. Format output as a structured table for network meta-analysis.”

These prompts enabled reliable, cross-validated extraction of quantitative endpoints from free-text PDF trial reports. The detailed workflow is provided in Supplementary Figures 1 & 2 in S1 File.

The NMA generated through this approach was fully scripted and implemented using AI-driven automation, with GPT-4o producing the analytical script in Python. To validate the accuracy and robustness of the AI-generated NMA, results were benchmarked against a manual Bayesian Evidence Generation, conducted using R-based statistical methodologies widely accepted in evidence synthesis. This dual-implementation strategy allowed for a direct comparison between AI-assisted and traditional statistical approaches, ensuring methodological rigor and alignment with best practices in clinical research and decision-making (Fig 1).

Metamind workflow was then compared to a published manual NMA, to further confirm findings between the two approaches across all 3 stages of information retrieval, data curation, and comparative effect estimation [8].

Retrieving NMA-relevant articles using promptriever

Promptriever is implemented in PyTorch atop a Llama-2 base Transformer with a Parameter-Efficient Fine-Tuning (PEFT) adapter. We then fixed both model and tokenizer to a maximum sequence length of 512 tokens (with padding to multiples of eight to maximize batch efficiency). We did not perform any additional training or fine-tuning of Promptriever; instead, we used the publicly released pre-trained model [6].

At inference time, user prompts are grouped into batches of up to four which tokenizes them with truncation and pad-to-max-length. We extract the first token embedding for each sequence—yielding a (B, 256) tensor—apply ℓ₂-normalization across the feature dimension, and concatenate results into an (N, 256) NumPy array for all inputs.

These normalized embeddings are then compared against our library of template embeddings stored in a FAISS index. We compute inner-product similarity scores between each query vector and every template, sort in descending order, and retain the top candidates (whereby the retrieval parameter default is 100). To capture fine-grained contextual fit, each retrieved template is concatenated with the original prompt and fed to a lightweight 4-layer MLP re-ranker (512 units, ReLU activations, dropout 0.1). This network outputs a scalar relevance score for each pair.

We benchmark against the pre-trained all-mpnet-base-v2 SentenceTransformer (224 M parameters). This model generates 768-dim embeddings for both user prompts and templates. We follow the identical retrieval pipeline: compute cosine similarities over all template embeddings, sort and retain the top results (default: 100), and re-rank via the same 4-layer MLP described above. All other are kept constant to ensure direct comparability.

To facilitate a controlled validation against previously published manual NMAs, we limited the retrieval scope to a maximum of 10 high-impact articles per disease area, specifically from The Lancet and NEJM. This selection allowed for benchmarking model performance against high confidence, widely cited RCTs with standardized reporting formats. However, this constraint reflects a design choice for comparability—not a limitation of MetaMind itself. As shown in S1 File, when applied without restrictions, Promptriever unrestricted application was performed against a comprehensive manual review. This demonstrates MetaMind’s capacity to scale beyond artificial benchmarks, enabling broader and more current real-world literature synthesis. We examined recall, precision, and F1 to evaluate our findings.

Our implementation reuses and adapts components from the Weller et al. publication, with modifications for domain-specific retrieval (Supplementary Figure 1 in S1 File) and integration into the NMA pipeline [6].

Text extraction and preprocessing

The next step of this methodology focused on extracting text from PDF documents, leveraging the Fitz library (PyMuPDF) to systematically gather all available textual data, such as abstracts and clinical study details. Many PubMed records are machine-readable (XML or JSON), but full-text availability is limited by subscription access, publisher restrictions, and inconsistent file formats. PDFs present unique challenges, including inconsistent structure, embedded images, and encoding issues, making them a rigorous test case for our Mixture of Agents approach. Tables formatted as text were successfully processed; however, images, scanned tables, and graphical elements (e.g., plots) were not used in this version of the pipeline. Thus, results rely on data being presented in machine-readable textual formats. Future extensions may incorporate optical character recognition-based extraction for image-based tables or scanned documents.

By prioritizing PDF extractions, this methodology demonstrated the robustness and adaptability of our system in handling real-world document retrieval scenarios. A structured list was created, where each entry corresponded to a single PDF document, and exception handling mechanisms were implemented to address issues such as unreadable files or unsupported formats. This step was critical in preparing raw text data for subsequent semantic processing by LLMs, ensuring high-quality information retrieval even from complex, unstructured sources.

Prompt engineering and layered aggregation

The methodology employed a MoA approach, utilizing multiple LLMs across distinct inference layers to optimize clinical parameter extraction from both structured PubMed data and unstructured full-text PDFs. In the first inference layer, models including Meta-Llama-3.1-8B-Instruct-Turbo, Mistral-7B-Instruct-v0.3, and Qwen2-72B-Instruct were tasked with extracting baseline and final clinical values, standard deviations, confidence intervals, and sample sizes. To improve accuracy and consistency, the second and third inference layers employed GPT-4o, which aggregated model outputs, cross-validated extracted clinical parameters, recalculated missing values when necessary, and structured the results into a standardized format suitable for NMA. This layered approach enhanced robustness, particularly in complex PDF extractions, where inconsistent formatting, missing values, and embedded numerical data posed significant challenges. The methodology applied a recursive approach where the GPT-4o model in each layer refined the aggregated outputs from the previous iteration (Supplementary Figure 2 in S1 File). The full MoA implementation, including agent prompts, aggregation logic, and decision rules, is provided in the Appendix. This design is model- and disease-agnostic and does not rely on condition-specific heuristics.

Final synthesis

In the final stage, the responses from the aggregation layers were synthesized into a comprehensive output using GPT-4o. In this final stage, the pipeline used GPT-4o’s streaming mode to generate outputs progressively from structured prompts. All prompts were fixed in advance and applied uniformly across documents. This streaming functionality refers solely to the LLM response interface and does not imply iterative fine-tuning, editing, or adaptive prompting. Extracted data were compared post hoc to manually curated NMA reference sets for evaluation.

PICO

The output of this methodology included a manual evaluation of PICO elements, assessing the accuracy (faithful reproduction) and completeness of the generated summaries in relation to the reference text [2]. Rather than relying on similarity scores, which proved insufficient due to the highly detailed and comprehensive nature of the model outputs, each PICO component was individually and manually reviewed to ensure factual correctness, alignment with the reference, and clinical relevance. A qualitative assessment approach was applied, ensuring that evaluations captured nuanced differences in medical evidence synthesis.

This manual approach is particularly valuable in clinical research and systematic reviews, where accurate representation of PICO elements is essential for high-quality evidence synthesis. By prioritizing expert-driven assessment over purely computational similarity measures, the methodology ensures a rigorous and context-aware evaluation of LLM-generated summaries, setting a higher standard for automated text summarization systems in medical applications.

FLASK

This methodology employed a multi-dimensional evaluation framework using the FLASK criteria, which assessed textual outputs based on correctness, factuality, efficiency, commonsense, comprehension, insightfulness, completeness, metacognition, readability, conciseness, and harmlessness. The evaluation process was fully automated, leveraging OpenAI’s GPT-4o to systematically score and analyse responses [5,9]. First, the reference text was extracted from clinical study PDFs, ensuring a structured comparison between the original document and the LLM-generated response. GPT-4o was then prompted with a predefined evaluation rubric, requesting numerical scores (1–5) for each FLASK criterion, along with a brief justification for each score. By integrating automated scoring and structured evaluation, this framework ensured objective, reproducible, and high-fidelity assessments of LLM-generated clinical summaries (Supplementary Figure 3 in S1 File).

In addition to automated evaluation, we performed manual spot checks on a random sample of outputs to ensure alignment between LLM-assigned FLASK scores and human clinical interpretation. This hybrid approach helped verify that model judgments were consistent with domain-specific expectations. LLM-based scoring can scale evaluation across large datasets, but the models may reflect biases present in their training data. To mitigate this, we used fixed rubrics and instructed the model to justify each score. This approach, along with manual spot validation, was designed to reduce variability and surface potential inconsistencies in scoring across different documents.

Automation of NMA generation

We used GPT-4o to write analytical code in R, using a Python-integrated workflow, to analyze the assembled dataset using brms, a validated and widely used software package for estimating comparative effects using Bayesian statistics (Supplementary Figure 4 in S1 File). This study employed an API-driven approach to dynamically generate and execute the code for Bayesian meta-analysis using GPT-4o [1]. Structured pseudocode instructions were provided to GPT-4o to generate the script that defines and analyses a Bayesian random-effects model. The model evaluated relative treatment effects with a binomial likelihood and logit link function, incorporating random treatment effects at the study level and specified priors. This script was executed within Python using the subprocess module, allowing seamless integration with the remaining Python-based workflow. AI-generated code in MetaMind serves as a scripting mechanism to faithfully reproduce a predefined, standard network meta-analysis workflow rather than to autonomously design or optimize statistical models. The underlying statistical approach was specified a priori and corresponds to established Bayesian NMA implementations. The role of the language model was limited to generating executable R scripts that implement this predefined analysis pipeline.

More specifically, the NMA code itself was written in R by GPT-4o using structured pseudocode instructions. The workflow runs primarily in Python, which calls an R-based Bayesian model through dynamically generated scripts executed via the subprocess module. This hybrid setup was chosen to preserve the statistical rigor of established R-based packages (e.g., brms), while enabling LLM-assisted automation and integration using Python.

To assess the statistical equivalence of AI-generated analysis code, results produced by GPT-4o–generated scripts were compared against manually implemented and vetted reference analyses using the same datasets and model specifications. Across these comparisons, treatment effect estimates, credible intervals, heterogeneity parameters, and treatment rankings were numerically equivalent, with no discrepancies in clinical conclusions observed. These checks confirm that AI-generated code faithfully reproduces standard Bayesian network meta-analysis workflows when executed under equivalent assumptions.

The generated script includes model definitions, priors, convergence diagnostics, and summary tables, and is available under Supplementary Materials in S1 File.

Bayesian evidence generation

The Bayesian analysis workflow was implemented using a Python-integrated workflow to evaluate treatment effects through a binomial-logit regression framework. A Bayesian random-effects model was defined with parameters for treatment-specific effects and between-study heterogeneity, using appropriately specified priors. The log-odds were estimated by modelling response counts over total sample size, incorporating treatment as a fixed effect and random treatment effects at the study level. The model output included posterior summaries with credible intervals, Rhat values for convergence diagnostics, and an estimate of between-study heterogeneity (tau). Between-study heterogeneity is naturally quantified by the posterior distribution of τ; I², which is derived from frequentist statistics, was therefore not the primary heterogeneity measure reported.

The entire analysis was executed in R-integrated Python workflow via the subprocess module, enabling seamless automation and reproducibility of Bayesian meta-analysis (Supplementary Figure 4 in S1 File).

In our analysis we utilized an approach within the Bayesian framework extension, which offers certain advantages like user-friendliness and familiarity over the more common NMA approach that examines treatment contrasts [10–12]. The treatment contrast method is extensively employed in health technology evaluations, such as those conducted by the UK’s National Institute for Health and Care Excellence (NICE). The NICE decision support unit has provided methodological guidelines on the practical application of this method [13].

Manual R-based NMA validation

To validate the results the manual implementation was performed for the set of selected publications [14–23]. Standard process was followed starting with comprehensive literature search and systematic review of selected studies. To enable fair comparison between manual and automated selection, we focused on selected studies. We extracted the data on study design, interventions, and outcomes. The results derived from our manual network meta-analysis closely mirrored those obtained via our automated pipeline approach, indicating a high degree of concordance between two approaches and affirming the robustness and reliability of the automated procedure (Supplementary Figures 5 & 6 in S1 File). In future endeavours, we plan to incorporate elements such as model evaluation, assessment of goodness-of-fit for the selected publications, and verification of underlying assumptions in the end-to-end pipeline [24,25].

Workflow

This entire methodology was implemented in Python, with R executed within this ecosystem, eliminating the need to rely exclusively on R or other statistical programming languages traditionally associated with NMA (Supplementary Figures 1, 2, 4, 9 in S1 File) [1,7]. This approach retained the strengths of R for Bayesian modelling while leveraging Python’s integration capabilities for automation, machine learning, and natural language processing. To ensure MetaMind meets both rigor and speed requirements, we benchmarked its entire workflow runtime against typical manual NMA workflows.

Framework for addressing study heterogeneity

Potential sources of clinical and methodological heterogeneity—such as dosing regimens, patient population characteristics, trial duration, and prior treatment exposure—were explicitly extracted as structured fields by MoA. In the current study, these variables were not incorporated as effect modifiers in the network meta-analysis model; however, their structured extraction ensures that heterogeneity sources are transparent rather than implicitly ignored.

MetaMind addresses study heterogeneity through two key mechanisms: (1) the MoA framework extracts detailed trial-level data—including baseline characteristics (e.g., disease severity, prior treatment exposure, sample size, age distributions)—into a structured tabular format, and (2) the final NMA stage models between-study heterogeneity explicitly using a random-effects Bayesian model, which accounts for variance across trials. While the current pipeline does not yet implement automated subgroup adjustment or meta-regression, the structured data output allows users to identify population differences and refine inclusion or stratification rules as needed. This modular design enhances interpretability and consistency across heterogeneous study sources.

For studies with incomplete or heterogeneous reporting (e.g., missing standard deviations or alternative summary statistics), supplementary prompts were used solely to flag missing information and to enumerate commonly used statistical imputation or transformation approaches (e.g., deriving standard deviations from confidence intervals or interquartile ranges). Importantly, these prompts did not autonomously apply statistical corrections or alter trial data used in the network meta-analysis. All such cases required explicit human review, and only standard, guideline-consistent methods were applied prior to analysis.

Implementation and reproducibility

All experiments were conducted using a consistent and explicitly specified LLM version throughout the study. Specifically, all LLM-driven components—including trial extraction, aggregation, and code generation—used GPT-4 (denoted as GPT-4o) without version switching or adaptive model selection. This avoided variability introduced by model updates. The temperature parameter was set to zero for critically challenging sections, ensuring deterministic token generation. Wherever supported by the underlying frameworks, random seeds were fixed to further reduce nondeterminism. Wherever repeated executions were performed, we observed no meaningful variability.

The full MetaMind pipeline implementation, including retrieval, extraction, aggregation, and analysis scripts, is provided in the Appendix. Due to reliance on proprietary LLM APIs, the pipeline is not released as a standalone executable repository; however, all core logic, prompts, and workflow components necessary for reproduction and adaptation are fully documented.

Automated code generation and execution

AI-generated code was executed in a secure, sandboxed computing environment with no external network access, preventing unintended data exfiltration. The execution environment was restricted to standard statistical libraries required for Bayesian network meta-analysis (e.g., brms). Second, all generated code was inspected by the authors prior to execution to verify that it performed only the intended analytical tasks. This inspection step ensured that the code adhered to established statistical practices and did not contain unsafe or extraneous operations. Third, the outputs of AI-generated analyses were systematically validated. Model results—including effect estimates, credible intervals, and treatment rankings—were cross-checked against known benchmarks and manually implemented reference analyses on selected datasets. Standard convergence diagnostics were reviewed to confirm proper model convergence and numerical stability.

MetaMind promptriever performance and output

We first examined, MetaMind performance in retrieving relevant articles. The performance of PEFT Promptriever and SentenceTransformer was evaluated based on their similarity scores for both UC and CD. The highest similarity score achieved by PEFT Promptriever was 0.7403, marginally outperforming SentenceTransformer’s 0.7049 (Table 1). Additionally, PEFT Promptriever demonstrated a narrower similarity range (0.0814) compared to SentenceTransformer (0.0959), indicating a tighter focus on relevant articles and higher specificity in aligning to nuanced query instructions. Similarly, on CD PEFT Promptretriever demonstrates more consistent performance with a narrower range (0.0353), focusing on high-relevance results (Table 2). A filter was applied to select only the top 10 most relevant articles.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Performance Comparison Between Retrieval Models for Ulcerative Colitis.

https://doi.org/10.1371/journal.pone.0342895.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Performance Comparison Between Retrieval Models for Crohn’s Disease.

https://doi.org/10.1371/journal.pone.0342895.t002

The top five articles on the induction and maintenance therapy for ulcerative colitis were filtered (Table 3). The study on Etrasimod achieved a similarity score of 0.7043. Upadacitinib scored 0.6998 and explored advanced therapies for moderate to severe ulcerative colitis These findings highlight the capability of PEFT Promptriever to retrieve high-quality, clinically relevant studies with nuanced semantic alignment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Promptriever Output-Key Ulcerative Colitis Articles Examples.

https://doi.org/10.1371/journal.pone.0342895.t003

A similar approach was taken for Crohn’s disease. The top four articles on the induction and maintenance therapy for Crohn’s disease were filtered (Table 4). The study on Risankizumab achieved the highest similarity score of 0.7142 and focused on clinical remission in moderate-to-severe Crohn’s disease.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Promptriever Output-Key Crohn’s Disease Articles Examples.

https://doi.org/10.1371/journal.pone.0342895.t004

To validate Promptriever’s retrieval accuracy, we compared its results against a published NMA which included 28 relevant trials (Supplementary Figure 7 in S1 File). Of these, Promptriever successfully retrieved 23, resulting in a recall of 82.1%. All retrieved trials were relevant, yielding a precision of 91.1% and an F1 score of 86.4%. The 5 studies missed by Promptriever included 1 without a PubMed ID, which was not retrievable. These findings demonstrate that Promptriever delivers high-recall, high-precision semantic retrieval while also surfacing more recent and diverse studies than the manual reference set. Across reference NMAs, retrieval performance was consistent, with no qualitative differences observed between UC and CD. Moreover, as not every alternative Promtriever frameworks was explored, it is plausible that due to workflow flexibility these summary statistics can be further improved. Precision and recall were computed deterministically with respect to fixed retrieved trial sets. Because no sampling or resampling procedure was used and model inference was not involved in metric computation, these performance measures are reported as point.

MetaMind MoA output

MetaMind’s MoA module was evaluated for its ability to extract structured clinical data from unstructured PDF trial reports. For selected trials in UC and CD, the system correctly extracted numerical endpoints (e.g., remission rates, confidence intervals, and sample sizes) across two timepoints per study, demonstrating successful mapping to NMA-ready tabular format (Table 5 & Table 6). These outputs were validated against manually curated references to assess extraction fidelity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Data Extraction for UC of Ustekinumab using MoA.

https://doi.org/10.1371/journal.pone.0342895.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Data Extraction for CD of Upadacitinib using MoA.

https://doi.org/10.1371/journal.pone.0342895.t006

We assessed the accuracy of extracted PICO elements by manually reviewing outputs from a subset of UC and CD trials. For this limited evaluation (focused on remission-related endpoints), the MoA framework correctly extracted and aligned all targeted data fields, achieving 100% accuracy on a limited set of remission endpoints. Similar extraction performance accuracy was reported for other relevant data elements, including study design features, baseline patient characteristics, intervention and dosing information, and reported adverse events. Across reference NMAs, extraction performance was consistent, with no qualitative differences observed between UC and CD. Similarly, extraction performance was 100% when applied on papers from a recent, peer-reviewed NMA publication on UC [8]. However, we acknowledge that broader validation across multiple endpoints and trial designs is needed before generalizing this performance metric further.

FLASK variability was observed due to study-level differences in reported endpoints and formatting (Supplementary Figure 3 in S1 File). However, the use of universal prompts across all studies was sufficient to capture all required clinical information, even in the presence of structural variability (Supplementary Figure 2 in S1 File). This demonstrates the robustness of the prompt design, which enabled consistent extraction without the need for study-specific tailoring.

MetaMind NMA output

To validate the accuracy of MetaMind’s automated NMA pipeline, we conducted two complementary comparisons. First, we compared the AI-generated NMA results (Figs 2a and 2b) to manually performed analyses by the co-authors (Supplementary Figure 5 in S1 File) [14–23]. This internal benchmark showed high concordance in effect estimates, credible intervals, and treatment rankings, confirming the reliability of MetaMind’s outputs. For both disease networks, model convergence diagnostics indicated stable estimation across all parameters, with R-hat values approximately equal to 1.00 and effective sample sizes sufficiently large for all treatment effect estimates. For the ulcerative colitis network, the residual deviance (20.4 on 20 data points) was closely aligned with the number of observed data points, and the deviance information criterion (DIC) was 39.8, indicating adequate model fit under the specified random-effects structure. Similarly, for the Crohn’s disease network, the residual deviance (13.5 on 15 data points) demonstrated good agreement with observed data, and the DIC was 26.7, supporting satisfactory model fit. Network plots were constructed which were well-connected and contained evidence loops (Supplementary Figure 6 in S1 File). Second, we assessed alignment with a recent publication, a peer-reviewed NMA in ulcerative colitis (Supplementary Figures 7 and 8 in S1 File) [8,15,19,26–31]. This demonstrates consistent treatment effect patterns and overlap in identified trials, supporting external validity. Among treatments common to both MetaMind and the published NMA, relative treatment rankings and effect directions were concordant, with overlapping credible intervals for all comparable endpoints. These findings indicate that MetaMind can reproduce both internally generated and published NMA results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. (a) UC NMA Summary & Forest Plot (b) CD NMA Summary & Forest Plot.

https://doi.org/10.1371/journal.pone.0342895.g002

Finally, MetaMind outputs can be generated within a week, while the entire workflow to be generated manually would require several months [32].

A stage-wise comparison of MetaMind against manual NMA workflows is summarized in Table 7, highlighting performance across retrieval accuracy, data extraction fidelity, statistical concordance, and execution time. Aggregate metrics are reported for clarity, as performance was consistent across benchmarks.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Summary table of MetaMind performance across all stages.

https://doi.org/10.1371/journal.pone.0342895.t007