Introduction

Radiology impressions guide clinical care. They don’t just restate findings, they turn them into an answer to the clinical question, often the only part a referring clinician reads closely [1].

Large language models (LLMs) perform well across different NLP tasks [2]. In radiology, they are being tested for report classification [3–6], clinical note summarization [7–9], structuring free text reports [10,11], and drafting reports, summaries, and impressions [12–14].

The problem is that fluent isn’t the same as right. For impression generation, LLMs can produce output that reads well but drifts into generic phrasing, inconsistent style, or, worse, unsupported statements and incorrect interpretations [15].

Workarounds exist. Domain adaptation and fine-tuning can improve performance [16], but inference-time strategies are gaining traction because they can change behavior without retraining [17]. Retrieval-augmented generation (RAG) is one such strategy: retrieve semantically similar prior cases and use them as high-fit few-shot examples at inference time, instead of relying on static examples [18].

This study tests a RAG-based dynamic few-shot approach for generating impressions from pulmonary CT angiography (CTPA) reports. We compared this method against zero-shot prompting and fixed, randomly selected few-shot prompting.

Methods

This retrospective study was approved by the Mayo Clinic Institutional Review Board (IRB), with a waiver of informed consent granted (approval no. 24–013325, March 10, 2025). We included CTPA reports from Mayo Clinic sites (Rochester, Arizona, Florida, and the Mayo Clinic Health System) performed between April 30, 2023, and June 17, 2024. Data were accessed for research purposes between March 15, 2025, and April 10, 2025. The authors had access to identifiable patient information during data extraction, which was handled within institutional firewalls and only for the analysis.

In total, 11,998 reports (10,681 unique patients) were analyzed. Eligible reports contained both findings and impression sections and included both structured and unstructured report formats.. Reports were randomly split at the report level: 95% (n = 11,399) formed the retrieval bank and 5% (n = 599) were held out for testing. Because the split was performed at the report rather than patient level, patients with multiple examinations could contribute more than one report across the overall dataset. The reference impressions served as the ground truth for evaluation.

RAG inserts case-matched examples at inference time [18]. We generated GPT-based embeddings for the findings text in the retrieval bank. For each test report, we embedded its findings, retrieved the top-k most similar findings from the bank, and inserted the paired findings–impression examples into the prompt as in-context demonstrations (Fig 1).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Retrieval-augmented prompting (retrieval-based few-shot) for impression generation from CT pulmonary angiography (CTPA) reports.

The findings text from the new report is used as the retrieval query and embedded to retrieve the top-k most semantically similar findings–impression examples from a vector database (retrieval set). The final LLM prompt includes task instructions, the new report findings, and the retrieved examples, and the LLM generates the output impression.

https://doi.org/10.1371/journal.pone.0354688.g001

We evaluated LLaMA 3.1-70B [19] and GPT-4o [20] for impression generation from the findings section under three inference setups: zero-shot prompting; fixed few-shot prompting with 3, 5, or 10 randomly selected examples (held constant across runs); and dynamic retrieval-selected few-shot prompting with k = 3, 5, or 10. We ran temperatures 0, 0.7, and 1, using default top-P and top-K settings.

Outputs were compared to the reference impressions using ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore F1 [21,22]; the full prompt is provided in the S1 File. We also performed qualitative analysis on a subset of cases to assess whether low similarity scores reflected superficial wording differences or clinically important errors, the 50 lowest-scoring generated impressions by ROUGE-L F1 for each model (overall 100 impressions) were evaluated by a radiologist. Error categories included wrong prioritization/redundant details, truncated or malformed output, hallucinated findings, overinterpretation/unsupported recommendations, omission, benign rephrasing, and other.

Statistical comparisons used the Wilcoxon signed-rank test with p < 0.05. All analyses were conducted in Python 3.11 within institutional firewalls, and each report was processed in an isolated run to prevent cross-case influence.

This study evaluated text-generation performance only; latency, throughput, compute cost, and clinical workflow feasibility were not assessed.

Results

Demographic characteristics

Baseline demographics for the retrieval bank and test set are summarized in Table 1. The cohorts were similar, with no significant differences.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Demographic characteristics of patients in the embeddings and test datasets (n = 11,998^*).

https://doi.org/10.1371/journal.pone.0354688.t001

Overall performance

Dynamic retrieval few-shot prompting beat both zero-shot and fixed few-shot across models, temperatures, and k on ROUGE-1/2/L and BERTScore F1 (all p < 0.05; S1 File: Supplemental Tables 1-8). Fixed few-shot also improved over zero-shot, but the gains were smaller (Figs 2–3, S1 File: Supplemental Tables 1–8).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. GPT-4o comparison of mean scores with dynamic, fixed and without few-shot learning across temperatures and top-k values.

Mean ROUGE-1 (a), ROUGE-2 (b), ROUGE-L (c), and BERTScore (d) F1 scores for generated impressions using dynamic few-shot (FS) learning (“Dynamic FS”), zero-shot (“Zero-shot”), and fixed few-shot (“Fixed FS”), across varying temperatures (0, 0.7 and 1) and top-k retrieved documents [3,5,10]. Solid lines represent results with dynamic few-shot results, dashed lines represent zero-shot and fixed few-shot results.

https://doi.org/10.1371/journal.pone.0354688.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. LLaMA-3.1-70B comparison of mean scores with dynamic, fixed and without few-shot learning across temperatures and top-k values.

Mean ROUGE-1 (a), ROUGE-2 (b), ROUGE-L (c), and BERTScore (d) F1 scores for generated impressions using dynamic few-shot (FS) learning (“Dynamic FS”), zero-shot (“Zero-shot”), and fixed few-shot (“Fixed FS”), across varying temperatures (0, 0.7 and 1) and top-k retrieved documents [3,5,10]. Solid lines represent results with dynamic few-shot results, dashed lines represent zero-shot and fixed few-shot results.

https://doi.org/10.1371/journal.pone.0354688.g003

Qualitative analysis

Example high- and low-scoring impressions (ROUGE-L) for both models at temperature 0 with top-10 retrieved examples are shown in Tables 2-3. In several cases, the generated impression exactly matched the reference (ROUGE-L = 1), reflecting consistent phrasing patterns in many CTPA impressions. Retrieval-selected few-shot examples helped both models capture this case-appropriate style more reliably. Lower-scoring examples illustrate where the generated impression diverged from the reference, highlighting clear targets for further prompt and retrieval refinement (Tables 2-3).

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. GPT-4o example cases for generated impressions against the ground truth, under temperature 0 and top-10 retrieved documents.

https://doi.org/10.1371/journal.pone.0354688.t002

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. LLaMA 3.1-70B example cases for generated impressions against the ground truth, under temperature 0 and top-10 retrieved documents.

https://doi.org/10.1371/journal.pone.0354688.t003

Review of the 50 lowest-scoring impressions per model showed different error profiles (S1 File: Supplemental Tables 9-12). GPT-4o failures were dominated by wrong prioritization or redundant details (37/50, 74%), whereas LLaMA 3.1-70B failures were dominated by truncated or malformed output (34/50, 68%). Hallucinations were observed in both models (GPT-4o: 5/50, 10%; LLaMA 3.1-70B: 9/50, 18%), while omissions were less common (GPT-4o: 3/50, 6%; LLaMA 3.1-70B: 4/50, 8%). Representative examples are provided in S1 File: Supplemental Table 12.

Discussion

In this study, we evaluated whether retrieval-based dynamic few-shot prompting could generate LLM-derived CTPA impressions that more closely resemble reference impressions. Across GPT-4o and LLaMA-3.1-70B, this method consistently outperformed zero-shot and fixed few-shot prompting, showing improved alignment on automated text-similarity metrics.

An impression is not a restatement of findings; it is the clinical answer, often compressed into a few lines that clinicians actually act on [1]. That makes impression generation harder than generic summarization and more sensitive to local style, ordering, and what a radiologist chooses to foreground. Zero-shot prompting has no way to lock onto that local style. Fixed few-shot helps, but it is blunt: the examples may be irrelevant to the case at hand. Dynamic retrieval changes the few-shots from “random examples” into “this is how we talk about this kind of case”, which is exactly the anchor a model needs when the target output is short and stylistically constrained.

Lower temperature and larger k produced the strongest alignment on text similarity metrics, consistent with more deterministic generation and more exposure to highly similar exemplars. The practical takeaway is simple: inference-time settings are not cosmetic, they materially shift automated similarity scores.

In the broader literature, this sits between two common approaches. Prior work has shown that zero-shot LLMs can generate radiology impressions, but performance is variable and often evaluated with subjective human scoring [12]. Fine-tuning can push performance further by teaching domain- and institution-specific patterns, as in PET impression generation [16], but it costs data, compute, and maintenance. Retrieval-based prompting offers a third path: adapt at inference time without retraining, while keeping the model closer to local reporting patterns through matched examples. This aligns with the growing focus on optimizing inference rather than only scaling models or training [17,23]. It also complements newer agent-style pipelines that pair retrieval with additional review steps to improve outputs beyond zero-shot baselines [24].

At the same time, the evaluation frame matters. ROUGE and BERTScore quantify overlap and semantic similarity, but they do not measure clinical correctness or safety; they can miss hallucinations, unsupported claims, and harmful omissions [21,22,25]. Because both retrieval and BERTScore rely on embeddings, BERTScore gains also deserve a careful read [22]. These metrics are useful for controlled benchmarking, but they do not answer the clinical question of “is this impression right?”. We did not measure end-to-end inference latency, retrieval overhead, token or compute cost, throughput, or implementation burden in clinical workflow. Accordingly, the present results speak to text-generation alignment rather than operational feasibility in real-world deployment.

This study has several limitations. It was retrospective, limited to a single exam type (CTPA), and evaluated only two LLMs. In addition, we did not perform human evaluation of the generated impressions, so we cannot determine whether higher similarity scores corresponded to better clinical quality, correctness, or safety. Evaluation relied exclusively on automated similarity metrics. ROUGE and BERTScore are useful for benchmarking textual overlap and semantic similarity to reference impressions, but they are not measures of diagnostic accuracy or clinical correctness. They do not reliably detect negation errors, clinically important omissions, unsupported statements, or harmful hallucinations. Accordingly, the improvements reported here should be interpreted as gains in textual alignment with reference impressions rather than evidence of clinical safety, diagnostic validity, or readiness for real-world deployment. In addition, the dataset was split at the report level rather than the patient level. Because some patients contributed multiple CTPA reports, strict independence between retrieval-bank and test reports may be reduced compared with a patient-level split, and this may have modestly favored retrieval of semantically similar prior cases. Finally, we did not assess latency, throughput, compute cost, or operational feasibility in clinical workflow; therefore, these findings should not be interpreted as evidence of deployment readiness.

In conclusion, dynamic, case-matched retrieval consistently improved alignment of LLM-generated CTPA impressions with reference impressions on automated text-similarity metrics. The largest gains were seen at temperature 0 and k = 10. Further prospective work with human evaluation will be needed to determine whether these improvements translate to clinically meaningful gains.

Supporting information