Background and significance

Adverse drug events (ADEs) refer to any physical or psychological injuries and unexpected events caused by medication use [1], and can be significant hospital complications impacting patient experience and costs [2]. It is challenging to detect ADEs at an early stage due to ambiguous and incomprehensive information about actions, symptoms and medications. Thus, information extracted from electronic health records (EHRs), including diagnoses, clinical notes, and laboratory tests, becomes important for supporting early treatment and primary and secondary prevention.

However, extracting relevant information from unstructured clinical notes is challenging. Over the past few decades, several approaches have been used including rule-based, machine learning-based, deep learning-based, and contextualized language model-based approaches [3]. More recently, contextualized language models have been widely applied in natural language processing (NLP). These models are pre-trained on large corpora and can effectively capture contextual patterns in the relevant domain, thereby facilitating their application in downstream tasks. The Bidirectional Encoder Representations from Transformer (BERT) model [4,5], one of the most widely adopted contextualized models, was first introduced in 2018. It employs a transformer architecture to pre-train a deep neural network on a large corpus of unlabeled text data using an unsupervised approach. Since BERT has continuously evolved, BERT-based NLP systems have shown great promise for ADE-related tasks, such as relation extraction and text classification [3,6]. Fan et al. [7] conducted a comparative analysis between BERT-based models, standard deep learning models, and current state-of-the-art models for ADE detection and extraction, and demonstrated that a BERT-based model achieved new state-of-the-art results. Hussain et al. [8] proposed an end-to-end system for adverse drug relation detection by fine-tuning BERT, showing good performance on Twitter and biomedical literature datasets (e.g., PubMed-indexed abstracts). The introduction of the SpanBERT (Span-based BERT) architecture [9] for ADE extraction task outperformed competing models [10] on the Social Media Mining for Health (SMM4H) and Adverse Drug Event Corpus (CADEC) datasets developed by the Commonwealth Scientific and Industrial Research Organisation (CSIRO) [11,12]. Narayanan et al. [13] evaluated various biomedical contextual embeddings and models using the 2018 National NLP Clinical Challenges (n2c2) shared task Track 2 dataset on ADEs and medication extraction [14], demonstrating the great potential of the BERT architecture. In recent work on adverse event (AE) detection, Chopard et al. [15] confirmed the feasibility of automating the coding of AEs described in the narrative section of serious AE report forms. Furthermore, Silverman et al. [16] demonstrated that large language models (LLMs), such as University of California, San Francisco (UCSF)-BERT, achieved higher accuracy in detecting serious AEs from clinical notes compared with previous methods. A review [17] of machine learning and deep learning approaches for ADE extraction using benchmark datasets, including n2c2 [14], highlighted BERT’s superior performance in end-to-end tasks. A scoping review by Golder et al. [18] demonstrated the potential of NLP for identifying pediatric ADEs from clinical notes, while also highlighting challenges related to methodological heterogeneity, limited external validation, and inconsistent reporting. A narrative review by Zitu et al. [19] summarized the current state of LLMs, including BERT and the Generative Pretrained Transformer (GPT) models, in ADE detection, showing that LLM-based methods often outperform traditional approaches, although limitations such as performance variability and interpretability challenges remain.

Despite the good performance of BERT model and its variants (hereafter referred to as BERT Short-Formers, in contrast to Longformer [20]), one limitation is that long texts may be truncated due to token limits. To mitigate this limitation, Longformer [20], a BERT variant that can handle long text sequences, was developed. The key innovation of Longformer is its attention mechanism, which considers window-based local-context self-attention and global attention to achieve powerful contextual and sequential representations.

Our work began with exploring effective data processing methods for extracting information from clinical notes in EHRs for transformer-based models, ultimately aiming to develop an NLP system for detecting ADEs in clinical notes. To achieve this goal, we systematically investigated two data processing methods, called window-based and split-based approaches, to generate model inputs for fine-tuned BERT Short-Formers (BERT-base-uncased [5], a general-domain pretrained language model; BioMed-RoBERTa [21], a RoBERTa-based model further pre-trained on biomedical text; BioClinicalBERT [22], a BERT model further pre-trained on both biomedical literature and clinical notes; BioBERT [23], a BERT model further pre-trained on biomedical literature; PubMedBERT [24], a BERT model pre-trained on PubMed abstracts and full-text biomedical articles; and SpanBERT [9], a BERT model pre-trained with span-based masking objectives), as well as Clinical-Longformer [25,26], a Longformer-based model further pre-trained on clinical text, using the n2c2 dataset. We then applied the optimal processing and modeling methods to a dataset extracted from Vanderbilt University Medical Center (VUMC) EHRs to further assess generalizability. The two datasets contain markedly different types of clinical notes: the n2c2 dataset comprises fewer but longer notes, with a higher number of ADEs derived from hundreds of drug mentions, whereas the VUMC dataset includes more notes of shorter length, featuring more homogeneous ADEs focused on two drugs of interest. By applying these differing approaches to two distinct datasets, we aim to provide guidance for identifying ADEs from clinical notes using transformer-based models.

Materials and methods

Data processing

Pre-trained transformer models can only take a limited length of free text, typically up to 512 tokens, whereas the Longformer can take up to 4,096 tokens. As clinical notes are typically long and unstructured, requiring preprocessing prior to modeling. We investigated several processing methods, ultimately focusing on the following two.

Window-based approach.

When drug names of interest are already annotated or can be easily located, this approach can be useful to effectively capture the contextual details surrounding drug names. It extracts a specific number of words before and after the drug name, called window size, by leveraging whitespace-based tokenization; for example, 20 words before and after vancomycin, a drug of interest in the n2c2 dataset. The window sizes we investigated include 10, 15, 20, 50, and 100 words. If the number of words before or after the drug name was insufficient, we extracted words up to the beginning or end of the note.

Split-based approach.

We explored a simpler alternative for settings where drug names are not yet annotated, as identifying them requires substantial effort. This approach exclusively partitions clinical notes into predefined chunk sizes, such as 2, 4, 10, and 20 chunks. We first used the Natural Language Toolkit (NLTK) tokenizer [35] to tokenize the text into sentences, and then divided the sentences into predefined chunk size. If the token length of a particular chunk exceeded a predefined token limit (e.g., 4,096 for Longformer), we further divided it into two smaller chunks at the word level. This additional step ensured that the chunks remained within the desired token limit, preventing truncation of context. Thereafter, no further splitting was required as most chunks were within the token limit. However, a small number of chunks still exceeded the token limit due to missing punctuation, especially at the beginning or end of some clinical notes.

For both approaches, newline characters were replaced with spaces to facilitate processing. A schematic diagram illustrating the data processing steps is shown in Fig 1. Examples of each strategy are provided in the Supporting Information.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic diagram of data processing steps.

Window-based approach: extracts a predefined number of words before and after annotated drug names. Split at the sentence level: divides clinical notes into predefined chunk sizes at sentence boundaries. Split at the word level: subdivides any chunk exceeding the token limit into two smaller chunks at word boundaries.

https://doi.org/10.1371/journal.pone.0350516.g001

Results

Table 1 presents the results of the fine-tuned BERT-base-uncased model on the n2c2 dataset using varying window sizes. Window sizes of 10–20 words performed well, with a 15-word window achieving the best results (mean ± SD: AUPRC 0.670 ± 0.006, micro F1 0.963 ± 0.001, macro F1 0.804 ± 0.005). This suggests that a 15-word window around the drug name provides sufficient context about ADEs associated with the drug. Our exploratory analysis regarding the distance between drug names and ADEs supported that a 15-word window covered approximately 80% of ADEs. Wider window sizes yielded poorer performance, likely due to the inclusion of longer and more complex contextual information (e.g., medical history and prescriptions), which may introduce noise. Although broader windows capture more context, they do not necessarily improve model performance. Conversely, excessively small windows may exclude ADEs. The SDs of the performance metrics were small across window sizes, indicating stable results. Given that our goal is to identify as many positive cases as possible, a moderate window size offers a better balance.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Effects of window size on BERT-base-uncased model performance using the n2c2 dataset.

https://doi.org/10.1371/journal.pone.0350516.t001

To leverage domain-specific knowledge, we fine-tuned several BERT Short-Formers pre-trained on biomedical corpora using the optimal 15-word window. Performance was evaluated on the n2c2 dataset (Table 2). BioClinicalBERT, BioBERT, PubMedBERT, and Clinical-Longformer demonstrated strong performance. Among them, PubMedBERT achieved the highest AUPRC (0.698 ± 0.018), followed by Clinical-Longformer (0.689 ± 0.058). These findings suggest that biomedical pre-training enhances a model’s ability to capture domain-specific information and improves classification performance. Therefore, in the subsequent evaluation, we selected PubMedBERT as the best-performing BERT Short-Former, BERT-base-uncased as the baseline, and Clinical-Longformer as the representative Longformer model.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Model comparison with window size of 15 words using the n2c2 dataset.

https://doi.org/10.1371/journal.pone.0350516.t002

The results of text partitioning for the two selected BERT Short-Formers and Clinical-Longformer on the n2c2 dataset are presented in Table 3. The number of chunks (i.e., chunk size) substantially affects model performance by determining the extent of text truncation imposed by token limits. Clinical-Longformer performed consistently well across most configurations: the 4-chunk setting achieved the best AUPRC (0.840 ± 0.056), the 20-chunk setting obtained the best micro F1 (0.939 ± 0.006), and the 10-chunk setting attained the best macro F1 (0.852 ± 0.007). We also compared two BERT Short-Formers and Clinical-Longformer using 10- and 20-chunk configurations on the n2c2 dataset. These chunk sizes were selected because most chunks remained within the token limit of BERT Short-Formers, given that the 95th percentile word count in the n2c2 training set was approximately 1,368. Results for 2- and 4-chunk settings were not reported for the BERT Short-Formers as most segments would exceed the token limits. With 10 chunks, the models achieved AUPRCs of 0.635 ± 0.060 (BERT-base-uncased), 0.745 ± 0.028 (PubMedBERT), and 0.819 ± 0.017 (Clinical-Longformer). With 20 chunks, the corresponding AUPRCs were 0.624 ± 0.023 (BERT-base-uncased), 0.733 ± 0.023 (PubMedBERT), and 0.757 ± 0.043 (Clinical-Longformer), respectively. The SDs for Clinical-Longformer with the 2- and 4-chunk settings were slightly higher than those of the other configurations, although the differences were modest. Overall, these findings highlighted the importance of selecting an appropriate partitioning strategy to preserve context integrity and optimize model performance.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Effects of split chunk size on model performance using the n2c2 dataset.

https://doi.org/10.1371/journal.pone.0350516.t003

We further conducted statistical testing using the Wilcoxon signed-rank test on paired results from 5-fold cross-validation to compare Clinical-Longformer (4-chunk setting) and PubMedBERT (10-chunk setting), which achieved the highest AUPRCs. The resulting p-values were 0.0625 for AUPRC, 0.0625 for micro F1, and 0.4375 for macro F1, indicating no statistically significant differences at the level of 0.05. However, the small number of folds (i.e., limited sample size) constrained the statistical power of these tests. Notably, for a paired sample size of 5 with no ties or zero differences, the minimum achievable p-value under a two-sided exact Wilcoxon signed-rank test is 0.0625. Thus, even if all paired AUPRC and macro F1 values for Clinical-Longformer exceeded those of PubMedBERT, statistical significance could not be reached. To further examine this limitation, we conducted additional analysis using 10-fold cross-validation. This yielded p-values of 0.0020 for AUPRC, 0.0020 for micro F1, and 0.1055 for macro F1. In this setting, 0.0020 represents the minimum possible p-value for a paired sample size of 10, consistent with all AUPRC values for Clinical-Longformer exceeding those of PubMedBERT. These findings suggested a potential performance difference between the models. However, we did not adopt 10-fold cross validation for all analyses, as it resulted in smaller validation sets, which may be suboptimal given the sample sizes. Therefore, we retained 5-fold cross-validation throughout and reported mean and SD values, without additional formal statistical testing due to the limited statistical power.

Table 4 shows the performance of the two selected BERT Short-Formers and Clinical-Longformer for classifying ADEs in the VUMC dataset. As clinical notes in the VUMC dataset are relatively short, we evaluated chunk sizes of 1 (i.e., entire note, so no split), 2, 4 and 6 chunks. The results of 1-chunk with BERT Short-Formers are not reported due to convergence issues caused by token limit exceedance. Although most VUMC notes fall within the token limit of Longformer, the 2-chunk, 4-chunk, and 6-chunk outperformed 1-chunk. This suggests that moderately partitioned text may provide more focused contextual information for ADE detection. Similar trends were obtained with BERT-base-uncased and PubMedBERT. Furthermore, Clinical-Longformer substantially outperformed BERT Short-Formers, likely due to its ability to process much longer token sequences and preserve more complete contextual information. However, increasing the number of chunks to 6 did not consistently improve performance across all models. Further partitioning may increase the chance that the ADE and the corresponding drug name are placed in different chunks, fragmenting contextual information. Excessive partitioning can therefore reduce performance by limiting a model’s ability to capture comprehensive context and introducing additional complexity.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Effects of split chunk size on model performance using the VUMC dataset.

https://doi.org/10.1371/journal.pone.0350516.t004

Discussion

We fine-tuned pre-trained transformer-based models for detecting ADEs from two distinct datasets. Models pre-trained on biomedical corpora demonstrated strong capability in ADE classification tasks by leveraging domain-specific knowledge. Among BERT Short-Formers, PubMedBERT achieved the best performance (AUPRC = 0.745, micro F1 = 0.963, macro F1 = 0.823 in the best-performing setting) on the n2c2 dataset. Clinical-Longformer generally outperformed BERT Short-Formers across most metrics on the n2c2 dataset (AUPRC = 0.840, micro F1 = 0.963, macro F1 = 0.852 in the best-performing setting) and the VUMC dataset (AUPRC = 0.536, micro F1 = 0.963, macro F1 = 0.762 in the best-performing setting), suggesting that the ability to model longer clinical context further enhances performance. To the best of our knowledge, this is the first study to fine-tune the Longformer model for ADE detection, demonstrating its applicability to clinical NLP tasks.

Because transformer models are constrained by token limits, we systematically evaluated window-based and split-based text processing strategies. The window-based approach performed optimally with a 15-word window, indicating that moderate contextual scope effectively balances information retention and noise reduction. The split-based approach was introduced as a practical alternative that does not require annotated drug names, as such annotation demands significant effort. We found that chunk size significantly affected performance, even when note length was within the token limit. Moderate partitioning improved results, whereas excessive partitioning reduced performance, likely due to fragmentation of contextual information. Small SDs across folds indicated stable model performance. Comparisons between architectures further highlighted important differences. BERT-base-uncased performed better with the window-based processing approach, likely due to its focused contextual learning around drug mentions. Clinical-Longformer benefited more from split-based processing, consistent with its design for longer sequences. These findings emphasize that optimal ADE detection depends on the joint selection of text partitioning and model architecture strategies.

There are a few prior studies on the n2c2 dataset focused on ADE-Drug relation using BERT-based models. Our models achieved similar or improved performance (micro F1 score of 0.96 on BERT-base-uncased, 0.97 on BioClinicalBERT and BioBERT) compared to previously reported results by Wei et al. (micro F1 score of 0.80 on BERT-large-uncased and 0.81 on MIMIC BERT) [5,6,37], and Mahendran et al. (micro F1 scores of 0.97 on BERT-base-uncased, BioClinicalBERT and BioBERT) [3]. In a recent work, Dai et al. [38] reported micro F1 scores (SDs) of 0.25, 0.52 (0.01), and 0.15 (0.22) on GPT-4, fine-tuned span-based models, and fine-tuned generative models, respectively, including generative artificial intelligence (AI) based approaches. Due to substantial class imbalance, we used AUPRC and macro metrics as the primary evaluation measures, as they better reflect performance on the minority class than micro F1 that is equivalent to accuracy in binary classification. Micro F1 scores were uniformly high and did not meaningfully differentiate model performance.

On institutional datasets, Silverman et al. [16] reported accuracy of 0.88–0.92 and macro F1 of 0.61–0.68 in drug-serious AE pair identification from annotated outpatient clinical notes at UCSF using Hierarchical UCSF BERT (H-UCSF-BERT). In comparison, our results on the VUMC dataset demonstrated improved performance with an accuracy of 0.97 using BERT-base-uncased, and a macro F1 of 0.76 using Clinical-Longformer. It is important to note that their notes focused on 8 non-steroid immunosuppressants, whereas our study using the VUMC dataset focused specifically on AEs associated with two medications, citalopram and escitalopram. Although we also evaluated our approaches on the n2c2 dataset, which contains a broader set of drug-ADE relations, direct comparison was not possible because Silverman et al. did not report H-UCSF-BERT results on the n2c2 dataset.

When using non-default attention windows with Clinical-Longformer, we occasionally observed training instability. In some runs, the model failed to converge during the early epochs despite adjustments to the warmup ratio and learning rate. In other cases, models converged but exhibited lower validation and test performance and greater variability across runs. These effects may be related to disrupted gradient flow when attention windows are too small, attention sink behavior where certain tokens absorb disproportionate attention, or task-specific threshold effects when local context becomes insufficient. Overall, this suggests that the default attention window is empirically tuned for stable training, and non-default configurations may require substantial additional tuning. Nevertheless, transformer-based models, particularly Clinical-Longformer, remain valuable for facilitating the extraction of meaningful signals from clinical notes, with their advantages outweighing potential challenges.

This study has several limitations. First, limited hyperparameter tuning of the transformer-based models may have constrained full exploration of their performance. Future work could investigate a broader range of training configurations. Second, this study utilized the n2c2 dataset and a single institution-specific dataset. These two datasets are inherently distinct, reflecting substantially different clinical contexts, documentation practices, medication coverage, and patient populations, and thus provide meaningful heterogeneity that supports the assessment of model generalizability across varied healthcare settings. Despite that, validation across additional datasets, institutions, and drug classes remains warranted, particularly for medications with rare or heterogeneous AE profiles. Institution-specific models such as UCSF-BERT, if made publicly available, represent promising avenues for future comparative studies, as they may offer further insight into model performance across diverse institutional note structures and patient populations, thereby strengthening the assessment of generalizability. Third, potential annotation bias may have introduced variability into the labeled data due to differences among annotators, training procedures, and human error. Finally, although we observed qualitative trade-offs between human annotation effort (i.e., the requirement for window-based vs. no requirement for split-based processing) and computational resources (e.g., runtime for BERT Short-Formers vs. Longformer), quantitative comparison was not feasible due to the lack of systematically recorded annotation time and computational costs.

Conclusion

In this study, we presented transformer-based models for detecting ADEs as well as different data processing methods and their impact on model performance. Window-based processing approaches rely heavily on annotations, which is labor-intensive, but can build precise context when the distances between drug names and their associated ADEs in the dataset are relatively consistent. Split-based processing approaches can easily split long texts into smaller chunks that transformer models can handle and do not require labor-intensive annotation for drug-ADE relations. However, chunk size determines the proportion of chunks that exceed the token limit (Tables 3 and 4), thereby affecting the classification task, and identifying the optimal chunk size can be challenging. The Longformer model requires more extensive fine-tuning, including but not limited to memory, runtime and hyperparameter settings, while for BERT Short-Formers, token limit is the major challenge.

Although our results are promising, the limitations highlight directions for further research and improvement to enhance the applicability and generalizability of transformer-based NLP models for ADE detection in real-world healthcare systems.

For future work, we aim to extend our findings to additional datasets and a broader range of drugs, enabling more comprehensive validation across diverse clinical scenarios. To improve the extraction of ADE-related information from EHRs, we plan to incorporate additional elements such as symptoms, actions, and dose changes, which may enhance understanding of drug-ADE relations and support clinical decision-making. Moreover, data augmentation strategies such as paraphrasing, synthetic data generation, oversampling, along with advanced long-document modeling techniques, including hybrid approaches that combine window- and split-based approaches, retrieval-augmented methods, and dual-pass architectures, could be explored to more effectively capture contextual information for ADE tasks. Continued methodological development in this area may contribute to more robust NLP-based ADE detection systems and ultimately support improvements in patient safety and health outcomes.

Supporting information

Acknowledgments

We thank Sara L. Van Driest and Ida Aka for providing the VUMC dataset and clinical context, which helped the annotation process.