Dataset preparation

MBKBase (https://mbkbase.org/rice/knownGene) [23] was used as the reference database to annotate stress response genes, including those specifically associated with drought response. Out of 2416 identified abiotic stress response genes, 336 were labelled as drought-responsive in MBKBase. The remaining 2080 genes were treated as an operational background set. Gene labels indicating ‘associated’ or ‘non-associated’ with drought were obtained from the KnownGene module.The KnownGene module in MBKBASE provides the most recent functional annotations of experimentally validated alleles compiled from sources such as funRiceGenes, GenBank, Oryzabase and Q-Taro. However, these gene labels may change as new high-throughput studies become available. Drought-responsive genes represent a subset of abiotic stress response genes [24]. Hence, this study narrows the scope of ML on prioritising genes specifically associated with drought response from the broader abiotic stress context.

Subsequently, to implement the ML model, a total of 44 features were used. 28 from genomics, 14 from transcriptomics and two from proteomics (S1 Table). The methods used to generate these features are explained below.

Genomic features

Genomic features were obtained from publicly available repositories and publications. The QTG-Finder [5] repository provided 27 features used to rank genes associated with abiotic and biotic stress response. This dataset incorporates seven single nucleotide polymorphism features retrieved from Rice-SNP-Seek [25], 17 functional annotation features based on Gene Ontology, one co-function network descriptor from RiceNet [26] and two evolutionary features from [27]. Additionally, as the 28th feature, genes were labelled as expression quantitative trait locus (eQTL) based on [28] due to their role in response to abiotic stress factors.

Proteomics features

Protein-protein interactions (PPI) reflect gene interactions, based on the involvement of the encoded proteins in a specific biochemical pathway. A higher score correlates with stronger interactions [15]. PPI data for Oryza sativa was obtained from STRING-DB (version 12.0). Two features were extracted from the database, namely PPI_Counts_400 and PPI_Counts_700, which represent the number of interactions for each gene, where the score is above 400 and 700 respectively.

Transcriptomics features

Transcriptomic features were derived from RNA-seq data. The RNA-seq data deposited in the Sequence Read Archive (SRA NCBI) were manually searched to categorize drought-related experiments performed on Oryza sativa. Sequence accessions were obtained from these BioProjects. PRJNA248474, PRJNA420056, PRJNA608550, PRJNA608552, PRJNA667047. In total, the selected studies included 42 drought-sensitive samples and 48 drought-tolerant samples. Studies were selected based on the following criteria.

1. availability of a corresponding peer-reviewed publication;
2. a minimum of two biological replicates per condition.
3. inclusion of both drought-tolerant and drought-sensitive rice varieties.
4. complete and well-annotated metadata in the public repository.
5. generation of paired-end sequencing data.

A standardized pipeline was established to convert the raw sequencing data into gene counts. The raw data were downloaded as FASTQ files using SRAToolkit (V2.8.2). FastQC [29] was used to perform quality checks. Adapters and low-quality bases were trimmed using Trimmomatic (V0.36) [30]. High quality reads were retained and aligned to the reference genome sequence (IRGSP-1.0) using HISAT2 [31]. HISAT2 resulted in BAM files which were then sorted and quantified using featurecounts [32]. Each experiment was processed separately and subsequently combined for feature generation.

To eliminate batch effects arising from experiments conducted in different laboratories, ComBat-seq [33] from the SVA package [34] in R was employed. This method applies a negative binomial regression model that preserves the integer nature of RNA-seq count data, ensuring compatibility with standard differential expression tools that require integer input [35]. ComBat-seq was selected for its demonstrated robustness in correcting unwanted variation during integrative RNA-seq analyses, as supported by previous studies [36,37]. Principal component analysis (PCA) was performed to examine whether the samples showed apparent group separation in the reduced-dimensional space (S1 Fig).

Then, the gene counts from tolerant and sensitive varieties were grouped and processed separately with the DESeq2 [38] package in R to generate features based on differential gene expression. Four features were extracted using DESeq2. basemean, log2-fold-change, significant upregulation, and significant downregulation, for both tolerant and sensitive varieties, respectively. Significance thresholds were established to determine upregulated and downregulated genes. genes were considered significantly upregulated if the log2-fold-change exceeded 1.5 with an adjusted p-value below 0.05, and significantly downregulated if the log2-fold-change was below −1.5 with an adjusted p-value below 0.05. A total of eight differential expression (DE) features were generated.

Weighted Gene Co-expression Network Analysis (WGCNA) is a systems biology method that facilitates the dissection of the underlying biological mechanisms for a co-expressed gene set [39]. WGCNA shows correlated genes among modules associated with a specific trait or observation, which may share important biological regulatory roles [40]. From WGCNA, three features were derived, each individually processed for both tolerant and sensitive varieties, resulting in a total of six co-expression (CE) features.

It should be noted that all differential expression and co-expression feature extraction steps were performed globally across the entire processed RNA-seq dataset prior to dataset splitting and cross-validation fold generation. These transcriptomic features were constructed without using the gene-level MBKBASE drought-association labels. Nevertheless, because feature construction preceded gene-level splitting, genes subsequently held out contributed to the estimation of parts of the fixed feature representation.

In total, 17 new features were added to the QTG-Finder dataset, to repurpose it for the classification of genes associated with drought response. The newly added features are listed and described in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. List of newly generated features.

https://doi.org/10.1371/journal.pone.0360052.t001

Data balancing

Upon generating the dataset with all features, the proportion of ‘associated’ and ‘non-associated’ genes was evaluated. The ‘non-associated’ genes were excluded based on DESeq2 adjusted p-values obtained from drought tolerant and susceptible rice varieties. Lower adjusted p-values reflect greater significance in gene dysregulation following drought treatment [41]. Filtering genes based on their level of differential expression is a recommended strategy for optimizing data mining [8]. As a result, the ‘non-associated’ records in the datasets were excluded based on the increasingly stringent adjusted p-values of 1 × 10 ⁻¹, 1 × 10 ^− 2, 1 × 10 ^− 3, 1 × 10 ^− 4 and 1 × 10 ^− 5, at 10-fold decrements. Table 2 shows the different proportion of associated and non-associated genes after the removal of ‘non-associated’ genes.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Description of Dataset Dimensions.

https://doi.org/10.1371/journal.pone.0360052.t002

The baseline dataset, referred to as Dataset 1 (S1 Data), contained 86.1% ‘non-associated’ (N.As.) and 13.9% ‘associated’ (As.) genes, including all abiotic stress-responsive genes from MBKBASE. Severe class imbalance necessitated the derivation of five additional datasets with reduced ‘non-associated’ genes, guided by adjusted p-values from DESeq2. Only Dataset 6 achieved near balance, as shown in Table 2, while the Datasets 2–5 still showed imbalance. As an internal consistency check, a small holdout set was created by randomly isolating five ‘associated’ and five ‘non-associated’ genes. Since these genes were drawn from the same pool and share the globally constructed transcriptomic feature representation, this set is not an independent validation cohort, and its size (n = 10) precludes inference about generalizability to unseen genes.

Model development

The study employed four supervised ML algorithms for classification, two EL algorithms. Random Forest (RF) and XGBoost (XGB), and two DL algorithms. 1-Dimensional Convolutional Neural Network (1D-CNN) and Multilayer Perceptron (MLP).

The EL algorithms were implemented using the scikit-learn library, while DL models were developed using the TensorFlow and Keras libraries. Initial architecture and hyperparameters were selected based on the approach described by [12], with systematic optimization conducted on Dataset 6 before application to multiple datasets. Various configurations of dense, convolutional, and max-pooling layers, as well as regularization parameters and dropout layers, were evaluated to determine optimal performance.

The selected 1D-CNN architecture consisted of a convolutional layer with 64 filters and a kernel size of 3, using ReLU activation and accepting input with the shape of training features. ReLU activation was chosen due to its ability to introduce non-linearity into the model while avoiding the vanishing gradient problem, thus improving training efficiency and performance [42]. This was followed by a max-pooling layer with a pool size of 2 to reduce dimensionality, and a dropout layer (rate = 0.2) to prevent overfitting. The features were then flattened and passed through a dense layer with 64 neurons and ReLU activation, followed by another dropout layer (rate = 0.2). The final dense layer, the output layer, used Sigmoid activation, as it is deemed suitable for binary classification [43].

For the MLP model, a sequential architecture was implemented with three hidden layers. The first layer contained 64 neurons with ReLU activation and L2 regularization, followed by a dropout layer (rate = 0.2). The second hidden layer expanded to 128 neurons with ReLU activation and L2 regularization, also followed by a dropout layer (rate = 0.2). The third hidden layer was reduced back to 64 neurons with ReLU activation and L2 regularization. Finally, the output layer consisted of a single neuron with sigmoid activation for binary classification. Both DL models were compiled with the Adam optimizer, binary cross-entropy loss function, and accuracy as the training metric for optimization. Models were trained for 10 epochs with a batch size of 32 for performance evaluation. To examine learning behaviour and convergence, separate diagnostic runs with extended training up to 50 epochs were conducted to generate training and validation loss curves.

In contrast, hyperparameters for EL algorithms were fine-tuned after applying them onto the Dataset 6 to ensure the suitability of the algorithms first. A paired t-test was performed to check if there is a significant improvement due to hyperparameter tuning.

Additionally, simpler baseline models were also tested, which includes Logistic Regression, Support Vector Machines, K-Nearest Neighbour and Naïve Bayes. However, results of baseline models were omitted due to poor performance.

Model evaluation

The models were evaluated using a nested cross-validation (NCV) framework, with performance assessed through metrics including accuracy, precision, recall, F1 score, and Area Under Curve – Receiver Operating Curve (AUC-ROC). NCV, with an inner loop for model selection and an outer loop for evaluation, reduces overfitting risks and provides more reliable performance estimates [44]. The inner loop was set to three-folds, whereas the outer loop was set to five-folds. Accuracy reflects the percentage of correctly classified data, while precision measures the percentage of correctly classified positive instances relative to all instances classified as positive. Recall denotes the ratio of positively classified instances to all actual positive instances, and the F1 score provides a balanced assessment of precision and recall. AUC-ROC evaluates the model’s ability to distinguish between classes, with values closer to 1 indicating better performance. Disparity between precision and recall indicates bias in performance due to class imbalance, which lowers F1 score [45].

The NCV was employed to limit leakage during model evaluation, but the scheme operates on a feature matrix assembled before gene-level splitting. The reported metrics are therefore estimates of classifier performance conditional on this fixed feature representation, rather than estimates of the end-to-end pipeline comprising feature construction and classification. All performance values reported in this study should be interpreted in that restricted sense.

To ensure reliability, the models underwent 20 repetitions of training, with adjustments to the random.seed number each time. This iterative approach mitigates the impact of randomness inherent in ML algorithms on model performance [46]. The performance measures reported is the average of 20 repetitions and along with its standard deviation. Standard deviation quantifies the average deviation of performance measures from their mean, offering insight into the spread around the mean value.

Performance stability was further assessed using three evaluation methods namely nested cross-validation method (NCV), the holdout method and K-fold cross-validation. A stable model is expected to produce results with minimal discrepancy across these different evaluation methods [47,48].

A sensitivity analysis was performed using 10 alternative datasets, each assembled by randomly sampling “non-associated genes” from the remaining non-associated pool, while completely excluding the original Dataset 6 negatives. Each dataset maintained the class proportions of Dataset 6.

Feature importance analysis

Upon identifying suitable algorithms, feature importance scores were obtained using wrapper methods. These scores were averaged across 20 repetitions with different random seeds to ensure result stability. Feature importance was assessed both before and after hyperparameter tuning to evaluate the impact of optimization on feature selection.

Following the identification of important features, an ablation study was conducted to systematically evaluate the contribution of different omics components to model performance. In this study, omics components (DE, CE, PPI, and genomic features) were iteratively removed from the feature set, and models were retrained and evaluated using the same performance metrics. This approach allowed for the assessment of each omics component’s relative contribution to the model’s predictive power, providing insights into which data types were most informative for identifying drought-responsive genes.

Functional enrichment analysis

Based on the suitable hyperparameters for the best performing classification algorithm, the genes are ranked using classifier generated probabilistic scoring. ShinyGO (http://bioinformatics.sdstate.edu/go/) [49] was used for the Gene Ontology (GO) terms to retrieve biological process (BP) annotations of the top 5% ranked genes. A statistical threshold (FDR ≤ 0.05) was applied to identify functional annotations significantly enriched by the queried genes.

Multi-omics Dataset

Dataset preparation is a crucial step in developing ML models to address complex questions. Features should be representative to enable supervised ML algorithms to distinguish between labels [16]. A total of 90 RNA-seq samples (1.4 TB) were processed to generate gene counts data, and 44 features were derived from genomics, transcriptomics, and proteomics data (Table 1).

Model performance

Four models were used to classify drought-associated genes based on 44 multi-omics features. The best performance was achieved with Dataset 6 (Fig 2) (S2 Table), shows the importance of using balanced datasets. In contrast, Datasets 1–5 showed high precision but low recall due to class imbalance, which led to the misclassification of minority class instances [50]. A threshold value of 0.60 for performance measures is indicated by the red line in the following figures.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Performance measures of machine learning algorithms on six different datasets (DS1-DS6) using nested cross-validation.

(A) Accuracy. (B) Precision. (C) Recall. (D) F1-Score. (E) AUC-ROC.

https://doi.org/10.1371/journal.pone.0360052.g002

RF and XGB consistently performed well, achieving scores above 0.60 across all performance measures, indicating their suitability for gene classification. Tree-based ensemble learning (EL) methods effectively handled dataset complexity, as previously reported [11,19]. Based on NCV, RF and XGB demonstrated robust classification performance on Dataset 6, with recall values of 0.62 and 0.70, respectively, and both achieving AUC-ROC above 0.86 (S2 Table). The superior recall of XGB suggests greater sensitivity in identifying ‘associated’ genes, likely due to its built-in regularization, which prevents overfitting by controlling tree complexity and reducing variance, as observed in previous studies [51]. RF, on the other hand, leverages the aggregation of multiple decision trees, making it inherently resistant to overfitting by averaging predictions and reducing bias [52]. The consistent performance of these ensemble models highlights their ability to handle complex, multi-omics datasets, unlike CNN and MLP, which struggled with feature extraction and learning stability. These findings suggest that EL methods are better suited for datasets of this size and complexity, as they effectively mitigate the risk of overfitting, which is often exacerbated in smaller datasets [13].

Conversely, the 1D-CNN and MLP underperformed, with recall values falling below 0.60, indicating a high rate of false negatives, despite the use of a balanced dataset. Recall is particularly important as it measures correctly classified ‘associated’ genes [53]. Limited dataset size likely constrained deep learning models’ generalizability, as reported in previous studies [51,54]. Training-validation loss plots over 50 epochs revealed early overfitting in 1D-CNN (Fig 3A), while MLP exhibited underfitting by slow learning progress (Fig 3B). The 1D-CNN showed early overfitting based on the training–validation loss curves exhibiting difficulty in extracting relevant features from the data [55]. In contrast, the MLP model showed limited performance improvement across epochs, indicating underfitting under the evaluated configuration [56]. These observations further reinforce the suitability of tree-based models for handling complex, multi-omics datasets.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Training and Validation Loss Plot of (A) 1-Dimensional Convolutional Neural Network and (B) Multi-layer Perceptron.

The 1D-CNN showed early overfitting, while MLP showed underfitting across 50 epochs.

https://doi.org/10.1371/journal.pone.0360052.g003

Stability analysis using three evaluation approaches (Table 3) showed that the tree-based models, RF and XGB, exhibited minimal performance fluctuations, while the DL models showed notable instability. A similar observation was reported [48], where tree-based model demonstrated consistent performance across multiple evaluation approaches, particularly with small dataset. Meanwhile, the F1-score of 1D-CNN varied significantly across approaches. 0.54 ± 0.11 (NCV), 0.74 ± 0.01 (Holdout), and 0.57 ± 0.10 (CV). Despite hyperparameter tuning, the DL models continued to underperform, underscoring the higher within-dataset performance and stability of RF and XGB. XGB’s built-in regularization helped mitigate overfitting, while RF’s ensemble architecture made it inherently more resilient to noise and variance [54]. The findings suggest that EL methods are better suited for this dataset, offering higher discriminative performance and stability under the evaluated fixed-feature setting.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Performance stability of ML algorithms tested with three different evaluation approaches.

https://doi.org/10.1371/journal.pone.0360052.t003

Hyperparameter tuning

Despite GridSearchCV is commonly used for hyperparameter optimization within the NCV framework, it was not adopted in this study. This is because GridSearchCV selects hyperparameters based on performance tied to a specific random.seed which can bias the results [48]. As a result, a manual grid search was performed within the two-tiered nested approach, with tested parameters as listed in Table 4. The optimal parameters were identified by averaging performance measures over 20 iterations. The combination that yielded the highest mean values in accuracy, F1-score, and AUC-ROC is presented in Table 4.

Download:

• PNG
  larger image
• TIFF
  original image

Table 4. Hyperparameters used for tuning the XGBoost (XGB) and Random Forest (RF) models.

https://doi.org/10.1371/journal.pone.0360052.t004

To validate the effectiveness of hyperparameter tuning, a t-test was conducted to compare the performance of fine-tuned models against their default parameter settings. The results indicated that fine-tuned XGB and RF models achieved significantly higher overall performance. Although accuracy, F1-score, and AUC-ROC improved for XGB, recall slightly decreased from 0.70 to 0.67. In contrast, the fine-tuned RF model outperformed XGB, with a notable increase in recall (Table 5).

Download:

• PNG
  larger image
• TIFF
  original image

Table 5. The Performance of Ensemble Algorithms with Default and Fine-tuned Parameters are Compared with T-Test.

https://doi.org/10.1371/journal.pone.0360052.t005

The reduction in XGB’s recall may be attributed to the lower learning rate and subsample ratio, combined with an increased max_depth, which can elevate the risk of overfitting [57]. However, this effect was counterbalanced by a higher value for the lambda regularization parameter, making the model more conservative [58]. In RF, although increasing max_depth can also lead to overfitting [59], the addition of more trees and a higher minimum sample split threshold improved generalization [60]. The max_features parameter was set to ‘sqrt’, allowing the algorithm to automatically determine the optimal number of features at each split [61]. Further analysis of feature contributions is recommended to better understand the impact.

Across repeated evaluations of the small internal holdout, mean accuracy, recall and AUC-ROC were 0.90, 0.63 and 1.00, respectively. Given the holdout size of only five associated and five operational-background genes, these estimates are highly uncertain. They are reported only as an internal consistency check and neither establish generalizability nor distinguish sample-size effects from overfitting. Furthermore, the resampled datasets used for sensitivity analysis achieved accuracy, precision, recall and AUC-ROC of 0.89, 0.96, 0.76 and 0.93 respectively. These results indicate that the observed classifier performance was not unique to the particular negative sample used in Dataset 6. This analysis varies only the composition of the negative class; it does not quantify possible information sharing arising from global feature construction.

Feature importance analysis

Feature importance was assessed for both RF and XGB before and after hyperparameter tuning (S4 Table). In RF, the most influential features primarily included the distribution, magnitude, and direction of gene expression derived from RNA-seq data in tolerant and sensitive samples, along with PPI_Counts_400 and network_weight features. Following fine-tuning, PPI_Counts_700 replaced paralog_copy_number in the importance ranking, although the latter remained within the top 15 features.

For XGB, the features is_inorganic_nutrients_metabolism and hub_genes_dr_Sen were consistently ranked among the top 10, along with other features also prioritised by RF. However, several features including is_redox_metabolism, is_receptor, is_start_lost, is_stop_lost, and is_splice_site_SNP, were completely disregarded by XGB, receiving zero importance scores across all 20 iterations. After hyperparameter tuning, an additional five features. mod_high_corr_drought_Tol, hub_genes_dr_Tol, is_nucleotides_metabolism, is_fatty_acids_lipids_metabolism, and is_cofactors_metabolism, were also excluded from the model. These results suggest that the fine-tuned RF model is less prone to noise and can make accurate predictions without entirely discarding any input features.

Ablation study

The ablation analysis for RF (Table 6) revealed that removing any feature group resulted in a decline in model performance, with the most significant drop observed when all features except genomics were excluded. The complete model comprising DE, CE, PPI, and genomics achieved the highest accuracy (0.83) and AUC-ROC (0.87), highlighting the importance of multi-omics integration. Removing CE or PPI alone had minimal impact, however, excluding DE led to a more noticeable decline in performance. The removal of both DE and CE further reduced recall compared to excluding DE alone. Notably, the lowest recall (0.23) was observed when only CE feature was used, suggesting its limited standalone predictive power. While CE may not be effective on its own for gene classification, it contributes meaningfully in combination with other features. These results suggest that DE contribute significantly to model robustness. The findings are consistent with the feature importance analysis, indicating that DE-derived features were highly informative for model predictions.

Download:

• PNG
  larger image
• TIFF
  original image

Table 6. Ablation Study on Random Forest.

https://doi.org/10.1371/journal.pone.0360052.t006

Functional enrichment analysis

As shown in the Fig 4 below, the BP GO terms reflect drought-response related terms, such as ‘response to water deprivation’ and ‘response to water’. This shows that drought-response genes were highly enriched. Subsequently, the classification pipeline has the potential to distinguish drought-responsive genes.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Enriched BP GO terms of the top 5% genes from RF.

GO terms associated with drought such as ‘response to water deprivation’ and ‘response to water’ have been enriched in higher folds.

https://doi.org/10.1371/journal.pone.0360052.g004