2.1 Software packages and tools

The code developed during this research was implemented in Python, running on a Linux platform and employing bioinformatics, data analysis, and ML packages. Specifically, the bioinformatics packages include ViennaRNA (v2.5.0) [35], Biopython (v1.80) [36], bowite2 (V2.3.5.1) [37], cutadapt (v3.5) [38] and NCBI Blast [39]; the data analysis packages include pandas (v1.3.4) [40] and numpy (v1.21.5) [41]; ML packages include scikit-learn (v1.0.2) [42], XGBoost (v1.7.1) [43], and the SHAP values packages (v0.41.0) [44].

2.2 Definitions

For each MTI, we define the following terms regarding the target molecule: fragment is the target region that is obtained in experiment (e.g., CLASH, see below) and is used to calculate the interaction; site is the actual binding region within the fragment obtained by duplex calculation; full mRNA refers to the 3’UTR sequence that contains the target site; interaction duplex represents the binding pattern between a miRNA and a site sequence and is calculated by RNAduplex tool from ViennaRNA suite [27]. We classified the duplexes based on their seed type: canonical seed, non-canonical seed, or other. Canonical seed interactions are defined as exact Watson-Crick pairing in positions 2–7 or 3–8 of the miRNA. Non-canonical seed interactions may contain GU base pairs and up to one bulged or mismatched nucleotide at these positions [12]. Given the existing focus of works in this field on canonical and non-canonical interactions, we opted to specifically explore these two seed types of interactions, collectively referred to as valid in our study. Our objective was to analyze high-quality interactions, ensuring that negative interactions are comparable to positive interactions and not overly artificial. Consequently, for negative interactions, we considered only valid interactions, as well.

2.3 Data retrieval and processing

Precursor and mature human miRNA sequences were downloaded from miRBase (releases 17–22) [45]. Human 3’UTR sequences were downloaded from the Ensembl Biomart database [46], as in our study, only interactions that fall in 3’UTR are considered.

CLASH

CLASH-based techniques, that are derived from AGO-CLIP methods, use an extra step to covalently ligate the 3’ end of a miRNA and the 5’ end of the associated target RNA within the miRISC. Subsequent cloning and sequencing of isolated chimeric miRNA–target reads facilitate the identification of direct MTIs. The portion of chimeric reads in such sequencing data is about 2%. The rest of the reads represent separate miRNAs and mRNA target fragments, as in CLIP studies. In CLIP datasets, determining the specific miRNA involved in each interaction relies on bioinformatic predictions [47, 48].

We denote by CLASH, the high-throughput datasets composed of chimeric MTIs retrieved by methods e.g., CLASH, CLEAR-CLIP. In our study, we used four such datasets generated from human cells (h1-h4, Table 1); where each dataset contains miRNA–fragment pairs. All datasets were processed as described in [23], considering only interactions that involved the 3’UTR region. The processing involved converting the datasets into a consistent format where each record includes: miRNA name, miRNA sequence, target fragment sequence, full mRNA sequence, and the fragment’s coordinates within the mRNA.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. High-throughput datasets of human chimeric MTIs.

https://doi.org/10.1371/journal.pcbi.1012385.t001

CLIP

We downloaded the complementary CLIP data for the dataset h3, generated by CLEAR-CLIP experiment in Huh-7.5 cells [13] (NCBI GEO database series GSE73059). This data provides separate miRNA and mRNA sequences associated with AGO complexes, such that the interacting miRNA-mRNA pairs are unknown. We developed a pipeline to match the sequences to the respective names of miRNAs and mRNAs. First, we removed the adapter sequences (as in [13]) and filtered out reads shorter than 18nts. Next, we created two index databases for 3’UTR sequences and miRNA precursor sequences using bowtie2 [37], and aligned the reads against the databases (resulting in miRNA.SAM and mRNA.SAM files, respectively). From the miRNA.SAM file, we further removed sequences longer than 25nts [50]. To obtain the mature miRNA name corresponding to the aligned region, we mapped the available mature miRNA sequences from miRBase against the precursors as well and selected the mature miRNA with the highest overlap.

From the mRNA.SAM file, we removed sequences shorter than 40 or those that mapped with mismatches. In cases where a read was aligned to multiple 3’UTR sequences, we kept the match to the longest one. Based on the alignment, we extended each fragment to 75 nts (similar to [13]). For each 3’UTR, we kept one alignment among those that overlapped by more than 85%. These steps resulted in two lists miRNA-clip and mRNA-clip, composed of miRNAs (name and sequence) and mRNAs (name, fragment sequence, and full mRNA), respectively.

TarBase

We downloaded experimentally labeled negative interactions from TarBase v8 [24]. Each interaction record in TarBase dataset consists of gene ID, miRNA name, source species, tissue, cell line, and experimental method. We discarded all non-human or unknown cell line records. We filtered duplicate records of the same miRNA-gene pair. Then, we supplemented each record with the corresponding miRNA and 3’UTR sequence from miRBase and Ensembl, respectively, resulting in a dataset denoted as TarBase, which contains miRNA-full mRNA pairs.

2.4 Generating interactions

Below, we describe how we used the above resources to generate positive and negative interactions for our analysis.

Positive interactions

Datasets h1-h4 are considered positive, as they contain experimentally identified miRNA-mRNA pairs. For all records in these datasets, interaction duplexes were calculated and classified and only valid interactions were kept (Table 2). In addition, we filtered duplicate interactions (i.e., the same miRNA, target gene, and duplex). We chose h3 as the positive interactions dataset for our experiments. In addition, we created full-positive-dataset (FPD) from h1-h4, which was used to filter negative interactions produced by various methods, as described below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Summary of positive datasets processing pipeline.

https://doi.org/10.1371/journal.pcbi.1012385.t002

Negative interactions

We collected methods for generating negative interactions from the literature (refer to Fig A in S1 Text for details on the comprehensive literature search). We grouped methods that share a common idea and utilize similar sources of information, although the implementation details may vary across studies. Below, we provide relevant details for each method and summarize all methods in Fig 1. Furthermore, the pseudocode of the algorithms is provided in S1 Text.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Summary of methods for generating negative interactions utilized in this study.

The first three methods use the positive dataset h3 (miRNA- mRNA fragment-full mRNA records) to produce a corresponding negative interaction for each positive interaction. The fourth method uses TarBase data (miRNA-full mRNA records), and the last two methods employ complementary CLIP data of h3 (miRNA-clip list, mRNA-clip list). The TarBase and CLIP-based methods utilize the full-positive dataset (FPD) to filter genuine interactions from potential negative interactions. Note that the black ellipse in the figure represents miRNA–target pairs; but it does not appear for the CLIP dataset, which provides separate miRNA and mRNA sequences.

https://doi.org/10.1371/journal.pcbi.1012385.g001

Method 1: Mock miRNA.

A mock miRNA is a random shuffling of a real mature miRNA sequence; for each positive interaction in h3, we employed a random shuffling (1-mer/mono, using uShuffle with k = 1 [51]) to the original miRNA sequence until the sequence at positions 2–7 and 3–8 in the shuffled miRNA did not match with the same regions of any real human miRNA (similar to, e.g., [20, 23]). Next, we provided the mock miRNA and the full mRNA sequence as inputs to RNAduplex and repeated the process until a valid duplex was obtained or the number of attempts reached a predefined limit. Successful attempts involve generating artificial miRNA sequences, which, although they do not naturally occur in human cells, form valid interaction duplexes. We anticipate detecting distinct feature patterns between genuine and artificial miRNA interactions. The dataset resulting from this process is referred to as Mock_miRNA (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of the negative datasets generated by different methods.

The method numbers correspond to method numbers in Fig 1. FPD denotes the full-positive-dataset.

https://doi.org/10.1371/journal.pcbi.1012385.t003

2.5 Features

To represent positive and negative MTIs, we used 500 features, of which 490 features were adopted from our previous study [23] and the remaining 10 features that characterize miRNA pairing were added in this study. The classification of the features into five categories is shown in Table 4. For the full list of features and their description, see S1 Text and S1 Table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Categories of features that are used to represent MTIs.

https://doi.org/10.1371/journal.pcbi.1012385.t004

2.6 Machine Learning methods

We employed the following ML techniques in our experiments.

2.7 Data leakage, imbalance and splitting into training and testing sets

Our ML approach is based on interaction features rather than the interacting sequences themselves. Similar sequences might form different interactions with variations in the duplex structures and their characteristics, and vice versa; different sequences can form duplexes with the same characteristics. Therefore, for each interaction in the negative datasets, we ensured that the positive dataset h3, contains no interactions with the same feature values.

Some of the methods generated less or more negative interactions compared to the positive dataset (Table 3). To address the class imbalance problem, we employed the commonly used methods of undersampling, which involves randomly selecting a subset of samples from the larger class.

Accurate determination of the training and testing sets is essential to ensure the reliability of results. Specifically, we ensured that all models were trained and tested on the same positive interactions while varying the negative interactions. We split the positive and negative datasets into 80% training and 20% testing according to a random state (Fig 2). Furthermore, to simulate cross-validation, we repeated the splitting procedure 20 times with different random states, resulting in 20 unique training and testing sets for each dataset. By adopting this comprehensive methodology, we obtained robust and reliable results for our analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Evaluation of negative data generation methods by intra- and cross-datasets analysis.

The models were trained and tested on identical positive interactions, while the negative interactions varied. The positive and negative datasets were randomly split into 80% for training and 20% for testing. Training and testing sets derive from the same negative dataset or two different datasets in intra- or cross-dataset classification analysis, respectively. To simulate cross-validation, we repeated the splitting procedure 20 times with different random states, resulting in 20 unique training and testing sets for each dataset.

https://doi.org/10.1371/journal.pcbi.1012385.g002

2.8 Evaluation

We conducted a comprehensive evaluation of the performance of XGBoost in classifying MTIs intra- and cross-negative datasets (Fig 2). Training and testing sets derive from the same negative dataset in intra-dataset classification or two different datasets in cross-dataset classification. Thus, we enumerated all possible pairs of training and testing datasets: (train_i, test_j) where 1 ≤ i,j ≤ N and N is the number of methods to generate negative interactions. For each pair, we trained 20 XGBoost classifiers with the training sets of dataset i (corresponding to 20 splits) and evaluated their performance on the testing sets of dataset j. We calculated the mean and the standard deviation values of the classification accuracy (ACC), True Negative Rate (TNR), True Positive Rate (TPR), and F1 score.

Additionally, to ensure a fair comparison of one-class SVM with SVM model and of Isolation Forest with Random Forest model, we executed the GridSearch algorithm with five-fold cross-validation to determine the optimal hyperparameters for the one-class models, while ensuring compatibility with the hyperparameters of the binary-class models. For the training of the binary-class models, we used the training sets generated above, while for the one-class models, we used partial training sets that included only the positive interactions. All the models were tested on the same testing sets. Here we computed the mean and the standard deviation of different measures of the 20 tests as well.

2.9 Data analysis techniques for interpreting classification model results

We used various techniques to gain a deeper understanding and biological insight into the classification results.

3.1 Intra-dataset analysis identifies inherent biases in some strategies for negative data generation

In this subsection, we evaluate the performance of ML-based binary classifiers to correctly classify positive and negative MTIs derived from the same dataset. For each dataset, we generated 20 random training-testing splits of the data. We then trained XGBoost classifiers on the 20 training sets and measured their average performance in the classification of their respective testing sets (Fig 3). Additionally, we examined sub-cases for some data generation methods to determine their optimal implementation and estimate the contribution of features to classifiers’ decisions. We found that the performance metrics of most datasets were similar and relatively high, ranging as follows: ACC [0.74–0.85], TPR [0.66–0.85], TNR [0.75–0.85], and F1 [0.72–0.85]; however, for some sub-cases, the performance reached much higher values and was subjected to a deeper investigation as described below. Since all metrics (ACC, TPR, TNR, F1) correlated, the primary measure reported below is ACC.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Evaluation of negative data generation methods based on intra-dataset ML.

The results are organized in a heatmap, where rows correspond to different methods of negative data generation and columns correspond to different performance metrics. Each cell provides the average value obtained from 20 classifiers. The datasets marked with asterisks were chosen for the next step of cross-dataset ML analysis. The boxed methods are sub-cases of the same method. For interpretation of the results refer to feature importance analysis in Fig 4, and Figs B,C in S1 Text.

https://doi.org/10.1371/journal.pcbi.1012385.g003

3.1.1 Comparative analysis of key features: Highlighting variations in the classification decision-making process.

To investigate the difference in the decision-making process of the classification models trained on different datasets of negative interactions, we conducted the following test. For each pair of models, we extracted the union of each dataset’s top 100 important features and applied Wilcoxon Signed Rank test on their normalized SHAP values (Fig 4). The decision-making process of the classification models exhibits divergences; notably, methods based on CLIP data (e.g.,CLIP_non_CLASH, NPS_CLIP_Random) are significantly different from other datasets. Furthermore, we observe notable differences among pairs of sub-cases of the same method (e.g., Mock mRNA, NPS_CLASH). For example, the significant difference between NPS_CLASH methods indicates that selection criterion (MFE or random) among multiple sites affects the impact of different features on the classification outcome. In the random case, miRNA pairing and seed features are the top important features, while in the MFE case, mRNA composition features take precedence.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Comparing top important features of different negative datasets.

Each cell (i,j) represents the -log10(p-value) of the Wilcoxon Signed Rank test applied on the normalized SHAP values of the union of the top 100 features of datasets i and j. Bonferroni correction was applied to the p-values. Non-significant P-values (greater than 0.05) are marked with 0. The boxed methods are sub-cases of the same method. Included are only methods that were chosen for cross-dataset analysis.

https://doi.org/10.1371/journal.pcbi.1012385.g004

3.2 The cross-dataset analysis demonstrates the models’ sensitivity to negative data in the MTI classification task

In the previous subsection, we conducted training and evaluation of dedicated classifiers for each dataset. To further understand how sensitive the models are to different types of negative data, we investigated cross-dataset relationships which are reported in this subsection. To that end, we first characterized the datasets in terms of base-pairing patterns and miRNA distributions. We also compared pairs of datasets by their miRNA distribution using KL distance. Subsequently, we used classifiers trained on specific negative datasets and assessed their classification performance on negative interactions from different datasets (Fig 2). We hypothesized that classifiers that are trained and tested on datasets with similar characteristics would perform better compared to datasets that are not.

3.2.2 Cross-datasets classification performance.

For each dataset, we loaded the XGBoost classifiers that we trained in subsection 3.1 and used them to classify the test sets of the other datasets (repeated 20 times for each train-test split, Fig 2). The primary measure reported below is mean ACC (Fig 5, Fig H in S1 Text). We provide other performance metrics TPR, FPR, TNR, and FNR in Figs I-L in S1 Text, respectively. The ACC values vary among the pairs, ranging from 0.48–0.82. The matrix exhibits symmetry, i.e., for most pairs of datasets, the classification performance is similar for (i,j) and (j,i), where datasets i and j serve as training and testing sets, and vice versa, respectively. In general, for most datasets, the intra-classification scores (diagonal) are similar or slightly higher than their scores of cross-classification (either as training or testing dataset) (e.g., Mock_di_fragment). However, some datasets, such as Mock_miRNA and NPS_CLIP_Random, exhibit lower ACC scores in the cross-classification with several datasets than in their intra-classification.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Cross-datasets classification results.

Cross-datasets classification ACC for pairs of negative datasets. Each cell (i,j) represents the mean ACC of the 20 classifiers that were trained on dataset i and tested on dataset j. The standard deviation (std) values can be found in Fig H in S1 Text. Note that, for ease of interpretation of the results, the color scale is the same as in Fig 3 for intra-dataset analysis and inverse to the scale used for the KL-divergence plot in Fig G in S1 Text. For interpretation of the results, refer to dataset characteristics and feature importance analysis in Figs F, G, and M in S1 Text.

https://doi.org/10.1371/journal.pcbi.1012385.g005

To explain the differences in the performance of the pairs we examined several factors. For example, (TarBase_microarray, Mock_di_mRNA) achieved relatively high ACC (0.7) while (TarBase_microarray, CLIP_Non_CLASH) achieved low ACC (0.55) scores. First, the KL divergence results indicate that the dataset TarBase_microarray is similarly divergent from the other two datasets (Fig G in S1 Text) with scores of 4.4 and 4.2. This observation is further supported by the CDF curves (Fig F in S1 Text), which indicate that miRNA distributions are not the main factor for the variations in ACC results.

Second, in most cases, the distribution of base-pairing patterns of the interactions within the datasets coincides with the ACC results (Fig E in S1 Text). When datasets exhibit a similar proportion of non-canonical interactions with high-density levels, their cross-classification ACC tends to be relatively similar to their corresponding intra-results. The above datasets TarBase_microarray, Mock_di_mRNA, CLIP_Non_CLASH contain 38%, 44%, and 18% non-canonical seeds with high-density pairing, respectively, reflecting that the dataset TarBase_microarray is more similar to the dataset Mock_di_mRNA in this regard, supporting the higher ACC for this pair compared to the second pair.

Third, features that characterize free energy and miRNA pairing of the interactions (see Methods: 2.5), and are directly related to the density level, were significantly important for both pairs (Fig M in S1 Text). However, the difference lies in their impact on classification success and cross-dataset knowledge transfer. For the first pair, these features contribute to successful classification and are consistent for both datasets. Although energy and miRNA pairing features remain crucial for the second pair, they do not aid in the classification task (low performance), suggesting a disparity in energy distribution between the datasets, which hinders effective information transfer from the training dataset to the testing dataset.

3.3 One-class classification performance

We investigated whether negative interactions are necessary for the classification of MTIs or if it is feasible to develop models that utilize only one class of available information. To that end, we evaluated the performance of one-class SVM and Isolation Forest algorithms that are trained on positive class examples only. We compared them with their corresponding binary learning models SVM and random Forest, respectively (see Methods: 2.6.2, 2.8), using multiple metrics. As the trends across different negative datasets were comparable (Fig P in S1 Text), we refer in detail only to TarBase_microarray dataset (Fig 6). Notably, one-class classifiers demonstrate a nearly perfect TPR. At the same time, the TNR is negligible, implying that these classifiers cannot differentiate between positive and negative interactions and instead categorize all negative interactions as positive.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparing the performance of One-class classifiers with their respective binary classifier.

The performance of four classifiers is measured by ACC, AUC, TPR, TNR, and FPR. The One-class SVM and Isolation Forest were trained on positive data only while the SVM and Random Forest were trained on both positive and negative datasets. All four models were tested for their ability to classify both positive and negative interactions. The negative dataset reported here is TarBase_microarray. For similar analyses with other negative datasets see Fig P in S1 Text.

https://doi.org/10.1371/journal.pcbi.1012385.g006

Isolation Forest achieved a low ACC score of 0.5 while Random Forest reached a high score of 0.78. Additionally, FPR was 0.186 and TPR was 0.814 for the Random forest, implying its proficiency in accurately identifying both positive and negative interactions, which is not reflected in the performance of the Isolation forest. The One-Class SVM and SVM classifiers exhibited comparable ACC scores of 0.49 and 0.57, respectively. However, SVM outperforms one-class SVM in identifying negative interactions, as evidenced by its lower FPR and higher TNR values.

Intra- and cross-dataset analyses indicate the difficulty of ML models to generalize interaction rules from training to testing sets

Having the negative datasets in hand, we aimed to first assess the quality of the datasets by training/testing the ML model on each one of them and (2) evaluate the generalizability of these models on independent test sets (i.e., other negative datasets). The first task proved difficult, with a maximum ACC of 0.82. This suggests that the negative interactions were of high quality, resembling positive interactions and avoiding excessive artificiality. When examining a specific classifier in its intra- vs cross-classification performance, we observed similar TPRs due to shared positive interactions in the test sets; and these TPRs were relatively high across most classifiers (correspond to rows in Fig I in S1 Text). Nevertheless, significant variations were observed in the FPRs of the same classifier. For example, for Mock_di_mRNA FPRs range from 0.19–0.77 (Fig J in S1 Text); highlighting the model’s variable risk of misclassification of negative interactions, which is undesirable in real-world applications. Furthermore, almost all classifiers, showed a high FPR on at least one dataset, indicating the inability of the model to generalize interaction rules from learning to testing sets.

Specific methodological characteristics explain the differences between datasets

The performance of a classifier can be highly affected by methodological decisions that introduce biases into the characteristics of negative datasets. In the case of mock_mono_mRNA method, we demonstrated that the nucleotide shuffling method influences the target composition properties. The intra-analysis ACC of this dataset is above 0.9, indicating that the model effectively learned the pattern of a specific small group of dominant features.

Another characteristic that causes differences between datasets is related to the search space of the binding site on the target sequence (i.e., experimentally derived fragment or on the full mRNA). Clustering of the datasets based on their cross-dataset ACC values, clearly shows that ACC values are higher for dataset pairs that use similar portions of the target gene (fragment or full mRNA) to allocate the sites (Fig Q in S1 Text). A larger search space enables RNAduplex to identify more energetically favorable regions, influencing the energy characteristics of the interactions. Not surprisingly, energy characteristics emerged as the most influential factor in the classification process of negative interactions that were generated using different portions of target genes (Fig R in S1 Text).

In some existing methods, negative candidates are filtered based on specific criteria. In our study, in NPS_CLASH, for example, we employed one of two selection options for candidates of specific miRNA-mRNA pair: interaction with the lowest MFE or random selection. The cross-dataset ACC scores of the pair NPS_CLASH_MFE and NPS_CLASH_Random in both directions are significantly lower than their respective intra-dataset results. These notable disparities arise because energy-based selection favors interactions with biased energy characteristics. To counterbalance this bias, we generated datasets with two additional selection methods that prioritize candidates with relatively good energy properties rather than optimal ones: median MFE and random selection from the top 20th percentile based on MFE. We then applied cross-dataset analysis between all four selection options (Fig S in S1 Text). We found that the selection method impacts the classification performance, as is evident by the lower ACC of the cross-dataset compared to all intra-dataset classifications. Nevertheless, the model trained on NPS_CLASH_20th_Percentile exhibited subtle differences in performance, suggesting it better generalized the interaction rules.

The use of artificial or real miRNA/mRNA sequences in negative datasets is another factor to consider. The literature has debated the impact of artificial sequences on the model’s ability to differentiate between positive and negative interactions, as it may learn to distinguish between the sequence patterns rather than the interaction patterns [23,28]. Surprisingly, our findings demonstrate that datasets composed of artificial sequences or real sequences achieve comparable ACC in intra-analysis, probably since we do not include sequence-based features [23]. Furthermore, in certain cases, datasets composed of artificial sequences exhibit strong generalization capability on datasets composed of real sequences in the cross-analysis.

Maintained factors in dataset evaluation

To ensure a rigorous comparison between the methods, we maintained specific parameters throughout our analysis. We used the RNAduplex tool to find the location of the binding site of miRNAs on target sequences. The duplex selection process by this tool introduces bias, as it chooses energetically favorable regions. Consequently, energy-related features influence the classification model. We focused on canonical and non-canonical interactions as these are considered functional. In addition, we employed XGBoost as our ML method. This method was shown to outperform other ML methods in MTI prediction [23] and to achieve better results compared to deep learning models when handling tabular data [52]. Performing similar analyses using different ML and deep learning algorithms, alternative duplex generation tools, seedless interactions, and additional interaction features could be a venue for future research.

Specifically, we demonstrated the analysis utilizing human experimental datasets, with a particular human MTI dataset (h3) serving as the positive dataset. Based on our previous research [23, 53], which shows that MTI interaction rules are transferable between evolutionary close species, we speculate that our conclusions– that different methods for negative datasets impact the performance of ML models– could also extend to other species. However, this requires validation in further studies.

Combination of negative datasets

To further examine the biases in negative data generation, we created a negative dataset composed of a balanced mixture of all the negative datasets proposed in this study and trained/tested a new model with this data along with the positive set. Interestingly, the model failed to classify the negative interactions correctly. This outcome reiterates the insight that the negative interactions are biased by the specific strategy of each generation method. Consequently, the model that was trained on this combined dataset struggled to identify a consistent pattern that characterized all negative interactions while distinguishing them from positive interactions. It is possible that other strategies, including ensemble methods for combining models trained on different negative datasets, would yield better performance and should be examined in future studies.

Recommendation for negative data selection and examination of new ML models

The process of generating negative interactions can introduce biases to the data, leading the model to easily distinguish between positive and negative interactions in the training set and then fail on the test set. To avoid such a scenario, we propose some guidelines: (1) Avoid bias in one or more features. For instance, if the k-mer composition is modeled in the features, maintaining the original ratio distribution of k-mer sequence composition in the artificial sequence is crucial, and can be achieved through k-mer shuffling (e.g., Mock mRNA sub-methods); (2) Keep negative interactions as similar as possible to positive interactions (such as maintaining canonical or non-canonical seed) and (3) Generate negative interaction from diverse miRNAs and mRNAs to enhance model generalizability (e.g., TarBase_Liver interactions originate from a limited set of miRNAs, and its model failed to generalize and exhibited model overfitting).

To assess the generalizability of a new ML model for MTI classification, it is essential to evaluate its ability to effectively capture positive interactions while minimizing the risk of misclassifying negative interactions as positive. This is characterized by achieving a high True Positive Rate (TPR) and a low False Positive Rate (FPR), respectively. As mentioned earlier, we identified the search space for the binding site on the target sequence (i.e., experimentally derived fragment or the full mRNA) as a key factor contributing to differences between negative datasets. To evaluate the model’s generalizability, it is recommended to test it on negative interactions derived from both strategies. Divergent results between these strategies would indicate low generalization, while similar results would affirm the model’s effectiveness. Notably, the Mock_di_fragment method demonstrated a notable success rate in identifying positive interactions in cross-classification analysis. However, it occasionally misclassified a few negative interactions as positive. Despite this, the method exhibited good generalization overall (Fig Q in S1 Text).

One-class classification models highlight the need for negative data in MTIs classification task

As an attempt to avoid generating negative datasets for training, we trained One-class SVM and Isolation Forest solely on positive interactions. However, when testing them on both types of classes, we observed very poor classification performances overall. This low performance suggests that these models struggled to learn solely from positive examples, as they misclassified almost all negative interactions, highlighting the challenge of distinguishing between both classes. Consequently, these models, commonly used for anomaly detection tasks to identify outliers, are unsuitable for MTI analysis due to the similarity between positive and negative interactions. This further underscores the importance of negative examples to create accurate classification models that can distinguish between the interactions despite the similarities.

Extension to other biological interaction data

While our research focuses on miRNA-mRNA interactions, the methodologies and insights gained can be extended to other types of biological interactions, such as Transcription Factor (TF)-DNA and RNA Binding Protein (RBP)-RNA interactions. In these domains, the challenge of generating reliable negative data remains critical for developing accurate predictive models. For TF-DNA interactions, several methods have been proposed for the preparation of negative datasets with low expected TF binding, including sequences not labeled as TF binding sites, downstream random exons, random selection of non-coding sequences, or DNA regions far away from genes [54]. For RBP-RNA interactions, methods for negative data generation include shuffling the coordinates of binding sites within all genes with at least one binding site or extracting sites that are not identified as interacting [55]. Following the results of our study, it is critical to compare and evaluate the impact of these methods on classifier performance and establish guidelines and standards in these fields as well.