1. Introduction

Automated function prediction (AFP) is the process of predicting the function of genes or proteins by learning patterns in high-throughput biological data using machine learning. The goal of AFP is to predict novel protein functions to increase the rate at which researchers can understand disease and other phenotypes. The history of AFP shows a trend of using increasingly sophisticated machine learning methodologies and making predictions in increasingly complex organisms. One conception of AFP uses supervised machine learning models trained on high-throughput biological data to predict gene or protein function in terms of Gene Ontology labels [1]. The first iterations of this approach occurred in 2003 and 2004 using models built with Bayesian networks. These models combined high-throughput gene co-expression, protein interaction, and genetic interaction data along with expert-curated priors to predict gene function in Saccharomyces cerevisiae [2,3].

SPELL, MEFIT, and bioPIXIE were subsequent models of automated function prediction in S. cerevisiae that leveraged gene co-expression data and innovated on many of the ideas found in prior work [4–6]. MEFIT and bioPIXIE employed naive Bayesian networks while SPELL used a directed query search through gene expression datasets. These were among the first AFP methods to use some conception of model training. MEFIT and bioPIXIE replaced expert-curated model priors with priors learned from training data, and SPELL dynamically generated model weights at runtime in response to each query. These three methods were trained on similar input data and were experimentally validated in a systematic study of mitochondrion organization and biogenesis in yeast. This led to the discovery of over 100 genes involved in that process. These studies demonstrated that AFP models can be useful for directing experimental biology; furthermore, they evaluated the impact of underlying data types (e.g., expression data vs protein interaction data) on AFP performance [7–9]. At the same time, models were developed that used more sophisticated machine learning methodologies, such as support vector machines [10–12]. Additionally, methodologies began to branch out to predicting gene and protein function in organisms other than yeast [13,14], and predictions were extended beyond GO annotations to predict other genomic features, such as knockout phenotypes and gene-disease associations [15–17].

In the past decade, much of the cutting edge research in machine learning, and its applications to biology, have transitioned towards the use of neural networks. Neural networks transform input data through layers of “artificial neurons” to produce an output prediction. Neural networks quantify the discrepancy (or “loss”) between the predicted output and the target output value of labeled training examples using a loss function. Given this loss value, neural networks can update their own parameters to make more accurate predictions via an algorithm known as “backpropagation.” Deep neural networks (those with several layers of artificial neurons) are responsible for breakthroughs in numerous problem domains including image recognition and natural language processing due to their ability to learn complex, hierarchical, and non-linear representations of data. However, the complexity of the functions learned by deep neural networks makes it challenging for humans to understand the rationale behind how these models make their predictions. Improving the explainability of neural network outputs and finding ways to interpret the inner workings of these models is an ongoing area of research [18].

Advancements in deep learning with neural networks are responsible for a paradigmatic shift in how researchers approach automated function prediction. Deep neural networks have displayed a marked increase in predictive accuracy over past AFP models [19]. Several variants of neural networks have been shown to be particularly well suited for modeling complex relationships in various modalities of biological input data. Some notable variants include convolutional neural networks, graph neural networks, and transformers [19]. Furthermore, neural networks have been associated with a trend in AFP towards predicting function primarily from a protein’s amino acid sequence (in the absence of species information) [20–23]. In parallel, a substantial body of work has explored network-integration-based AFP approaches by combining multiple biological networks using techniques such as matrix factorization, graph embedding, and deep representation learning [24–28].

Researchers lack a nuanced understanding of how the differences between specific deep learning architectures and earlier, widely-used machine learning methodologies affect the quality of predictions of co-expression-based AFP models. In the field of AFP, models lose the attention of researchers once they have been surpassed in predictive accuracy by other state-of-the-art techniques, and this trend has only been exacerbated by the rise of deep learning in AFP. There is little analysis of how deep learning versus classical machine learning techniques are suited for tackling the domain-specific challenges of AFP (such as noisy input data and changing Gene Ontology annotations). Furthermore, most AFP models lack systematic experimental validation beyond evaluating performance on newly annotated proteins (that were not present when the model was trained) [1]. Therefore, it is unclear how most deep learning and classical AFP models differ, especially in the context of directing novel biological experimentation.

The large variety of machine learning techniques and input data modalities in AFP make it difficult to determine how differences between deep learning and classical machine learning methodologies affect prediction quality. In service of this goal, it is useful to establish a baseline by comparing the performance of the early, classical AFP models against the simplest deep learning architecture. Therefore, we chose to compare the classical AFP models MEFIT and SPELL to a model built with feedforward neural networks trained on the same gene co-expression input data. While none of these models represent the state-of-the-art in contemporary AFP, we believe that this analysis will establish a baseline that will be useful for comparisons with more complex deep learning architectures and other input data modalities in the future. We emphasize that our comparison is not meant to be exhaustive of all “classical” machine learning techniques; rather, it is to help establish a historically-grounded baseline of comparison to examine methodological behaviors and variance.

In this paper, we directly compare feedforward neural networks to experimentally-validated, classical machine learning techniques for co-expression-based AFP in yeast to assess how deep learning can improve prediction quality in this domain. First, we describe our methodology for building a model of AFP using feedforward neural networks that predicts 79 GO biological process labels from co-expression data of pairs of genes. We trained our model with the same gene expression data and GO labels that MEFIT and SPELL were trained with in 2007. Second, we compared each model’s ability to predict genes involved in mitochondrion organization. We found that our model is more performant than MEFIT and SPELL, in part because it better distinguishes mis-annotated negatives in its training data. Third, we trained our model with current-day gene expression data and GO labels to assess how the accumulation of more input data and higher-quality training labels affects our model’s performance. Finally, we discuss the relative merits of the approaches investigated and place our results in the broader context of more modern AFP methods.

2. Methods

Our model trains feedforward neural networks to predict the functional relationship between pairs of genes in the context of 79 Gene Ontology biological processes (Fig 1). As input, our model uses the correlations between the gene expression profiles of pairs of genes in 113 microarray assay datasets. Once trained, our model uses its predictions to construct a functional relationship graph for each GO term that can be converted into experimentally-testable, single-gene functional predictions. Our initial model is trained using the same Gene Ontology annotations and gene expression datasets that SPELL and MEFIT trained with as of April 15th, 2007.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of methodology.

(A) Training data is created by calculating pairwise gene expression similarities in 113 gene expression datasets and by curating a set of “gold standard” training labels for genes with known functions. (B) Feedforward neural networks are trained to predict the Gene Ontology co-annotations of pairs of genes given feature vectors of their gene expression similarity. (C) Our method translates pairwise functional relationship predictions into testable single gene rankings for 79 biological processes.

https://doi.org/10.1371/journal.pone.0322689.g001

2.1 Curation of labeled training data

The input data for our model represents each pair of genes as a vector of 113 z-scores derived from Pearson correlations across experimental conditions for each of the 113 gene expression datasets. To train our model, we curated training labels for gene pairs with known functional relationships and gene pairs thought to be unrelated. Positive and negative labels are determined for each of the 79 predicted GO terms based on each gene pair’s co-annotation to a term, as detailed below.

2.1.1 Gene expression compendium curation.

In order to accurately compare between approaches, our model uses the same gene expression compendium as MEFIT and SPELL, which we obtained from the Saccharomyces Genome Database (SGD) [29,30]. This compendium consists of 113 gene expression datasets publicly available and curated by SGD as of April 15th, 2007 and comprises 1842 experimental conditions. These raw gene expression levels are processed for use as input data by normalizing Fisher-transformed Pearson correlations between pairs of genes for each of the 113 datasets in the compendium. For each gene expression dataset, the Pearson correlation between a pair of genes’ expression values is calculated as:

where x and y are the gene expression vectors of size n from a given dataset, μ_x and μ_y are their means, and σ_x and σ_x are their standard deviations. In the case of missing measurements, gene expression vectors x and y only contain the experimental conditions that have observations for both gene x and gene y (Further details and other minor adjustments are described in Section 1.1 of S1 Text). As the distribution of Pearson correlations can vary greatly between gene expression datasets, the Fisher transformation is applied to improve comparability across datasets [31]:

After Pearson correlations have been Fisher transformed, the resulting values are normalized by the following equation to produce a standard normal score (z-score) for each pair of genes:

where μ_f and σ_f are estimated mean and standard deviations calculated using a subset of 200,000 random Fisher transformed Pearson correlations from that dataset. The resulting distribution is approximately standard normal . In all, each value in each dataset is a z-score that represents the number of standard deviations away a gene pair’s gene expression correlation is from the dataset’s mean of pairwise correlations [5]. This is the same preprocessing and normalization approach MEFIT and SPELL used for their gene expression compendiums [4,5].

2.1.2 Gold standard curation and cross validation.

To train our model, we need training labels consisting of gene pairs known to be functionally related (positive examples) and gene pairs thought to lack functional relationships (negative examples). To derive these labels, we first curate a set of single genes that have known functions, which we will call our “gold standard” genes. We defined genes with known functions as genes annotated to at least one of a set of 79 biological process Gene Ontology terms curated by experts as being informative enough to direct laboratory experiments [32]. These GO terms form a subset of biological process terms (distributed as a GO slim), and we used annotations to this subset from January 1st, 2007 in order to enable comparison to prior methods trained at that time. We further filtered the GO slim to only include terms with a sufficient number of annotations for model training (n >= 10). Genes not annotated to any of these 79 GO terms were considered functionally uncharacterized.

Our model uses 4-fold cross validation to assess overfitting and make its final predictions. The gold standard genes are randomly divided into four independent subsets. Each “fold” of the model will use three of these subsets as training genes while holding out the remaining subset as testing genes. Each fold generates a set of related (positive) gene pairs and unrelated (negative) gene pairs from the training genes or testing genes. Related pairs are those that are co-annotated to at least one of the 79 GO terms. Unrelated pairs are those whose smallest co-annotated GO term contains at least 10% of the genes in the yeast genome, indicating that each gene has a known function, and those functions are different from each other. This approach is not a perfect measure, since it assumes that two genes with known, different functions are unlikely to share a functional relationship. Note that to avoid data leakage and circularity, gene pairs used as training examples are only pairs where both genes are in the training gene set, and testing examples are only pairs where both genes are in the testing geneset [33,34].

2.1.3 Features and training labels.

Our model represents each gene pair as a vector of input features associated with a vector of training labels. The input features for each gene pair is a vector of 113 z-scores derived from the gene expression datasets, and the training labels are one-hot encoded vectors representing the co-annotations of a gene pair to 79 Gene Ontology terms. For each GO term, a gene pair is labeled positive (encoded as 1) if the genes are co-annotated to that GO term and negative (encoded as 0) otherwise.

The full gene expression data files compendium and Gene Ontology files used for all training are available in S1 Dataset.

2.3 Post-processing for testable biological predictions

The aggregated outputs of our neural network model are functional relationship graphs for each predicted GO term that contain information about how confident our model is that two genes are functionally related in the context of each respective GO term. Importantly, our predicted GO terms were collected from a list of biological processes experts deemed to be informative for directing experimental biology [32]. While our model’s pairwise predictions contain a wealth of potentially novel biological information, we expect that they are also fairly noisy. We can reduce this noise by aggregating pairwise predictions into predictions of the functions of single genes. Furthermore, this allows us to compare against the performance of other automated function prediction models (such as MEFIT and SPELL) that also make single gene functional predictions. Similar to MEFIT and SPELL [7,9], we process our functional relationship graphs to generate testable predictions by ranking individual genes by their predicted involvement in a GO term as measured by the mean of their connections to other genes annotated to that term (Fig 2).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Diagram illustrating process of generating testable predictions from predicted pairwise relationships.

For each GO term, every gene in the yeast genome is ranked by the weighted average of its connections to genes annotated to that GO term in that GO term’s functional relationship graph.

https://doi.org/10.1371/journal.pone.0322689.g002

To predict an individual gene’s involvement in a GO term’s biological process, we use the confidence of the gene’s predicted relationship to other genes annotated to that term. For each GO term’s functional relationship graph, each gene in the yeast genome is ranked by its average weighted connection to all other genes annotated to that GO term. Therefore, for each GO term, our model produces a ranked list of individual genes where the top ranked genes are predicted to be more likely to be involved in that biological process. The performance of the model is measured by generating ROC and precision-recall curves from these single gene rankings [35]. Genes annotated to a GO term are labeled positive, all other genes in the gold standard set are labeled negative, and all uncharacterized genes are unlabeled and do not affect the performance evaluation.

3. Results

3.1 Model performance

We evaluated the performance and overfitting of our neural network model by comparing the model’s average pairwise training AUC, average pairwise testing AUC, and single gene AUC for each of the 79 predicted GO terms. We plotted each GO term’s testing AUC against its training AUC with the size and color of each point being proportional to its GO term’s single gene AUC (Fig 3). The overfitting for each GO term (measured as the difference between its training and testing AUC) is represented by its point’s distance from the black diagonal line. We found that a GO term’s pairwise testing AUC was positively correlated with its single gene AUC (r = 0.847; p << 0.001) and negatively correlated with the overfitting (r = −0.859; p << 0.001). Together, this indicates that while our model does suffer from some overfitting, it is significantly less for the GO terms that have high testing AUC performance. The single predicted rankings for each predicted GO term for this model can be found in S1 File; further analysis of overfitting can be found in S1 Text.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Comparison of performance of neural network model for all 79 output GO terms.

For each GO term, the term’s average testing AUC is plotted against its average training AUC, and the size and color of the point is proportional to the AUC of the GO term’s single gene ranking. The black diagonal line shows where the testing AUC and training AUC are the same, which indicates there is no overfitting. The red circle and text denoted the six GO terms with the worst testing AUC, and the green circle and text denoted the six GO terms with the best testing AUC. This figure was constructed using data found in S1 Table.

https://doi.org/10.1371/journal.pone.0322689.g003

There is great variation in the testing AUC performance and overfitting across all predicted GO terms of the model. We expected that this is the result of our input gene expression data being fundamentally more informative for predicting some biological processes compared to others. To examine this claim, we first analyzed the top six best performing and the top six worst performing GO terms by testing AUC, which are the terms circled in green and red respectively (Fig 3). We found that the highest ranking terms were enriched for processes related to ribosome biogenesis, which is a process whose genes are highly co-regulated by gene expression [36]. Conversely, we expected that the weakest performing terms likely lacked strongly coordinated gene co-expression among most pairs of genes annotated to these processes among our data compendium. As such, we found that the lowest ranking terms were pseudohyphal growth (a process not commonly occurring in laboratory yeast strains [37]) and terms encompassing a diverse set of biological pathways (organelle fusion, protein acylation and dephosphorylation, etc.).

These findings align with our expectation that the testing AUC performance and overfitting for each GO term are correlated with the extent to which a GO term’s genes are co-regulated at the level of gene expression. To evaluate this quantitatively, we examined the difference in co-expression distributions for co-annotated gene pairs to the background distribution for all gene pairs. For each GO term, we derived a probability distribution, T, of the average z-scores (across our 113 gene expression datasets) by randomly sampling 200,000 gene pairs co-annotated to that GO term. Additionally, we created a background probability distribution, B, by sampling the average z-scores for 200,000 of all the training gene pairs. For each GO term, we measured the Kullback-Liebler (KL) divergence between the term’s probability distribution and the background probability distribution as D_KL(T,B). KL divergence measures the extent of discrepancy between two distributions by quantifying the amount of information lost by approximating one distribution with another [38]. The KL divergence of each term was positively correlated with the testing AUC of the term (r = 0.735; p << 0.001) and negatively correlated with the term’s overfitting (r = −0.660; p << 0.001) (S4 Fig in S1 Text). This indicates that as a GO term’s pairwise co-expression differs more from the background co-expression distribution, the GO term tends to make more accurate predictions and suffer from less overfitting. Therefore, these findings support our expectation that our model performance for a given GO term is correlated with the extent to which the term’s genes have coordinated co-expression.

3.2 Comparing mitochondrion organization predictions to SPELL and MEFIT

We set out to compare the performance between different machine learning models when predicting functional involvement in mitochondrion organization (GO:0007005) when trained on the same gene co-expression data. We compared the performance of our neural network model, SPELL, and MEFIT by comparing the ROC curves generated by each model’s rankings of single genes predicted to be involved in mitochondrion organization using Gene Ontology labels from January 1st, 2007 (Fig 4). We found that our neural network model performed better than both SPELL and MEFIT in the TPR-FPR tradeoff as reflected by over a 0.04 increase in AUC compared to both models.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Comparison of ROC performance to MEFIT and SPELL.

ROC curve comparing tradeoff between recall and false positive rate for rankings of genes predicted to be involved in mitochondrion organization for MEFIT, SPELL, and our neural network model. GO labels used in ROC curve calculations were derived from the January 1st, 2007 Gene Ontology. This figure was constructed using data found in S2 Table.

https://doi.org/10.1371/journal.pone.0322689.g004

We also assessed the precision of each model on their top predictions, since those predictions are the most ripe for experimental validation. Using the same rankings mentioned above, we generated precision-recall curves for each model’s rankings (Fig 5A). We found that the neural network model had worse precision at low recall (0–0.2) compared to SPELL and MEFIT when using labels derived from the 2007 Gene Ontology, despite our model’s overall improvement in ROC performance. In order to assess which model has better precision on top predictions in terms of our current understanding of biology, we relabeled the aforementioned rankings from each model using labels derived from June 16th, 2022 GO annotations. Using these modern labels, we found that our neural network model had higher precision than SPELL and MEFIT at low recall (Fig 5B).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Comparison of precision-recall tradeoff with MEFIT and SPELL.

Convex-hulled precision-recall curves with logarithmic x-axes comparing the tradeoff between precision and recall for rankings of genes predicted to be involved in mitochondrion organization for MEFIT, SPELL, and our neural network model trained on Gene Ontology annotations from using labels from January 1st, 2007 Gene Ontology. (A) Rankings labeled using January 1st, 2007 Gene Ontology annotations (solid lines) and (B) using June 22nd, 2022 Gene Ontology annotations (dashed lines). This figure was constructed using data found in S2 Table.

https://doi.org/10.1371/journal.pone.0322689.g005

In order to understand our model’s performance increase on top predictions using 2022 GO labels, we analyzed the distribution of genes that had their annotations changed from 2007 to 2022. Partially due to the efforts of SPELL and MEFIT, the genes and proteins annotated to mitochondrion organization in yeast have significantly changed since 2007. One particularly interesting set is the class of 117 genes that were labeled as negatives using 2007 GO annotations (in the gold standard but not annotated to mitochondrion organization in 2007) but are labeled as positives using 2022 GO annotations (annotated to mitochondrion organization in 2022). These genes were labeled as negatives for mitochondrion organization in our training data (and in the training data of MEFIT and SPELL) but according to our current understanding of biological reality, they should be labeled as positive. Due to these mis-annotations in the Gene Ontology, these examples contaminated our negative training set. As such, these 117 “contaminated negatives” are penalized for high-ranking predictions based on the 2007 evaluation, even though their high rank is biologically accurate. As such, we analyzed the distribution of rankings for these contaminated-negative mitochondrion organization genes ranked within the top 20% of genes predicted to be involved in mitochondrion organization for each model (Fig 6). Our neural network approach ranks these mis-annotated genes significantly greater rank than either MEFIT or SPELL (two sample KS test, Neural network–MEFIT p-value = 0.0274; Neural network–SPELL p-value = 0.0268), suggesting that our approach is more robust to label noise in its training data, at least in this particular case.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Distribution of mitochondrion organization mis-annotated negatives.

Density plots of relative rank of each model of predictions of high confidence mis-annotated negative mitochondrion organization genes. A gene’s rank is calculated as its relative placement in the single gene ranking of genes involved in mitochondrion organization. For each model, n represents the number of mis-annotated negatives predicted with a relative rank between 0.8 and 1.0. Two sample right-tailed Kolmogorov-Smirov tests were performed between the distribution of the neural networks high confidence mis-annotated negatives and SPELL or MEFIT high confidence mis-annotated negative distribution. The null hypothesis was rejected for both tests (Neural networks–MEFIT p-value = 0.0274; Neural network–SPELL p-value = 0.0268).

https://doi.org/10.1371/journal.pone.0322689.g006

3.3 Model performance using modern gene expression data and Gene Ontology labels

In the previous section, we found that our model outperformed SPELL and MEFIT when trained with the same 2007 gene expression data and Gene Ontology annotations. However, since 2007, the scientific community has accumulated significantly more gene expression data and the annotation quality of the Gene Ontology continued to improve. Therefore, we wanted to assess the performance of our approach when using modern gene expression data and Gene Ontology annotations and to explore the effects of additional input data versus improved label accuracy. Models used in this section have an architecture consisting of three layers of 80 hidden neurons each. This shared architecture was chosen for consistency of performance between the models in this section that have different sized input vectors.

We trained two models with the same 2007 Gene Ontology labels, but different gene expression input data. The first model used the 113 gene expression datasets available in 2007 and the second model used 430 gene expression datasets curated by SGD in 2022. When comparing each model’s AUC for each GO term’s single gene rankings, we found that across all GO terms, the AUC tended to be higher for the model trained with 2022 gene expression data (right-tailed paired t-test; p << 0.001) (Fig 7A). This indicates that additional input data generally improves the accuracy of predictions.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Comparison of performance with different combinations of gene expression input data and Gene Ontology training labels.

(A) Scatterplot of the AUCs derived from single gene rankings for each GO term derived from two models trained with different input data and the same 2007 training labels. The x axis represents AUCs from a model trained with all gene expression datasets available in 2007 and the y axis represents AUCs from a model trained with all gene expression datasets available in 2022. The black diagonal line separates terms that perform better with 2007 vs 2022 gene expression data. (B) Scatterplot of the AUCs derived from single gene rankings for each GO term derived from two models, one trained with 2007 Gene Ontology labels (x axis) and one with 2022 Gene Ontology labels (y axis). The black diagonal line separates terms that perform better with 2007 vs 2022 Gene Ontology labels (C) For each GO term, probability distributions were created by sampling the average z-score of 200,000 genes pairs each from the set of genes annotated in 2007 (T₂₀₀₇) and the set of genes annotated in 2022 (T₂₀₂₂). Background probability distributions were created from samples of the average z-score of 200,000 gene pairs from the set of all training gene pairs in 2007 and 2022 (B₂₀₀₇ and B₂₀₂₂). For each GO term, the 2007 KL divergence was measured as D_KL(T₂₀₀₇,B₂₀₀₇) and the 2022 KL divergence was measured as D_KL(T₂₀₂₂,B₂₀₂₂). The difference between the 2022 Ontology AUC and the 2007 Ontology AUC is plotted against the difference between the 2022 KL Divergence and the 2007 KL divergence with a regression line (dashed line). This figure was constructed using data found in S3 Table.

https://doi.org/10.1371/journal.pone.0322689.g007

We also trained two models using the same 2022 gene expression input data, but with different Gene Ontology training labels. The first model (as described in the previous paragraph) was trained and evaluated on labels from the 2007 Gene Ontology while the second model used labels from the 2022 Gene Ontology. We found that there was a high disparity in performance between the two models across all GO terms (Fig 7B). Some GO terms had better AUCs when training and evaluating on 2022 labels while others performed better with the 2007 labels. Given our findings in section 3.1 that each GO term’s performance is correlated with the extent its genes are co-regulated at the level of gene expression, we decided to investigate if a similar effect could explain the disparities in performance between these models.

We hypothesized that these performance disparities could be partially explained by the change from 2007 and 2022 in how correlated the gene sets of each GO term were compared to the background distribution. For each GO term, probability distributions were created by sampling the average z-score (across 430 gene expression datasets) of 200,000 genes pairs each from the set of genes annotated in 2007 (T₂₀₀₇) and the set of genes annotated in 2022 (T₂₀₂₂). Background probability distributions were created from samples of the average z-score of 200,000 gene pairs from the set of all training gene pairs in 2007 and 2022 (B₂₀₀₇ and B₂₀₂₂). For each GO term, the 2007 KL divergence was measured as D_KL(T₂₀₀₇,B₂₀₀₇) and the 2022 KL divergence was measured as D_KL(T₂₀₂₂,B₂₀₂₂). We found that the difference in a GO term’s AUC from 2022 labels and 2007 labels was positively correlated with the difference in the KL divergence for the 2022 and 2007 z-score probability distributions (r = 0.485; p << 0.001) (Fig 7C). This suggests that terms that had worse performance with 2022 labels had z-score distributions that became more similar to the background distribution between 2007 and 2022, making them harder to predict.

4. Discussion

In this work, we set out to compare feedforward neural networks to two classical machine learning methodologies for co-expression-based automated function prediction in yeast. We created a model of AFP in yeast using neural networks that can predict the functional relationship between pairs of genes in the context of 79 biological processes. Our model uses these pairwise predictions to rank individual genes by their predicted involvement for these 79 different biological processes. When training our model with the same gene expression input data and Gene Ontology labels as SPELL and MEFIT, we found that our model’s predictions of genes involved in mitochondrion organization had better ROC performance than these models. This indicates that if our model had been created at the same time as MEFIT and SPELL, it could have been used in conjunction with them to successfully direct an experimental study of mitochondrion organization [7–9]. However, our model still needs to be experimentally validated to determine if it is useful for directing experimental biology in the modern day. In future work, we plan to use petite frequency assays to test functionally uncharacterized genes our model predicts to be involved in mitochondrion organization (as did MEFIT and SPELL) [7–9].

Revisiting automated function prediction with gene expression data and Gene Ontology labels from 2007 revealed how the application of neural networks for automated function prediction interacts with our accumulation of biological knowledge. We found that the accumulation of more high-throughput gene expression data between 2007 and 2022 led to a significant increase in the performance of our model across most biological processes (Fig 7A). This suggests that despite the large corpus of gene expression data for yeast we have already amassed, acquiring more data can still improve co-expression-based AFP’s accuracy.

The conclusions for updates to Gene Ontology annotations are more complicated. We found that the difference in performance between a model trained with 2007 Gene Ontology annotations and 2022 Gene Ontology annotations varied greatly between GO terms (Fig 7B). We found that GO terms that performed better with 2007 GO annotations had co-expression z-score distributions that were less similar to the background training distributions in 2007 compared to 2022 (Fig 7C).

These results highlight an important limitation of our approach that is shared by most supervised AFP methods: prediction quality is highly sensitive to the annotations used to derive training labels. GO annotations are derived from the published literature and therefore reflect a bias toward well-studied genes and pathways, with limited coverage for poorly characterized genes. Additionally, the GO structure itself encodes strong assumptions about functional organization that may not fully reflect biological reality in all organisms or circumstances. These factors may further constrain the discovery of genuinely novel functional relationships using GO-based supervised approaches. Our results indicate that AFP methods must recognize these limitations when constructing training labels from GO annotations for supervised models. Furthermore, our results highlight the benefits of the fully unsupervised or weakly supervised approaches to AFP from network integration, particularly for uncovering function among uncharacterized genes [24].

Our analyses of past GO annotations highlights that the way our model, and many others, define negative training examples is somewhat naïve and potentially problematic. Recall that in our model, MEFIT, and SPELL, genes and pairs of genes are labeled as negative for a term if they are not annotated to that term but are annotated to another term in the GO slim. Yeast proteins often exhibit multiple distinct functions; thus, making the assumption that a gene’s involvement in one biological process implies a lack of involvement in another is reductionary. As such, there may be great value in the development of new approaches to defining negative training examples or approaches that forgo defining negative training examples entirely [39–41]. In the face of this problem, we note our model’s ability to better distinguish mis-annotated negative mitochondrion organization genes in its training set compared to SPELL and MEFIT (Fig 6). This is evidence that neural networks may have better potential to overcome the issue of problematic negatives compared to other machine learning models in the domain of AFP. However, further experiments and analyses are warranted to more fully explore this relationship.

In this study, we established a baseline for how feedforward neural networks and two classical machine learning methods compare for co-expression-based AFP. As we look to expand this analysis to more complex deep learning architectures, we feel it is important to place our methodology within the broader context of contemporary AFP research. Our methodology represents a different paradigm of supervised AFP than contemporary AFP approaches. Contemporary AFP largely focuses on predicting Gene Ontology labels of proteins from amino acid sequence embeddings generated from protein language models [19,23]. Some other common input data modalities for contemporary AFP models are protein structure [42] and text embeddings from large language models trained on scientific literature or GO term descriptions [23,43]. These models are typically trained on proteins from multiple species and are therefore predicting function in a species-agnostic fashion. Early AFP approaches, as well as our own, predict Gene Ontology labels by learning patterns in the co-expression profiles of pairs of genes. They learn the patterns of interactions between genes of the same species and are therefore species-specific. A proper comparison of these paradigms would require two kinds of analyses. First, what biological functions are gene expression versus amino acid sequence informative of? Second, what biological processes are species-specific and species-agnostic AFP models each better at predicting? While these analyses were out of the scope of our study, we do believe they are highly valuable and worthy of future study.

Our study shows that there is room for innovation in co-expression-based, species-specific AFP. First, we showed that, for some biological processes, feedforward neural networks are more performant at predicting GO labels from co-expression data than the classical machine learning methods of Bayesian belief networks and directed query searches. As we will discuss more, other deep learning architectures may achieve even better performance and are worthy of further inquiry. Second, our findings suggest the accumulation of more genome-wide gene expression data significantly improves our ability to predict gene function (Fig 7A). Given the greater abundance of data and innovations in machine learning methodologies, we believe that there is still a wealth of functional insights to be mined from gene expression data.

5. Future work and conclusions

Our future work consists of improving the interpretability of our model, exploring other deep learning architectures, and integrating data modalities other than gene expression into our methodology comparisons. First, while our model displayed better predictive accuracy than MEFIT and SPELL for mitochondrion organization, we note that our neural network model is less explainable and interpretable than these other methods. Given a predicted gene function, MEFIT and SPELL both display the input gene expression datasets that were most important for making that prediction [4,5]. Our model lacks this explainability. We did improve the interpretability of our model by finding that a GO term’s performance is correlated with the extent the co-expression of a term’s gene pairs diverge from the background training distribution (S4 Fig in S1 Text). That being said, our future work includes making our model more interpretable and explainable by implementing strategies like feature importance attribution [44].

Second, we want to explore other neural network architectures for co-expression-based AFP. We chose to build our model using feedforward neural networks because we wanted to compare the simplest deep learning architecture to MEFIT and SPELL. We found that feedforward neural networks can provide a modest increase in ROC performance over Bayesian networks and directed query searches (Fig 4). Now that we have established this baseline for deep learning models, we plan to compare feedforward networks to other neural network architectures. In particular, we believe we can improve performance by using graph neural networks to directly learn functional relationship graphs rather than constructing them from the pairwise predictions made by our current model.

The last portion of our future work lies in testing the performance of models that integrate other modalities of biological data into their input other than just co-expression. As illustrated in Fig 3, gene co-expression data alone is not informative of every biological process a gene can participate in. That being said, integrating other data modalities into a model’s input can compensate for the lack of signal in gene expression data for some biological processes. For instance, BioPIXIE is a Bayesian network based model (similar to MEFIT) that includes physical interaction, genomic interaction, and cellular localization data in addition to co-expression data in its input. We plan to augment our model by including these data modalities in our input features [6]. Then, we want to compare this model to bioPIXIE to see if neural networks can improve performance by learning patterns between data modalities (which naïve Bayesian networks are unable to do).

In summary, we directly compared neural networks to two classical, experimentally validated machine learning models for co-expression-based AFP. We found that our model outperformed these prior methodologies at predicting the involvement of yeast genes in mitochondrion organization, in part due to being robust to mis-annotated negatives in its training data. As such, we suspect that the predictions for other biological processes made by our model are similarly useful for directing laboratory biology experiments. Overall, we found that a simple feedforward neural network can compete with or outperform earlier co-expression-based AFP models such as MEFIT and SPELL, which highlights how architectural choices and differences can influence robustness to annotation noise and data availability for predicting a gene’s biological function.

Supporting information

Acknowledgments

We would like to thank Dr. Bethany Strunk and Dr. Brian Teague for their help in editing the manuscript. We would also like to thank the members of Strunk Lab for their thoughtful conversations.