Descriptors

As hinted in the introduction, descriptors are numerical values that quantify several characteristics (either biological or physico-chemical) of a protein sequence. Below, we review the two major categories of descriptors.

Protein sequence embeddings.

Protein sequence embeddings are a type of representation for proteins that embodies the biological information inherent in the sequence. They are inspired by techniques in natural language processing, particularly word embeddings [28], which represent words as vectors in a way that captures semantic relationships between them.

Protein sequence embeddings are derived from protein language models (PLMs). These are essentially deep neural networks trained on vast collections of protein sequences by minimizing a self-supervised objective known as masked language modeling [17]. In simple terms, the model is tasked with predicting amino acids in the sequence that have been intentionally hidden or “masked,” forcing the model to learn from the context in which the missing amino acid appears. This approach is similar to the original training of word embeddings, where a word is predicted based on its context within a sentence. As a result of this process, each amino acid in the sequence is represented as a high-dimensional vector.

Each entry in the vector can be thought of as a feature that captures some aspect of the protein’s biological or physico-chemical properties. The exact interpretation of these features can be complex, as they are learned by the model and not directly tied to specific, predefined characteristics. However, the key is that proteins (or amino acids) that are similar in some biological sense will have similar embeddings. PSEs have been shown to encode various hallmarks of biological sequences, ranging from simple physico-chemical properties to more complex ones such as remote homology [29]. Thus, they have been used to learn a variety of sequence-based predictive tasks [18–20], although their use for antigen prediction has not been explored yet.

Related work

In the past years, several RV methods to identify potential PAs out of bacterial proteomes have been proposed. They can be roughly categorized into filtering approaches and ML approaches.

Filtering approaches such as NERVE [30], Vaxign [31], Jenner-predict [32], VacSol [33], work by applying filters to the candidates until a viable subset is found. These filters are typically based on applying a cut-off value to biological or physico-chemical descriptors. Multiple filters are applied sequentially, in no particular order among themselves. For example, one could decide to exclude (from an initial set of candidates) protein sequences with predicted probability of being an adhesin below 0.5. On the remaining set, one could further decide to exclude those with predicted number of transmembrane domains below 2, and so on.

Traditional ML approaches such as VaxiJen [34], the methods by Heinson [35] and Bowman [25], VaxignML [21], and and deep learning-based approaches such as Vaxi-DL [36], Vaxign-DL [37] learn a model that classifies protein sequences as PAs or non-PAs from a labeled dataset of protein sequences. Crucially, in contrast with filtering approaches, these methods do not discard candidates, but rather assign a score (generally a probability) that indicates the likelihood of being PAs. Typically, trained ML models are applied to the entire proteome of a bacterial species for which a vaccine is sought. The proteins can be ranked according to the probability assigned by the model, indicating which ones are more likely to be protective.

In this work, we set our focus on this second category. To the best of our knowledge, using PSEs to represent sequences in this applied context has not yet been explored by the RV community, which usually relies on representing protein sequences as vectors of descriptors.

Data collection and preprocessing

We collected 708 unique PA sequences from databases Protegen [38] and VaxiJen [34] with a corresponding UniProt entry. We removed homologs by clustering the sequences with mmseqs-cluster [39] (sequence identity > 0.3, coverage > 0.5), taking one representative sequence for each cluster found. This narrowed down the number of PA sequences to 458 (comprising 82 different bacterial species).

While Protegen and VaxiJen only contain PA sequences, ground truth non-PA sequences are not available for this predictive task. Therefore, we artificially created non-PA samples following previous works [21,35]. The process to add a protein sequence to the set of non-PA sequences is exemplified in Fig 2. For a given PA sequence, we randomly sampled from UniProt one candidate non-PA sequence of the same bacterial species. We rejected the candidate if found to be similar to any PA sequence in the set of antigenic sequences or to non-PA sequences already accepted (sequence identity > 0.3, coverage > 0.5, e-value < 0.001). Otherwise, we accepted the candidate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Flowchart of the non-PA sequences selection process.

Given a reference PA sequence, a non-PA candidate sequence is drawn at random from the reference’s proteome in UniProt. The two sequences are checked for similarity: if they are similar, the candidate is rejected and a new one is drawn. If they are not, the candidate is tested for similarity against all previously accepted non-PAs. If they are, the candidate is rejected and a new one is drawn. Otherwise, the candidate is added to the set of non-PA sequences. This process is repeated for each PA sequence available.

https://doi.org/10.1371/journal.pone.0323895.g002

We finally assigned to each PA sequence a positive label (1), and to each non-PA sequence a negative label (0). In total, the constructed dataset contains 916 labeled sequences.

Computational pipeline

The core object of this study is a predictive pipeline that accepts protein sequences in FASTA format as input and yields a probability value for each protein sequence, indicating their likelihood of being a PA. Irrespective of the numerical representation of protein sequences (be it PSEs or descriptors), the pipeline can be deconstructed into three distinct sub-modules. The initial module, referred to as the Extraction module, processes protein sequences in FASTA format and converts them into numerical vectors with components, depending on whether a PSE-based or descriptor-based representation is chosen. The subsequent module, known as the Preprocessing module, applies various transformations to the vectors, such as scaling and optional feature selection. The final module, termed the Classification module, accepts the preprocessed vectors and inputs them into a calibrated classifier that generates the ultimate probabilistic predictions. Importantly, the exact configuration of the Preprocessing and Classification modules (e.g., the amount of feature selection, the specific classifier) is chosen during model selection to maximize performances. The pipeline is depicted in Fig 3. Notice that, once trained, the pipeline can be applied to predict the proteins of any bacterial species without retraining, regardless of whether the species is rare or under-studied.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Overview of the computational pipeline.

The pipeline proposed in this study is composed of three modules, which vary depending on how protein sequences are represented (blue path: PSE-based; red path: descriptor-based). Green boxes represent fixed operations that are applied once to the sequences, while red boxes are tuned during the experiments, meaning that their configuration is chosen during model selection to maximize performances. Orange boxes indicate numerical quantities, either vectors or scalars.

https://doi.org/10.1371/journal.pone.0323895.g003

In the remainder of this section, we discuss the specific modules in detail in relation to the protein representation of choice (PSE- or descriptor-based).

Classifier calibration

While this work deals with a probabilistic binary classification task, not all the classifiers described above return proper probabilities. For example, the SVM predicts a score which is related to the distance of the input from the margin. To ensure that the classifiers output proper probabilities, we applied calibration to the models after training. Specifically, model calibration tries to enforce the following property on the outputs of a classifier:

where P indicates probability, Y is a discrete random variable which ranges in the set of possible labels, and is a continuous random variable ranging in the interval [0,1]. In words, calibration consists of adjusting the classifier’s outputs so that they match the observed class frequencies. In this work, calibration is performed using Platt’s Scaling [47]. Basically, the method produces calibrated probabilities with the following logistic regression model:

where parameters are estimated with maximum likelihood.

Model selection

During the evaluation process, we used randomized hyperparameter search [48] to optimize the pipeline steps. This included the scaling method (applicable only for the descriptor-based pipeline), the decision to apply feature selection and its extent, the choice of classifier and its hyper-parameters. Randomized hyper-parameter search entails sampling hyper-parameter combinations from predefined distributions and assessing the model performance with each combination. In this study, each hyper-parameter combination was evaluated using 5-fold Cross-Validation (CV). The combination that produced the best average negative log-likelihood (NLL) across the 5 validation folds was chosen. We used NLL to select the best pipeline since it has been shown to consistently drive the selection towards well-calibrated classifiers [49]. The resulting optimized pipeline was then trained on the complete training set and subsequently evaluated on a separate test set to obtain an unbiased performance estimate. The full table of the hyperparameters that were optimized during model selection is shown in S1 Table. Notice that in this study we do not take into account class imbalance, meaning that PAs and non-PAs are in the same proportion within the training set. The rationale of this choice is two-fold. On the one hand, previous studies in PA classification found that performances were not significantly impacted when the dataset is artificially imbalanced [21]. On the other hand, dealing with imbalanced data with over-sampling or under-sampling is known to negatively affect model calibration [61].

PSE-based vs. descriptor-based model evaluation

The initial experiment involved a comparative analysis of the descriptor-based pipeline and the PSE-based pipeline. The objective was to ascertain which data representation – descriptors or PSEs – offers superior predictive accuracy in identifying protective antigens. It is important to remark that we are not evaluating a single classifier, but the entire pipeline, which also includes the model selection of the classifier. We employ a Leave-One-Bacteria-Out (LOBO) evaluation scheme: for a given bacterial species, all protein sequences in the labeled dataset belonging to that species are allocated to the test set, while the remaining sequences are assigned to the model selection set. This approach tests the model’s ability to generalize to unseen bacterial species, aligning with methodologies used in prior studies [21,36,37] and to the typical RV stage of vaccine research and development projects in the pharmaceutical industry. We performed LOBO evaluations on 10 bacterial species (6 Gram positive, 4 Gram negative): Actinobacillus pleuropneumoniae, Campylobacter jejuni, Chlamydia muridarum, Escherichia coli, Mycobacterium tuberculosis, Neisseria meningitidis, Staphylococcus aureus, Streptococcus pneumoniae, Streptococcus pyogenes, and Yersinia pestis. Fig 4 shows the workflow of the chosen LOBO evaluation procedure.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Leave-One-Bacteria-Out (LOBO) experimental setup.

NLL: negative log-likelihood; AUROC: Area Under the Receiver Operating Characteristics curve; AUPR: Area Under the Precision-Recall curve; WF1: Weighted F1 score; MCC: Matthews Correlation Coefficient.

https://doi.org/10.1371/journal.pone.0323895.g004

Metrics. For each bacterial species tested, we used the trained model to predict its (unseen) protein sequences. We employed the following metrics for the comparison, similarly to previous works such as [21,36,37]: Area Under the Receiver Operating Characteristic (AUROC), Area Under the Precision-Recall curve (AUPR), Weighted F1 (WF1), and Matthews Correlation Coefficient (MCC). To smooth out the effect of randomness, the training procedure was repeated 10 times, each time using a different fixed seed for random numbers generation. We report the mean of each performance metric across these 10 trials. Once a random seed is fixed during a trial, the two performances (PSE-based vs. descriptor-based) are comparable fairly since they use the same data splits and undergo an identical model selection.

Benchmark evaluation

In this experiment, we compare the PSE-based pipeline to methods from the literature on the iBPA benchmark [21]. The original iBPA benchmark includes 249 proteins, but we were only able to recover 243 upon request to the authors. To ensure a fair comparison, we trained the PSE-based pipeline using the same dataset as the reference work. Essentially, the training dataset for this task consisted of 397 antigens and 397 non-antigens. We excluded 3 proteins from the original dataset due to unclear or non-bacterial species. During training, the PSE-based pipeline was optimized with the same hyper-parameter tuning procedure as in the LOBO evaluation. However, we fixed the classifier to SVM since in the LOBO evaluation it performed better than the alternatives 71% of times on the validation set. We compared against 8 different methods from the literature, reusing the results from the work of [37]. We remark that the PSE-based pipeline is trained with 394 antigens out of the original 397 (99.2%), and tested on 243 proteins out of the original 249 (97.5%). We therefore consider the comparison fair to the best of our possibilities.

Candidate selection simulation

In this experiment, we simulate a situation where a researcher, provided with a bacterial proteome, must determine the order in which candidate antigens should proceed to pre-clinical tests (assuming a sequential testing protocol). The optimal approach would prioritize actual antigens for testing, thereby saving pre-clinical resources by minimizing tests on non-antigens. We compare two distinct strategies for establishing this testing sequence:

• one that ranks the candidates according to their biological descriptors, similar to the filtering approach used in RV, which we term the RV-based approach,
• and another that exploits the output probability given by the PSE-based pipeline trained during the LOBO evaluation to rank candidates, which we call the likelihood-based approach.

We downloaded the entire proteomes of the 10 test species mentioned above from UniProt. These proteomes were enriched by adding, for a given species, all PA sequences found in our dataset (taking care of removing duplicate sequences). Since surface exposure is the gold standard that all candidates must adhere to in order to be eligible, the comparison was restricted to protein sequences whose subcellular localization was predicted as “extracellular space” or “outer membrane” by the BUSCA software package [50]. After filtering for surface exposure, we assigned a positive label to known PAs (1), and a negative label (0) to the remaining proteins. Note that this approach is conservative, since proteins that we deem as negative are actually untested, and therefore it is unknown whether they are PA or not.

The RV filtering strategy was implemented by arranging the protein sequences using 4 descriptors as sorting keys: probability of having signal peptides (from high to low), probability of being an adhesin (from high to low), number of transmembrane helices (from large to small), immunogenicity score (from high to low). The order of precedence among the 4 keys was randomly determined in each trial (since there is no universal preference criteria among them). Conversely, the likelihood-based strategy was implemented by sorting the protein sequences in descending order according to their likelihood of being PAs. More precisely, given a bacterial species, the likelihoods were assigned from the corresponding PSE-based pipeline trained during the LOBO evaluation (meaning that the protein sequences of the bacterial species under study were never seen during training).

Metrics.

To compare performances in this experiment, we devised a novel metric designed to reward assigning higher rank to known PAs. Given a bacterial proteome composed of m protein sequences , let be a vector that labels the sequences in S as known PAs (1) or non-PAs (0). Let be the “ideal” permutation of that rearranges all known PAs first, and be the label vector rearranged accordingly. Similarly, let be a permutation to evaluate, and be the label vector rearranged accordingly. In practice, the permutation is obtained by arranging the sequences in S according to some precomputed ranking (be it RV-based or likelihood-based). Finally, let be a cumulative sum function defined over vectors as follows:

(1)

Conceptually, the vector returned by is a histogram where the i-th bar represents the running sum up to the i-th element of the input vector. With these information, we can define the normalized Antigen Discovery Rate (nADR) as:

(2)

where the i subscript is used to denote the i-th component of the vector returned by . In practice, the first calculates the area of the two histograms (by summing up the histogram bins) and then computes the ratio of the two areas obtained. Intuitively, it quantifies how closely the evaluated ranking matches the ideal ranking, with a higher score indicating a better match, akin to information retrieval metrics like the normalized Discounted Cumulative Gain [51]. A simple illustration of how the metric is computed is shown in Fig 5. Note that has many desirable characteristics: it ranges in the interval (0,1] since holds trivially for ; it is independent of L since it is normalized; and it equals 1 if and only if (i.e., when the evaluated ranking is optional). It is not defined when the denominator is 0, i.e., when the proteome does not contain known PAs, in which case we simply force it to be 0. In our experiments, is computed 10 times with different random seeds for each held-out bacterial species. We report the mean of these trials.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Example of the nADR metric computation.

We employ a toy proteome composed of m = 5 protein sequences, of which 3 are known PAs. In the picture, transforms the input vector in a histogram of running sums, while sums the histogram bins.

https://doi.org/10.1371/journal.pone.0323895.g005

To further characterize the quality of the rankings, during the experiments we also monitor an index that we term the First Hit Index (), defined as:

(3)

which stores the position of the first known antigen in ranking order. Intuitively, it can be interpreted as the minimum number of pre-clinical tests needed to recall a known antigen. Referring back to Fig 5, we have that = 2.

Fold enrichment analysis

To further characterize the results of the previous experiment, we conducted a fold enrichment analysis of known PAs within the 90-th percentile of the average (across 10 different random seeds) ranking produced by the likelihood-based strategy. In simpler terms, we checked the statistical over-representation of known PAs within the high-probability subset of the proteome. The fold enrichment is defined as the ratio between the observed number of PAs in the 90th percentile of the ranking and the expected number across the whole proteome. On the same subset, we computed the recall metric, defined as the percentage of known PAs found in the 90-th percentile. To add statistical significance to the results, we also computed the p-values (at the conventional 0.05 cutoff) associated to the enrichment with a hypergeometric test.

PSEs outperform descriptors in LOBO evaluation

Table 2 summarizes the results of the evaluation as described in the Experiments section. In 7 out of 10 species (A. pleuropneumoniae, C. jejuni, C. muridarum, E. coli, M. tuberculosis, N. meningitidis, S. pneumoniae), the PSE-based pipeline outperforms the descriptor-based pipeline across all measured metrics. In two cases (S. aureus and Y. pestis), PSE-based performs better on 3 metrics out of 4. Only in one case (S. pyogenes), the descriptor-based pipeline performs better than the PSE-based baseline (in 3 metrics out of 4). Lower generalization of S. pyogenes’ PSEs with respect to descriptors might be caused by an underrepresentation of sequence characteristics or distributional properties in the sequences used to pretrain the ESM-2 model. More precisely, it could be that the S. pyogenes sequences, while antigenic, are different from all other sequences (perhaps due to peculiarities of the sequencing process). In that case, it might be that this variability has been overfitted by PSEs, while descriptors are able to better generalize. However, ours is merely a hypothesis that needs to be tested in further studies. Indeed, the gap between PSEs and descriptors, while in favor of the latter, is still narrower than the other cases. Also, the MCC metric, which measures correlation between the predictions and the actual antigenicity, is slightly in favor of PSEs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Leave-One-Bacteria-Out evaluation of descriptor-based and PSE-based pipelines.

https://doi.org/10.1371/journal.pone.0323895.t002

Averaging performances across species (last row of the table), we found that for all metrics, the PSE-based baseline performed better than descriptor-based models. This result provides clear evidence that PSEs are a more effective means to represent proteins for RV tasks, with superior performance in general.

PSE-based pipeline outperforms competitors in iBPA benchmark

Table 3 reports the results of the benchmark evaluation on the iBPA dataset, as detailed in the Benchmark evaluation section. The PSE-based pipeline outperformed all the competitors in all the benchmark metrics. Notably, we report a 6.75% improvement in WF1 and a 11.8% improvement in MCC with respect to the previous best model. Summing up, while the previous experiment demonstrated that the PSE-based pipeline proposed in this study generalizes better than descriptor-based pipelines in a LOBO setting, this result demonstrates that it generalizes better than all other methods in the literature on unseen proteins, regardless of the bacterial species.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Comparison of our PSE-based pipeline with existing works on the iBPA benchmark.

https://doi.org/10.1371/journal.pone.0323895.t003

Likelihood-based ranking outperforms RV-based ranking in simulated candidate selection

Table 4 shows the results of the simulated candidate selection for pre-clinical tests, where we compare the proposed likelihood-based approach against the RV-based approach described in the Candidate selection simulation section. In all the 10 test species under study, the likelihood-based strategy significantly improves on the metric with respect to the baseline, by 49% on average. By inspecting the FHI, we can also observe that in 8 cases out of 10 (i.e., excluding M. tuberculosis and Y. pestis), using the likelihood-based strategy leads to the re-discovery of the first known PA within at most 4 pre-clinical trials on average, while the RV-based strategy, in the same cases, requires at least 6 and at most 41. Even in the most difficult cases of Y. pestis (resp. M. tuberculosis), our strategy requires 18 (resp. 12) pre-clinical tests to re-discover the first known PA, while the RV-based strategy requires 125 (resp. 38). Overall, our approach allows to save (on average across the 10 species) up to 83% of pre-clinical tests with respect to the RV-based strategy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Simulated candidate antigen selection for pre-clinical testing

https://doi.org/10.1371/journal.pone.0323895.t004

The evolution of the simulated pre-clinical tests is shown in Fig 6, where in the x-axis we indicate the number of pre-clinical trials (assuming they are performed sequentially), and on the y-axis we plot the cumulative distribution of known PAs re-discovered according to the likelihood-based strategy (green line) versus the RV-based strategy (gray line). As can be seen in all the species under study, the rate at which PAs are re-discovered with the likelihood-based approach is consistently above that of the RV-based approach, which indicates that to re-discover the same number of known PAs, the proposed method requires a smaller number of pre-clinical tests.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison between likelihood-based strategy and RV-based strategy at discovering novel antigens (simulation).

The plots display the number of pre-clinical trials (x-axis) in relation to the cumulative distribution of known PAs (y-axis) re-discovered with the proposed likelihood-based strategy (green line), compared to the RV-based strategy (gray line). Both strategies are described in the Candidate selection simulation section. Shading around the lines identify a 95% confidence interval around the mean of the cumulative distribution. In all cases, the green line is above the gray line, indicating that to re-discover the same number of known PAs, our method requires a smaller number of pre-clinical tests.

https://doi.org/10.1371/journal.pone.0323895.g006

High-probability subset of likelihood-based ranking is enriched with known PAs

The fold enrichment analysis, presented in Table 5, shows a statistically significant over-representation of known PAs within the 90th percentile of the ranking (i.e., the subset containing high-probable PAs according to the likelihood-based strategy) compared to the expectation under random sampling. The result is consistent across the 10 species evaluated, with p-values below the 0.05 cutoff. Remarkably, the proposed method is capable of recalling 36% of known PAs within the 90th percentile of the ranking (on average across the 10 species). These results suggest that the PSE-based pipeline is assigning known PAs to the top positions of the ranking at a higher rate than a random model, which conforms to our expectations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Fold enrichment analysis of Protective Antigens (PA) in the 90th percentile likelihood-based ranking.

https://doi.org/10.1371/journal.pone.0323895.t005