Fig 1.
IV-KAPhE is a multi-label kinase-phosphosite assignment method that nests a multi-label model of in vitro assignment in a model of in vivo functional association.
Naïve Bayes+ consists of sub-models for each kinase, trained from kinase-specific, high-throughput in vitro kinase-substrate relationships. These sub-models together comprise a final, multi-label Naïve Bayes model. IV-KAPhE is a monolithic, multi-kinase Random Forest model trained from all literature-derived kinase-substrate annotations and random pairs as negative cases.
Fig 2.
Naïve Bayes models of in vitro kinase-phosphosite assignment have important performance differences from PSSM-based methods.
a) PSSM methods and Naïve Bayes perform similarly in cross-validation of multi-label kinase-substrate assignment via macro-averaged precision versus recall. The expanded Naïve Bayes+ model outperforms the other methods. Points indicate the scores at the cutoff that maximizes that macro-F1 score. Black error bars showing 95% confidence intervals at these points are indiscernible in most cases, indicating highly robust performance across cross-validation folds. b) The macro-averaged F1 scores behave differently with score/probability cutoff for scoring matrix-based models versus Naïve Bayes. PSSM and PFM-based models require a strictly defined cutoff. Naïve Bayes+ again outperforms the others and retains the same flat relationship with cutoff as basic Naïve Bayes. Points indicate the maximum value. Bands indicate the 95% confidence interval. Color assignments are the same as in (a). c) Example score distributions for a S/T kinase (AKT1) and a Y kinase (FYN) from one round of cross-validation. For S/T kinases, Naïve Bayes probabilities are largely distributed close to 0.0 and 1.0 while PSSM scores take more intermediate values, notably including scores for Y sites. Y kinases show better separation for both methods. d) Left: Logistic curves relating phosphoproteome-backed PSSM scores to Naïve Bayes probabilities. Each curve represents a fitted logistic function for each kinase. The color of the curve represents the number of kinase substrates used to fit each specificity model. Right: The fitted logistic curve parameters versus number of substrates. S/T and Y kinases have negative relationships between inflection point and numbers of substrates. e) Min-max normalization of PSSM scores does not produce a stable inflection point independent of the number of substrates.
Fig 3.
Additional predictive features of in vitro kinase substrates can be discriminated in the training data.
Distributions of empirical probabilities are represented by the kernel density estimate of the kinases’ respective Bernoulli probability parameters. Each feature trends towards higher probability in in vitro substrates than in other sites.
Fig 4.
IV-KAPhE performs well in cross-validation and significantly outperforms previously published methods on an external validation set.
a) Naïve Bayes+ posterior probability, GO BP semantic similarity, and STRING experimental score had the greatest importance when training the Random Forest models. Error bars show standard error across cross-validation runs. b) Predictive performance of individual quantitative features, as assessed by average macro-precision and macro-recall across 10 folds of the training data, reveals GO BP semantic similarity and STRING experimental score as being the most predictive individual features. c) Cross-validation evaluated via macro-averaged precision, recall and F1 all reflect strong performance by IV-KAPhE. d) IV-KAPhE’s coverage of the external test data set is similar to LinkPhinder’s but is lower than that of GPS 5.0. e) Kinase-specific F1 scores reveal IV-KAPhE’s consistently strong performance across most kinases, with similar performance for S/T and Y kinases, compared to other methods. f) IV-KAPhE outperforms the simpler PSSM-based and Naïve Bayes+ methods as well as other previously published methods in kinase-substrate assignment of an external validation set. Points indicate the scores for simple assignments (GPS) or the scores at nominal cutoffs for quantitative predictions (cutoffs—IV-KAPhE: 0.5, PSSM: 0.75, Naïve Bayes+: 0.5, LinkPhinder: 0.672 [27], NetworKIN 3.0: 1.0 [40]). Error bars show the 95% confidence intervals at these points. g) IV-KAPhE has a higher macro-averaged F1 score than the other methods. Points and color assignments are as in (e). Bands indicate the 95% confidence interval. h) IV-KAPhE similarly outperforms the other methods in Receiver Operating Characteristic (ROC) curve analysis for this balanced test set. Points and color assignments are as in (e). Error bars show 95% confidence intervals. i) Focusing on multi-label assignment for sites in the test set with known kinases, the macro-averaged false discovery rate (FDR; i.e. rate of novel assignments) dominates the average true positive rate (TPR). The curves are similar for most methods. At its nominal cutoff, IV-KAPhE has the second-highest FDR, but it is matched by the highest TPR.
Fig 5.
Phosphoproteome-wide kinase assignments suggest more widespread multi-kinase phosphorylation than existing literature annotations do.
a) Histograms of the number of kinases associated with sites in the phosphoproteome reveal different views of the phosphorylation network. Literature annotations in PhosphoSitePlus suggest most sites are regulated by one or two kinases. In vitro Naïve Bayes+ predicts some S/T sites are “hubs” and all Y sites can be phosphorylated by most Y kinases. LinkPhinder and IV-KAPhE, in contrast, predict a long tail of hub sites. b) Histograms of the median number of kinases assigned per site for all proteins likewise show different predictions for hub proteins. Literature annotations suggest most proteins are phosphorylated by one kinase at each site. The computational methods all hypothesize multiple kinases per site, with some substrate proteins being very promiscuous at all their sites.
Fig 6.
IV-KAPhE can identify instances where literature-derived kinase-phosphosite assignments are probably assigned in error to commonly studied isozyme, instead of a lesser-studied one.
a) IV-KAPhE assignments of ribosomal protein S6 kinase alpha isozymes for all sites annotated in PhosphoSitePlus as being phosphorylated by at least one of the isozymes. Red colors indicate assignments predicted as likely by IV-KAPhE. Highlighted sites, discussed in the text, are examples that IV-KAPhE predicts are more likely to be phosphorylated by a different isozyme than the one annotated. b) IV-KAPhE assignments of calcium/calmodulin-dependent protein kinase type II subunits to annotated sites, as described in (a).
Fig 7.
IV-KAPhE enables more robust and consistent kinase-activity inference.
a) Target kinases and their downstream substrate kinases in a multi-inhibitor quantitative phosphoproteomics experiment [41] are expected to have altered enzymatic activities. Assignments derived from NetworKIN 3.0, GPS 5.0, and IV-KAPhE make stronger inferences than those from LinkPhinder or in vivo literature-derived annotations from PhosphoSitePlus, however these methods also erroneously predict increased activity in some target kinases and downstream substrate kinases that they enzymatically activate. Each column represents a different kinase inhibitor condition (see Table A in S1 Text), in which green dots are direct targets of the inhibitor, orange triangles are kinases that are enzymatically activated by a target kinase, and violet squares are kinases that are enzymatically inhibited by a target kinase. Gray, dashed lines indicate activity levels of -2.5 and 2.5, corresponding to Z-test p-values of 10−2.5. b) IV-KAPhE provides more consistent inference of negative activity in target kinases and substrate kinases that they enzymatically activate than other computational methods as well as in vivo literature-derived annotations from PhosphoSitePlus. Point colors and shapes, as well as the gray dashed lines, are as described for panel (a).