An overview of ThermoNet

To predict the ΔΔG of a point mutation, we take advantage of recent advances in deep learning for computer vision [29] and the successes of deep CNNs in biophysical problems [27,28,30–32]. We treat protein structures as if they were 3D images where voxels are parameterized using atom biophysical properties [27,28] (Fig 1A and 1B, and Table 1). For a given single-point mutation, ThermoNet requires that a 3D structure (either experimentally determined or modeled via homology modeling) of one of the alleles is available. As a first step, ThermoNet constructs a structural model for the mutant from the structure of the wild type using the Rosetta macromolecular modeling suite (Methods) [9,33]. ThermoNet assumes that the ΔΔG of a point mutation can be sufficiently captured by modeling the 3D biophysical environment around the mutation site. It thus extracts predictive features by treating protein structures as if they were 3D images and voxelizing the space around the mutation site of both the wild-type structure and the corresponding mutant structural model (Fig 1C). Each voxel is parameterized with seven predefined rules (Table 1) to characterize the biophysical nature of its neighboring atoms. The feature maps are then stacked to create a tensor with size [16, 16, 16, 14] as input to the trained ensemble of ten deep 3D-CNNs, which generate a prediction of the ΔΔG the given mutation causes to the wild-type structure (Fig 1D and 1E, and Methods). Each of the component 3D-CNN models consists of three 3D convolutional layers with 16, 24, and 32 neurons respectively and one densely connected layer of 24 neurons (Fig 1E). These architectural hyperparameters (i.e. number of neurons in convolutional and densely connected layers and sizes of the input voxel grid) were tuned via five-fold cross-validation (Methods, S1 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Biophysical property channels for protein structure voxels.

https://doi.org/10.1371/journal.pcbi.1008291.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. An overview of the ThermoNet computational framework.

(A) Protein structures are treated as if they were 3D images. A 16 Å × 16 Å × 16 Å cubic neighborhood centered at the C_β atom (red sphere) of the mutated residue (or C_α atom in the case of a glycine) of an example protein (PDB ID: 1L63) is discretized into a 3D voxel grid at a resolution of 1 Å. Each voxel is represented by a gray dot. (B) Just as an RGB image has three color channels, the 3D voxel grid is parameterized with seven biophysical property channels: hydrophobic, aromatic, hydrogen bonding donor, hydrogen bond acceptor, positive ionizable, negative ionizable, and occupancy. The saturation level of each voxel ranges from 0.0 to 1.0 and is colored accordingly (Methods). (C) To predict the change in thermodynamic stability caused by a given single-point mutation, ThermoNet calls Rosetta to refine the wild-type structure and to create a structural model of the mutant protein. (D) ThermoNet voxelizes the space around the mutation site of both the Rosetta-refined wild-type structure and the corresponding mutant structural model. Both the 3D voxel grid of the wild-type structure and that of the mutant model are parameterized accordingly to create two [16, 16, 16, 7] feature maps. (E) The feature maps are then stacked to create a [16, 16, 16, 14] tensor as an input to the trained deep 3D convolutional neural network. The final output of the network is the predicted ΔΔG the given mutation causes to the wild-type protein structure.

https://doi.org/10.1371/journal.pcbi.1008291.g001

Creating data sets for robust training and testing of ThermoNet

The ability of a machine-learning model to generalize can be overestimated when there is data leakage between the training set and the test set. In structural bioinformatics problems such as ΔΔG prediction, such data leakage can result when the training set contains proteins that are homologous to proteins in the test set. This is because the effects of different mutations in the same protein or homologous proteins are not necessarily independent. In most previous methods for ΔΔG prediction, this data leakage issue was not fully appreciated, and homologous proteins were present between training and test sets. For example, a recent method used randomly selected subsets of mutants from the widely used S2648 data set for training and testing [34]. Not surprisingly, the training and test sets of shared 61 identical proteins (S1 Table).

The issue of having mutations from the same protein in both training and test sets was addressed in developing the mCSM and INPS methods, where the cross-validation procedure ensured that mutations of the same protein remained together in either the training or test set [16,19]. While this was a step in the right direction, grouping mutations at the protein level is not sufficient to remove the homology between training and test sets. For example, we found substantial homology between 132 proteins within S2648 (S2A Fig). Thus, splitting the S2648 data set for training and testing at the protein level is likely to end up with shared homology between the splits. In fact, using the PISCES server [35] to remove redundancy from the S2648 data set resulted in only 104 non-redundant proteins (out of 132) at the level of < 25% sequence identity (S2 Table). Homology is common among data sets used in training ΔΔG predictors; for example, many proteins in the VariBench [36] data set also share substantial homology (S2B Fig).

We further highlight that the widely used S2648 data set includes fourteen proteins from S^sym data set, which was previously used to evaluate a wide range of ΔΔG predictors [24,37], and another eight proteins that are putative homologs to proteins in S^sym (Fig 2A, S3 Table). In addition, a similar level of overlap also exists between the VariBench and Q3421 data sets and S^sym (Fig 2A, S4 and S5 Tables). Thus, performance estimates on S^sym of methods trained using S2648 or parameterized using VariBench are likely to be overly optimistic. For example, in a recent publication, a version of INPS trained using a data set obtained by removing all proteins with > 25% sequence identity to proteins in S^sym showed substantially reduced performance [38].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Data set curation and identification of shared homology.

(A) Venn diagrams showing the amount of overlap at the protein level between three widely used training sets S2648, VariBench, and Q3421 for ΔΔG predictors and the S^sym test set. Numbers in these diagrams indicate protein counts. Upper panel and lower panel indicate that both S2648 and Q3421 share 14 identical proteins with S^sym; middle panel indicates that VariBench and S^sym share 11 identical proteins. All three data sets share additional homology with S^sym, which is presented in S3, S4, and S5 Tables, respectively. (B) Creating data sets for robust training and testing of ThermoNet. We started with the Q3421 set of 3421 mutations from 150 proteins. (Numbers in data set names indicate the number of unique mutations the data set contains.) After homology reduction and anti-symmetry data augmentation (Methods), this data curation workflow gives a training set of 3488 mutations with an equal representation of stabilizing and destabilizing changes and reduced homology to the S^sym test set. A separate data set called Q6428 was also created by augmenting the Q3214 data set before homology reduction to train ThermoNet*.

https://doi.org/10.1371/journal.pcbi.1008291.g002

Thus, to train and evaluate ThermoNet, we implemented a rigorous procedure to reduce the sequence similarity between the training and test sets (S^sym, p53, myoglobin) by removing duplicate data points and pruning protein-level homology (Fig 2B). Our rigorous pruning of the starting Q3421 data set [15] resulted in a data set consisting of 1,744 distinct mutations. This data set was then augmented by creating a reverse mutation data point for each of the 1,744 direct mutations according to the anti-symmetry property of ΔΔG (Methods), thus giving to a total of 3,488 data points for the training of ThermoNet (Fig 2B). While the pruning nearly halved the size of available training data, as discussed in the following section, models trained using the resulting augmented Q3488 data set will be less likely to have an overestimated performance when evaluated on S^sym. Finally, to explore the influence of homology between training and test sets on estimates of model performance, we also augmented Q3214, the intermediate data set before the step of homology reduction, to train a different version of ThermoNet, called ThermoNet* (Fig 2B).

ThermoNet achieves state-of-the-art performance on blind test set

We systematically compared ThermoNet and ThermoNet* with seventeen ΔΔG predictors on the S^sym balanced data set to evaluate their performance and degree of bias with respect to the ΔΔG anti-symmetry between direct and reverse mutations (Methods). A brief summary of the characteristics of these ΔΔG predictors and their references are given in S6 Table. In short, the predictors are based on diverse features and strategies with some, like ThermoNet based only on structural information, while others like DDGun3D [37] and STRUM [15], integrate structural information with sequence and evolutionary features. The S^sym data set, which was constructed previously for assessing the biases of ΔΔG predictors [24], consists of experimentally measured ΔΔG values for 342 direct and the corresponding reverse mutations (a total of 684 mutations) from fifteen protein chains for which the structures of both the wild-type and mutant proteins have been resolved by X-ray crystallography with a resolution of 2.5 Å or better. This data set is by construction balanced with respect to stabilizing and destabilizing mutations, thus enabling the evaluation of prediction bias. However, as noted in the previous section, many proteins in S^sym overlap or are homologous to proteins in commonly used training sets (Fig 2A).

To evaluate performance, we computed the root-mean-square error σ and the Pearson correlation coefficient r separately for direct and reverse mutations. We measured prediction bias by two statistics, the Pearson correlation coefficient r_dir−rev between the predictions for direct and those for reverse mutations and the δ value, defined as: δ = ΔΔG_rev + ΔΔG_dir. A perfectly unbiased predictor would give r_dir−rev = −1 and 〈δ〉 = 0 kcal/mol, where 〈δ〉 is the average of δ.

ThermoNet achieves strong prediction accuracy that is comparable for direct mutations (r_dir = 0.47 and σ_dir = 1.56 kcal/mol; Table 2, Fig 3A) and the corresponding set of reverse mutations (r_rev = 0.47 and σ_rev = 1.55 kcal/mol, Fig 3C). This suggests that ThermoNet did not overfit to direct mutations. The fractions of mutations for which the prediction error is within 0.5 kcal/mol and 1.0 kcal/mol are 36.3% and 58.8% for direct mutations and 36.5% and 58.5% for reverse mutations (Fig 3B and 3D). ThermoNet successfully reduces prediction bias with a near-perfect r_dir−rev (-0.96) and a negligible 〈δ〉 (−0.01 kcal/mol) (Fig 3E). We also report the distribution of δ, since 〈δ〉 cannot distinguish large, but symmetric, bias from low bias (Methods). As shown in Fig 3F, 40.9% and 96.2% of mutations have a prediction bias < 0.1 kcal/mol and < 0.5 kcal/mol, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Comparative analysis using the balanced test set S^sym.

https://doi.org/10.1371/journal.pcbi.1008291.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Performance of ThermoNet on the blind test set.

(A) Performance of ThermoNet on predicting ΔΔG for direct mutations; The Pearson correlation coefficient (r) between predicted values and experimentally determined values is 0.47, and the root-mean-square error (σ) of predicted values from experimentally determined values is 1.56 kcal/mol. The dots are colored in gradient from blue to red such that blue represents the most accurate prediction and red indicates the least accurate prediction. (B) Cumulative distribution of ThermoNet prediction error on direct mutations. (C) Performance of ThermoNet on predicting ΔΔG for the reverse mutations (r = 0.47, σ = 1.55 kcal/mol). (D) Cumulative distribution of ThermoNet prediction error on reverse mutations. (E) Direct versus reverse ΔΔG values of all the mutations in the blind test set predicted by ThermoNet. A perfectly unbiased predictor would give r = −1 and 〈δ〉 = 0 kcal/mol. ThermoNet successfully reduces prediction bias with r = −0.96 and 〈δ〉 = −0.01 kcal/mol. (F) Distribution of ThermoNet prediction bias.

https://doi.org/10.1371/journal.pcbi.1008291.g003

We also report the performance of ThermoNet*, a different version of ThermoNet trained using the Q6428 data set augmented from the intermediate data set Q3214 before the step of homology reduction in our data set curation procedure (Fig 2B and Methods). ThermoNet* was trained in the exact same way as ThermoNet except that the homology of its training set Q3214 to S^sym was retained. Thus, the parameterization of ThermoNet* is comparable to previous methods that did not consider homology. As expected, evaluation of ThermoNet* on S^sym shows even better performance in σ_rev (1.38 vs. 1.55 kcal/mol) and r_rev (0.59 vs. 0.48) than ThermoNet, which was trained using the data set obtained after homology reduction (Table 2, Fig 2B, and Methods). Thus, the performance of many previously developed methods is likely to be substantially lower if they had been trained using a data set that shared no homology with S^sym. In contrast, ThermoNet’s strong performance, even after removing homology reduction, suggests robust generalization in real-life applications.

Compared to other ΔΔG predictors, ThermoNet* achieves the best performance on reverse mutations, and the methods that outperform it on direct mutations all have substantial bias against reverse mutations (σ_rev > 2.09 kcal/mol and 〈δ〉 < -0.58). The seemingly good performance of many machine learning-based methods on direct mutations, but poor performance on reverse mutations suggests potential overfitting due to unbalanced training sets [24–26,39]. ThermoNet also performs well, but as a result of the reduction in performance due to removing homology between training and test sets, the DDGun and DDGun3D methods outperform it on direct and reverse mutations. Unfortunately, it is not possible to retrain and evaluate all the other methods on the homology pruned training set, so we cannot directly compare the other methods to ThermoNet. Nonetheless, the fact that it still outperforms most suggests its utility and robustness.

Structural models of reverse mutations are necessary for unbiased ΔΔG predictions

To evaluate whether the inclusion of the reverse mutations is necessary for the reduction in prediction bias, we trained a predictor following the same procedure for training ThermoNet but using a data set consisting of only the 1,744 direct mutations and their associated experimental ΔΔGs (i.e. the Q1744 data set in Fig 2B). We applied this predictor to predict the ΔΔGs of the direct and reverse mutations of the S^sym test set. As shown in S3 Fig, ΔΔGs of direct mutations predicted by these models correlate reasonably well (r = 0.47 and σ = 1.38 kcal/mol) with the experimental values and are comparable to the performance of the ensemble of networks trained using the balanced data set Q3488. In contrast, these models perform poorly (r = −0.06 and σ = 2.40 kcal/mol) in predicting the ΔΔGs of the corresponding set of reverse mutations (S3 Fig). This suggests that the models were biased toward the training set which is dominated by destabilizing mutations. This is confirmed by the strongly positive correlation between the predictions for direct mutations and those for reverse mutations and the large prediction bias (r_dir−rev = 0.35 and 〈δ〉 = 1.63 kcal/mol) (S3 Fig). Compared to the performance of ThermoNet, which was trained using the balanced data set Q3488, the results highlight the necessity of a balanced data set for correcting prediction bias.

Case studies: The p53 tumor suppressor protein and myoglobin

We further tested ThermoNet by predicting the ΔΔGs of single-point mutations in the p53 tumor suppressor protein and myoglobin whose thermodynamic effects have previously been experimentally measured. The TP53 tumor suppressor gene encodes the p53 transcription factor that is mutated in ~45% of all human cancers. Unlike most tumor suppressor proteins that are inactivated by deletion or truncation mutations, single amino acid substitutions in p53 often modify DNA binding or disrupt the conformation and stability of p53 [40]. Myoglobin is a cytoplasmic globular protein that regulates cellular oxygen concentration in cardiac myocytes and oxidative skeletal muscle fibers by reversible binding of oxygen through its heme prosthetic group [41]. The p53 data set consists of 42 mutations within the DNA binding domain of p53 [16]. The myoglobin data set consists of 134 mutations scattered throughout the protein chain [42]. We note that none of the mutations in these two data sets were present in the training set and that proteins that are likely to be homologous to p53 and myoglobin were also removed from the training set of ThermoNet (Fig 2B, Methods). We used published crystal structures of p53 (PDB ID: 2OCJ) and myoglobin (PDB ID: 1BZ6) to create one structural model for each of the mutations in these two data sets respectively using the FastRelax protocol in Rosetta [43]. These predictions were compared directly with the experimentally determined ΔΔGs (Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. ThermoNet predicted well the ΔΔGs of mutations in the p53 tumor suppressor protein and myoglobin.

(A) Performance of ThermoNet on predicting ΔΔG for the direct mutations in p53 (r = 0.45, σ = 2.01 kcal/mol). (B) Performance of ThermoNet on predicting ΔΔG for the reverse mutations in p53 (r = 0.56, σ = 1.92 kcal/mol). (C) Direct versus reverse ΔΔG values of all p53 mutations predicted by ThermoNet (r_dir−rev = −0.93 and 〈δ〉 = −0.04 kcal/mol). (D) Performance of ThermoNet on predicting ΔΔG for the direct mutations in myoglobin (r = 0.38, σ = 1.16 kcal/mol). (E) Performance of ThermoNet on predicting ΔΔG for the reverse mutations in myoglobin (r = 0.37, σ = 1.18 kcal/mol). (F) Direct versus reverse ΔΔG values of all myoglobin mutations predicted by ThermoNet, with a Pearson correlation of r_dir−rev = −0.97 and 〈δ〉 = −0.02 kcal/mol. The dots are colored in gradient from blue to red such that blue represents the most accurate prediction and red indicates the least accurate prediction.

https://doi.org/10.1371/journal.pcbi.1008291.g004

For p53, ΔΔGs of both direct and reverse mutations predicted by ThermoNet correlate well with the experimental measurements (r = 0.45 and 0.56) and have little bias (r_dir−rev = −0.93 and 〈δ〉 = −0.04 kcal/mol). However, the predicted ΔΔGs of myoglobin mutations correlate less well (r = 0.38 and 0.37 for direct and reverse mutations respectively) compared to those of p53, though the bias is also low (r_dir−rev = −0.97 and 〈δ〉 = −0.02 kcal/mol). The poorer correlations for myoglobin are likely because the myoglobin data set consists of ΔΔG measurements obtained under various experimental conditions which ThermoNet does not explicitly account for. In fact, after excluding four data points (L29N, A130L, and two data points corresponding to A130K) with the biggest prediction error (> 3 kcal / mol), the Pearson correlations increase to 0.52 and 0.51 for direct and reverse mutations respectively. While the correlation between predicted and experimentally measured ΔΔGs is not perfect, the predictions are generally conservative–no mutations with low measured ΔΔG are predicted to have a high ΔΔG. These results demonstrate the utility of ThermoNet as rapid unbiased predictor of ΔΔG for mutations in clinically relevant proteins.

We also compared ThermoNet with four other biophysics-based methods: FoldX, Rosetta, SDM, and CUPSAT (see S6 Table for a summary and references of these methods). We were not able to include all methods because both p53 and myoglobin are already in the S2648 set that was used for training most of the other machine learning-based ΔΔG predictors and they are also in the VariBench [36] data set used to derive parameters of the DDGun model [37]. We recognize that there is also a possibility that mutation ΔΔG values from p53 and myoglobin were used in the derivation of parameters of the biophysics-based methods tested here. Our comparison indicates that both FoldX and Rosetta predictions have a better correlation than ThermoNet while also reasonably anti-symmetric (S7 and S8 Tables). However, as shown in Table 2 and demonstrated in previous studies [24,25], both FoldX and Rosetta are likely to show bias toward predicting destabilization when tested on larger data sets.

ΔΔG landscape of ClinVar missense variants

Previous work has shown that variant deleteriousness can only be partially attributed to ΔΔG [44] and that both stabilization and destabilization can cause disease [45]. We sought to use ThermoNet to obtain a less biased picture of the impact of benign and pathogenic variants on protein stability. We applied ThermoNet to predict the ΔΔG distributions of pathogenic and benign missense variants in ClinVar, a widely used resource of medically important variants [46]. For comparison, we also applied FoldX, a popular and freely available ΔΔG predictor [13,22] to the ClinVar set. We first examined the overall predicted ΔΔG distribution of ClinVar variants. The ΔΔGs of ClinVar variants predicted by ThermoNet range from -2.75 kcal/mol to +3.75 kca/mol (Fig 5A). Experimentally measured ΔΔGs generally fall within -5 kcal/mol to +5 kcal/mol [13,47]; thus, ThermoNet’s predictions are consistent with the range of observed values. In contrast, the ΔΔGs of the same set of variants predicted by FoldX range from -6.64 kcal/mol to +57.2 kcal/mol (Fig 5A), and 15.2% of ΔΔGs predicted by FoldX are outside the expected range of -5 kcal/mol to +5 kcal/mol.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Predicted ΔΔG distributions of ClinVar missense variants.

(A) The overall ΔΔG distributions of ClinVar variants predicted by ThermoNet and FoldX. ThermoNet’s predictions are consistent with the expected range based on experimentally determined ΔΔG values (-5 kcal/mol to +5 kcal/mol). In contrast, more than 15% of ΔΔGs predicted by FoldX are outside the expected range. (B) The ΔΔG distributions for ClinVar benign variants predicted by ThermoNet and FoldX. (C) The ΔΔG distributions of ClinVar pathogenic variants predicted by ThermoNet and FoldX. The ΔΔGs of 80.2% of benign variants predicted by ThermoNet fall within the neutral zone (-0.5 to +0.5 kcal/mol, region between dashed lines), in which variants are not expected to influence fitness. FoldX only predicted 39.7% of benign variants to be in the neutral zone. Further, the ΔΔGs of pathogenic variants predicted by ThermoNet suggest pathogenic variants are nearly equally likely to be stabilizing (47.3%) as destabilizing (52.7%). In contrast, FoldX predicted that 83.2% of pathogenic variants are destabilizing. Variants for which FoldX ΔΔG is > 20 kcal/mol are omitted for clarity. Percentages represent the fractions of variants whose ΔΔGs are predicted to be in the neutral zone.

https://doi.org/10.1371/journal.pcbi.1008291.g005

ΔΔGs predicted by ThermoNet are also consistent with the expected ΔΔG distributions according to a biophysical model of protein evolution [47]. This model hypothesizes that fitness is a non-monotonic, concave function of protein stability, meaning that fitness decreases with increasing deviation from an optimal stability. The model also suggests that there is a “neutral zone” of 1.0 kcal/mol around the optimal stability in stability space, and mutations whose impact on stability fall within the neutral zone will have little effect on fitness [47]. We thus reasoned that the ΔΔGs of benign variants should fall within a narrow range from -0.5 kcal/mol to +0.5 kcal/mol and that pathogenic variants should be equally likely to be destabilizing or stabilizing [47]. To test this hypothesis, we examined the ΔΔG distributions of pathogenic and benign variants separately. The ΔΔGs of 80.2% of benign variants predicted by ThermoNet fall within the neutral zone, whereas FoldX only predicted 39.7% of benign variants to be in the neutral zone (Fig 5B). Further, the ΔΔGs of pathogenic variants predicted by ThermoNet suggest pathogenic variants are nearly equally likely to be destabilizing (52.7%) as they are to be stabilizing (47.3%) (Fig 5C). In contrast, FoldX predicted that 83.2% of pathogenic variants are destabilizing. As already demonstrated in previous studies, the bias is likely because FoldX was parameterized on an experimental ΔΔG data set dominated by destabilizing mutations [23–25].

ThermoNet’s predictions are also consistent with the fact that variant pathogenicity can only be partially attributed to impacts on protein stability. ThermoNet predicts that 61.4% of pathogenic variants that have ΔΔGs within the neutral zone and 19.8% of benign variants have ΔΔGs outside the neutral zone (Fig 5B and 5C). This is expected based on previous biochemical characterizations of pathogenic mutations. For example, Bromberg et al. collected 66 mutations with experimentally measured ΔΔG and functional annotations from the literature. The ΔΔGs of this set of mutations range from -4.3 to 4.96 kcal/mol and the authors found that 31% of mutations affecting function had ΔΔGs within the neutral zone while 19% functionally neutral mutations had ΔΔGs outside the neutral zone [44].

Protein thermodynamic stability

The thermodynamic stability ΔG of the folded form of a two-state protein, which is its Gibbs free energy of folding, is defined in relation to the concentration of folded [folded] and the concentration of unfolded [unfolded] forms: where T is the temperature, and R is the gas constant. More stable proteins, meaning that a higher fraction of the protein is in the folded form, have more negative values of ΔG. The impact of mutations on protein stability, ΔΔG, is defined in terms of the change in ΔG between the wild-type and mutant proteins: such that a destabilizing mutation has a positive ΔΔG, whereas a stabilizing mutation has a negative ΔΔG. The values of ΔΔG resulting from single-point mutations usually range from -5 to 5 kcal/mol [13]. A mutation that destabilizes a protein with a typical stability (ΔG = -5 kcal/mol) by 1 kcal/mol will reduce the equilibrium constant for the folding reaction of this protein by a factor of 5.1 at physiological temperature (310 K).

Thermodynamics of direct and reverse mutations

Consider a pair of proteins whose sequences differ only at a single position where the amino acid is X in one protein and Y in the other. Let the Gibbs free energies of folding of this pair of proteins be ΔG_X and ΔG_Y respectively. For such a pair of proteins, one can think of the protein Y as being generated by a “direct” mutation at the sequence location from amino acid X to Y and the change in the Gibbs free energy of folding caused by X to Y mutation is ΔΔG_X→Y = ΔG_Y − ΔG_X. One may also think of the protein X as being generated by a “reverse” mutation from amino acid Y to X and the change in the Gibbs free energy of folding caused by this reverse mutation is ΔΔG_Y→X = ΔG_X − ΔG_Y = ΔΔG_X→Y. A well-performing, “self-consistent” method for predicting ΔΔGs would not only give accurate ΔΔG predictions for the direct mutations, but also for the reverse mutations. This self-consistency requirement has been largely ignored by previously developed ΔΔG predictors [23–25].

Data sets and symmetry-based data augmentation

The data set used to train ThermoNet was derived from the Q3421 data set compiled in a previous study [15]. The Q3421 data set contains 3,421 distinct single-point mutations in 150 proteins collected from the ProTherm database [53]. The effects of these mutations on the stability of protein structure have been measured experimentally and expressed quantitatively as ΔΔG values. We first excluded those mutations from the Q3421 data set that were also in the S^sym test set (see following). To reduce the sequence similarity between proteins in the training set of ThermoNet and the proteins it was tested on, we also removed all proteins that are likely homologous (BLAST e-value < 0.001) to p53, myoglobin, and proteins in S^sym from Q3421. Estimation of homology was accomplished by running the blastp program [54] using protein sequences in the S^sym data set and the sequences of p53 and myoglobin as queries against protein sequences in Q3421. Our rigorous pruning of the Q3421 data set resulted in a final data set consisting of 1,744 distinct mutations in 127 proteins. This data set was augmented by creating a reverse mutation data point for each of the 1,744 direct mutations, thus giving to a total of 3,488 data points for training ThermoNet. The data set was randomly divided into ten equally sized, mutually exclusive subsets each consisting of 10% of the direct mutations and the corresponding 10% of reverse mutations. In the training of each component model of ThermoNet, nine subsets were combined to form a training set and the remaining one subset was used as a validation set. The data set used to test ThermoNet and to compare it with fifteen previously developed ΔΔG predictors was a common, balanced data set called S^sym consisting of 342 pairs of proteins with known crystal structures [24]. The members forming each pair differ at only a single position in the protein sequence. The ΔΔG values of the 342 direct mutations have been experimentally measured and the ΔΔG values of the corresponding 342 reverse mutations were assigned using anti-symmetry.

Modeling mutant structures

We treat each mutation as a pair of proteins whose sequences differ only at a single sequence position. For each pair of proteins, we designate the one whose structure has been experimentally resolved as protein X and the other as protein Y. Structures of the X proteins were collected from the Protein Data Bank (PDB) [55] and were relaxed in the Rosetta all-atom energy function ref2015 [56] using the Rosetta FastRelax protocol [43]. To prevent large-scale conformational shift from the input PDB structure, atoms were constrained to their starting locations with a harmonic penalty potential during relaxation. The same Rosetta FastRelax protocol was also employed to create structural models for each of the Y proteins from the corresponding relaxed structure of the X protein by supplying a Rosetta resfile specifying the mutation X → Y to make. The structures of both proteins of each protein pair in the S^sym test set were collected from the PDB [55] and were also relaxed using the same Rosetta FastRelax protocol.

Voxelization of the neighborhood of mutation site

We treated each protein structure as a collection of volume elements (voxels) in 3D space. Just as pixels element in an image have color channels, we parameterized voxels in a protein structure by a set of k biophysical property channels: [v₁, v₂, …, v_k] where the value v_i of each property channel indicates the level of saturation of property i at this voxel (Fig 1A and 1B). For each mutation (a pair of proteins), we superimposed the mutant structure onto the wild-type structure such that the root-mean-squared distance between them is minimized and collected a grid of 16 × 16 × 16 voxels from both structures. We parameterized each voxel with seven property channels each of a distinct chemical nature according to AutoDock4 atom types [57] as in the work of Jimenez et al. [28] (Table 1). This resulted in a tensor of the shape [16, 16, 16, 7] for a single structure and a tensor of the shape [16, 16, 16, 14] when the two tensors from both structures are concatenated to represent the mutation. The grid was centered at the C_β atom of the mutation site amino acid (or C_α atom if it’s a glycine) where each voxel is a unit cube whose sides are 1 Å long. The level of saturation f(d) of each property channel at each voxel is determined by the van der Waals radius r_vdw of the atom designated to have that property and its distance d to the center of the voxel through the following formula:

The computation of tensors from protein structures was performed using routines implemented in the HTMD Python library (version 1.17) for molecular simulations [58]. A Python program for creating input tensors from a list of mutations is provided in GitHub repository at https://github.com/gersteinlab/ThermoNet. The orientations of protein structures were taken directly from the retrieved PDB files without further adjustment, although our Python program does provide an option for rotating input structures about all three Cartesian axes.

Model architecture

CNNs are a type of deep-learning model commonly used in computer vision applications. They have recently proven to perform well in residue contact prediction [59–62] and protein tertiary structure prediction [63–66]. We selected CNNs for protein structure-based ΔΔG prediction because we formulated this problem as a computer vision problem by treating protein structures as if they were 3D images. Each of the component models of ThermoNet features a sequential organization of three 3D convolutional layers (Conv3D), one 3D max pooling layer (MaxPool3D), followed by one fully connected layer (Dense) (S1 Fig). Convolutions operate over 4D tensors, called feature maps, with three spatial axes (length, width, and height) as well as a depth axis. The convolution operation extracts 3D patches of shape [3, 3, 3] with stride 1 from its input feature map and applies the same transformation to all patches, producing an output feature map. This output feature map is still a 4D tensor: it has a length, height, and a width whose values are determined by the shape of convolution patches and size of the stride, but its depth, which is also called number of filters, is a hyperparameter of the layer (see the section on hyperparameter search below). The number of filters in the three Conv3D layers were 16, 24, and 32 respectively. All convolution operation outputs from each Conv3D layer are transformed by the rectified linear activation function (ReLU). The transformed outputs from the last Conv3D layer are pooled by taking the maximum activation of each 2 × 2 × 2 grid. The max-pooled activations are then flattened into a 1D vector of features which are fully connected with a dense layer of 24 ReLU units. This model architecture was implemented using Keras [67] with TensorFlow [68] as the backend.

Training ThermoNet

ThermoNet is an ensemble of ten deep 3D-CNNs, each trained to perform the best on a validation set. Each of the component models was trained on nine subsets (collectively known as the training set), and its generalization performance was monitored on the remaining one subset (validation set). This process was iterated ten times each using a different one of the ten subsets as the validation set and the remaining nine subsets as the training set. We employed this procedure to obtain an ensemble of ten models because model ensembling has been suggested to produce better predictions [69]. Evaluation of this ensemble was performed on a separate test set (see below). All component models of ThermoNet were trained using the Adam optimizer [70] for 200 epochs with default hyperparameters (maximum learning rate = 0.001, β₁ = 0.9, β₂ = 0.999). Kernel weights of the model were initialized using the Glorot uniform initializer and updated after processing each batch of eight training examples. The mean squared error (MSE) of the predicted ΔΔG values from the experimental measurements was used as the loss function during training. When training a deep neural network, one often cannot predict how many epochs will be needed to get to an optimal validation loss. We monitored the MSE of the predictions on a separate validation set consisting of 10% variants randomly selected from the training set during training. Training was stopped when the MSE on the validation set stopped decreasing for ten consecutive epochs. To regularize the model, the dense layer was placed between two dropout layers with a dropout rate of 0.5 in each layer (S1 Fig). Each of the final component models was the one that produced the lowest MSE on the validation set. The final predicted ΔΔG value is the average of predictions from the ten models.

Hyperparameter search

The design of deep CNNs entails many architectural choices to account for number of hidden convolutional layers and fully connected layers, number of filters, filter size, strides, padding, dropout rate among many other hyperparameters. We initially created a voxel grid with size 16 × 16 × 16 at a resolution of 1 Å for each chemical property channel following the procedure described in [27] to cross-validate our network architectures. Considering the limited training data set available in our study, we tried some smaller architectures with cross-validation to decide the optimal one, rather than simply adapting the widely used, much larger, network architectures in computer vision applications. We restricted our deep CNNs to have three convolutional layers and one fully connected while considering several hidden layer sizes. Our results from five-fold cross-validation suggest that the architecture of the 16 × 24 × 32 convolutional configuration combined with a fully connected layer of size 24 achieved the best performance (S1 Fig). An additional consideration for our deep 3D-CNN architecture is the dimension of the local box. The size of the local box specifies the structural information accessible by the network and therefore is a hyperparameter of our method. We cross-validated the voxelization scheme with grid of sizes 8 × 8 × 8, 12 × 12 × 12, 16 × 16 × 16, and 20 × 20 × 20 at a resolution of 1 Å. All voxelization schemes draw a cubic box around the mutation site with lateral length of l Å where l equals 8, 12, 16, or 20 Å. Our results from five-fold cross-validation indicate that the voxel grid with size 16 × 16 × 16 gives the best prediction performance (S1 Fig).

Performance evaluation

The following measures were adopted to evaluate the performance of ThermoNet and to facilitate comparison with previously developed methods. The primary measures for evaluating prediction accuracy were the Pearson correlation coefficient (r) between experimental and predicted ΔΔGs and the root-mean-squared error (σ) of predictions. For a set of n data points (x_i, y_i), the formula for calculating r and σ are defined as follows: where the (x_i, y_i) tuple denotes the experimental and predicted ΔΔG values of mutation i, respectively, and n denotes the number of mutations in the data set. The measures for evaluating prediction bias were the Pearson correlation coefficient between the predictions for direct mutations and those for reverse mutations and the parameter δ which is defined as:δ = ΔΔG_rev + ΔΔG_dir and was previously used to quantify prediction bias [23]. An unbiased predictor should have δ = 0 for every mutation. The average of δ, 〈δ〉, taken over all mutations in the S^sym data set was used in two previously studies to quantify prediction bias [24,37]. While we report 〈δ〉 in this work, we note that 〈δ〉 is flawed because biases toward opposite directions will be washed out when summed. To give a more transparent presentation of prediction bias, we also plot the distribution of δ.

ClinVar variants

We retrieved the ClinVar database [47] in VCF format on August 15, 2019 and ran the VCF file through the Variant Effect Predictor (version 97) [71] to annotate the consequences of all ClinVar variants. We created a set of missense variants that can be mapped to protein structures to demonstrate the applicability of ThermoNet to clinically relevant variants. Our evaluation set consists of solely ClinVar missense variants that are labeled as "pathogenic" or "likely pathogenic" for the pathogenic group and "benign" or "likely benign" for the benign group. All variants were required to have a review status of at least one star and no conflicting interpretation. All ClinVar variants designated as “no assertion criteria provided”, “no assertion provided”, “no interpretation for the single variant”, or not covered by protein structure were excluded from the evaluation set. Due to the dependency of ThermoNet on 3D structures, we also require variants in the evaluation set to be mappable to available protein structures. The residue-level mapping of ClinVar variants onto protein structures was based on the SIFTS resource that provides residue-level mapping between UniProt and PDB entries [72]. Collectively, these restrictions resulted in 3,510 pathogenic variants and 950 benign variants that can be mapped to experimental structures deposited in the PDB [55]. The mapped variants along with PDB IDs can be found in our GitHub repository at https://github.com/gersteinlab/ThermoNet.