Table 1.
Summary of the methods analyzed in the present work and their respective fields of successful application.
Fig 1.
Binding site modeling approaches for different comparison algorithms.
The binding site of coagulation factor Xa (PDB ID 1f0r, chain A) bound to the nanomolar inhibitor RPR208815 is shown together with a schematic representation of the ways in which binding site features are modeled. The methodologies are connected with the corresponding underlying data structures used for the comparison. Binding site visualizations were generated using UCSF Chimera[60].
Table 2.
Summary of benchmark data sets used to analyze the individual strengths and weaknesses of different binding site comparison tools.
Similar and dissimilar binding site pairs are referred to as active and inactive pairs, respectively. The average overall G-factors were calculated with PROCHECK[66] and PROCHECK-NMR[67] to enable the quality comparison between all data sets including data set 2. Mean, standard deviation (stddev), minimum and maximum are given for all quality critieria.
Fig 2.
Evaluation of different binding site comparison tools with respect to the data set of structures with identical sequences.
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to the AUC. Thin lines represent the resulting ROC curve for the scoring scheme that yielded the highest AUC. (A) A slightly higher AUC for SiteAlign was obtained if distance d2 was applied for binding site pair ranking. (B) For the surface-based methods, the Tanimoto (color) for Shaper or VolSite/Shaper and the ColorTanimoto for SiteHopper led to the highest AUC. (C) The use of the Tanimoto coefficient as similarity measure led to the highest AUC for TIFP(PDB). D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Fig 3.
Changes in the molecular interaction patterns of dihydrofolate reductase ligands and changes in the binding site upon ligand binding (PDB ID 1ohk, chain A (A and D); PDB ID 4kd7, chain A (B and E); PDB ID 1drf, chain A (C and F)). A-C) Representation of the binding site structures. Figures were generated using UCSF Chimera[60]. D-F) Schematic view of the crucial interactions between protein and ligand. Figures were generated using LigPlot+[69].
Fig 4.
Evaluation of different binding site comparison tools with respect to the data set of NMR structures.
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to the AUC. (A) The highest AUC was obtained for SiteAlign when using distance d1. (B) All Shaper comparisons led to higher AUCs for the scoring measure Tanimoto (color). SiteEngine results slightly improved the AUC for the distance scoring scheme. D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Fig 5.
Different interaction patterns for structures from the solution NMR ensemble of ileal lipid-binding protein (PDB ID 1eio, chain A).
The ensemble contains five conformers in total. Models 1, 3, and 4 (from left to right) were used to generate this illustration. Residues with alternating interaction patterns in the different conformations are highlighted and occupy nearly half of the pocket. The remaining part of the pocket is mainly engaged in hydrophobic contacts and three hydrogen bond interactions with the small molecule glycocholate. The figure was generated using LigPlot+[69].
Fig 6.
Evaluation of different binding site comparison tools with respect to data set 3 (five substitutions by physicochemically different residues).
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to their AUC. (A) PocketMatch showed the best AUC for the score PMScoremin (thin orange line). (B) The scores SVA, RefTversky (color), RefTversky (color), RefTversky (color), RefTversky (color), and ColorTanimoto led to the highest AUC values for ProBiS, Shaper, Shaper(PDB), VolSite/Shaper, VolSite/Shaper(PDB), and SiteHopper, respectively (thin lines). (C) The highest AUC was obtained for IsoMIF and TIFP(PDB) when using taniM and the Tanimoto coefficient as similarity measure (thin lines). D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Table 3.
The Spearman’s Rho correlation coefficients for a comparison between the original crystal structures and their corresponding decoy structures with one, two, three, four, and five substitutions of randomly chosen residues by physicochemically different ones for residue-based, surface-based, and interaction-based binding site comparison methods.
A color gradient was applied to highlight a prominent negative correlation (green), a correlation coefficient of -0.5 (yellow), and no correlation at all or a positive correlation (red). No cavities could be extracted for Cavbase and RAPMAD comparisons for decoy structures generated from the structure with the PDB ID 4ca7 (n.d.).
Fig 7.
Evaluation of different binding site comparison tools with respect to the data set of rational decoy structures (five mutations).
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to their AUC. (A) PocketMatch showed the best AUC for the score PMScoremin (thin orange line). (B) The scores SVA, Tanimoto (color), Tanimoto (color), RefTversky (color), RefTversky (color), and ColorTanimoto led to the highest AUC values for ProBiS, Shaper, Shaper(PDB), VolSite/Shaper, VolSite/Shaper(PDB), and SiteHopper, respectively (thin lines). (C) The highest AUC was obtained for TIFP(PDB) when using the Tanimoto coefficient as scoring measure (thin dark green line). D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Table 4.
The Spearman’s Rho correlation coefficients for a comparison between the original crystal structures and their corresponding decoy structures with one, two, three, four, and five rational mutations of randomly chosen residues for residue-based, surface-based, and interaction-based binding site comparison methods.
A color gradient was applied to highlight a prominent negative correlation (green), a correlation coefficient of -0.5 (yellow), and no correlation at all or a positive correlation (red). No cavities could be extracted for Cavbase and RAPMAD comparisons for decoy structures generated from the structure with the PDB ID 4ca7 (n.d.).
Fig 8.
Evaluation of different binding site comparison tools with respect to the data set of Kahraman structures [63] after the exclusion of phosphate binding sites.
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to the AUC. (A) The best AUC for SiteAlign resulted from the d1 distance (thin red line). (B) For ProBiS, VolSite/Shaper, SiteEngine, and SiteHopper the scores SVA, Tanimoto (color), TotalScore, and ShapeTanimoto yielded the best AUC values (thin lines). (C) For TIFP(PDB), the use of the Hamming distance led to the best results with respect to AUC (thin line). D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Fig 9.
Similarity score matrices for the Kahraman data set generated from the SiteHopper (left) and IsoMIF (right) results.
Both methods are able to find clusters of binding sites with identical ligands. The combination of both methods might even give rise to an improved differentiation. Similarity scores (tani) above 0.4 are colored green for the matrix obtained with IsoMIF. Similarity scores (PatchScore) above 0.65 are colored green for all SiteHopper-derived site alignments.
Fig 10.
Evaluation of different binding site comparison tools with respect to the data set of Barelier et al.[64].
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to the AUC. (A) The thin red line represents the resulting ROC curve for SiteAlign when using the distance d1. (B) Thin lines represent the ROC curves for ProBiS, Shaper, Shaper(PDB), VolSite/Shaper, VolSite/Shaper(PDB), SiteEngine and SiteHopper when using the scoring schemes SVA, FitTversky (color), FitTversky (color), RefTversky (color), Tanimoto (fit), distance, and ShapeTanimoto, respectively. (C) The thin line represents the resulting ROC curve for IsoMIF and the taniMW score. D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Fig 11.
Alignments of high-scoring binding site pairs of the Barelier data set generated by (A) Cavbase, (B) TM-align, (C) SMAP, and (D) Shaper. (A) Superposition of angiotensin-converting enzyme (PDB ID 2x8z, green) and leukotriene A4 hydrolase (PDB ID 4dpr, purple) in complex with captopril (ball-and-sticks representation). Red spheres denote hydrogen bond acceptor features while purple spheres represent mixed hydrogen bond acceptor/donor features. Metal coordination sites are marked by orange spheres and blue and yellow spheres denote residues with aromatic and aliphatic characteristics, respectively. The Cavbase similarity score for this match is 11.37. (B) Alignment of leukotriene A4 hydrolase (PDB ID 3fty, green) and mitogen-activated protein kinase 14 (PDB ID 1w7h, purple) crystallized with the small molecule fragment 3-(benzyloxy)-pyridine-2-amine (3IP, ball-and-sticks-representation). Residues within a 4 Å radius of any ligand atom are depicted in stick representation. This alignment yields a TM-score of 0.32. (C) Superposition of adipocyte lipid-binding protein (PDB ID 2ans, green) and pheromone-binding protein (PDB ID 1ow4, purple) in complex with the fluorescent probe 8-anilino-1-naphthalene sulfonate (2AN, ball-and-sticks). The residues shown in stick representation represent only a fraction of all matched residues. The SMAP RawScore for this site pair is 63.44. (D) Shaper-based alignment of the neocarzinostatin (PDB ID 2g0l, green) and tankyrase-2 (PDB ID 4hki, purple) flavone (FLN, ball-and-sticks) binding sites (TanimotoCombo = 0.92). Residues within a 4 Å radius of the ligand are represented as sticks. Hydrogen bond interactions are depicted as green springs.
Fig 12.
Results of a binding site feature analysis for the class A, B, and C pairs of Barelier et al.[64] and a sequence-culled subset of druggable binding sites. The relative frequencies of the binned properties are presented in light gray for class A structures, in dark gray for structures belonging to class B and C pairs, and in black for the sequence-culled sc-PDB[65] subset. The binding site features were calculated using DoGSite[73].
Fig 13.
Evaluation of different binding site comparison tools with respect to the data set of successful applications.
A-C) The ROC curves for residue- (A), surface- (B), and interaction-based (C) comparison methods. The name of the tool is colored according to its corresponding ROC curve. The binding site comparison tools are sorted in descending order with respect to the AUC. (A) SiteAlign yielded a slightly better AUC if the distance d1 was used (thin line). (B) The best AUC values for ProBis, Shaper, Shaper(PDB), VolSite/Shaper, VolSite/Shaper(PDB), SiteEngine, and SiteHopper resulted from the scoring measures Zscore, Tanimoto (color), Tanimoto (color), Tanimoto (color), Tanimoto (color), TotalScore, and ShapeTanimoto, respectively (thin lines). D-F) EFs for residue- (D), surface- (E), and interaction-based (F) comparison methods. A linear color gradient ranging from white for the highest value to gray to black for the lowest value was applied for the EFs at different percentages of screened data set.
Table 5.
Sensitivity (se) and specificity (sp) values for data set 7 using different score cut-offs.
The score thresholds to discriminate similar and dissimilar site pairs were determined using the ROC-based Youden’s J statistic[85] based on data set 1 and data set 7. Both thresholds were applied for sensitivity and specificity calculations. The rank column gives the rank of the respective method within all methods for the corresponding sensitivity. Sensitivity and specificity values for thresholds that were assigned by the methods’ developers are given in brackets. For SiteAlign, Shaper, VolSite/Shaper, and TIFP, the corresponding scoring measure is different from the one used in our study.
Fig 14.
Binding site alignments for similar cavity pairs which most tools failed to identify.
(A) Alignment of human carbonic anhydrase II (PDB ID 1bn4, green) and cyclooxygenase-2 (PDB ID 6cox, purple) as obtained with IsoMIF. (B) The binding sites of synapsin (PDB ID 1aux, green) and PIM-1 kinase (PDB ID 3a99, purple) as aligned by SiteEngine. All illustrations were generated using UCSF Chimera[60].
Table 6.
Run time evaluation of different binding site comparison methods with respect to the data set of Kahraman et al.[63].
The numbers in brackets highlight the number of successfully prepared pockets or the number of comparisons, respectively. The average run times are colored by a gradient ranging from green (minimum run time) to yellow to red (maximum run time).
Fig 15.
Guiding the choice of appropriate binding site comparison tools.
(A) Venn diagram illustrating differences in the strengths of the comparison methods based on a subset of quality criteria. An asterisk marks methods which provide a binding site alignment for a visualisation of site similarities. (B) Venn diagram of successful applications of the evaluated residue-, surface-, and interaction-based tools in different research scenarios. Both diagrams were generated using DrawVenn[100].
Table 7.
Overview of substitutions to generate a data set of rationally modified decoy structures.
Table 8.
Redundancy of the benchmark entries.
The PDB IDs of redundant entries are given with the corresponding data sets and the number of structures in both sets.
Table 9.
Summary of the scoring schemes provided by different comparison tools.
An asterisk marks the scheme that was used for the evaluation of tools that offer more than one similarity or distance measure.