Systematic benchmarking of deep-learning methods for tertiary RNA structure prediction
Fig 14
Box and violin plots showing the comparison of the methods based on different datasets.
The median values are labelled with blue text and the whiskers denote the interquartile range a) The datasets are on the X-axis and the RMSD is on the Y-axis. We pooled all the models from different methods datasets together and only compared the RMSD of the models based on their datasets. The median RMSD for the CASP dataset was the highest (26.12 Å), New dataset was in the middle (14.66 Å) and RNA-puzzles had the lowest median RMSD (6.75 Å). b) This plot shows the same comparison, but we look at each method separately. The Average plot (in grey) is the average of the models predicted by all the methods for a particular target. Generally, CASP15 dataset has the highest median RMSD for all methods, New dataset was in the middle and RNA-puzzles dataset has the lowest median RMSD for all the methods. The reason for RNA-puzzles being the easiest is because 35/36 targets are X-ray crystallographic structures and many of the targets were published before 2020, thus they might be included in the training sets of the ML-based methods. CASP dataset is the hardest because most of the targets are synthetic and Cryo-EM structures. The new dataset provides the most realistic performance estimates as it is a well-balanced dataset (comprising all kind of RNAs with representation from both X-ray crystallographic and Cryo-EM structures) and none of its targets are present in the training sets of the ML methods. DRFold has a median RMSD of 2.73 on RNA-puzzles dataset possibly because it has already seen most of the targets in the RNA-puzzles dataset while training thus giving an overinflated performance.