Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift
Fig 9
Reliability diagram and expected calibration error (ECE, displayed as Equation 9 100) for each soft labeling method on image classification using the CINIC10 dataset.
Black bars indicate the accuracy in the given bin and red bars indicate the gap between accuracy and confidence. Perfect calibration, where confidence and accuracy are equal, would follow the dotted line (i.e. a black bar over the line indicates under-confidence and a black bar under the line indicates over-confidence). We use the average of the logits produced by models trained with 10 different random seeds with no temperature scaling. Aggregating soft labels results in the best overall calibration.