Fig 1.
Screenshot of the image classification task.
The object to be classified is highlighted by a rectangular frame. The number on the right (“2”) denotes the time remaining to answer the question in seconds. The bottom bar indicates the progress toward completing the classification of all images. The correct answer of this image is “no threat” (art installation).
Fig 2.
Performance of image classification as a function of the entropy threshold.
(a) Accuracy and (b) number of image processed. Square: case 1, where image entropy is computed from two labels (“threat” and “no threat”), filled circle: case 2, where image entropy is computed from three labels (“threat”, “no threat”, and “I don't know”), triangle: case 3, where image entropy is computed from two labels (“threat” and “no threat”) after reassigning “I don’t know” to either class proportional to “threat” and “no threat” by all participants. Points and vertical lines are means and standard errors of 1,000 runs. Dotted lines correspond to the case, where no entropy threshold was applied (that is, the image is retired from the repository when it receives three labels of “threat” and “no threat” combined).
Fig 3.
Performance of image classification over different numbers of volunteers.
(a) Accuracy at entropy threshold 0.2, (b) at 0.5, and (c) at 0.8. (d) The number of processed images at entropy threshold 0.2, (e) at 0.5, and (f) at 0.8. Colors correspond to Fig 2 (square: case 1, filled circle: case 2, triangle: case 3, open circle: no entropy threshold).
Fig 4.
(a) True positive rates and (b) true negative rates over entropy threshold. Colors correspond to Fig 2 (square: case 1, filled circle: case 2, triangle: case 3).
Fig 5.
Gray bars are observed proportions when participants labeled positive (threat) and negative (no threat). Black bars are the proportion of true answers when participants opted for “I don’t know”.