Mutual influence between language and perception in multi-agent communication games
Fig 2
Creating perceptual bias with relational label smoothing.
(A) Example of how the training targets (labels) are adapted to induce a color bias. To generate a CNN with a color bias, some of the target weight is spread across all other classes that have the same color as the target object. In our data set, there are 64 different object classes. The first sixteen classes comprise red objects (classes 1–16), followed by yellow objects (classes 17–32), turquoise objects (class 33–48), and purple objects (classes 49–64). For example, if the input image belongs to class 2 (“tiny red cylinder”), the usual target label, y0, is a one-hot vector where the entire weight lies on the true class index. The relational component, yr, spreads the target weight onto all other red objects. The target vector used for training is a weighted average of the original target and the relational component. Analogously, to introduce a scale/shape bias, some of the target weight is spread onto all other objects of the same scale/shape as the input object. (B) Representational similarity matrix for the color CNN after training (σ = 0.6). Entries at position (i,j) correspond to the average cosine similarity between the CNN activations for images of class i and the CNN activations for images of class j (based on the penultimate fully-connected layer). The white 16 × 16 blocks on the diagonal indicate that objects of the same color are perceived as very similar to each other.