Perceptual datasets.

To understand the human visual system, stimuli from perceptual datasets can be used in an attempt to relate the evoked perception to the visual input. We can roughly categorize three types of stimuli used for visual perception: natural, synthetic and manipulated.

The first represent ‘normal’ photos of objects, materials and scenes as they can be found in reality. Experimental design with such stimuli often attempts to relate the evoked perceptions to natural image statistics within the images or physical characteristics of the contents captured in the images. Some examples of uses of natural stimuli datasets include, but are not limited to, the memorability of pictures in general [18] or more specifically the memorability of faces [19]. In another example, images of natural, but novel objects were used to understand what underlies the visual classification of objects [20].

The second type, synthetic stimuli, are created artificially, such as digital renderings, drawings and paintings. Synthetic stimuli might represent the real world, but often contain image statistics that deviate from natural image statistics. Paintings have for example often been used to study affect and aesthetics [21–23]. In another example [24], used a set of synthetic stimuli to test for memorability of data visualizations.

Both natural and synthetic images can be manipulated, which leads to the third type of stimuli. Manipulated stimuli are often used to investigate the effect of image manipulations by comparing them to the original (natural or synthetic) image. Here the manipulations function as the independent variables. For example [25] created a database of images that contain scene inconsistencies that can be used to study the compositional rules of our visual environment. In another example, a stimulus set consisting of original and texture (i.e., manipulated) versions of animals found that perceived animal size is mediated by mid-level image statistics [26].

The advantage of using manipulated or synthetic images is that perceptual judgments can be compared to some independent variable, which is typically not available for natural images. Paintings are a special case here. They are a synthetic image of a 3D scene that is rendered using oils, pigments and artistic instruments. However, the painting is also a mostly flat, physical object. Retrieving the veridical data is usually impossible for paintings. In other words, objects or materials depicted in photos can often be measured or interacted with in the real world but this is rarely possible for paintings. However, the advantage of using paintings is that it can often be seen, or (historically) inferred, how the painter created the illusory realism. Even if it cannot be seen with the naked eye, chemical and physical analysis can be performed. In [27] a perceptually convincing depiction of grapes was recreated using a 17th century recipe. In this reconstruction, the depiction was recreated by a professional painter one layer at a time, where each layer represents a separate and perceptually diagnostic image feature that together lead to the perception of grapes. The physical limitations of painterly depictions relative to the physical 3D world, such as for example due to luminance compression in paintings [28–31] may lead to systematically different strategies for material depiction. Despite this [15], has shown that the perceptions of materials and material properties depicted in paintings are similar to those previously reported for photographic materials [14].

Therefore, studying paintings in addition to more traditional stimuli like photos or renderings, can enrich our understanding of human material perception. It should be noted that in this paper we focus on the image structure of the painting instead of the physical object. In other words, we focus on what is depicted within paintings and our data and analysis is limited to pictorial perception. In the remainder of this paper, when we mention paintings, we mean images of paintings.

Throughout history, painters have studied how to trigger the perceptual system and create convincing depictions of complex properties of the world. This resulted in perceptual shortcuts, i.e., stylized depictions of complex properties of the world that trigger a robust perception. The steps and painterly techniques applied by a painter to create a perceptual shortcut can be thought of as a perception-based recipe. Following such a recipe results in a perceptual shortcut, which is a depiction that gives the visual system the required inputs to trigger a perception. Many of the successful depictions are now available in museum collections. As such, the creation of art throughout history can be seen as one massive perceptual experiment. Studying perceptual shortcuts in art, and understanding the cues, i.e., features required to trigger perceptions, can give insights into the visual system. We will demonstrate this idea by analyzing highlights in paintings and photos.

Ecological datasets.

To understand how the human visual system works it is important to understand what type of visual input is given by the environment. Visual ecology encompasses all the visual input and can be subdivided into natural and cultural ecology. Natural ecology reflects all which is found in the physical world. For example, to understand color-vision and cone cell sensitivities it is relevant to know the typical spectra of the environment. For this purpose, hyperspectral images [32, 33] can be used, in this case to investigate color metamers (perceptually identical colors that originate from different spectra) and illumination variation. In another example, a dataset of calibrated color images were used to understand color constancy [34] (the ability to discount for chromatic changes in illumination when inferring object color). The SYNS database was used to relate image statistics to physical statistics [35]. Another dataset contains photos taken in Botswana [36] in an area that supposedly reflects the environment of the proto-human and was used to investigate the evolution of the human visual system. Spatial statistics of today’s human visual ecology are clearly different from Botswana’s bushes as most people live in urban areas that are shaped by humans. For example, a dataset from [37] was used to compute the distribution of spatial orientations of natural scenes [38].

The content depicted within paintings only loosely reflects the natural visual ecology, but they do strongly represent cultural visual ecology. They have influenced how people see and depict the world and have influenced visual conventions up to contemporary cinematography and photography. Both perceptual scientists and art historians have looked for and studied compositional rules and conventions within art. A good example is the painterly convention that light tends to originate from the top-left [39, 40], which is likely related to the human light-from-above prior [41–44].

New developments in cultural heritage institutions have made the measurement and study of paintings much more accessible. In recent years the digitization of cultural heritage has led to a surge in publicly available digitized art works. Many individual galleries and art institutions have undertaken the admirable task to digitize their entire collection, and have often make a portion, if not the whole collection digitally available with no or minor copyright restrictions. The availability of digitized art works, combined with advancements in image analysis algorithms, has lead to Digital Art History, which concerns itself with the digitized analysis of artworks by for example analyzing artistic style [45] and beauty [46], or local pattern similarities between artworks [47]. In [48], the authors for example developed a system that automatically detects and extracts garment color in portraits, which can for example be used for the digital analysis of historical trends within clothes and fashion.

Crowley and Zisserman [49] pointed out that art historians often have the unenviable task of finding paintings for study manually. With an extensive dataset of material depictions within art, this task might become slightly easier for art historians that study the artistic depiction of materials, such as for example stone [50, 51]. The ability to easy find fabrics in paintings and it’s fine-grained subclasses such as velvet, silk and lace could be used for the study of fashion and clothes in paintings in general [52, 53] or for paintings from a specific cultural context, such as Italian [54], English and French [55] or even for the clothes worn by specific artists [56]. The human body and it’s skin, which clothing covers, is often studied within paintings [52, 57, 58]. For example, the Metropolitan Museum, published an essay on anatomy in the Renaissance, for which artworks depicting the human nude were used [59]. In this work on anatomy, only items from the Metropolitan Museum were used but with an annotated database of material depictions this could be extended and compared to other museum collections. Furthermore, through material categories such as food and flora category, the MIP could give access to typical artistic scenes such as stillives [60, 61] and floral scenes [62] respectively. It should be noted that ‘stuff’ like skin and food might not appear like a stereotypical material, however in this paper we adhere to the view of Adelson, where each object, or ‘thing’, is considered to consist of some material, i.e., ‘stuff’ [63]. Within this view non-stereotypical ‘stuff’ such as skin and food can certainly be considered as a material.

Computer vision datasets.

Today, the majority of image datasets originate from research in computer vision. One of the first relatively large datasets representing object categories [64] has been used to both train and evaluate various computational strategies to solve visual object recognition. The ImageNet and CIFAR datasets [65, 66] are regarded to be standard image recognition datasets for the last decade of research on deep learning vision systems.

Traditionally much visual research has been concerned with object classification but recently material perception has received increasing attention [63, 67–69]. A notable dataset that contains material information is OpenSurfaces [67], which contains around 70k crowd-sourced polygon segmentations of materials in photos. The Material In Context database improved on OpenSurfaces by providing 3 million samples across 23 material classes [68]. To our knowledge, no dataset exists that explicitly provides material information within paintings.

The majority of image datasets contain photographs, but various datasets exist that contain artworks. The WikiArt dataset for instance, which is created and maintained by a non-profit organisation, with the admirable goal “to make world’s art accessible to anyone and anywhere” [70]. The WikiArt dataset has been widely used for a variety of scientific purposes [45, 71–74]. The Painting-91 dataset from [75] consists of around 4000 paintings from 91 artists and was introduced for the purpose of categorization on style or artist. More recently, Art500k was released, which contains more than 500k low resolution artworks which were used to automatically identify content and style [76] within paintings.

The visual difference introduced by painterly depiction does not pose any significant difficulties to the human visual system, however it can be challenging for computer vision systems as a result of the domain shift [77–79]. Differences between painting images and photographic datasets include for instance composition, textural properties, colors and tone mapping, perspective, and style. As for composition, photos in image datasets are often ‘snapshots’, taken with not too much thought given to composition, and typically intended to quickly capture a scene or event. In contrast, paintings are artistically composed and are prone to historical style trends. Therefore, photos often contain much more composition variation relative to paintings. Within paintings, composition can vary greatly between different styles. The human visual system can distinguish styles—for example, Baroque vs. Impressionism—and also implicitly judge whether two paintings are stylistically similar. Research in style or artist classification, as well as neural networks that perform style transfer, attempt to model these stylistic variations in art [45, 80].

Humans can also discount stylistic differences, for example, identifying the same person or object depicted by different artists. Similarly, work in domain adaptation [77–79] focuses on understanding objects or ‘stuff’ across different image styles. Models that learn to convert photographs into painting-like or sketch-like images have been studied extensively for their application as a tool for digital artists [80]. Recent work has shown that such neural style transfer algorithms can also produce images that are useful for training robust neural networks [81]. However, photos that have been converted into a painting-like image are not identical to paintings; paintings can contain spatial variations of style and statistics that are not present in photos converted into paintings. Furthermore, painterly convention and composition are not taken into account by style-transfer algorithms.

Depending on the end goal for a computer vision system, it can be important to learn from paintings directly. Of course, when the end goal is to detect pedestrians for a self-driving car, learning from real photos, videos, or renderings of simulations can suffice. However, if the goal is to simulate general visual intelligence, multi-domain training sets are essential. Furthermore, if the goal is to create computer vision systems with a perception that matches human vision, training on paintings could be very beneficial. Paintings are explicitly created by and for human perception and therefor contain all the required cues to trigger robust perceptions. Therefor, networks trained on paintings are implicitly trained on these perceptual cues.

The multifaceted nature of datasets.

While we have distinguished the broad purposes of datasets and exemplified each with representative datasets, it is important to keep in mind that these datasets can serve multiple goals across the taxonomy. For example, the Flickr Material Database [12] was initially created as a perceptual dataset to study how quickly human participants were capable of recognizing natural materials. However, since then it has also often been used as a computer vision dataset, including by the original authors themselves [82]. In this study paintings are considered as especially interesting as they can be used for perceptual experiments, for digital art history, i.e., cultural visual ecology and can furthermore be used to train and test computer vision networks. The dataset presented in this paper is explicitly designed with this multidisciplinary nature in mind.

Automatic bounding boxes.

While we consider our dataset to be quite larger, it only covers a small but representative portion of art history. It might be required to access materials in paintings that are not part of our dataset. To allow for this, we have trained a FasterRCNN [89] bounding box detector to localize and label material boxes in unlabelled paintings. We use the publicly available implementation from [90] with a ResNet-50 backbone and feature pyramid network (R50-FPN). The model is finetuned from a COCO-pretrained model for 100 epochs using the default COCO hyperparameters from [90]. First we trained the detector on 90% of annotated paintings in the dataset. In section Automatically detected bounding boxes in the Results and demonstrations section below, we show our evaluation of the network, which was performed on the remaining 10% of annotated paintings. While we created this network to be able to detect paintings outside our dataset, we decided to apply the network on our dataset in order to more densely annotate our paintings. Therefor, after the evaluation, we ran the detection network on the entire set of paintings, i.e., training and testing data, in an attempt to more exhaustively annotate materials within paintings. From the automatic detected bounding boxes we first removed all boxes that scored <50% confidence (as calculated by FasterRCNN). Next, we filtered out automatic boxes that were likely already identified by human annotators, be removing automatic bounding boxes that scored ≥ 50% on intersection over union, i.e., automatic boxes that shared the majority of it’s content with human boxes. This resulted in an additional 94k bounding boxes.

Perception-based recipes in painterly depictions.

As previously argued, we believe that painterly techniques are a sort of perception-based recipe. Applying these recipes results in a stylized depiction that can trigger a robust perception of the world. Studying the image features in paintings can lead to an understanding of what cues the visual systems needs to trigger a robust perception.

Here we explore a perceptual shortcut for the perceptions of glass by annotating highlights in paintings and comparing these with highlights in photos. In paintings, it has previously been observed that highlights on drinking glasses are typically in the shape of windows, even in outdoor scenes [91]. This highlight-shape can even often be found in contemporary cartoons [92, 93]. This convention can be considered as a perception-based recipe, where the result is a window-shaped highlight that appears to be a robust cue that triggers the perception of gloss for drinking glasses.

We used bounding boxes from our dataset and photographs sourced from COCO [94]. Participants for this study included 3 of the authors, and one lab-member naive to the purpose of this experiment.

Images. We used 110 images of drinking glasses, split equally across paintings and photos. First, we selected all bounding boxes in the glass, liquid container category in our dataset. From this set, we manually selected drinking glasses, since this category can also contain items such as glass flower vases. Next, we removed all glasses that were mostly occluded, were difficult to parse from the background—for example when multiple glasses were standing behind each other, and removed images smaller then 300x300 pixels. This resulted in a few hundred painted drinking glasses.

Next, we downloaded all images containing cups and wineglasses from the COCO [94] dataset, from which we removed all non-glass cups, occluded glasses, blurry glasses and glasses that only occupy a small portion of the image, and small images. This left us with 55 photos of glass cups and wineglass. Next, we randomly selected 55 segmentations from our painted glass collection. Each image was presented in the task at 650 × 650 pixels, keeping aspect ratio intact.

During this selection phase, we did not base our decision on the shape of the glass. After the experiment, as part of the analysis, we divided the glasses into three shapes, namely spherical, cylindrical, and conical glasses. See Fig 3 for an example of each shape.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Examples of the three glass shapes.

From left to right: spherical, cylindrical and conical. The red geometry annotations were manually created by the authors, and were used to standardize across glasses for the highlight analysis. Paintings used, from left to right: Portret van een jongen, zittend in een raamnis en gekleed in een blauw jasje, by Jean Augustin Daiwaille. 1840. Still Life with a gilded Beer Tankard, by Willem Claesz. Hed, 1634. The White Tablecloth, by Jean Baptiste Siméon Chardin, 1731. The left and middle images courtesy of The Rijksmuseum. The right image courtesy of The Art Institute of Chicago. All images reproduced under a CC0 license.

https://doi.org/10.1371/journal.pone.0255109.g003

Task. Participants annotated highlights on drinking glasses using an annotation interface. In the annotation interface, users would be presented with an image on which the annotated geometry was visible. This made it clear which glass should be annotated, in case multiple glasses were visible in the image. Users were instructed to instruct all visible highlights on that glass. Once the user started annotating highlights, the geometry would no longer be visible. Annotations could be made by simply holding down the left-mouse button and drawing on top of the image. Once a highlight was annotated a user could mark it as finished and continue with the next highlight, and eventually move to the next image.

Results. To compare the highlights between photos and paintings, we resized each glass to have the same maximum width and height, and then overlaid each glass on the center. Initially, we overlaid all images for both types of media (not visualized here) and found the resulting figure quite noisy. However, when we split the glasses on media and shape, clear patterns emerge for painted glasses Fig 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The overlaid highlights created by users, split on media and glass shape.

In general, the photographic glass shapes display more variability and do not display a clear pattern. Note that for photos, no stimuli existed with a conical shape in our set which leads to a black image, since there were no highlight-annotations. On the right, for painted glasses, we see clear patterns in the placement of highlights for each glass shape.

https://doi.org/10.1371/journal.pone.0255109.g004

As can be seen, painters are more likely to depict highlights on glasses adhering to a stylized pattern, at least for spherical and conical glasses. This pattern of highlights is perceptually convincing and very uniform in comparison with the variation found within reality.

Furthermore, we calculated the agreement between each pair of participants, as the ratio of pixels annotated by both participants (i.e., overlapping area) divided by the number of pixels that was an annotated by either participant (i.e., total area). Averaged across participants, the agreement on paintings (0.33) was around 50% higher relative to the average agreement between participants on photos (0.21). This means that for our stimuli, highlights in paintings are less ambiguous when compared to photos.

Material presence.

Within the 19,325 paintings, participants exhaustively identified the presence of 123,244 instances of 15 coarse materials. In other words, for each painting, participants indicated if each material is or is not present. The distribution of unique materials per painting is normally distributed with an average of 5.7 unique coarse materials present per painting (std = 2.8 materials). The most frequent materials are skin and fabric. The least frequent are ceramics and food. The relative frequency of each coarse material is presented in Fig 5. We did not exhaustively identify fine-grained materials within paintings, so we will not report those statistics here.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The proportion of paintings in our dataset that depict at least one instance of each material.

https://doi.org/10.1371/journal.pone.0255109.g005

Based on prior knowledge of natural ecology, one might assume that some materials, such as skin and fabric might often be depicted together in paintings. To quantify the extent to which materials are depicted together, we create a co-occurrence matrix presented in Fig 6, where each cell is the co-occurrence for each pair of materials as the number of paintings where both materials are present, divided by the number of paintings where either (but not both) materials are present. We can see for example, that if skin is depicted, there is a 94% change to also find fabric in the same painting.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Co-occurrence matrix.

Each cell equals the number of paintings where both materials are present divided by the number of paintings where one or the other material is present.

https://doi.org/10.1371/journal.pone.0255109.g006

Furthermore, one might expect that the presence of one material can have an influence on another material. For example, one might expect that gem might almost always be depicted with skin, but that skin is only sometimes depicted with gem. To quantify these relations, we calculated the occurrence of a material given that another material is present. We visualize this in Fig 7. Here we see that if gem is present, then skin is found in 99% of the paintings, but that if skin is present, then gem is found in only 20% of the paintings. The same relationship is true for gem and fabric. This implies that gems are almost always depicted with human figures, however that human figures are not always shown with gems. Another example, when liquid is present, in 85% of the paintings, wood is also present. One might be reminded of typical naval scenes, or landscapes with forests and rivers. Inversely, when wood is present, only 34% of the paintings depict liquid. For food and ceramics, two materials which are present in less then 10% of paintings, we see that if food is present, ceramics has a 53% change to be present as well, but the inverse is only 33%. This implies that food is served in, or with, ceramic containers half of the time, but that this is only 1/3rd of what ceramics is used for.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Likelihood matrix.

This matrix visualizes the influence a material has on the likelihood of finding another material within the same painting, i.e., if one material on the y-axis is present, then how does this impact the presence of other materials on the x-axis? Calculated as the number of paintings where both materials are present, divided by the number of paintings that contain only one of the materials.

https://doi.org/10.1371/journal.pone.0255109.g007

Material presence over time.

We have previously shown the distributions of materials in paintings in Fig 5. When we created similar distributions (not visualised) for temporal cross-sections, for example for a single century, we found that these distributions were remarkably similar to the average distribution in Fig 5. We used t-tests, to see if the distribution for any century was significantly different from the average distribution in Fig 5 and found no significant effect. This means that despite the changes in stylistic and artistic techniques over time, the distribution of materials (such as in Fig 5 remained remarkably stable over time for the period covered in our dataset.

The spatial layout of materials.

Paintings are carefully constructed scenes and it follows that a painter would carefully choose the location at which to depict a material. With the knowledge that spatial conventions exists within paintings (e.g., lighting direction [39, 40]), one can assume that these might extend to materials. The average spatial location and extent of materials is visualized by taking the (normalized) location of each bounding box for a specific material and subsequently plotting each box as a semi-transparent rectangle. The result is a material heatmap, where the brightness of any pixel indicates the likelihood to find a material at that pixel. In this section, we limit the material heatmaps to only include the bounding boxes created by human annotators. In the next section, we visualize the material heatmaps for automated boxes too.

Material heatmaps for the 15 coarse materials are shown in Fig 8. The expected finding that sky and ground are spatially high and low within images serves as a simple validation or sanity-check of the data. It is interesting to see how skin and gem are both vertically centered within the canvas. It appears to suggests a face, with necklaces and jewelry adorning the figure. In general, each material heatmap appears to be roughly vertically symmetric. For glass, there does however appear to be a minor shift towards the top-left. This might be related to an artistic convention, namely that light in paintings usually comes from a top-left window [39]. When we look at the heatmaps for the sub-categories for glass in Fig 9, we see that it is indeed glass windows that show the strongest top-left bias.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Material heatmaps, which illustrate the likelihood at any given pixel to find the target material at that pixel.

Brighter colors indicate higher likelihoods.

https://doi.org/10.1371/journal.pone.0255109.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Material heatmaps for glass sub categories.

For glass windows, it is interesting to see the clustering in the top-left corner, which is in agreement with the artistic convention of having light come from the top left.

https://doi.org/10.1371/journal.pone.0255109.g009

Automatically detected bounding boxes.

Besides the bounding boxes created by humans, we also trained a FasterRCNN network to automatically detect bounding boxes with 90% of the data as training data. On the remaining unseen 10% of paintings, the network detected 90,169 bounding boxes. We removed those with a confidence score below 50%, which resulted in 24,566 remaining bounding boxes. In the section below, all references to the automated bounding boxes refer to these 24,566 bounding boxes.

A qualitative sample of detected bounding boxes is given in Fig 10. Our human bounding boxes are non-spatially exhaustive in nature meaning that not every possible material has been annotated. As a result, the automatically created bounding boxes cannot always be matched against our human annotations and thus we cannot use this to evaluate their quality. In order to validate the automatic bounding box detection, we performed a simple user study to get an estimate of the accuracy per material class, which is visualized in Fig 11. In the user study, a total of 50 AMT participants judged a random sample of 1500 bounding boxes. The 1500 bounding box stimuli were divided into 10 sets of 150 stimuli, where each set contain 10 boxes per course material class. Each individual participant only saw one set and each set was seen by 5 unique participants. Therefore, this can be thought of as 10 experiments, each with 5 participants and 150 stimuli, where participants performed the same task across each set/experiment. The participants were tasked to rate whether each stimuli is either a good or a bad bounding box, where a bad bounding box was defined as either 1) having the wrong material label (e.g., “I see wood, but the label says stone”) or as having a bad boundary where the edges of the bounding box were not near the edges of the material. The order of stimuli was randomized between sets and participants. This leads to a total of 7500 votes, 500 per material classes. The ratio of good to bad votes per material classes can serve as a measure of accuracy, which has been visualized in Fig 11.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Examples of detected materials in unlabeled paintings.

Automatically detecting materials can be useful for content retrieval for digital art history and for filtering online galleries by viewer interests.

https://doi.org/10.1371/journal.pone.0255109.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. In the bar graph, the accuracy for automatically detected bounding boxes is displayed in the same order as in Fig 5.

The values were derived from human quality votes. On the right, we compare the material heatmaps for fabric between the automated and the human annotation bounding boxes. From left to right, top to bottom: Lake George, by John William Casilear, 1857. Man with a Celestial Globe, by Nicolaes Eliasz Pickenoy, 1624. A Seven-Part Decorative Sequence: An Interior, by Dirck van Delen, 1631. Thomas Howard, 2nd Earl of Arundel, by Anthony van Dyck, 1620. The Poultry Seller, by Cornelis Jacobsz. Delff, 1631. First and second digital image courtesy of The Metropolitan Museum of Art. Third and last image courtesy of The Rijksmusuem. Fourth image courty of the Getty’s Open Content Program. All images reproduced under a CC0 license.

https://doi.org/10.1371/journal.pone.0255109.g011

The participant agreement averaged across bounding boxes was found to be 80%, i.e., on average 4 out of 5 participants agreed on their rating per bounding box. As a result of the user study, we found a mean average accuracy of 0.55 across participants. While not high, these results are somewhat interesting in that they show that a FasterRCNN model is capable of detecting materials in paintings, without any changes to the network architecture or training hyperparameters. It is certainly promising to see that an algorithm designed for object localization in natural images can be readily applied to material localization in paintings. Likely, the accuracy could be further improved by finetuning the network which we have not done in this paper.

It is interesting to note that the spatial distribution of automatically detected bounding boxes looks very similar to the spatial distribution of the human annotated bounding boxes. We have visualized the material heatmap for one material, fabric, for the automated bounding boxes to show the similarity with the material heatmap for the same material created from human annotation bounding boxes. This has been visualized in the right side of Fig 11

Extracting polygon segmentations.

A natural extension of material bounding boxes is material segments [67–69]. Polygon segmentations are useful for reasoning about boundary relationships between different semantic regions of an image, as well as the shape of the regions themselves. However, annotating segmentations is expensive and many modern datasets rely on expensive manual annotation methods [67, 69, 94–96]. Recent work has focused on more cost effective annotation methods (e.g. [97–100]). One broad family of methods to relax the difficulty of annotating polygon segmentations is through the use of interactive segmentation methods that transform sparse user inputs into a full polygon masks.

For this dataset, we apply interactive segmentation with the crowdsourced extreme clicks as input. To evaluate quality, we compared against 4.5k high-quality human annotated segmentations from [15], which were sourced from the same set of paintings. We find that both image-based approaches like GrabCut (GC) [101] and modern deep learning approaches such as DEXTR [98] perform well. Surprisingly, DEXTR transfers quite well to paintings despite being trained only on natural photographs of objects. The performance is summarized in Table 3. The performance is summarized using the standard intersection over union (IOU) metric. IOU is computed as the intersection between a predicted segment and the ground truth segment divided by the union of both segments. IOU is computed for each class, and mIOU is the mean IOU over all of the classes. Samples are visualized in Fig 12. Segments produced by these methods from our crowdsourced extreme points will be released with the dataset.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Segmentation visualizations.

Left to right: Original Image, Ground Truth Segment, Grabcut Extr Segment, DEXTR COCO Segment. Both Grabcut and DEXTR use extreme points as input. For evaluation, the extreme points are generated synthetically from the ground truth segments. The IOU for each segmentation is shown in the bottom right corner. Top image: Dance before a Fountain, by Nicolas Lancret, 1724, Digital image courtesy of the Getty’s Open Content Program. Bottom image: Still life with fish, by Pieter van Noort, 1660, Het Rijksmusuem. Images reproduced under a CC0 license.

https://doi.org/10.1371/journal.pone.0255109.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Segmentations from extreme clicks.

Grabcut [101] rectangles use bounding-box only initialization as a reference baseline. Grabcut Extr is based on the improved GC initialization from [88] with small modifications: (a) we compute the minimum cost boundary with the cost as the negative log probability of a pixel belonging to an edge; (b) in addition to clamping the morphological skeleton, we also clamp the extreme points centroid as well as the extreme points; (c) we compute the GC directly on the RGB image. DEXTR [98] Pascal-SBD and COCO are pretrained DEXTR ResNet101 models on the respective datasets. Note that Pascal-SBD and COCO are natural image datasets of objects, but DEXTR transfers surprisingly well across both visual domains (paintings vs. photos) and annotation categories (materials vs. objects).

https://doi.org/10.1371/journal.pone.0255109.t003

Learning robust cues for finegrained fabric classification.

The task of distinguishing between images of different semantic content is a standard recognition task for computer vision systems. Recently, increasing attention is being given to “fine-grained” classification, where a model is tasked with distinguishing images of the same coarse-grained category (e.g., distinguishing different species of birds or different types of flora [102–104]). Classifiers for material categories can perform reasonably well on coarse-grained classification by relying on context alone. In comparison, fine-grained classification is more challenging for deep learning systems as contextual clues are often equal within fine-grained classes. For example, one might reason that the material of a shirt might be recognized as fabrics partly because of the context, i.e., being worn by a figure. However, in fine-grained classification context can be held consistent across classes (for example, both velvet shirts and satin shirts are worn). To successfully distinguish between these two fine-grained classes in a context-controlled setting, a classifier should use non-contextual features (at least more so relative to uncontrolled settings).

The rational above leads to two interesting possibilities. First, we hypothesize that the painted depictions of materials can be beneficial for fine-grained classification tasks. Since artistic depictions focus on salient cues for perception, i.e., paintings are explicitly created for and by perception, it is possible that a network trained on paintings is able to learn a more robust feature representation by focusing on these cues.

Second, visualising the features used by a successful fine-grained classifier could potentially lead to the uncovering of latent perceptual cues. For example, in the Perception-based recipes in painterly depictions section above, we showed that window-shaped highlights are a robust cue for the perception of gloss on drinking glasses. However, it is assumed that the visual system used many such cues which are as of now unknown. Visualising what cues are used by classifiers might lead to the finding of cues used by the perceptual system.

Task. We experimented with the task of classifying cotton/wool versus silk/satin. The latter can be recognized through local cues such as highlights on the cloth; such cues are carefully placed by artists in paintings. To understand whether artistic depictions of fabric allow a neural network to learn better features for classification, we trained a model with either photographs or paintings. High resolution photographs of cotton/wool and silk/satin fabric and clothing (dresses, shirts) were downloaded and manually filtered from publicly available photos licensed under the Creative Commons from Flickr. In total, we downloaded roughly 1K photos. We sampled cotton/wool and silk/satin samples from our dataset to form a corresponding dataset of 1K paintings. We analyzed the robustness of the classifier trained on paintings versus the classifier trained on photos in two experiments below. Taken together, our results provide evidence that a classifier trained on paintings can be more robust than a classifier trained on photographs, and that visualizing these features could lead to discovering perceptual cues utilized by the human visual system.

Generalizability of classifiers. Does training with paintings improve the generalizability of classifiers? To test cross-domain generalization, we test the classifier on types of images that it has not seen before. A classifier that has learned robust features will outperform a classifier that has learned features based on more spurious correlations. We tested the trained classifiers on both photographs and paintings across the two classes using 1000 samples per domain.

In Table 4, the performance of the two classifiers are summarized. We found that both classifiers perform similarly well on the domain they are trained on. However, when the classifiers are tested on cross-domain data, we found that the painting-trained classifier performs better than the photo-trained classifier. This suggests that the classifier trained on paintings has learned a more generalizable feature representation for this task.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Classifier performance across domains.

Classifiers are trained to distinguish cotton/wool from silk/satin. The first column represents the classifier trained on photographs, and the second column represents the classifier trained on paintings. In the first row, the classifiers are tested on images of the same type they were trained on (i.e., trained and tested on photos, and trained and tested on paintings). In the second row, the classifiers are tested on the other medium, i.e., trained on photos and tested on paintings and vice versa.

https://doi.org/10.1371/journal.pone.0255109.t004

We have reported the confusion matrices in Table 5. The photo classifier applied to paintings is heavily biased towards satin predictions. We hypothesize that this is because the photo classifier is relying on spurious cues (such as image background or clothing shape) over more robust cues and thus that the shift from photos to paintings causes its mispredictions. Only 21% of cotton samples are correctly identified as cotton while 79% are identified as satin. This skew in precision/recall across the classes is also reflected by the F1 scores for each class. On the other hand, the painting classifier applied to photos is much more balanced in its predictions, with 57-59% of predictions being correct. The precision/recalls are also much better balanced as reflected by the F1 scores.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Confusion matrix for the two classifiers.

The top represents the classifier trained on photos, tested on paintings which is heavily biased towards satin. The bottom represents the classifier trained on paintings, tested on photos, which is more balanced in its predictions.

https://doi.org/10.1371/journal.pone.0255109.t005

Human agreement with classifier cues. How indicative are the cues used by each classifier to humans? We hypothesized that training networks on paintings might lead to the use of more perceptually relevant image features. If this is true, then the features used by the classifier trained on paintings should be preferably by humans.

We produced evidence heatmaps with GradCAM [105] from the feature maps in the network before the fully connected classification layer. We extracted high resolution feature maps from images of size 1024 × 1024 (for a feature map of size 32 × 32). The heatmaps produced by GradCAM show which regions of an image the classifier uses as evidence for a specific class. If the cues (i.e., evidence heatmaps, such as in Fig 13) are clearly interpretable, this would imply the classifier has learned a good representation. For both models, we computed heatmaps for test images corresponding to their ground truth label. We conducted a user study on Amazon Mechanical Turk to find which heatmaps are judged as more informative by users. Users were shown images with regions corresponding to heatmap values that are above 1.5 standard deviations above the mean. Fig 13 illustrates an example. Users were instructed to “select the image that contains the regions that look the most like <material>”, where <material> was either cotton/wool or silk/satin. We collected responses from 85 participants, 57 of which were analyzed after quality control. For quality control, we only kept results from participants who spent over 1 second on average per trial.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Visualization of cues used by classifiers.

Left to right: Original Image, Masked Image (Painting Classifier), Masked Image (Photo Classifier). The unmasked regions represent evidence used by the classifiers for predicting “silk/satin” in this particular image. See main text for details. Image from Interior of the Laurenskerk at Rotterdam, by Anthonie De Lorme, with figures attributed to Ludolf de Jongh, 1662. Digital image courtesy of the Getty’s Open Content Program, reproduced under a CC0 license.

https://doi.org/10.1371/journal.pone.0255109.g013

Overall, we found that the classifier trained on paintings uses evidence that is better aligned with evidence preferred by humans (Fig 14). This implies that training on paintings allows classifiers to learn more perceptually relevant cues, and it shows that this method might be useful to detect previously unknown perceptual cues.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Human preference for classification cues used by each classifier.

The y-axis represents how often humans prefer the cues from a classifier trained on the same domain as the test images. For example, the first bar indicates that in 73.2% of the cases, humans preferred cues from the classifier trained on paintings when classifying wool/cotton paintings (and thus, the inverse, that in 26.8% of the cases, humans preferred cues from the photo classifier.) Interestingly, note the last column—humans equally prefer cues used by both classifiers for classifying silk/satin photos despite the painting classifier never seeing a photo during training.

https://doi.org/10.1371/journal.pone.0255109.g014

Due to the domain shifts, training and testing a classifier on a single type of images will outperform a classifier trained and tested on different kind of images. Based on this, if paintings do not lead to a more robust feature representation we would expect the painting classifier to do best on paintings and the photo classifier to perform best on photos. Interestingly however, this does not hold when testing on photos of the satin/silk category (see last column of Fig 14). We found that users actually have no preference for the cues from either classifier, i.e., the cues from the painting classifier appears to be equally informative as the cues from the photo classifier for categorizing silk/satin in photos. This suggests that either (a) the painting classifier has learned human-interpretable perceptual cues for recognizing satin/silk, or (b) that the photo classifier has learned to classify satin/silk based on some spurious contextual signals that are difficult to interpret by humans. We asked users to elucidate their reasoning when choosing which set of cues they preferred. In general, users noted that they preferred the network which picks out regions containing the target class. Therefore, it seems that the network trained on paintings has learned better to distinguish fabric through the appearance of such fabrics in the image over other contextual signals (see Fig 13).