Perceptual holistic color combination analysis of Papilionidae butterflies as aesthetic objects

In this study, we clarified the holistic color combination rules of human-preferred Papilionidae butterflies by examining the hue, lightness, and chroma. A set of 118 Papilionidae butterfly images used in our previous study was analyzed. These images were classified via hierarchical density-based spatial clustering based on perceptual similarities of colors that were obtained from a subjective image classification experiment. The color combinations of the clustered images were determined based on representative colors that were analyzed by a Gaussian mixture model with minimum message length and the color combination types defined in our previous study. Consequently, we obtained the following holistic color combination rules for Papilionidae: 1) contrasting lightness, similar chroma, and similar hue, 2) contrasting lightness, contrasting chroma, and similar hue, 3) similar lightness, similar chroma, and complementary hue, and 4) similar lightness, similar chroma, and similar hue. These rules suggest that minority color harmony theories are valid under particular conditions.


Introduction
Color harmony studies contribute not only to developing color design but also to clarifying the mechanisms of human aesthetic responses. Conventional color harmony theories developed in the past were not always consistent with psychological results [1][2][3][4]. Thus, to achieve consistency between conventional color harmony theories and psychological results, the concept of computational aesthetics has had to be applied to color harmony studies, e.g., clarifying the color combination structures of paintings and butterflies as aesthetic objects [5,6]. In a previous study, the common color combination rules in paintings were not clarified [5]. In contrast, our previous work [6] focused on the beautiful colors in nature that have been applied to color design [7][8][9][10][11], and the artistic quality of Papilionidae butterflies [12][13][14][15][16][17]. Therein, the color combination rules of 118 human-preferred Papilionidae butterfly images were clarified [6], i.e., contrasting lightness, similar chroma, and similar hue, which agreed with a part of the psychological color harmony principles [1][2][3][4]. Nevertheless, similar lightness, contrasting chroma, and complementary hue have also been obtained as a minor proportion of results that agree with a part of the conventional color harmony principles [1,18]. These results suggest that the minority color combinations that do not appear in the psychological a1111111111 a1111111111 a1111111111 a1111111111 a1111111111

Image classification experiment
To determine the perceptual similarity between each pair of images, we conducted a subjective image classification experiment. Experimental methods to measure the perceptual similarities of images include the following: (a) table scaling experiment, in which observers arrange the stimuli on the table according to their perceptual similarities; (b) computer scaling experiment, where a reference image is compared with some images, and the most similar one to the reference image is selected [19]; and (c) ViSiProG test, wherein some stimuli are presented simultaneously, and observers select and place the similar stimuli in a separate box to form the clusters [20]. To classify images of similar colors, as required in this study, a modified version of the ViSiProG test was designed.
We selected 118 Papilionidae butterfly images, which were used in our previous work as human-preferred Papilionidae butterflies (Fig 1) [6], as the stimuli. These images were centered on a white square-background (R, G, B = 1.0, 1.0, 1.0) covering an area of 12 cm 2 .
As the experimental environment, two monitors (50 in and 32 in) connected to a MacBook Pro (Retina, 13-inch, Early 2015) (Apple, Inc., Cupertino, CA, USA) were placed in a darkened room (Fig 2). Additionally, a 50 in monitor (TH-50LFE7J, Panasonic, Corp., Osaka, Japan) was positioned to the right of a 32 in monitor (BL3201PT, BenQ, Inc., Taipei, Taiwan). An i1 Display Pro (X-Rite, Inc., Grand Rapids, MI, USA) was used to correct the colors of both monitors. On the 50 in monitor, thumbnail images of all stimuli were presented simultaneously against a black background (R, G, B = 0, 0, 0). The arrangement was in random order and the size was 5 cm 2 (Fig 3(A)). Those images were enlarged to 12 cm 2 upon double-clicking. On the 32 in monitor, the enlarged images and two folders were presented, and one folder was opened (Fig 3(B)). The folders had a black background (R, G, B = 0, 0, 0) and displayed no extra information (e.g., toolbar and sidebar). The left (32 in) monitor's background was "Solid Gray Pro Ultra Dark" (R, G, B = 0.188, 0.188, 0.188) in Mac. The viewing distance between the monitor and the observer was 75 cm. Thirty-one observers with normal color vision from Chiba University participated in the experiment (16 males and 15 females, aged from their late teens to 30s).
In the experiment, each observer took Ishihara's test for color deficiency and was taught how to operate the experimental window. They were instructed to classify the stimuli based on only color, independent of shape and pattern. During the experiment, each observer dragged similarly colored images from the right monitor into the same folder on the left monitor. They could not include the same stimulus in the other folders, but could add new folders as necessary. This trial was repeated until all 118 stimuli were classified. The observers were briefed on the experiment and its safety, and provided written consent before beginning the experiment. For minor observers, consent from parents or guardians was not required by the ethics committee.

Image classification
To classify the 118 Papilionidae images, we calculated the similarities of the images based on the experimental results. First, to obtain the similarities, the frequencies at which a pair of images was classified into the same folders in the experiment were divided by the number of observers. The distance matrix was defined by the similarity of each pair of images subtracted from 1. The matrix consisted of 117 rows and columns, and each element of this matrix denotes the distance of each pair of images. Table 1 shows a part of the distance matrix as an example. Using the distance matrix, HDBSCAN was performed with the dbscan package of R (The R Foundation, Vienna, Australia) to classify the images [21]. The minimum size of the clusters was two. HDBSCAN is a hierarchically modified density-based spatial clustering of applications with noise (DBSCAN) [22]. It can automatically determine the clusters while removing noise. Because HDBSCAN improve the hierarchical cluster analysis used in our previous work, it was used in this study.

Color combination characteristics
In our previous work, the representative colors of the classified images were extracted by a GMM with the mclust package of R (The R Foundation, Vienna, Australia) [23]. The GMM considers the color distributions as mixed multiple Gaussians and presents the probability density function of the color distributions in each Gaussian mixture component by using three variables, namely, mean values (μ), mixing proportions (πk, the estimated populations that configure a Gaussian), and covariance matrix (S), which were considered as the values, sizes, and ranges of a representative color, respectively. In this work, because the proportions under 3% were almost invisible [24], they were excluded from the analysis [6]. The number of components in each cluster was determined from the minimum message length criterion (MML) instead of the Bayesian information criterion (BIC), which was used as the reference value in our previous work [6]). The MML values were calculated for 20 components, because further components would complicate the results. In the MML, the shorter the code build for the data, the better the data generation models are [25]. GMM with MML outperformed those with other criterion, including BIC, in estimating the number of components [25]. Furthermore, it was effective in unsupervised color image segmentation with GMM and automatic estimation of the number of components [26]. Therefore, because GMM with MML improves upon the results obtained in our previous work using GMM with BIC, it was used in this study.
To determine the color combination characteristics of each cluster, we used the same color combination types as defined in our previous work (Table 2) [6]. The representative colors were classified into any one of the categories listed in Table 2 based on the mean values in the CIELCh color space (lightness, chroma, and hue scale in CIELAB) [27]. The color combinations of the clusters were determined from the color combination types in Table 2 based on the category combinations of representative colors.

Results
The selected images were classified into 24 clusters based on their perceptual similarity. Fig 4  depicts the dendrogram of the classified images and their cluster numbers. Table 3 lists the number of components (representative colors) and the best MML values in each cluster. Table 4 shows the μ, πk, and S values of each component in the clusters. From an interpretation of Table 4 according to the definition of Table 2, the color combinations of the clusters tend to exhibit the following four patterns: 1. contrasting lightness, similar chroma, and similar hue  5-7 depict the relative frequencies of CIELCh categories in each cluster of the color combination patterns 1-4. In Fig 5, 11 clusters were included in pattern 1. In terms of lightness (Fig 5(A)), low-category colors were more frequent than their high-category counterparts; there were a few mid-category colors. This result indicates the dominant low lightness and contrasting lightness combinations. In terms of chroma (Fig 5(B)), the low categories were the most frequent. There were a few middle categories, but no high categories. This indicates the presence of dominant low chroma and similar chroma combinations. In terms of hue (Fig 5  (C)), the hues 2-4 occurred relatively more frequently; additionally, no further than the sixth hue was present. This indicates the dominant orange to yellow (the ranges of hue names in CIELCh were determined based on [28,29]) and similar hue combinations. Therefore, these clusters mainly have contrasting lightness, similar chroma, and similar hue combinations.
As shown in Fig 6, five clusters were included in pattern 2. In terms of lightness (Fig 6(A)), the same tendency as that in Fig 5(A) was observed. As for chroma (Fig 6(B)), the same The model is parameterized by geometric characteristics determined using the covariance matrix of each component.
"VVV" shows an ellipsoidal distribution and varying volume, shape, and orientation. "VVI" shows a diagonal distribution and varying volume and shape. "VVE" shows an ellipsoidal distribution and equal orientation.
(a)                    tendency as that of lightness appeared. These results indicate the dominant low lightness (chroma) and contrasting lightness (chroma) combinations. In terms of hue (Fig 6(C)), the hues 2-4 occurred relatively more frequently, and no further than the fourth hue appeared. This indicates the dominant orange to yellow and similar hue combinations. Therefore, these clusters mainly have contrasting lightness, contrasting chroma, and similar hue combinations. μ and ∑ in each component were arranged vertically in order of L � , C � ab , and h ab . Note that as the components that have πk < 3% were excluded from analysis, the number of components in each cluster differs from those in Table 4. As shown in Fig 7, three clusters were included in pattern 3. In terms of lightness (Fig 7  (A)), the low categories were the most frequent. There were few middle categories, and no high categories. In terms of chroma (Fig 7(B)), the same tendency as that of lightness appeared. These indicate the dominant low lightness (chroma) and similar lightness (chroma) combinations. In terms of hue (Fig 7(C)), the hues 2-4 occurred relatively more frequently, and there were a few ninth and tenth hues. These indicate the dominant orange and complementary hue combinations. Thus, these clusters mainly have similar lightness, similar chroma, and complementary hue combinations.
In Fig 8, two clusters were included in pattern 4. In terms of lightness and chroma (Fig 8(A)  and 8(B)), the same tendencies as those in Fig 7(A) and 7(B) appeared, respectively. These indicate the dominant low lightness (chroma) and similar lightness (chroma) combinations. In terms of hue (Fig 8(C)), the same tendency as that in Fig 6(C) appeared. These indicate the dominant orange and similar hue combinations. Accordingly, these clusters mainly have similar lightness, similar chroma, and similar hue combinations. The above four patterns did not appear in clusters 9, 11, and 16. In cluster 9 (Table 4(i)), the middle lightness and chroma were dominant. The hues 2-4 were frequent. Thus, cluster 9 only has similar hue combinations. In cluster 11 (Table 4(k)), the middle and high lightness were frequent. The low chroma was most frequent. There were a few middle chroma and no high chroma. The hues 2-4 were frequent. Thus, cluster 11 mainly had similar chroma and similar hue combinations. In cluster 16 (Table 4(p)), the low and high lightness were frequent, along with the low and middle chroma. Further, the hues 2-4 were frequent. Therefore, cluster 16 mainly has contrasting lightness and similar hue combinations.
In addition, these color combination patterns are not always consistent with the characteristics of the dendrogram in Fig 4. In this dendrogram, the higher the similarity of colors of images, the greater the belongingness of the corresponding images to the adjacent branches. However, a few clusters that have the same color combination pattern are found farther from

Discussion and limitations
We obtained the holistic color combination rules of human-preferred Papilionidae butterflies for 4 different categories, as mentioned previously. In the previous psychological color harmony studies, the following robust color harmony principles were invariably obtained: "High lightness," "Unequal lightness values" (large lightness difference), "Equal chroma" (same or similar in chroma color), and "Equal hue" (same or similar hue color) [1][2][3][4]. The above three principles, except "High lightness," qualitatively agreed with the results of our previous work, i.e., contrasting lightness, similar chroma, and similar hue [6]. In addition to our previous work, the color combination pattern 1 is also consistent with the above three principles (e.g., "Unequal lightness values," "Equal chroma," and "Equal hue") in this study. As they appeared most frequently in the clusters, the contrasting lightness, similar chroma, and similar hue are the most dominant color combinations of human-preferred Papilionidae butterflies. Similar lightness, contrasting chroma, and complementary hue differ from the above principles; however, they appeared in the color combination patterns 2-4 and in the minority of results in our previous work [6]. The similar lightness and complementary hue qualitatively agree with the "equal lightness" and "complementary hue" that are part of the conventional color harmony principles [1]. The contrasting chroma appears in the results of Chuang and Ou as well, and their results exhibited large 95% confidence intervals [18]. It is likely that these color combinations did not appear because the color appearance attributes were handled independently and those results were simplified in previous psychological color harmony studies. In contrast, this study investigated the results by integrating color appearance attributes. Therefore, these color combinations may not be universal, but may harmonize to a limited extent. The conditions for this are as follows. The similar lightness harmonizes by combining similar chroma. The contrasting chroma harmonizes by combining contrasting lightness and similar hue. The complementary hue harmonizes by combining similar lightness and similar chroma. Furthermore, this study has several limitations. Initially, the conditions that harmonize similar lightness, contrasting chroma, and complementary hue were shown. As these color combination rules were obtained from the human-preferred Papilionidae butterflies for which the color harmony was demonstrated in our previous work, these rules are valid. However, we cannot conclude these results for color harmony theories based on limited samples. Psychological experiments to investigate the harmonies of the color combinations under those conditions are required in future work.
Subsequently, the color combination types were defined based on the ranges of segmented color space. These definitions of similar and contrasting color combinations may not always agree with perception. To achieve a more perceptual color combination analysis, the threshold of similarity and contrast must be determined experimentally in future work.
Moreover, it is suggested that the results of image clustering based on the human visual perception in this study were influenced by the conspicuous chromatic colors. Therefore, in the future work, the conspicuous chromatic colors should also be included in the color combination analysis.
Similarly, the concepts of the methods of our previous study and this study must be regulated, to compare the results. In our previous study, we employed a simple and conventional approach to obtain standard results (i.e., data comparable with the results of the future works) because the color combination analysis method has not been established until now. On the other hand, because a fuzzy logic is applied to color image segmentation and color planning system based on the image [8,[30][31][32], the same can be applied to color combination analysis in future works as well.
Finally, the method used in this study can be improved to enable application to the color combination analysis of other aesthetic objects (e.g., flowers, jewelries, etc.). We will analyze and clarify the color combination rules of other aesthetic objects in our future works. If the aesthetic objects reflect the human psychological aesthetic responses, the color combination rules of other aesthetic objects may agree with the results of this study and previous color harmony studies. Moreover, the color combination rules peculiar to other aesthetic objects could also be shown.

Conclusion
In this study, we aimed to clarify the perceptual holistic color combination rules of humanpreferred Papilionidae butterflies. To achieve this, the Papilionidae butterfly images were classified via hierarchical density-based spatial clustering based on experimentally obtained perceptual color similarities. The color combinations of the clustered images were determined based on representative colors extracted by the GMM with minimum message length. We obtained the following holistic color combination rules of Papilionidae: 1. contrasting lightness, similar chroma, and similar hue