An fNIRS-based investigation of visual merchandising displays for fashion stores

This paper investigates a brain-based approach for visual merchandising display (VMD) in fashion stores. In marketing, VMD has become a research topic of interest. However, VMD research using brain activation information is rare. We examine the hemodynamic responses (HRs) in the prefrontal cortex (PFC) using functional near-infrared spectroscopy (fNIRS) while positive/negative displays of four stores (menswear, womenswear, underwear, and sportswear) are shown to 20 subjects. As features for classifying the HRs, the mean, variance, peak, skewness, kurtosis, t-value, and slope of the signals for a 20-sec time window for the activated channels are analyzed. Linear discriminant analysis is used for classifying two-class (positive and negative displays) and four-class (four fashion stores) models. PFC brain activation maps based on t-values depicting the data from the 16 channels are provided. In the two-class classification, the underwear store had the highest average classification result of 67.04%, whereas the menswear store had the lowest value of 64.15%. Men’s classification accuracy for the underwear stores with positive and negative displays was 71.38%, whereas the highest classification accuracy obtained by women for womenswear stores was 73%. The average accuracy over the 20 subjects for positive displays was 50.68%, while that of negative displays was 51.07%. Therefore, these findings suggest that human brain activation is involved in the evaluation of the fashion store displays. It is concluded that fNIRS can be used as a brain-based tool in the evaluation of fashion stores in a daily life environment.


Introduction
Customer behavior due to marketing is strongly influenced by the perception of the surrounding environment as shown by dandyism [1]. Visual merchandising display (VMD) is a marketing strategy to develop floor plans and displays to attract, engage, and stimulate the customer towards making a purchase. If done effectively, the marketers are able to maximize sales. It can also help marketers develop a unique marketing identity and brand, differentiating themselves in the market [2]. In this paper, by using functional near-infrared spectroscopy (fNIRS), we investigate the brain activation of potential customers when viewing various VMDs of four fashion stores.
From the early twenty-first century, VMD has become a science and one of the main tools for increasing sales. A good display can attract the consumer's attention, which is essential in the decision-making process leading to a purchase. To capture a consumer's mind, the needs of the consumer when viewing the VMD should be considered. Law et al. [3] investigated, from a psychological perspective, the affective responses of the consumers to visual stimuli in stores by considering the aesthetic and symbolic aspects of function-oriented products. The conformity between the preconceived fashion image and the image projected from a fashion store works as a mediator that affects the actual purchase decision. Shin et al. [4] investigated the benefits of eco-friendly VMDs on satisfaction as well as the effect of satisfaction on store displays. They demonstrated experimental evidence that psychological and social satisfaction play a fundamental role in the benefit of eco-friendly VMDs leading to positive store characteristics. Such results indicate that social information results in positive psychological and emotional satisfaction, which consequently leads to a positive view of the store for retail customers.
Currently, VMD studies of the influence of brain activity from a visual stimulus has become an important research issue. To investigate the activated brain region when viewing positive and negative stimuli, Zhang et al. [5] used functional magnetic resonance imaging (fMRI) and reported that the local connections in the brain increased while viewing affective pictures. In the positive emotional network, a positive affect score was correlated with the betweenness value of the right orbital frontal cortex (OFC) and the right putamen, while in the negative emotional network, a negative affect score was correlated with the betweenness value of the left OFC and the left amygdala. Local efficiencies in the left superior and inferior parietal lobes were correlated with the subsequent arousal ratings of the positive and negative pictures, respectively. Metereau et al. [6] found that the activities in the medial OFC and the ventromedial prefrontal cortex (PFC) were correlated positively in terms of the expected values of the cues in anticipation of rewards and punishments. This finding indicates that the medial OFC and the ventromedial PFC encode a generally unsigned anticipatory value signal, regardless of the valence of anticipation (positive/negative) and type (appetitive/aversive events). Bercik et al. [7] investigated the impact of color illumination (an essential factor in food shoppers' perception) using EEG. Schmeichel et al. [8] found that exercising (vs. not exercising) self-control increases left frontal cortical activity during picture viewing, particularly among individuals with a relatively more active behavioral activation system than behavioral inhibition system, and particularly when viewing positive pictures. A similar but weaker pattern emerged when viewing negative pictures. Using fNIRS to measure the brain hemodynamic responses (HRs), the finding from Kreplin et al. [9] indicated that the right PFC was involved in positive evaluation of visual art that may be related to the judgment of pleasantness or attraction. This result suggests that emotional processing takes precedence over digressing viewing discipline during visual deliberation of art. In this case, the emotional nodule was brought to the beginning because of the participant's search for a personal association with the art images. Hence, participants relied on their own judgement and personal relationship with the art to make a subjective evaluation of the price.
In this paper, the HRs evoked by viewing positive and negative pictures of four different store categories (menswear store, womenswear store, underwear store, and sportswear store) are measured. The obtained results are portrayed in brain activation maps using the t-values of the measured HRs and the desired HRs [24]. Two-class and four-class linear discriminate analysis (LDA) [25] classifiers are used to distinguish the HRs from different picture stimuli, for which seven features (mean, peak, variance, skewness, kurtosis, t-value, and slope) are utilized.

Ethics statement
The experiment was conducted upon the approval of the Pusan National University Institutional Review Board. Written consent was obtained from all subjects prior to starting. The experimental procedure was conducted in accordance with the ethical standards described in the latest Declaration of Helsinki.

Participants
Twenty healthy subjects (age 28 ± 3, all right handedness, 10 males and 10 females) participated in this experiment. All of them were healthy with no previous history of neurological disorders. All participants were informed about the procedure and operation of the fNIRS system prior to providing written consent. They were asked to avoid body motion and remain relaxed during the experiment. When viewing the stimuli, the subjects were asked to focus on each picture presented on the computer screen and indicate either like or dislike.

Visual stimuli
The visual stimuli were projected onto a monitor screen (Samsung LED Model: LS24A300) in front of the participants. The viewing distance to the screen was approximately 50 cm. The size of screen is 23 inch (16:9) and the resolution is 1,980 × 1,080 pixels. Four categories of fashion stores (menswear, womenswear, underwear, and sportswear) were shown on the screen to prompt negative and positive images. One trial consists of a 15 sec picture stimulus followed by a 30 sec rest for individual categories as shown in Fig 1. One session includes an initial 60 sec baseline correction period and a 450 sec task period (5 trials of positive and negative images in random order). The entire experiment for 4 categories takes a total of 2,040 sec.

Stimulation picture selection
The pictures used in this work were initially attained from the Internet (from a collection of standardized photographic materials) by typing "menswear store display," "womenswear store display," "underwear store display," and "sportswear store display" into http://google.com [26,27]. Only those pictures with good resolution were first screened. To narrow down the number of pictures to be used for experiment, an online survey was performed, in which two hundred sixteen people participated. In this survey, students and workers (all participants are adults) with different majors and positions were invited, yielding a total of 216 answers. In evaluating the quality of the pictures, a 10 points scale was used (10 means the most positive, and 1 means the most negative), see Fig 2 for instance. Lastly, 10 pictures (5 positive, 5 negative) were selected for each category, which resulted in a total of 40 pictures for 4 categories (menswear, womenswear, underwear, sportswear). It is noted that the scores of the selected 5 positive pictures ranged from 8 to 10 on average, and those of the 5 negative pictures ranged from 1 to 3. It is also remarked that four out of 10 underwear pictures were from men's, and six out of 10 sportwear pictures were from women's.

Signal acquisition and processing
To measure brain activity, a multi-channel continuous fNIRS imaging system (DYNOT: Dynamic Near-infrared Optical Tomography; two wavelengths: 760 and 830 nm; NIRx Medical Technologies, NY) with a sampling rate of 1.81 Hz was used. A 16-channel probe configuration was placed on the prefrontal cortex [28][29][30]  To investigate the fNIRS data, the open source software NIRS-SPM [31] was used together with our own MATLAB codes (Mathworks, USA). The obtained intensity signals of nearinfrared lights were first converted to HbO and HbR concentration changes using the modified Beer-Lamberts law.
where ΔA denotes the variation in the optical density of the near-infrared light emitted at wavelength λ j , α HbX (λ 1 ) refers to the extinction coefficient of HbX in μMmm −1 , d denotes the differential path length factor (DPF) that is unitless, and l indicates the emitter-detector distance (in mm). To remove the physiological noise related to respiration and cardiac signals, a 4th order Butterworth low-pass filter with a cutoff frequency of 0.15 Hz was used [32]. Additionally, to eliminate drift in the signal, de-trending was carried out inside the NIRS-SPM software [33].

Regions of interest
To visualize the brain activity from fNIRS data, t-value maps are the most intuitive method. In this paper, the t-values were computed by using the robustfit function available in Matlab TM , which is an iterative reweighted least squares estimation algorithm with a bisquare weighting fNIRS-based investigation of VMD for fashion stores function. Let y j i 2 R N�1 be the measured data at the i-th channel for the j-th stimulus (i.e., 10 stimuli for one category), and let N be the data size per stimulus (in this study, N = 64 since 15 sec stimulus and 20 sec rest were adopted using a sampling frequency of 1.81 Hz). It is noted that out of the 30 sec rest period, only 20 sec data were used. Then, the linear regression model is defined as follows [24].
where X r 2R N×1 is the expected hemodynamic response upon viewing the given stimuli, 12R N×1 is a column vector made of 1's to correct the offset of the baseline, β is the unknown coefficient indicating the activity strength of the expected hemodynamic response, and ε2R N×1 denotes white Gaussian noise. The following equations describe the steps to obtain the t-values.
whereb j i indicates the estimate of b j i and stats refers to the statistical data including t-value, p-value, standard error (SE) of the estimated coefficient. The brain activity-related coefficient was related to the t-value to determine if its value was statistically greater than the t-critical (t crt ) value. The t crt value depends on the degrees of freedom (i.e., 63, which is N-1), and it is 1.67 in this case. If the shape of the HbO response is closer to that of the expected HR (y r ), the t-value is high. If the HbO response is different, the t-value is low (or negative). If a channel with the computed t-value is greater than t crt , the channel is considered active. After obtaining all the t-maps from the 4 categories, the regions of interest (ROIs) [34,35] for individual categories are investigated through the maps. The ROI denotes the area where the t-value(s) are higher than t crt .  Fig 6A, 6B and 6C, the ROIs when viewing positive/negative displays from the four categories are clearly different. For further analysis of feature selection and classification, only the active channels from each ROI were used because using the signals only from the associated ROI (rather than using the data from all channels) results in better classification accuracy [36,37].

Feature extraction and classification
In this study, the averaged HbO signals from the active channels were considered for classification [38,39] (see Fig 7 and Fig 8). For features, the following seven statistical signal properties (mean, peak, variance, skewness, kurtosis, t-value, and slope) were considered [22,40].
• Mean: the average signal value (for 20 sec from the start of the display), • Peak: the maximum value of the selected window (for 20 sec), • Variance: a measure of the signal spread (for 20 sec), • Skewness: a measure of the asymmetry of the signal around its mean relative to a normal distribution, fNIRS-based investigation of VMD for fashion stores • Kurtosis: a measure for the sharpness of the peak relative to a normal distribution, • t-value: a measure of a test statistic for the difference between the dataset and the expected HR, • Slope: the value that fits a regression line to the given data set.
To enhance the classification accuracy, a 20 sec window signal was used rather than using the entire 45 sec task-rest period. The previous study [41] used a 2-7 sec time window for a 10 sec stimulus for classification purposes of brain-computer interface (BCI) based on two features: mean and slope. However, in this study, we applied a different stimulus period (15s) with different features selection. Therefore, a 20 sec time window is selected for further classification. Additionally, the feature values have been rescaled between 0 and 1 using the following equation.
where z2R n , z 0 is the rescaled value between 0 and 1, min z is the smallest value, and max z is the largest value. For efficient classifications, several classifiers have been discussed in the literature [42][43][44][45][46][47][48]. Here, we used an LDA classifier to verify the HbO signals. fNIRS-based investigation of VMD for fashion stores

Results
Fig 5 shows all brain activation maps from 20 participants for menswear, womenswear, underwear, and sportswear by projecting positive and negative pictures of store displays and whose data were obtained from the prefrontal cortex. There is some similarity across the subjects for a given category. Therefore, using the data from the ROI of each experiment, the averaged brain activation maps for the 20 subjects were obtained as shown in Fig 6A. In Figs 6B and 6C, the gender-based brain activation maps are drawn.
In the menswear experiment shown in Fig 6A, channels 1 , 5, 8, 10, 11, 12, 13, 14, 15, and 16 were activated when viewing positive displays, and channels 12, 14, and 16 were activated when viewing negative displays. In the womenswear experiment, channels 3, 6, 8, and 14 were activated when viewing positive displays, and channels 1, 9, 10, 13, 14, and 15 were activated when viewing negative displays. In the underwear experiment, only channel 16 was activated when viewing both positive and negative displays. In the sportswear experiment, channels 2, 5, 10, 11, and 14 were activated when viewing positive displays, and channels 1, 2, 7, 12, and 16 were activated when viewing negative displays. Figs 6B and 5C present the male and female PFC activation maps. For male subjects, all channels with MP display stimulation were activated. Channel 8, 9, 10, 11, 12, 16 were activated by UP displays. In other stimulations, a few channels were activated (WP: channel 8, SP: channel 11, MN: channels 12 and 16, WN: channels 13 and 14, UN: channels 8 and 10, SN: channels 10 and 12). For female subjects, MP, WP, WN, and SP showed a large number of activated channels. Conversely, no channels were activated in UP, and only channel 16 was activated in UN. Based on averaging the activated fNIRS-based investigation of VMD for fashion stores channels, either among all subjects or by gender, through the intuitive observation a significant asymmetry of the t-maps is shown in the prefrontal cortex.

Fig 7. Comparison of the averaged HRs of positive and negative displays (the x-axis indicates the time series, in which 0-15 sec is the stimulus period and 16-35 sec is the rest period; the y-axis indicates the magnitude of HbO).
https://doi.org/10.1371/journal.pone.0208843.g007  The following findings can be summarized as follows. i) For menswear, the peak HbO values from positive stimuli were higher than those of negative stimuli. ii) In contrast, the peak value of positive displays of underwear and sportswear were lower than that of negative displays. iii) However, peak values of positive/negative displays of womenswear were not distinguishable. Overall, out of four categories, the negative sportswear display was the most influential to the customer, showing the highest HbO response at its peak value. Fig 8 compares all the HbO responses of positive/negative displays from the four categories. We conclude that the order of hemodynamic activation levels for positive displays are sportswear, womenswear, menswear, and underwear. Table 1 shows the two-class classification accuracies of all feature combinations (averaged over 20 subjects). In individual positive and negative classifications, the highest accuracies are marked in bold (blue): 64.15% (menswear), 64.95% (womenswear), 69.12% (underwear), and 66.27% (sportswear). Based on these best accuracies, Table 2 lists the subject-wise results of two-class classifications between positive and negative displays, in which good/bad looking underwear stores were classified most accurately, at 67.04%, and menswear stores were classified least accurately at 64.15%. Table 3 compares the four-class classification accuracies based on the average of 20 subjects with all feature combinations. In classifying four-store positive displays, the highest accuracy was 50.70%. In the case of negative displays, the highest accuracy was 51.10%. Table 4 shows the subject-wise four-class classification accuracies. The accuracy averaged across 20 subjects for the four positive displays was 50.68% (Sub.10 shows the highest accuracy of 76.67%). In the case of negative displays, the averaged accuracy was 51.07% (Sub.7 had 66.95%). Table 5 shows the gender-based two-class and four-class classification accuracies. For male subjects, the best classification category was underwear (71.38%), and the  fNIRS-based investigation of VMD for fashion stores fNIRS-based investigation of VMD for fashion stores classification accuracy for four-class positive displays was 51.69%. The order of classification accuracies was underwear (71.38%), sportswear (68.33%), menswear (58.50%), and womenswear (p = 0.069 is considered to be not statistically significant). For female subjects, the womenswear display showed the best classification accuracy (73.00%). The order was womenswear (73.00%), menswear (69.81%), underwear (64.71%), and sportswear (61.70%). For fourstore classification, negative displays had 52.34% classification accuracy, and positive displays had 49.67% accuracy. For the statistical test, the one-sample t test was applied to evaluate the accuracies based on its chance levels ( Table 6).

Discussion
In the present study, the authors used fNIRS to monitor brain signals for visual merchandising displays because the fNIRS machine is portable, silent, noninvasiveness, and low cost and allows an experiment comparable to fMRI. The adopted fNIRS technique has high potential as fNIRS-based investigation of VMD for fashion stores a brain imaging modality. In examining the brain activation, the maps and classification results were discussed in relation to the types of stores and genders. In this study, positive and negative displays from four categories (menswear store, womenswear store, underwear store, and sportswear store) were investigated. Gender-based classification was typically distinguishable in the categories of womenswear and sportswear. We found that the activated brain regions were asymmetrical across subjects (see Figs 5 and 6). This kind of asymmetry in functional responses in the prefrontal cortex was observed in other reports as well [5,8,49]. The results from Achterberg et al. [49] showed more activation in the right dorsal lateral PFC during negative social pictures. Our results are consistent with their report since the negative displays of three of four categories (menswear, womenswear, and sportswear) showed higher activation in the right PFC (see Fig 6A), which illustrates the asymmetry of brain responses. The activation channel number based on t-test analysis shows that the number of positive display channels is greater than that of negative displays in the menswear and sportswear cases. Some previous studies [50,51] reported that activation in the PFC is positively correlated with positive and negative art picture stimulation. Vessel et al. [51] found greater activation in the PFC for esthetically pleasing images. The gender-based t-maps were different (Fig 6B and 6C). This conclusion is consistent with previous research [52]. There are more active channels in menswear and underwear displays in the case of male subjects. Very few active channels were found in the womenswear displays. However, in the case of female subjects, womenswear displays produced more activated channels. Underwear displays yielded the poorest activation (see Fig 6C). The active brain region was asymmetrical in gender-based maps as well.
We applied an LDA classifier to the two-class classification problem of positive and negative displays (in which 21 different feature combinations were examined) and to the four-class classification problem with four stores. In classifying positive and negative menswear stores (see Table 1), the best accuracy was obtained from the combination of variance and slope, which fNIRS-based investigation of VMD for fashion stores was 64.15%. In the case of womenswear, the best accuracy was achieved from the combination of mean and peak values. In the case of underwear, the combination of skewness and t-value resulted in the highest accuracy, and lastly, in the case of sportswear, the combination of variance and slope gave the highest accuracy. It is noted that no common feature combination covering all four categories was found. However, as seen in Table 2, all averaged accuracies were higher than chance level (50%). Some subjects (subs. 11 and 15) had very high accuracy, up to 83.33%. Additionally, some subjects' data showed poor accuracy lower than the chance level, which are marked in parentheses.
In the case of four-class classification for the four different stores, the results are more meaningful because the accuracies are much higher than the chance level (25%). Furthermore, gender-based classification accuracies provide a direct indication of vulnerable categories: females are sensitive to womenswear displays, men are sensitive to underwear displays, and both males and females are less emotional with menswear and sportswear displays. Based upon classification accuracies and activation maps, it is concluded that gender-specific store displays should be pursued. Positive and negative displays for different products are conveyed differently to the customer by different emotions [52,53]. Such brain responses are recognized differently, as seen in the maps. From this view, the overall intensity of a map may be related to the customer's interest. Therefore, the brain response from the PFC related to emotion or pleasantness by viewing positive or negative store pictures can be scrutinized in the early stages of interior design. This study indicates that the examination of brain activation will help fashion marketing in the future.
Even though some conclusions like the above can be made, current research reveals that the relationship between VMD and brain signals is still vague. There are only a few papers reporting the effectiveness of brain-based approaches. This lack of research is not because VMDs do not influence consumers, but because the machines to measure brain signals do not provide enough data. Therefore, a new technique to investigate VMDs and brain activation should be developed for neuro-marketing applications.
There are some factors that influence classification accuracy. i) Signal strength variation is due to several factors, including skull scalp thicknesses and hair pigment. ii) In the present study, we measured the PFC as a brain region without hair, but a more emotion-related brain region needs to be investigated. iii) A subject's concentration level is another important factor to consider. During the experiment, some subjects reported that they felt sleepy and fatigued during the experiment closer to the end. Therefore, minimizing such subject-wise variation is another issue to consider. iv) Additionally, the surrounding environments also should be considered, such as machine noise. v) In this work, the pictures used in the experiment were obtained from the Internet. However, pictures attained from a planned store will provide a direct evaluation of the planned store. Finally, vi) cognitive effects between fixed and moving displays would be interesting [54].

Conclusions
In this study, we examined the possibility of using fNIRS as a brain-based tool to investigate the consumer's response for fashion store displays. The hemodynamic responses from the prefrontal cortices of twenty subjects were quite distinctive when viewing positive and negative store images. This finding implies that consumer responses to tentative plans of new stores can be evaluated in advance using fNIRS. Because new interior design costs a lot of money, the proposed brain-based approach can reduce total expenditure and construction time. In this study, even though hemodynamic responses to general (uncoordinated) store displays were examined, the hemodynamic responses of coordinated displays for a planned item would provide more design-specific evaluation for prospective consumers. Overall, consumer behavior analysis related to VMDs deserves to be evaluated using fNIRS. Future research should include a design-specific evaluation of VMDs focusing on store layouts, mannequins, light, points of purchase displays, colors, atmospherics, exterior displays, and others. As fashion changes quickly, to follow the directions of consumer behaviors, it is expected that we will see a rapid growth in interdisciplinary research and subsequent development of VMDs utilizing a brain-based approach in fashion marketing.
Supporting information S1 File. Dataset. The dataset for the four stores with positive/negative display stimulation (averaged over 20 subjects). To understand our work, fNIRS task-related data were provided as supporting information.