Creative problem solving and facial expressions: A stage based comparison

A wealth of research indicates that emotions play an instrumental role in creative problem-solving. However, most of these studies have relied primarily on diary studies and self-report scales when measuring emotions during the creative processes. There has been a need to capture in-the-moment emotional experiences of individuals during the creative process using an automated emotion recognition tool. The experiment in this study examined the process-related difference between the creative problem solving (CPS) and simple problem solving (SPS) processes using protocol analysis and Markov’s chains. Further, this experiment introduced a novel method for measuring in-the-moment emotional experiences of individuals during the CPS and SPS processes using facial expressions and machine learning algorithms. The experiment described in this study employed 64 participants to solve different tasks while wearing camera-mounted headgear. Using retrospective analysis, the participants verbally reported their thoughts using video-stimulated recall. Our results indicate differences in the cognitive efforts spent at different stages during the CPS and SPS processes. We also found that most of the creative stages were associated with ambivalent emotions whereas the stage of block was associated with negative emotions.


Introduction
The cognitive mechanisms underlying the creative process have been the focus of creativity research for decades [1]. Numerous sources of information, knowledge, skills, and emotions are utilized in different ways during the creative process [2]. Therefore, understanding how individuals take different approaches and learn to be creative becomes crucial to support their creative development.
Recent years have seen a rise in the number of studies in understanding the creative subprocesses and stages [3][4][5]. While most of the studies have focused on understanding the granularity of the creative process, it is essential to identify the creative act in its totality [6]. It is crucial that we evaluate the similarities and differences between the creative and non-creative acts to a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 convergent thinking involved in CPS instead of regular problem-solving. Finally, the CPS process involves reorganizing and restructuring the information, while the routine, non-creative process just recalls the information based on the existing knowledge. These differences, though, require further investigation as there is no empirical evidence that can be used as a comparison between CPS and the simple problem solving (SPS) processes.
Emotions and the creative process. Similarly, understanding the emotional processes that contribute to creativity becomes essential as they influence higher-level cognitive functions during the creative process, such as perception of the stimuli, judgment, decision making, and reasoning [16,17]. Phenomenology research into creativity and emotions shows a wide range of emotions occurring during the creative process across many domains. Several creative individuals, e.g., artists, musicians, scientists, designers, describe the feelings of mixed emotions such as joy, happiness, and pain during the long processes of working and reworking on the creative problem to realize the idea of the product or outcome [18,19]. Some long-standing views have examined the role of positive emotions in enhancing cognitive flexibility and creativity [8]. Others have explored the relationship between negative affects compared to neutral mood states in promoting creativity by helping individuals be more focused, critical, and determined in producing a creative outcome [20,21]. The third paradigm produced reliable evidence for ambivalent emotions association with creativity [22], where highly activated positive (excited) and negative (angry) states were linked to high creative engagement and deactivated positive (relaxed) and negative (discouraged) states were linked to lower creative engagement [23].
Notably, a few studies have also investigated the dynamics of emotions for different creative stages. Peilloux and Botella [24] for example, found that the creative stages of immersion, thinking, research, inspiration, and insight were associated with positive emotions and the stages of judgment, experimentation, and planning were associated with negative emotions. Most recently, Kumar et al. [25] examined the dynamics of emotions using the eight design subprocesses, where positive affects dominated the conceptual phase of the design process whereas negative affects dominated the embodiment phase of the design process.
Existing measures of the creative process. Protocol analysis. Protocol analysis has emerged as an important tool for behavior analysis during the dynamics of the creative process [3,5,26,27]. Concurrent verbal protocols or the think-aloud method involves participants verbalizing their thought processes while performing the task in real-time using their short-term memory [28]. In the retrospective protocol method, participants retrieve the information of the already completed task that has been stored in their memory. Both these methods have their own advantages and limitations. Gero and Tang [29] have emphasized that both these methodologies achieve similar results with respect to examining the process-oriented aspects of the creative design process.
Markov's chain analysis. Recent years have seen a rise in utilizing the Markov's chain analysis to study the creative subprocess sequence. A Markov Chain is a stochastic model for predicting, estimating, or guessing the result of an event based on the preceding state and its action [30][31][32]. Pringle and Sowden [5] compared shifts of the associative mode of thinking with the analytical mode of thinking during the processing of emotional input. Moreover, Kan and Gero [33] examined design protocols from the standpoint of the sequential order of the Function Behavior-Structure (FBS) processes using the first-order Markov chain for various design processes. Nevertheless, the Markov analysis can be performed on any coding scheme. Ergodic Markov chains have the property of achieving a single stationary state distribution as time progresses. Generally, it is represented as a row vector π whose entries are probabilities that add to 1, and given the transition matrix P, it satisfies where π is referred to as the equilibrium distribution of a chain. Computing a stationary state distribution makes it easy to statistically compare different distribution probabilities. Existing measures of emotions during the creative process. Most of the published research in measuring creativity and emotions together has relied primarily on diary studies and experience sampling methods [34,35]. Even though these methods offer greater ecological validity, they are costly, time consuming, and do not account for in-the-moment emotional experiences of the creators. A further step in this direction should include the use of various technological advances that enable the investigation of the physiological and behavioral measures of the creative process and emotions [36]. A recent study employed eight body postures to decode an individual's emotions during the creative design process using Kinect and a machine learning classifier [37]. Several articles have also asserted that facial expressions can be used to study emotions during the creative process [17,38]. Maybe the concept of measuring facial Action Units (AUs) can be utilized to map the corresponding emotions that arise at different stages of the creative process.

Present study
The research gap. Several articles have asserted that facial expressions can be leveraged to study emotions of the individuals during the creative process. To the best of our knowledge, no study has examined the dynamics of emotions during the CPS process using facial expression data.
Consequently, the two primary goals of this study are to examine the process related difference between the CPS and SPS processes and to capture in-the-moment emotional experiences of individuals using facial expressions during the CPS process.
Objective of this study. More specifically, the objectives of this study are -1. To compare the CPS and SPS processes using Protocol analysis and Markov chains.

2.
To classify different stages of CPS process based on automatic facial AU data combinations.

Participants
A total of 69 participants volunteered for the study, of which two opted out of the study, citing discomfort in wearing the headgear during the experiment, while the data of three participants were excluded due to the wrong orientation of the facial camera during the experiment. Thus, the study was carried out on a convenience sample of 64 adults randomly assigned to the CPS and SPS task groups. The CPS and SPS groups comprised 33 (

Materials
Tasks. Two different sets of tasks, CPS and SPS, each comprising of three different activities, were used in this study. The CPS task consisted of three different tasks (Fig 1), a drawing task [39], a writing task [40], and a structure making task [41]. The diversity of these creativity tasks allowed more room for creative engagement. They were chosen owing to the fact that working on a variety of creative tasks and switching between them enhances divergent and convergent thinking by reducing cognitive fixation [42]. The SPS task consisted of a drawing task, a writing task, and a structure-making task that were tailored to demand the same amount of effort as solving the creative tasks (Fig 2). The difficulty levels of the two sets of tasks were assessed beforehand by eight judges who rated the creative components (1 = not at all creative, 5 = very creative) and difficulty levels (1 = very easy, 5 = very difficult) of the tasks on a 5-point Likert scale. The two tasks did not differ in terms of their difficulty levels (t = −-0.323, df = 14, p = 0.751). In contrast, the CPS tasks were rated significantly higher on creativity as compared to the SPS tasks (t = 15.82, df = 14, p < .001).
Capturing the facial expressions. Participants' facial expressions were captured using a custom-made head-worn camera. The aim was to ensure that the frontal faceview was always in view of the camera, regardless of the participant's head movements during the task. An AKASO V50 pro sports camera was mounted on the headgear facing the participant at an approximate distance of 12.25 inches from their nose tip. The face was also illuminated by a strip of LED lights embedded in the headgear, as shown in Fig 3. The participants' facial expressions were recorded while solving the tasks for 30 minutes at the rate of 25 frames per second in 720p.
A manipulation check was performed to ascertain that the headgear did not interfere with the task performance by administering the same to 18 randomly selected matched participants who were not part of the main study. Nine participants were required to wear the headgear, while the remaining nine did not and were asked to complete the alternative usage test [43] where they were asked to provide 'five different usages of a brick' within two minutes. The zscores indicated no difference between those wearing the headgear and those who did not on fluency (z = −0.97, p = 0.39), flexibility (z = −0.225, p = 0.87) and originality scores (z = −0.94, p = 0.39).

Procedure
The experiment was conducted in the one-way glass room of the psychology lab at IIT Kanpur. This room was equipped with three dome cameras to record all the activities of the participants. Besides observing the participant through the one-way glass, the experimenter could observe the session from the observation room by controlling the camera movements using a control panel. The three cameras captured the whole process providing a close-up view, a 3/4 profile, and a backside profile (see Fig 4). The one-way glass room had only a working table, and a chair in addition to a table meant for the raw materials. After establishing the rapport, the participants were provided with the informed consent form upon reaching the venue. Upon signing it, they were instructed to wear the headgear and continue the interaction with the experimenter. This was done in order to make them accustomed to the headgear. As soon as the participant approved of working for 30 minutes on the tasks while wearing the headgear, they were instructed to proceed to the one-way glass room. The instructions were read aloud to them, and thereafter the CPS or SPS set was provided to them. The participants could start with any task and switch between tasks or work simultaneously on different problems. At this point, the participants were instructed not to read the three tasks until they heard the 1st buzzer. Doing so ensured that we did not lose out on any possible observation of the spontaneous occurrence of stages just after reading the task/tasks. The participants began working on the tasks following the first buzzer, and their sessions were recorded. While the dome cameras recorded the entire session, the head-mounted camera made separate recordings of the facial expressions. The participants were instructed not to think aloud to avoid any manipulation of the facial AUs. Participants were also periodically reminded of the time passed every 10 minutes through the buzzer sound. The total time for completion of all three tasks was 30 minutes. The task activity was followed by a short break of five minutes. Following the short break, the participants were asked to describe their thinking process during the activity based on some questions and instructions. Using the video stimulated recall, participants watched the whole video recording of their activity, where they were asked to reflect retrospectively on the process, speak aloud, and report their experiences. They were free to pause the video and speak about the process. This session was audiotaped. The total time to administer the whole experimental procedure, including the retrospective recall, was approximately 110 minutes per participant.

Data analysis
Segmenting the protocols. The video of facial expressions was synchronized with the timeline of the three dome camera recordings from the beginning to the end of the activity and the audio recordings with the video timelines. Adobe Premiere Pro CC was used to trim and synchronize videos and audios using the frame-matching features. Audio MP3 for each participant was mastered and boosted for better sound quality. This was followed by transcription of the verbal utterances on a word file. The transcribed verbal utterances were further  divided into small units/segments/moves based on one main verb per parse [26] along with the timestamps in the datasheet for each participant. The start and end of a verbal utterance were timestamped based on the time interval of the recorded video that the participants were referring to. After parsing the verbal utterances, video recording observations that were not reported as verbs or actions during the retrospective verbal utterance process were separately segmented and were added to the same excel sheets with timestamps. They can be categorized as task-related actions (e.g., reading the task description, sketching, making the prototype, arranging the raw materials, etc.) and other actions (e.g., standing up, relaxing the body, etc.). The segmentation of actions based on video observation became more robust by capturing details that were obscured by verbal utterances alone.
Coding the segments. Following the review of the segmented protocols, seven categories, namely preparation, ideation, illumination, evaluation, verification, production, block, or other were developed as coding scheme with the attributes of the different modes of thinking based on the literature review. Each segment was assigned to one category only. Table 1 summarizes the coding scheme based on which a total of 18,355 segments were coded for all the 64 participants.
When dealing with a large number of categories, coding requires a high level of reliability. Accordingly, after the lead author completed this exercise, an independent coder who was blind to the experiment but was proficient in coding schemes and creativity research coded 1476 segments randomly chosen from the CPS and SPS groups. Cohen's kappa was computed for these 1476 segments to determine intercoder reliability. We observed strong agreement in 82% of the segments (k = .816, p < .001) after correction for chance. In the event of disagreements between the coders, the coding categories were discussed between the two coders in order to categorize the protocols into different stages based upon different modes of thinking. Upon resolution of the discrepancy between the coders, the codes were adjusted to be consistent across all protocols for each participant.
Markov's analysis to study the transition between different stages. Based on the coding for the segments in the excel sheet, we calculated the probability of transition from one state to the next for each of the eight categories, providing us with an 8 by 8 matrix for each participant. Using these probability matrices, a stationary state distribution matrix with one row and eight columns was created using R studio for each participant to represent the differences in cognitive effort exerted at different stages of the task solving process. The analysis of Markov's stationary state distribution in this study was based on the work by Kan and Gero [33].
Transition state diagrams. The transition diagrams provide a visual representation of the transition of events with their probabilities. Based on the analysis of Jeong's work [54], the transition state from one stage to another, for each participant, the sum of these frequencies was taken, and a relative frequency was computed for each transition. This provided us with a single transition matrix containing 64 probabilities of the transitions for the CPS and SPS groups, respectively.
In these transition diagrams, the creative stages are represented by the nodes that are linked to the other nodes using directional arrows. These directional arrows represent the relative frequency from one stage to another and the width of the arrow represents the strength of the transitional probability. Numbers in the transitional state diagrams represent the probability of one stage being followed by another stage.
Facial expression analysis. Automatic facial action coding system. Facial Action Coding System [55] is a rigorous and psychometrically robust system that utilizes facial muscle movements to represent them as action units (AUs) to describe facial activity. Emotion-specific AUs or the combination of AUs can be found in EMFACS [Ekman & Friesen, 1983 [Unpublished]]. Several researchers across different domains have used automatic facial expression recognition software to characterize a specific set of emotions [56].
This study leveraged the OpenFace 2.0 [57] program to extract the facial action units from the synchronized videos of all the sessions. Based on comparing the performance of automatic facial expression recognition tools, OpenFace has been shown to be superior to the high-paid commercial softwares, e.g., Nodlus FaceReader, and Affectiva in detecting automatic facials. AUs with better accuracies [58]. Another advantage of OpenFace is that it is available in the open-source domain (https://github.com/TadasBaltrusaitis/OpenFace). This program is trained on different facial datasets and uses a linear kernel support vector machine to recognize individual AUs. The system recognizes seventeen specific AUs (1, 2, 4, 5, 6, 7, 9, 10, 12, 14, 15, 17, 20, 23, 25, 26, and 45). This study used only 12 AUs and their combinations that Table 1. Details of information category, example segments, and code explanation from participants' protocols, all coded into seven stages.

Protocol examples taken from the CPS and SPS groups
Code Category

Task-related action
Reads the Instruction sheet and the question paper

Preparation
One prepares to solve the problem by acquiring information, searching for answers, setting the problem, and find the solution approach [11,12] Task-related action Picks up the raw materials from the chair Verbal Utterance "I needed some time to think on this topic" Verbal Utterance "I saw this hut as going into that shape Ideation It's a generative process that includes the mental construction of various types of pre-inventive structures, that can become potential novel solutions [3,44,45] Verbal Utterance "you know this curve is like I felt that about making a violin" Verbal Utterance "so the first thing that to my mind was a kettle" Verbal Utterance "and while folding this sheet I suddenly came up with the 2nd task ka final answer here" Illumination It's the sudden appearance of an idea with immediate certainty [46,47] Verbal Utterance "this came very suddenly to me" Task-related action picks up another sheet and makes the cone Production Production corresponds to the elaboration and transformation of an idea into a physical entity through action [48,49] Task-related action Draws the 2nd element's rectangle Task-related action shades the 2nd element's lower right triangle Verbal Utterance "This base was somehow able to hold the structure still" Evaluation During this stage, the individual assesses the idea and the produced output [50,51] Verbal Utterance "Here I made it by mistake because this part confused me" Verbal Utterance "the sheet was very thin that is why my structure was falling down Task-related action redraws the eye Refinement It is the process, the creative goals or work is revisited, refined and reformulated [3,52] Verbal Utterance "So I redrew this part again to match this section" Verbal Utterance "I cut this part because it was longer than 81 cms" Verbal Utterance "here I was confused about making the star shape"

Block
Creative block is an inability to work due to external or internal factors [45,53] Verbal Utterance "but I was not sure what to make" Verbal Utterance "After much thinking, nothing was coming to my mind for this task" Verbal Utterance "I have never done such a task before Other Utterances or moments that were not related to the task.
Verbal Utterance "Haha, I don't know why I did that" https://doi.org/10.1371/journal.pone.0269504.t001 correspond to seven emotions. Details regarding the combination of AUs related to specific emotions used in this study can be found in [Ekman & Friesen, 1983 [Unpublished], 59,60]. For each participant, OpenFace generated a total of 45000 frames based on the video duration of 30 minutes. These 45000 frames (rows) contained continuous AU data from 0 to 5 (intensity of AUs) for 12 facial AUs, along with the timestamp and confidence level. In most cases, AUs is regressed with high levels of confidence, usually 98%. To increase internal validity, frames with confidence levels less than 95% were removed from the analysis. Support vector machine to classify the creative stages based on facial AU combinations. The dataset generated by OpenFace required further processing before classification, as the facial AU datasets were highly imbalanced (typically a result of an unequal distribution of classes within a dataset) with some AUs with 0 values indicating no movement of specific muscles, while other AUs displayed continuous values (above 0), indicating movement intensity of the muscles at different time intervals.
As part of the feature set in this study, seven different combinations of AUs are used to represent seven different emotions to predict the target class, comprising of six stages. During the data analysis, the datasets of 'illumination' and 'others' stage were discarded as 'illumination' occurs within a fraction of a second [61] and is, therefore, inaccessible by retrospective protocol analysis and 'others' did not contain tasks-related information. Further, by eliminating these stages, the model's performance improved by reducing the data imbalance issue. The final dataset for the CPS and SPS groups used for the SVM classification is presented in Table 2.
Before the model generation in SVM, we used a random undersampling approach to balance our facial AUs dataset using python libraries. We leveraged the imblearn.undersam-pling_sampling package in python programming language to undersample the data and generate a new dataset. The RandomUnderSampler function results in random selections of examples from the majority class. The parameter 'replacement' is set to false to prevent the repetition of the same example that requires deletion from the training dataset. This is followed using the fit_resample function to under sample the dataset and thereafter, train_test_split function has been leveraged to split into the training and testing dataset. 80% of the data were split into training sets and 20% as test sets.
We used a polynomial kernel function with degree 4 using Python to classify the stages, as other kernels did not provide high accuracy in predicting the stages based on different combination of AUs. Beyond this degree, the model became saturated. Moreover, our SVM model utilized the C parameter to avoid misclassification of the training data. Decision function shape is used as an ovo (OnevsOne) classifier to serve as a binary classifier for all pairs of classes, as well as the training utilizes the fit function to obtain a fitting of the model to the input training instances while testing utilizes the predict function to make predictions about the testing instances.

Comparing the distribution of stages during the SPS and CPS processes
Since the computed stationary state distributions of the stages were not normally distributed, the Mann Whitney U test was conducted. The results of the statistical analyses are summarized in Tables 3 and 4.
Statistical analysis revealed a significant difference in the distribution of stages during the CPS and SPS processes. Participants in the CPS group predominantly spent their cognitive efforts towards the stages of Ideation (z = −6.572, p < 0.05), Illumination (z = −2.696, p < 0.05), and Refinement (z = −3.437, p < 0.05).
Task A's ideation phase involved participants generating pre-inventive structures from incomplete elements provided to them, which encouraged them to diverge and consider alternate solutions. For example, Participant 01 or P01 (CPS) commented, "This curvy shape looked like a girl and a vase" whereas, P02 (CPS) associated the shape with surrealism and reported, "It looked like a Dali painting." The majority of participants for Task B expressed ideas based on the combination of the previously stored schema for having common festivals as interaction points between two countries that could facilitate cross-cultural exchanges. For example, P18 (CPS) reported, "we can have movies like Bajrangi Bhaijaan which will bring more unity among each other" while P12 (CPS) reported, "then I thought about the food as a medium to connect these two countries." During Task C, most participants were interested in creating a strong and stable foundation for the structure. For example, P28 (CPS) reported, "to fir I got the idea about making a strong base that will hold the tower," and P07 (CPS) reported, "I will have to make many legs where this long heightened thing can stand." In the illumination stage, some of the participants credited the emergence of insight to random associations during idea generation that somehow emerged spontaneously as an appropriate solution. For example, P02 (CPS) during Task A reported, "you know when I saw this, I was thinking of many things, and immediately this idea suddenly came." Some participants A substantial number of enhancements were made during the refinement stage to produce a better output. Many participants returned to the previous tasks and refined their work as they worked on the current task. Since they were occupied with other tasks, they were more likely to incubate, evaluate their previous responses, and then make changes.
Participants in the SPS group spent majority of their cognitive efforts during the Preparation (z = −6.878, p < 0.05) and the Production stage (z = −5.914, p < 0.05). The preparation stage required a considerable amount of concentration and attention. Task A demanded considerable focus on the deliberation of mentally converting the shapes. For example, P06 (SPS) reported, "these shapes were simple but required a lot of attention as you need to think the reverse of it" while P12 (SPS) reported, "I became habituated with the drawings and the conversion thing." During the preparation stage of Task B, participants worked on gathering the information from the graphs, translating the data, and arriving at relevant conclusions. For example, P15 (SPS) reported "these graphs were simple but I was reading them again and again to find the connection". Similarly, during Task C, P13 (SPS) reported, "so I read these instructions one by one and it required focus." During the production stage, participants spent their cognitive efforts in creating or transforming their ideas or thoughts into a tangible solution. In task A, for example, participants focused most of their efforts on implementing their ideas through sketching on the grid Table 3. Mean values of the stationary state distribution of the creative stages between the CPS and SPS groups. answer sheet. We observed an average increase in participants' drawing speed after the 6th to 7th element. As participants became familiar with the grid and the reference sheet, their drawing speed increased. During the task B production stage, participants spent their efforts in writing their conclusions after making sense of the data. During Task C, participants tried to develop the structure prototype based on the instructions provided. It is interesting to note that some participants spent considerable time unraveling the adhesive tape, while others cut multiple strands of tape and stuck them to the table for convenience and time savings.

Stage transition diagrams between different stages
As illustrated in Figs 6 and 7, the participants of CPS group expressed more thought sequence from the production to the ideation stage (57%) than the SPS group (1%). As participants developed the idea they produced them, and then immediately returned to the mode of ideation. By generating the outputs, the participants were able to generate a greater number of ideas, for instance, P03 in the CPS group stated-"I was doodling without much thought and started to visualize it when doodling". This was followed by transition from preparation to preparation, which was 57% for the CPS group and 40% for the SPS group. Consequently, participants in the CPS group transitioned rapidly from the preparation-to-preparation stage to gather relevant information before moving on to the ideation stage. This is an important step as it is said that preparation requires 99% of perspiration which leads to 1% of inspiration [62].
For the CPS group, the transition from illumination to ideation was 65%, whereas there were no such instances in the SPS group. The reason for this is that the tasks in the SPS group did not require illumination activities, whereas in the CPS group, when the idea suddenly emerged, participants expected it to be developed further. For example, P06 in the CPS group reported-"the idea that came instantly was the biryani competition" (illumination) and then immediately reported-"This was funny idea and I thought to develop it further" (ideation). More events of transition from production to preparation (82% vs. 12%), preparation to the production (50% vs. 19%), others to preparation (49% vs. 33%), illumination to preparation (100% vs. 5%), refinement to preparation (50% vs. 22%) happened for the SPS group than the CPS group, respectively. As part of the SPS group, it was necessary to examine sample sheets (preparation) to produce appropriate outputs. The results indicate that the preparation stage was most likely to occur for the SPS and was at the center of most of the transitions. It is likely that this result is due to the nature of the tasks where the participants spent most of their time collecting information and setting up the problem.

SVM classification of stages based on facial AU combinations
For selecting the best separation between classes, the highest possible accuracy scores for each stage were considered. Among all the stages, only the target class block was classified with the highest accuracy scores (above 95%). The remaining stages were classified with moderate accuracy scores. The results are presented in Table 5.
Preparation stage and associated emotions. The stage preparation observed moderate accuracy scores for the ambivalent emotions, happiness (67.65% and 64.73%), and fear (68.95% and 66.52%) during the CPS and SPS process respectively. Positive emotions at the beginning of the task led some of the participants to approach the problem with high levels of motivation. For example, P01 (CPS) during task A reported-"this question was very interesting and I was just staring at them to understand the question" and P15 (CPS) reported,-"I was totally excited so immediately I started counting the shapes." During task B, P21 (CPS) reported -"I like literature and writing and I can use my knowledge so this was really exciting," and P03 (CPS) reported-"this question was interesting as well as tough." During task C, P16 (CPS) reported-"this task was full of creativity and fun," and P17 (CPS) reported-"I thought why not first solve the 3rd task as it looks very interesting". For some of the participants, preparation was accompanied by a sense of apprehension after appraisal of the task about failing to complete the task. For example, during task A, P27 (CPS) reported-"I was worried because I was losing on a lot of time in searching for the pen to write"; P10 (CPS) reported-"I was a bit skeptical and stressed when I read this question." Nevertheless, some participants also experienced negative emotions when attempting to complete the tasks on time. For example, during task A, P06 (CPS) reported-"but these shapes were complex and I got scared to finish it on time."

PLOS ONE
During task B, P13 (CPS) reported-"basically this was a political question and I realized that this task was going to take a lot of time." During task C, P21 (CPS) reported-"I was also thinking about how much time I have left for this task. Participants in the SPS reported negative feelings in response to the focus and attention demanded by the tasks. For example, P15 (SPS) during task A reported-"this one demanded focus and attention and I had to be careful in converting the shapes"; P31 (SPS) during task B reported-"so I was also calculating the values in my head after looking at these graphs, that's why it was taking time for me to see and then write

PLOS ONE
Capturing Creative Cognition and Emotions Table 5. Accuracy scores of various stages for 7 emotions for the SPS and CPS groups. the answers"; P17 (SPS) during the task C reported-"this kind of task is like want you to be very attentive because you will have to measure everything and make them." Alternatively, the other perspective could be perhaps due to the nature of the tasks in the SPS group that were not exploratory and did not demand high divergent thinking abilities. For example, P06 (SPS) reported during task A-"there was nothing much to think about this task"; P31 (SPS) reported-"actually this was a complicated problem due to the shapes and you can't come up with ideas here."

Anger
Ideation stage and associated emotions. The ideation stage also involved ambivalent emotions. These observations are in accordance with previous findings [8]. Moderate accuracy scores were observed for happiness (73.81% and 70.19%) for the CPS and SPS groups, respectively. For fear, accuracy scores were significantly higher (76.05%) for the SPS group than for the CPS group (56.90%). Perhaps the gap between ideation and implementation of those ideas during the SPS process seemed farfetched due to the nature of the tasks that resulted in such a difference. Some of the participants dedicated their cognitive efforts on making richer associations with positive emotions, thereby facilitating flexibility. For example, P27 (CPS) during task A reported-"so at first glance I thought that it was something like Picasso's work and I was like very happy that I have direction now"; P02 (CPS) during task A reported-"it looked like a Dali painting somehow from which I got inspired." During task B, P15 (CPS) reported-"I felt really happy that I came up with this idea of one nation one flag." While during task C, P06 (CPS) reported-"I was thinking of making something similar to the Eiffel tower." Perhaps, for some of the participants, not being able to transform pre-inventive structures into a concrete mental representation or idea caused fear during ideation. For example, during task B, P33 (CPS) reported-"I was scared if I will be able to complete this vague idea of UN ka intervention with Pakistan." In the SPS group, negative affects encouraged participants to develop self-efficacy necessary to reach the right solution, resulting in positive affect. The participant initially struggled with converting a few shapes in Task A, but later came up with an idea for transforming the shapes. For example, P02 (SPS) stated-"I was not sure here but then I came across a rough idea of converting these arcs into semi triangle that made me feel lighter".
Block stage and associated emotions. Block is the only stage that observed very high accuracy scores for negative emotions-anger (96.04% and 97.26%), fear (94.67% and 97.26%), sadness (95.01% and 95.95%), and disgust (96.90% and 95.57%) respectively for the CPS and SPS groups. This makes sense as there are several factors that can be attributed to an impasse, for example, creative fixation, fear of failure, or lack of faith in oneself [63]. For example, during the task A, P1 (CPS) reported-"this was a sad moment because I couldn't think of anything at this moment"; P2 (CPS) reported-"I was not able to come up with an idea right away which pissed me off"; P11 (CPS) reported-"It was very irritating that why I am not able to move forward". During the task C, P19 (CPS) reported-"this paper was frustrating me because there was nothing I can do with this". It is interesting to note, despite this, that some of the participants learned to accept these negative emotions, helping them to persist and complete their tasks, while others switched to an alternative activity.
Several participants in the SPS group were stuck on task A's shapes, specifically the third element with two semi-circular shapes and the eleventh element with a star shape. For example, participant P01 (SPS) during task A reported-"at this point I was confused and irritated about converting this circle or not"; P02 (SPS) reported-"then at this figure the confusion happened and I was fearing that I could make this one wrong"; P30 (SPS) reported-"this was not triangle not circle both which was frustrating to decode". In addition, some participants had difficulty rolling an 81-cm paper in Task C and encountered an impasse. For example, P04 (SPS) reported-"First I was not satisfied and I did not understand what I have to do to make it 81 cms"; P31 (SPS) reported-"even after trying so hard it was not 81 cms which was irritating"; P22 (SPS) reported-"I was like there was no option to make the length bigger which was very sad". Participants in both the groups either abandoned the tasks immediately and switched to the other task or worked on the task for some time.
Evaluation stage and associated emotions. The evaluation stage observed moderate accuracy scores for ambivalent emotions, happiness (72.25% and 74.09%), pride (64.63% and 68.08%), fear (60.74% and 67.83%) and stress (59.72% and 68.58%) respectively for the CPS and SPS groups. This state, which requires focused attention to determine whether the idea is weak, is an emotionally taxing process. For example, P19 (CPS) during task C reported-"but then this was not standing which gave me a lot of stress." P05 (CPS), during task A reported -"but something was not very right here and it was very disappointing." Similarly, in the SPS group, P01 (SPS) during task A reported-"in between I was checking whether I am doing it correctly as I was little worried"; P11 (SPS) reported-"I was having a lot of problem and stress in drawing this circle on this line". For task C, most of the participants utilized their cognitive efforts on measuring the length of the rolled sheet. For example, P13 (SPS) reported-"mine was coming a bit long which was mind boggling"; P27 (SPS) reported-"when I realized this one to be 10 cms short I was like shit. . ." It is interesting to note that some participants appraised their results positively which perhaps resulted in experiencing happiness. For example, P03 (CPS), during task A, reported-"I just looked at this thing and it came out to be nice." P22 (CPS) during task C, reported-"here, when it stood na, I was very happy"; P06 (CPS) reported-"here I was happy that this structure could stand". Our findings are consistent with previous research that showed designers experience high levels of arousal and positive emotions during the verification stage [64].
Refinement stage and associated emotions. For the state of refinement, the CPS and SPS groups scored higher accuracy scores for ambivalent emotions-happiness (85.81% and 83.49%), pride (81.58% and 82.13%), and stress (77.19% and 81.19%) respectively. During this stage, additional layers of work, refinements, and finishing touches were added. Some participants were successful in resolving the issues and thus experienced happiness. For example, P01 (CPS) during task A reported-"I felt extremely happy when I made more flames to provide a completeness in this photograph." Nonetheless, there were some participants who struggled to resolve the issues, which resulted in additional time and stress. For example, P14 (CPS) during task C reported-"even after spreading these legs, my idea did not work that tensed me." P19 (SPS), during task A, reported-"I shaded the rest of the shapes to be free from tension"; P18 (SPS) reported-"I was having problem in redrawing this circle again and again." Production stage and associated emotions. This stage observed moderate accuracy scores for ambivalent emotions-happiness (74.35% and 74.09%), fear (60.55% and 65.71%), pride (70.29% and 71.31%), and stress (60.71% and 71.83%) for both the CPS and SPS groups respectively. At this stage, all the tasks required the participants to produce tangible outcomes to implement an idea they had already conceptualized. Perhaps, those who were successful in transforming their ideas into successful outcomes felt happiness and pride. For example, during task A, P17 (CPS) reported-"once I had this face idea, I was successful in making it here." During task C, P02 (CPS) reported-"I was enjoying making firm base out of this newspaper sheet." During task A, participant P18 (SPS) reported-"I kept drawing like this because I was enjoying it." During task C, P22 (SPS) reported-"I folded the sheet into exactly three parts and woaahhhh!!". Perhaps, some of the participants may have realized that they could not produce a quality outcome and, as a result, might have experienced fear and stress. The realization of this ongoing process of production differs from the process of evaluation where the produced outcome or an idea is verified. It becomes difficult to capture the evaluation of the ongoing production process during the retrospective think-aloud protocol.

General discussion and conclusion
For a more comprehensive understanding of creativity, it is necessary to capture creativity act in its totality and the underlying emotional mechanisms. This work presents a novel method of capturing in-the-moment emotional experiences of the individuals during the creative process using the facial expressions data and a headgear setup. Previous studies have leveraged self-report scales and diary studies in examining the emotion influences on the creative work. This study is first to capture the emotional states associated with each stage during the CPS process using real-time facial action unit combinations and support vector machine classification. Moreover, this work also compared the CPS and SPS processes using the protocol analysis and Markov chains.
In order to answer the first research question (RQ1-what stages of the CPS and SPS require the greatest amount of effort from the individuals?), protocol analysis and Markov's analysis results revealed that the participants in the CPS group spent most of their cognitive effort during the stages of 'ideation,' 'illumination,' and 'refinement,' whereas the participants in the SPS group spent most of their cognitive effort during preparation and production. Perhaps, this difference is attributable the tasks assigned, as the tasks in the CPS group offered more opportunities for divergent thinking. In the 'ideation' process, participants spent most of their time deriving pre-inventive structures, although some participants were more successful than others in generating pre-inventive structures quickly and solving the tasks on time. Different components, such as task motivation, and skillsets impacted the quality of output produced for a specific task as not all tasks were attended with equal efforts. The same is true for the other stages. For example, most participants showed more interest in solving tasks A and C than task B for the CPS group. Although all these tasks encouraged divergent thinking, drawing and structure-making tasks were considered more playful and enjoyable than the written tasks. Task A and Task B were given priority over task C for the SPS group. This requirement arose from the requirement for greater focus and attention in reading the instructions and then developing the prototype. Furthermore, all the tasks in the SPS group involved high attentional demands, which meant some participants were able to focus more and produce better results. As a result of this study, one of the approaches proposed by [2] to differentiate between the CPS and SPS is supported: these two processes entail the same stages, but the variation is in the level of execution at each stage. These results provide the first empirical evidence in understanding the difference between the CPS and SPS process at a micro-level and lend support to studies suggesting that the creative process must allow higher degrees of divergent and convergent thinking [6]. There were also differences in transition state diagrams between the CPS and SPS processes. The CPS group experienced more transitions from 'preparation' ! 'preparation,' 'illumination' ! 'ideation' and 'production' ! 'ideation.' The SPS group, on the other hand, experienced more transitions from 'production' ! 'preparation,' 'refinement' ! 'preparation,' 'illumination' ! 'preparation,' 'others' ! 'preparation' and from 'preparation' ! 'production. ' To address the second research question (RQ2-What kinds of emotions occur at different stages of the creative process?), the exploratory part of this study investigated the in-themoment experience of emotions during the CPS processes by fulfilling the research gap posited in the literature regarding the use of automatic facial expression analysis in researching the role of emotions in creativity. Six stages were classified using SVM based on seven facial expressions (or facial AU combinations). Although ambivalent emotions were majorly present during most of the creative stages, considering the results, specific emotions were found to be present for different stages where the negative and positive emotions have a predominance presence. Happiness and pride were found to be predominant during the stages of evaluation, refinement, and production whereas negative emotions were predominant during the stage of block. These results are unique and corroborate with the previous findings. During the evaluation and refinement process, one may see a silver lining that induces happiness whereas, during a creative block, the idea becomes fixated, and the emotions become predominantly negative.
The findings of this study are consistent with theories relating to the role of ambivalent emotions in creative work [18,22,25]. The appraisal of the tasks and the appraisal of one's performance during the completion of the tasks caused most participants to experience both positive and negative emotions at different time intervals. Furthermore, the time pressure to finish the three tasks on time allowed some participants to stay motivated and experience ambivalent emotions to be more creative, but for others, this time pressure acted as a guillotine to their creative performance. On the contrary, the stage of 'block' was highly dominated by negative emotions, and high accuracy scores were observed for anger, fear, sadness, and disgust. This makes sense as the previous research has indicated that a diverse set of negative thoughts is encountered during the resolution of this state, e.g., loss of focus, fear of failure, worry, depression, etc [53]. Interestingly, we did not observe much difference in the emotional profiles for each stage between the CPS and SPS processes, even though CPS and SPS have different distributions of stages.

Limitations of the study
Unlike any other study, this study has its limitations. We did not postulate a dichotomy between the CPS and SPS processes nor examine the different categories of stages between the two processes. Segmenting and coding different categories of stages for the CPS and SPS processes might pose a construct validity issue since the literature fails to distinguish stage-based differences for the CPS and SPS processes. In addition, conducting a statistical test becomes difficult based on the difference in categories of stages between the CPS and SPS stages. Another limitation of the study is its lack of ecological validity. The creative process in the real physical world is influenced and mediated by many factors, and thus, the laboratory-based study may yield a different result from those in the natural situation [65]. As of now, the prospect of recording and analyzing moment-to-moment facial expressions in a real-life situation seems distant at this moment. Additionally, this research was constrained by the retrospective protocol analysis, which lowered the accuracy of the in-the-moment CPS and SPS process analyses. Nevertheless, concurrent verbal utterances could have introduced bias in the data due to redundant facial muscles movements. Furthermore, the small sample size didn't permit us to account for the differences between the creative performances of the high and low scorers.

Implications of this research
Despite these limitations, this research can have significant practical implementations in creativity research. Novel methods and tools could be developed to train individuals to recognize and regulate their emotions during the creative process to create a deeper understanding of what emotions to expect at each stage and implement effective strategies to manage those emotions. The process of ideation could, for instance, be enhanced by inducing positive emotions and allowing flexibility. Similarly, creative individuals can also be trained to regulate their negative emotions during the creative block to stay motivated and find ways to solve the problem. Few studies have developed personalized gaming experiences based on different emotions. E.g., the game alters its difficulty level by sensing negative emotions in gamers through recognizing facial expressions in real-time to help them stay motivated throughout the gameplay [66,67]. Consequently, the results of this study may serve as a baseline for developing creative support systems for various professionals. For instance, an individual could receive personalized visual cues or affective cues in real-time based on the changes in their facial expression captured by their personal device to enhance their creative performance. This claim may be considered immature at this point due to the limitations of current technological advances, as well as laws and policies about the monitoring of sensitive facial information. Furthermore, designing such affective solutions must demonstrate excellent reliability and validity to be generalized.

Future work
There is a need for further research regarding creativity and facial expressions. Replication of our study with a larger sample size among various ethnic groups may demonstrate the similarities and differences with our findings. Future work could include longitudinal studies that capture facial expressions at different stages of the creative process to clarify the relationship between creativity and emotions. Moreover, the benefits of such studies could also improve ecological validity by determining the causal connection. A further step could be to triangulate this study by utilizing other technological interventions, e.g., EEG and biofeedback sensors. In addition, different automatic facial expression recognition algorithms could be used to crossvalidate the accuracy scores obtained in this study.