1.1 Requirements

To address more critical domains, such as psychotherapy or the service industry, AI applications should describe, understand or detect more emotions in real life. For instance, one can imagine training to express self-confidence before a job interview or dealing with anxiety in a therapeutic context [14]. Thus the FEM-based process needs to adapt to more various and fine-grained requirements.

• Flexibility. The process should be capable of personalizing facial expressions for observers. That is, generating the desired expression that meets the need of the observer. As in most literature, we refer to the person whose face is being manipulated as an “actor”, and to the person supervising/designing the manipulation as an “observer”. In case they want to personalize their own facial appearance, the actor and the observer can be the same person.
• Exhaustiveness. The process should be applicable to any expressions, and not limited to basic facial expressions. This can be complex emotions or social attitudes (e.g., self-confidence) as well as more general expressions (e.g., how do you want to be seen during your job interview?).
• Expertise-free. The process should be controllable by any observers in a precise and consistent manner without the need for expert knowledge (e.g., FACS-certified coders, and knowledge in affective computing).

1.2 Contribution

In this paper, we propose a novel interdisciplinary approach (see Fig 1) to personalize facial expressions by combining Generative Adversarial Networks (GANs) [15] with a technique inspired by the cognitive sciences, psychophysical reverse correlation [16, 17]. Reverse correlation is both an experimental procedure and an analysis technique able to extract facial prototypes, or ‘mental representations’ of any given desired facial expression from the observer. In other terms, our approach identifies which attributes need to be modified to better fulfill the need of the observer, resulting in the generation of personalized facial expressions. This can meet the requirement of Flexibility.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. The framework of our approach to personalize facial expressions.

We combine the recent deep learning technique, i.e., Generative Adversarial Network (highlighted in blue), with psychophysical reverse correlation, a recently emerging technique from the cognitive sciences (highlighted in green). We employ the same GAN to extract personalized control parameters (i.e., mental representation) and to personalize facial expressions.

https://doi.org/10.1371/journal.pone.0290612.g001

Differing from typical GANs that can manipulate facial expressions, our approach has the following strengths.

• Exhaustiveness. One can address any emotion or social attitude, with no need to build a dedicated training database for each emotion or social attitude. Rather, we use local attribute to manipulate facial expressions, which can cover a wide range of local facial movements, in order to reproduce the observer’s prototype, regardless of what the prototype’s target expression is.
• Expertise-free. No expert knowledge in affective computing or certified FACS coder [13]) is needed to create the personalized prototype since our approach only requires the observer’s perception (i.e., subjective judgment) rather than the observer’s expertise.

Conversely, differing from typical reverse correlation approaches which use 3D virtual avatars [18], using FEM techniques (such as GANs) allows manipulating real faces (2D pictures), thus providing an easier and more intuitive way to edit facial expressions. In detail, we use the same tool (GAN, for instance) twice: first, to generate experimental stimuli for the reverse correlation procedure, and then to apply the resulting prototype for the manipulation. This can ensure that the manipulation is consistent with the mental representation of the observer.

Finally, the mental prototype extracted in the intermediate step (i.e., reverse correlation procedure) does not especially fit any so-called universal prototype. It is specific to the observer. To enhance the definition of facial expression prototypes, we introduce below the concept of dominant and complementary action units to precisely describe facial expression prototypes.

1.3 Related work

Here, we briefly review the facial expression manipulation techniques (FEM) from computer science, and the reverse correlation procedure from cognitive science.

2.1 Stimuli generation

To generate input stimuli (random facial expressions) for reverse correlation, we can employ any tool that can control objective local attributes. Here, we choose GANimation [3] controlled by facial action units (AUs) [13] to synthesize random facial expressions (i.e., reverse-correlation stimuli). In this step, GANimation takes as input an image of the actor’s face (e.g. captured with an emotionally neutral expression) and a vector of AUs to create a deformed face (i.e., stimulus).

In terms of the vector of AUs, GANimation is capable of manipulating 16 AUs by activating or deactivating the corresponding AU. While simultaneously activating too many AUs typically will create visual artifacts. Therefore, we generate stimuli by only activating 3 AUs. Combining 3 out of 16 AUs, there can be possible AU vectors, where C is the mathematical combination function. For more details about GANimation, please see the literature [3] and S1 Appendix.

2.2 Perceptual experiment

The second step of our approach is the perceptual experiment. For each perceptual experiment, observers perform m trials. In each trial of the perceptual experiment, a pair of randomly generated stimuli is presented to the observer. Each pair of randomly generated stimuli is displayed only once. The observer is asked to choose which stimulus of the given pair best corresponds to the target expression (e.g., “which of these two faces looks happier?”). Note that for one trial, if we randomly select a pair of 3 AU-activated stimuli, there are possible combinations. For each perceptual experiment, the set of m trials is randomly selected from the trials.

2.3 Mental representation computation

Here, we obtain the mental representations by a two-step computation: dominant action unit computation and complementary action units computation. The purpose of the dominant action unit computation is to determine which action unit has a significant effect on the perception of the observer. We then determine the complementary action units, i.e., in combination with the dominant action unit, which action units also drive the perception of the observer.

We’ll use mathematical notation to avoid any ambiguity. To this end, we define Ω as all the trials within a perceptual experiment, where |Ω| = m. Note that in this paper, |.| represents the cardinality of the set. According to the activation or deactivation of a given action unit AU_i (i is the subscript number of AU), each perceptual experiment Ω can be divided into three subsets.

• Ω_{i*}: the subset of trials in which one of the paired stimuli has AU_i activated and another one has AU_i deactivated.
• Ω_{i}: the subset of trials in which both stimuli have AU_i activated.
• : the subset of trials in which both stimuli have AU_i deactivated.

Dominant action unit computation. We first define as the set of stimuli selected by the observer from the subset Ω_{i*}, and Φ_i as the set of all stimuli in which AU_i is activated. We then count P(i|Ω_{i*}) the proportion of the selected stimuli that have AU_i activated in the subset Ω_{i*}, i.e., how likely an activated AU_i is to drive the observer’s perception. (1)

Finally, we can determine the action unit AU_i with the largest proportion P(i|Ω_{i*}) as the dominant action unit denoted by AU_d. d is the subscript number of dominant AU. (2)

Complementary action units computation. Similar to the definition of Ω_{i}, we specify Ω_{d} as the subset of trials where both stimuli have dominant AU_d activated. We continue to divide subset Ω_{d} into three subsets according to the activation status of the non-dominant AU (denoted by AU_j, where j is the subscript number of AU).

• Ω_{d,j*}: under the premise that each pair of stimuli has AU_d (the dominant action unit) activated, the subset of trials in which one of the paired stimuli has AU_j activated and another one has AU_j deactivated.
• Ω_{d,j}: under the premise that each pair of stimuli has AU_d (the dominant action unit) activated, the subset of trials in which each pair of stimuli has AU_i activated.
• : under the premise that each pair of stimuli has AU_d (the dominant action unit) activated, the subset of trials in which each pair of stimuli has AU_i deactivated.

Since previously the dominant action unit has been determined, the complementary action units can not be the dominant one (i.e., ∀AU_j ≠ AU_d). We define as the set of stimuli selected by the observer from the subset Ω_{d,j*}, and Φ_j as the set of all stimuli in which AU_j is activated. We compute P(j|Ω_{d,j*}) the proportion of selected stimuli in subset Ω_{d,j*} that have AU_j activated, i.e. how likely the addition of AU_j to dominant AU_d is to drive the observer’s perception. (3) In practice, we limit the number of complementary action units by introducing a threshold T_q (to separate complementary AUs and non-complementary AUs). (4) Note that is the set of all the subscript numbers of complementary AUs.

The output of this step is the mental representation (i.e., personalized control parameters for facial expression manipulation). We construct the mental representation of the observer (as aforementioned in Stimuli generation, a 16-dimensional binary AUs) by activating the dominant action unit and all the complementary action units.

2.4 Personalized manipulation

Once the mental representation of the observer is extracted, we apply personalized manipulation on each frame. To be consistent with the mental representation and the final manipulation, we employ the same tool (GANimation [3]) for the stimuli generation and the personalized manipulation. To make the video compatible with GANimation (especially with the dimension of the face), we crop, align and resize the face of the actor in each frame.

2.5 Experiment setting

Implementation: GANimation model. We choose GANimation [3] as the tool to generate facial expressions by editing local attributes, namely action units (AUs) [13]. We use the code of GANimation released by its authors. All settings are unchanged. The input image and the output image are 148px × 148px. To crop, align, and resize the face, we employ OpenFace [42].

Implementation: Mental representation computation. As aforementioned, we determine a dominant AU for one emotion as the action unit that dominantly drives the observer’s perception. For the complementary AUs, we need to determine which AUs combined with the dominant AU have a significant effect on driving the observer’s perception. Therefore, we need to set a relatively high threshold T_q to eliminate most AUs with less significant proportions P(j|Ω_{d,j*}). Indeed, T_q = 50% corresponds to the situation in which, among each pair of AU_d-activated stimuli, the observer selects as many AU_j-activated stimuli as AU_j-deactivated stimuli. This means that AU_j carries no information content for this experimental task. To identify which AU is activated that positively influences the perception of the observer, the threshold T_q should be significantly higher than 50%. Considering that state-of-the-art prototypes [13, 18] have 2 to 5 AUs activated, to align with state-of-the-art prototypes, we manually set the threshold T_q to 80% for happiness, sadness and anger and 70% for self-confidence. Thus all the personalized prototypes can have 2 to 5 AUs activated.

Experimental protocol: Observers. Four observers (one female) participated (in October 2021) in the perceptual experiment, all relatively young (mean = 27.7yo) adults of three cultural groups: Brazil (1), China (2), and France (1), respectively denoted by observers #1 to #4. Only one observer had experience in affective computing, and nobody is a certified coder in Facial Action Coding System [13] or a psychologist. Each observer signed informed consent, and the experimental data were anonymous.

Experimental protocol: Perceptual experiment. The perceptual experiment aims to illustrate that our approach can personalize the facial expressions of a given emotion, even though this emotion is not available in existing deep-learning databases. Note that the purpose of the perceptual experiment is not to give extensive results or to discuss facial expression prototypes. We chose three basic emotions (happiness, sadness, and anger) that existed in deep-learning databases and one non-basic emotion (self-confidence) that is not explicitly available in existing deep-learning databases. Each of the four observers participated in four different experimental tasks to extract his/her mental representation of happiness, sadness, anger, and self-confidence. In related work using reverse correlation [10, 36, 37, 39], the average number of trials for the perceptual experiment of one emotion varies from 700 to 1800. For each experimental task, we decided that observers performed m = 840 trials. The question was fixed and unique, e.g., “Which of these two faces looks happier?” The order of the four experimental tasks was counterbalanced among observers, and all experimental tasks used the same actor’s photograph. It took about 40 to 60 minutes for one observer to complete a task. The time interval between experimental tasks was set to half a day. All experiments were conducted in a quiet room in the lab, using a custom computer graphic interface implemented in PsychoPy.

Ethics statement. Our work does not require an ethics statement, since the risk in our work is minimal. 1) The identity of individuals is not known. 2) There is no way to track them from the data in the database. 3) There is no social or physical risk. 4) The psychological risk is absent since we asked participants according to their perceptions. 5) All the observers and participants signed informed consent forms and all the data were totally anonymous.

3.1 Dominant and complementary AUs computation

Fig 2 details dominant and complementary AUs computations of each emotion (happy, sad, angry, and confident) from observer #2. See S1–S3 Figs for the computations from other observers. As mentioned in Mental representation computation, the proportion of each AU is computed based on the corresponding subset of trials (see Eqs (1) and (3)).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. The results for dominant and complementary AUs computation from observer #2.

Note that “Dom.” = dominant AU computation; “Comp.” = complementary AUs computation; “Conf.” = self-confident. In each chart, the proportion for each AU is computed based on the corresponding subset of trials. We highlight the dominant AU in red and the complementary AUs in yellow. The thresholds for the complementary AUs computation are marked by yellow dashed lines.

https://doi.org/10.1371/journal.pone.0290612.g002

The concept of dominant and complementary AUs contains more information about emotional prototypes than a list of activated AUs in the universal prototypes [13]. Here are our observations. Similar observations can be found in supporting information (S1–S3 Figs).

The dominant AU drives the observer’s perception. As defined in Mental representation computation, the dominant AU is the AU_i with the largest proportion P(i|Ω_{i*}), where i ∈ μ. As shown in the first row of Fig 2, the corresponding proportions exceed 80%. This means the observer has a significant probability to choose the facial expression that has the dominant AU activated.

We can observe the dependency between the dominant AU and the complementary AUs. For the dominant AU computation shown on the first row of Fig 2, the complementary AUs have much lower proportions than that of the dominant AU. For instance, in the histogram for dominant AU computation of “happy”, the proportion of AU6 being selected is just 52% (note that AU6 is later determined to be the complementary AU). That is to say in the following situation when the observer saw a pair of stimuli that one had AU6 activated and another one did not have AU6 activated; the observer #2 made a nearly random selection (i.e., 52% to select the stimulus that had AU6 activated and 48% to select the stimulus that did not have AU6 deactivated). This indicates that a single complementary AU can not drive the observer’s perception as much as the dominant AU. However, for the complementary AU computation shown on the second row of Fig 2, when the dominant AU is activated, the facial expressions that have the complementary AUs activated have a significant probability of being selected by the observer. For instance, in the histogram for complementary AUs computation of “happy”, the proportion of AU6 being selected is 100%. That is to say in the following situation when the observer saw a pair of stimuli, both stimuli had AU12 activated, one of them had AU6 activated but another one did not have AU6 activated; the observer #2 always chose the stimulus that have AU6 activated. This means that complementary AUs can drive the observer’s perception only in combination with the dominant AU.

3.2 Convergence efficiency

We discuss here the convergence efficiency of our approach by monitoring the convergence of 1) dominant AU computation and 2) complementary AUs computation as we increase the number of trials used in the reverse correlation procedure.

To do so, we compute 1) the correlation between the histogram for dominant AU computation using the first n trials from the entire perceptual experiment (i.e., Ω), and the final histogram of dominant AU computation (in Fig 2); 2) the correlation between the histogram of complementary AUs computation using the first n trials from the subset of trials in which all the stimuli have the dominant AU activated (i.e., Ω_{d}), and the final histogram of complementary AUs computation (in Fig 2).

Fig 3 shows the convergence of dominant AU computation and complementary AUs computation from the perceptual experiment of confidence. Similar converging curves from the perceptual experiments of happiness, sadness, and anger can be found in S4–S6 Figs. These curves reflect the typical reverse-correlation convergence (see e.g., Figure 6 in [39]).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. Example from the perceptual experiment of confidence to monitor the convergence of our approach.

(a) Correlation between the result of dominant AU computation after the first n trials (x-axis) and the result after 840 trials. The average correlation for all observers is marked by the red dashed line. (b) Correlation between the result of complementary AUs computation after the first n trials (x-axis) in the corresponding subset and the result using all trials in the corresponding subset.

https://doi.org/10.1371/journal.pone.0290612.g003

The dominant AU can be determined with only a 12-minute experiment. For the convergence of the dominant AU computation (in Fig 3a), while all 840 trials are considered, it takes less than 170 trials to reach a correlation of 0.9. As our approach just takes the AU with maximum proportion to determine the dominant AU (see Eq 2), performing 170 trials is enough. That is to say, only 170/840 ≈ 20% of the trials are necessary (equivalent to 12 minutes if the entire perceptual experiment needs 60 minutes).

Only a few data are used for complementary AUs computation. For each observer, only a small subset of trials (i.e., Ω_{d}) are, in effect, used to estimate complementary AUs. For instance, Fig 3b illustrates that the largest subset, which is from observer #2, only includes 31 trials, and these trials are distributed throughout the entire perceptual experiment.

A prototype can be determined accurately in about 20 minutes. Although in our approach, the number of trials is set based on related works [10, 36, 37, 39], it appears unnecessary to randomly generate as many as 840 trials to determine dominant and complementary AUs for an observer. In fact, the duration of the perceptual experiment can be largely reduced. If our approach only randomly generates the first 170 trials to determine the dominant AU and then generates another 100 trials only from Ω_{d} to determine the complementary AUs, it will be less than 20 minutes (270 trials, instead of 840 trials) to obtain the mental prototype.

3.3 Personalized prototypes

We reconstruct the personalized prototypes of each observer by activating the dominant and complementary AUs. Fig 4 shows the personalized prototype of each observer, as well as state-of-the-art prototypes from the literature [13, 18] (denoted by “Ek” and “Yu”). The corresponding activated AUs are listed at the bottom of the faces. For the personalized prototypes, we highlight the dominant AU in square brackets; the others are the complementary AUs. Note that there is not any published database of confidence, and there is no state-of-the-art prototype for confidence.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. Personalized prototypes of the four emotions from observers and state-of-the-art prototypes [13, 18].

For each personalized prototype, we indicate the emotion category followed by the observer (denoted by “#1” to “#4”) or the state of the art (denoted by “Ek” and “Yu”). We detail the dominant AU in square brackets and the others are complementary AUs. For the state-of-the-art prototypes, we only list the activated AUs. All facial expressions are reconstructed by the same GAN (i.e., GANimation [3]) and the same actor. Note that GANimation [3] cannot edit AU16 (lower lip depressor). We replace AU16 with AU25 (lips part) to reconstruct the angry prototype of Yu [18] (“angry-Yu”).

https://doi.org/10.1371/journal.pone.0290612.g004

According to the results in Fig 4, we observe that the personalized prototypes are compatible with state-of-art prototypes. All 12 prototypes of basic emotions generally convey expressions similar to that of Ekman [13] and Yu [18]. All dominant AUs: AU12 for happiness, AU4 or AU15 for sadness, and AU4 or AU9 for anger, can be found in state-of-the-art prototypes.

The prototypes are personalized. In each perceptual experiment task, although observers were asked the same questions, all observers acquired subtly different mental representations and, especially, different complementary AUs. With the exception of observer #1-sad and Ekman [13], all synthesized facial expressions also differed from the state-of-the-art prototypes by at least one AU. For instance, observer #2-happy is the same as Yu [18], plus the addition of AU20 (lip stretcher), resulting in a wider smile.

Our manipulations can be extended to the emotions that are not available in existing databases, such as confidence. We had no comparison prototypes for confidence. Although all confidence manipulations had the same dominant AU12 as happiness, the expressions remained different from any of the listed prototypes of happiness, notably because of the involvement of AU4 (brow lowerer), AU5 (upper lid raiser), AU9 (nose wrinkler) and AU17 (chin raiser).

4.1 Subjective evaluation by mean opinion score

To assess what observers thought of their own personalized facial expression prototypes, we employed the Mean Opinion Score, which is a popular indicator of perceived media quality [43]. We asked the four observers to rate their personalized facial prototypes from 1 to 5 representing bad satisfaction to excellent satisfaction. In Table 1, we listed all the scores rated by observers and presented the mean opinion score for each emotion (happy, sad, angry and confident) in the last column.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Subjective evaluation by mean opinion score.

https://doi.org/10.1371/journal.pone.0290612.t001

All observers rated their personalized facial prototypes with “4” and “5”, meaning that observers were satisfied with their personalized facial prototypes.

4.2 Subjective evaluation by ranking

To quantify the acceptance of the personalized prototypes, we added the state-of-art prototypes of Ekman and Yu [13, 18] for comparison. We conducted a ranking study involving two distinct groups of people: 1) The “observers” group consisted of the observers who had previously performed Perceptual experiment (i.e., observers #1, #2, #3, and #4), and 2) The “non-observers” group comprised participants recruited from Amazon Mechanical Turk (AMT) who had not performed the previous Perceptual experiment.

Based on the ranking results obtained from the “observers” group, we aimed to investigate whether each observer still preferred his/her own personalized prototype when presented with a set of prototypes mixed with other observers’ personalized prototypes and state-of-the-art prototypes. Additionally, by analyzing the ranking results obtained from the “non-observers” group, we aimed to gain insights into the preferences of the other people (bystanders) who did not participate in the previous perceptual experiments.

4.2.2 Results.

In terms of the “observers” group, Table 2 illustrates that each observer ranks his/her personalized prototypes in each position. Note that there are only 4 prototypes of self-confidence (no state-of-the-art prototype of self-confidence), whereas, for the other emotions, there are 6 prototypes. That is why the probability of “5th” and “6th” by random ranking is different from the others.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. The probability that each observer ranks his/her own personalized prototype in each position.

https://doi.org/10.1371/journal.pone.0290612.t002

We notice that all the observers rank their personalized prototypes in the top-3. All the observers have at least a 50% probability to rank their personalized prototypes in the first place. However, if the ranking is random, the probability to select their personalized prototypes is only 18.75%. This indicates that the prototypes are personalized. They can reflect the observer’s mental image and well answer the question in the perceptual experiment.

In terms of the “non-observers” group, in Table 3, we present the preferences between each pair of prototypes computed by the Schulze method and the final rankings. Note that due to cyclic preferences (such as the aforementioned “#2”, “#4”, and “Yu.” of sadness) the sum of the paired preferences is not always equal to 100%. For instance, in Table 3(b), the sum of the preference from “#4” to “Yu.” (57%) and the preference from “Yu.” to “#4” (51%) is 108%. Considering state-of-the-art prototypes as the baselines and for these 217 participants, our observations are as follows.

• The low-ranking personalized prototypes are about equally preferred to at least one of the state-of-the-art prototypes. “#3”, “#4”, and “Ek.” in Table 3(a) and 3(c), and “#2”, “#3”, and “Yu.” in Table 3(b) are low-ranking (ranked in the last three). The paired preferences between them are around 50%. For instance, in Table 3(a), 46% of the participants preferred “#4” to “Ek.”, and 54% of the participants preferred “Ek.” to “#4”. That is to say, these low-ranking personalized prototypes are about equally preferred to the state-of-the-art prototype (“Ek.” or “Yu.”). This also validates that our approach can generate personalized prototypes.
• Emotional prototypes are not universal. As shown in Table 2, the prototypes are not universally preferred among participants. Although in Table 3(a) and 3(c), the top-ranking prototypes are much preferred over the others (“#2” of happiness and “#1” of anger), most preferences are far from 100% (and 0%). Especially in Table 3(b), most prototypes of sadness (including state-of-the-art prototypes) are about equally preferred among the hired participants. Indeed, most preferences are close to 50% which is quite far from 100%. Even there are cyclic preferences. Hence, there can be many prototypes of one emotion.

Download:

• PNG
  larger image
• TIFF
  original image

Table 3. Preferences between each pair of prototypes computed by the Schulze method and the final rankings.

https://doi.org/10.1371/journal.pone.0290612.t003

To sum up, our approach generated personalized prototypes which both differed from each other and from state-of-the-art prototypes. According to rankings from the “observers” group, we can validate that the prototypes are personalized. According to the rankings from the “non-observers” group, personalized prototypes are either close to state-of-the-art prototypes (e.g. “#1” and “Ek.” of sadness are identical) or preferred to state-of-the-art prototypes. Rankings also suggest that preferred prototypes of any single emotion are not unique across different people.