Figures
Abstract
Numerous studies indicate that listening to music and reading are processes that interact in multiple ways. However, these interactions have rarely been explored with regard to the role of emotional mood. In this study, we first conducted two pilot experiments to assess the conveyed emotional mood of four classical music pieces and that of four narrative text excerpts. In the main experiment, participants were asked to read the texts while listening to the music and to rate their emotional state in terms of valence, arousal, and dominance. Subsequently, they rated text and music of the multisensory event in terms of the perceived mood, liking, immersion, and music-text fit. We found a mutual carry-over effect of happy and sad moods from music to text and vice versa. Against our expectations, this effect was not mediated by the valence, arousal, or dominance experienced by the subject. Moreover, we revealed a significant interaction between music mood and text mood. Texts were liked better, they were classified as of better quality, and participants felt more immersed in the text if text mood and music mood corresponded. The role of mood congruence when listening to music while reading should not be ignored and deserves further exploration.
Citation: Hauck P, Hecht H (2023) Emotionally congruent music and text increase immersion and appraisal. PLoS ONE 18(1): e0280019. https://doi.org/10.1371/journal.pone.0280019
Editor: Thiago P. Fernandes, Federal University of Paraiba, BRAZIL
Received: May 10, 2022; Accepted: December 19, 2022; Published: January 12, 2023
Copyright: © 2023 Hauck, Hecht. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files. They are also uploaded on OSF: DOI 10.17605/OSF.IO/SF47R.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Multitasking is a phenomenon of our time. Smartphones and other digital devices are omnipresent in our daily lives. Apart from making phone calls, we send text messages, check the news, read books, make photos, etc. all the while listening to music. Our increasing mobility and the option to be constantly online contribute to a continuous sensory overload. Visual input via the display, auditory signals delivered by headphones, and haptic stimuli like the smartphone vibration merge with sensations from the surrounding world into complex multisensory impressions. Reading texts and simultaneously listening to music has become a common combination of activities, sometimes intending to create a musical background for the reading content, sometimes just to seal oneself off from surrounding noise. But what does this new level of intersensory fusion do to our emotions? Both music [1] and text [2] are known to be able to transport distinct emotions. But what happens to our emotion recognition when we are consuming music and text at the same time? Does the music influence our impression of the mood expressed in the reading material, and/or vice versa? In the following, we will give an overview of the relevant research on the interplay of music and text perception and then report an experiment that tests the hypothesis of the mutual emotional impact of music and text.
Influence of music on cognitive performance
The influence of music on cognitive performance has been extensively investigated. Evidence about the disturbing, enhancing, or neutral nature of this influence, however, is mixed (see [3], for a meta-analysis). Important factors that have been identified in this context are task difficulty [4], music complexity [5], and the personal preference for external stimulation, along with extraversion [6, 7]. As to emotional factors, arousal and mood conveyed by the music have been shown to differentially influence memory performance [8, 9]. In a recent study, Ward et al. [10] found that mood-matching music can improve recall in older adults. However, there is still no consensus about the distracting or supporting impact of music on cognitive performance, and a closer look at the literature reveals a number of gaps and shortcomings. We want to address the issue with the aim of closing one of these gaps by specifically concentrating on the influence of music on emotion perception while reading narrative texts.
Influence of background music on the reading process
Several studies have examined the influence of music on cognitive aspects of the reading process, such as reading speed, fluency, and comprehension. Findings, however, are inconsistent. For instance, music has mostly been found to distract the reader, but it can also promote concentration (see [4, 11], for reviews). Several studies demonstrate that text comprehension is impaired by background music or noise when compared to reading in silence [12, 13]. Employing an eye-tracking approach, Zhang et al. [14] confirmed this disrupting effect, showing that background music is associated with more re-reading of the text. However, not every music piece has the same impact on the reading process. Thompson et al. [15] discovered that loud and fast background music led to reduced text comprehension, while soft and slow music did not make a difference in this regard. Moreover, the distracting effect has been shown to depend on whether or not the music was preferred as study music [16]. Participants scored significantly lower in a reading comprehension test when they had listened to non-preferred study music while reading than when they had read in silence. Music previously categorized as preferred study music did not have this effect. As to the differentiation of music types, Vasilev et al. [17] concluded in their meta-analysis that background noise, music, and speech have a stronger distracting effect on reading performance when background speech contains meaningful content and when music has meaningful lyrics. However, their analysis found small but stable adverse effects for all kinds of background sounds, which was true for adults as well as for children. Besides the influence of tempo, loudness, preference, type, and other characteristics of the music, interindividual differences have also been shown to affect the strength of the music’s impact on reading performance. As such, extraverts and persons with higher working memory capacity seem to be less disturbed by background music than introverts and those with lower working memory capacity [18, 19].
Emotion perception in music and text
In contrast to the relatively well-studied influence of music and noise on the reading process, there are no reports in the literature on whether music also changes the emotional evaluation of the reading content, and vice versa. We therefore merely provide a brief overview of studies addressing related research questions.
Gfeller et al. [20] investigated the affective response to music, text, or the combination of both. In their design, both text and music stimuli were presented acoustically, as they employed a recorded recitation of a poem and recordings of commercial-style or atonal music. Their results show that some emotional ratings changed when music was combined with text, as compared to music only or text only. For instance, the text was liked better when combined with popular commercial-style music than in combination with unpopular atonal music. The authors conclude that listening to music can influence our emotional response to simultaneously received texts.
Another study on emotional factors in the context of reading with background music by Kallinen [21] showed that valence assessment of texts depended on music tempo. Focusing on gender effects, they found that men rated news texts to be more positive when listening to fast rather than slow background music, whereas women’s ratings were most positive in the no-music condition. Unfortunately, the researchers employed only one music piece with different tempo modifications instead of different music stimuli, so it is difficult to generalize their results.
Turning the tables, some attempts have been made to explore the inverse emotional impact of text in the form of lyrics on music perception [22, 23]. In an early work, Stratton and Zalanowski [24] revealed a relative dominance of emotions conveyed by the lyrics compared to emotions carried by the music. More recently, Ali and Peynircioğlu [25] demonstrated that sad music with sad lyrics is perceived as sadder than sad music without lyrics. Interestingly, happy music with happy lyrics was rated as less happy compared to happy music without lyrics. The authors concluded that congruent lyrics detract from and thus attenuate happy emotions, while they bolster sad emotions conveyed by the music. In contrast to the findings by Stratton and Zalanowski [24], Ali and Peynircioğlu found that the emotions carried by the music are dominant when music and lyrics convey incongruent emotions (see also [26]). In agreement with the latter results, Brattico et al. [27] showed that happy music without lyrics activates the limbic system whereas happy music with lyrics is reflected in auditory regions only. Except for the described research on music and lyrics, an examination of the influence of nonmusical text on music perception has not yet been made.
Influence of expertise on multisensory perception
In multisensory research, it is of great interest to examine differences between experts and novices. For instance, it has been shown that persons who scored higher on a test that assessed their touching expertise by measuring ‘need for touch’ were less susceptible to the nondiagnostic haptic experience of a container when rating the gustatory qualities of its contents [28]. In the context of music and reading, there are mixed results regarding the impact of musical expertise on cognitive tasks. Patston and Tippett [29] found that musicians but not non-musicians were disturbed by background music in the completion of a language comprehension task (see also [30]). However, musicians benefit more from soft background music in terms of attention performance than do non-musicians [31]. As there is no clear evidence about musical expertise as a factor in the susceptibility to background music, we want to address this issue in the present research.
Emotional mediation of multisensory processes
In the literature on multisensory associations or crossmodal correspondences [32], it is widely acknowledged that the emotional state a person experiences during the reception of the multisensory input plays an important role. When stimuli from one sensual modality influence the perception in another, or when multisensory stimuli are consistently associated with each other, emotional states have often been found to explain the effects [33, 34]. The emotion mediation hypothesis [35] suggests that stimuli which trigger the same emotion when presented separately, are more likely to be perceived as fitting together (see [36], for a review). This connection was demonstrated for associations among, inter alia, odor and color [37], music and color [38], music and taste [39], taste and shape [40], and touch and color [41].
As we want to examine the potential influence of music mood on perceived text mood and vice versa, we must separately consider the emotions attributed to the stimuli on the one hand, and the recipient’s mood state on the other hand. Only if both affective dimensions are independently measured, can emotional mediation be examined. In our study, the stimulus attributes music mood and text mood were assessed in a pre-test with regard to the extent to which they conveyed a happy or a sad mood. The respective texts and music pieces were then categorized as ‘happy’ or ‘sad’. The dependent variables ‘perceived text mood’ and ‘perceived music mood’ were measured by numeric rating scales. We employed a pictorial scale to determine the mood state of the participant. According to the emotion mediation hypothesis, the independent mood variables should lead to a change in experienced emotions, which in turn would correlate with the dependent mood measure. That is, sad music would make the participant feel sadder, which in turn would lead to higher sadness-ratings of the text, and vice versa.
Inducing emotions through music and texts
Numerous studies have shown that music reliably evokes emotions in the listener [1, 42]. This has proven very useful when it comes to the focused elicitation of specific emotions as an experimental method [43, 44]. Especially for happiness and sadness, playing music to participants turned out to be a very effective induction method [45]. Likewise, narrative texts have been used to induce emotions in experimental studies [2], as speech per se is known to easily convey mood, be it via single words, sentences, or whole stories [46–48].
Emotions influence music and text perception
Research on the impact of mood on emotion perception in texts or music has received only limited attention so far. As to music, Vuoskoski and Eerola [49] found that participants’ current mood is associated with mood-congruent music ratings. However, this effect was moderated by extraversion, suggesting a complex relationship between mood and personality. Another study reported distinct age differences in the influence of listener mood on affect perception in music. Younger adults’ perception of negative valence in the music was mood-congruent, whereas it was mood-incongruent for older adults. Arousal perception in the music was mood-congruent for both age groups, but the effect was much more pronounced in younger participants [50]. In the context of text perception, the prevailing mood has thus far only been considered in terms of reading performance. Mills et al. [51] showed that a sad mood leads to more profound text comprehension (deep-reasoning) compared to a happy mood (see also [52, 53]). Opposite results were found by Scrimin and Mason [54], whose participants showed better text comprehension and better learning of factual knowledge when they were in a positive mood. Apparently, evidence about the connection between the receiver’s emotions and music and text perception is still limited. Exploring this relationship will be part of this study.
Our study
The main objective of this work is to investigate the relationship between perceived emotions in music and text when simultaneously consumed. We want to examine if a distinct mood in the music is reflected in the mood evaluation of a text being read at the same time, and vice versa. We are also interested in exploring a potential influence of emotions conveyed by music or text on the perceived music-text fit, as well as on the receivers’ felt immersion into the music or the text. As we are interested in both directions of the potential crossmodal effects, we applied a symmetrical research design. A secondary aim of the study is to regard the impact of relative expertise on multisensory transfer and association. We furthermore examine the assumption of an emotional mediation for all mentioned effects.
We hypothesized that the emotional mood of the music (or the text) influences the perceived mood conveyed by the text (or the music) by way of an emotional transfer. We assumed that this effect is less pronounced in experts than in novices.
To test these hypotheses, we performed two pre-tests and one laboratory experiment with three separate samples. The pre-tests validated the emotional value of music pieces and narrative text excerpts and were the basis to choose the stimuli for the main experiment. In a fully crossed within-subjects-design, participants were asked to read the texts while listening to the music and to rate both text and music in terms of the conveyed emotional mood (happy vs. sad), artistic quality, perceived immersion, and general liking. We also assessed participants’ momentary emotional state by measuring valence, arousal, and dominance at the beginning of the experimental session and after every trial.
Method
Sample
40 students (31 female, 9 male; Mage = 23.95, SD = 4.27) volunteered to participate in the experiment, amongst which 39 were psychology students and one studied another subject. As it is mandatory for psychology students at the Johannes Gutenberg University Mainz to participate in studies conducted in the department, most of them had prior experience with cognitive experiments. However, there was no overlap or potential interference with designs tested here. All of them gave written informed consent in accordance with the Declaration of Helsinki. 38 participants had self-reported normal or corrected-to-normal vision and hearing; only two were slightly myopic and did not wear glasses or contact lenses. However, their data did not deviate substantially from those of the rest of the sample, so they were included in the subsequent analyses. All participants reported being non-synesthetes. As all subjects were university students in good standing, we assumed that their cognitive performance was not impaired. Thus, we did not carry out cognitive tests, nor did we test them for previous substance use. Musical expertise was assessed by the Goldsmith Musical Sophistication Index Version 1, a self-report inventory for individual differences in musical sophistication (Gold-MSI; [55]). It measures the ability to engage with music independently of musical style, preferences, or regularity of musical practice. In order to avoid overloading the questionnaire, we picked 18-items assessing the General Factor of Musical Sophistication. In our sample, the mean General Musical Sophistication Score (GMS) was below average compared to the norm (MGMS = 68.97, SDGMS = 23.13; 28th percentile; norm: MGMS = 81.58, SDGMS = 20.62). One participant scored extremely high (GMS = 122); however, her data did not stand out and were thus not dismissed. For analysis, we performed a median split to divide the sample into more (experts) and less (novices) musically experienced participants (median GMS = 69.25). We identified extreme outlier values in terms of music and text ratings with the help of boxplots (criterion: > 3 * IQR from 1st or 3rd quartile) and replaced these outliers by the group mean. In total, this was the case for five data points representing 0.21% of all ratings. In addition to the Gold-MSI questionnaire, we had asked for daily music listening and reading routines. Most of the sample reported listening to music (almost) every day (n = 35), the remaining participants indicated to do so less frequently. In terms of reading, 15 participants reported to read (almost) every day, 10 read several times a week, 13 read several times a month and two participants indicated to (almost) never read for pleasure. 6 participants found reading (more or less) tiring, and one participant indicated only to read when necessary. Most of the sample (n = 33) reported that they did (rather) not like to listen to music while reading.
As the group of those used to simultaneous reading and listening to music was very small (n = 7), we could only exploratively examine if habituation might be an influencing factor for the crosstalk between music and text. The experiment was conducted in accordance with the Declaration of Helsinki. Informed consent was obtained before the experimental session, subjects were told that they could abort the experiment at any time without consequences, and they were debriefed about the purposes of the study immediately after the session. The study was approved by a local Departmental Ethics committee under a blanket approval.
Apparatus and stimuli
Setup.
The experiment took place in a darkened laboratory room equipped with a table and an office chair. The table was standing in front of a wall of LED panels with movable walls to the left and right of the table so that the walls constituted an open booth (see Fig 1). The LED panels emitted slightly reddish light (RGB: 255, 55, 0), creating a comfortable, warm atmosphere. An additional desk lamp on the table provided customary reading light. Noise-canceling headphones lay ready on the table.
The LED wall emitted a slightly reddish light creating a comfortable atmosphere, a table lamp facilitated reading in the otherwise dark room.
Hard- and software.
Stimuli were generated with a Dell Optiplex 980 PC (Core i5) using Python 2.7.13. Data were analyzed with the statistics software IBM SPSS statistics version 27.
Text stimuli.
We employed excerpts from four different belletristic texts that have been shown to convey either sad or happy emotions, drawing on Zupan’s and Babbage’s list of narrative texts for emotion elicitation [2]. In a small pilot experiment with an independent sample (n = 8, 6 female; Mage = 31.38, SDage = 9.47), we tested the emotional value and impact of six happy, six sad, and six neutral excerpts from novels that had been translated into German. The texts had a length of 500 to 600 words and required approximately three minutes of reading time on average. Participants rated the emotional mood of the text on a bipolar 11-point scale from -5 (sad) to +5 (happy) and the intensity of the mood on a 10-point scale from 0 (‘not at all’) to 9 (‘very much’). As there is no generally accepted tradition of mood assessment for crossmodal stimuli, we have opted for an integration of qualitative and quantitative measurements. For the analysis, we generated a mood score by multiplying the two ratings. Participants furthermore rated valence, arousal, and dominance they felt at that moment on a 9-point version of the Self-Assessment Manikin scales (SAM scales [56]). We chose the two texts each that had been rated as most happiness- and sadness-conveying as stimuli for the main experiment. For the experimental sessions, the texts were printed on custom white print paper (size Din A 4). Details about the chosen stimuli–two sad and two happy text excerpts–can be retrieved from S1 Table.
Audio stimuli.
In analogy to the text stimuli, we chose four music pieces as audio stimuli said to carry either happy or sad emotions. We pretested six happy and six sad pieces that had been used to induce emotions in previous studies (see [57], for a review). All pieces were classical string music from the 18th or 19th century, which we cut or looped to a length of four minutes each. To reassure that the musical character would not vary too much throughout each piece, we cut the audio track just before a distinct mood change, e.g., a change of key, and restarted the piece from the beginning, until four minutes were reached. In an online-pretest, 45 participants (33 women; Mage = 25.64, SDage = 7.15; MGMS = 65.68, SDGMS = 23.7) rated the pieces on the same scales as had been used for pretesting the text stimuli. We chose the music pieces that scored highest in terms of conveying a happy and a sad mood (two music pieces per mood) for the main experiment. S2 Table lists further information about the chosen stimuli. The resulting mp3-files were set to an equivalent continuous sound level of about 60 dB using an SPL-meter (NTi AL1). During the experimental session, they were presented through headphones.
Design and procedure
We applied a fully crossed 2 (music mood) x 2 (text mood) within-subjects design, resulting in four experimental conditions. To avoid the repetition of a text or music piece for any given subject, there were two texts and two music pieces per mood (two happy, two sad each). In order to counterbalance potential order effects, all possible stimulus orders were presented to the same number of subjects. The four combinations per condition resulted in eight possible permutations of the four conditions, each presented to 5 of the 40 participants. Their assignment to a given stimulus order was randomized.
Participants were tested individually. First, their prevailing mood was assessed through the SAM scales (see stimuli section), then they were asked to put on the headphones and read the first text. The music started as soon as they began reading. Participants were told to focus on the text, just as if they were reading for pleasure in their leisure time. They were instructed to continue reading for as long as the music would play, and start from the beginning of the text in case they would have finished reading it before the end of the music. With this instruction, we ensured that every participant read the full text and heard the entire music piece. When the music stopped, participants filled in the SAM scales again, followed by a questionnaire measuring their liking of the text (‘How do you like the text?’), their evaluation of its artistic quality (‘How would you rate the artistic level of the text?’), their perceived immersion (‘How much did the text intrigue you?’), and their assessment of the text’s emotional mood (‘How would you rate the emotional mood of the text?’). Afterward, the same set of questions was filled in to evaluate the music, supplemented by an indication of familiarity with the piece (‘How well did you know the music before this experiment?’), and followed by a question on the perceived fit of text and music (‘How would you personally rate the fit between the text and the music?’). All constructs, except for emotional mood, were measured on 10-point scales from 0 (‘not at all’) to 9 (‘very much’). Like in the pretests, the character of the emotional mood was rated on a bipolar 11-point scale from -5 (sad) to +5 (happy), followed by the measurement of emotional intensity on the noted 10-point scale. Half of the participants first rated the texts and then the music, while the other half received the questionnaires in reverse order. Reading speed was measured with a stopwatch. The whole sequence was repeated for all four experimental conditions.
At the end of the experiment, participants filled in a final questionnaire (see Supplementary Material) assessing demographic data as well as different control variables, namely eyesight, hearing, synesthesia, gender, age, profession, highest completed level of education, reading habits, music listening habits, reasons for listening to music, activities combined with listening to music, attitudes towards reading, and the short version of the Gold-MSI [55] measuring musical expertise. The experimental session ended after approx. 35 minutes. The temporal sequence of the experimental session can be seen in Fig 2. Data were collected between July 29th 2020 and January 17th 2021.
Note that the flow chart is exemplary for 50% of the participants that were instructed to rate first text then music. The other 50% of the participants started with music ratings. Participants were randomly assigned to the ‘text first’ or ‘music first’ groups.
Results
In the first step, we will describe analyses on the influence of music on text perception, and in the following, we will consider the reverse influence of text on music perception. In addition to the included graphs and tables, boxplots further illustrating participants’ ratings are shown in S1 Fig.
Influence of music mood on text perception
We calculated a 2 (music category) x 2 (text category) repeated measures multivariate analysis of variance (rmMANOVA) for the dependent variables perceived mood score, quality, immersion, and liking of the text. The factors ‘music category’ and ‘text category’ refer to the classification as ‘sad’ or ‘happy’ according to the pre-experiments (see methods section). Both multivariate main effects and the multivariate interaction were significant (see Table 1 for the complete results). Post-hoc univariate rmANOVAS (LSD-corrected) showed that the main effect of text category was significant for perceived text-mood score, immersion, and liking, but not for quality. As expected, happy texts were perceived as conveying a happier mood, they were liked better, but they were less immersing than sad texts. The main effect of music category was significant for perceived mood score, that is, texts were rated as conveying a happier mood with happy background music than with sad background music. There were no significant univariate effects of music category for text quality, immersion, and liking. Univariate interactions of music category x text category were significant for text quality, immersion, and liking, but not for text-mood score. As illustrated in Fig 3, texts were perceived as higher in quality, as more immersing, and they were liked better when read under congruent music. All effects were clearly more pronounced for sad compared to happy texts.
Means for perceived mood score (a), artistic quality (b), immersion (c), and liking (d) of the text as a function of happy vs. sad music and text categories. Error bars show ± 1 SEM.
Multivariate and univariate main effects and interactions.
We then tested if the found effects of music mood on text evaluation were mediated by the experienced emotions of the participant. As mentioned above, experienced emotions were measured according to their valence, dominance, and arousal at the beginning of the experiment (baseline) and after every trial. For analysis, we subtracted all values from the baseline measurement and hence displayed the relative change in the emotional state throughout the experimental session. Then, we first calculated a 2 (music category) x 2 (text category) rmMANOVA on valence, arousal, and dominance, and, second, examined the interrelation of the significant emotion measures with the different text evaluation variables by calculating Pearson correlations.
The rmMANOVA revealed a significant main effect for music category, thus the stimulus mood did influence the participants’ emotional condition, F(3, 36) = 4.5, p = .009, ⴄ2 = .27. Post-hoc univariate rmANOVAs showed that the effect was driven by changes in valence and dominance ratings, while arousal was not significantly affected (valence: F(1, 38) = 11.25, p = .002, ⴄ2 = 0.228; dominance: F(1, 38) = 5.74, p = .022, ⴄ2 = .131; arousal: F(1, 38) = 0.85, p = .364, ⴄ2 = 0.022). In the next step, however, valence and dominance did not reveal any relation to text evaluation: Among all Pearson correlations between valence and perceived text mood, only one out of four possible music-mood-text-mood-combinations was significant (sad music with happy texts, r(38) = .32, p = .042). In addition, for none of the four combinations did we find a significant correlation between dominance and text mood, nor between valence or dominance and liking of the text. Nevertheless, text immersion correlated positively with valence for happy-music-happy-text-combinations, and negatively for happy-music-sad-text-, and sad-music-sad-text-combinations (see S3 Table, for all correlation values). Data indicated that valence mediates the effect of music mood on text immersion.
Influence of text mood on music perception
To examine the reverse influence of text mood on music perception, we calculated a 2 (music category) x 2 (text category) rmMANOVA for the dependent variables perceived mood score, quality, immersion, and liking of the music. Both main effects of music category and text category were significant, as well as the respective interaction (see Table 2, for the complete results). Univariate rmANOVAS showed that text category significantly influenced the perceived mood score of the music. Music-mood was rated as happier when participants were reading a happy text while listening. The univariate interaction of music category x text category was significant for liking and immersion of the music. Music was liked better while reading a congruent compared to an incongruent text. Similarly, music was rated as more immersing when presented simultaneously with a congruent rather than an incongruent text. As illustrated in Fig 4, this effect was stronger for sad than for happy music.
Means for perceived mood score (a), artistic quality (b), immersion (c), and liking (d) of the music as a function of happy vs. sad music and text categories. Error bars show ± 1 SEM.
Multivariate and univariate main effects and interactions.
To test the emotional mediation of the influence of text mood on music perception, we mirrored the analytical procedure described above for the reverse effect. The rmMANOVA on emotional measures revealed a main effect of text category, F(3, 36) = 14.06, p < .001, ⴄ2 = .053, which showed significance in all three univariate analyses, indicating that the stimulus mood had an influence on the participants’ emotional state (valence: F(1, 38) = 33.78, p < .001, ⴄ2 = .471; arousal: F(1, 38) = 11.23, p = .002, ⴄ2 = .228; dominance: F(1, 38) = 29.28, p < .001, ⴄ2 = .435). Sad texts were associated with lower valence, but higher arousal and dominance ratings compared to happy texts. However, there were no significant correlations of valence, arousal, and dominance ratings with perceived music-mood score for none of the four music-text-mood-combinations. For sad-music-sad-text-combinations, there was a significant positive correlation between arousal and music liking and a significant negative correlation between valence and music immersion (see S4 Table, for all correlation values). A general mediation by emotion ratings could not be observed.
Music-text fit
To test if music and text are more likely to be perceived to fit together when they convey the same mood, we pursued two alternative approaches. First, we inspected the interrelation between our stimulus mood categories (happy vs. sad) and music-text fit ratings. Therefore, we calculated a 2 (music category) x 2 (text category) rmANOVA with music-text fit as the dependent variable, and found a strong music category x text category interaction, F(1, 39) = 111.55, p < .001, ⴄ2 = .741. Fit ratings were higher when music and text categories corresponded. Complete results can be retrieved from S5 Table.
In a second approach, we took participants’ mood ratings as a reference and determined the absolute difference between music-mood ratings and text-mood ratings for each combination of music and text categories. We then calculated Pearson correlations between this difference value and fit ratings. All four correlations were negative, with significant negative correlations for happy-music-sad-text combinations, r(38) = –.33, p = .039, and for sad-music-happy-text combinations, r(38) = –.56, p < .001. The more similar the respective moods conveyed by text and music were rated, that is, the smaller the perceived mood difference, the better music and text were perceived to fit together.
Exploration of the influence of habituation and expertise on the crosstalk between music and text
In our post-questionnaire, we collected data on musical and reading expertise, as well as music listening and reading habits. We used these data to explore if expertise and habituation had an impact on the found crossmodal effects.
Musical and reading expertise.
To test the influence of musical expertise on the effect of music mood on text perception, we calculated the same 2 (music category) x 2 (text category) rmMANOVA for perceived mood score, quality, immersion, and liking of the text, this time with the between-subjects-factor musical expertise. This factor has been built by performing a median split on the Gold-MSI Musical Sophistication Score (see methods section) to receive the two groups of musical experts and non-experts. There was no significant effect of musical expertise for any of the dependent variables (perceived mood score: F(1, 38) = 0.37, p = .547, ⴄ2 = .01; quality: F(1, 38) = 0.003, p = .995, ⴄ2 = .0; immersion: F(1, 38) = 0.19, p = .666, ⴄ2 = .005; liking: F(1,38) = 0.001, p = .976, ⴄ2 = .0), nor an interaction with one of the main factors (see S6 Table). We calculated another 2 (music category) x 2 (text category) rmMANOVA on perceived mood score, quality, immersion, and liking of the music with the between-subjects factor reading frequency. The latter is a dichotomous variable created from the answers to the 4-point question ‘How often do you read for pleasure (books, newspapers, magazines, news texts; analog or digital)?’ by contrasting frequent readers (‘several times a week’ and more) with seldom readers (‘several times a month’ and less). There was a significant between-subjects effect of reading frequency for liking of the music (F(1, 37) = 5.11, p = .030, ⴄ2 = .121). Frequent readers gave generally higher ratings for music liking than seldom readers. There were no other significant univariate between-subjects effects (perceived mood score: F(1, 37) = 0.41, p = .526, ⴄ2 = .011; quality: F(1, 37) = 0.43, p = .518, ⴄ2 = .011; immersion: F(1, 37) = 1.15, p = .291, ⴄ2 = .030), nor significant interactions of reading frequency with the main factors (see S7 Table).
The habit of listening to music while reading.
We had asked participants to indicate their degree of agreement or disagreement with the statement ‘I like to listen to music while reading’. Most of the participants (n = 33 out of 40) said (rather) not to do so. Even though the remaining group of ‘music-readers’ was very small, we were interested in potential differences between the ‘music-readers’ and the ‘silence-readers’ in terms of music- and text-mood interplay. Thus, in an explorative approach, we calculated the 2 (music category) x 2 (text category) rmMANOVA for perceived mood score, quality, immersion, and liking of the text again, now with the between-subjects-factor habit. We carried out an analogous rmMANOVA with the same factors and dependent variables referring to music perception. There was a significant effect of habit for text mood, F(1, 38) = 5.525, p = .024, ⴄ2 = .0127. As can be seen in Fig 5, text-mood ratings by music-readers were more strongly influenced by the music than those of silence-readers. There were no other significant between-subjects effects for habit, nor an interaction of habit with the main factors (see S8 Table).
Error bars show ± 1 SEM.
Discussion
The present research examined the interrelation between emotional appraisals of classical music and narrative texts. In a fully crossed within-subjects design, participants were asked to read two sad and two happy texts while listening to sad or happy music. Subsequent ratings referred to the emotions perceived in the stimuli, but also to the participants’ current emotional state. We furthermore assessed perceived immersion, as well as artistic quality, liking, and perceived music-text fit. The study revealed three key findings: First, we showed that both music and text have a mutual influence on the emotional appraisal of the other, suggesting a transfer of emotions. Second, it turned out that both music and text are perceived as more immersing and are liked better when music and text mood correspond. Third, we saw that the more similar the separate mood ratings of music and text, the better they were perceived to fit together. Importantly, only the crossmodal effect on text immersion was mediated by valence, whereas all other effects were not mediated by the emotions experienced by the participant. We also examined potential group differences, but neither musical expertise nor reading frequency made a difference in terms of the reported outcomes. However, an explorative examination of reading habits hints at a difference between those who are used to reading while listening to music compared to those who are not combining these two activities.
Multisensory transfer of emotional attributions
As expected, our findings indicate that happy or sad mood is transferred from music to texts and vice versa. That is, sad (happy) music makes written stories appear sadder (happier), and sad (happy) stories make music sound sadder (happier). This observation agrees well with numerous studies in the field of multisensory research that have examined the influence of music on the perception of diverse sensual stimuli. For instance, it has been found that listening to ‘soft’ music increases haptic softness perception [58], listening to ‘light’ music makes red wine taste lighter [59], music can enhance the emotions conveyed by pictures [60], and coffee tastes less bitter in the presence of ‘sweet’ music [61]. In a recent study by Ehret et al. [62], musical attributions were also shown to affect the perceived overall valence of the atmosphere. Accordingly, Yamasaki et al. [63] demonstrated that the evaluation of the environment can be shifted in the direction of specific characteristics of the music. Our results integrate these previous outcomes, as they demonstrate the mutual impact of auditory input on visually received information, and vice versa, which then influence the overall impression of a fictive environment and atmosphere.
Previous authors have often solicited emotion as the mediator and key to deciphering the crossmodal transfer of stimulus characteristics (see introduction). Our research disagrees with this interpretation, as emotional mediation did not explain the effects of crossmodal transfer and association in the case of music and text. However, the comparability of study designs is limited. Narrative texts are complex stimuli and the task of rating the conveyed mood of the text implies not only visual processing but also the cognitive effort of reading. In contrast, it is common in the field of multisensory research to employ simple stimuli like color patches, single tones, and basic taste samples. With our approach, we offer a first assessment of crossmodal emotional mediation involving texts as stimuli.
Regarding the underlying emotions, we can conclude that music and text did not primarily affect the participants’ mood but rather did the music impinge on the assessment of the text. And vice versa did the text alter the appraisal of the music. Thus, we can assume that we found rather pure appraisal effects here. We may ask how likely such intersensory appraisal changes are to influence our actions. Negative emotions were recently shown to influence our actions, specifically our inhibitory control [64, 65]. Note, however, that the crossmodal appraisal effect in the current study are once removed from the actor’s emotional state. We can thus only speculate if negative emotions identified in a given text or music piece will impact our behavior. Further studies building on this hypothesis would be a promising path forward.
In our paradigm, most effects were more pronounced for sad than for happy stimulus moods. This is once more in line with the insights by Ehret and colleagues [62], who revealed that adding a negative aspect to a positive ambiance has a bigger impact on the overall evaluation of the atmosphere than does add a positive aspect to a negative status quo. This pattern might be explained by a general negativity bias, suggesting that negative information is given more weight than positive information, it is processed faster and commands more attention [66].
Now that the mutual influence of text and music mood is established, which are the underlying processes? We will refer to two approaches to explain the crossmodal effects. First, it is conceivable that multisensory integration takes place, despite the instruction to merely evaluate one modality. Information that is received in one sensory channel may spill over to the other channel. For the recipient, it is difficult if not impossible to distinguish which information is transmitted via which sensory channel. Impressions and characteristics belonging to one stimulus merge with the other, equivalent to a direct transfer of attributions [67, 68]. In our study, the crosstalk of text and music was reciprocal, that is, music had an influence on text perception, and text mood, in turn, affected music perception. This indicates that the mood conveyed by music and text is indeed regarded as one unified experience [69]. Second, Kahneman’s capacity model for attention [70] can be used to explain our findings on a broader level. It claims that cognitive capacity is limited and therefore attention has to be allocated among possible mental activities. As attentional resources become insufficient to closely attend to both text and music, the perceptual system may devote the resources to the recognition of emotion at the expense of linking the perceived emotion to its source. Moreover, it was highly probable within the scope of our 2 x 2-design that text and music transmitted a similar emotional message. This coincidence may have been exploited in the face of high attentional demand.
Influence of mood congruency on perceived quality, immersion, and liking
In our study, music and text were rated higher in immersion and were liked better when they were presented together with mood-congruent text or music. Moreover, the perceived text quality was higher when text and music mood corresponded, while music quality ratings were not influenced by the text. These findings on quality and liking are not surprising, as studies agree that stimuli are evaluated more positively when accompanied by congruent information in another apparently irrelevant sensory modality [71–73]. The fact that perceived music quality (in contrast to perceived text quality) was not systematically affected by congruency could be explained by the idea that it is generally more difficult to evaluate a classical music piece rather than a belletristic text. Those who are not educated in classical music might not hear quality differences at all, or they might not dare to judge it. The small variance of music quality ratings in our data supports this assumption.
Let us now find explanations for the outcome that participants felt more immersed in music and text when the stimuli were congruent in mood. We could show that the effect of happy music on text immersion is mediated by experienced valence. That is, happy music makes us happier, which in turn promotes our feeling of being absorbed in the happy text. The feeling of happiness is possibly connected to a certain light-heartedness and momentary open-mindedness which enables us to better engage in the positive reading content and to dive down into the happy story. In the case of sad texts valence is negatively correlated with immersion, that is, the worse we feel, the more we get involved with sad texts. We conclude that a text is more captivating if it is congruent with our own emotional state. As to the reverse connection, our current findings on emotional mediation concerning the influence of text mood on music immersion as well as on music liking remain inconclusive at the point and require further investigations.
Our results show that not only immersion but also liking and perceived quality were higher in the mood-congruent conditions. Thus, it seems that a text or a music piece intrigues us more if we generally like it and if we consider it of good quality. However, the causality between these three constructs is yet to examine. Thinking laterally, interesting analogies can be found in film music research. Studies revealed that music and sound effects influence emotions experienced by the viewer [74] and enhance immersion and suspense [75]. Film music furthermore affects the semantic evaluation of film scenes [76], it can foster empathy with the characters [77], and the content of film excerpts is remembered more easily when music and pictures convey a congruent mood [78]. Just like in our paradigm, music appears as an influential factor for the involvement in the story.
The perceived fit between music and text
Our analysis confirmed distinct patterns of perceived music-text fit. The more similar music and text were evaluated in mood, the better the stimuli were regarded to match. This result is in agreement with our recent study showing that music and colored ambient lighting are judged to fit particularly well if they carry similar emotional connotations [79]. In the same vein, numerous investigations have identified emotional attributions as drivers for the mapping of music with various stimuli from other sensual domains (see [36], for a review).
The influence of expertise and habit
In contrast to previous research (see introduction), we did not discover a difference between experts and novices in terms of their susceptibility to nondiagnostic crossmodal cues. Even though there is no consensus about experts being more or less influenced by background music, it is generally agreed that they are not affected in the same way as novices. We conducted additional explorative analyses to examine if there is a difference between those who regularly listen to music while reading and those who do not. Indeed, the ‘music-readers’ were more strongly influenced by background music than the ‘silence-readers’. This is in contrast with our expectations based on the finding that students are less disturbed by background music if they are used to studying with music [80, 81]. Even though the group of ‘music-readers’ was very small in our sample, we still consider this outcome as highly relevant and worth investigating in future studies. As mentioned before, the deviation of our results from the literature might be explained by the limited comparability of our design with classic crossmodal paradigms, but also by different practices to distinguish the more from the less experienced participants.
Limitations and future directions.
The research design of the present study was well-suited to examine how emotions in texts are related to emotions in the music listened to while reading. However, some methodical limitations should be considered in potential follow-up studies. As the experiment was carried out in a university setting, the sample was rather homogenous in terms of age, education, musical expertise, and reading and music listening behavior. Potentially confounding factors such as psychological and neurological disorders, substance use, and current infections were not assessed, which might be recommended for future samples. Further, a more diverse composition of participants would facilitate the comparison of group differences. In terms of expertise, participants should be specifically recruited as true experts and novices rather than distinguishing them post-hoc by a median split. According to a recent review, music psychologists agree that a musician should be defined by at least six years of musical expertise with at least one hour of practice per week [82]. Only a few of our subjects fell into this category. Beyond expertise and habits, it would be interesting to consider different levels of reading fluency and literacy, other cognitive abilities, as well as personality factors. As to the latter, it has been found that extraverts are less prone to be disturbed by background music than introverts when it comes to fulfilling cognitive tasks (see [19, 83, 84], but also [85, 86]). Moreover, the examination of age differences is promising as it is probable that older adults will differ from younger adults in their way to react to multisensory overload [87, 88]. For practical reasons, we employed a very limited range of stimuli, namely classical string music and American belletristic literature. These were chosen to clearly convey a happy or a sad mood. Future research should seek to diversify the scope of variables to investigate if the results can be generalized to other domains.
On a more technical level, authors interested in conducting studies including musical and reading stimuli, should take into account that different reading speed of participants leads to differing combinations of text passages with musical phrases. In our study, we addressed this issue by cutting and looping relatively uniform music excerpts excluding more important changes in loudness, tempo, instrumentation, or mode. A different approach could be to control the reading speed by presenting single sentences on a screen or using monotonous sound bites rather than music to reduce variance in the auditory domain. However, this was not suitable for our study as we wanted to create a reading and listening situation that is as realistic as possible in a laboratory setting. Further, the question of the auditory delivery is crucial to any crossmodal study design including auditory stimuli. We chose over-ear headphones to approximate everyday reading-while-listening situations and to prevent distraction through external noise. Nevertheless, the influence of music transmitted through loudspeakers and thereby influencing the overall room ambiance would be interesting to examine.
Despite of the described limitations, the present study is of great value and an important addition to the field. Its strengths include its novelty in terms of stimulus combinations, its applicability, and its timely relevance. The design is well-balanced between practical relevance and control of conditions and the results can be regarded as a solid base for further investigations.
Practical implications
The findings presented in this article are especially interesting for the development of digital multimedia tools that create multimodal experiences. For instance, music streaming services might use the insights of our study and potential follow-ups to create specific playlists going with the mood of a given reading content. In this sense, an app could be set such that it instantly adjusts the music background appropriate for the reading content the user has selected. Moreover, the idea of creating specific background music for fiction books is obvious. As we showed that listening to mood-congruent music fosters text immersion, soundtracks for written love stories, dramas, and thrillers would be a genuine enrichment for passionate readers. Transforming the reading of a book into an audiovisual experience might even make it more attractive to those who hitherto prefer to spend their leisure time watching TV and video streaming services.
Conclusion
The emotional appraisal of the mood conveyed by music and text that are presented simultaneously is of crossmodal nature. We show that listening to music while reading a text enhances immersion into the text, but also into the music, if both convey the same mood. That is, we feel more immersed in a sad story when we listen to sad music, and correspondingly, sad music is more moving in combination with a sad text. The same crossmodal amplification is true for happy stimuli. Thus, music and texts are liked better, and they are perceived to fit together when they are mood-congruent. Mood congruency also enhances emotion perception in the stimuli. Sad (happy) texts appear sadder (happier) together with sad (happy) background music and vice versa. These results were, however, not mediated by the valence, arousal, or dominance experienced by the participant, which makes this research stand out from the majority of crossmodal studies. Future research should investigate the generalizability of the discovered phenomenon to other text and music genres. Thereafter, the exploration should be moved from the laboratory to the field to confirm its practical value. Application fields for these findings range from the multisensory enhancement of private settings to the creation of highly immersive multimedia experiences.
Supporting information
S1 Table. List of novel excerpts presented as text stimuli derived from Zupan and Babbage [2].
https://doi.org/10.1371/journal.pone.0280019.s001
(DOCX)
S2 Table. List of music pieces presented as audio stimuli [44, 89–98].
https://doi.org/10.1371/journal.pone.0280019.s002
(DOCX)
S3 Table. Pearson’s correlation coefficients of valence, arousal, and dominance experienced by the participants with perceived text-mood score, text liking, and text immersion.
https://doi.org/10.1371/journal.pone.0280019.s003
(DOCX)
S4 Table. Pearson’s correlation coefficients of valence, arousal, and dominance experienced by the subjects with perceived music-mood score, music liking, and music immersion.
https://doi.org/10.1371/journal.pone.0280019.s004
(DOCX)
S5 Table. Results of a 2 (music category) x 2 (text category) rmANOVA on music-text fit.
https://doi.org/10.1371/journal.pone.0280019.s005
(DOCX)
S6 Table. Multivariate effects and interactions of a 2 (music category) x 2 (text category) rmMANOVA with the between-subjects factor musical expertise on perceived mood score, quality, immersion, and liking of the text.
https://doi.org/10.1371/journal.pone.0280019.s006
(DOCX)
S7 Table. Multivariate effects and interactions of a 2 (music category) x 2 (text category) rmMANOVA with the between-subjects factor reading frequency on perceived mood score, quality, immersion, and liking of the music.
https://doi.org/10.1371/journal.pone.0280019.s007
(DOCX)
S8 Table. Multivariate effects and interactions of a 2 (music category) x 2 (text category) rmMANOVA with the between-subjects factor ‘habit to listen to music while reading’ on perceived mood score, quality, immersion, and liking of the text.
https://doi.org/10.1371/journal.pone.0280019.s008
(DOCX)
S1 Questionnaire. EM–music first.
To evaluate music and text for the group “music first”.
https://doi.org/10.1371/journal.pone.0280019.s009
(PDF)
S2 Questionnaire. ET–text first.
To evaluate music and text for the group “text first”.
https://doi.org/10.1371/journal.pone.0280019.s010
(PDF)
S3 Questionnaire. Post-questionnaire.
To assess music listening and reading behavior and demographic data.
https://doi.org/10.1371/journal.pone.0280019.s011
(PDF)
S4 Questionnaire. Pre-questionnaire.
To assess the prevailing mood before the confrontation with any stimuli as a baseline.
https://doi.org/10.1371/journal.pone.0280019.s012
(PDF)
S1 Data. Data of the experiment.
Dataset before outlier correction.
https://doi.org/10.1371/journal.pone.0280019.s013
(SAV)
S1 Fig.
Boxplots to illustrate participants’ ratings of dependent variables concerning a) music and b) text stimuli.
https://doi.org/10.1371/journal.pone.0280019.s014
(TIF)
S2 File. Syntax containing the computation of new variables.
https://doi.org/10.1371/journal.pone.0280019.s016
(SPS)
Acknowledgments
We thank Agnes Münch for technical support, Ariane Wilhelm and Paavo Käseberg for assisting in recruitment and experimental testing, and Daniel Auras for his help with data entry. The second author (HH) acknowledges the support of the Wissenschaftskolleg zu Berlin (https://www.wiko-berlin.de).
References
- 1.
Juslin PN, Sloboda JA. Handbook of music and emotion. Theory, research, applications. Oxford: Oxford University Press; 2010.
- 2. Zupan B, Babbage DR. Film clips and narrative text as subjective emotion elicitation techniques. J Soc Psychol. 2017; 157:194–210. pmid:27385591.
- 3. Kämpfe J, Sedlmeier P, Renkewitz F. The impact of background music on adult listeners: A meta-analysis. Psychol Music. 2011; 39:424–48.
- 4. Landay K, Harms PD. Whistle while you work? A review of the effects of music in the workplace. Human Resource Management Review. 2018; 29:371–85.
- 5. Kang HJ, Williamson VJ. Background music can aid second language learning. Psychol Music. 2014; 42:728–47.
- 6. Gonzalez MF, Aiello JR. More than meets the ear: Investigating how music affects cognitive task performance. J Exp Psychol Appl. 2019; 25(3):431–444. pmid:30688499
- 7. Reynolds J, McClelland A, Furnham A. An investigation of cognitive test performance across conditions of silence, background noise and music as a function of neuroticism. Anxiety Stress Coping. 2014; 27:410–21. Epub 2013/12/10. pmid:24215564.
- 8. Hallam S, Price J, Katsarou G. The effects of background music on primary school pupils’ task performance. Educational Studies. 2002; 28:111–22.
- 9. Tesoriero M, Rickard NS. Music-enhanced recall: An effect of mood congruence, emotion arousal or emotion function. Music Sci. 2012; 16:340–56.
- 10. Ward EV, Isac A, Donnelly M, van Puyvelde M, Franco F. Memory improvement in aging as a function of exposure to mood-matching music. Acta Psychol (Amst). 2021; 212:103206. Epub 2020/11/12. pmid:33189951.
- 11. Dalton BH, Behm DG. Effects of noise and music on human and task performance: A systematic review. Occupational ergonomics. 2007; 7:143–52.
- 12. Furnham A, Strbac L. Music is as distracting as noise: the differential distraction of background music and noise on the cognitive test performance of introverts and extraverts. Ergonomics. 2002; 45:203–17. pmid:11964204.
- 13. Daoussis L, Mc Kelvie SJ. Musical preferences and effects of music on a reading comprehension test for extraverts and Introverts. Percept Mot Skills. 1986; 62:283–9.
- 14. Zhang H, Miller K, Cleveland R, Cortina K. How listening to music affects reading: Evidence from eye tracking. J Exp Psychol Learn Mem Cogn. 2018; 44:1778–91. pmid:29389184.
- 15. Thompson WF, Schellenberg EG, Letnic AK. Fast and loud background music disrupts reading comprehension. Psychol Music. 2012; 40:700–8.
- 16. Johansson R, Holmqvist K, Mossberg F, Lindgren M. Eye movements and reading comprehension while listening to preferred and non-preferred study music. Psychol Music. 2012; 40:339–56.
- 17. Vasilev MR, Kirkby JA, Angele B. Auditory distraction during reading: A bayesian meta-analysis of a continuing controversy. Perspect Psychol Sci. 2018; 13:567–97. pmid:29958067.
- 18. Christopher EA, Shelton JT. Individual differences in working memory predict the effect of music on student Performance. Journal of Applied Research in Memory and Cognition. 2017; 6:167–73.
- 19. Furnham A, Bradley A. Music while you work: the differential distraction of background music on the cognitive test performance of introverts and extraverts. Appl Cognit Psychol. 1997; 11:445–55.
- 20. Gfeller K, Asmus E, Eckert M. An investigation of emotional response to music and text. Psychol Music. 1991; 19:128–41.
- 21. Kallinen K. Reading news from a pocket computer in a distracting environment: effects of the tempo of background music. Comput Hum Behav. 2002; 18:537–51.
- 22. Mori K, Iwanaga M. Pleasure generated by sadness: Effect of sad lyrics on the emotions induced by happy music. Psychol Music. 2014; 42:643–52.
- 23. Barradas GT, Sakka LS. When words matter: A cross-cultural perspective on lyrics and their relationship to musical emotions. Psychol Music. 2022; 50:650–69.
- 24. Stratton VN, Zalanowski AH. Affective impact of music vs. lyrics. Empir Stud Arts. 1994; 12:173–84.
- 25. Ali SO, Peynircioğlu ZF. Songs and emotions: Are lyrics and melodies equal partners. Psychol Music. 2006; 34:511–34.
- 26. Sousou SD. Effects of melody and lyrics on mood and memory. Percept Mot Skills. 1997; 85:31–40. pmid:9293553.
- 27. Brattico E, Alluri V, Bogert B, Jacobsen T, Vartiainen N, Nieminen S, et al. A functional MRI study of happy and sad emotions in music with and without lyrics. Front Psychol. 2011; 2:308. pmid:22144968.
- 28. Krishna A, Morrin M. Does touch affect taste? The perceptual transfer of product container haptic cues. J Consum Res. 2008; 34:807–18.
- 29. Patston LLM, Tippett LJ. The effect of background music on cognitive performance in musicians and nonmusicians. Music Percept. 2011; 29:173–83.
- 30. Yang J, McClelland A, Furnham A. The effect of background music on the cognitive performance of musicians: A pilot study. Psychol Music. 2016; 44:1202–8.
- 31. Wu C-C, Shih Y-N. The effects of background music on the work attention performance between musicians and non-musicians. Int J Occup Saf Ergon. 2021; 27:201–5. Epub 2019/02/13. pmid:30758255.
- 32. Spence C. Crossmodal correspondences: A tutorial review. Atten Percept Psychophys. 2011; 73:971–95. pmid:21264748
- 33. Collier GL. Affective synesthesia: Extracting emotion space from simple perceptual stimuli. Motiv Emot. 1996; 20:1–32.
- 34. Palmer SE, Langlois TA, Schloss KB. Music-to-color associations of single-line piano melodies in non-synesthetes. Multisens Res. 2016; 29:157–93. pmid:27311295
- 35. Palmer SE, Schloss KB, Xu Z, Prado-León LR. Music-color associations are mediated by emotion. Proc Natl Acad Sci U S A. 2013; 110:8836–41. pmid:23671106.
- 36. Spence C. Assessing the role of emotional mediation in explaining crossmodal correspondences Involving Musical Stimuli. Multisens Res. 2020; 33:1–29. pmid:31648195.
- 37. Schifferstein HNJ, Tanudjaja I. Visualising fragrances through colours: The mediating role of emotions. Perception. 2004; 33:1249–66. pmid:15693669.
- 38. Lindborg P, Friberg AK. Colour association with music is mediated by emotion: evidence from an experiment using a CIE lab interface and interviews. PLoS One. 2015; 10:e0144013. pmid:26642050.
- 39. Wang QJ, Spence C. Assessing the role of emotional associations in mediating crossmodal correspondences between classical music and red wine. Beverages. 2017; 3:1.
- 40. Velasco C, Woods AT, Deroy O, Spence C. Hedonic mediation of the crossmodal correspondence between taste and shape. Food Qual Prefer. 2015; 41:151–8.
- 41. Wright O, Jraissati Y, Özçelik D. Cross-modal associations between color and touch: Mapping haptic and tactile terms to the surface of the Munsell Color Solid. Multisens Res. 2017; 30:691–715.
- 42. Imbir K, Gołąb M. Affective reactions to music: Norms for 120 excerpts of modern and classical music. Psychol Music. 2017; 45:432–49.
- 43. Lench HC, Flores SA, Bench SW. Discrete emotions predict changes in cognition, judgment, experience, behavior, and physiology: a meta-analysis of experimental emotion elicitations. Psychol Bull. 2011; 137:834–55. pmid:21766999.
- 44. Krumhansl CL. An exploratory study of musical emotions and psychophysiology. Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale. 1997; 51:336–53. pmid:9606949.
- 45. Siedlecka E, Denson TF. Experimental methods for inducing basic emotions: A qualitative review. Emotion Review. 2019; 11:87–97.
- 46. Hsu C-T, Jacobs AM, Citron FMM, Conrad M. The emotion potential of words and passages in reading Harry Potter—an fMRI study. Brain Lang. 2015; 142:96–114. Epub 2015/02/11. pmid:25681681.
- 47. Proverbio A. M., & Russo F. Multimodal recognition of emotions in music and language. Psychol Music. 2022; 50:54–68.
- 48. Bestgen Y. Can emotional valence in stories be determined from words. Cogn Emot. 1994; 8:21–36.
- 49. Vuoskoski JK, Eerola T. The role of mood and personality in the perception of emotions represented by music. Cortex. 2011; 47:1099–106. Epub 2011/04/30. pmid:21612773.
- 50. Cohrdes C, Wrzus C, Wald-Fuhrmann M, Riediger M. “The sound of affect”: Age differences in perceiving valence and arousal in music and their relation to music characteristics and momentary mood. Music Sci. 2020; 24:21–43.
- 51. Mills C, Wu J, D’Mello S. Being sad is not always bad: The influence of affect on expository text comprehension. Discourse Processes. 2019; 56:99–116.
- 52. Bohn-Gettler CM, Rapp DN. Depending on my mood: Mood-driven influences on text comprehension. J Educ Psychol. 2011; 103:562–77. pmid:21927504.
- 53. Trevors G, Bohn-Gettler C, Kendeou P. The effects of experimentally induced emotions on revising common vaccine misconceptions. Q J Exp Psychol (Hove). 2021; 74:1966–80. Epub 2021/05/15. pmid:33926324.
- 54. Scrimin S, Mason L. Does mood influence text processing and comprehension? Evidence from an eye-movement study. Br J Educ Psychol. 2015; 85:387–406. Epub 2015/05/23. pmid:26010020.
- 55. Schaal NK, Bauer A-KR, Müllensiefen D. The Gold-MSI: Replication and validation of a questionnaire instrument for the measurement of musical sophistication on the basis of a German sample. Music Sci. 2014; 18:423–47.
- 56. Bradley MM, Lang PJ. Measuring emotion. The Self-Assessment Manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994; 25:49–59.
- 57. Västfjäll D, Larsson P, Kleiner M. Emotion and auditory virtual environments. Affect-based judgments of music reproduced with virtual reverberation times. Cyberpsychol Behav. 2002; 5:19–32. pmid:11990972.
- 58. Imschloss M, Kuehnl C. Feel the music! Exploring the cross-modal correspondence between music and haptic perceptions of softness. J Retail. 2019; 95:158–69.
- 59. Hauck P, Hecht H. Having a drink with Tchaikovsky: the crossmodal influence of background music on the taste of beverages. Multisens Res. 2019; 32:1–24. pmid:31059491
- 60. Baumgartner T, Lutz K, Schmidt CF, Jäncke L. The emotional power of music: How music enhances the feeling of affective pictures. Brain Res. 2006; 1075:151–64. pmid:16458860
- 61. Galmarini MV, Silva Paz RJ, Enciso Choquehuanca D, Zamora MC, Mesz B. Impact of music on the dynamic perception of coffee and evoked emotions evaluated by temporal dominance of sensations (TDS) and emotions (TDE). Food Res Int. 2021; 150. Epub 2021/11/02. pmid:34865810.
- 62. Ehret S, Schroeder C, Bernet J, Holzmüller A, Thomaschke R. All or nothing: The interaction of musical and spatial atmosphere. Psychol Music. 2021; 49:513–28.
- 63. Yamasaki T, Yamada K, Laukka P. Viewing the world through the prism of music. Effects of music on perceptions of the environment. Psychol Music. 2013; 43:61–74.
- 64. Mirabella G. The Weight of Emotions in Decision-Making: How Fearful and Happy Facial Stimuli Modulate Action Readiness of Goal-Directed Actions. Front Psychol. 2018; 9:1334. Epub 2018/08/02. pmid:30116211.
- 65. Kiss M, Raymond JE, Westoby N, Nobre AC, Eimer M. Response inhibition is linked to emotional devaluation: behavioural and electrophysiological evidence. Front Hum Neurosci. 2008; 2:13. Epub 2008/10/03. pmid:18958213.
- 66. Rozin P, Royzman EB. Negativity bias, negativity dominance, and contagion. Pers Soc Psychol Rev. 2001; 5:296–320.
- 67.
Stein BE, Meredith MA. The merging of the senses. The MIT Press; 1993.
- 68.
Welch RB, Warren DH. Intersensory interactions. In: Boff KR, editor. Handbook of perception and human performance. New York: Wiley; 1986.
- 69. Winkielman P, Ziembowicz M, Nowak A. The coherent and fluent mind: how unified consciousness is constructed from cross-modal inputs via integrated processing experiences. Front Psychol. 2015; 6:83. Epub 2015/02/09. pmid:25741297.
- 70.
Kahneman D. Attention and Effort. Englewood Cliffs: Prentice Hall; 1973.
- 71. North AC, Sheridan LP, Areni CS. Music congruity effects on product memory, perception, and choice. J Retail. 2016; 92:83–95.
- 72. Seo H-S, Lohse F, Luckett CR, Hummel T. Congruent sound can modulate odor pleasantness. Chem Senses. 2014; 39:215–28. pmid:24368256.
- 73. Reber R, Winkielman P, Schwarz N. Effects of perceptual fluency on affective judgments. Psychol Sci. 1998; 9:45–8.
- 74. Steffens J. The influence of film music on moral judgments of movie scenes and felt emotions. Psychol Music. 2020; 48:3–17.
- 75. Kock M, Louven C. The power of sound design in a moving picture: an empirical study with emoTouch for iPad. Empir Musicol Rev. 2018; 13:132–48.
- 76. Lipscomb SD, Kendall RA. Perceptual judgement of the relationship between musical and visual components in film. Psychomusicology. 1994; 13:60–98.
- 77. Hoeckner B, Wyatt EW, Decety J, Nusbaum H. Film music influences how viewers relate to movie characters. Psychol Aesthet Crea. 2011; 5:146–53.
- 78. Boltz MG. The cognitive processing of film and musical soundtracks. Mem Cognit. 2004; 32:1194–205. pmid:15813500.
- 79. Hauck P, von Castell C, Hecht H. Crossmodal Correspondence between Music and Ambient Color Is Mediated by Emotion. Multisens Res. 2022; 35(5):407–446. pmid:35985652
- 80. Etaugh C, Michals D. Effects on reading comprehension of preferred music and frequency of studying to music. Percept Mot Skills. 1975; 41:553–4.
- 81. Etaugh C, Ptasnik P. Effects of studying to music and post-study relaxation on reading comprehension. Percept Mot Skills. 1982; 55:141–2.
- 82. Zhang JD, Susino M, McPherson GE, Schubert E. The definition of a musician in music psychology: A literature review and the six-year rule. Psychol Music. 2020; 48:389–409.
- 83. Dobbs S, Furnham A, McClelland A. The effect of background music and noise on the cognitive test performance of introverts and extraverts. Appl Cognit Psychol. 2011; 25:307–13.
- 84. Cassidy G, MacDonald RA. The effect of background music and background noise on the task performance of introverts and extraverts. Psychol Music. 2007; 35:517–37.
- 85. Kou S, McClelland A, Furnham A. The effect of background music and noise on the cognitive test performance of Chinese introverts and extraverts. Psychol Music. 2018; 46:125–35.
- 86. Avila C, Furnham A, McClelland A. The influence of distracting familiar vocal music on cognitive performance of introverts and extraverts. Psychol Music. 2012; 40:84–93.
- 87. Carrier LM, Cheever NA, Rosen LD, Benitez S, Chang J. Multitasking across generations: Multitasking choices and difficulty ratings in three generations of Americans. Computers in Human Behavior. 2009; 25:483–9.
- 88. Commodari E, Guarnera M. Attention and aging. Aging Clin Exp Res. 2008; 20:578–84. pmid:19179843.
- 89. Jefferies LN, Smilek D, Eich E, Enns JT. Emotional valence and arousal interact in attentional control. Psychol Sci. 2008; 19:290–5. pmid:18315803
- 90. Cook JW, Spring B, McChargue D. Influence of nicotine on positive affect in anhedonic smokers. Psychopharmacology (Berl). 2007; 192:87–95. Epub 2007/02/03. pmid:17277935.
- 91. Konečni VJ, Brown A, Wanic RA. Comparative effects of music and recalled life-events on emotional state. Psychol Music. 2008; 36:289–308.
- 92. Eich E, Metcalfe J. Mood dependent memory for internal versus external events. J Exp Psychol Learn. 1989; 15:443–55.
- 93. Gorn G, Pham MT, Sin LY. When arousal influences ad eEvaluation and valence does not (and vice versa). J Consum Psychol. 2001; 11:43–55.
- 94. Martin MA, Metha A. Recall of early childhood memories through musical mood induction. The Arts in Psychotherapy. 1997; 24:447–54.
- 95. Barbiere JM, Vidal A, Zellner DA. The color of music. Correspondence through emotion. Empir Stud Arts. 2016; 25:193–208.
- 96. Cunningham JG, Sterling RS. Developmental change in the understanding of affective meaning in music. Motiv Emot. 1988; 12:399–413.
- 97. Roisman GI, Fortuna K, Holland A. An experimental manipulation of retrospectively defined earned and continuous attachment security. Child Dev. 2006; 77:59–71. pmid:16460525.
- 98. Schubert E. The influence of emotion, locus of emotion and familiarity upon preference in music. Psychol Music. 2007; 35:499–515.