Influence of music on cognitive performance

The influence of music on cognitive performance has been extensively investigated. Evidence about the disturbing, enhancing, or neutral nature of this influence, however, is mixed (see [3], for a meta-analysis). Important factors that have been identified in this context are task difficulty [4], music complexity [5], and the personal preference for external stimulation, along with extraversion [6, 7]. As to emotional factors, arousal and mood conveyed by the music have been shown to differentially influence memory performance [8, 9]. In a recent study, Ward et al. [10] found that mood-matching music can improve recall in older adults. However, there is still no consensus about the distracting or supporting impact of music on cognitive performance, and a closer look at the literature reveals a number of gaps and shortcomings. We want to address the issue with the aim of closing one of these gaps by specifically concentrating on the influence of music on emotion perception while reading narrative texts.

Influence of background music on the reading process

Several studies have examined the influence of music on cognitive aspects of the reading process, such as reading speed, fluency, and comprehension. Findings, however, are inconsistent. For instance, music has mostly been found to distract the reader, but it can also promote concentration (see [4, 11], for reviews). Several studies demonstrate that text comprehension is impaired by background music or noise when compared to reading in silence [12, 13]. Employing an eye-tracking approach, Zhang et al. [14] confirmed this disrupting effect, showing that background music is associated with more re-reading of the text. However, not every music piece has the same impact on the reading process. Thompson et al. [15] discovered that loud and fast background music led to reduced text comprehension, while soft and slow music did not make a difference in this regard. Moreover, the distracting effect has been shown to depend on whether or not the music was preferred as study music [16]. Participants scored significantly lower in a reading comprehension test when they had listened to non-preferred study music while reading than when they had read in silence. Music previously categorized as preferred study music did not have this effect. As to the differentiation of music types, Vasilev et al. [17] concluded in their meta-analysis that background noise, music, and speech have a stronger distracting effect on reading performance when background speech contains meaningful content and when music has meaningful lyrics. However, their analysis found small but stable adverse effects for all kinds of background sounds, which was true for adults as well as for children. Besides the influence of tempo, loudness, preference, type, and other characteristics of the music, interindividual differences have also been shown to affect the strength of the music’s impact on reading performance. As such, extraverts and persons with higher working memory capacity seem to be less disturbed by background music than introverts and those with lower working memory capacity [18, 19].

Emotion perception in music and text

In contrast to the relatively well-studied influence of music and noise on the reading process, there are no reports in the literature on whether music also changes the emotional evaluation of the reading content, and vice versa. We therefore merely provide a brief overview of studies addressing related research questions.

Gfeller et al. [20] investigated the affective response to music, text, or the combination of both. In their design, both text and music stimuli were presented acoustically, as they employed a recorded recitation of a poem and recordings of commercial-style or atonal music. Their results show that some emotional ratings changed when music was combined with text, as compared to music only or text only. For instance, the text was liked better when combined with popular commercial-style music than in combination with unpopular atonal music. The authors conclude that listening to music can influence our emotional response to simultaneously received texts.

Another study on emotional factors in the context of reading with background music by Kallinen [21] showed that valence assessment of texts depended on music tempo. Focusing on gender effects, they found that men rated news texts to be more positive when listening to fast rather than slow background music, whereas women’s ratings were most positive in the no-music condition. Unfortunately, the researchers employed only one music piece with different tempo modifications instead of different music stimuli, so it is difficult to generalize their results.

Turning the tables, some attempts have been made to explore the inverse emotional impact of text in the form of lyrics on music perception [22, 23]. In an early work, Stratton and Zalanowski [24] revealed a relative dominance of emotions conveyed by the lyrics compared to emotions carried by the music. More recently, Ali and Peynircioğlu [25] demonstrated that sad music with sad lyrics is perceived as sadder than sad music without lyrics. Interestingly, happy music with happy lyrics was rated as less happy compared to happy music without lyrics. The authors concluded that congruent lyrics detract from and thus attenuate happy emotions, while they bolster sad emotions conveyed by the music. In contrast to the findings by Stratton and Zalanowski [24], Ali and Peynircioğlu found that the emotions carried by the music are dominant when music and lyrics convey incongruent emotions (see also [26]). In agreement with the latter results, Brattico et al. [27] showed that happy music without lyrics activates the limbic system whereas happy music with lyrics is reflected in auditory regions only. Except for the described research on music and lyrics, an examination of the influence of nonmusical text on music perception has not yet been made.

Influence of expertise on multisensory perception

In multisensory research, it is of great interest to examine differences between experts and novices. For instance, it has been shown that persons who scored higher on a test that assessed their touching expertise by measuring ‘need for touch’ were less susceptible to the nondiagnostic haptic experience of a container when rating the gustatory qualities of its contents [28]. In the context of music and reading, there are mixed results regarding the impact of musical expertise on cognitive tasks. Patston and Tippett [29] found that musicians but not non-musicians were disturbed by background music in the completion of a language comprehension task (see also [30]). However, musicians benefit more from soft background music in terms of attention performance than do non-musicians [31]. As there is no clear evidence about musical expertise as a factor in the susceptibility to background music, we want to address this issue in the present research.

Emotional mediation of multisensory processes

In the literature on multisensory associations or crossmodal correspondences [32], it is widely acknowledged that the emotional state a person experiences during the reception of the multisensory input plays an important role. When stimuli from one sensual modality influence the perception in another, or when multisensory stimuli are consistently associated with each other, emotional states have often been found to explain the effects [33, 34]. The emotion mediation hypothesis [35] suggests that stimuli which trigger the same emotion when presented separately, are more likely to be perceived as fitting together (see [36], for a review). This connection was demonstrated for associations among, inter alia, odor and color [37], music and color [38], music and taste [39], taste and shape [40], and touch and color [41].

As we want to examine the potential influence of music mood on perceived text mood and vice versa, we must separately consider the emotions attributed to the stimuli on the one hand, and the recipient’s mood state on the other hand. Only if both affective dimensions are independently measured, can emotional mediation be examined. In our study, the stimulus attributes music mood and text mood were assessed in a pre-test with regard to the extent to which they conveyed a happy or a sad mood. The respective texts and music pieces were then categorized as ‘happy’ or ‘sad’. The dependent variables ‘perceived text mood’ and ‘perceived music mood’ were measured by numeric rating scales. We employed a pictorial scale to determine the mood state of the participant. According to the emotion mediation hypothesis, the independent mood variables should lead to a change in experienced emotions, which in turn would correlate with the dependent mood measure. That is, sad music would make the participant feel sadder, which in turn would lead to higher sadness-ratings of the text, and vice versa.

Inducing emotions through music and texts

Numerous studies have shown that music reliably evokes emotions in the listener [1, 42]. This has proven very useful when it comes to the focused elicitation of specific emotions as an experimental method [43, 44]. Especially for happiness and sadness, playing music to participants turned out to be a very effective induction method [45]. Likewise, narrative texts have been used to induce emotions in experimental studies [2], as speech per se is known to easily convey mood, be it via single words, sentences, or whole stories [46–48].

Emotions influence music and text perception

Research on the impact of mood on emotion perception in texts or music has received only limited attention so far. As to music, Vuoskoski and Eerola [49] found that participants’ current mood is associated with mood-congruent music ratings. However, this effect was moderated by extraversion, suggesting a complex relationship between mood and personality. Another study reported distinct age differences in the influence of listener mood on affect perception in music. Younger adults’ perception of negative valence in the music was mood-congruent, whereas it was mood-incongruent for older adults. Arousal perception in the music was mood-congruent for both age groups, but the effect was much more pronounced in younger participants [50]. In the context of text perception, the prevailing mood has thus far only been considered in terms of reading performance. Mills et al. [51] showed that a sad mood leads to more profound text comprehension (deep-reasoning) compared to a happy mood (see also [52, 53]). Opposite results were found by Scrimin and Mason [54], whose participants showed better text comprehension and better learning of factual knowledge when they were in a positive mood. Apparently, evidence about the connection between the receiver’s emotions and music and text perception is still limited. Exploring this relationship will be part of this study.

Our study

The main objective of this work is to investigate the relationship between perceived emotions in music and text when simultaneously consumed. We want to examine if a distinct mood in the music is reflected in the mood evaluation of a text being read at the same time, and vice versa. We are also interested in exploring a potential influence of emotions conveyed by music or text on the perceived music-text fit, as well as on the receivers’ felt immersion into the music or the text. As we are interested in both directions of the potential crossmodal effects, we applied a symmetrical research design. A secondary aim of the study is to regard the impact of relative expertise on multisensory transfer and association. We furthermore examine the assumption of an emotional mediation for all mentioned effects.

We hypothesized that the emotional mood of the music (or the text) influences the perceived mood conveyed by the text (or the music) by way of an emotional transfer. We assumed that this effect is less pronounced in experts than in novices.

To test these hypotheses, we performed two pre-tests and one laboratory experiment with three separate samples. The pre-tests validated the emotional value of music pieces and narrative text excerpts and were the basis to choose the stimuli for the main experiment. In a fully crossed within-subjects-design, participants were asked to read the texts while listening to the music and to rate both text and music in terms of the conveyed emotional mood (happy vs. sad), artistic quality, perceived immersion, and general liking. We also assessed participants’ momentary emotional state by measuring valence, arousal, and dominance at the beginning of the experimental session and after every trial.

Sample

40 students (31 female, 9 male; M_age = 23.95, SD = 4.27) volunteered to participate in the experiment, amongst which 39 were psychology students and one studied another subject. As it is mandatory for psychology students at the Johannes Gutenberg University Mainz to participate in studies conducted in the department, most of them had prior experience with cognitive experiments. However, there was no overlap or potential interference with designs tested here. All of them gave written informed consent in accordance with the Declaration of Helsinki. 38 participants had self-reported normal or corrected-to-normal vision and hearing; only two were slightly myopic and did not wear glasses or contact lenses. However, their data did not deviate substantially from those of the rest of the sample, so they were included in the subsequent analyses. All participants reported being non-synesthetes. As all subjects were university students in good standing, we assumed that their cognitive performance was not impaired. Thus, we did not carry out cognitive tests, nor did we test them for previous substance use. Musical expertise was assessed by the Goldsmith Musical Sophistication Index Version 1, a self-report inventory for individual differences in musical sophistication (Gold-MSI; [55]). It measures the ability to engage with music independently of musical style, preferences, or regularity of musical practice. In order to avoid overloading the questionnaire, we picked 18-items assessing the General Factor of Musical Sophistication. In our sample, the mean General Musical Sophistication Score (GMS) was below average compared to the norm (M_GMS = 68.97, SD_GMS = 23.13; 28^th percentile; norm: M_GMS = 81.58, SD_GMS = 20.62). One participant scored extremely high (GMS = 122); however, her data did not stand out and were thus not dismissed. For analysis, we performed a median split to divide the sample into more (experts) and less (novices) musically experienced participants (median _GMS = 69.25). We identified extreme outlier values in terms of music and text ratings with the help of boxplots (criterion: > 3 * IQR from 1^st or 3^rd quartile) and replaced these outliers by the group mean. In total, this was the case for five data points representing 0.21% of all ratings. In addition to the Gold-MSI questionnaire, we had asked for daily music listening and reading routines. Most of the sample reported listening to music (almost) every day (n = 35), the remaining participants indicated to do so less frequently. In terms of reading, 15 participants reported to read (almost) every day, 10 read several times a week, 13 read several times a month and two participants indicated to (almost) never read for pleasure. 6 participants found reading (more or less) tiring, and one participant indicated only to read when necessary. Most of the sample (n = 33) reported that they did (rather) not like to listen to music while reading.

As the group of those used to simultaneous reading and listening to music was very small (n = 7), we could only exploratively examine if habituation might be an influencing factor for the crosstalk between music and text. The experiment was conducted in accordance with the Declaration of Helsinki. Informed consent was obtained before the experimental session, subjects were told that they could abort the experiment at any time without consequences, and they were debriefed about the purposes of the study immediately after the session. The study was approved by a local Departmental Ethics committee under a blanket approval.

Apparatus and stimuli

Setup.

The experiment took place in a darkened laboratory room equipped with a table and an office chair. The table was standing in front of a wall of LED panels with movable walls to the left and right of the table so that the walls constituted an open booth (see Fig 1). The LED panels emitted slightly reddish light (RGB: 255, 55, 0), creating a comfortable, warm atmosphere. An additional desk lamp on the table provided customary reading light. Noise-canceling headphones lay ready on the table.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The experimental setup.

The LED wall emitted a slightly reddish light creating a comfortable atmosphere, a table lamp facilitated reading in the otherwise dark room.

https://doi.org/10.1371/journal.pone.0280019.g001

Design and procedure

We applied a fully crossed 2 (music mood) x 2 (text mood) within-subjects design, resulting in four experimental conditions. To avoid the repetition of a text or music piece for any given subject, there were two texts and two music pieces per mood (two happy, two sad each). In order to counterbalance potential order effects, all possible stimulus orders were presented to the same number of subjects. The four combinations per condition resulted in eight possible permutations of the four conditions, each presented to 5 of the 40 participants. Their assignment to a given stimulus order was randomized.

Participants were tested individually. First, their prevailing mood was assessed through the SAM scales (see stimuli section), then they were asked to put on the headphones and read the first text. The music started as soon as they began reading. Participants were told to focus on the text, just as if they were reading for pleasure in their leisure time. They were instructed to continue reading for as long as the music would play, and start from the beginning of the text in case they would have finished reading it before the end of the music. With this instruction, we ensured that every participant read the full text and heard the entire music piece. When the music stopped, participants filled in the SAM scales again, followed by a questionnaire measuring their liking of the text (‘How do you like the text?’), their evaluation of its artistic quality (‘How would you rate the artistic level of the text?’), their perceived immersion (‘How much did the text intrigue you?’), and their assessment of the text’s emotional mood (‘How would you rate the emotional mood of the text?’). Afterward, the same set of questions was filled in to evaluate the music, supplemented by an indication of familiarity with the piece (‘How well did you know the music before this experiment?’), and followed by a question on the perceived fit of text and music (‘How would you personally rate the fit between the text and the music?’). All constructs, except for emotional mood, were measured on 10-point scales from 0 (‘not at all’) to 9 (‘very much’). Like in the pretests, the character of the emotional mood was rated on a bipolar 11-point scale from -5 (sad) to +5 (happy), followed by the measurement of emotional intensity on the noted 10-point scale. Half of the participants first rated the texts and then the music, while the other half received the questionnaires in reverse order. Reading speed was measured with a stopwatch. The whole sequence was repeated for all four experimental conditions.

At the end of the experiment, participants filled in a final questionnaire (see Supplementary Material) assessing demographic data as well as different control variables, namely eyesight, hearing, synesthesia, gender, age, profession, highest completed level of education, reading habits, music listening habits, reasons for listening to music, activities combined with listening to music, attitudes towards reading, and the short version of the Gold-MSI [55] measuring musical expertise. The experimental session ended after approx. 35 minutes. The temporal sequence of the experimental session can be seen in Fig 2. Data were collected between July 29th 2020 and January 17th 2021.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Temporal sequence of the components of one experimental session.

Note that the flow chart is exemplary for 50% of the participants that were instructed to rate first text then music. The other 50% of the participants started with music ratings. Participants were randomly assigned to the ‘text first’ or ‘music first’ groups.

https://doi.org/10.1371/journal.pone.0280019.g002

Influence of music mood on text perception

We calculated a 2 (music category) x 2 (text category) repeated measures multivariate analysis of variance (rmMANOVA) for the dependent variables perceived mood score, quality, immersion, and liking of the text. The factors ‘music category’ and ‘text category’ refer to the classification as ‘sad’ or ‘happy’ according to the pre-experiments (see methods section). Both multivariate main effects and the multivariate interaction were significant (see Table 1 for the complete results). Post-hoc univariate rmANOVAS (LSD-corrected) showed that the main effect of text category was significant for perceived text-mood score, immersion, and liking, but not for quality. As expected, happy texts were perceived as conveying a happier mood, they were liked better, but they were less immersing than sad texts. The main effect of music category was significant for perceived mood score, that is, texts were rated as conveying a happier mood with happy background music than with sad background music. There were no significant univariate effects of music category for text quality, immersion, and liking. Univariate interactions of music category x text category were significant for text quality, immersion, and liking, but not for text-mood score. As illustrated in Fig 3, texts were perceived as higher in quality, as more immersing, and they were liked better when read under congruent music. All effects were clearly more pronounced for sad compared to happy texts.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3.

Means for perceived mood score (a), artistic quality (b), immersion (c), and liking (d) of the text as a function of happy vs. sad music and text categories. Error bars show ± 1 SEM.

https://doi.org/10.1371/journal.pone.0280019.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Results of a 2 (music category) x 2 (text category) rmMANOVA on perceived mood score, quality, immersion, and liking of the text.

Multivariate and univariate main effects and interactions.

https://doi.org/10.1371/journal.pone.0280019.t001

We then tested if the found effects of music mood on text evaluation were mediated by the experienced emotions of the participant. As mentioned above, experienced emotions were measured according to their valence, dominance, and arousal at the beginning of the experiment (baseline) and after every trial. For analysis, we subtracted all values from the baseline measurement and hence displayed the relative change in the emotional state throughout the experimental session. Then, we first calculated a 2 (music category) x 2 (text category) rmMANOVA on valence, arousal, and dominance, and, second, examined the interrelation of the significant emotion measures with the different text evaluation variables by calculating Pearson correlations.

The rmMANOVA revealed a significant main effect for music category, thus the stimulus mood did influence the participants’ emotional condition, F(3, 36) = 4.5, p = .009, ⴄ² = .27. Post-hoc univariate rmANOVAs showed that the effect was driven by changes in valence and dominance ratings, while arousal was not significantly affected (valence: F(1, 38) = 11.25, p = .002, ⴄ² = 0.228; dominance: F(1, 38) = 5.74, p = .022, ⴄ² = .131; arousal: F(1, 38) = 0.85, p = .364, ⴄ² = 0.022). In the next step, however, valence and dominance did not reveal any relation to text evaluation: Among all Pearson correlations between valence and perceived text mood, only one out of four possible music-mood-text-mood-combinations was significant (sad music with happy texts, r(38) = .32, p = .042). In addition, for none of the four combinations did we find a significant correlation between dominance and text mood, nor between valence or dominance and liking of the text. Nevertheless, text immersion correlated positively with valence for happy-music-happy-text-combinations, and negatively for happy-music-sad-text-, and sad-music-sad-text-combinations (see S3 Table, for all correlation values). Data indicated that valence mediates the effect of music mood on text immersion.

Influence of text mood on music perception

To examine the reverse influence of text mood on music perception, we calculated a 2 (music category) x 2 (text category) rmMANOVA for the dependent variables perceived mood score, quality, immersion, and liking of the music. Both main effects of music category and text category were significant, as well as the respective interaction (see Table 2, for the complete results). Univariate rmANOVAS showed that text category significantly influenced the perceived mood score of the music. Music-mood was rated as happier when participants were reading a happy text while listening. The univariate interaction of music category x text category was significant for liking and immersion of the music. Music was liked better while reading a congruent compared to an incongruent text. Similarly, music was rated as more immersing when presented simultaneously with a congruent rather than an incongruent text. As illustrated in Fig 4, this effect was stronger for sad than for happy music.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4.

Means for perceived mood score (a), artistic quality (b), immersion (c), and liking (d) of the music as a function of happy vs. sad music and text categories. Error bars show ± 1 SEM.

https://doi.org/10.1371/journal.pone.0280019.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Results of a 2 (music category) x 2 (text category) rmMANOVA on perceived mood score, quality, immersion, and liking of the music.

Multivariate and univariate main effects and interactions.

https://doi.org/10.1371/journal.pone.0280019.t002

To test the emotional mediation of the influence of text mood on music perception, we mirrored the analytical procedure described above for the reverse effect. The rmMANOVA on emotional measures revealed a main effect of text category, F(3, 36) = 14.06, p < .001, ⴄ² = .053, which showed significance in all three univariate analyses, indicating that the stimulus mood had an influence on the participants’ emotional state (valence: F(1, 38) = 33.78, p < .001, ⴄ² = .471; arousal: F(1, 38) = 11.23, p = .002, ⴄ² = .228; dominance: F(1, 38) = 29.28, p < .001, ⴄ² = .435). Sad texts were associated with lower valence, but higher arousal and dominance ratings compared to happy texts. However, there were no significant correlations of valence, arousal, and dominance ratings with perceived music-mood score for none of the four music-text-mood-combinations. For sad-music-sad-text-combinations, there was a significant positive correlation between arousal and music liking and a significant negative correlation between valence and music immersion (see S4 Table, for all correlation values). A general mediation by emotion ratings could not be observed.

Music-text fit

To test if music and text are more likely to be perceived to fit together when they convey the same mood, we pursued two alternative approaches. First, we inspected the interrelation between our stimulus mood categories (happy vs. sad) and music-text fit ratings. Therefore, we calculated a 2 (music category) x 2 (text category) rmANOVA with music-text fit as the dependent variable, and found a strong music category x text category interaction, F(1, 39) = 111.55, p < .001, ⴄ² = .741. Fit ratings were higher when music and text categories corresponded. Complete results can be retrieved from S5 Table.

In a second approach, we took participants’ mood ratings as a reference and determined the absolute difference between music-mood ratings and text-mood ratings for each combination of music and text categories. We then calculated Pearson correlations between this difference value and fit ratings. All four correlations were negative, with significant negative correlations for happy-music-sad-text combinations, r(38) = –.33, p = .039, and for sad-music-happy-text combinations, r(38) = –.56, p < .001. The more similar the respective moods conveyed by text and music were rated, that is, the smaller the perceived mood difference, the better music and text were perceived to fit together.

Exploration of the influence of habituation and expertise on the crosstalk between music and text

In our post-questionnaire, we collected data on musical and reading expertise, as well as music listening and reading habits. We used these data to explore if expertise and habituation had an impact on the found crossmodal effects.

The habit of listening to music while reading.

We had asked participants to indicate their degree of agreement or disagreement with the statement ‘I like to listen to music while reading’. Most of the participants (n = 33 out of 40) said (rather) not to do so. Even though the remaining group of ‘music-readers’ was very small, we were interested in potential differences between the ‘music-readers’ and the ‘silence-readers’ in terms of music- and text-mood interplay. Thus, in an explorative approach, we calculated the 2 (music category) x 2 (text category) rmMANOVA for perceived mood score, quality, immersion, and liking of the text again, now with the between-subjects-factor habit. We carried out an analogous rmMANOVA with the same factors and dependent variables referring to music perception. There was a significant effect of habit for text mood, F(1, 38) = 5.525, p = .024, ⴄ² = .0127. As can be seen in Fig 5, text-mood ratings by music-readers were more strongly influenced by the music than those of silence-readers. There were no other significant between-subjects effects for habit, nor an interaction of habit with the main factors (see S8 Table).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Mean ratings for text mood score as a function of the habit to listen to music while reading.

Error bars show ± 1 SEM.

https://doi.org/10.1371/journal.pone.0280019.g005

Multisensory transfer of emotional attributions

As expected, our findings indicate that happy or sad mood is transferred from music to texts and vice versa. That is, sad (happy) music makes written stories appear sadder (happier), and sad (happy) stories make music sound sadder (happier). This observation agrees well with numerous studies in the field of multisensory research that have examined the influence of music on the perception of diverse sensual stimuli. For instance, it has been found that listening to ‘soft’ music increases haptic softness perception [58], listening to ‘light’ music makes red wine taste lighter [59], music can enhance the emotions conveyed by pictures [60], and coffee tastes less bitter in the presence of ‘sweet’ music [61]. In a recent study by Ehret et al. [62], musical attributions were also shown to affect the perceived overall valence of the atmosphere. Accordingly, Yamasaki et al. [63] demonstrated that the evaluation of the environment can be shifted in the direction of specific characteristics of the music. Our results integrate these previous outcomes, as they demonstrate the mutual impact of auditory input on visually received information, and vice versa, which then influence the overall impression of a fictive environment and atmosphere.

Previous authors have often solicited emotion as the mediator and key to deciphering the crossmodal transfer of stimulus characteristics (see introduction). Our research disagrees with this interpretation, as emotional mediation did not explain the effects of crossmodal transfer and association in the case of music and text. However, the comparability of study designs is limited. Narrative texts are complex stimuli and the task of rating the conveyed mood of the text implies not only visual processing but also the cognitive effort of reading. In contrast, it is common in the field of multisensory research to employ simple stimuli like color patches, single tones, and basic taste samples. With our approach, we offer a first assessment of crossmodal emotional mediation involving texts as stimuli.

Regarding the underlying emotions, we can conclude that music and text did not primarily affect the participants’ mood but rather did the music impinge on the assessment of the text. And vice versa did the text alter the appraisal of the music. Thus, we can assume that we found rather pure appraisal effects here. We may ask how likely such intersensory appraisal changes are to influence our actions. Negative emotions were recently shown to influence our actions, specifically our inhibitory control [64, 65]. Note, however, that the crossmodal appraisal effect in the current study are once removed from the actor’s emotional state. We can thus only speculate if negative emotions identified in a given text or music piece will impact our behavior. Further studies building on this hypothesis would be a promising path forward.

In our paradigm, most effects were more pronounced for sad than for happy stimulus moods. This is once more in line with the insights by Ehret and colleagues [62], who revealed that adding a negative aspect to a positive ambiance has a bigger impact on the overall evaluation of the atmosphere than does add a positive aspect to a negative status quo. This pattern might be explained by a general negativity bias, suggesting that negative information is given more weight than positive information, it is processed faster and commands more attention [66].

Now that the mutual influence of text and music mood is established, which are the underlying processes? We will refer to two approaches to explain the crossmodal effects. First, it is conceivable that multisensory integration takes place, despite the instruction to merely evaluate one modality. Information that is received in one sensory channel may spill over to the other channel. For the recipient, it is difficult if not impossible to distinguish which information is transmitted via which sensory channel. Impressions and characteristics belonging to one stimulus merge with the other, equivalent to a direct transfer of attributions [67, 68]. In our study, the crosstalk of text and music was reciprocal, that is, music had an influence on text perception, and text mood, in turn, affected music perception. This indicates that the mood conveyed by music and text is indeed regarded as one unified experience [69]. Second, Kahneman’s capacity model for attention [70] can be used to explain our findings on a broader level. It claims that cognitive capacity is limited and therefore attention has to be allocated among possible mental activities. As attentional resources become insufficient to closely attend to both text and music, the perceptual system may devote the resources to the recognition of emotion at the expense of linking the perceived emotion to its source. Moreover, it was highly probable within the scope of our 2 x 2-design that text and music transmitted a similar emotional message. This coincidence may have been exploited in the face of high attentional demand.

Influence of mood congruency on perceived quality, immersion, and liking

In our study, music and text were rated higher in immersion and were liked better when they were presented together with mood-congruent text or music. Moreover, the perceived text quality was higher when text and music mood corresponded, while music quality ratings were not influenced by the text. These findings on quality and liking are not surprising, as studies agree that stimuli are evaluated more positively when accompanied by congruent information in another apparently irrelevant sensory modality [71–73]. The fact that perceived music quality (in contrast to perceived text quality) was not systematically affected by congruency could be explained by the idea that it is generally more difficult to evaluate a classical music piece rather than a belletristic text. Those who are not educated in classical music might not hear quality differences at all, or they might not dare to judge it. The small variance of music quality ratings in our data supports this assumption.

Let us now find explanations for the outcome that participants felt more immersed in music and text when the stimuli were congruent in mood. We could show that the effect of happy music on text immersion is mediated by experienced valence. That is, happy music makes us happier, which in turn promotes our feeling of being absorbed in the happy text. The feeling of happiness is possibly connected to a certain light-heartedness and momentary open-mindedness which enables us to better engage in the positive reading content and to dive down into the happy story. In the case of sad texts valence is negatively correlated with immersion, that is, the worse we feel, the more we get involved with sad texts. We conclude that a text is more captivating if it is congruent with our own emotional state. As to the reverse connection, our current findings on emotional mediation concerning the influence of text mood on music immersion as well as on music liking remain inconclusive at the point and require further investigations.

Our results show that not only immersion but also liking and perceived quality were higher in the mood-congruent conditions. Thus, it seems that a text or a music piece intrigues us more if we generally like it and if we consider it of good quality. However, the causality between these three constructs is yet to examine. Thinking laterally, interesting analogies can be found in film music research. Studies revealed that music and sound effects influence emotions experienced by the viewer [74] and enhance immersion and suspense [75]. Film music furthermore affects the semantic evaluation of film scenes [76], it can foster empathy with the characters [77], and the content of film excerpts is remembered more easily when music and pictures convey a congruent mood [78]. Just like in our paradigm, music appears as an influential factor for the involvement in the story.

The perceived fit between music and text

Our analysis confirmed distinct patterns of perceived music-text fit. The more similar music and text were evaluated in mood, the better the stimuli were regarded to match. This result is in agreement with our recent study showing that music and colored ambient lighting are judged to fit particularly well if they carry similar emotional connotations [79]. In the same vein, numerous investigations have identified emotional attributions as drivers for the mapping of music with various stimuli from other sensual domains (see [36], for a review).

The influence of expertise and habit

In contrast to previous research (see introduction), we did not discover a difference between experts and novices in terms of their susceptibility to nondiagnostic crossmodal cues. Even though there is no consensus about experts being more or less influenced by background music, it is generally agreed that they are not affected in the same way as novices. We conducted additional explorative analyses to examine if there is a difference between those who regularly listen to music while reading and those who do not. Indeed, the ‘music-readers’ were more strongly influenced by background music than the ‘silence-readers’. This is in contrast with our expectations based on the finding that students are less disturbed by background music if they are used to studying with music [80, 81]. Even though the group of ‘music-readers’ was very small in our sample, we still consider this outcome as highly relevant and worth investigating in future studies. As mentioned before, the deviation of our results from the literature might be explained by the limited comparability of our design with classic crossmodal paradigms, but also by different practices to distinguish the more from the less experienced participants.

Practical implications

The findings presented in this article are especially interesting for the development of digital multimedia tools that create multimodal experiences. For instance, music streaming services might use the insights of our study and potential follow-ups to create specific playlists going with the mood of a given reading content. In this sense, an app could be set such that it instantly adjusts the music background appropriate for the reading content the user has selected. Moreover, the idea of creating specific background music for fiction books is obvious. As we showed that listening to mood-congruent music fosters text immersion, soundtracks for written love stories, dramas, and thrillers would be a genuine enrichment for passionate readers. Transforming the reading of a book into an audiovisual experience might even make it more attractive to those who hitherto prefer to spend their leisure time watching TV and video streaming services.