General theoretical background

One dominant theme in current linguistic theories is that sounds themselves have no meanings. This thesis—also known as the arbitrariness of the relationship between meanings and sounds—was declared by Saussure to be one of the organizing principles of natural languages [1, 2], which has had significant impacts on modern thinking about languages. In a recent review article on speech perception [3], while acknowledging some exceptions, the authors argue that “[i]n their typical function, phonetic units have no meaning” (p. 129), which shows that the arbitrariness thesis is still prevalent in the current thinking about speech perception. After all, it does not seem to be the case, at least at first glance, that, for example, /k/ itself has any inherent meanings. If there are fixed sound-meaning relationships, so the argument goes, then the same object (or the concept) should be called by the same name across all the languages (assuming that languages use the same set of sounds). This prediction is obviously false, because different languages use different strings of sounds to mean the same object/concept; e.g., the same animal is called /dɔg/ in English, /hƱnt/ in German, // in French and /inu/ in Japanese, etc. The following quote from Saussure summarizes this view succinctly (pp. 67-68):

The link between signal and signification is arbitrary. Since we are treating a sign as the combination in which a signal is associated with a signification, we can express this more simply as: the linguistic sign is arbitrary.

(Emphasis in the original.)

…

There is no internal connexion, for example, between the idea ‘sister’ and the French sequence of sounds s-ö-r which acts as its signal. The same idea might as well be represented by any other sequence of sounds. This is demonstrated by differences between languages, and even by the existence of different languages. [2]

However, a growing body of experimental and corpus-based studies show that there is at least a stochastic tendency—or bias—for particular sounds to be associated with particular meanings—the association which is often referred to as “sound symbolism” or “sound symbolic associations” [4]. The argument for sound symbolic associations at least dates back as far as Plato’s Cratylus [5, 6]. Modern studies of sound symbolism were inspired by the pioneering work by Sapir [7], which shows that English speakers tend to associate /a/ with big images and /i/ with small images. There is now a substantial body of work showing that this size-related sound symbolism holds not only for English speakers, but also for speakers of other languages; generally, back and low vowels—those with low second formant—are associated with big images, whereas front and high vowels—those with high second formant—are associated with small images [8–17] (though cf. [18]). See also [8, 19, 20] for some classic discussions on sound symbolism, and [17] for a recent informative review, which presents a more nuanced view of non-arbitrariness in natural language.

Another well-studied case of a sound-meaning correspondence originates from the insights by Köhler [21, 22]. He pointed out that given two nonce words, maluma and takete, round shapes are more likely to be associated with the former, whereas angular shapes are associated with the latter (Fig 1(a)). These associations have been studied and replicated by a number of studies [20, 23–29] (see also [30–32] for the related “bouba-kiki” effect). A later study [27] demonstrated that this relationship is more general—the relationships hold between round shapes in general and sonorant consonants, and between angular shapes and obstruents (as those shown in Fig 1b1 and 1b2). These studies show that sounds have associations not only with linguistic meanings but also with static visual shapes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Round and angular shapes.

Shapes that are associated with maluma and takete. Rounded shapes on the left tend to be associated with maluma and angular shapes on the right tend to be associated with takete. (a) Reproductions of Köhler’s original figures, (b1) shapes used in [27], and (b2) shapes used in [27].

https://doi.org/10.1371/journal.pone.0163525.g001

One important emerging insight in the studies of sound symbolism is that sound symbolism is nothing but an instance of a more general cross-modal iconicity association between one perceptual domain and another [33–35]. The study by [27], for example, demonstrates that sounds can be associated not only with linguistic meanings, but also with visual shapes. Other studies have shown that particular sounds can be associated with the images of personalities [24, 36, 37], and furthermore, even shapes themselves can be associated with linguistic meanings or particular personalities (even without being mediated by sounds) [36, 37]. These results imply that a cross-modal association, of which sound symbolism is one instantiation, is a general feature of our cognition. If this hypothesis is correct, then the demonstrated examples of the sound-meaning relationships are just a tip of the iceberg.

Given this general theoretical background, one main question that is addressed in this research is as follows: If particular sounds can be associated with visual images, are such associations limited to static visual images, or can they also be associated with dynamic visual gestures like body movements? In answer to this question, we demonstrate that sounds can be associated with particular gestural motions.

Before closing the introduction, we would also like to raise one cautionary remark about what our findings—and the results of other studies of sound symbolism—would really mean to the arbitrariness thesis of Saussure [1, 2]. We are not challenging the thesis that linguistic symbols can be arbitrary. For example, even if the English word big contains a “small vowel” [I], it does not prevent the learner of this language from learning that it means “big” (though cf. [38, 39] for evidence that sound symbolism might facilitate word learning). On the other hand, we know that sound symbolic effects do affect the word-formation patterns in such a way that words that follow sound-symbolic patterns are more frequently found than expected by chance [15, 40]. Therefore, we do not believe that sound-symbolic mechanisms are completely outside of the linguistic system. In short, then, how the effects of sound symbolism “sneaks into” the system of arbitrary signs is an interesting issue for the cognitive science of languages (see [39] for relevant discussion). However, we do not attempt to resolve this issue in this paper.

The current study

The current study addressed whether gestural motions can be directly associated with sounds, partly inspired by existing studies of sound symbolism in sign languages (see e.g. [41–43]). This question has been addressed by a few existing studies, which presented some video images to the participants and examined if particular motions are associated with particular sounds [29, 44] (see also [45]). Especially, the current study can be understood as an extensive follow-up study of the one conducted by Koppensteiner et al [29] with a few substantial differences. While [29] used a forced-choice paradigm, the current study used a free elicitation method, in which the participants named the given gestures rather freely. A forced choice method is amenable to a potential concern raised by Westbury [46]: “[t]he sound symbolism effects may depend largely on the experimenter pre-selecting a few stimuli that he/she recognizes as illustrating the effects of interest” (p.11). A free elicitation method deployed in the current experiment avoids this potential concern, because the sounds elicited are not pre-determined by the experimenters. (We hasten to add that we are not arguing that a forced-choice method is useless or deeply flawed in studying sound symbolic patterns. At the very least it serves to objectively confirm the intuitions that the experiments have with a large number of naive participants.)

Another aspect in which our study differs from [29] is that we used native speakers of Japanese as the target participants to address the question of how general the relationships between gestural motions and sounds are. ([29] do not report the native language of the participants. However, since the experiment was run at the University of Vienna, we conjecture that they are mainly native speakers of German.) To the extent that there is a possibility that sound symbolic patterns can partly be language-specific [14, 18, 44, 47], testing speakers of different languages is important. In addition, we also tested whether the magnitude of manual gestures can affect participants’ judgments; for example, is a large manual gesture more likely to be associated with /a/ than with /i/, a la Sapir’s [7] finding? This is a topic that was not explored by [29].

Generally, also relevant to the current study is the observation by Kunihara [48] that sound symbolism works stronger when the participants of the experiments actually pronounce the stimuli; i.e. using articulatory gestures enhances the effects of sound symbolism. This result suggests that there is a non-trivial sense in which sound symbolism is grounded in actual articulatory gestures [7, 8, 14, 26, 48–51]. For example, /a/ is considered to be large, maybe because the jaw opens the most for this vowel [52, 53]. It does not seem to be unreasonable to generalize this insight into a more general hypothesis: sounds themselves are associated with bodily gestures in general, whether they are articulatory or not (see also [41–43]). Extending on this hypothesis, at the end of the paper we address a potential practical application of this sort of research—if gestural motions have direct connections with sounds, we can make use of those associations in sports instructions [54].

Stimulus movies

Apparatus and setups.

To record the stimuli, a right-handed male served as an actor (Fig 2). The actor is the fourth author of this paper, who is able to manipulate details of his body movement very well. The actor wore a black long-sleeved shirt, a black balaclava, and a white glove on his right hand. His eyes were covered with glasses with black lenses. A spherical infrared-reflective marker (15 mm in diameter) was attached on the tip of his middle finger on the glove. In a dimly lit room, the actor sat on a high-stool in front of a black curtain and performed maluma and takete gestures with the right hand.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The actor.

The actor who recorded the gestural stimuli.

https://doi.org/10.1371/journal.pone.0163525.g002

A digital movie camera (HDR-CX720, SONY, Japan) was placed in front of the actor; the distance between the camera and the actor was approximately 3 m. The recording covered the actor’s whole upper body, in order to capture the whole hand motions. The movie camera recorded the right hand motions with a shutter speed of 1/1000 s and a sampling rate of 60 frames per second (fps). These recording conditions were expected to provide clear cues to the movements of the white-gloved hand. See supporting information for all of the stimulus movie files that were used in the experiment.

Three high-speed cameras (OptiTrack Prime13, NaturalPoint, USA) recorded all movements of the reflective marker at 120 fps. The three-dimensional (3D) coordinates of marker position were automatically computed using motion capture system software (Motive: Body, NaturalPoint, USA). After the spatial calibration, the position errors of computed values were no more than +/- 2.5 mm in the 3D space.

Recording of the movie stimuli.

Köhler’s original maluma and takete drawings [21, 22] were printed on a piece of paper, and placed right above the movie camera, which helped the actor to trace them with his right hand. The actor traced the shapes of maluma and takete in one stroke. The actor tried to keep the velocity of his hand movement as constant as possible, but for the takete movement, the acceleration profiles necessarily changed, because of the changes in directionality of the movement.

To examine whether the magnitude of gestures would influence their association with sounds, the actor performed each of the maluma and takete gestures in two different kinematic conditions. In the first condition (henceforth, the SMALL condition), the actor kept a motion tempo at 60 beats per minute (bpm) and completed the action within 6 s. The range of his hand movements was fit within a square range, whose length was roughly equal to his shoulder width. In the second condition (henceforth the LARGE condition), the motion tempo was 40 bpm and movement duration was 9 s. For this large condition, the actor moved his right hand approximately 1.5 times as large as the square range whose length was his shoulder width. See Fig 3a. The actor practiced these gestures until he became familiar with each condition and then repeated five recording trials for the main recording.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Properties of the visual stimuli.

Line drawings of motion paths (a) and acceleration profiles (b) of the middle finger tip in the frontal plane for the maluma (the top panel) and takete (the bottom panel) gestures.

https://doi.org/10.1371/journal.pone.0163525.g003

Kinematic properties of the maluma and takete gestures.

Fig 3 illustrates line drawings of motion paths and acceleration profiles of the middle finger tip on the frontal plane for the four selected gestures. The acceleration values were calculated as the second order derivative of the reflective marker position on the frontal plane against time. In the acceleration profiles, the x-axis shows the time course of the gestures in percentages; the y-axis represents the acceleration at each point in the standardized time. Fig 3a indicates that the actor reproduced hand motions that are very close to the original maluma and takete drawings of Köhler. Note however that these motion traces were not presented to our participants—they only observed the movements. Fig 3a is provided here for the sake of illustration.

Acceleration profiles were considerably flatter for the maluma gestures (the top panel, Fig 3b) than for the takete gestures (the bottom panel, Fig 3b). Fig 3b also shows that the hand was moving at an approximately constant speed in the maluma gestures (the top panel), reflecting smooth motion pattern. On the other hand, the takete gestures involve a waveform with six cycles; each cycle is a reminiscent of a sinusoid with local maximum and local minimum (the bottom panel). These waveform profiles reflect six acute changes of movement direction and movement velocity on the frontal plane in the takete gestures. The acceleration profiles for the SMALL and LARGE conditions are comparable, both in the takete and maluma conditions. All the kinematic measurements for the selected gestures are summarized in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Kinematic properties of the gestures used in the experiment.

https://doi.org/10.1371/journal.pone.0163525.t001

Finally, the 3D position data of the reflective marker were analyzed using motion analysis software (BENUS3D, Nobby-Tech, Japan) to produce Point-Light Display (PLD) movies of the middle finger tip. These PLD stimuli show only movements of the reflective marker, excluding any images of the actor. The motion paths and kinematic features of the PLD stimuli were identical to those of the corresponding gesture movies. While the original videos were clearly gestural movements of a human body, the PLD stimuli only involved movements of a point-light. The contrast between these two conditions was designed to address the question of whether there is a difference between human body movements and more general non-human movement patterns. (It shares the same spirit as those phonetic experiments which use non-speech stimuli for speech perception experiments [56]—see [57] for an experiment on sound symbolism using non-speech sounds).

The elicitation task

Summary

The current experiment revealed several associations between particular types of motions and particular sets of sounds: (1) front vowels are more likely to be associated with small motions than with large motions; (2) front vowels are more likely to be associated with the takete motions than the maluma motions; (3) obstruents are more likely to be associated with the takete motions than with the maluma motions; (4) voiced obstruents are more likely to be associated with large motions than small motions. Overall, the current study lends further support to the idea that dynamic motions can invoke particular sounds [29, 44, 45]. Although [29] has already shown this association, we confirmed the existence of the association using a different—and arguably better—methodology and using a set of speakers with different language background. Finding correlations between gesture sizes and some particular types of sounds—back vowels and voiced obstruents—is also new.

There was little if any difference between the ACTOR and PLD conditions. The fact that both the ACTOR condition and the PLD condition showed similar results is also interesting in that both gestural movements of a human body and non-human light movements caused similar sound symbolic patterns (cf. [64, 65]). Our participants were able to associate sounds with dynamic motions, even when the motions were movements of a point-light without any bodily gestures.

Gestural patterns and sound symbolism

The current study has shown that back vowels are more likely to be associated with the maluma motions while front vowels are more likely to be associated with the takete motions. This finding replicates Berlin’s study who found similar sound-shape associations. The fact that the same sort of sound symbolic association holds for static visual shapes (Berlin’s study) and for dynamic movements (current study) suggests that sound symbolism is not limited to perception of static images, but also holds for the perception of dynamic motions.

The current study also demonstrated that the takete motions are often associated with obstruents, while sonorant sounds are often associated with the maluma motions. These associations again replicate the previous studies on the shape-based sound symbolism effects [26, 28, 33]. Yet again, these associations demonstrate that dynamic gestural motions can be projected onto particular sounds, expanding the scope of the traditional sound symbolism studies [29, 44].

A post-experimental questionnaire indicated that all of the participants could discriminate between the maluma gestures and takete gestures—recall, however, that no trace lines representing the motion path were presented. The current results thus raise the possibility that smooth movement patterns (for the maluma motions) themselves are associated with sonorant consonants and back vowels, while jagged acute movement patterns are associated with obstruents and front vowels.

Gestural size and sound symbolism

The current study finds that back vowels are more often associated with the larger motions than the smaller motions. This finding also accords well with previous finding that back vowels are perceived to be larger [7, 10, 12, 14], arguably because the resonance cavity for the second formant is bigger for back vowels [11, 12, 14]. Yet again, this parallel suggests that dynamic motions, not just static images, can cause sound symbolic associations.

The effect of obstruent voicing on the perception of size is less well studied than the effect of vowels—however, there are a few studies suggesting that voiced obstruents are more likely to be associated with larger images than voiceless obstruents [10, 14, 17, 26, 62], and there is a reasonable articulatory basis for this association. Voicing with obstruent closure involves the expansion of the intraoral cavity due to the aerodynamic conditions imposed on voiced obstruents [66]. This articulatory movement to expand the intraoral cavity can be the source of large images.

Further issues on sound-symbolic relationships

One issue that remains to be resolved is how direct the mapping between motions and sounds are. Do the participants directly map the actor’s gestures and PLD movements onto particular set of sounds? Or are the motion images mediated by static representations of the motion paths? The current experiment was not designed to tease apart these two possibilities, but we believe that this is an important question. If movements are directly mapped onto sounds, this connection would ultimately be related to the question of the bodily basis of sound symbolism—can sound symbolism have its roots in bodily—articulatory—movements themselves [8, 14, 30, 48]? We would like to explore this issue in more depth in future research. In order to do so, we need to examine other known cases of sound symbolism, and explore whether bodily movements can be a basis of each sound symbolic pattern.

A more general question for future research is whether it is possible that non-linguistic gestures (presented as stimuli in this experiment) are directly mapped onto articulatory gestures (cf. studies on iconicity in sign languages, e.g. [41]). We find this hypothesis to be possible, and it points to a general issue in a cross-modal perception. We often find that a cross-modal relationship holds not just between two domains, but among more than two-domains. For example, [37] finds that obstruents are associated with angular shapes and inaccessible personal characteristics, and moreover, angular shapes themselves can be associated with inaccessible personal characteristics. Given these results, an interesting question arises: how direct is the cross-modal relationship between one perceptual domain to another?