Vowel formant frequencies

Regarding the array of parameters commonly assessed in forensic speaker comparison (FSC) practice, formant frequencies are among the most common [1]. Previous experiments also suggest that, when dealing with high-quality recordings, formant frequencies are among the most speaker-discriminatory parameters when compared to melodic and timing estimates [2].

In general, the FSC task is commonly carried out considering F1, F2, and F3 due to the inaccessibility imposed by the telephone bandwidth to higher frequencies. However, recent technological advances in online messaging communication (e.g., WhatsApp and Telegram) have widened the possibilities of assessing higher frequencies in evidence materials [4].

Regarding speech production, vowel segments, like other speech sounds, are assumed to convey information in three dimensions: linguistic, social, and idiosyncratic, as mentioned by [5]. These dimensions are the driving forces of variation in speech production.

Regarding acoustic vowel specification, the quality of a vocalic segment (e.g., /i/, /u/, or /a/) is primarily correlated to the frequency of the first and second speech formants [6], namely F1 and F2, produced by the proper manipulation of mouth opening and constriction location. Variations in these dimensions are related to the degree of articulatory precision required for producing a given vowel [7]. Furthermore, higher formants such as F3 (i.e., somewhat related to vowel configuration, as in the case of front rounded vowels) and F4 are commonly referred to as being more speaker-specific, conveying more speaker-discriminant information [4]. These formants are also seen as related to voice quality aspects in spoken and singing voice [8].

As mentioned by [9], the position of the higher formants in the spectrum, such as F3 and F4, is largely determined by the vocal tract characteristics. Moreover, in the experiment conducted by [7], the authors observed that while F2 tended to increase in frequency as the point of constriction moved forward from the glottis, there was only a small increase in F3 as the mouth opening increased in size and became less rounded during the referred movement. The rate of increase in F3 depended mostly on the constriction size.

Studies suggest that the laryngeal cavity (LC) has its unique acoustic properties and is independent of the rest of the vocal tract [10, 11], showing that the elimination of the LC results in the suppression of F4 while other formants remain unaffected [11]. Furthermore, experiments using acoustic models have demonstrated that changes in the shape of the LC can have a significant impact on F4. However, it should be acknowledged that those experiments were carried out in an experimental setting using a particular artificial vocal tract, which limits the possibility of generalizations to real vocal tracts.

Some common sources of inter-speaker variability concerning vowel configuration are reported in the literature, including differences as a function of sex, dialect, age, speaking rate, and speaking style [12–15]. For the present study, we will focus on the latter variable: speaking style, mainly on its impact on the speaker-discriminatory power of formants.

Speaking style

Some studies have explored the levels of speaker variability concerning vowel space configuration as a function of different speaking tasks. Among these, some attention has been paid to the effects of slow and fast speech [14, 16], “clear speech” [17, 18], the nature of the spoken material– whether scripted or not [15], and “loud speech”. There are, of course, a plethora of other factors associated with changes in speaking style; however, we will focus on those that are pertinent to our study.

The study conducted by [14] aimed to examine the size of the acoustic vowel space in talkers identified as having slow and fast habitual speaking rates. Thirty participants, 15 slow and 15 fast talkers aged 18 to 35 years, were selected for the experiment. All speakers had dialects from the upper Midwest of the USA. Participants were asked to read a text at habitual and maximal speaking rates. Based on the total reading time and overall speaking rate of each participant’s performance, talkers were classified into slow and fast groups.

The results indicated no difference in the average size of the vowel space between slow and fast talkers and no relationship across talkers between vowel duration and formant frequencies. One observed difference between the slow and fast talkers was in the inter-talker variability of the vowel spaces, which was greater for the slow talkers of both sexes. However, the hypothesis of a cross-speaker, rate-dependent size of the vowel space was not supported by the data, as the average vowel spaces of the two groups of speakers were virtually identical, as were the average formant frequencies for the individual vowels.

“Clear speech” defined as “a speaking style that is used by talkers when they know they may have difficulty being understood” [19], which is typically slower than conversational speech [20], has also been studied. In a comprehensive study conducted by [18], it was observed that across all 41 talkers, the first formant (F1) increased significantly in clear speech for 7 out of the 10 English vowels assessed. The predominant trend in the F1 data showed an overall increase, which suggests that speakers typically lower their jaw and tongue more in clear speech, selectively enhancing the acoustic distance between high and low vowels. For the second formant (F2), the direction of change varied depending on the vowel category. Specifically, front vowels exhibited significantly higher F2 values, while back vowels displayed lower F2 values in clear speech. This combination of increased F2 for front vowels and decreased F2 for back vowels resulted in an expanded F2 range.

In [15], F1 and F2 formants of the French vowels /i/, /a/, and /u/ were assessed in six speech tasks: (i) describing “bizarre details” in a picture; (ii) providing a more comprehensive picture description with a narration style; (iii) same as in situation (ii), but the speaker was asked through a headset to provide other complementary information in the picture; (iv) the speaker was told that a new person who had never seen the picture would have to reproduce it based on the information he/she delivers; (v) the speaker engages in a spontaneous conversation with an experimenter; (vi) the speaker was asked to read a word list presented in random order.

The results revealed that the four tasks where the speaker had to describe a picture shared common characteristics. The spontaneous speech task appeared different from the other situations. In general, tasks in which the speaker had to interact frequently with an interlocutor tended to show substantial contrast with non-interactional tasks. As suggested by topological observations of the vowels in the bi-dimensional acoustic plane, there was a clear tendency for more centralized formant values in spontaneous speech compared to the other tasks (i.e., picture descriptions).

Experiments have also been conducted to study how the acoustic configurations of vowel formant frequencies are affected by “loud speech”. In the study carried out by [21] with eleven German-speaking female speakers, the subjects were instructed to produce normal and loud speech in 3 tasks: i. reading sentences with the target vowels /i: a: u:/, ii. answering questions that included target words with controlled consonantal contexts with varying vowel types and a text recall task. Loudness variation was elicited naturalistically by changing interlocutor distance. F1 and F2 formant frequencies were obtained from the stressed vowels in the targets.

As reported by [21], comparisons across vowels indicated that high, tense vowels showed limited formant variation as a function of loudness. Analysis of /i:, a:, u:/ across speech tasks revealed a reduction in vowel space during the retelling task compared to the other tasks (i.e., reading and answering specific questions). Loudness changes for F1 were consistent in direction but variable in extent, with few significant results for high, tense vowels. Moreover, results for F2 were quite varied and seldom significant. Overall, the authors observed that speakers differed in how loudness and task affected formant values, suggesting that loud speech does not always lead to changes in vowel formant patterns.

A crucial aspect, particularly concerning speaker characterization with implications for forensics, involves the mobile phone speaking condition and its possible effects on acoustic-phonetic parameters. Studies such as [22] suggest that speaking on a mobile phone constitutes a unique speaking style due to the specific characteristics of this condition and the significant changes it imposes on acoustic-phonetic descriptors. However, most available studies on speaker discrimination have primarily focused on the artificial effects of telephone/mobile phone bandwidth transmission, which may represent a limitation. This focus makes it challenging to isolate the specific impact of speaking style alone from the artificial changes imposed by the transmission channel.

Among some studies involving mobile phone-transmitted speech, we highlight the one conducted by [23] on the effects of mobile phone transmission on vowel formant frequency measurement. In that study, six male and six female English speakers (with average ages of 23 and 24, respectively) read a short passage while speaking through a mobile phone. Two simultaneous recordings were made and compared: one at the far end of the phone line (mobile phone-transmitted) and the other via a microphone directly in front of the speaker (direct recording). Measurements of F1, F2, and F3 were taken from stressed vowels in both recordings.

Results from that study suggested that, due to the filtering effect of the phone transmission, measured F1 frequencies for most vowels were higher than their counterparts in the direct recordings. On average, measured F1 values in the mobile phone condition were 29 percent higher than in the direct condition. Furthermore, F2 measurements were not significantly affected. Measured F3 frequencies were also generally unaffected by mobile phone transmission. However, some speaker-related exceptions were found, particularly for subjects with relatively high F3s. In these cases, the mobile phone recordings tended to yield significantly lower values.

Regarding Brazilian Portuguese, the study by [24] focused on determining the extent of acoustic-phonetic changes caused by mobile phone band-pass filters and transmission effects on oral vowels. Recordings of 10 male speakers were analyzed in two conditions: mobile phone-transmitted and direct speech. The analyses showed that mobile phone band-pass filtering shifted vowels with low F2 values upwards and vowels with high F2 values downwards. Furthermore, it was verified that telephone transmission affected speakers differently. The analysis of the effects on the vowel space demonstrated that the increase in measured F1 values in the mobile phone situation resulted in a global downward displacement of the vowel space. Additionally, the decrease in measured F2 values for front vowels and the increase for back vowels tended to reduce the area of the vowel space for most subjects.

In light of previous research, it is reasonable to ask: what is the impact of speaking style alone on the discriminatory power of vowel formant frequencies when removing the artificial changes imposed by the transmission channel? This study aims to address that question.

In contrast to [3], the present study also explores the combined discriminatory power of different formants. Furthermore, a different speaking style (Interview) was included in the analyses, allowing us to compare findings obtained from spontaneous dialogue with those from the interview style. Comparing the same subjects across two different speaking styles enables the assessment of formant discriminatory power when mismatched speech materials are confronted. Furthermore, the impact of data sampling on the estimation of discriminatory power metrics has also been considered, as described in the methods section.

Some forensic implications

Theoretical and methodological standards have been carefully examined and recommended to determine the most effective parameters for applications in speaker comparison. A set of criteria proposed by [25] encompasses the following desirable attributes that an acoustic measure should exhibit: a large degree of inter-speaker variability, minimal intra-speaker variability, robustness against attempts at disguise or imitation, robustness against the effects of transmission channels, frequent occurrence (availability), and ease of measurement (measurability).

Concerning the first two criteria (intra- and inter-speaker variation), there is a notable lack of studies in Brazilian Portuguese that examine the extent of variation in phonetic-acoustic parameters across different speaking styles, particularly those commonly encountered in forensic contexts, such as spontaneous dialogues and interviews.

A recurring practice in forensic speaker analysis is the comparison of speech materials from the same subject in at least two different speaking styles: (i) establishing a spontaneous dialogue with a familiar interlocutor, often resulting from a telephone interception, called “questioned” speech material; and (ii) being subjected to an interrogation (i.e., interview) with an unfamiliar interlocutor, resulting in a speech sample called “reference” material. The process of comparing such materials collected under different conditions is accompanied by a reduction in the dimensionality and representativeness of the samples [6].

As highlighted in [6], assessing differences between samples becomes a major challenge as there is generally no control over the situations in which data are collected. In most cases, the expert is led to confront a reference recording collected in a context that is almost or entirely different from the context of the standard sample acquisition due to the presence and/or absence of condition-specific variables. Therefore, in this experiment, we aim to isolate and assess one such condition-specific variable: speaking style.

Participants

The participants are 20 subjects, namely 10 identical twin pairs, all male, Brazilian Portuguese (BP) speakers from the same dialectal area (state of Alagoas). The participants’ ages ranged between 19 and 35 years, with a mean of 26.4 years.

Although the participants in this study were originally part of a twin study project that analyzed potential acoustic-phonetic differences in the speech productions of identical twin pairs, this study will focus on comparing non-twin subjects. For more information on the experiments involving twin speakers, the reader is referred to [3, 26–28].

The inclusion and exclusion criteria established for the referred original project were the following. Inclusion criteria: i. Identical twins; ii. male speakers; iii. same dialect; iv. aged between 18–45 years; v. with at least elementary school completed (High School). Exclusion criteria: i. Reported hearing loss or speech disorder, ii. identical twins raised apart; iii. identical twins that lived apart from each other for more than five years.

Recordings

The recordings were carried out in silent rooms in the cities where the participants resided. All recordings were carried out with a sample rate of 44.1 kHz and 16-bit amplitude resolution, using an external audio card (Focusrite Scarlett 2i2) and two headset condenser microphones (DPA 4066-B).

The speech material analyzed regards: i. spontaneous telephone conversations (dialogues) between familiar speakers (twin pairs); and ii. interviews between the participants and the experimenter (the first author). The recordings were carried out in two different contemporaneous sessions (i.e., in the same day). The specificities regarding each speech material and the context for their elicitation are presented in the following.

Data transcription and extraction

The data segmentation and transcription were performed manually using the Praat software [30]. All data were transcribed and reviewed by the first author, who is a phonetician, a certified speech-language pathologist, and a native speaker of the dialect studied. Speech portions pertinent to the present analysis were divided into three different tiers in the Praat TextGrid, as follows:

1. Dialogue part: different portions/parts of the dialogues throughout the recordings, e.g., beginning, middle, and final parts;
2. Speech chunks: speech intervals, in most cases corresponding to inter-pause intervals (i.e., stretches of speech between long silent or filled pauses);
3. Oral monophthongs: oral monophthongs produced within speech chunks;

The focus of the present research was the seven phonemic vowels of BP /i, e, ε, a, ͻ, o, u/. We refer to these vowels as possessing different qualities, namely, different perceptual and acoustic characteristics to them. Hence, the term ‘vowel quality’ is used. Regarding the number of tokens for each vowel in both speaking styles, these were as follows (from higher to lower): 5,995 /a/, 2,910 /i/, 1.838 /u/, 1,684 / ε/, 1,479 /e/, 1,063 /o/, 973 /ͻ/, resulting in a total of 15,942 tokens.

Vowels were manually segmented and transcribed based on auditory and acoustic criteria, specifically by carefully listening to the speech segments and observing the appearance and disappearance of F2 energy in the broad-band spectrogram. F1-F4 values were automatically extracted from the midpoint of the labeled vowels using a Linear Predictive Coding (LPC) technique. The maximum frequency range for formant extraction was set to 5000 Hz, identifying five formants using an LPC order of 10. Possible measurement errors were subsequently checked using the Interquartile Range to detect outliers within speakers, styles, and vowels of the same speaker.

The parameter extraction was performed using a Praat script cf. [31] developed by the third author. The script generates a text file containing speaker identity, vowel labels, vowel duration in seconds, and formant frequencies in Hertz. Statistical analyses were conducted using formant frequency measurements in Hertz.

Statistical analysis

All statistical analyses were carried out in the R environment [32]. The main justification for focusing on comparisons among non-twins is that such data can be considered more realistic from a general standpoint. In everyday life, individuals may share similarities in aspects such as sex, age, dialect, and education level, but the extreme similarities seen in twin pairs are relatively rare.

The order of the subsections presented below reflects the steps taken during the statistical analysis process.

Dialogue versus interview

As shown in Fig 2A, there is almost a complete overlap between the acoustic spaces of different speaking styles, despite some slight differences in average ‘F1 x F2’ values based on vowel quality. The surface areas of the vowel space polygons for dialogue and interview are very similar. For some back vowels, particularly the low and high-mid back vowels, the acoustic space is slightly wider in the interview style compared to dialogue. Conversely, the vowel /u/ appears slightly more centralized in the interview style than in dialogue.

Despite the overall similarity in vowel space configurations shown in Fig 2A, speaker-specific particularities can still be observed. As depicted in Fig 3, speakers exhibit different patterns in how speaking style affects their vocalic spaces. Visualizing Fig 3 and numerically comparing surface vowel space areas reveals two categories: individuals with a slightly wider acoustic space in the interview style compared to dialogue (e.g., A1, B1, B2, C2, D1, D2, E1, E2, G1, I1, I2, J1), and those with a slightly wider vowel space area for dialogue (e.g., A2, C1, F1, F2, G2, H1, H2, J2). In some cases, there is primarily a displacement of the entire acoustic space either vertically or horizontally.

The observation that different speakers may exhibit distinct visual trends in their vowel space configurations across various speaking styles (Fig 3) might shed some light on the less pronounced differences observed when comparing the acoustic spaces of the general speaker group (Fig 2A), as individual differences may counterbalance each other. However, these trends should be further explored in future analyses dealing primarily with the study of style to probe their statistical and potential linguistic significance.

Vowel quality and formants

Fig 4 illustrates the impact of vowel quality on the density distribution of F1-F4 formants. It is clear that vowel quality affects the average values of the lower formants in both speaking styles to a greater extent. For example, in the dialogue speaking style, the average F1 difference among vowels ranges from approximately 331 Hz to 586 Hz (a difference of 56%), and the average F2 difference ranges from 1.074 Hz to 2.027 Hz (a difference of 67%). The variability of vowels’ F1 x F2 is also evident in Fig 2, allowing us to distinguish between different vowel categories.

Conversely, while differences across formants are generally expected to become more pronounced in higher spectral regions, lower variability was observed for the higher formants (F3 and F4). In the dialogue speaking style, F3 values ranged from 2.368 Hz to 2.655 Hz, and F4 values ranged from 3.387 Hz to 3.570 Hz, with differences of approximately 10% and 6%, respectively. If we had a visual representation of the F3 x F4 relationship (as in Fig 2B), it would show vowel points centered around specific locations in the two-dimensional acoustic space.

Having made some qualitative remarks on the data, we now address the statistical outcomes concerning the analysis of the speaker discriminatory power of formants as a function of speaking style and vowel quality.

The discriminatory power of vowel formants

When inspecting the C_llr and EER boxplots in Fig 5 for individual formant frequencies assessed in style-matched conditions, it was evident that F3, F4, and F1 (in order of relevance) were the best-performing formants in both speaking styles in most cases. Notably, F2, when assessed in isolation, demonstrated lower speaker discriminatory performance. When narrowing down the analysis to compare results between styles, it is possible to observe that system performance was higher for dialogue in relation to interview as demonstrated by lower C_llr and EER values in Fig 5A and in Fig 5B, respectively.

Regarding mismatched speaking style analyses, Fig 6 illustrates C_llr and EER scores for individual and combined parameters. Comparing F1-F4 boxplots in this figure with F1-F4 boxplots in Fig 5 reveals that discriminatory performance is higher in style-matched conditions. Notably, average C_llr and EER scores shifted towards higher values in Fig 6. It is evident that all formants are affected by style, with F3 showing a slightly lower median C_llr (Fig 6A) and a larger distribution towards lower EER values (Fig 6B).

However, it is worth noting that the combined discriminatory power of F3 and F4 is higher than that of F1 and F2, as indicated by lower C_llr and EER median values in Fig 6. It is safe to say that multivariate analysis is the most recommended approach when dealing with mismatched speech materials. Combining different formants proved to be the best method for modeling speaker differences. Even the combination of F1 and F2, which performed the least effectively among all formant combinations, outperformed the best-performing single variable, F3, in Fig 6.

As can be observed in Figs 6 and 7, the combination of higher formants ‘F3 + F4’ seemed to outperform the combination of lower formants ‘F1 + F2’, as suggested by lower C_llr and EER median values. For instance, C_llr scores decreased from 0.49 to 0.39, and EER scores decreased from 0.19 to 0.11 when comparing ‘F3 + F4’ against ‘F1 + F2’.

In the multivariate analysis, it was possible to verify that the magnitude of improvement in C_llr and EER scores increased as more formants were incorporated into the model, especially when high formants (i.e., F3 and F4) were included. Notably, the best discriminatory performance was achieved when most formants were combined. However, removing or adding F2 from the final model did not impact system performance substantially. As can be observed by comparing results between models (‘F1 + F3 + F4’ vs. ‘F1 + F2 + F3 + F4’) in Fig 7, the differences were minimal.

As shown in Fig 7, when all formants were included in the model, C_llr and EER scores dropped to 0.35 and 0.10, respectively, in relation to the best performing single formant, i.e., F3 (C_llr = 0.56, EER = 0.20). The overall C_llr and EER distributions were also shifted, revealing a score variability within lower C_llr and EER ranges in Fig 6.

Concerning the assessment of discriminatory power as a function of vowel quality based on the formant relation ‘F1 x F2’, a general pattern was suggested in Fig 8. Overall, front vowels tended to outperform back vowels in terms of their discriminatory power. Front vowels and central vowel /a/ tended to depict the lowest C_llr and EER scores when looking at the overall results without considering specific styles in Fig 8A, and when narrowing down the analysis to dialogue in Fig 8B, and interview in Fig 8C.

The dispersion and variability of different vowels were presented in Fig 2B. By inspecting this figure, one can observe that front vowels have larger ellipse areas compared to back vowels. As already mentioned, these areas represent the 95% confidence interval for speakers’ average F1 x F2 values when data from both dialogue and interview speaking styles are combined. The possible relationship between vowel dispersion and discriminatory power will be addressed in the discussion.

For reasons already explored, namely, the higher impact of articulatory constraints on lower formants [6, 7] (see Fig 4), only F1 and F2 were considered for the assessment of the discriminatory power of specific vowels in Fig 8. The outcomes may not be generalized to conditions where higher formants, such as F3 and F4, are incorporated into the models. However, for making a linguistic sense of the differences in performance among vowels, an ‘F1 x F2’-based comparison suffices. Regarding the potential forensic implications of the present analysis, it is preferred to refer to Fig 6, where all vowels’ combined discriminatory power is considered as per different formant combinations.