1.1 Tonal languages

For tonal languages, the Lombard effect may interact with the dynamic F0 patterns that carry lexical meaning. While spectral cues such as shifts in F0 and formants are likely important for the LG, the contribution of temporal information is still questionable.

Recent findings for Mandarin Chinese [7] suggest that spectral measures, such as -ratio (ratio of the speech energy between 1–15 kHz and 50–1000 Hz) and speech-weighted signal to noise ratio (SNR), correlate with the LG, while F0-related changes are also predictive. However, the relative contributions of spectral versus spectro-temporal cues to absolute SI and LG in tonal languages remain unresolved.

To address this question, we explore computational SI models as a controlled analysis tool that allows us to isolate specific acoustic features. The models are not the primary object of investigation in this study; instead, they allow us to manipulate the available acoustic information selectively for a prediction, thereby disentangling spectral and spectro-temporal contributions under otherwise identical conditions.

1.2 SI models of this study

The primary model was the framework of auditory discrimination experiments (FADE) of [12]. FADE implements a machine learning backend, adapted from automatic speech recognition (ASR) systems, similar to the ”optimum detector” approach [13]. Stimuli are first preprocessed by a frontend, which is inspired by auditory-system mechanisms, allowing precise control over the type of information entering the model. This control makes it possible to directly compare speech recognition threshold (SRT) predictions using spectral or spectro-temporal representations of the input signals. Importantly, FADE does not assume any fixed or predefined combination of spectral and temporal cues; instead, it automatically selects the features that lead to the best speech recognition performance.

FADE has previously demonstrated high accuracy across several non-tonal languages in both stationary and fluctuating noise conditions [14], outperforming the extended speech intelligibility index (SII) (eSII, see [15]). It also provides accurate predictions for normal-hearing and hearing-impaired listeners [16,17], and it still outperforms state-of-the-art commercial ASR systems [18] in the task of human SI prediction. In the context of tonal languages, FADE successfully predicted the average LG for Cantonese speakers [19], which further motivated its use in the present study.

Despite these advantages, the ASR backend of FADE is implemented as a hidden Markov model with restricted transitions (see method section). As a result, even a spectral-only input representation may still permit some spectro-temporal inference if its changes across time frames are decoded. Therefore, the classical SII was used as a second model approach, which decomposes speech and noise into long-term spectral components, providing a strictly spectral model of intelligibility [20].

The SII has proven to be a simple yet powerful SI model for non-tonal languages [21,22]. If the LG in Mandarin Chinese is fully explainable by spectral changes in the speech signal only, a strictly spectral model such as the SII is expected to be sufficient to account for the changes in speech intelligibility.

1.3 Aim of this study and rationale

The presented study investigates whether spectral information alone is sufficient to predict the LG in Mandarin Chinese, or whether additional spectro-temporal cues are required. In addition, we examine whether the same conclusions hold for absolute SI, quantified by the SRT.

Because spectral features are a subset of spectro-temporal features, accurate predictions based solely on spectral inputs would imply that temporal cues introduced by the Lombard effect do not provide additional, independent intelligibility-relevant information. Under the assumption that human listeners make optimal use of the available cues, this would indicate that spectral changes are the main driving factor of the LG in Mandarin Chinese.

To address this, two complementary comparisons were carried out:

1. (1) FADE with spectro-temporal versus FADE with spectral-only features

This comparison isolates the contribution of the input representation, as both conditions share the same computational backend. Any differences in predicted LG, therefore, stem solely from the presence or absence of temporal information in the frontend.

This is done by a comparison of the predictions with separable Gabor filter bank (SGBFB) features and log-mel features. Although both representations are extracted from time-varying speech signals, each log-mel feature vector contains only the spectral information of the current analysis frame, whereas SGBFB features explicitly encode spectro-temporal modulation patterns across adjacent frames (see Section 2.2.1). Consequently, within the present study, log-mel features are regarded as spectral-only information.

1. (2) FADE with spectral features versus SII

This comparison ensures that predictions based on spectral information are evaluated with a strictly spectral model. As the models (i.e., FADE and SII) differ in their computational architectures, this step verifies that any effects observed in comparison (1) are not artifacts of residual temporal sensitivity in FADE’s ASR-based backend.

Together, the two comparisons allow us to determine whether spectro-temporal cues provide indispensable information for predicting the LG, while also controlling for potential confounds arising from the underlying model architecture.

2.1 Empirical data

The empirical data of SRT of plain and Lombard speech in Mandarin were previously published by [7]. The speech material consisted of Mandarin matrix test sentences [23], a syntactically fixed, five-word sentence test with a constrained vocabulary. Recordings were obtained from six male and five female native Mandarin speakers for both plain and Lombard speech. To induce Lombard speech, unmodulated speech noise with a long-term spectrum of a male speaker at normal vocal effort [24] (ICRA1) was presented to the speaker at a fixed level of 80 decibel (dB) sound pressure level (-SPL). This level was chosen based on previous studies, which showed that it reliably elicits Lombard speech while remaining below levels associated with hearing damage or excessive auditory or vocal fatigue [6]. The use of a fixed noise level was motivated by the aim of controlled elicitation of Lombard speech rather than ecological validity. The plain and Lombard speech recordings were subsequently used to measure speech recognition and assess the speaker-specific LG with a group of 13 native normal-hearing listeners. The SRT was measured adaptively with the procedure proposed by [25] in stationary ICRA1 noise. Twenty-two measurements were conducted with each listener so that the SRT was obtained for each speaker and speaking style (11 speakers x two speaking styles).

The average across listeners was used as speaker-specific SRT for each condition, and the uncertainty of the mean was estimated from the standard deviation across listeners. [7] reported speaker-specific SRTs in the range of dB to dB for plain speech and dB to dB for Lombard speech. The speaker-specific LGs varied from 3.2dB to dB, which indicated substantial inter-speaker variability. The smallest Lombard gain was on the same order as the average measurement uncertainty ( dB). Despite this variability, the data showed a significant mean LG across speakers.

2.2 FADE prediction

[12] proposed the framework of auditory discrimination experiments (FADE) for SRT modeling. FADE uses a hidden Markov model with a Gaussian mixture model (HMM-GMM) backend adapted from ASR systems. This design allows the model to evaluate intelligibility from the acoustic input while retaining the flexibility to exploit spectral and temporal cues without any a priori assumptions about their combination.

2.2.2 FADE backend: the generalized optimum detector.

Central to FADE is the concept of an optimum detector, which extracts the most relevant information from the input feature representation to perform the detection or discrimination task. Unlike the optimum detector approach of [29], FADE generalizes to any psychoacoustic discrimination task, such as speech recognition, using a HMM-GMM as ASR backend. This flexibility makes FADE a suitable tool for testing the importance of cues that contribute to LG in Mandarin Chinese. In contrast, traditional SI-models, such as the SII, make fixed assumptions about the importance of signal features.

The optimum detector operates in two steps: in the first step, the speech signals of the test material (i.e., the sound files of the speaker for the condition of interest) are mixed with random segments of masking noise across a range of SNR. HMM-GMMs are then trained on the labeled mixtures for each SNR. The underlying Markov model assumes eight consecutive states per word, three states representing silence between words, and six states representing the onset or offset silence of a sentence. For the definition of the grammar in the language model, word transitions are restricted according to the grammar of the matrix test. Importantly, this is not an additional modeling assumption, but rather reflects the experimental condition of listeners who are trained before actual measurements to get familiar with the limited vocabulary of the test and with the sentence structure. Including the same grammar in FADE therefore ensures that the model operates under the same structural constraints as the human listeners. Accordingly, FADE models an optimally trained human listener.

In the second step of the optimum detector, a new set of data, constructed in the same manner but using new segments of noise, is used to test the performance of the HMM-GMMs. The resulting model performance as a function of training- and testing-SNR defines the predicted SRT: the lowest test SNR at which word recognition rate reached 50% is returned as the SRT. Uncertainties of the predictions are estimated by bootstrapping the word-recognition performance.

For SRT predictions, FADE is run independently for each speaker and speaking style (plain, Lombard), ensuring that the optimum detector is always trained on the specific material for which the prediction is made. The LG predictions are the difference between the predictions for plain and Lombard speech:

(1)

Higher accuracy of LG predictions than of the absolute SRT predictions is theoretically possible because any systematic prediction bias that affects both plain and Lombard speech similarly cancels when computing the LG prediction. Consequently, a large bias of the absolute SRT prediction does not necessarily imply an equally bad performance of the LG prediction. While this mechanism provides a plausible explanation for the present results, the existence of such a shared prediction bias was not evaluated explicitly in this study.

The Markov assumption of the FADE backend with its eight states per word warrants consideration when interpreting the predictions based on spectral features. This is because, although only spectral information may enter the backend, the Markov chain could, in principle, model limited temporal structure through state transitions across successive frames. However, this temporal modeling is strongly constrained by the small number of states per word, which prevents a detailed representation of fine-grained temporal dynamics. For example, an upwards shift or second tone in Mandarin over 500 ms with a window shift of 10 ms would require in the order of 50 states for log-mel features, exceeding the available state range. To address this limitation and ensure a comparison with a model that does not rely on any implicit temporal features, a purely spectral model (the SII, see following Section 2.3) was included in this study for comparison.

2.3 SII-based prediction

To complement the FADE predictions, which may be sensitive to limited temporal information, the speech intelligibility index (SII) [20] was used as a purely spectral model. Unlike FADE, the SII does not incorporate temporal modulations or spectro-temporal interactions and assumes that the filtered long-term spectrum fully determines SI.

In this study, third-octave band-pass filters were implemented according to [30] and used for the calculation of the SII. While these filters differ from the log-mel or MFCC features of FADE, they share the essential property that only long-term spectral information is available to the model.

2.3.2 Reference condition for SII-based predictions.

To convert the SII into a predicted SRT, a reference SII (SII_ref) was calculated from empirically measured SRTs of the original Mandarin matrix test [23] in stationary ICRA1 noise. For a new speech in noise condition, the SNR is adjusted such that the SII matches the reference value:

(2)

The SNR that corresponds to this match is taken as predicted SRT.

2.4 Statistical measures for a comparison of models

When comparing a model prediction with empirical data, it is important to account for measurement uncertainties and to understand how well a ”perfect” model would perform.

Here, the ”perfect” SI-model is defined as one that predicts the empirical SRT exactly when the measurement uncertainties are zero. In practice, both measurement and model predictions are associated with uncertainties, so even a perfect model deviates from the empirical data.

To quantify the agreement between model predictions and empirical data while considering these uncertainties, the chi-squared measure of goodness of a model () was used:

(3)

where and are the empirical and predicted values, respectively, while and are the standard deviations of measurement and prediction uncertainties. Good agreement of the prediction B with a measurement A is reflected by a value of close to the mean of the underlying -distribution. This is equal to the number of independently predicted , also called the degrees of freedom .

A ratio (also called normalized ) close to one indicates that the model predictions deviate from empirical data by an amount consistent with the reported uncertainties. Values above one suggest that the model does not fully explain the data. can only happen if measurement and prediction errors are correlated or overestimated.

The likelihood that the observed deviation arises purely from uncertainty can be evaluated using the -distribution. The probability for the observed is the likelihood that the mismatch between observation and prediction is the result of measurement errors and prediction uncertainties.

Commonly used statistical measures such as Pearson correlation coefficient (R), bias, and root mean square error (RMS) are included for comparability. In general, a good model should exhibit high R, low RMS, and bias and in the range of unity.

3.1 Absolute SRT50 prediction

Statistical comparisons of empirical and predicted SRTs are summarized in Table 1.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Absolute SRT and LG prediction of empirical SRTs for Mandarin Chinese in ICRA1: Displayed are SII-based prediction and FADE predictions with different frontends. Pearson correlation coefficient (R), root mean square error (RMS), and bias are provided. gives the normalized chi-squared measure. gives the probability for above or equal to the reported value for a -distribution with degrees of freedom. (See section 2.4).

https://doi.org/10.1371/journal.pone.0356236.t001

For absolute SRT predictions with the FADE model, the prediction accuracy strongly depended on the feature representation used. The best overall agreement with the empirical absolute SRTs was observed for FADE using SGBFB features with a high correlation (R = 0.94), the lowest RMS error (0.75 dB), and negligible bias ( dB).

The normalized chi-squared was comparatively small (, ), indicating that, although deviations remain statistically significant, the prediction errors were substantially closer to the expected uncertainty range than for any other model–feature combination. FADE using spectral features (log mel or MFCC) achieved similarly high correlations; however, the RMS error increased by a factor of about two for log-mel features and by almost a factor of three for MFCC features. Similarly, the bias increased markedly: predictions based on log-mel features showed a systematic offset of 1.15 dB, whereas the MFCC-based predictions exhibited an even larger bias of 2.03 dB. The normalized chi-squared values reflected these deviations, with for log-mel and for MFCC, indicating that both spectral feature sets failed to reproduce the empirical SRTs within the expected uncertainty range.

The SII performed less accurately than any FADE variant. Correlation was moderate (R = 0.89), with an RMS error of 1.3 dB. The bias ( dB) was also noticeably larger in magnitude than for the FADE model with SGBFB. Most importantly, the normalized chi-squared remained high (, ), indicating that the discrepancies between SII predictions and measured SRTs were larger than expected from the measurement uncertainty.

The measured and predicted SRTs, separated by speaker and speech style, are presented in Figs 1–4. Although both models produced SRTs that were generally too low for speakers 10 and 11, these speakers were not excluded from the primary analysis because there was no indication of systematic deviation associated with speech style or speaker gender. Instead, they were treated as valid observations representing the natural variability of the dataset. To assess the influence of speakers 10 and 11 on the study conclusions, all statistical analyses were repeated after excluding these speakers (Table 2). The resulting performance metrics, statistical significance, and overall conclusions remained essentially unchanged, indicating that the reported findings are robust to their exclusion.

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Absolute SRT and LG prediction of empirical SRTs for Mandarin Chinese in ICRA1 without speakers 10 and 11: Displayed are SII-based prediction and FADE predictions with different frontends. (See Table 1). To test the influence of speakers 10 and 11 on the study conclusions, all statistical measures were repeated after excluding these speakers (see Table 2)..

https://doi.org/10.1371/journal.pone.0356236.t002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. SII-based absolute SRT prediction for Mandarin Chinese in ICRA1 with a Mandarin female reference speaker from the original Mandarin Matrix sentence test of [23] with gender-specific BIF of [31].

Each speaker is assigned to a number. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement uncertainties (reported as standard deviation) are displayed as error bars.

https://doi.org/10.1371/journal.pone.0356236.g001

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. FADE prediction of the absolute SRT based on log-mel features for Mandarin Chinese in ICRA1.

Log-mel features capture only spectral information. Each speaker is assigned to a number. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement as well as prediciton uncertainties (reported as standard deviation) are displayed as error bars.

https://doi.org/10.1371/journal.pone.0356236.g002

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. FADE prediction of the absolute SRT based on MFCC features for Mandarin Chinese in ICRA1.

MFCC features capture only spectral information. Each speaker is assigned to a number. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement as well as prediction uncertainties (reported as standard deviation) are displayed as error bars.

https://doi.org/10.1371/journal.pone.0356236.g003

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. FADE prediction of the absolute SRT based on SGBFB features for Mandarin Chinese in ICRA1.

SGBFB features encode spectro-temporal information within a single time-frame. Each speaker is assigned to a number. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement as well as prediction uncertainties (reported as standard deviation) are displayed as error bars.

https://doi.org/10.1371/journal.pone.0356236.g004

3.2 Lombard gain prediction

LG predictions are shown in Figs 5–8. Statistical comparisons are summarized in Table 1. FADE showed overall good agreement with the measured data. The log-mel features achieved the highest correlation (R = 0.87) together with the lowest error (RMS = 0.48 dB) and negligible bias (0.05 dB). The normalized chi-square value (,) indicated no significant difference between prediction mismatch and uncertainties.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. SII prediction of the Lombard gain for Mandarin Chinese in ICRA1 with a Mandarin female reference speaker from the original Mandarin Matrix sentence test of [23] with gender-specific BIF of [31].

Each speaker is assigned to the same number as in Fig 1–4. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement uncertainties (reported as standard deviation) are displayed as error bars. Overall, all approaches show equally accurate LG predictions.

https://doi.org/10.1371/journal.pone.0356236.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. FADE prediction of the Lombard gain for Mandarin Chinese in ICRA1 based on log-mel features.

Log-mel features capture only spectral information. Each speaker is assigned to the same number as in Fig 1–4. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement as well as prediction uncertainties (reported as standard deviation) are displayed as error bars. Overall, all approaches show equally accurate LG predictions.

https://doi.org/10.1371/journal.pone.0356236.g006

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. FADE prediction of the Lombard gain for Mandarin Chinese in ICRA1 based on MFCC features.

MFCC features capture only spectral information. Each speaker is assigned to the same number as in Fig 1–4. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement as well as prediction uncertainties (reported as standard deviation) are displayed as error bars. Overall, all approaches show equally accurate LG predictions.

https://doi.org/10.1371/journal.pone.0356236.g007

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. FADE prediction of the Lombard gain for Mandarin Chinese in ICRA1 based on SGBFB features.

SGBFB features encode spectro-temporal information within a single time-frame. Each speaker is assigned to the same number as in Fig 1–4. The dashed line represents a linear interpolation with a slope of unity. The bias is the offset between the dashed and continuous lines. Measurement as well as prediction uncertainties (reported as standard deviation) are displayed as error bars. Overall, all approaches show equally accurate LG predictions.

https://doi.org/10.1371/journal.pone.0356236.g008

FADE with SGBFB features, which was the best-performing configuration for absolute SRT predictions, also performed well for LG estimation. Its correlation (R = 0.82) and error (RMS = 0.57 dB) were slightly poorer than those obtained with log-mel features. The fit quality (, ) likewise indicated no statistically significant deviation from the data. Despite the small differences across metrics, SGBFB features maintained a low bias (0.11 dB) and stable performance across tasks. FADE with MFCC features showed the weakest performance among FADE variants, reflected in lower correlation (R = 0.78), slightly increased prediction error (RMS = 0.72 dB). However, the chi-squared statistic (, ) indicates that the mismatch is only slightly larger than expected from measurement and prediction uncertainties. Its bias was more pronounced (0.34 dB) but remained small in practical terms.

For comparison, the SII showed a similar correlation (R = 0.86) but a slightly higher error (RMS = 0.54 dB). Its bias remained small (0.17 dB), and the normalized chi-square value (, ) indicated an acceptable, though less precise, fit than the best FADE variants.

4.1 Absolute SRT50 prediction

The present results demonstrate that spectro-temporal features, i.e., spectro-temporal modulation patterns encoded within a single time-frame, are crucial for accurate, absolute SRT prediction. Speaker-specific SRTs can be predicted by both models; however, FADE with SGBFB features clearly outperformed all tested approaches. The SGBFB configuration of FADE showed the highest correlation with empirical data, the smallest RMS error, and no bias ( dB). Nevertheless, the measurement uncertainties were sufficiently small to reveal that neither FADE nor SII is a perfect model, as the remaining variance is higher than the expected variance from measurement uncertainties (p for all models; see Table 1). This indicates that factors beyond measurement uncertainties contribute to the mismatch and that more accurate models are theoretically possible.

One possible factor could be the limited frequency range of the FADE frontend, which sampled at 16kHz. This is supported by the findings of [32], who have shown that the frequency range above 8kHz significantly improves SI for normal hearing listeners. Improved SRT predictions within the FADE framework might be achieved by higher sampling rates in the frontend.

Nonetheless, the superior performance of FADE with SGBFB features for absolute SRT prediction suggests that encoding temporal information is essential for a speaker-specific SRT prediction, since only SGBFB capture dynamic spectro-temporal cues. Features such as individual time-frames of MFCC, and log-mel spectrograms, or the long-term spectrum used in the SII cannot encode these temporal aspects, which limits their predictive accuracy.

These findings align with [33], who demonstrated the advantage of gabor filter bank (GBFB) over MFCC in ASR tasks, highlighting the importance of spectro-temporal cues. Similar conclusions have been drawn across multiple languages, including German, Polish, Russian, Spanish, English, and Cantonese [14,19], especially in fluctuating noise conditions [28]. Together, these results reinforce the role of spectro-temporal encoding as a key determinant of absolute SRT predictions for speech in stationary noise.

4.2 Lombard gain prediction

In contrast to absolute SRT predictions, the prediction of the LG relied primarily on spectral cues, i.e., cues within a single time-frame did not need to contain information about temporal modulations. FADE predicted the LG accurately even when using only spectral features (log-mel or the derived MFCC), with normalized chi-squared close to unity, indicating that the remaining mismatch could be fully explained by prediction- and measurement uncertainties.

FADE with SGBFB features, although superior for absolute SRT predictions, did not provide additional predictive value for the LG, suggesting that the temporal information is largely redundant for this measure. The SII showed comparable results, demonstrating that the information in the filtered long-term spectrum is sufficient to capture the relative improvements in SRT due to Lombard speech.

These findings are consistent with the acoustical properties of Lombard speech in Mandarin [7], where spectral properties of Lombard speech have been shown to correlate with the LG.

4.3 General remarks

This study examined the role of spectral and spectro-temporal (i.e., spectral cues together with spectro-temporal modulation patterns encoded within a single time-frame.) acoustic cues in predicting both absolute SI and the LG in Mandarin Chinese. As only a single tonal language was considered here and a comparison to non-tonal languages is beyond the scope of the current study, we cannot resolve whether tonal languages make use of spectro-temporal encoding differently from non-tonal languages. However, we found that the spectro-temporal features of speech are essential for accurate SRT prediction of Mandarin Chinese, similar to non-tonal languages [14]. This suggests that spectro-temporal representations are required to capture speaker-specific intelligibility differences, even when lexical tone is an inherent property of the language.

Previous work with bilingual speakers of English and Cantonese [19] supports the view that spectro-temporal features are not language-specific but rather encode the physical properties of speech production adequately, such as time-dependent spectral filtering due to articulatory movements in the vocal tract.

In contrast, the prediction of the LG was equally accurate using FADE with either spectro-temporal or purely spectral features, as well as with the SII model. Within the measurement uncertainties of this study, spectro-temporal features did not provide additional predictive information beyond spectral cues for LG. This indicates that Lombard-induced intelligibility benefits in Mandarin are primarily driven by spectral changes, rather than by additional temporal information introduced by speaking rate or other dynamic modifications. At least in continuous ICRA1 noise, the Mandarin Chinese LG can be fully accounted for by spectral changes in the speech signal.

A limitation of this study is the size of the empirical dataset, which included 22 complete matrix tests from 13 listeners each. At the same time, it represents, to the authors’ knowledge, the most extensive and most precise Mandarin plain/Lombard SRT dataset published. Nevertheless, this sample size may still be insufficient to detect subtle contributions of temporal cues to the LG, given the measurement uncertainty of approximately 0.5 dB. Accordingly, the absence of a spectro-temporal effect on LG should be interpreted as an upper bound on its potential contribution rather than definitive proof of its absence. Future studies with more listeners may clarify whether spectro-temporal features contain a small, previously undetectable influence on LG.

Similarly, although stationary, speech-shaped noise (ICRA1) was used here, other types of maskers, such as fluctuating or competing speech, may highlight temporal contributions that were not observed in the present study. The use of the Mandarin matrix test with monosyllabic and disyllabic words provides a controlled stimulus set. Still, future studies with everyday sentences or connected speech may uncover additional temporal effects, particularly those related to prosodic or contextual speech dynamics.