Identifying the research question

The primary research question addressed was:’Have any previously developed or researched AV SIN tests been used to measure SRT?’ A secondary question investigated which of these tests incorporated remote or automated functionalities?

Identifying relevant studies

In this phase of study, we sought to establish the criteria for selecting publications to be included in the scoping review. Although scoping studies are inherently broad in scope, we deliberately identified specific criteria to guide our search process. The search strategy utilised for electronic databases was formulated based on our research questions and key concept underpinning the study. Prior to commencing the searches, two authors (AH and AG) reached a consensus on the relevant keywords for article retrieval. Searches were conducted from earliest records available up to February 18, 2025. The databases selected for this review included the Cochrane Library, IEEExplore, PubMed, Science Direct, and Web of Science, chosen for their comprehensive coverage of topics pertinent to health, engineering, social sciences, and psychology. Repositories containing grey literature were excluded from this systematic search, due to the absence of peer review which raised concerns regarding the authenticity, reliability, and reproducibility of the included work [35]. This exclusion was enforced to ensure that the retrieved literature would directly contribute to addressing the research question. The search terms employed in the database queries included “speech in noise,” “speech intelligibility,” “speech perception,” “speech recognition in noise,” “audiovisual,” “audio-visual,” and “auditory-visual.” These terms were consistently applied across all databases using boolean operators to structure the queries. Specifically, the key terms within each axis were combined using the “OR” operator and the search strategies for both axes were linked using the “AND” operator. The search commands were as follows: (“speech in noise” OR “speech intelligibility” OR “speech perception” OR “speech recognition in noise”) AND (“audiovisual” OR “audio-visual” OR “auditory-visual”). To minimise the risk of excluding relevant studies, no restrictions were placed on publication dates or languages.

Study selection

All studies published in English that addressed AV SIN tests which used SRT as measurement were considered for inclusion in this review. The primary objective was to identify any AV SIN tests that had been developed without imposing any restrictions on the date range for inclusion. Additionally, no limitations were placed on sample size or study design, as the focus was on analysing the methodologies and development employed in previous research within the scope of this review.

Incomplete papers, opinion pieces, book chapters, editorials, and grey literature were excluded from further review. The management of all search results citations including the identification and merging of duplicates was conducted using Paperpile LLC 2024 software. Subsequently, three stage selection process was implemented to evaluate the articles.

The inclusion criteria were established based on the following parameters: (1) studies involving AV SIN testing (2) no age restriction for participants, (3) measurement of the SRT, 4) no limitation on the language of stimuli, and (5) articles written in English. Articles excluded were: (1) measurements without SRT, (2) were not written in English, and 3) consisted of clinical commentaries, editorials, interviews, letters, newspaper articles, abstracts only, or non-peer-reviewed literature (e.g., thesis).

During stage 1, two authors (AH and KD) independently reviewed the titles and abstracts of all articles related to AV SIN tests. In the second stage, the full-text articles were meticulously examined by the same authors (AH and KD) to assess their eligibility. The final stage involved a third author (AG) who evaluated the articles flagged by the previous reviewers, due to discrepancies, resolving any disagreement to make the final determination on inclusion.

Ultimately, fully eligible articles were chosen for further analysis and synthesis (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A PRISMA flow chart that shows the process of article selection [48].

https://doi.org/10.1371/journal.pone.0338600.g001

Charting the data

The subsequent phase involved systematically organising the data extracted from the primary research reports under review. The data was structured according to the main research question to emphasise the primary findings. Each paper was meticulously read multiple times to ensure a comprehensive understanding of its aims, objectives, and findings, to ensure no relevant information was overlooked. The following information was systematically documented from each article: author(s), year of publication, study aim, research context, study participants, outcome measures, stimuli, language, test procedures and main findings.

Collating, summarising and reporting results

Papers meeting the inclusion criteria were meticulously examined, with their content subjected to rigorous analysis and assessment. Recurring themes, the various methodologies employed for recording the AV material, and the procedures utilised for determining SRT were critically scrutinised and presented in the following section.

Study selection

Articles were retrieved from five online databases, resulting in an initial set of 5261 papers. After eliminating duplicates and excluding irrelevant studies, 3536 articles remained. Title screening then led to the exclusion of 3340 articles, followed by abstract screening which further excluded 109 articles. This process resulted in a final selection of 87 articles for a comprehensive full-text review. Throughout this process, 76 articles were excluded based on the inclusion criteria due to reasons such as not having an SRT measurement, lack of AV integration or not being written in English. Consequently, the final number of studies included to address the research question was 11.

Study characteristics

The following reviews each of the 11 included studies, summarising their aims, methodologies and findings (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Study characteristics and main findings.

https://doi.org/10.1371/journal.pone.0338600.t001

Stimuli and measures

An analysis of the studies revealed substantial variability in both the speech stimuli and measurement methodologies employed for AV SIN testing. The speech materials ranged from basic digit triplets to more intricate sentence-based stimuli. Several studies utilised standardised speech tests, such as matrix sentence tests, BKB sentences, and IEEE sentences, while others developed bespoke materials, including passages from the CST or novel sentence lists. In terms of measurement, most studies focused on determining SRTs, albeit with varying target percentages. Whilst many studies employed the conventional 50% correct performance criterion, others explored alternative thresholds. such as 80% SRT, whilst another study examined multiple thresholds, including 5%, 50%, 80%, and 95%. Furthermore, another study calculated SRTs based on the mean of the final six SNR ratio values. These variations in SRT percentages reflect ongoing efforts to optimise sensitivity and mitigate ceiling effects in AV testing. Adaptive procedures were commonly used for efficient SRT estimation, although some studies incorporated fixed SNRs to characterise performance across diverse listening conditions. Scoring methodologies encompassed both keyword and whole-sentence approaches (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Critical appraisal of methodological strengths and limitations of included studies.

https://doi.org/10.1371/journal.pone.0338600.t002

Visual integration

The studies reviewed utilised a range of methods to present visual speech information and assess its integration with auditory cues. Several studies employed video recordings of real speakers L. Arnold et al. [36]; Bernstein and Grant [37]; Cox et al. [38]; Llorach et al. [39, 40]; Van de Rijt et al. [41]; Le Rhun et al. [42] providing naturalistic visual speech cues that facilitated the integration process. In contrast, other studies employed virtual human speakers Choudhary et al. [43]; Devesse et al. [44]; Schreitmüller et al. [45], which despite sacrificing some realism, allowed for enhanced control over visuals elements such as lip contrast and head scaling. Although the extent of visual benefit varied across studies, it was consistently significant, with improvements in SRTs ranging from 1.5 to 5 dB when visual cues were incorporated. Furthermore, Van de Rijt et al. [41] and Bernstein and Grant [37] provided empirical support for the principle of inverse effectiveness, demonstrating that the benefit of visual cues was maximised at intermediate SNRs where AO performance was neither excessively high nor low.

Remote testing

A key finding of this review was the lack of remote testing capabilities in the AV SIN tests examined. None of the 11 studies incorporated methods for administering tests remotely or via telehealth platforms. All tests were conducted in controlled laboratory or clinical environments, with participants required to be physically present for the testing sessions.

Future innovations could leverage virtual reality (VR) to create immersive and standardised testing environments or employ AI-generated, photorealistic avatars to offer precise control over visual speech cues, overcoming the inconsistencies of video recordings.

Speech material and masking noise

The reviewed studies demonstrate significant progress in developing AV SIN tests across different languages and populations, using a diverse array of speech materials and masking noise. The studies included matrix sentence tests [39,42,45], which are beneficial due to their highly controlled vocabulary and syntactic structure, ease of adaptability across languages, and extensive number of possible combinations, rendering them suitable for repeated measures. Additionally, more naturalistic sentence materials [44] have been validated and standardised for clinical application, while word-level stimuli (L. [36]) represent an alternative approach.

Choudhary et al. [43] used digit triplets, which offer simplicity and rapid administration, thereby minimizing linguistic confounds. However, this material does not accurately reflect the complexities of real-world communication challenges, nor does it assess sentence-level processing. MacLeod and Summerfield [40] utilised BKB sentences, which are standardised and widely used in research. Despite their utility, these sentences may be constrained by the limited number of available material and may not adequately reflect the complexity of adult conversational discourse. Bernstein and Grant (2009) utilised IEEE sentences, which are phonetically balanced and commonly employed in speech testing. Although this corpus is known for its low context predictability, it could be limited by the number of sentences available when compared to matrix sentences (720 sentences versus the possibility of 100,000 possible sentence combinations). Cox et al. [38] used material from the CST test, which consists of 48 passages of conversational speech. While this test has high ecological validity, the use of conversational speech makes it more difficult to control for linguistic factors, and scoring can be more complex and time-consuming.

This variety reflects a tension between the need for controlled, comparable stimuli and the desire for ecological validity. Future development of AV SIN tests should strategically balance competing factors by incorporating a diverse range of speech materials. These materials should include single words and sentences, thereby enabling a comprehensive assessment of AV speech perception across multiple levels of linguistic complexity, from phoneme recognition to contextual comprehension.

The diversity of masking noises used across studies, including speech-shaped noise, multi-talker babble, and modulated noise, whilst some reflect the complexity of real-world listening environments others are used for laboratory experiments. While speech-shaped noise offers consistent energetic masking, it may not fully capture the informational masking effects encountered in everyday listening situations. The comparative analysis of various noise types within individual studies, as done by Bernstein and Grant [37], is particularly valuable in elucidating how different masker characteristics interact with AV integration processes. These maskers often fall into two categories; energetic masking, where noise overlaps spectrally with the speech signal, obstructing it at the auditory periphery and informational masking which arises from cognitive interference, such as competing speech signals that are perceptually similar to the target. Future AV SIN tests could benefit from incorporating adaptive technologies that manipulate both the type and intensity of background noise to offer a more comprehensive assessment of AV speech perception in challenging listening conditions.

It is critical to note that the studies identified in this review were exclusively conducted in Western countries, utilising materials in English, German, French, and Dutch. This reveals a significant geographic and linguistic gap in the literature. There is a notable absence of AV-SIN test development and validation in languages from Asia, Africa, and South America. Given that linguistic and cultural factors can significantly influence speech perception, the direct application of existing tests to diverse global populations is not appropriate. Future research should prioritise the development of culturally and linguistically adapted AV-SIN tests to ensure that the benefits of this assessment methodology are accessible globally and relevant to diverse patient populations.

Procedural aspects and scoring methods

The review underscored several important methodological considerations for AV SIN testing, including the choice between adaptive procedures and fixed SNR measurements, as well as differences in scoring methods and the number of trials. Adaptive procedures for estimating SRTs offer efficiency and precision but may not capture the full range of performance across different SNRs. The approach taken by Van de Rijt et al. [41] of measuring performance across a fixed range of SNRs provides valuable insights into the shape of the psychometric function for AV speech perception, revealing important effects such as inverse effectiveness [41]. Inverse effectiveness is the principle that the benefit of combining auditory and visual information grows as the individual signals become less reliable.

Another crucial aspect to consider is the selection of an appropriate SRT percentage. Studies have demonstrated that a 50% threshold may be too easy to reach when visual cues are present due to the benefits of lipreading [39]. Scoring methods also varied, ranging from keyword scoring for sentences to more detailed phoneme-level scoring, each offering different balances between efficiency and the depth of information obtained. Additionally, studies differed in trial count, with test lists ranging from 10 to 30 sentences, underscoring the need for further research to find the optimal balance between test reliability and administration time.

Future advancement in AV SIN tests could benefit from integrating diverse approaches. Employing adaptive procedures to rapidly estimate SRTs while alongside fixed SNRs to characterise the full psychometric function, may enhance test accuracy. Additionally, incorporating multiple scoring levels could provide a comprehensive assessment, offering both overall intelligibility measures and detailed insights into AV integration.

Visual integration

The exploration of both video recordings and virtual human speakers for presenting visual speech information signifies an important area of innovation in AV SIN testing. Although video recordings of real speakers currently offer the most naturalistic visual cues, the potential benefits of virtual humans, in terms of stimulus control and flexibility, make this a promising avenue for future research. An important consideration is the simultaneous recording of both visuals and auditory, as studies have shown that dubbing during post-processing can introduce inconsistencies [39, 42].

The consistent finding of significant visual benefits across studies, with SRT improvements ranging from 1.5 to 5 dB, underscores the importance of incorporating visual cues in speech intelligibility assessments. The evidence for inverse effectiveness observed in several studies has significant implications for test design and the results interpretation. Future AV SIN tests should be designed to capture this phenomenon, by adjusting SNRs to optimise AV integration for each individual. Additionally, further research is needed to examine individual differences in the capacity to integrate auditory and visual speech cues, which have meaningful implications for developing rehabilitative strategies in clinical populations.

Remote testing

The absence of remote testing capabilities in current AV SIN assessments highlights a significant gap in the field. As technology has advanced, particularly in recent years, the potential for remote administration of speech perception tests has grown significantly. With the growing importance of telehealth in audiology and the global shift towards remote healthcare delivery [47,50], it is imperative to prioritise the development of AV SIN tests that can be reliably administered in remote settings. Leveraging improved connectivity and emerging telehealth platforms, future research may increasingly incorporate remote testing methods to address this need.

However, the transition to remote AV SIN testing presents several challenges. Ensuring consistent audiovisual presentation across diverse devices and varying internet connections is critical. Additionally, safeguarding test integrity in unsupervised settings poses unique difficulties. Potential solutions may involve the development of specialised software or web-based platforms for AV SIN test delivery, integrating automated calibration procedures to standardise device output, and employing advanced encryption and authentication technologies to protect test materials.

Rigorous research will also be necessary to validate remote versions of AV SIN tests by comparing them with in-person administration to ensure equivalence of results. The successful advancement of remote AV SIN testing capabilities would substantially enhance the accessibility and clinical applicability of these assessments, facilitating broader adoption in both research and clinical practice. This development has the potential to revolutionise speech perception testing, bridging gaps in accessibility and aligning with the broader trends in telehealth innovation.

Strengths and limitations

A primary strength of this scoping review is its specific focus on SRTs measured through AV tests. This focus, however, inherently excluded a substantial body of research that examines AV speech perception without formally quantifying thresholds. Much of the research within the fields of psychology and neuroscience explores behavioural and neurological mechanisms underlying the integration of auditory and visual inputs during language processing without calculating speech reception scores. As a result, our review may have overlooked significant insights that this broader literature on AV speech perception could have provided. Nevertheless, we chose to focus specifically on SRTs because they offer a quantifiable and clinically relevant measure of speech intelligibility, which is particularly important for evaluating and comparing the performance of AV-based hearing assessments and interventions. This strategic focus allowed for the identification of AV assessments that were expressly designed and validated to measure speech intelligibility thresholds. Although the application of the SRT criterion resulted in a more limited pool of eligible studies, it allowed for a more focused mapping of AV testing methodologies specifically designed to quantify visual speech intelligibility gains. By narrowing the scope in this way, the review highlights the range of existing tools, their key characteristics, and areas where further methodological development or validation may be needed.

Furthermore, the decision to include only studies published in peer-reviewed journals may have introduced a degree of publication bias. Research on AV speech testing that is documented in unpublished manuscripts, conference proceedings, and dissertations was likely excluded. Expanding the scope of future reviews to include this so-called grey literature could yield additional perspectives and insights. An additional advantage of our inclusion criteria was the lack of restrictions regarding publication dates. This inclusive approach provided a comprehensive view of the development trajectory of AV speech testing, encompassing efforts that span several decades. By reviewing foundational works, we were able to identify early pioneers who explored the use of visual cues to enhance speech perception, striving towards the optimisation and validation of AV speech tests.

Similarly, our review identified a lack of research focusing on diverse populations. Although one study developed a test for paediatric use, most of the research cantered on monolingual adults with normal hearing or post-lingual hearing loss. The unique challenges faced by multilingual individuals, pre-lingually deaf children, or individuals with comorbid cognitive or visual impairments were largely unaddressed. The applicability of these findings to populations in low-resource settings, where access to technology and clinical expertise is limited, also remains unexplored. This limits the global generalisability of the current body of evidence and underscores the need for research that is more inclusive of diverse participant groups.