The EyeMusic algorithm

In the present study we used the EyeMusic a visual-to-auditory SSD developed in our lab, which transforms whole-visual images into auditory inputs, termed soundscapes, preserving shape, location and even color of the objects in the scene [3]. Specifically, the EyeMusic algorithm down-samples every image to a 30x50 pixels matrix and conveys the x-axis visual information via a left-to-right sweep-line, such that visual features on the left of the image are heard before those on the right. The y-axis positions are conveyed through pitch manipulations, e.g., high-pitched musical notes represent high locations in the image. Different colors are conveyed via different musical instruments (see [3] for full description of the algorithm). In this experiment we used three colors, red, white and blue. Silence is conveyed by an additional forth color, black (see Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. EyeMusic description.

The EyeMusic visual-to-auditory SSD transforms visual information into auditory soundscapes. X-axis information is conveyed through time, such that information on the left side of the image is heard before the information on the right. Y-axis information is conveyed through pitch manipulations on the pentatonic scale, such that objects ‘features positioned in the higher portions of the image are sonified with a higher pitch than lower features. Colors are conveyed through timbre variations using different musical instruments. In the current experiment, we used the colors red (piano), white (choir) and blue (trumpet), while silence conveyed black. The orange box, sweeps from left to right, sonifying one column at a time.

https://doi.org/10.1371/journal.pone.0250281.g001

The EyeMusic has been used successfully for a variety of tasks exploring questions such as sensory-motor information transfer [31], testing visual acuity [32], examining the neural correlates of SSD-presented letters and numbers [33], focusing on particular details of the visual scene and then integrating them into a combined whole [34], and even in practical real world tasks based on shape and color information, such as finding vegetables at the supermarket [13].

The online version of the EyeMusic

To maximize the usability and distribution of the self-training program, we created an online version of the EyeMusic, which could be accessed via a dedicated website. This website, which includes step-by-step lessons of increasing difficulty for self-training on the EyeMusic SSD, among other EyeMusic related content, was written using ASP.NET MVC technology. One main advantage of an online EyeMusic training platform is that users do not need to install any program to train on the EyeMusic SSD and can train with the algorithm by themselves and at their own pace. Additionally, all the activities of the users are saved automatically on a SQL server database for analyses purposes.

Participants

Fifty sighted individuals (25 females), aged 26.64±5 years (mean ± SD), participated in this study. The participants were randomly assigned to five groups: Auditory only unisensory feedback (N = 10, 6 females, mean age 24.9+1.57); Interleaved audio-visual, multisensory, feedback (N = 10, 4 females, mean age 25.3+2.9); Simultaneous audio-visual, multisensory, feedback (N = 10, 5 females, mean age 28+4.87); Simultaneous textual description, multisensory, feedback (N = 10, 4 females, mean age 29.5+8.12); Control (N = 10, 6 females, mean age 25.5 +2.45). All participants were naïve to the EyeMusic SSD algorithm as well as to any other SSDs. All participants stated they have normal or corrected-to-normal hearing and vision.

Participants were compensated for their time and received an additional motivation bonus depending on the lesson-level of EyeMusic they reached at the end of the experiment or, for the control group, on their success rate in the second repetition of the SSD-stimuli identification test, which the other participants performed after EyeMusic training (see details in the next paragraph and see Fig 2). The research protocol was approved by the ethics committee of the Interdisciplinary Center (IDC), Hertzeliya. All participants signed an informed consent form before starting the experiment.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Experimental flow.

The experiment included 5 groups of sighted participants, 4 experimental groups, and 1 control group. All participants performed a baseline SSD identification test, and repeated the same test after ~75 minutes. Between tests, the 4 experimental groups participated in a self-learning online training program consisting of 9 step-by-step lessons of increasing difficulty guiding them through the basic principles of the EyeMusic. The feedback method deployed during training to teach the participants to interpret the auditory stimuli of the EyeMusic, varied among groups: 1 Auditory only unisensory group receiving an auditory description of the stimuli after each EyeMusic stimulus; and 3 Audio-visual multi-sensory groups—2 groups perceiving visual images appearing either simultaneously or following the EyeMusic stimuli; 1 group receiving textual descriptions of the stimuli alongside hearing the auditory stimulus; In the control group participants were instructed to free reading from the computer during the ~75 minutes between the two SSD identification tasks.

https://doi.org/10.1371/journal.pone.0250281.g002

Experimental setup & procedure

The experiment was conducted on standard PCs (laptop or desktop computers), using standard off-the-shelf headphones, keyboard and mouse.

We developed a self-training program to teach the basic principles of the EyeMusic SSD and tested its efficacy using two identical SSD-stimuli identification tests, interleaved by ~75 minutes of self-training, which comprised a series of 9 step-by-step lessons of increasing difficulty. Before moving on to the next lesson, participants were presented with two self-assessment questions regarding their perceived learning and difficulty of the lesson they just concluded, and were required to answer a short forced-choice SSD-stimuli identification quiz on the material covered during the concluded lesson, aiming at quantitatively assessing the learning of the participants (see Fig 2 for the experimental flow).

Before starting the experimental procedure, participants received a brief verbal explanation on the concept of SSD and on the basic principles of the EyeMusic algorithm. Then, without ever hearing any EyeMusic soundscape, they performed the first SSD-stimuli identification test, which lasted ~7 minutes. Then they started the training procedure which was stopped after ~75 minutes, independently of whether participants completed all the lessons. Finally, the post-training test, which lasted ~7 minutes, started automatically (see Fig 2 and next paragraphs for details on the training procedure). To minimize tiredness of participants, we set the total duration of the experiment, including pre- and post-training tests, to ~90 minutes and this is why the training was automatically stopped after ~75 minutes.

Multisensory vs. unisensory training

Contrary to the wealth of evidence reporting better learning during multisensory than unisensory stimulations [17–22, 45], our results showed no significant difference between multisensory and unisensory training approaches. Specifically, we did not observe any significant difference among the improvement levels reached at the end of the four training programs in the post-training test. Nonetheless, we observed some tendencies for an advantage of multisensory audio-visual over auditory unisensory training strategies. For instance, when looking at individual success rate in the pre- and post-training SSD identification test, we observed that in the interleaved audio-visual and simultaneous audio-visual multisensory training groups, 100% of the participants improved their success rate in the post-training test compared to their score in the pre-training test, while in the textual group, and especially in the auditory unisensory training group, such improvement was less consistent across participants (see Fig 4). Additionally, when considering how many training lessons participants successfully completed before the training was stopped after ~75 minutes, we observed that participants from all three multisensory groups (interleaved audio-visual, simultaneous audio-visual and textual) tended to complete more lessons of our self-training program. Interestingly, we showed that the individual number of successfully completed lessons significantly correlated with the overall success rate achieved in the post-training test. This latter result, in turn, corroborates the conclusion that the multisensory approach tended to be more effective than the unisensory one. Note however, that this result might be at least partially due to differences in the speed of processing between vision and audition. Indeed, the perception of the visual feedback (received in all three multisensory training groups) is quicker than the auditory one (received in the unisensory training), thus potentially making the advancement in the entire training program faster.

Additionally, we observed that participants in the interleaved audio-visual and auditory unisensory training groups tended to repeat overall less lessons (i.e., more often reached an accuracy >70% in the end-lesson quiz in their first attempt) compared to participants in the simultaneous multisensory and reading groups. Moreover, the individual repetition ratio significantly correlated with the success rate in the post-training test. This suggests, in turn, that the auditory and audio-visual training programs tended to be more effective in teaching the basics of the EyeMusic. Nicely, these results fit well with participants’ end-lessons self-assessment, where we observed a tendency of participants from the interleaved audio-visual and auditory unisensory groups to scale their learning as higher, and the level of difficulty as lower. Thus, both these tendencies together suggest that an interleaved training approach, whether unisensory or multisensory, seems to be more efficient than simultaneous training approaches for the initial stages of SSD training. This result is also in line with previous evidence showing the effectiveness of interleaved multisensory training in improving sound localization of deafened ferrets [46]. The effectiveness of an interleaved training strategy is probably due to the fact that this approach forces participants to focus more on the novel sensory information which is presented alone, compared to simultaneous training strategies where the focus of attention towards the novel sensory information might diminish in favor of the supplementary, more familiar sensory input.

All the aforementioned results together, highlight a tendency for the interleaved audio-visual approach to be the most efficient feedback strategy during training (i.e., highest number of completed lessons; lowest number of lessons’ repetition ratio and better end-lesson self-assessment scores), even though at a pure statistical level, we did not find any difference among the various training approaches. One possibility is that our group sizes were too small to catch potential significant differences in this regard. Another possibility for this lack of a statistical advantage of multisensory strategies is that we trained here the basic principles of the EyeMusic SSD, using relatively simple stimuli (i.e., lines and simple shapes). Indeed, the inverse effectiveness rule which is used to determine the effectiveness of multisensory stimulations on perception, postulates that multisensory enhancement has higher efficacy in perceptual situations in which one of the two sensory inputs is either deteriorated or very complex [47, 48]. It might be that in the case of basic SSD properties, the auditory signal is not complex enough to significantly benefit from additional multisensory inputs. This option is further strengthened by the average accuracy at baseline, without any SSD training, in identifying SSD stimuli, which resulted significantly higher than chance level.

Currently, the training duration was relatively short, ~75 minutes. We choose a relatively short training duration since we were interested in investigating the efficacy of relatively quick self-training programs. A short training program is important, as many potential SSD users are reluctant to use these devices due to the long training required for mastering them. Overall, our current findings show that self-training on the initial learning phases of an SSD algorithm is indeed possible and can efficiently occur relatively quickly. However, in our study, participants’ accuracy level after training was only at around 63%, namely still far from ceiling (see [15] for similar findings). Thus, we assume that with longer training, participants’ overall accuracy rate could further increase. Additionally, longer training programs will enable to introduce more complex SSD soundscapes, making the learning more useful for real-life tasks. Possibly, with longer training and a bigger sample, differences in the final outcomes among the various training strategies will become more apparent and might unravel an advantage of multisensory over unisensory approaches. We think that an initial shorter and entirely autonomous training program, might serve the crucial function of intriguing the users, ultimately encouraging them to further train on the device with a longer training program aimed at achieving more benefits from the use of SSDs.

Testing sighted participants as a proof-of-concept for the efficacy of SSD self-training

As visual-to-auditory SSDs are mainly aimed at being assistive technology for the blind and visually impaired population to convey visual information and ultimately maximize their independent interactions with the environment, this stake-holder population is also the final target of the current self-training SSD program. Here however we tested only sighted individuals, as a proof-of-concept for the feasibility of this approach and as a first step towards the identification of the most suited training strategy. Specifically, testing the sighted population allowed us to easily create a self-training platform which was able to deliver multisensory training lessons based on audio-visual pairing, which is easily and freely available in an online platform. Implementing multisensory training for blind individuals would have entailed the involvement of audio-tactile inputs, requiring an additional hardware component (i.e., to deliver tactile SSD stimulations), which both raises costs and is harder to adapt to an online platform as it would require constant maintenance. It is important though to note, that multisensory audio-visual approaches can potentially impact the rehabilitative aspects of visually impaired individuals, for instance, individuals with residual vision, or with degenerative visual loss (see for instance [49, 50] suggesting the coupling of SSDs with sight restoration approaches). Our current results on the sighted population show that for this initial stage of SSD training, the unisensory and multisensory, perceptual and descriptive training methods were equally efficient. This suggests that a self-training program tailored to the blind population using a unisensory descriptive auditory feedback training strategy to teach them the basic principles of the visual-to-auditory EyeMusic SSD, might be effective.

However, the fact that we used only sighted participants is obviously also a limitation for the translational aspect of this work. Note though, that our platform has been already designed in a fully accessible manner, thus making the testing of the blind and visually impaired populations relatively straightforward to implement in future works. Furthermore, the fact that we did not find any significant difference in the learning outcomes following perceptual (interleaved and simultaneous audio-visual) and descriptive (auditory-only textual reading) feedback training strategies, strengthens the hypothesis that audio-only descriptive feedback might be effective for conveying basic SSD-transformed visual content to both congenital and late blind visual-to-auditory SSD users. Indeed, the visual content delivered in the current training programs relates to visual concepts that are familiar also to congenital blind individuals (e.g. size, line orientations, simple shapes are commonly perceived by blind people via the tactile modality). We hypothesize auditory description might suffice also for successfully conveying color: although obviously fully blind individuals cannot perceive color via other sensory modalities, the concept of color is semantically familiar to them (i.e., they are constantly exposed to colors during linguistic interactions). Importantly, the current training only requires associating specific colors with specific sounds, thus remaining in a conceptual domain. Therefore, we predict that such an online self-training program will be successful also in blind individuals. Note that, in addition, previous studies comparing the use of SSDs between blind and sighted participants have shown that SSD learning can be effective in both groups [5, 32, 51–54]. Therefore, future studies directly testing the applicability of such online training in blind individuals could also add to this latter literature. Potentially, the online (and free) nature of this self-training we developed, and its possible use without sighted assistance, will significantly increases its availability to the blind community, a problem often limiting previous training initiatives which required travel, and the high cost of the offered training programs [9]. Such an online platform, together with further development of the training programs, might then succeed in loosening the training bottleneck and in spreading the use of SSDs among blind people. For instance, this online self-training can be extended to include also more active training, which can potentially boost the users performance [55–57].

Finally, while the primary use of most SSD transformations is for sensory rehabilitation, they can also be used potentially for sensory augmentation. In such use cases, sighted users learning a visual-to-auditory tranformation (e.g. where the visual information might be coming from a heat sensor) might consider using visual input as part of their training, and our results are relevant to these use cases as well.

Dynamic vs. static training

The training used here, in the self-training program, is the basic and most common method, namely using a series of static stimuli [36, 58, 59]. However, while effective, as demonstrated here, this type of learning shows limitations when aiming at training for real world scenarios, and can quickly become boring for the users ultimately harming users’ motivation. Many evidence suggest that adding more dynamic aspects to training environments boosts learning and also increases users’ enjoyment [60–66]. Thus, the next steps of this training program should include dynamic scenarios such as games, and tailored virtual environments [67], which we are currently testing. Finally, the last stage of SSD training will involve full immersive use of SSDs in the real world. Note that perceiving objects or full scenes via SSDs is a very complex task, requiring dedication and often personalized feedback. Thus, we propose that for promoting the use of SSDs in real-life, the final training solution might probably entail a combination of an initial relatively-short and entirely self-monitored training program, followed by a longer training program carried out in a mixture of self-learning and supervised training with an instructor (and potentially in the future using artificial intelligence allowing the individualization of the training content and strategies based on the user’s performances). Potentially, this combined training approach will promote the overall and everyday use of SSDs. These training programs should keep in mind the blind target population, including congenitally blind individuals, to whom some of the visual concepts, such as depth can be novel. Special thought needs to be given towards the translation of these concepts to their available sensory experiences (see the work of Renier & De Volder regarding depth perception via SSDs [68]).

Another important aspect of our self-training approach is the possibility to measure training parameters, and to control for the exact training history and/or training level of participants while using the SSD. Despite numerous imaging studies which have shown the recruitment of the deprived visual cortex by auditory SSD inputs after training (cross-modal plasticity) [69–73], there has yet to be a sufficiently systematic exploration of the neural correlates of the different stages of this cross-modal recruitment, based on the level of proficiency in using SSDs. This tool can be crucial in providing controlled parameters for this exploration.