Fig 1.
The EyeMusic SSD and experimental setup.
(A) A 3rd person view of the user's avatar and field of view, the contents of which will be sonified in the EyeMusic's algorithm shown below it. (B) Illustrates the principles of the EyeMusic visual-to-audio transformation: after down-sampling images to 24*40, each pixel in the image is assigned a value such that its y-axis value is converted to a musical note (the higher the value the higher the note), its x-axis value to time (the closer to the left the earlier in the soundscape), its brightness to volume and its color to musical instrument (C) Illustration of the experimental setup.
Table 1.
Information on the congenitally blind participants of experiment 1.
Fig 2.
(A) Two screenshots from Task 1, from above demonstrating the level's layout (top) and from the user's 1st person view demonstrating a triangular door and a square door seen at an angle (bottom) (B) Two screenshots from Experiment1:Task 2, from above demonstrating the level's layout (top) and from the users 1st person view demonstrating trees, a house and a door (bottom).
Fig 3.
Screenshots from Experiment1:Task 3.
Including views from directly above (top left), from above at an angle (top right) and screenshots to be sonified from the user's 1st person perspective showing houses, trees and a crosswalk (bottom).
Fig 4.
Scores for Detection of the different door, Explaining why it is different and Navigating to it correctly in Experiment1:Task 1 (blue), and for Detection of the correct door and navigating through it in Experiment1:Task 2 (green). Error bars represent SD.
Fig 5.
Results of the two sighted-blindfolded groups of Experiment 2.
Detection of the different door, Explaining why it is different and Navigating to it correctly in Task 1. Error bars represent SD.
Fig 6.
Auditory representation from different distances and angles.
A demonstration of how the same scene or object can have very different representations both visually (shown here as screen shots) and auditorily (shown here as the corresponding EyeMusic waveforms) from different distances and angles, creating a significant challenge to maintaining object constancy and object identification. ((A) From Experiment1:task1, (B) From Experiment1:task2).