2.1.1 Hardware.

The FAD was developed in collaboration with ARIA Research. The FAD was co-designed with O&M specialists and multiple people with BLV (four pilot testers). We developed the FAD to explore the functionality of the acoustic touch sensory augmentation paradigm. Fig 3 demonstrates the functionality of the FAD app during the study with the mobile app view. The system consists of the NReal Light AR Glasses [42] (Fig 2A) and an OPPO Find X3 Pro Android phone [43]. The NReal glasses were chosen because of the weight (88g), computer vision support (RGB and 2x5MP cameras), 6-degree-of-freedom inertial measurement unit (IMU), binaural speakers, and compatibility with the Android Unity SDK. We attached motion capture reflective markers to the NReal smart glasses (as shown in Fig 2A) that enabled the tracking of head movement and rotations during the study. The OPPO smartphone was chosen based on processor performance and RAM size to support the deep learning object recognition model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. A demonstration of the NReal glasses, OPPO android smartphone, and the sonification app being used to detect a cup and bottle within the user’s FOV.

https://doi.org/10.1371/journal.pone.0290431.g003

2.1.2 Object recognition.

The FAD Fig 4 was developed using the Unity Game Engine 2022.1.0b2 version [44]. The Unity SDK managed the audio input and camera/head-tracking output of the NReal Glasses. Image object recognition was implemented within the Unity scene using the YOLOv5m [45] classification model. We also tested the YOLOv5s, YOLOv5l, and YOLOv5x models. but found that the m model provided the best balance of classification accuracy, inference speed, and app stability on the smartphone. The model used pre-trained weights from the COCO dataset, which contained 80 object classes [46]. For performance and app stability, the input size was set to 512x320, and the aspect ratio to 16:9. Higher resolution of the input data improved the performance but reduced the inference speed and app stability. The object recognition performed at an average of 66 ms inference speed with a power consumption of 4.5 W.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. A block diagram outlining the connections between the technical components and the study’s users (researchers and participants).

The arrows represent the flow of information with the text specifying the type of information.

https://doi.org/10.1371/journal.pone.0290431.g004

In addition to object recognition, the object’s distance from the camera position is also determined (as shown in Fig 3). The distance is calculated using the stereo cameras on the glasses. We used the Semi-Global Block Matching (SGBM) [47] stereo-matching algorithm in OpenCV to calculate the depth map. Then, we used the Weighted Least Squares (WLS) filter [48] to smooth the noise in the disparity map. We aligned the depth map to the RGB frame using the cameras’ calibrations. For each object, the distance was determined by calculating the average depth inside a small region (50x50) within the centre of the bounding box. Detected objects are placed into the Unity virtual environment in a position relative to the head (tracked by the IMU). For the study, we limited the object classification to four selected household items, a book, bottle, bowl, and cup. Each item was assigned an appropriate sound: the book—page turning sound, bottle—scraping a glass bottle sound, bowl—lid placed on a bowl sound, and cup—a cup placed on a wooden table sound.

2.1.3 Auditory icon trigger parameters.

The auditory FOV and sonification parameters were achieved through a trapezoidal frustum-shaped FOV by the Unity game engine. As the user’s head rotates, the repetition rate of the auditory icon sound changes based on the position of the item within the auditory FOV (the fastest repetition rate in the center of the view). During the app’s development stage, we invited four pilot participants with BLV to test the FAD and the early protocols of the study. Based on the participant feedback, we selected the size of the auditory FOV, the number of FOV layers, and the sound-based repetition rate. We determined an auditory FOV size of 36.6° horizontally and 20° vertically. The FOV contained 10 layers with a 20 ms incremental delay (at the outer edge, there is a 100 ms delay between sound repetitions). This functionality improved the user’s ability to precisely localize an item by fixing it to the center of their view.

2.3.1 Seated task.

Fig 5 outlines the flow of a single trial (three different conditions) for the seated task. The seated task was intended to compare the performance of the FAD against two idealized conditions (Clock and Speakers). This task contains the following three conditions.

• FAD: This condition involves using the FAD device to scan and sonify the items on the table, creating a mental spatial map of the items on the table. During the item-searching phase, the participants were asked to identify the items they heard/recognized verbally.
• Clock: Simulating a sighted guide or smart glasses instructing the participant on the location of each item using the spatial layout of a clock face. The instruction speaker informed the participants of the location of each item using the format of item name followed by the location on the clock (“12 o’clock Bowl”). As shown in Fig 5, from left (L, Position 1) to the right (R, Position 5): Position 1 is “9 o’clock”, Position 2 is “between 10 and 11 o’clock”, Position 3 is “12 o’clock”, Position 4 is “between 1 and 2 o’clock”, and Position 5 is “3 o’clock”.
• Speakers: Real spatial sound is played by a loudspeaker co-located with each item on the table. This condition involves the placement of Bluetooth speakers in front of the items. These speakers will sequentially play the auditory icon three times before moving to the next item. The play sequence was ordered from left to right.

This task functionally measures three components of FAD: the ability to detect an item, recognize the sound, and memorize the position of the items. The seated task consisted of 60 trials equally distributed across three conditions (20 FAD, 20 Clock, 20 Speakers). We also varied the number of items (2 or 3) on the table for the Seated task. The task contained three conditions that were interwoven in a pseudo-randomized manner. A rest break is given after each block of 20 trials. The trial sequence was pseudo-randomized under the following rules:

• each set of 20 trials is distributed between the three conditions in a 7:7:6 distribution, with the blocks counter-balanced to ensure equal overall distribution;
• no condition appeared more than three trials in a row;
• each item was the target item (the item to be picked up) for each position (3 conditions x 4 items x 5 positions);
• each condition had 10 trials x 2 items on the table and 10 trials x 3 items on the table.

The automated manager dictates the study’s flow, informs the researcher of the order of item placement, and plays instructions to the participants. The item placement stage involved the researcher placing the items on the table based on the order provided on a screen (as seen in Fig 5. In the Start Trial stage, a speaker provided computer-synthesized oral instruction to inform the participant of the current condition of the trial. Depending on the condition, the participant would create a mental spatial map of the items on the table using the information provided. The participant then pressed a button indicating they were ready to reach the target item. The speaker then played an instruction to pick up a specific target item the participant would reach. No auditory stimuli were provided during the reaching stage; the participant must rely on their spatial memory alone to reach the object. The participants were instructed only to use their right hand and were allowed one reach attempt per trial. Minor adjustments to their hand trajectory were allowed. The reach was considered successful if the participant touched the item and unsuccessful if their arm was extended incorrectly.

2.3.2 Standing task.

The standing task is a functional test that explored the performance of the FAD when searching for an item with full mobility and within a cluttered environment. As shown in Fig 1B, the standing task involved three tables placed around the standing participant in a U shape. Twelve differently shaped and sized bottles are placed around the participant, with four bottles per table. The participant was asked to search and pick up a non-bottle item among multiple bottles, which served as distractors. This task consists of 3 trials which involved reaching for the book, bowl, or cup. In each trial, the researcher replaced one bottle with the target item according to the placement position on the screen. Then, the automated manager instructed the participant to “look for the” target item (e.g., “look for the bowl”), prompting the participant to search and reach it. Participants were given unlimited reach attempts but were allowed to give up if they could not find the item.

2.5.1 Task performance.

For the seated task, we assessed the participant’s ability to generate and memorize a spatial map (when exploring the items on the table) and the ability to use the spatial map to locate a specific item. We assessed the participant’s ability to find a specific item in a cluttered environment for the standing task. The seated and standing task performance is measured using an annotated trial run sheet of events during the trials. For every trial, the observing researcher noted the trial number, trial condition, the outcome (success or failure) of contacting the target item, the recognized items (spoken verbally during the FAD condition), and any other noteworthy events. After the session, the trial run sheet was manually digitized and processed for each participant. The participant data were separated into groups (BLV and blindfolded sighted).

The task performance results were calculated using the following equation:

The time required for the participant to build a mental spatial map was recorded through an event log with timestamps. The exact time was calculated by comparing the timestamp between the start of the trial and the timestamp when the participant pressed the user response button. It would not be meaningful to compare the conditions as the Clock and Speakers conditions required no active tasks while the FAD condition required active scanning, which naturally took longer. We mainly used this data point to explore the differences between BLV and blindfolded sighted participant groups.

2.5.2 Motion capture body tracking.

Twelve Optitrack Flex13 [50] cameras were used as the optical motion capture to track the item positions, participants’ right-hand movement, and their head/FAD rotation. As shown in Fig 6A, the motion capture system used reflective markers to translate physical items/body parts to rigid bodies, which were converted to trackable game objects. The motion capture data for the hand, head, and items were saved to a local JSON file through the Unity Game Engine. For the seated task, we measured the hand and head motion capture data to track the functional behaviors of the participants. We did not use the motion capture system for the standing condition because of limitations in the tracking area size.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6.

(A) depicts the hand, head, and item tracking through motion capture in this study. (B) Plots of the hand trajectory (z-x plane, overhead view) for every FAD trial by the participants. The blue dots represent the hand’s starting position with an offset always initialized at (0,0). The red dots represent reached item position. Position 1–5 refers to reaching positions from the left of the user to the right.

https://doi.org/10.1371/journal.pone.0290431.g006

We collected the hand kinematic data to evaluate the reaching behavior of the participants. The trajectories were sorted by the z-x plane (overhead view) with an offset applied so that all the trajectories started from (0,0). We observed the inconsistent ending points of the data (see Fig 6B). This inconsistency occurred due to the inconsistency in the starting position of the hand and the ending position of the item placement. We explored using techniques such as dynamic time warping [51] to normalize trajectories to the same length, scale, and start/end points. However, the resulting data became too distorted to attain any meaningful information. Instead, we calculated the hand-reaching time, reaching speed, and trajectory-to-optimal-path ratio. The Hand-reaching time is the time between the participant receiving the instruction and the participant’s hand touching the item. The Hand reaching speed was calculated by dividing the total trajectory length by the reaching time. The trajectory-to-optimal ratio refers to the percentage difference between the trajectory the participant takes and the optimal (direct) route from the start to the endpoint. The trajectory-to-optimal path ratio was calculated using the following equation:

The head kinematic data provides insight into the search patterns and the efficiency of the FAD for identifying items. We only used the FAD condition data for head rotation because the head movement was not required for the Clock and Speakers conditions during the spatial mapping stage. We sorted the head rotation into a single-axis orientation (y-axis) and calculated the values for both successful and unsuccessful (Missed) reaching trials. We calculated the total head rotation sorted by groups and the rotation speed (total head rotation/user response time) sorted by groups. This measurement helps understand the relationship between head rotation and the success of a task. One limitation of this measure was the imbalance between the successful and missed trials. Overall, most of the trials were successful (85.53±14.93%) in reaching the target item; fewer missed trials influenced the reliability of the date for the missed trials.

2.5.3 NASA-TLX.

We investigated the participant’s cognitive state to explore whether the FAD increased the participant’s mental workload levels. We used the NASA Task Load Index (NASA-TLX) questionnaire to gauge the participant’s mental workload. The NASA-TLX survey provided a reliable, self-reported metric to assess an individual’s subjective mental workload [52]. We assessed the seven aspects of mental, physical, temporal (feeling rushed/hurried), performance, effort, frustration, and the total score (RAW-TLX). The participants responded to the questionnaire in three stages after the Training session: the Seated and Standing tasks. We used the unweighted 7-point (scaled to 21 points) NASA-TLX questionnaire [52]. We averaged scores across groups and compared the mental, physical, temporal, performance, effort, frustration, and total (RAW-TLX) metrics over the three stages. This questionnaire indicates the participant’s level of perceived workload throughout the study.

2.5.4 Physiological measurements.

The participant’s physiological state plays a vital role in understanding the usability of the FAD device. The FAD device should not unintentionally increase the participant’s stress level or induce anxiety. We suspect factors such as uncertainty of the device performance and frustration/discomfort when using the device could increase the participants’ stress/anxiety levels. We used two physiological measurement devices in this study. Firstly, we used the Zephyr bioharness module [53] to measure the participant’s Heart Rate (HR), Heart Rate Variability (HRV), Breathing Rate (BR), and Skin Temperature (TEMP). The HR and HRV data were filtered through the HR confidence (remove when HR Confidence<20%) metric provided by Zephyr [54]. The BR and TEMP were filtered for outliers using two standard deviations from the median as the boundary condition. The second device was the Empatica E4 wristband [55], which measured the participant’s Electrodermal Activity (EDA) response. The EDA data were filtered for outliers using two standard deviations from the median as the boundary condition. The physiological measures were processed and averaged across the participant groups. We compared the physiological measures by condition to observe any specific condition that increased activity. We also compared the Seated and Standing tasks to observe for changes in physiological activity. These measurements provided helpful information regarding any increase in stress levels.

4.1.1 FAD: Head scanning, item sonification, and spatial mapping.

From the total head rotation results, it is clear that in the successful trials, both participant groups required significantly less head rotation compared to the unsuccessful trials. This result seems counter-intuitive; if a person scans the environment more, it would be reasonable to have more information. However, we believe the data suggests a crucial factor in the reliability of the FAD’s object recognition. If the unsuccessful trials were due to issues with the acoustic touch technique, then it would be expected to be observed a decrease in task success rate compared to the Clock and Speaker conditions, which involve no head scanning. It could be reasoned that the participants may rely on an ineffective head scanning method, such as scanning too fast. However, the head rotation speed did not yield any significant difference between successful and unsuccessful trials. This suggests that the participants were consistent with their head scanning speed. We believe the unsuccessful trials were due to errors in object recognition rather than the acoustic touch. We argue that the increased head rotation/sweeps may have introduced a higher risk of incorrect/misinformation from the object recognition, which led to some unsuccessful trials. An essential future consideration will be implementing more robust object recognition methods that can compensate for the head scanning motion and the quality of the head-mounted cameras.

We evaluated the item detection rate (Fig 7) of the seated task and the standing performance to understand better the reliability of using auditory icons (instead of automated text, i.e., Clock condition) in item identification. The standing task indicates the participants’ ability to detect specific items within a cluttered environment. All participants successfully identified the book, bowl, and cup within an environment filled with bottles. When observing the seated task success rate by items, we found that the sighted participants had a lower success rate for the bottle and bowl sounds when compared to the automated speech of the Clock condition. The bottle and bowl sounds (glass scraping/collision sounds) were confusing to the participants. One interesting observation is that participants could correct confused items in trials containing one glass item (i.e., book + bottle/cup/bowl) as they could deduce the item identity even though it was incorrectly recognized. Based on this assessment, we can conclude that auditory icons can convey the identity of an item. However, careful consideration should be given to selecting the auditory icon to ensure a unique item identity.

One drawback of the clock face instructions is the lack of precise spatial information. Each clock face zone represents roughly 36° (180° across 5 clock regions) of the area and does not provide any distance information. The FAD and Speakers can covey more precise directional information to the participant. From the observation and development, the FAD allows directional information through the variable repetition of the sound (higher repetition rate when closer to the center of FOV), and the distance can be gauged using vertical head scanning. In contrast, the Speakers condition requires apriori knowledge that is not translatable to real-world use. The task performance and reaching accuracy results indicate that the acoustic touch technique can convey accurate spatial information. In summary, the acoustic touch paradigm can successfully enable the recognition and localization of an item. However, future iterations of wearable devices should emphasize the reliability of machine object recognition and carefully design auditory icons for objects.

4.1.2 Cognitive workload and physiological stress.

When observing the NASA-TLX scores (Fig 1), we found that the sighted participant group experienced a higher cognitive workload than the BLV participant group. These findings are unsurprising, as the sighted participants are likely adjusting and adapting to the blindfolded. The BLV participants, on average, rated an increased perceived effort over the study stages. This result is likely due to the standing task stage’s more active nature than the training, which was a relatively relaxed task (participants were given unlimited time/trials to learn the protocol). Overall, the NASA-TLX scores suggest that the prolonged use of the FAD did not significantly increase the participants’ workload level.

When we compared the physiological measures (Table 1) between the seated task conditions, we did not observe a significant increase in physiological activity. The BR is the exception to this observation in that a significant increase was observed for the BLV participant group between the FAD and Speakers conditions. While an increase in BR could be interpreted as a mild hyperventilation [61] behavior indicating an increase in anxiety, it can more likely be attributed (due to lack of change within the other metrics) to increased bodily movement [62]. The significant increase in HR, BR, and TEMP between the seated and standing tasks for the blindfolded sighted participant group suggest an increase in acute stress levels [63]. The increase in HR can likely be attributed to the uncertainty of moving into an unfamiliar environment without their mobility aid. Additionally, the blindfolded sighted participants would be contending with the uncertainty of walking with a blindfold resulting in increased BR and TEMP. If given more time or repeated sessions, these users could become more familiar with the device and experience less stress during the standing task.