Natural sounds can be reconstructed from human neuroimaging data using deep neural network representation
Fig 2
(A) Training dataset. Waveform examples and category labels are displayed, with each category—human speech, animal sounds, musical instruments, and environmental sounds—represented in distinct colors. (B) Test dataset. Examples selected from each of the four categories are shown, with corresponding waveforms and labels. (C) Examples of spectrograms from the test dataset. One example from each category is displayed. (D) Data samples for machine learning analyses. Each 8-s sound stimulus is divided into three overlapping 4-s windows, and corresponding fMRI responses (3 volumes) are averaged within each window to create data samples. Single-trial fMRI volumes are used for training data, while test data utilize either single-trial volumes or volumes averaged across eight repetitions. (E) Definition of auditory cortex (AC). The AC, outlined by blue lines, is delineated as a combination of two regions: the early auditory cortex, shown in brown, and the auditory association cortex, shown in orange.