Fig 1.
Representational similarity analysis.
(a) A set of experimental conditions or stimuli are presented to participants. In this example, recordings of English words are presented aurally. (b) For each experimental condition, EMEG data is collected from participants’ regions of interest for a specified epoch. (c) Dissimilarities between each pair of responses are computed and stored in a representational dissimilarity matrix. Potential dissimilarity measures include Pearson’s correlation distance or Euclidean distance between response vectors. Rows and columns of the matrix are indexed by the condition labels, making the matrix symmetric with diagonal entries all 0 by definition. In this example there are four conditions in total, and the responses to the condition pair (bulb, tribe) is compared, with the value stored in the indicated matrix entry, and its diagonally-symmetric counterpart. (d) A model of the experimental conditions or stimuli is used to compute a model RDM. The model RDM can be computed in several ways, e.g. by comparing representations of the stimuli under the model; or by modelling the dissimilarities directly. (e) Data and model RDMs are statistically compared, e.g. by computing Spearman’s rank correlation of their upper-triangular vectors.
Fig 2.
Mapping from GMM–HMM triphone log likelihoods to phone model RDMs.
(a) Each 10 ms frame of audio is transformed into MFCC vectors. From these, a GMM estimates triphone log likelihoods, which are used in the phonetic HMMs. (b) We used the log likelihood estimates for each triphone variation of each phone, concatenated over a 60 ms sliding window, to model dissimilarities between input words. Dissimilarities modelled by correlation distances between triphone likelihood vectors were collected as entries in phonetic model RDMs. (c) These phone-specific model RDMs were computed through time for each sliding window position, yielding 40 time-varying model RDMs.
Fig 3.
The mapping between articulatory features and phonetic labels.
Columns describe phones and rows describe features, with a filled-in cell indicating that a phone exhibits a given feature. The features are grouped into descriptive categories, from top to bottom: Broad phonetic categories, place-of-articulation features for consonants, manner-of-articulation features for consonants, vowel frontness, vowel closeness, and vowel roundedness.
Fig 4.
Second-order similarity structure of phone models.
Entries in the matrix are Spearman’s rank correlations between model RDMs. The second-order similarity structure of the phone models for a representative time window centred over 90 ms after word onset, given by a correlation matrix between phone model RDMs.
Fig 5.
Similarities between model RDMs and phonetic features.
(a) Davies–Bouldin indices for each feature. Error bars indicate one standard deviation of values over the epoch. (b) η2 values for each feature assignment. Rule-of-thumb guides for small, medium and large effect sizes are indicated by horizontal lines. Error bars indicate one standard deviation of values over the epoch. (c) The arrangement of phone models plotted using MDS (distance–dissimilarity correlation 0.94). Points are labelled according to the presence or absence of the sonorant feature. Models with the sonorant feature are represented with red circles, and models without the sonorant feature are represented with green triangles. The MDS arrangement of points as displayed was independent of the feature labelling.
Fig 6.
(a) Each stimulus’s evoked EMEG response is captured within a fixed time window and regular searchlight patch, which moves continuously in space inside the searchlight mask. (b) The dissimilarity between a pair of conditions is taken to be the correlation distance (1 − Pearson’s correlation) between the vectors of activation within the spatiotemporal searchlight. (c) The modelled dissimilarities between pairs of conditions are collated into a model RDM (see Fig 2). (d) Model RDMs are compared to the data RDM, with the resulting statistic mapped back into the central vertex of the searchlight. This is repeated for each spatiotemporal position of the searchlight.
Fig 7.
Relating brain data dRDMs to phone model dRDMs and converting to feature fits.
(a) At each vertex and time point, all phone model RDMs are computed (Fig 6) and fitted against the data RDM in a GLM, yielding coefficients βϕ. (b) The rows of the phone-feature matrix of Fig 3 describe for each feature f the phones ϕ exhibiting f, providing a labelling function χf. The example given here is for the feature sonorant, the top row of the feature matrix in Fig 3. (c) The coefficients βϕ were aggregated by sum over each feature f to produce a map of fit for each feature, which was mapped back to the central location of the spatiotemporal searchlight.
Fig 8.
(a) Thresholded maps of fit for each feature for which at least one vertex showed significant fit between 100 ms and 370 ms. We report p < 0.05 using *, p < 0.01 using **, and p < 0.001 using ***. Light and dark greys represent gyri and sulci respectively. The miniature figures show the location of the larger diagrams. Anatomical landmarks are superior temporal gyrus (STG), superior temporal sulcus (STS) and Heschl’s gyrus (HG). The dotted white lines indicate the outline of HG. (b) With a fixed threshold of p < 0.01 for both hemispheres, only broad-category features remain on the right, and within-category distinctive features are dominant on the left.