Corpus

To construct semantic trajectories of words, we use the downloadable version of the Google Books Ngram Viewer database version 2, downloaded from https://storage.googleapis.com/books/ngrams/books/datasetsv2.html, as time-labelled corpora available in multiple languages [8, 36]. In this work, we included five languages from the Indo-European family with the most available data, English, French, German, Spanish, and Italian. Notable limitations of the Google Ngram corpus include the lack of meta-data, and correspondingly, the equal importance assigned to a large variety of sources, including those by non-native speakers. As a special case, the English dataset suffers from significant overrepresentation of academic literature [37, 38]; in order to correct for this bias, we use the English fiction corpus as a representation of English [37].

Language processing

We used the snowball stemmer of the natural language toolkit (nltk) module of Python [39] to lemmatize words, and then we filtered out stopwords [40], before collecting all co-occurrences within a window size 2, as suggested by Levy et al. [15] and Hamilton et al. [20], illustrated by Fig 2a–2c.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Key steps of the pipeline that generates diachronic embeddings from google ngram data.

(a) Lemmatization of words. (b) Filtering stopwords. (c) Counting co-occurrences with context window of length 2. (d) Smoothening co-occurrences by applying a sliding time window to co-occurrence counts. (e) Vocabulary, i.e., the final set of lemmas we include in our analysis, is defined by the lemmas that occur at least 100 times at all time windows. (f) Training dataset was created by subsampling all co-occurrences in order to have a constant number of co-occurrences N_c = 10⁷ at each time window. (g) Word embedding by Word2vec with Skip-gram. (h) Alignment of different embeddings (both within and across years). (i) Randomization of the temporal order of step sizes of each word trajectory separately. (j) Randomization of the temporal order of step directions of each word trajectory separately.

https://doi.org/10.1371/journal.pone.0298650.g002

Temporal grouping

Since temporal annotation of ngram occurrences follow a yearly resolution, we first collected co-occurrences by year, and then we applied a 10-year sliding window to increase vocabulary size and to smoothen data. Although setting the sliding window size to 10 years is an arbitrary choice, we show in S1 File (namely in S5 Fig in S1 File) that as window size increases gradually from 1 year to 10 years, the exponent α approximately converges to its value at 10 years. We then filtered out words that do not appear at least 100 times in each time window.

While Google’s raw database initially provides ngram data from almost all years in the past few centuries, the number of occurrences decreases significantly as we look further back in time. This is a crucial factor in our methodology, since a certain amount of co-occurrence statistics is required for the embedding model to find a proper representation for the word meanings. Since the minimum number of words known by a native speaker is roughly 10 000 [41], we prescribed that we have a vocabulary with at least that size in each time step by only keeping data from years after 1950. If we went back further in time, we would have had a much smaller set of words to work with. This process is shown in Fig 2d and 2e.

Semantic embedding

For constructing word embeddings, we use Word2vec’s Skip-gram model with negative sampling (SGNS), which is one of the most widely used word embedding algorithms due to its computational efficiency and ability to capture semantic relationships in a simple mathematical form [16, 42]. It uses a two-layer neural network to represent each word i with two D-dimensional vectors called ‘word vector’ (v_i) and ‘context vector’ (w_i) such that a global cost function C, depending on all word vectors and all context vectors is (approximately) minimized. The objective of the Skip-gram model is to optimize the estimated empirical log-probabilities (2) of word j occurring in the context of word i, using the cost function C defined as (3) where L is the length of the text and is the linguistic context of word i. Following the recommendations of [15, 20], we set the dimension D of the embedding space to D = 300 and the context size to 2. Fig 1a visualizes a single embedding projected to 2 dimensions by t-sne [35], a non-linear dimension reduction method.

Measuring semantic distance

In this paper, we self-consistently define semantic distance |x − y| between vectors x and y in the D-dimensional embedding space as Euclidean distance, . This is a departure from the most commonly used semantic similarity measure, cosine similarity |x − y|_cos, which is favored because it accounts for the rescaling symmetry, x → λx, by setting |λ₁x − λ₂y|_cos = |x − y|_cos [13, 42]. We argue, however, that finding the most general class of underlying symmetries and measuring distance separately is a more principled procedure, especially when semantic distances have numerical and not just ordinal meaning. In particular, we choose Euclidean distance for the following reasons.

1. In the Skip-gram model, the difference of two word vectors, v_i − v_j, has semantic meaning. By interpreting the space of context words as a dual space (see S1 File for details), the meaning of v_i − v_j can be understood by projecting it to various context words, akin to the definition of linear functionals in dual space, (4) Since this result is valid for any context word k, v_i − v_j measures the relative log-probability of word i versus word j occurring in the context of word k, for all contexts k. This is in line with the starting hypothesis of distributional semantics, which states that two words have a similar meaning if their co-occurrence statistics are similar.
2. Euclidean distance is a metric, obeying triangle inequality, commutativity, and other axioms, whereas other measures, such as cosine similarity are not. This is important when semantic trajectories are interpreted as trajectories in metric spaces, for example, by measuring their squared displacement over time, |Δx|²(t) = |x(t) − x(t = 0)|².
3. As discussed above, embedding dimensions are arbitrary. They need to be aligned for meaningful comparison across multiple embeddings (regardless of whether it coming from the same co-occurrence data or a different one). Here we follow the most commonly used method, called the Procrustes algorithm [43], that finds an orthogonal transformation that minimizes the total Euclidean distance between corresponding points in the two embeddings. Measuring semantic distance as Euclidean distance is consistent with this alignment method.
4. As a consequence of keeping the size of the word cloud constant, defined by Eq (11), two embeddings belong to the same equivalence class if they can be orthogonally transformed to each other, discussed above and shown below in Methods. Orthogonal transformations are defined by keeping Euclidean distance, also consistent with our definition.

Diachronic embedding

We first subsample the co-occurrences corresponding to each time window to eliminate systematic sample size bias (the amount of data included in the Google Ngram database steadily increases with time in all five languages) as shown by Fig 2f. We set this sample size to N_c = 10⁷ co-occurrences, set by the first (few) time windows with the least amount of data. We observe that N_c = 10⁷ sets a good tradeoff between sample size (and thus trajectory noise), vocabulary size, and trajectory length for this database. As shown in S1 File (namely in S1 Fig in S1 File), using larger samples will not give us embeddings with more information. This is because increasing the number of input co-occurrences only makes the word cloud larger without altering its structure.

After generating the D = 300 dimensional embeddings for each time window M = 80 times by Word2vec with Skip-gram, we first align all M embeddings within a time window. Then, word positions across embeddings are averaged to obtain a more robust estimate of semantic positions for each word at a given time window. These average embeddings, one for each time window, are then aligned across time, as shown by Fig 2g and 2h.

The alignment of two embeddings consists of two steps. First, one has to define the class of transformations that keep semantic relationships invariant, and second, the transformation within this class that minimizes the difference between the two embeddings has to be selected. For the first step, we find (see below for a proof) that the requirement of keeping the total size of the word cloud constant, as defined by Eq (11), restricts all linear transformations to those that are orthogonal. This is also in line with both measuring semantic distance as Euclidean distance, as discussed above, and also with the most commonly used alignment method called the orthogonal Procrustes method, which finds that the best transformation R that accounts for this rotational (orthogonal) freedom is given by (5) where the rows of the N × D matrix W′ and W contain the word vectors of the aligned and reference embedding, respectively, and ||⋅||_F is the Frobenius matrix norm. We use the analytical solution [43] for R to find optimal alignments both within a time window and among time windows.

Constant word cloud size defines an orthogonal embedding symmetry

Word cloud size, defined by Eq (11), is written as (6) where V(W) is the empirical covariance matrix of word positions, extracted from W, the N × D matrix with its rows corresponding to the D-dimensional word vectors of all N words, and Var_i (W_ij) is the variance of the word positions along dimension j. Using the definition of variance, (7) The size of the transformed word cloud W′, defined as W′ = RW is (8) The difference between Tr V(W) and Tr V(W′) is (9) The transformation R satisfies the requirement of constant cloud size Tr V(W) − Tr V(W′) = 0 only if (10) also written in matrix form as RR^T = 1, i.e., R needs to be orthogonal.

Mean versus ensemble average anomalous diffusion exponent

As explained in the Introduction, distributional semantics does not directly define the meanings of words; it treats them as statistical properties based on their contexts. Studying word meanings in this framework only makes sense in relation to other words, which are also defined relationally. This means that one cannot extract all the semantic information of an individual word from its embedded trajectory alone, but other word trajectories also have to be considered, and their collective behaviour should be studied. When dealing with measurable quantities, such as anomalous diffusion exponents, this requires analyzing their distribution and averages.

Averages of anomalous diffusion exponents corresponding to single-word trajectories are calculated in two different ways. The mean anomalous diffusion exponent is computed by first fitting an anomalous diffusion exponent α_i to the trajectory of each word i, and then averaging the fitted exponents, , where N is the total number of words. To compute the ensemble average anomalous diffusion exponent, we first average |Δx_i(t)|², the squared displacement of word i over time to obtain the mean squared displacement over time, , and then we fit an anomalous diffusion exponent to |Δx(t)|², called ensemble average anomalous diffusion exponent 〈α〉.

Trajectory randomization methods

Fig 2i and 2j illustrate the two types of trajectory randomization methods we use in this paper to generate the results shown by Fig 3c and 3d and Table 1: randomization of step sizes and randomization of step directions. Within each type, “randomization” refers to drawing step sizes and directions from various distributions constructed from the original trajectories, described as follows.

1. Random sizes: step sizes were sampled from a normal distribution with mean and standard deviation corresponding to that of the step sizes of the original trajectory; sizes from distribution: step sizes were sampled from the set of all step sizes corresponding to all words; shuffled sizes: step sizes were sampled from the set of step sizes corresponding to the same word; in other words, the temporal order of step sizes of a trajectory has been shuffled; original sizes: step size for every word at every time step was set to its original value.
2. Random directions: directions were sampled uniformly over a D-dimensional sphere; shuffled directions: the temporal order of step directions of a trajectory has been shuffled; original directions: the direction for every word at every time step was set to its original value.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Anomalous diffusion analysis of semantic trajectories.

(a) Squared displacement (Δx)² of selected English words over time, along with average squared displacement over all words, shown in blue, which can be approximated by 〈(Δx)²〉 ∼ t^〈α〉 with ensemble-average anomalous diffusion exponent 〈α〉≈0.44. (b) Average squared displacement of words over time in five languages, English, French, German, Italian, and Spanish. This is compared to an ensemble of random walkers under the same constraint given by a constant total word cloud size, defined by Eq (11). (c) Average squared displacement 〈(Δx)²〉 of English words under various combinations of trajectory randomization methods. Columns and rows differ in the way step sizes and directions are randomized, respectively, see details in text. Keeping the original sequence of step directions is necessary to recover strongly subdiffusive behavior (bottom row), but not sufficient: the more information regarding step sizes is kept, the more its best-fit diffusion exponent approaches its original value of 〈α〉 ≈ 0.44. (d) Distribution of diffusion exponents α fitted to the trajectory of each word separately. Their average is close to the ensemble averages 〈α〉 shown in c).

https://doi.org/10.1371/journal.pone.0298650.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Ensemble-average anomalous diffusion exponent 〈α〉 and the average of the anomalous diffusion exponents of all words, under various combinations of trajectory randomization methods, for all five languages.

Grey cells show the exponents of the original, non-randomized trajectories. Note that the exponents for English are extracted from Fig 3c and 3d; for the other four languages, the analogous Figures are included in S3 and S4 Figs in S1 File.

https://doi.org/10.1371/journal.pone.0298650.t001

Maximally representation-agnostic temporal embedding

While constructing semantic trajectories of words from time-labelled co-occurrence data, one needs to account for two sources of arbitrariness in the process: stochasticity and symmetry. The first, stochasticity, comes from the nature of modern data-efficient semantic embedding methods, such as Word2vec. Since these approaches use a neural network to find the best possible embedding, sampling techniques, initial weights, and the choice of numerical optimization method (such as stochastic gradient descent) lead to non-deterministic embeddings. Second, the cost C itself exhibits multiple global minima associated with the same input data. In particular, the form of the estimated log-probabilities defined in (2) implies that any transformation that leaves all dot products v_i ⋅ w_j invariant results in the exact same cost C. The simplest example of such transformation is a linear rescaling of word vectors v ↦ λv and inverse rescaling of context word vectors w ↦ λ⁻¹w, but in principle, any invertible linear transformation of word vectors v ↦ Rv, together with (the transpose of) its inverse applied to context vectors w ↦ (R⁻¹)^Tw leaves the dot product invariant (see Methods for details).

However, an additional constraint comes from focusing solely on words (and not contexts) when constructing temporal trajectories: the constraint that ensures that the ambient embedding space does not shrink or expand over time, formalized as (11) where V is the D × D empirical covariance matrix of word positions (with D = 300 being the embedding dimension) and σ_i is the standard deviation of word positions along principal dimension i. As shown in Methods, this reduces the possible transformations R to the orthogonal ones, obeying R⁻¹ = R^T. Consequently, a single embedding is identified with an equivalence class containing {Rv_i, Rw_j} with any orthogonal R.

We use this orthogonal freedom to define maximally representation-agnostic trajectories in two steps as shown in Fig 1b. First, we minimize the effect of stochasticity, described below, in any single time. Without stochasticity, any embedding starting from the same co-occurrence data would belong to the same equivalence class, and therefore it would be possible to find a transformation R_j for each embedding j such that they all numerically coincide. With stochasticity, however, different embedding realizations, starting from the same data, land in (slightly) different equivalence classes, and as a consequence, perfect alignment among them is not possible. The best one can do is to find a transformation for each embedding such that an overall distance measure between all embeddings is minimized (see Methods for details). This allows us to “average out”, i.e., minimize the effect of stochasticity at any single timestep by taking the average of all aligned embeddings. Second, we align averaged embeddings corresponding to different times in the same way: we find an orthogonal transformation that minimizes the distance between the embedding at subsequent times to construct word trajectories. Such word trajectories are thus maximally smoothened. Although this raises the question of whether maximal smoothening washes away real phenomena, we will see that this smoothening procedure applied to random walk does not change measured observables such as the anomalous diffusion exponent α of the process. Fig 1c shows the measured semantic change of three selected words within a decade, projected to 2 dimensions, visualized over a static background.

Semantic subdiffusion across languages

As detailed in Methods, to measure the actual value of the anomalous diffusion exponent α, we need to take averages over the whole vocabulary. We proceed from single trajectories to an ensemble of trajectories in two alternative ways: (i) we first average |Δx|²(t) over individual trajectories to obtain 〈|Δx|²(t)〉 and then we fit the ensemble-average anomalous diffusion exponent, 〈α〉, based on 〈|Δx|²(t)〉 ∼ t^〈α〉; (ii) alternatively, we first fit the anomalous diffusion exponent α to single trajectories and then we average over words to obtain the mean anomalous diffusion exponent . Although individual trajectories deviate considerably from the simple scaling behavior given by |Δx|² ∼ t^α, somewhat surprisingly, their ensemble average, 〈|Δx|²(t)〉, follows the scaling given by t^〈α〉 with high accuracy. This is illustrated in Fig 3a, showing the squared displacement |Δx|²(t) of several individual English words as well as the ensemble average 〈|Δx|²(t)〉 of all English words.

The grey cells of Table 1 list obtained exponents 〈α〉 and for five languages, English, French, German, Italian, and Spanish. In all languages, both the ensemble-average anomalous diffusion exponent 〈α〉 and the mean anomalous diffusion exponent are significantly lower than α = 1, i.e., semantic trajectories show subdiffusive behaviour on this time scale. In particular, we measured an ensemble-average anomalous diffusion exponent 0.4 < 〈α〉 < 0.5 for all five languages. This is in strong contrast with a random walk generated using the same parameters (see Methods for details), corresponding to a fitted α ≈ 1, as shown by Fig 3b.

Comparison with randomized trajectories

Given the robust observation that the ensemble-average semantic behaviour of words follows subdiffusion with 〈α〉 ≈ 0.4 − 0.5 across languages, one might ask the following two questions. (i) Is this an artifact of the diachronic alignment procedure? (ii) If not, what is behind the observed subdiffusion? In other words, what (combination of) microscopic models of stochastic collective dynamics might explain this macroscopic result? In the following, we focus on these two questions. We apply a series of randomization methods to the original trajectories, gradually removing temporal correlations in step sizes and step directions of individual trajectories to see which of these, if any, play a role behind subdiffusion. As shown in Fig 3c and Table 1, step sizes of a trajectory are randomized three different ways, which we call random sizes, sizes from distribution, and shuffled sizes; step directions are randomized two ways, random directions, shuffled directions (see Methods), which, together with the original trajectories, gives three times four combinations.

Fig 3c and 3d show the average squared displacement 〈(Δx)²〉 of all English words, and the distribution of anomalous diffusion exponents α fitted to individual word trajectories separately, under all twelve combinations of trajectory randomization methods (including the original trajectories). The same plots for French, German, Italian, and Spanish are shown in S3 and S4 Figs in S1 File; fitted anomalous diffusion exponents 〈α〉 and are listed for all five languages in Table 1. The top left panel (random step sizes, random step directions) corresponds to a random walk; the bottom right panel (original step sizes, original step directions) corresponds to the original, non-randomized semantic trajectories.

Comparing the results of original trajectories with the randomized trajectories (Fig 3c) conveys three important messages: (i) temporal alignment, applied to an ensemble of uncorrelated trajectories (top left panel), results 〈α〉 = 1 (see top left panel), equivalent to that of uncorrelated trajectories without alignment. (ii) Both correlations in step sizes and step directions are important factors behind subdiffusion. Neither of them alone can produce trajectories with an α lower than 0.8, yet the two combined give α ≈ 0.5. (iii) If step sizes do not vary significantly, the shuffling of the directions can alter the total displacement along a trajectory only by a small amount, thus, the last data points in the middle row (shuffled directions) must be very close to the last data points of the corresponding panels of the last row (original directions). On the other hand, shuffled directions do remove temporal correlations in step directions, making trajectories follow approximately diffusion-like behavior until the constraint on total displacement does not affect them. This can be clearly seen in the middle row of Fig 3c; in Fig 3d, we decided not to fit exponents to individual word trajectories in the middle row (shuffled directions) to avoid systematic bias in the exponents depending on the fitting range.

We further investigate a related effect, the effect of keeping the total size of the word cloud constant, as formalized by Eq (11). Ensembles of randomized trajectories in Fig 3c and 3d do not obey this constraint; we, therefore, generated a random walk, corresponding to random step directions and random step sizes, with the additional constraint on keeping the total cloud size constant. This simulated trajectory is illustrated in Fig 3b (“random walk”). While the total displacement is limited by the size of the word cloud, this only appears to affect the random walk trajectory when the displacement reaches the radius of the cloud (and then converges to approximately to the square root of two times the radius, corresponding to two orthogonal vectors with length equals to the radius). This, along with the middle row of Fig 3c, suggests that global constraints on total displacement do not cause the observed subdiffusive behavior.

Finally, we compare the original semantic trajectories to another randomized model, where time labels of the original data are randomly reshuffled (see details in Section S6 in S1 File). As shown by S5 Fig in S1 File, such randomized “trajectories” display a non-zero anomalous diffusion exponent, α ≈ 0.2, which can be attributed to the alignment of embeddings corresponding to subsequent times. This is an unavoidable effect whenever embedding dimensions are arbitrary and thus alignment is necessary; this baseline non-zero anomalous diffusion exponent is present even when time labels of diffusive trajectories (α = 1) are reshuffled before the same pipeline, including alignment, is applied (S8 Fig in S1 File). Trajectories built from reshuffled time labels, however, display significantly larger fluctuations (S6 Fig in S1 File), and correspondingly, the goodness of fit of the anomalous diffusion exponent is significantly lower (S7 Fig in S1 File).