Table 1.
Characteristics of the books analyzed.
The length of each book L is measured in millions of tokens.
Fig 1.
(a) Probability mass functions f(n) of the absolute frequencies n of words and lemmas in La Regenta, together with their fits. (b) The same, from top to bottom, for Clarissa, Moby-Dick, Ulysses (all three in English), and Don Quijote (in Spanish). The distributions are multiplied by factors 1, 10−2, 10−4 and 10−6 for a clearer visualization.
Table 2.
Power-law fitting results for words and lemmas, denoted respectively by subindices w and l.
V is the number of types (vocabulary size), nm is the maximum frequency of the distribution, Na is the number of types in the power-law tail, i.e., with n ≥ a, a is the minimum value for which the power-law fit holds, and γ and σ are the power-law exponent and its standard deviation, respectively. 2σd, the double of the standard deviation σd is also given. σd is the standard deviation of γl−γw assuming independence, which is . The last column provides ℓ1, the number of lemmas associated to only one word form. Notice that the lemma exponent is very close to the one found in Ref. [29] for the tail of a double power-law fitting, except for Moby-Dick and Ulysses.
Fig 2.
γl (the exponent of the frequency distribution of lemmas) versus γw (the exponent of the frequency distribution of word forms).
As a guide to the eye, the line γl = γw is also shown (solid line). Error bars indicate one standard deviation.
Table 3.
Analysis of the association between random variables using Pearson and Spearman correlations as statistics.
ρ is the value of the correlation statistic and p is the p-value of a two-sided test with null hypothesis ρ = 0, calculated through permutations of one of the variables (the results can be different if p is calculated from a t–test). The sample size is 𝓝 = 10 in all cases. Only the Spearman correlation between aw and al/aw is significantly different from zero.
Table 4.
The fit of a linear model for the relationship between exponents (γw and γl) and the relationship between cut-offs (aw and al).
c1 and c3 stand for slopes and c2 and c4 stand for intercepts. The error bars correspond to one standard deviation. A Student’s t-test is applied to investigate if the slopes are significantly different from one and if the intercepts are significantly different from zero. The resulting p-values indicate that in all cases the slopes are compatible with being equal to one. The intercepts are compatible with zero for the exponents, but seem to be incompatible for the cut-offs.
Fig 3.
(a) Probability mass functions f(n) of the absolute frequencies n of words and lemmas in La Regenta, together with their fits, under rescaling of both axis. The collapse of the tails indicates the compatibility of both power-law exponents. (b) The same for, from top to bottom, Artamène, Bragelonne (both in French), Seitsemän v., Kevät ja t., and Vanhempieni r. (all three in Finnish). The rescaled distributions are multiplied in addition by factors 1, 10−2, etc., for a clearer visualization.
Fig 4.
The lower cut-off for the frequency distribution of lemmas (al) versus the lower cut-off for the frequency distribution of word forms (aw).
The line al = aw is also shown (solid line).
Fig 5.
(a) Number of words per lemma as a function of lemma absolute frequency nl in Vanhempieni romaani (in Finnish) and in La Regenta. The figures for the former have been slightly shifted up for clarity sake. (b) Frequency of words nw as a function of the frequency of their lemmas nl in La Regenta.
Fig 6.
Probability density D(nl/nw) of the frequency ratio for lemmas and words, nl/nw, in La Regenta.
Values of nl smaller than nw are disregarded, as they arise from words associated to more than one lemma. Bending for the largest nl/nw is expected as the maximum of the ratio is given by nl, which is not constant for each distribution but has a variation of half an order of magnitude (see plot legend).
Table 5.
Coverage of the vocabulary by the dictionary in each language, both at the word-type and at the token level.
The average for all texts is also included. Remember that we distinguish between a word type (corresponding to its orthographic form) and its tokens (actual occurrences in text).
Table 6.
Size of vocabulary V (i.e., number of types) when texts are decomposed in different sorts of types, being these: word-lemma-tag (w-l-t), plain words, lemma-POS (l-pos), lemma-POS of words in the dictionary (l-pos dic), lemmas, and lemmas of words in the dictionary (lemma dic).
The latter provide the most radical transformation, as it yields the largest reduction in resulting vocabulary.