Limitations of the Study, Open Questions, and Future Work

The reported results can be argued to provide quantitative support for a generative origin of timing deviations, i.e., that these respond to the structure of the musical piece and its psycho-perceptual consequences for interpretation. As mentioned in the Introduction, there is evidence that musical aspects such as tempo, phrasing, metrical accents, musical form, and harmonic structure can determine timing deviations. In our experiments, mixing different compositions and their interpretations, we found scarce evidence for the dependence of onset deviations on individual score elements (specifically, of note intervals and relative durations). However, this does not preclude other score information like the aforementioned musical aspects having a direct influence on onset deviations. The fact that onset deviations perform similarly or slightly better than relative note durations, combined with the fact that the former were independent and uncorrelated to the latter, also suggests that onset deviations encapsulate information that goes beyond duration/temporal aspects of the score. Additional medium- or large-scale quantitative studies with real-world commercial recordings of classical guitar could provide more insight into this question. An alternative plausible hypothesis for the obtained results would be that onset deviations were so specific to the musical composition, that deterministic rules inferred from a pool of compositions could not be generalized to cover all the variability in the pool. As existing research suggests, this hypothesis cannot be completely ruled out (cf. [11]). Thus, one should also be open to the possibility that timing deviations encapsulated some contextual aspects specific to the composition but not related to the score (e.g., composer-specific performance rules, historical performance considerations, etc.). Regarding this latter hypothesis, it could perhaps be interesting to replicate the experiments carried out here considering cover songs or jazz versions, as these retain the essence of the original composition while usually introducing important changes in timbre, harmony, tempo, or rhythm [41] (although semi-automatic onset extraction could be more involved).

As our study was not designed to do so, the reported results only provide weak evidence regarding the additional hypotheses on the origins of timing deviations mentioned in the Introduction. It is true that the considered music collection contains a number of recordings of different pieces by the same performer. Hence, if performer-specific deviations dominated the raw onset sequences, one would expect much worse piece identification accuracies, as recordings would tend to cluster around performers and not around pieces. However, the lack of a sufficient number of performers having more than one recording in the considered collection seriously challenges clustering across performers and does not allow any strong claim regarding this hypothesis to be made (it is worth mentioning nonetheless that, as some works indicate [19], [24], [25], performer-specific aspects may be better studied after subtracting a global, average performance template like the one we consider here). A similar argument holds for emotion-based hypotheses. Assessing the biological or instrument-related motion hypothesis would require a different collection containing recordings of the same piece played with different instruments (perhaps the cover songs or jazz versions mentioned above). In the present study, we wanted to focus on the classical guitar, as this is an almost unexplored area. Finally, our results suggest that randomness is a minor component of the considered onset deviations, reinforcing their largely voluntary nature. Indeed, if noise were very present in the considered onset deviations, we would not be able to achieve the reported classification accuracies. A precise quantification or estimation of the amount of noise in timing deviations is, nonetheless, out of the scope of the present study (it may moreover have a lot of dependencies: instrument, genre, performance difficulty, etc.).

Regarding the reported preliminary assessments, we are aware that many further improvements can be done, specially for the case of long-range correlations. A complete characterization of note onset deviations, their relation to all possible score elements, their distribution, and their long-range correlations is beyond the scope of the present study. Nonetheless, we believe our assessments are some of the necessary first steps towards these goals and could motivate future research and discussion (for instance, the fact that onset deviations do not conform to a standard Gaussian distribution could lead researchers in machine-based music rendering to explore other distributions that could result in a more plausible listening experience). Hence, we opt to include, link to some literature and briefly explain such assessments.

Conclusions

In summary, the obtained results show (a) that onset deviation sequences are a powerful predictor of the musical piece being played, (b) that they are at least as powerful as direct music score information corresponding to relative note durations, if not better, (c) that such predictive power is robust to classification scheme choices, to the size of the considered data set, and to the length of the considered sequences, (d) that even very short sequences provide statistically significant accuracies, and (e) that temporal dependencies between onset deviations are key to obtaining such accuracies. Moreover, we quantify how the length of onset deviation sequences and, to a lesser extent, the size of the data set, impact classification accuracy. Some additional preliminary experiments are reported. In particular, our results show non-significant correlations between onset deviations and relative note durations or pitch intervals, and indicate that onset deviations do not obey a Gaussian distribution. Finally, we discuss existing open issues and some of the limitations of our study, while linking our findings to the existing literature.

As a main objective, this article wants to provide a new and fresh view on the topic of music timing variations. We believe that by taking quantitative medium-scale approaches, considering real-world commercial recordings, and different instruments apart from piano is a necessary step towards a better understanding of it. Here, the focus is on the utility of onset deviation sequences as musical piece signatures, and on the predictive power of those sequences. Hence, our main contribution relies on showing this predictive power and studying it under different temporal windows, even very short ones. With this in mind, the results found are encouraging and open new research perspectives. We hope future work will bring more evidence on the connection between musical pieces and the onset deviations extracted from their performances.

Music Collection

In our music collection we have 10 different compositions, and each composition is performed by 10 different guitarists, thus yielding a total of 100 recordings. However, some performances of different compositions have been interpreted by the same musician. In total, we have 82 different guitarists, with some of them playing between 2 and 5 pieces. The collection includes well-known guitarists such as Andrés Segovia, John Williams, Manuel Barrueco, Rey de la Torre, Robert Westaway, and Stanley Myers. In order to encompass different epochs, we chose compositions spanning four different periods: baroque, classical, romantic, and modern (Table S1). Recording years go from 1948 to 2011, and the number of onsets per score measure varies between 1 and 16. A table relating compositions, recordings, and performers is provided (Table S2).

Semi-automatic Onset Detection

In music signal processing, different techniques of varying complexity for automatic onset detection exist [36], [37]. These usually work on the time domain, the frequency domain, or both [37], [38]. Due to the difficulty of the task, it is not expected that a single method or parameter combination will work for all possible specific cases [37]. Therefore, we needed to choose the correct onset detection algorithm according to our needs, and tuned its parameters appropriately for the data at hand. In our case, we considered the 7 available algorithms in the Aubio library (http://aubio.org) [38]. To choose one of the algorithms and its best-fitting parameters we implemented particle swarm optimization [42] and ran it over an independent, out-of-sample set of 12 classical guitar audio files with manually annotated onsets [35]. The best performing combination was found to be the Kullback-Leibler algorithm [43] with a window length of samples, a hop size of samples, a peak-picking energy threshold of , and a silence threshold of dB (the sample rate was 44.1 KHz; for further explanations we refer to the Aubio documentation (http://aubio.org/doc/onsetdetection_8h.html) and [38]).

After detecting the onsets in our collection using the algorithm and parameters above, we implement an additional onset validation step. For that, we first manually annotated the score measure positions of all recordings in our collection. This way, we could unambiguously synchronize each measure in the audio file with the corresponding measure positions in the written score. The reference onset positions were then assigned by distributing the onsets between each measure according to strict score notation (Fig. S8). Additionally, we checked whether there were missing onsets. If a score onset did not match an audio onset , we imputed the temporal location corresponding to 7 milliseconds before the highest audio signal magnitude (absolute values) closest to and within a short-time window (Fig. S8). We used a window centered at whose length corresponded to the 90-th percentile value of the composition's note durations (the 7-millisecond offset was manually determined by visual inspection of a small subset of the real data).

To check the accuracy of the obtained , we manually validated 223 onsets, randomly sampled from the whole data set. Specifically, we annotated the temporal difference between what we considered to be the true onset location and the one determined by our approach (Fig. S7A). The vast majority of the inspected onsets were at their correct locations. Using a threshold evaluation strategy to determine the percentage of correct onset placements [38], we estimated that only a 6.7% of them were not placed on the exact location they should be. This number drops to 2% if we consider a threshold of 150 milliseconds (Fig. S7B).

After extracting onset positions , onset deviations are computed as in Eq. 1, obtaining a sequence for a composition with note onsets. Notice that, as mentioned in the Introduction, this is an event-shift representation of timing. Notice furthermore that, due to the manual synchronization of each score measure with the audio signal, the first onset of each measure would result in , thus losing several meaningful onset deviations. To alleviate this problem we consider a 4-measure window synchronization, and average the onset deviations obtained when moving this window in steps of one measure (i.e., the final in the -th measure is obtained by averaging the four obtained by synchronizing the beginning of measures and , and , and , and and ). Thus, with the exception of the onsets at the beginning and end of the piece, would be obtained as the average over four deviation values (we however made a further refinement and avoid the extremes, i.e., the maximum and minimum values, and compute the average between the two central ones). The raw onset deviations for the considered recordings can be found online (http://www.iiia.csic.es/~tan/downloads/2013_OnsetDeviations_Data.tar).

Preliminary Side Checks

All compositions provided similar numbers for the statistics of raw onset deviation values (Table S3). Additionally, we manually confirmed that maximal anticipations/delays generally corresponded to full cadences, usually ritardandos found in piece endings or strong structural locations (cf. [19]–[21], [25]). For instance, in the middle of the twenty-first measure of C02 (J.S. Bach, BWV 1007), for all performances, we observed a long pause between 0.5 and 1 seconds, which does not correspond to any existing annotation in the written score. Also, in C03 (A. Barrios, La Catedral–Prelude), the notes corresponding to the melody in the arpeggios are significantly delayed in most of the performances.

As a separate preliminary check, we ran some tests in order to assess the nature of the distribution of the samples in . First, we checked whether such distribution could be assumed to be a standard Gaussian distribution(2)where here stands for a single onset deviation value and and correspond to the mean and standard deviation of all values in , respectively. For each recording we ran an Anderson-Darling test [33] for the null hypothesis that was drawn from a Gaussian distribution. The Anderson-Darling test is known to be one of the most powerful statistical tools for detecting most departures from normality [44]. Under such test, the null hypothesis was rejected for 93 of the 100 recordings at a significance level of and for 88 of the 100 recordings at . Visual inspection of the data also gives us a qualitative confirmation of this result (Fig. S3A).

Next, as a further separate preliminary check, we wanted to assess whether we could find some indication of long-range correlations in . For that, inspired by [9], [39], we performed a small qualitative investigation of whether a power law could explain the power spectral densities obtained from . In particular, we considered whether(3)where is the power law exponent. A visual inspection of individual linear fits to the power spectral densities obtained from different recordings suggests the possibility of a power law (Fig. S3B). Noticeably, the fits yielded exponents in the ranges provided by [9]. Since, as mentioned, we only wanted to have an impression of the behavior of , we did not pursue more robust power law fitting strategies such as the ones followed in our previous work [45], [46] or elsewhere [47], nor did we consider more advanced techniques for determining the existence of long-range correlations such as the ones employed, e.g., in [9], [39].

Feature Extraction

The features we use as input for classification are normalized onset deviation subsequences or n-grams. First, the entire sequences for each recording are normalized to have zero mean and unit variance, , where again and correspond to the mean and standard deviation of all values in . Next, for each composition , the one to which the -th recording belongs, an integer note index is uniformly chosen, . This, together with the predefined sequence length , determines a subsequence . The final data that serves as input for the classifier consists of the union of n-gram feature sequences plus the composition labels across all recordings. Formally,(4)where denotes the union operator and, as mentioned, indicates the composition index of the -th recording. Notice that, due to the random choice of and the fact that (Table S1), the for each composition might be different. However, notice also that is the same for every recording of composition . Hence, the same subsequence position is taken for all recordings of a composition.

Apart from onset deviations, some of the performed experiments consider other information from the score. This is the case for pitch intervals and relative note durations. Pitch intervals, which we denote by , are expressed in semitone differences between consecutive notes (e.g., semitone, semitones, semitones). Relative note durations, which we denote by , are taken as the written note duration with respect to the beat (e.g., beat, beat, beat). In the latter case, for classification, we compute the 10 different sequences (one for each composition) and then produce 10 copies of each in order to emulate 100 performances. The rest of the process is the same as explained above except that, in the normalization step, we replace the mean by the mode of the distribution. We believe this is a more sensible approach, as the distribution of relative note durations is discrete and often discontinuous (see Fig. S1). Notice that, in the case of relative note durations and in contrast to onset deviations, there will be no differences between performances. This makes relative note durations a very strong adversary against which the predictive power of onset deviations can be compared.

Classification

We cast the problem of identifying the piece from its onset deviations as a 10-class classification problem [30]–[32]. To show that the predictive power of the considered feature sequences is generic and not biased towards a specific classification scheme, we employ basic algorithms exploiting five different machine learning principles [30]–[32]: decision tree learning, instance-based learning, logistic regression, probabilistic learning, and support vector machines. The algorithm implementations we use come from scikits-learn (http://scikit-learn.org) version 0.10 and, unless stated otherwise, their default parameters are taken. Since our focus is on assessing the predictive power of onset deviation sequences rather than obtaining the highest possible classification accuracies, we make no tuning of the classifiers' parameters. In total we use 7 algorithms [30]–[32] plus a random classifier:

• NN: -nearest neighbor classifier. We use the Euclidean distance (NN-E) and dynamic time warping dissimilarity (NN-D). For dynamic time warping we use a standard implementation with a global corridor constraint of 10% of the sequence length [48]. The number of neighbors is arbitrarily set to .
• Tree: classification and regression tree classifier. We use the Gini coefficient as the measure of node impurity and arbitrarily set a minimum number of instances per leaf.
• NB: naive Bayes classifier. We employ a Gaussian function to estimate the likelihood of each onset deviation.
• LR: logistic regression classifier. We use L2-regularized logistic regression with automatically-scaled intercept fit.
• SVM: support vector machine. We consider a linear kernel (SVM-L) and a radial basis function kernel (SVM-R).
• Random: random classifier. We additionally consider a random classifier as the baseline. It outputs a randomly selected class from the pool of all available training labels.

Evaluation Strategy

For each data set we perform standard 20-times, 10-fold, out-of-sample cross-validation [30]–[32]. Even if our music collection is already balanced (10 performances per piece), we force internal training and testing data sets to be balanced as well. Hence, we train with 9 performances per piece and test with 1. We additionally ensure that all classifiers observe the same training/testing sets. As different selections of could affect the results, we repeat the whole process 100 times, in order to obtain a reliable estimation of all possible accuracies (not only for average accuracies and their standard deviations, but also to have a proper idea of maximum/minimum values and reliably assessing statistical significance). In summary, we generate 100 data sets and test each classifier with them. This yields a total of accuracy values (100 for each classifier, including the random baseline) computed from folds.

As we use matched samples , we assess statistical significance with the well-known Wilcoxon signed-rank test [33]. The Wilcoxon signed-rank test is a non-parametric statistical hypothesis test used when comparing two matched samples (or related samples, or repeated measurements) in order to assess whether their population mean ranks differ. It is the natural alternative to the Student's -test for dependent samples when the population distribution cannot be assumed to be normal [33]. We use as input the 200 accuracy values obtained for one classifier and the random baseline. To compensate for multiple pairwise comparisons, we apply the Holm-Bonferroni method [34], a post-hoc statistical analysis method controlling the so-called family-wise error rate that is more powerful than the usual Bonferroni correction [49].