Generative models

The goal of a generative model is to capture the distribution of a set of data to then create new samples in said distribution. Within this category, we find Generative Adversarial Networks [30] (GANs); their later successors, Wasserstein GANs [31] (WGANs); and Variational Autoencoders [32,33] (VAEs). All these architectures use two networks that either compete or collaborate, ultimately creating a latent space to sample from later. The last two of them are employed in our method. Among later architectures, we encounter diffusion models [9], which learn to generate data from noise by learning a denoising function. They condition the process on a target class, descriptor, or prompt to obtain their results. Though these models are widely used for image generation, other data types, like text or music, can be generated with them. In fact, there are specialized architectures for different data types. For example, Generative Pretrained Transformers (GPTs [34,35]) exploit attention mechanisms [8] to generate text word by word, conditioning it on both a prompt and their previously generated text. They are often tuned with Reinforcement Learning from Human Feedback [27,28,36] (RLHF), which profiles preferred outputs based on human judgment, in a way similar to our proposed method.

For the particular case of artificial music composition and generation (referred to as just “music generation”, onward), the first artificial neural networks used for composition started with MultiLayered Perceptrons (MLPs) [37,38], then evolved to Short-Term Memory Networks [39,40] (LSTMs) and into GANs [26,41]. Found among the latter, MuseGAN [26] is a WGAN trained over MIDI files to create piano rolls, binary matrices with positive values representing each sounding note, with up to five different instruments. Coconet [42], a later model that also works on composition, approaches the process by trying to imitate human composers. Instead of writing in the direction of time like other models, a Neural Autoregressive Density Estimator (NADE) [43] fills gaps in Bach chorales. This teaches the model to rewrite and iterate over its pieces, not tying the process to any causal direction. Later models shift focus from the symbolic domain to the audio domain, abstracting the composition process. MelNet [44], a model contemporary to Coconet, generates realistic piano solos and is trained with Mel-spectrograms, 2D decompositions of the soundwave on different frequencies remapped to a quasi-logarithmical perception-based scale. Later in time, we encounter models that exploit the Transformer architecture, like AudioCraft, which combines audio compression [3] and text-to-sound generation [4,5], or Jukebox [1], which can also generate vocal parts from written lyrics. One of the latest trending examples is Bark [2], which is the text-to-speech model used in the AI music generator Suno and is capable of generating music and sound effects next to the lyrics from the text prompt. Lastly, generative models should achieve results that evoke certain emotions, especially if asked to generate personalized content. Hence, architectures that focus on recognising these emotions, such as the graph neural networks HGLER [45] or ENMI [46], can become important additions to music generation applications.

Bayesian optimization

Bayesian optimization has been widely used in many applications and scenarios. Examples include hyperparameter tuning for neural networks [47], robust policy search, planning, and motion in robotics [20,48,49], autonomous systems [12], game agent optimization [50], aircraft design [51], physics modeling and simulation [52], biochemistry [53], ecology [54], and medicine [55–57] among others.

Some works focus on Bayesian optimization as a way to take into account human judgment for tasks like image generation [58], photography color enhancement [16,59], material appearance modeling [16,60], procedural animations [17] and prosthesis calibration [15,61]. These cases usually lean towards preferential Bayesian optimization (PBO), which captures relations between evaluations instead of assigning scalars individually. This takes care of some biases, such as drift or anchoring, when working with human judgment. PBO has been used for the optimization of music generation [29], where users manipulate an interface for melody composition, onto which they can select a two-bar section for the system to propose variations, which they can also edit themselves. However, high-dimensional PBO is still an open problem with few preliminary results addressing it [62] limited to 1D preferences.

High-dimensional vanilla BO, on the other hand, is a well-known issue, and many techniques tackle its limitations [63]. Some assume the effective dimensionality of the problem is smaller than its domain, and hence some dimensions may be uninformative. These dimensions may be axis-aligned [64,65] or not [21,22]. Thus, some authors opt to cull down the search space [21], while others enforce sparse priors on the dimensions [65,66]. Another option is breaking down the GP into a sum of several smaller and independent component functions, which receives the name of Additive kernels [24,67,68]. This approach makes the total complexity of fitting all kernels lower than fitting a single global kernel. Some others choose to change the exploration strategy. Some works propose to impose locality on the optimization [25], which simplifies the process, while others prefer transforming the geometry of the search space by making it non-Euclidean, e.g., through cylindrical kernels [69], which effectively expands the center of the search space while shrinking the borders.

Music generation model

Our method’s foundation is a music generation model. Following former literature [29] on music-oriented BO, we chose symbolic music generation as the domain of our problem.

Symbolic music is most often represented in score sheets: Several instruments play different parts of the score, each with their own dynamics, but interacting with each other. Notes unfold along time following structural patterns, grouping in bars and forming phrases. Lastly, notes are played in different pitches, and stack forming arpeggios and chords.

Our data is a set of piano rolls, which imitate the structure of score sheets. A piano roll is a multi-dimensional binary tensor, which is made of I instrument tracks that play for B bars of T timesteps. Instruments play within a range of P pitches, with zeros meaning silence, and ones meaning sound. These are obtained from and convertible to MIDI files. Mathematically, a piano roll is defined as . This multi-dimensional setup allows for several instruments to play several notes at the same time. Compared to melodies, where only up to one note is playing at a time, it increases the range of possible generations, as well as complexity.

For our selected model, we take inspiration from MuseGAN [26], a WGAN capable of generating piano rolls for up to five instruments. Formally, a WGAN is composed of two networks, a generator G, and a critic C, facing each other in a minmax game

(1)

where G takes an input z from a latent space Z that follows a distribution P_Z, and creates counterfeit samples G(z) that should emulate the real samples x from the real distribution P_X. Meanwhile, C examines both the real and counterfeit samples and outputs values C(x) and C(G(z)) for each of them. C must maximize the difference between the scores of the real samples and the fakes, while G must reduce it. To accelerate convergence, a gradient penalty [70] term is added to the critic’s objective function, which becomes

(2)

where P_G is defined uniformly sampling interpolations between sample pairs (x,G(z)).

Our model, which we named PowGAN, is a lighter version of the “Composer” model of the original MuseGAN that can generate piano rolls for drums, guitar, and bass (called a ‘Power trio’ in rock music, hence the name), taking Gaussian noise as input. The original MuseGAN model also included strings and piano, which we removed since we deemed five instruments were difficult for humans (potentially non-experts) to evaluate within sensible time limits. The piano rolls used for training the model are a subset of the Lakh Pianoroll Dataset [26], which is derived from the Lakh MIDI Dataset [71]. The reasoning for which tracks to remove comes from analyzing this dataset. The piano track was the one with the most silent segments, and the strings track included all other instruments other than the five listed.

Regarding the architecture, both the generator and critic are multi-stream 3D Convolutional Neural Networks. A schematic overview of the generator is depicted in Fig 2. The generator’s architecture consists of an initial part shared among instruments that upsamples the input and later splits into private streams for each instrument. Each instrument stream performs inverse-3D convolutions over bars, time, and pitch, and is also split into two halves where dimension expansion occurs in different orders: time-first, and pitch-first. Each stream’s halves are merged at the end to form the final track for each instrument. The critic follows an architecture mostly symmetric to the generator, with the addition of two global streams that capture harmony and rhythm features for all the composition [26].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. PowGAN’s generator architecture overview.

The model takes a Gaussian noise vector, upsamples it through the shared network, and creates common features that are replicated for all instruments. Features run through private parallel instrument streams. Each instrument stream (drums, guitar, and bass) splits in two, upsamples these features, and merges again, each creating the bars for the corresponding track. Bars in each track are concatenated and form the final piano roll Ψ.

https://doi.org/10.1371/journal.pone.0335853.g002

The features of the generated pieces, and the adequacy of the distribution and the model, are profiled through the following metrics:

• Empty Bar Ratio (EBR). Measures the percentage of completely silent bars. It ranges from 0 to 1. Applies to all tracks.
• Used Pitch Classes (UPC). Measures the number of different notes used in each pitched track. Classes circle back every scale, so the same note played in different scales counts as only one note. It ranges from 0 to 12. Applies to Guitar and Bass.
• Tonal Distance [72] (TD). Measures harmonic relations between pitched tracks. Intuitively, the lower the TD, the stronger the relation and the better the tracks sound together. Applies to the Guitar-Bass pair.

Bayesian optimization

Bayesian optimization is our tool of choice for exploring the latent space of the developed model. BO seeks to optimize an unknown function f using a probabilistic surrogate model P(f), commonly a Gaussian process (GP), and an acquisition function.

A Gaussian process is a collection of random variables such that every finite subset of them has a multivariate normal distribution, and it is defined by its mean and covariance functions. The GP is built on a dataset of n previous observations of f, and its posterior probability is updated every time a new observation is acquired. The posterior model allows predictions y_q at query points which follow a normal distribution such that

(3)

where y is the vector of evaluation results, k is a vector with components , K is the kernel matrix with components for and is the noise variance. There exist multiple options for , but in this work we use the Matèrn kernel which is known to perform well for Bayesian optimization [12,47]. The Matèrn kernel is defined as:

(4)

where is the signal variance and r is the Mahalanobis distance between points x and . In particular, we use the Matèrn kernel with automatic relevance determination (ARD), such that:

(5)

where is a vector of the lengthcales of the kernel resulting in a diagonal matrix with one lengthscale per dimension of x. Each lenghtscale measures the scale of the distance in that dimension. For equally distant coordinates and , a larger lengthscale results in a higher Mahalanobis distance and a higher correlation between and . For regression and optimization, this high correlation means that knowing already allows you to predict with high accuracy and vice versa. Therefore, sampling that coordinate becomes unnecessary, resulting in irrelevant dimensions. Contrary, a small lengthscale means a lower correlation, making independent of . In practice, this requires having multiple samples of that dimension to correctly approximate the function, resulting in a relevant dimension for regression and optimization.

The kernel hyperparameters, which includes the lengthscales, signal variance and noise variance are usually estimated through an empirical Bayesian approach, meaning they are estimated from either maximum marginal likelihood (ML) or maximum a posteriori: . However, empirical Bayes results in an overconfident uncertainty estimation of the Gaussian process as it ignores the uncertainty of the lenghtscale estimates [12,47]. Instead, we estimate the lengthscale posterior distribution through a fully Bayesian approach based on Markov chain Monte Carlo (MCMC) that with better uncertainty estimation and more robust results [12,47,73]. While MCMC is more computationally intensive than ML, the extra cost is negligible compared to the cost of evaluating . Note that in this work, corresponds to generating a piece of music, reproduce it to the user and wait for the rating.

In addition to the surrogate model, BO also uses an acquisition function for efficiently choosing the next evaluation point at each step t of the algorithm, such that . Our choice for is the Expected Improvement (EI) function

(6)

where is the improvement function, ρ is the current best result among the observations of the GP, ϕ and Φ are the Gaussian probability density function (PDF) and cumulative density function (CDF), and and are obtained from Eq 3.

A representative initial dataset is key to fitting a meaningful initial model. We obtain our initial query points through Sobol sampling [74], a quasi-random series that ensures a good cover of the domain. After the initial evaluations, the algorithm runs iteratively by fitting the GP to all available observations and selecting new query points at the maxima of the acquisition function, as portrayed in Fig 3, adding the observations to the dataset. A pseudocode overview is provided in Algorithm 1.

Algorithm 1 Bayesian optimization.

Require: Objective function f, acquisition function a, budget N

1: Initialize data set with the data points and their responses , using a preliminary design.

2: for do

3:   Fit the surrogate model to :

4:  

5:   // With observation noise

6:  

7: end for

8: Return

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Bayesian optimization overview.

The GP (mean in blue, variance in pink) is fit to all previous observations (red dots) that follow the target function (dashed line) and the acquisition function a (EI, in green) indicates where the next query will be sampled.

https://doi.org/10.1371/journal.pone.0335853.g003

Gaussian spaces.

In order to explore a Gaussian space through BO, we need to adapt it from its assumed box-bounded search space into a Gaussian one. Some works [75] extend BO to iteratively increase the bounded volume, potentially to infinity, though it is still a hypercube. In our Gaussian latent space, the distribution of the data is concentrated around the origin, which means our search should be more thorough around this center area, as we can expect greater variation.

To solve this, we use the Box-Muller (BM) transform to map an exploration space [0,1]ⁿ to a Gaussian one . We show an example of how this transformation affects a two-dimensional space in Fig 4, though it works for any even n number of dimensions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Two-dimensional example of the Box-Muller transform.

The Sobol-sampled data (left) ends up following a Gaussian distribution (right) after being transformed. This allows Bayesian optimization to explore Gaussian spaces while respecting the prior of the distribution.

https://doi.org/10.1371/journal.pone.0335853.g004

High-dimensional Bayesian optimization

PowGAN’s 256-dimensional latent space can be considered a high-dimensional space for performing global optimization. General non-linear optimization in high-dimensional problems is an open problem. Most strategies assume that the optimization space can be simplified so that the optimization can be performed in a simpler or lower-dimensional space. For example, some authors [23,24,68] assume that the optimization space has some additive structure  +  f(x₂)  +  . Others perform only optimization in a local region [25] or assume the optimizer can be found in a lower-dimensional embedding [21,65,76]. For our setup, we hypothesize that a user’s preferred sample can be found in a lower-dimensional embedding S^d compared to the model latent space S^D, with dimensionality , considering that some variables might be imperceptible for users not proficient in music or that certain regions of the search space might not capture harmony or proper musical structure.

In this section, we present a couple of baselines for dimensionality reduction along with random rotational embedding Bayesian optimization (ROMBO), our proposed method for high-dimensional optimization. For the baselines, we first build a nonlinear decoder from S^d to S^D by training a variational autoencoder on top of the PowGAN model, which preserves the Gaussian distribution of the latent space. However, that increases the complexity of the model and the training process. Second, we use random embedding Bayesian optimization (REMBO) [21] as a baseline for high-dimensional Bayesian optimization. REMBO also builds a decoder to enable sampling on a lower-dimensional embedding, which in this case is a random linear projection. This guarantees finding a valid optimizer for the high-dimensional problem under certain conditions. REMBO is a fast and efficient method, but once again assumes a box-bounded input space, which conflicts with our Gaussian latent space. To solve this, we lastly introduce our REMBO-inspired algorithm, which we call rotational random embedding Bayesian optimization (ROMBO). The main advantages of ROMBO are its suitability for latent spaces with rotational symmetries, such as the Gaussian latent space of a GAN, and that its low-dimensional space bounds are simpler and fitter while still guaranteeing to find the optimum. Lastly, it is important to note that both REMBO and ROMBO are model-agnostic, meaning that they work with any model, not just PowGAN. Unlike a neural decoder, which requires changes in the architecture and re-training, the BO algorithms can be used out of the box.

Neural decoder.

As a baseline to create a decoder from the low-dimensional space to the high-dimensional space, we propose a neural decoder extending the generator architecture of our music generation model following a variational autoencoder (VAE) methodology.

An autoencoder is composed of two networks: an encoder E that reduces its input to fewer-dimensional features and a decoder D that must reconstruct the original data from these features. Variational autoencoders extend this by turning the latent encoded feature points into latent distributions by imposing a Gaussian prior. This ultimately forms a Gaussian latent space which can later be sampled. This new, smaller latent space and the following layers that upsample it back to a piano roll make up our baseline. In our proposal, we salvage a pre-trained PowGAN and use the generator as D; the critic, sans the head, as E; and place a fully connected core that connects them and learns the latent space, including mean and standard deviation layers to enforce the Gaussian prior. In order to maintain the expressiveness of the GAN, we freeze the layers of PowGAN’s generator and train just the encoder and core of the model. This is portrayed in Fig 5. At the end of training, we keep the neural encoder that has learned to map the feature space S_d to PowGAN’s latent space S_D.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Variational-autoencoder-inspired architecture.

We use PowGAN’s beheaded critic C as encoder E, and generator G as decoder D. Features obtained through E are mapped to a low-dimensional Gaussian latent space (S_d) through fully connected layers. The feature vectors are then mapped back to PowGAN’s high-dimensional latent space (S_D) and restored into piano rolls. All parts but PowGAN’s generator are trained.

https://doi.org/10.1371/journal.pone.0335853.g005

In contrast to a GAN, where the two networks compete with each other, here E and D must both minimize a reconstruction loss . Common choices are the mean absolute error (MAE) and mean square error (MSE). However, these reconstruction errors are unfit for unbalanced, sparse, and heavily structured data, such as our piano rolls, where most values are silence and notes must follow specific patterns. We explore two alternatives for our training losses:

Asymmetrical loss [77]. The first one is an asymmetrical loss (), which focuses on element-wise correspondence while trying to compensate for note/silence imbalance, defined as

(7)

where Ψ and are the original and decoded piano rolls and is the element at track i, bar b, timestep t and pitch p of a piano roll Ψ. m is a negative clipping parameter, with values lower than m being set to zero. and , with , are the asymmetric focus parameters that control how much negative predictions affect positive values and vice versa. Intuitively, this loss reduces the reward for easy true negatives, which prevents local minima such as complete silence (all zeros).

Perceptual loss. Element-wise losses can still lead to reconstructions with small errors that, perceptually, sound very different. This issue is tackled by using perceptual losses, which compare features, rather than single elements. One such example in the image domain is the structural similarity index measure (SSIM) [78], which compares patterns in pixel intensities. Thus, we designed a perceptual loss based on the previously commented musical features.

We first modify the EBR, as it can only output multiples of within the [0,1] interval for a single composition. We substitute it for an alternative, Silence Ratio (SR), calculated as

(8)

for each track i in a piano roll, which measures the percentage of silent elements rather than bars. Then, we formulate our musical feature loss as

(9)

where is a vector composed of the z-scores of the features calculated over the training data distribution for each of them. This includes SR for all instruments and TD and UPC for bass and guitar.

KL divergence. Whether we use or as , a Kullback-Leibler divergence (KL) term is always added to the loss to enforce a latent standard Gaussian distribution. The final loss of the architecture is

(10)

where is the latent space distribution, is a d-dimensional standard Gaussian, and β is a weighting term, which increases and resets regularly by applying KL annealing [79]. This practice prevents the model from ignoring the reconstruction error in favor of minimizing KL divergence, especially at the early stages of training.

Random Embedding Bayesian Optimization.

Random Embedding Bayesian Optimization (REMBO) [21] works under the assumption that, for certain problems, most dimensions do not change the objective function. That is, the effective dimensionality of the problem is smaller than its input domain. One such example is shown in Fig 6, where the objective function is represented as a 2D function where there is only one important dimension, but we do not know which dimension is the important one at the beginning of optimization. Instead, we can build an embedded 1-dimensional subspace (represented as a red line in Fig 6) where we can perform optimization and we are guaranteed to find the optimum.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Example of a function over a 2-dimensional domain and one effective dimension.

Searching over the 1-dimensional embedding is more efficient than searching over the whole domain.

https://doi.org/10.1371/journal.pone.0335853.g006

Formally, a problem with low effective dimensionality is defined as the following:

Definition 1. [21] A function is said to have effective dimensionality d_e, with , if

• there exists a linear subspace of dimension d_e such that for all and , we have , where denotes the orthogonal complement of .
• d_e is the smallest integer with this property.

We call the effective subspace of f and the constant subspace.

REMBO works by building a random linear projection , where the elements of A are independently sampled from a standard normal distribution . Assuming that , Theorem 2 of Wang et al. [21] shows that for any point in the original input space there exists a point , such that . Therefore, for any optimizer there exists a point such that .

A limitation of REMBO is how to define the search space for the lower-dimensional space to avoid missing any point. The authors show that, for any origin-centered box constraint space where is rescaled to [–1,1]^D, and an effective dimensionality of d_e, then with probability at least 1 − ϵ [21]. In practice, the effective dimensionality d_e is unknown, and we can only assume a dimension d > d_e. After empirical testing, they find a suitable bound for the lower-dimensional space [21] which makes .

Our problem differs in that we want to find an optimizer in a high-dimensional space S^D with a standard Gaussian prior . To take into account this prior, we would want our embedding A and the samples to ensure that Ay follows a multivariate Gaussian distribution and that its confidence bounds match those of S^D. We show how achieving this with REMBO and BM alone is not possible, and provide graphic examples in Fig 7 of their results and the desired embeddings, which we obtain through our own adapted method proposed below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Example embedding strategies.

The 1D data is embedded in a 2D space, where the 95% confidence bound of is drawn in magenta. From left to right, data is embedded using REMBO on uniform data, REMBO on Gaussian data, Box-Muller after REMBO on uniform data, and our proposed method. The first two present scaling issues, and the third creates spiral patterns with undesirable artifacts. Our method embeds data as multivariate Gaussians with coincident confidence bounds with the standard distribution.

https://doi.org/10.1371/journal.pone.0335853.g007

REMBO(BM(y)) (Fig 7, left). Assuming our y is bounded within [0,1]^d, this method results in REMBO projecting the points into d-dimensional Gaussians in S^D. This satisfies the multivariate Gaussian requirement. However, due to the random scaling of the embedding, the distribution does not necessarily match the confidence bounds of S^D.

BM(REMBO(y)) (Fig 7, middle). Embedding data within the interval and applying the box constraint, as proposed in REMBO, projects Ay within [–1,1]^D. Re-scaling them to (0,1] and applying Box-Muller forms spiral-like patterns. They provide no guarantee regarding the distribution of the data nor the confidence bounds, and sometimes create artifacts due to the box-constraint clipping, with many points being mapped either to infinity ( with some very small positive ϵ) or the origin.

Since none of these approaches fulfill the requirements, we propose a novel methodology (Fig 7, right) that adapts REMBO to work as intended in Gaussian spaces.

Random rotational embedding Bayesian optimization

Below, we explain how to adapt REMBO to improve its performance when working with Gaussian spaces and how to overcome some of its assumptions, like the box constraint.

Our proposed algorithm, which we call random rotational embedding Bayesian optimization (ROMBO), solves the scaling issue by redefining the embedding matrix so it acts as a rotation rather than a random projection. We provide our process for obtaining this new embedding and proof of its effectiveness below.

Theorem 2. (based on Theorem 2 from Wang et al. [21]) Assume we are given a function with effective dimensionality d_e and a matrix , with , whose columns form a random orthonormal basis for some d-dimensional subspace . Then, with probability 1, for any there exists a such that .

Proof: Following Wang et al. [21], we just need to prove that the matrix , where is an orthonormal basis for , has rank d_e. The rest of the proof applies for any matrix Q that satisfies that. Both and Q have orthonormal columns, and hence are full rank d_e and d, respectively. Their product is a d_e × d matrix, with rank at most d_e. Since Q is randomly oriented relative to , will have rank d_e with probability 1.

For our algorithm, we obtain Q from the QR-decomposition of A, such that . We use the same strategy as REMBO and use a random A matrix whose entries are independent normal samples from . Then, as per the results from Steward [80], we can see that for any matrix A whose elements are independently normally distributed with mean zero and common variance , then Q is an orthonormal basis with random orientation. Although the result from Steward [80] is defined for full rank matrices , with , we can see that our matrix Q is just the first d columns of representing the orthogonal basis for the column space of A. Thus, if has random orientation, then Q has random orientation as well in the column space of A.

Next, we need to consider the optimization problem in a compact space and how we can define the bounds in the embedding , which, as discussed before, is one of the limitations of REMBO.

Theorem 3. (based on Theorem 3 from Wang et al. [21]) Suppose we want to optimize a function with effective dimension . Suppose further that the effective subspace of f is such that is the span of d_e basis vectors, and let be an optimizer of f inside . If Q is a matrix such that its columns form a random orthonormal basis for , there exists an optimizer such that and .

Proof: In this case, our proof is simpler than Theorem 3 [21]. Since Q is an orthonormal basis, is a rotation of in S^D, as all d singular values of Q are 1. Therefore, .

This result is quite important. Contrary to REMBO, our projection matrix Q does not change the size of our exploration space. This is intuitive when we consider that REMBO builds its input space in a hypercube. Meanwhile, our input space is a hypersphere thanks to the rotational symmetry of the latent space as a Gaussian distribution, and the Box-Muller transformation. Therefore, we can maintain the same constant bounds independently of the effective dimension, and therefore, we can find tight bounds even if the effective dimension is unknown. As shown in Fig 7, our method preserves the bounds of the search and input spaces.

Music generation model

The model architecture follows the schema displayed in Fig 2. As discussed, our final piano rolls have three instruments instead of five. The temporal resolution of our output is also coarser, going down from 48 timesteps per bar to 16. The final output of our model is a piano roll . We provide a layer breakdown of the generator in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Layer breakdown of PowGAN’s generator.

‘Shared’ layers are common for all instruments, while ‘Private’ ones are replicated per instrument. Private pitch-first and time-first layers are parallel, and they end up merging with a 1x1 convolution.

https://doi.org/10.1371/journal.pone.0335853.t001

For the training data, each song is divided into 4-bar fragments, which are then sampled uniformly up to a maximum of 18 fragments per song. We ensure that removing the piano and strings does not accidentally create silent fragments, though we maintain them if they were already silent, even with all instruments. Our final training dataset is made of 57.201 fragments. For evaluation, we compare the musical features of the generated songs and the dataset used for training. We measure the EBR, UPC, and TD of a new dataset of 64000 fragments (2000 batches of 32) generated by sampling PowGAN’s high-dimensional Gaussian latent space. We report the metrics obtained by the PowGAN generations in Table 2, and provide the ones reported by MuseGAN [26] as reference. The overall absolute error between our dataset and our model for each of these metrics is similar in scale to the errors reported for MuseGAN, and even improves them in the case of UPCs and EBR for the drum track. With these results, we deem the training as correct and use PowGAN as our music generation model for the optimization tests.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Metric comparison between the used dataset and the closest values reported in evaluation, both for the MuseGAN baseline and PowGAN.

Values in the ‘Model’ row are better the closer to the ‘Dataset’ row. Notice that both datasets show different metrics due to random sampling, track trimming, and sample rejection, but the error still shows a similar scale, which suggests correct training. For the columns, ‘D’, ‘B’, and ‘G’ correspond to Drums, Bass, and Guitar.

https://doi.org/10.1371/journal.pone.0335853.t002

The code for training and evaluation was implemented using PyTorch and based on MuseGAN’s implementation. We train the model for 25000 steps, with a batch size of 32 and using an Adam optimizer with and and otherwise PyTorch default parameters. The generator and critic each use a different optimizer, and at each step, the critic performs 5 forward passes for every forward pass of the generator. For the first 10 steps, these passes are increased to 50, to give the critic a head start. The training was performed on a free Google Colaboratory environment equipped with an Intel Xeon CPU with 2 vCPUs and 13GB of RAM, as well as an NVIDIA Tesla T4 GPU with 16GB of VRAM. The whole execution of the notebook takes about an hour to complete under these conditions.

High-dimensional optimization

The neural decoders are trained using the dataset generated with PowGAN, which is split in a 90/10 ratio for the train and test sets. We use this and not PowGAN’s training data because a GAN generator tends not to replicate the individual data points used to train it [81]. Our decoder is a frozen pre-trained generator, so we cannot confirm whether it can generate the exact pieces used to train PowGAN or not. With a training set made of its generations, we ensure the decoder can replicate them once again.

We use both and to train two neural decoders, referred as ND_AS and ND_MF below. The training was performed on a 12th Gen Intel Core™ i7-12700K CPU with 32GB of RAM and an NVIDIA GeForce RTX 4090 GPU with 24GB of VRAM. We train for 10 epochs and a batch size of 32. The execution takes between fifteen and thirty minutes under these conditions, depending on the loss used for training. REMBO and ROMBO do not need additional training, as discussed earlier. To run the experiments, we develop an application based on the BO library BayesOpt [73]. We use the Python interface for BayesOpt, which provides a wrapper for custom optimization problems.

We design a series of experiments to test the efficacy of the trained neural decoders, REMBO, and ROMBO. The objective of each experiment is to recreate a song generated by forward-passing a point randomly sampled from S^D through the PowGAN generator. This song represents a hypothetical subject’s favorite song, which acts as the target. A total of 20 targets are generated, which are the same for all decoders.

Our objective function starts by taking a query point from our . For REMBO, it rescales it to the recommended interval. For the others, it performs the Box-Muller transform into . The resultant query is projected onto S^D through the selected decoder, and the generator produces the resultant song. The loss (either or ) between this song and the target is the function output. For we chose , and m = 0.05 as our hyperparameters.

We select d = 10 for our latent space dimensionality. All experiments use the 5/2 Matèrn kernel with ARD and the Expected Improvement acquisition function. For REMBO and ROMBO, all trials were done with a single random embedding per target, and the embedding for ROMBO was the Q matrix obtained from the QR-decomposition of the A matrix used for REMBO for that same target. For completeness, we also tested standard BO with no adaptation for high dimensionality, and a naïve approach where we don’t perform optimization. This last method randomly queries the high-dimensional latent space 70 times and saves its best result, but the model receives no feedback. All other methods start each trial with Sobol sampling, collecting 10 data points, and the optimization budget after initialization is set to 60 samples.

The results of the automatic trials, collected in Figs 8 and 9 Table 3, show the evolution and final average values of each loss function through optimization. The results for standard BO have been omitted from these for clarity, as its performance was worse than random due to the high dimensionality of the sampling space.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Average historic best (with 95% confidence bounds) showcasing the evolution of (top) and (bottom) during optimization of (left) and (right) for ROMBO (green), REMBO (red), and naïve random search (cyan).

ROMBO outperforms both in all metrics and targets.

https://doi.org/10.1371/journal.pone.0335853.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Average historic best (with 95% confidence bounds, plotted in logscale) showcasing the evolution of (top) and (bottom) during optimization of (left) and (right) for ROMBO (green) and the NDs trained minimizing (yellow) and (blue).

https://doi.org/10.1371/journal.pone.0335853.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Final average loss after optimization for each decoder and loss combination.

ROMBO outperforms REMBO, naive random search, and both NDs in both metrics when optimizing either of the two loss functions.

https://doi.org/10.1371/journal.pone.0335853.t003

Firstly, the random baseline obtains far higher losses, especially , than ROMBO and REMBO. This shows that high-dimensional BO algorithms help navigate the latent space of the model. ROMBO outperforms, on average, both REMBO and the neural encoders. This is true for both and even when optimizing with a metric and evaluating the other. ROMBO shows an early advantage that persists throughout optimization, suggesting better exploration, as intended when formulating the method.

However, random search beats the neural encoders, the one exception being the obtained by ND_AS. This suggests that the models fail to capture the features of the final songs in a reduced learned space. The curves of the neural decoders also offer additional insight into their behavior. Despite being trained to minimize their respective losses, the one trained on is significantly worse at both. This may be due to the model itself not being complex enough or the loss not being fit for gradient-descent-based strategies. As for the ND_AS, its performance is closer to ROMBO, though it still doesn’t reach the same values, and the latter has less variance in the results. Both metrics follow similar curves during optimization, with slight improvements for the optimized metric at each run.

The results show that ROMBO is the best method out of those we could develop and are proof of our method’s effectiveness. All further tests are carried out using ROMBO as our decoder.

Experimental design

We switch our evaluation method to human judgment, a numerical evaluation from 0 to 10, and they are asked to try and find a favorite generation. Participants were recruited in February 2025. The only requirement for enrollment was a healthy hearing. Musical experience was not addressed. A total of sixteen people enrolled, four female and twelve male participants between the ages of 23 and 43 (average 27.44, standard deviation 5.82), who were divided into two groups of eight (from now on the ‘Bayesopt’ and ‘Random’ groups). The first group uses the whole pipeline with ROMBO, and the second listens to random samples from PowGAN. Participants are not told this information. The objective is to evaluate the optimization’s effectiveness compared to randomly asking the model to generate pieces and to see whether users think the sample refinement process is useful, regardless of the selected method.

Ethics statement

The research process was conducted according to the ethical recommendations of the Declaration of Helsinki and was approved by both our institution’s data protection unit and the Aragón Autonomous Community Research Ethics Committee (CEICA), which supervises fairness in research projects and the use of biological and personal data. Our protocol is non-invasive and involves the gathering of opinion data only, with no risk for any participant, who gave their informed written consent right after explaining the experiment. The consent allowed the abandonment of the experiment, and all data was analyzed anonymously.

Procedure

Once the volunteer has given their written consent, the supervisor tells them the following pieces of information:

• They will be listening to several computer-generated, 10-second-long musical compositions. The objective is to find the song that is best suited to their taste among the possible generations.
• They will be giving a score to each of the compositions, from 0 to 10, and with 0.1 precision. Scores are said aloud to the researcher. Once they confirm their decision, the scores can’t be changed. If they give the same highest score twice, the latter will be considered better.
• They can ask the supervisor to replay each composition once, before giving the score, if they need to. They can also ask to replay the one they gave the current best score to and be reminded of what the score was.
• The experiment lasts for approximately 30 minutes, accounting for short breaks every few songs. The supervisor will tell them when a break starts and stops.

Then, the supervisor generates and plays two random songs. The user is told not to give a score for these songs. This serves two purposes: First, letting the participant know how the generated songs sound, to control expectations. We transform our output piano rolls to MIDI and play them using standard sound fonts. Hearing some samples in advance gives users a preamble to palliate common biases of opinion problems, such as drift and anchoring. Second, giving them the chance to adjust the volume. The user is instructed to turn it up enough so they can clearly hear the music, but not so much as to be uncomfortable for them to listen to.

We choose the experiments’ parameters taking into account the limitations of real users, which mainly involve evaluation time, capping the initial dataset size and iteration budget to a 30-minute time limit. The final number of observations is 22 samples using Sobol and 42 samples using optimization, for a total of 64 samples per participant. Compared to the automatic trials (10 Sobol samples and 60 acquired samples), this gives volunteers more chances to hear different kinds of compositions, potentially finding one that caters to their taste. It is also a way to prepare for noisy observations, as automatic trials used an exact loss, but we cannot expect the same for human evaluation. The breaks are distributed uniformly along the duration of the experiment, with two thirty-second breaks when reaching 16 and 48 evaluations, and a one-minute break when reaching 32 of them. These breaks both reduce the possible fatigue of the user and reset the hearing, so evaluations are not that conditioned on how long users have been listening to music without stopping.

Then, after completing the optimization process, volunteers are introduced to a survey stating four affirmations about the model’s general performance and the optimization process. All statements can be answered using Likert scale answers: “Strongly disagree”, “Disagree”, “Neither agree nor disagree” (onward “No preference”, for short), “Agree”, and “Strongly agree”. Users are asked to tell how much they agree with the following points:

• Q1: “Overall, I like the generated songs”
• Q2: “I like the song I gave the best score to”
• Q3: “I like the song I gave the best score to better than the first song I listened to”
• Q4: “After listening to all the songs, I am alright with the scores I gave to the best and first songs”

Question Q1 asks about the general performance of the composition model. Questions Q2 and Q3 evaluate the need for an optimization pipeline. Finally, question Q4 asks about the users’ ability to act as judges, since they are not given a chance to correct themselves and may not agree with their own judgment after the experiment ends.