https://orcid.org/0000-0002-8956-8529
Figures
Abstract
Brain-computer interface (BCI) technology has gained recognition in various fields, including clinical applications, assistive technology, and human-computer interaction research. BCI enables communication, control, and monitoring of the affective/cognitive states of users. Recently, BCI has also found applications in the artistic field, enabling real-time art composition using brain activity signals, and engaging performers, spectators, or an entire audience with brain activity-based artistic environments. Existing techniques use specific features of brain activity, such as the P300 wave and SSVEPs, to control drawing tools, rather than directly reflecting brain activity in the output image. In this study, we present a novel approach that uses a latent diffusion model, a type of deep neural network, to generate images directly from continuous brain activity. We demonstrate this technology using local field potentials from the neocortex of freely moving rats. This system continuously converted the recorded brain activity into images. Our end-to-end method for generating images from brain activity opens new possibilities for creative expression and experimentation. Notably, our results show that the generated images successfully reflect the dynamic and stochastic nature of the underlying neural activity, providing a unique procedure for visualization of brain function.
Citation: Yamashiro K, Matsumoto N, Ikegaya Y (2024) Diffusion model-based image generation from rat brain activity. PLoS ONE 19(9): e0309709. https://doi.org/10.1371/journal.pone.0309709
Editor: Anwar P. P. Abdul Majeed, Sunway University, MALAYSIA
Received: December 22, 2023; Accepted: August 16, 2024; Published: September 6, 2024
Copyright: © 2024 Yamashiro et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All program files are available from the Github repository (https://github.com/UT-yakusaku/BrainDiffusion/tree/main).
Funding: Funder Name: JST ERATO Grant No.: JPMJER1801 Funding Recipient: YI Funder Name: AMED CREST Grant No.: 24wm0625401h0001 Funding Recipient: YI Funder Name: JSPS Grants-in-Aid for Scientific Research Grant No.: 20K15926 Funding Recipient: NM Funder Name: KOSÉ Cosmetology Research Foundation Grant No.: N/A Funding Recipient: NM.
Competing interests: The authors have declared that no competing interests exist.
1. Introduction
Brain-computer interface (BCI) technology has seen remarkable advancements in recent years, revolutionizing domains such as clinical applications, assistive technology, and human-computer interaction research [1–5]. By establishing a direct communication pathway between the human brain and external devices, BCI systems enable users to control and interact with their surroundings using neural activity signals.
BCI has shown great potential in motor rehabilitation, allowing individuals with motor disabilities to control external devices using their brain signals [2, 6–8]. For example, stroke patients with impaired hand function can use BCI to control a robotic exoskeleton or a virtual avatar, enabling them to perform rehabilitative exercises and regain motor skills [9–12]. BCIs have also advanced assistive communication, where individuals with severe motor disabilities, such as those with amyotrophic lateral sclerosis (ALS) or locked-in syndrome, can communicate by translating their brain activity into text or speech output [13–15]. By focusing on specific tasks or commands, users can spell out words, select options on a screen, or generate synthetic speech.
BCI technology has shown promise in the field of mental health by providing insight into brain activity patterns associated with conditions such as depression, anxiety, and attention deficit hyperactivity disorder (ADHD) [16–24]. Neurofeedback using BCI allows individuals to self-regulate their brain activity by providing real-time feedback. This technique can help train individuals to modify specific brain wave patterns, leading to improved cognitive function and emotional well-being [3].
Notably, the potential of BCI technology extends beyond functional applications and has gained traction among healthy individuals. Some applications include controlling smartphone apps with EEG and playing games using brain activity. BCI has also created a link between brain activity and art [16, 25]. Recently, artists have used BCI to create art dynamically influenced by their own brain activity or that of the audience. This includes exhibitions influenced by audience brain activity and music created according to measured activity [26–29]. The process often involves an intermediate step that decodes the user’s intentions and then adjusts the output based on those decoded actions. While this painting BCI provides users with a novel way to create art, the experience can be enhanced by allowing users to transform their brain activity into paintings in a single step, free from intentional intervention. By completely removing the painting output from conscious control, entirely new forms of art can be created.
In recent years, there has been growing interest in diffusion models (DMs), which are deep generative models [30–32]. DMs have shown remarkable performance in various tasks, such as text-conditional image generation [33–35], image resolution upsampling [36, 37], and image colorization [38, 39]. To make training and inference more efficient, latent diffusion models (LDMs) have been introduced, which compute diffusion processes in the latent space generated by autoencoders [40]. This reduces computational costs and enables the generation of high-resolution images. Diffusion models have expanded the boundaries of art by introducing new ways to create and manipulate visual content.
In this paper, we present a novel approach that leverages LDMs to generate images directly from brain activity signals. Our method provides an end-to-end solution that seamlessly transforms neural activity into visual art, eliminating the need for intermediate steps or manual interpretation. By presenting our experimental results with recorded brain activity from freely moving rats, we demonstrate the feasibility and potential of our approach for translating neural signals into visual artistic output.
2. Related work
The exploration of creative expression through the fusion of art and neural processes has been the subject of research [41, 42], including modalities such as music and painting. Overall, this translating framework first extracts the subject’s intentions, which are then used to control various interfaces, such as an audio mixer and a painting palette [26–28, 43]. Previous research has attempted to help people with disabilities and has successfully enabled paralyzed patients to express their artistic creativity.
2.1 Composing music with neural signals
Musical composition is one of the transformations of neural activity into a form of art [25, 44]. In the context of these investigations, the transformation of neural signals into music is based on the analysis of electroencephalography (EEG) data. Specifically, previous studies have focused on EEG power spectrums, as well as the topological patterns exhibited by EEG data over an array of electrode assemblies. This computed power spectrum is used to control audio mixing and the tuning of musical parameters [43]. It can also facilitate direct engagement with musical instruments, thereby enriching the interaction between individuals and their musical creations.
2.2 Painting using neural signals
Painting using neural activity is another application that enables creative expression. In these paradigms, a specific neural component triggered by an internal or external event is used. The conversion of neural activity into painting is accomplished by decoding of the subject’s intentions from the specific neural component. The subject’s intentions are then used to control a palette on a screen. In previous research, two types of neural components have been considered, the P300 wave [26–28] and the steady state visual evoked potential (SSVEP) [29].
The P300 wave is detectable as a positive shift in the EEG signal 200–400 ms after the external stimulus is presented to the subject. The P300 wave is associated with the subject’s response to a stimulus and is considered an endogenous potential. The application of P300 waves to BCI is most commonly used as a speller for locked-in patients. The P300 speller uses a speller matrix containing letters and digits. In this system, the user directs attention to a character within a matrix, where each row or column is rapidly highlighted in a pseudo-random manner. If the highlighted row or column contains a letter desired by the subject, the P300 wave is detected in the EEG signal. By analyzing the evoked responses, the BCI is able to identify the desired letter by determining the row and column that evoke the most significant response. P300 brain painting uses this P300 speller matrix to create the P300 palette matrix, where different colors and shapes can be selected from the P300 response. The drawback of the P300 painting application was the slow response, which affected the efficiency of the system.
Some researchers have turned to SSVEP-based applications to overcome the challenges of P300 systems. The SSVEP is a continuous response generated in the visual cortex that synchronizes with the frequency of visual stimuli presented to the subject. The subject is exposed to a matrix of cells, each flickering at a different frequency. By directing attention to a specific target cell, the subject can effectively make a choice. This SSVEP-driven approach provides an enhanced rate of information transfer. The integration of hybrid brain-computer interface (BCI) systems that combine P300 and SSVEP has led to advances in real-time human-computer interaction. The brain painting application that used this system has successfully improved the subject’s real-time interaction.
Previous research on brain painting systems has effectively enabled users to express their artistic creativity by facilitating the translation of their intentions into artistic compositions. These systems typically begin by extracting the subject’s intentions, which are then used as input to control the painting palette on a computer screen. Consequently, the resulting artwork represents the subject’s intentions. Thus, the development of a novel application of brain painting depends on the development of a system that directly translates neural signals into works of art, which represents a promising avenue for artistic expression in BCI.
3. Methods
3.1 Datasets
3.1.1 Ethical approvals.
Animal experiments were performed with the approval of the Animal Experiment Ethics Committee at the University of Tokyo (approval number: P29-7) and according to the University of Tokyo guidelines for the care and use of laboratory animals. These experimental protocols were carried out in accordance with the Fundamental Guidelines for the Proper Conduct of Animal Experiments and Related Activities of the Academic Research Institutions (Ministry of Education, Culture, Sports, Science and Technology, Notice No. 71 of 2006), the Standards for Breeding and Housing of and Pain Alleviation for Experimental Animals (Ministry of the Environment, Notice No. 88 of 2006) and the Guidelines on the Method of Animal Disposal (Prime Minister’s Office, Notice No. 40 of 1995). Although our experimental protocols require that animals be humanely euthanized if they show signs of pain, marked lethargy, and discomfort, we did not observe such symptoms in any of the rats tested in this study. Every effort was made to minimize animal suffering.
3.1.2 Animal strains.
Local field potentials (LFPs) were recorded from Long-Evans rats (Japan SLC, Shizuoka, Japan). They were individually housed under conditions of controlled temperature and humidity (22 ± 1°C, 55 ± 5%) and maintained on a 12:12-h light/dark cycle (lights off from 7:00 to 19:00) with ad libitum access to food and water unless otherwise specified.
3.1.3 Surgery.
A custom 32-channel electrode was used to record LFPs from rats. This electrode was designed to cover the right S1 region, specifically the forelimb and hindlimb subregions. At the beginning of surgery, a rat was anesthetized with 2–3% isoflurane gas. A square craniotomy (2–6 mm posterior and 1–5 mm lateral to bregma) was performed using a dental drill. In addition, two stainless steel screws were implanted in the bone above the cerebellum as ground and reference electrodes. The recording device and electrodes were secured to the skull with stainless steel screws and dental cement. After surgery, the rat was housed in a transparent Plexiglas cage with free access to food and water for one week.
3.1.4 Recording.
After one week of recovery from surgery, LFPs were recorded from a freely moving rat in an open field. The rat was placed in a 40×40 cm open field. Several plastic objects were placed inside the open field to motivate the rat to move and not stay in one place. LFPs were recorded using the Open Ephys recording system (http://open-ephys.org). The recorded LFPs were referenced to the ground and digitized at 3000 Hz. The digitized data were simultaneously recorded with an overhead camera at 30 Hz.
3.2 Latent diffusion models
3.2.1 Basic architecture of diffusion models.
Diffusion models (DMs) are probabilistic generative models trained to recover a sample variable from Gaussian noise. Given a Gaussian noise input, the model transforms it into a sample from a learned data distribution through an iterative denoising process. The diffusion model is defined by two processes, the forward diffusion process and the backward diffusion process. To further explain this concept, we will review the mechanics of the diffusion process.
The forward diffusion process starts with a sample from the known data distribution. Small amounts of noise, usually Gaussian noise, are added to this initial sample stepwise. The addition of noise at each step produces a sequence of noisy samples. For each step t of the original data, Gaussian noise is added incrementally up to step T. The larger the T, the final xT will be nearly an isotropic Gaussian distribution.
In the backward diffusion process, the diffusion model learns the reverse process of the forward diffusion process to acquire a sample from the Gaussian distribution. Essentially, at each step in this backward diffusion process, a noisy sample xT−1 is predicted based on the preceding sample xT. This recovery of the original data from the Gaussian distribution is accomplished through the implementation of a neural network. The widely adopted architecture for this neural network in the reverse diffusion process is the U-net. Once the model is trained to recover data from the Gaussian distribution, the model is capable of generating a new sample from the original data distribution given a noisy input. The application of diffusion models to images has allowed the generation of realistic images from initial Gaussian noise data.
3.2.2 Model used in this study.
A variant of diffusion models known as LDM is used in this research. In conventional diffusion models, image generation takes place in high-dimensional pixel space, which requires significant computational resources. In contrast, the LDM performs diffusion processes within the latent diffusion space, a compressed image representation generated by an autoencoder. Initially, Gaussian noise is compressed into this latent space by an encoder, preserving maximum information while reducing sample dimensions. The subsequent diffusion process mirrors that of standard diffusion models. Finally, a decoder reconstructs an output image based on a sample from the latent space. In this study, the dimension of the latent space was 64×64×4, and the decoder reconstructed this latent matrix into image space, returning a 3-channel RGB image with size 512×512.
As described in the previous section, diffusion models are trained on a large set of images. Since diffusion models are large models, training a model can be resource intensive. However, there are many already trained open-source models available online, and we took advantage of these models. In our implementation, we used a pre-trained model available online called Stable Diffusion version 1.5. The original model was adapted to manipulate generated images based on textual input via a pre-trained text encoder (Contrastive Language-Image Pretraining: CLIP). However, in our implementation, no conditioning by textual input was applied. Consequently, the generated images were entirely dependent on the input neural activity.
3.3 Seamless transformation of neural signals to images
3.3.1 Preprocessing of neural signals.
To convert the LFP into images, the LFP was segmented and processed through a denoising U-net and a latent-to-image decoder provided by the author of the LDM (Fig 1A). The LDMs require input data with dimensions of 64×64×4. First, rat LFPs were filtered with a low-pass filter at 30 Hz. This cutoff was chosen to reduce high-frequency noise, such as muscle artifacts and electromagnetic interference, and to ensure that the data primarily represent neural activity. This frequency range includes the most relevant neural oscillations, such as delta (0.5–4 Hz), theta (4–8 Hz), alpha (8–12 Hz), and beta (12–30 Hz) rhythms, which are essential for analysis of cortical LFPs. Filtering at 30 Hz also smooths the signal, making it more suitable for further processing and analysis. The filtered signal was segmented into windows of 512 points with a sliding step of 100, creating a continuous data matrix of 512×32 points every 1/30 second (Fig 1B). The segment size of 512 points was selected to balance temporal resolution and computational load. This size captures meaningful patterns in the LFP data while remaining computationally efficient. The segment size also meets the input requirements for LDMs, allowing for effective transformation of LFP data into latent space representations suitable for image generation.
A. A photo of an experimental setup. B. Detailed illustration of the steps taken to generate image from rat LFP. The recorded LFPs were mapped onto noisy latent vector zT, which was then processed by denoising U-Net to produce a denoised latent vector z. This latent representation of the denoised image was then processed by a latent-to-image decoder to produce image X. C. Schematic of the generation of morphing images from rat LFP. A 512×32 matrix was sampled every 1/30 second with overlaps. The cropped LFPs were then processed through the denoising U-Net and the decoder to generate images at a rate of 30 Hz.
3.3.2 Image generation from LFPs.
The data matrix was then resized to 64×64×4 to match the dimension of the latent space and shuffled with a fixed random seed, mapping the segmented LFP to a noisy latent vector zt. The fixed seed ensured that changes in the same element in successive data matrices were aligned with the temporal signal of the LFP. Next, the reshaped and shuffled data matrix was standardized by subtracting the mean and then dividing by the standard deviation. Through these steps of reshaping, shuffling, and standardization, the LFP segment was transformed into a noisy latent vector that served as input to the latent diffusion model.
The noisy latent vector (zt) was then subsequently processed by the LDM. First, the noisy latent vector was processed by denoising U-Net to produce a denoised latent vector z. In this process, text input is commonly inserted into the denoising process for text conditioning. However, our implementation deliberately excluded any text input, thus generating images that only reflect the fluctuation of LFPs. Finally, the denoised latent vector of size 64×64×4 was converted to image space using the latent-to-image decoder. Therefore, the model returned a 512×512 image for each LFP segment. The resulting image sequence was transformed into a 30-fps video clip.
The model was executed using a Python script adapted from the official StableDiffusion GitHub repository and ran on the NVIDIA RTX A6000 platform. For the diffusion processes, we used five denoising steps as parameters of the diffusion models, with the inference time of each frame ranging from 2 to 3 seconds. Default parameters provided by the authors of LDM were used in all implementations unless otherwise noted.
4. Results and discussion
In this work, we developed a pipeline that enables the generation of art from brain activity using LDM (Fig 1A). In this pipeline, the recorded brain activity (LFPs, herein) was mapped to a latent vector, which was then processed by a denoising U-net. The denoised latent vector was then expanded into image space by a latent-to-image decoder. This entire process runs seamlessly without any user or operator input or intervention, enabling a fully automated generation of images derived from brain activity.
We have further extended our LFP-to-image transformation framework to account for the continuous fluctuation of brain activity. Fluctuations in brain activity are experimentally observed and can be characterized as non-deterministic and deterministic, along with oscillations at different frequencies [45–48]. In the cortex, these stochastic processes are known to be non-Gaussian [49] and unpredictable. We applied our LFP-to-image transformation to capture the fluctuations in continuously changing LFPs to visualize the stochastic process in the rat cortex (Fig 1B). In this approach, we first extracted LFP segments from the continuous recording with a sliding step of 100 points, resulting in similar matrices when they were close together. Consequently, each matrix was positioned close to its neighboring matrices in multidimensional space. To illustrate the arrangement of these matrices, we used Uniform Manifold Approximation and Projection (UMAP) on the converted LFP matrix (Fig 2A). Since the noisy latent vectors converted from segmented LFPs also reflected continuity in multidimensional space, the output of the LDM was also similar between successive frames. This allowed the generation of morphing images that captured fluctuations in brain activity in the rat cortex (Fig 2B). We observed that the structural similarity index of consecutive image pairs generated from consecutive LFP segments was higher than that of image pairs generated from two random latent vectors derived from the Gaussian distribution (p = 1.50 × 10−198, t1485 = 38.65, nGauss = 486, nLFP = 486, Student’s t-test: Fig 2C). This suggests that our model captures the continuity of neural activity over time, resulting in more similar images for successive LFP segments. By exploiting these continuous changes in neural state and local dynamics, our diffusion model generates diverse and unique images that reflect the complex and variable nature of brain activity.
A. 3D visualization of latent vector zT using UMAP. Embeddings are from a 5-minute recording of LFPs. Pseudocolor labels indicate the elapsed time since the start of the recording. B. Samples from morphing images over a 10-s duration sampled every 3 frames. C. Structural similarity index of the two consecutive image pairs generated from the Gaussian and LFP latent matrices (* p = 1.50 × 10−198, t1485 = 38.65, nGauss = 486, nLFP = 486, Student’s t-test).
Because the global state of neural activity is constantly changing within the multidimensional neural manifold [50, 51] and combined with local dynamics such as oscillation strength and subtle shifts in neuronal firing patterns [52, 53], similar images are rarely generated, especially when their latent vectors are temporally separated. Consequently, our LFP-to-image pipeline has the potential to generate an infinite number of image variations (Fig 3A and 3B). Interestingly, we found that images generated from LFP-derived latent vectors have different characteristics compared to commonly used latent vectors derived from Gaussian distributions. Even after standardization, the LFP latent vectors mapped to different locations compared to the Gaussian latent vectors, and the variance of each element of the vectors was higher compared to the baseline (Fig 4A and 4B). Furthermore, the analysis of the generated images showed that color entropy (p = 1.74 × 10−110, t1485 = -24.35, nGauss = 1000, nLFP = 487, Student’s t-test: Fig 4C) and texture entropy (p = 2.22 × 10−51, t1485 = -15.68, nGauss = 1000, nLFP = 487, Student’s t-test: Fig 4D) were higher in the LFP-derived images, indicating a rich variety of colors and elements that come together in a single image to convey a wealth of visual information.
A. Spatial distribution of selected frames in the embedding space. B. Noisy latent vector zT and the corresponding generated image obtained by the diffusion process and the latent-to-image decoder.
A. Spatial distribution of the latent matrix in the embedding space. Gray represents the baseline latent matrix derived from a Gaussian distribution, while red represents the LFP-mapped latent matrix. B. Variance of each element in the LFP-derived and Gaussian-derived latent matrices. C. Color entropy of the images generated using the Gaussian and LFP latent matrices (* p = 1.74 × 10−110, t1485 = -24.35, nGauss = 1000, nLFP = 487, Student’s t-test). D. Same as C, but for texture entropy (* p = 2.22 × 10−51, t1485 = -15.68, nGauss = 1000, nLFP = 487, Student’s t-test).
Furthermore, because open-source models can be downloaded and fine-tuned for LDM customization, a wide variety of LDMs are readily available today. As an example of the customization of our model, Fig 5 shows an example of changing the style of the generated images. While our primary focus has been on image generation using LDMs, recent advances have broadened the scope to include the generation of audio through diffusion processes [54, 55]. It is conceivable that audio-generating LDMs could be integrated with brain activity recordings, allowing the composition of music that reflects fluctuations in neural activity. This wealth of possibilities underscores the virtually limitless potential for art generation using BCI technology.
Morphing images generated from LFPs using different models. From top to bottom: OpenJourney, Stable-Diffusion v1.4, OilPainting, FruitFusion. All are available on Civitai.com.
If our framework were extended to humans, the potential to generate works of art without the need for conscious control holds promise for enhancing creativity [56, 57]. The integration of art and BCIs has advanced in recent years, spanning domains such as music [44, 58–61] and painting [56, 57, 62]. In brain painting, previous research has successfully used characteristic responses in brain activity to directly control a palette for drawing paintings [27–29, 57, 63, 64]. Our approach differs from other work on image generation in that it eliminates the need for an intermediate step in decoding intentions. The seamless transformation of brain activity into images provides a unique experience for the audience. In addition, our system may make it easier for artists to generate creative ideas because the output is independent of their consciousness. Physiological measures of creative processes are often assessed using divergent thinking [65] and remote association [66, 67]. The unconscious generation of images from one’s brain activity introduces a novel concept to the user. For example, divergent thinking is a cognitive process used to generate creative ideas by exploring numerous potential solutions. Offering a new perspective on painting through the capabilities of artificial neural networks expands the range of solutions beyond conscious ideas, thereby fostering heightened creativity.
Generating art from neural activity could also create interactive installations for the viewer’s artistic experience, where the visitor’s brain activity influences the visual display in real time. Although this is possible with previous research, our method of generating images that directly reflect the viewer’s neural activity will make the art unique to each visitor. For example, an exhibition space could have screens that morph and change based on the neural activity of individuals as they move through the space.
The proposed pipeline differs from previous methods in that it does not require user interfaces such as a painting palette. This enables personalized artistic therapy, where individuals could use this technology to create personalized art based on their brain activity, providing a visual outlet for emotions and thoughts that may be difficult to express verbally. This could be particularly beneficial in psychological and therapeutic settings, providing new avenues for self-expression and emotional release. In addition, the images generated could be used as visual feedback in neurofeedback training programs to help individuals improve cognitive function or manage mental health conditions.
This proposed framework of using images to reflect stochastic fluctuations in neural activity could be extended beyond art generation. This technology could be used in educational settings to help students visualize and understand brain function and neural processes. For example, neuroscience students could see real-time visualizations of their own brain activity during various cognitive tasks, enhancing their learning experience. In addition, integrating this technology into virtual reality (VR) and gaming could create new forms of interactive entertainment where the game environment changes based on the player’s brain activity, providing a more immersive and responsive experience.
The fusion of art and BCI technology offers innovative possibilities, as evidenced by our development of frameworks that enable the direct translation of brain activity into visual art. Although our implementation has been in rats, the concept of "brain painting" not only provides a novel means of expression for artists and non-artists alike, but also offers a unique avenue for the unconscious generation of creative ideas. At the intersection of neuroscience, artificial intelligence, and artistic creation, the continuing evolution of technology promises not only to redefine our approach to art, but also to unlock new dimensions of creativity.
Acknowledgments
The authors thank the laboratory members for helpful discussions, valuable comments, and technical support.
References
- 1. Allison BZ, Wolpaw EW, Wolpaw JR. Brain-computer interface systems: progress and prospects. Expert Rev Med Devices. 2007 Jul;4(4):463–74. pmid:17605682
- 2. Lebedev MA, Nicolelis MAL. Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation. Physiol Rev. 2017 Apr;97(2):767–837. pmid:28275048
- 3. Brickwedde M, Krüger MC, Dinse HR. Somatosensory alpha oscillations gate perceptual learning efficiency. Nat Commun. 2019 Jan 16;10(1):1–9.
- 4.
Tan D, Nijholt A. Brain-Computer Interfaces and Human-Computer Interaction. In: Tan DS, Nijholt A, editors. Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction. London: Springer London; 2010. p. 3–19.
- 5.
Zander TO, Kothe C, Jatzev S, Gaertner M. Enhancing Human-Computer Interaction with Input from Active and Passive Brain-Computer Interfaces. In: Tan DS, Nijholt A, editors. Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction. London: Springer London; 2010. p. 181–99.
- 6. Chaudhary U, Birbaumer N, Ramos-Murguialday A. Brain–computer interfaces for communication and rehabilitation. Nat Rev Neurol. 2016 Aug 19;12(9):513–25. pmid:27539560
- 7. Pichiorri F, Morone G, Petti M, Toppi J, Pisotta I, Molinari M, et al. Brain-computer interface boosts motor imagery practice during stroke recovery. Ann Neurol. 2015 May;77(5):851–65. pmid:25712802
- 8. Bundy DT, Souders L, Baranyai K, Leonard L, Schalk G, Coker R, et al. Contralesional Brain–Computer Interface Control of a Powered Exoskeleton for Motor Recovery in Chronic Stroke Survivors. Stroke. 2017 Jul 1;48(7):1908–15. pmid:28550098
- 9. Millán J del R, Renkens F, Mouriño J, Gerstner W. Noninvasive brain-actuated control of a mobile robot by human EEG. IEEE Trans Biomed Eng. 2004 Jun;51(6):1026–33. pmid:15188874
- 10. Gandhi V, Prasad G, Coyle D, Behera L, McGinnity TM. EEG-Based Mobile Robot Control Through an Adaptive Brain–Robot Interface. IEEE Trans Syst Man Cybern. 2014 Sep;44(9):1278–85.
- 11. Tidoni E, Abu-Alqumsan M, Leonardis D, Kapeller C, Fusco G, Guger C, et al. Local and Remote Cooperation With Virtual and Robotic Agents: A P300 BCI Study in Healthy and People Living With Spinal Cord Injury. IEEE Trans Neural Syst Rehabil Eng. 2017 Sep;25(9):1622–32. pmid:28026777
- 12. Arpaia P, Duraccio L, Moccaldi N, Rossi S. Wearable Brain–Computer Interface Instrumentation for Robot-Based Rehabilitation by Augmented Reality. IEEE Trans Instrum Meas. 2020 Sep;69(9):6362–71.
- 13. Anumanchipalli GK, Chartier J, Chang EF. Speech synthesis from neural decoding of spoken sentences. Nature. 2019 Apr;568(7753):493–8. pmid:31019317
- 14. Moses DA, Metzger SL, Liu JR, Anumanchipalli GK, Makin JG, Sun PF, et al. Neuroprosthesis for decoding speech in a paralyzed person with Anarthria. N Engl J Med. 2021 Jul 15;385(3):217–27. pmid:34260835
- 15. Moses DA, Leonard MK, Makin JG, Chang EF. Real-time decoding of question-and-answer speech dialogue using human cortical activity. Nat Commun. 2019 Jul 30;10(1):3096. pmid:31363096
- 16. Andujar M, Crawford CS, Nijholt A, Jackson F, Gilbert JE. Artistic brain-computer interfaces: the expression and stimulation of the user’s affective state. Brain-Computer Interfaces. 2015 Apr 3;2(2–3):60–9.
- 17. Nijboer F, Birbaumer N, Kübler A. The influence of psychological state and motivation on brain-computer interface performance in patients with amyotrophic lateral sclerosis—a longitudinal study. Front Neurosci [Internet]. 2010 Jul 21;4. Available from: https://www.frontiersin.org/articles/10.3389/fnins.2010.00055/full pmid:20700521
- 18. Amaral C, Mouga S, Simões M, Pereira HC, Bernardino I, Quental H, et al. A Feasibility Clinical Trial to Improve Social Attention in Autistic Spectrum Disorder (ASD) Using a Brain Computer Interface. Front Neurosci. 2018 Jul 13;12:477. pmid:30061811
- 19. Mayberg HS, Lozano AM, Voon V, McNeely HE, Seminowicz D, Hamani C, et al. Deep brain stimulation for treatment-resistant depression. Neuron. 2005 Mar 3;45(5):651–60. pmid:15748841
- 20. Drobisz D, Damborská A. Deep brain stimulation targets for treating depression. Behav Brain Res. 2019 Feb 1;359:266–73. pmid:30414974
- 21. Schlaepfer TE, Bewernick BH, Kayser S, Mädler B, Coenen VA. Rapid effects of deep brain stimulation for treatment-resistant major depression. Biol Psychiatry. 2013 Jun 15;73(12):1204–12. pmid:23562618
- 22. Lim CG, Lee TS, Guan C, Sheng Fung DS, Cheung YB, Teng SSW, et al. Effectiveness of a brain-computer interface based programme for the treatment of ADHD: a pilot study. Psychopharmacol Bull. 2010;43(1):73–82. pmid:20581801
- 23. Rohani DA, Sorensen HBD, Puthusserypady S. Brain-computer interface using P300 and virtual reality: a gaming approach for treating ADHD. Conf Proc IEEE Eng Med Biol Soc. 2014;2014:3606–9. pmid:25570771
- 24. Lim CG, Lee TS, Guan C, Fung DSS, Zhao Y, Teng SSW, et al. A brain-computer interface based attention training program for treating attention deficit hyperactivity disorder. PLoS One. 2012 Oct 24;7(10):e46692. pmid:23115630
- 25. Straebel V, Thoben W. Alvin Lucier’s Music for Solo Performer: Experimental music beyond sonification. Organised Sound. 2014 Apr;19(1):17–29.
- 26. Holz EM, Botrel L, Kübler A. Independent home use of Brain Painting improves quality of life of two artists in the locked-in state diagnosed with amyotrophic lateral sclerosis. Brain-Computer Interfaces. 2015 Apr 3;2(2–3):117–34.
- 27. Münßinger JI, Halder S, Kleih SC, Furdea A, Raco V, Hösle A, et al. Brain Painting: First evaluation of a new brain-computer interface application with ALS-patients and healthy volunteers. Front Neurosci. 2010 Nov 22;4:182. pmid:21151375
- 28. Botrel L, Holz EM, Kübler A. Brain Painting V2: evaluation of P300-based brain-computer interface for creative expression by an end-user following the user-centered design. Brain-Computer Interfaces. 2015 Apr 3;2(2–3):135–49.
- 29. Tang Z, Wang X, Wu J, Ping Y, Guo X, Cui Z. A BCI painting system using a hybrid control approach based on SSVEP and P300. Comput Biol Med. 2022 Nov;150:106118. pmid:36166987
- 30. Ho Jain, Abbeel . Denoising diffusion probabilistic models. Adv Neural Inf Process Syst [Internet]. Available from: https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html
- 31.
Sohl-Dickstein J, Weiss E, Maheswaranathan N, Ganguli S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In: Bach F, Blei D, editors. Proceedings of the 32nd International Conference on Machine Learning. Lille, France: PMLR; 07–09 Jul 2015. p. 2256–65. (Proceedings of Machine Learning Research; vol. 37).
- 32.
Generative modeling by estimating gradients of the data distribution [Internet]. [cited 2023 Dec 5]. Available from: https://proceedings.neurips.cc/paper_files/paper/2019/hash/3001ef257407d5a371a96dcd947c7d93-Abstract.html
- 33. Batzolis G, Stanczuk J, Schönlieb CB, Etmann C. Conditional Image Generation with Score-Based Diffusion Models [Internet]. arXiv [cs.LG]. 2021. Available from: http://arxiv.org/abs/2111.13606
- 34. Ho J, Saharia C, Chan W, Fleet DJ, Norouzi M, Salimans T. Cascaded diffusion models for high fidelity image generation. J Mach Learn Res. 2022 Jan 1;23(1):2249–81.
- 35. Ramesh A, Dhariwal P, Nichol A, Chu C, Chen M. Hierarchical text-conditional image generation with CLIP latents [Internet]. arXiv [cs.CV]. 2022 [cited 2023 Dec 5]. Available from: https://3dvar.com/Ramesh2022Hierarchical.pdf
- 36. Saharia C, Ho J, Chan W, Salimans T, Fleet DJ, Norouzi M. Image Super-Resolution via Iterative Refinement. IEEE Trans Pattern Anal Mach Intell. 2023 Apr;45(4):4713–26. pmid:36094974
- 37. Li H, Yang Y, Chang M, Chen S, Feng H, Xu Z, et al. SRDiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing. 2022 Mar 28;479:47–59.
- 38.
Saharia C, Chan W, Chang H, Lee C, Ho J, Salimans T, et al. Palette: Image-to-Image Diffusion Models. In: ACM SIGGRAPH 2022 Conference Proceedings. New York, NY, USA: Association for Computing Machinery; 2022. p. 1–10. (SIGGRAPH ‘22).
- 39. Peter P, Kaufhold L, Weickert J. Turning Diffusion-Based Image Colorization Into Efficient Color Compression. IEEE Trans Image Process. 2017 Feb;26(2):860–9. pmid:27849527
- 40. Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-Resolution Image Synthesis with Latent Diffusion Models [Internet]. arXiv [cs.CV]. 2021. Available from: http://arxiv.org/abs/2112.10752
- 41. Siler T. Neuroart: picturing the neuroscience of intentional actions in art and science. Front Hum Neurosci. 2015 Jul 23;9:410. pmid:26257629
- 42. Plioplys AV. Fusion of neuroscience and art. Lancet Neurol. 2010 Apr;9(4):350–1.
- 43. Miranda ER. Plymouth brain-computer music interfacing project: from EEG audio mixers to composition informed by cognitive neuroscience. International Journal of Arts and Technology. 2010 Jan 1;3(2–3):154–76.
- 44. Matthias J, Ryan N. Cortical Songs: Musical Performance Events triggered by artificial spiking neurons. Body Space Technol [Internet]. 2007 Jul 1;7(1). Available from: https://www.bstjournal.com/article/id/6710/
- 45. Deco G, Rolls ET, Romo R. Stochastic dynamics as a principle of brain function. Prog Neurobiol. 2009 May;88(1):1–16. pmid:19428958
- 46. McKinstry-Wu AR, Wasilczuk AZ, Harrison BA, Bedell VM, Sridharan MJ, Breig JJ, et al. Analysis of stochastic fluctuations in responsiveness is a critical step toward personalized anesthesia. Elife. 2019 Dec 3;8:e50143. pmid:31793434
- 47. Bogler C, Grujičić B, Haynes JD. Clarifying the nature of stochastic fluctuations and accumulation processes in spontaneous movements. Front Psychol. 2023 Oct 12;14:1271180. pmid:37901069
- 48. Vázquez-Rodríguez B, Avena-Koenigsberger A, Sporns O, Griffa A, Hagmann P, Larralde H. Stochastic resonance at criticality in a network model of the human cortex. Sci Rep. 2017 Oct 12;7(1):1–12.
- 49. Freyer F, Aquino K, Robinson PA, Ritter P, Breakspear M. Bistability and non-Gaussian fluctuations in spontaneous cortical activity. J Neurosci. 2009 Jul 1;29(26):8512–24. pmid:19571142
- 50. Altan E, Solla SA, Miller LE, Perreault EJ. Estimating the dimensionality of the manifold underlying multi-electrode neural recordings. PLoS Comput Biol. 2021 Nov;17(11):e1008591. pmid:34843461
- 51. Feulner B, Clopath C. Neural manifold under plasticity in a goal driven learning behaviour. PLoS Comput Biol. 2021 Feb;17(2):e1008621. pmid:33544700
- 52. Pina JE, Bodner M, Ermentrout B. Oscillations in working memory and neural binding: A mechanism for multiple memories and their interactions. PLoS Comput Biol. 2018 Nov;14(11):e1006517. pmid:30419015
- 53. Battaglia D, Witt A, Wolf F, Geisel T. Dynamic effective connectivity of inter-areal brain circuits. PLoS Comput Biol. 2012 Mar 22;8(3):e1002438. pmid:22457614
- 54. Liu H, Chen Z, Yuan Y, Mei X, Liu X, Mandic D, et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models [Internet]. arXiv [cs.SD]. 2023. Available from: http://arxiv.org/abs/2301.12503
- 55. Schneider F, Kamal O, Jin Z, Schölkopf B. Moûsai: Text-to-Music Generation with Long-Context Latent Diffusion [Internet]. arXiv [cs.CL]. 2023. Available from: http://arxiv.org/abs/2301.11757
- 56.
Todd DA, McCullagh PJ, Mulvenna MD, Lightbody G. Investigating the use of brain-computer interaction to facilitate creativity. In: Proceedings of the 3rd Augmented Human International Conference. New York, NY, USA: Association for Computing Machinery; 2012. p. 1–8. (AH ‘12).
- 57.
van de Laar BLA, Brugman I, Nijboer F, Poel M, Nijholt A. BrainBrush, a multimodal application for creative expressivity. In: Sixth International Conference on Advances in Computer-Human Interactions (ACHI 2013). IARIA XPS Press; 2013. p. 62–7.
- 58. Sanyal S, Nag S, Banerjee A, Sengupta R, Ghosh D. Music of brain and music on brain: a novel EEG sonification approach. Cogn Neurodyn. 2019 Feb;13(1):13–31. pmid:30728868
- 59. Lutters B, Koehler PJ. Brainwaves in concert: the 20th century sonification of the electroencephalogram. Brain. 2016 Oct;139(Pt 10):2809–14. pmid:27543971
- 60. Hermann T, Meinicke P, Bekel H, Ritter H, Muller HM, Weiss S. SONIFICATIONS FOR EEG DATA ANALYSIS [Internet]. [cited 2023 Dec 4]. Available from: https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=4b2faf4b681d010dc2e9f44fad47362182451f4d
- 61.
Guide to Brain-Computer Music Interfacing. Springer London; 18 p.
- 62. Nijholt A, Nam CS. Arts and Brain-Computer Interfaces (BCIs). Brain Comput Interfaces (Abingdon). 2015 Apr 3;2(2–3):57–9.
- 63. Cioli N, Holloman A, Crawford C. NeuroBrush: A Competitive, Artistic Multi-Modal BCI Application [Internet]. [cited 2023 Dec 3]. Available from: https://artisticbci.files.wordpress.com/2018/04/nicholas-cioli.pdf
- 64. Won K, Kwon M, Jang S, Ahn M, Jun SC. P300 Speller Performance Predictor Based on RSVP Multi-feature. Front Hum Neurosci. 2019 Jul 30;13:261. pmid:31417382
- 65.
Runco MA. Divergent thinking, creativity, and ideation. Ed. by Kaufman JC, Sternberg RJ Cambridge handbook of creativity. New York: Cambridge University Press; 2010.
- 66. Olteţeanu AM, Zunjani FH. A Visual Remote Associates Test and Its Validation. Front Psychol. 2020 Jan 28;11:26. pmid:32047460
- 67. Mednick SA. The associative basis of the creative process. Psychol Rev. 1962 May;69:220–32. pmid:14472013