Fig 1.
Overview of the method used to continuously generate images from rat LFP.
A. A photo of an experimental setup. B. Detailed illustration of the steps taken to generate image from rat LFP. The recorded LFPs were mapped onto noisy latent vector zT, which was then processed by denoising U-Net to produce a denoised latent vector z. This latent representation of the denoised image was then processed by a latent-to-image decoder to produce image X. C. Schematic of the generation of morphing images from rat LFP. A 512×32 matrix was sampled every 1/30 second with overlaps. The cropped LFPs were then processed through the denoising U-Net and the decoder to generate images at a rate of 30 Hz.
Fig 2.
Examples of morphing images generated from rat LFPs.
A. 3D visualization of latent vector zT using UMAP. Embeddings are from a 5-minute recording of LFPs. Pseudocolor labels indicate the elapsed time since the start of the recording. B. Samples from morphing images over a 10-s duration sampled every 3 frames. C. Structural similarity index of the two consecutive image pairs generated from the Gaussian and LFP latent matrices (* p = 1.50 × 10−198, t1485 = 38.65, nGauss = 486, nLFP = 486, Student’s t-test).
Fig 3.
Temporally spaced sample selection from a 5 min LFP recording.
A. Spatial distribution of selected frames in the embedding space. B. Noisy latent vector zT and the corresponding generated image obtained by the diffusion process and the latent-to-image decoder.
Fig 4.
Comparison of LFP-derived and Gaussian-derived images.
A. Spatial distribution of the latent matrix in the embedding space. Gray represents the baseline latent matrix derived from a Gaussian distribution, while red represents the LFP-mapped latent matrix. B. Variance of each element in the LFP-derived and Gaussian-derived latent matrices. C. Color entropy of the images generated using the Gaussian and LFP latent matrices (* p = 1.74 × 10−110, t1485 = -24.35, nGauss = 1000, nLFP = 487, Student’s t-test). D. Same as C, but for texture entropy (* p = 2.22 × 10−51, t1485 = -15.68, nGauss = 1000, nLFP = 487, Student’s t-test).
Fig 5.
Various styles of generated images.
Morphing images generated from LFPs using different models. From top to bottom: OpenJourney, Stable-Diffusion v1.4, OilPainting, FruitFusion. All are available on Civitai.com.