Text-to-video generation

The large-scale text-to-image (T2I) diffusion models has achieved unprecedented success, so researchers focus on applying the diffusion models to the field of video generation, and have achieved a series of remarkable research results. While many research [1,4,6,11] bring visual impact to people, but super computing resources make it difficult for individual researchers or small research teams, so researchers try to study how to better achieve text-to-video generation on the basis of lightweight resource utilization. For example, many T2V studies based on Zero-Shot, One-Shot, Few Shot and Training-Free have achieved better generation effects. This paper will also focus on lightweight resource utilization research for smooth editing.

Many major T2V generation models use the T2I model’s merging time module to generate video. Video Diffusion Models [14] extend the basic U-Net architecture of 2D image modeling to 3D space-time by factorizing the spatiotemporal attention module, and realize use the standard equation of diffusion model, which can obtain an effective process of video data generation model. Sora [15] explores large-scale training of generative models on video data, and leverages a transformer architecture that operates on spacetime patches of video and image latent codes, Sora demonstrates the feasibility of scaled video generation models. Imagen Video [16] improves video diffusion models by cascading them and using V-parametric decision making. Make-A-Video [17] begins with learning content description and move features and then extends a diffusion models-based T2I model [18] to T2V using a spatiotemporally factorized diffusion model. Lumiere [19] introduces a Space-Time U-Net architecture based on the pre-trained T2I U-Net to generate the entire temporal duration of the video, which synthefies state-of-the-art diverse and coherent videos. Follow Your Pose [20] also applies the image generation diffusion model to the video generation process, proposing a pose-guided text-to-video generation model to create a character video generation model with editable text and pose management. In addition to the guiding conditions such as pose, edge, and semantic segmentation, T2V tasks are evolving toward multi-conditional guidance. Therefore, in the face of extensive developing T2V generating tasks, research on video smoothing methods will be conducted throughout and is very important.

Text-to-video editing

The video editing based on diffusion models is often guided by text, so as to change the content, style and background layout of the generated video. Dreamix [21] proposed a text-based general appearance and action editing method for realistic scene videos, which achieved good video editing effects, but the model implementation process required relatively high computing resources. Patrick Esser [22] et al. proposed a structure and content perception model, which can modify video through sample images or text guidance, and demonstrated for the first time the control of time consistency by the joint training of image and video data. The experimental results are impressive, but the implementation process requires large-scale text image data resources. FateZero [23] captures the intermediate attention graph in the inversion process to retain the structure and motion information, fuse self-attentions with a blending mask obtained by cross-attention features from the source prompt, and changes the self-attention in the UNet to spatio-temporal attention, so as to realize the change of the video foreground content and the editing of the video style. Rerender A Video [24] presents a novel video-to-video translation framework, which translates the source video into the target video based on the style description of the text. It can be widely used in video style transfer. TokenFlow [25] finds that enhancing the consistency of internal diffusion features between frames can generate videos with temporal consistency during video editing. Video-P2P [26] achieve video editing by introducing cross-attention control and optimizing shared unconditional embedding of video inversion, as well as using different guides for source and editing prompts, and merging their attention. Slicedit [27] runs DDPM sampling using the extracted source video noise space, while injecting the guided conditional attention maps at specific timesteps to achieve video editing. FLATTEN [28] extends the T2I diffusion model and incorporates flow-guided attention into the DDIM sampling and inversion process, resulting in consistent video editing results. With the vigorous development of T2V tasks, T2V editing tasks do not require huge computing resources for the generation process, and can be flexible in the output video style, so T2V editing tasks will also be widely developed.

Latent diffusion models

Diffusion models have recently emerged as new leaders in deep generation models, owing to their high-quality generation effects in image generation tasks. They have excellent performance in many application fields, such as computer vision, NLP (Natural Language Processing), multimodal modeling, medical image reconstruction and so on [29]. Since the original diffusion models still has some shortcomings, such as slow sampling speed, poor maximization likelihood and weak data generalization ability, many optimization studies based on diffusion models have been conducted based on the above problems [30–36].

In this paper, video smoothing is studied on the basis of latent diffusion models [18]. Compared with other space compression methods, latent diffusion models propose a method to perform the diffusion process in latent space, which can reduce the computational complexity and achieve better image generation. Instead of training the model on pixel space, latent diffusion models process the original images through self-coding models, retaining only some features of the important foundation, thus greatly reducing the complexity of the training and sampling phases. latent diffusion models mainly consist of a self-coding model (including an encoder and a decoder) and diffusion operations of the latent space.

Specifically, given an image , H is the height, W is the width, and 3 is the number of color channels (R, G, B). First use an encoder ∈ encode the image into the latent space , where , then use a decoder to reconstruct the image from the latent space , the downsampling factor is f = H ∕ h = W ∕ w. In the latent space, noise prediction network need to be trained to implement the denoising process, is a noisy sample of at timestep t, t = 1 , . . . , T. In the latent diffusion models, a self-coding model is introduced, so the obtained by the encoder can be used to make the model learn in the latent space during training, expressed as:

(1)

where noise ε is added to according to step t to obtain . For the conditional guided image generation process, LDM introduces a domain-specific encoder for pre-processing condition y, where learning can be expressed as:

(2)

Denoising diffusion implicit models (DDIMs)

The denoising diffusion implicit models (DDIMs) [30] are more efficient iterative implicit probability models that build on the denoising diffusion probability models (DDPMs) [37] to enhance sampling speed even further. In DDPMs, the generative process is defined as the reverse of a particular Markovian diffusion process, so the generation steps are continuous and non-skippable. However, DDIMs break the Markovian process through mathematical reasoning and realize multi-step sampling, which improves sampling speed. The non-Markovian process is derived as follows:

Analyze the inference diffusion process indexed by real vectors :

(3)

where is a latent variable, timestep t = 1 , . . . , T, is a data distribution, when t>1 and diffusion hyperparameter sufficiently close to 0, , therefore, the following equation can be obtained:

(4)

According to Bayes’ rule, the following forward process can be derived:

(5)

where the forward process is no longer a Markovian process, because every can depend on , .

The generation process is as follows:

Define a trainable generation process , where each leverages knowledge of . Given a noise observation , , model predict from , which is a prediction of given :

(6)

The generation process can be defined as follows:

(7)

According to the generation process, the implicit model equation of denoising diffusion is further obtained as follows:

(8)

By setting to 0, the sampling process is deterministic, the latent variables are consistent, and the sampling steps are reduced. The following equation is obtained:

(9)

Research on the interframe noise relation

The training and sampling process of the diffusion models are carried out around the noise. As the core content of the diffusion models, noise brings infinite possibilities to researchers. For the traditional T2I model, the process of add noise or denoising is only for static images, but the video generated by the T2V model is composed of a series of static images, so the process of T2V generation will increase resources according to the length and resolution of the video. We often consider the noise relation of each step in the process of adding noise and denoising for T2I, so what is the relation between the noise of different frames in T2V, and how does it affect the consistency of the video? At present, relatively few studies [12,13] have been carried out on the noise relationship between video frames, and our research work is carried out in this aspect.

For a single frame image, in the forward process, noise is progressively added to the observation , and is obtained after t steps, for any step t, can be expressed as a linear combination of and random noise variable :

(10)

In the sampling process, after progressively denoise from , the observation is finally obtained, then can also be expressed as a linear combination of and estimated noise variable :

(11)

Discuss the relation between the noise of different frames in T2V. In the forward process, , and represent the original observations corresponding to any adjacent three frames, n is the frame index, after t steps, random noise variables , , are added to the original observations, then , and are obtained.

In the sampling process, after progressively denoise from the adjacent three frames, , and are obtained; , , respectively represent the estimated noise variable of the three frames during the denoising process. The diffusion hyperparameter values for the adjacent three frames in the same step are equal, so we can express any adjacent three frames as follows:

(12)

(13)

(14)

For forward and sampling processes, the ultimate prediction objective is to reach or and construct loss from there, so according to the prediction objective, we can construct the following equation:

(15)

(16)

Transform the two above equations as follows:

(17)

(18)

As shown in the above equations, for any adjacent three frames, when the steps are same, the increased noise variable and the estimated noise variable have a certain constraint relation, which will certainly affect the consistency of each frame.

INR Smooth: Video smoothing editing method for T2V training models

In order to improve the alignment of text and video and the consistency of different frames, the above results of the interframe noise relation can be applied to the T2V training models. Fig 3 depicts the proposed video smoothing editing framework for T2V training models.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. INR Smooth: Video smoothing editing framework for T2V training models.

The green arrow indicates the direction from the first frame to the last frame to build the interframe noise constraint, and the blue arrow indicates the opposite process.

https://doi.org/10.1371/journal.pone.0321193.g003

As shown in Fig 3, construct interframe noise constraints from both directions simultaneously, then obtain the mean square error (MSE) of the interframe noise variables:

(19)

(20)

where n indexes the corresponding video frame, N is the total number of frames of the video, MSE(1) represents the interframe noise constraint from the last frame to the first, MSE(2) represents the interframe noise constraint from the first frame to the last. The video smoothing loss function of the training model is constructed as follows:

(21)

where and are the weights of the constraint relation between frames in two directions respectively, and the values of and can be set to 0.5 respectively, which can be adjusted appropriately according to the characteristics of experimental objects during the experiment. Thus, the overall loss function of the T2V training model can be obtained as follows:

(22)

where λ is the weight of video smoothing loss, and is the original loss function of the diffusion models. The pseudocode of the video smoothing editing method for the T2V training models is shown in Algorithm 1.

INR Smooth: Video smoothing editing method for T2V training-free models

On the basis of large-scale diffusion models, more and more researchers begin to explore lightweight optimization research that can be achieved only through model fine-tuning without adding additional computing resources and training. The training-free T2V generating process uses fewer computing resources and produces results faster. However, due to the lack of training and learning of the study samples, it is often difficult to control the smooth performance of the generated video. To address such research challenges, DDIM Inversion is introduced based on the above interframe noise relation, and control the inversion noise to improve the smoothness of the generated video.

The purpose of DDIM Inversion is to reverse the sampling process, resulting in a noisy implicit representation that serves as the starting for the sampling process, and the generated image will be close to the source image. The whole process can be modeled as the process of adding random noise to gets , then using as the starting point for DDIM sampling to generate and predict inversion noise (represented by ), and then gradually sampling to get the final generated image. From , we can generate a sample from a sample via:

(23)

Algorithm 1: Video smoothing editing method for T2V training models.

Input:   

Output:   

1:   Hyperparameters: ,

2:   Diffusion process

3:   for t = 1 , 2 , . . . , T do

4:   

5:   end for

6:   Denoising process

7:   for t = T , ( T − 1 ) , . . . , 1 do

8:   

9:   end for

10:   Deriving the noise relationship between video frames

11:   from

12:   

13:   

14:   

15:   to

16:   

17:   

18:   Constructing smoothing loss

19:   

20:   

21:   

22:   return

According to the DDIM Inversion process and the interframe noise relation studied in this paper, a video smoothing editing framework for training-free models is constructed, as shown in Fig 4.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. INR Smooth: Video smoothing editing framework for training-free models.

Different from the smoothing editing method of the training models, inversion noise is used to replace the estimated noise for the training-free models.

https://doi.org/10.1371/journal.pone.0321193.g004

As shown, random noise is added in the diffusion forward process, and inversion noise is removed by the DDIM Inversion sampling process, represents the inversion noise from the last frame to the first, represents the inversion noise from the first frame to the last, a and b represent the step, n indexes the corresponding video frame. The noise constraint from both ends of the video is conducive to improving the consistency between video frames. According to Eqs 17 and 18 and DDIM Inversion process, and can be constructed as follows:

(24)

(25)

For training-free models, inversion noise is used to replace the estimated noise in the method for training models. Inversion noise has a value starting from the second-to-last frame of the video, inversion noise has a value starting from the second frame of the video, they both depend on the added random noise and the removed inversion noise of the adjacent frame. Weighting the inversion noise in both directions to obtain the inversion noise of the NTH frame:

(26)

where is inverse noise weight, by adjusting the inverse noise weight, the interframe consistency and text alignment of the video generated by the training-free T2V models can be improved. The pseudocode of the video smoothing editing method for the T2V training-free models is shown in Algorithm 2.

Algorithm 2. Video smoothing editing method for T2V training-free models.

Input:   

Output:   

1:   Hyperparameters:

2:   DDIM for inversion latents

3:   for t = 1 , 2 , . . . , T do

4:   

5:   end for

6:   Deriving the noise relationship between frames by inversion noise and random noise

7:   

8:   

9:   Constructing the inversion noise

10:   

11:   return

Implementation details

The proposed two smoothing editing methods can be applied to T2V baselines which require training or inversion-based baselines which are training-free, respectively. The implementation is based on each baseline’s official codes, original data frame rate, frame number, learning rate, batch size and resolution requirements. The comparison experiments with baselines are representative. In the inference phase, we use DDIM sampler with classifier-free guidance. As the internal parameters of different baselines are obviously different, in order to adapt to the optimization of different baselines, it is necessary to adjust the weights according to specific baselines. So the smoothing loss weight λ and inverse noise weight can be set according to the concrete experiment. The number of sampling steps in the experiment is 50. All experiments are conducted on an NVIDIA RTX 4090 GPU.

Dataset and metrics

This study uses official implementation datasets of each baseline for smoothing tests, which mainly include the DAVIS dataset [38]. We evaluate the quality of the generated video from two aspects: text alignment and frame consistency. According to the previous research [8,9,22,23,26], we select CLIP (T) (text alignment evaluation), CLIP (F) (frame consistency evaluation), and Optical flow [39] as the corresponding evaluation metrics. CLIP (T) refers to the average cosine similarity of text and image embedded across all frames of the generated video. CLIP (F) refers to the average cosine similarity between all consecutive frame pairs of the generated video. Optical flow computes the motion vector of each pixel between two frames using image information from succeeding frames. For the smoothing optimization effect of the baselines, the source videos provided by the official implementation process will be used as the main evaluation and comparison.

Baselines

Based on the generated data from the state-of-the-art baselines: Tune-A-Video [8], ControlVideo [9], FateZero [23] and Video-P2P [26], this paper carries out smoothing editing experiments. Smoothing editing methods include Smooth Video [13] and the proposed methods. At the same time, two advanced methods Slicedit [27] and STEM [40] are selected for extended comparison. Among them, Smooth Video [13] applies a video frame noise prediction method, which is in contrast with the proposed methods. Slicedit [27] and STEM [40] apply DDPM inversion and Spatial-Temporal Expectation-Maximization inversion, respectively, in contrast to the DDIM inversion applied in this paper.

Smoothing results based on training baselines

Comparison with Tune-A-Video. The experiments are based on the Stable Diffusion 1.4, with the guidance_scale set at 12.5. The experiments compare the baseline methods Tune-A-Video [8], Slicedit [27], and STEM [40], the smoothing editing method Smooth Video [13], and the proposed video smoothing editing method (Ours(T)) which is based on the training models. The smoothing loss weight of Ours(T) is set to 2.0.

As shown in Fig 5, each row in the figure is the middle 8 frames with obvious changes selected from the generated video. As shown, the video generated by Tune-A-Video can maintain the semantic alignment with the text, but there are still obvious flickers between frames (as shown in the yellow box of Tune-A-Video). With the introduction of Smooth Video, the smoothness of Tune-A-Video is improved, but the video results do not present a cartoon style and thus are not aligned with the text semantics. After introducing the proposed smoothing method (Ours(T)), the generated video exhibits better frame consistency, while the generated video has better quality and obvious cartoon style. The generated results of Slicedit do not present cartoon style and do not achieve text alignment. The STEM generation effect is remarkable, the text is well aligned, but there are slightly unsmooth details (as shown in the yellow box of STEM).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. Comparison of smoothing editing results of the cartoon style video generated by Tune-A-Video.

https://doi.org/10.1371/journal.pone.0321193.g005

Fig 6 shows the experimental results of the first 8 frames selected from the generated video. As shown by the red box of Tune-A-Video, there are obvious background differences between the frames, resulting in flickering and unsmooth video. Smooth Video improves the smoothness of Tune-A-Video, but there are obvious generation anomalies in the first three frames (as shown in the yellow boxes of Smooth Video). Similarly, Slicedit does not achieve text alignment. The generation results of STEM are abnormal, as shown by the yellow boxes, additional cars that do not exist in the source video appear. Compared with the above methods, the proposed method (Ours(T)) can make the background texture of the generated video clearer and can improve the interframe consistency. Meanwhile the proposed method achieves text alignment, and preserves the temporal consistency with the source video.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Comparison of smoothing editing results of outdoor scene video generated by Tune-A-Video.

https://doi.org/10.1371/journal.pone.0321193.g006

Comparison with Control Video. The experimental results of the baseline method ControlVideo [9], the comparison method Smooth Video [13] and Ours(T) are compared. The smoothing loss weight of Ours(T) is set to 0.2. Fig 7 shows the first 7 frames results of the experiments. As the figure shows, while ControlVideo can generate text-aligned videos, there are still shortcomings in the shadows and leg structure of the person in the video (as shown in the yellow boxes). Smooth video, improved character quality in some frames, but due to over-smoothing, the background texture disappears, some action poses are distorted, and cannot be aligned with the source video (as shown in the red boxes). In contrast, the proposed method aligns better with the background, color, and texture of the source video, while improving the person’s shadows and poses, and maintaining temporal consistency with the source video.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Comparison of smoothing editing results of human pose guided video generated by ControlVideo.

https://doi.org/10.1371/journal.pone.0321193.g007

Smoothing results based on training-free baselines

Comparison with FateZero. Due to FateZero [23]’s advantages in video content style and structural shape editing, the generated videos have better temporal consistency and smoothness. Under the condition that guidance_scale is set to 7.5, an style transfer experimental comparison is conducted between FateZero, Slicedit [27], STEM [40] and the proposed video smoothing editing method (Ours(TF)) which is based on the training-free models. The inverse noise weight of Ours(TF) is set to 1.0.

Fig 8 shows that the first three frames of FateZero have a strong stylized effect, whereas the last three frames have a weak stylized effect, displaying inconsistent stylized effect. The generation effect of Slicedit is close to the source video and does not achieve the Van Gogh style. The artistic effect of STEM needs to be strengthened, and the background does not realize the corresponding artistic style. In terms of style editing, the proposed method (Ours(TF)) demonstrates overall consistency, and the texture artistic effect of leaves and petals in the last three frames is consistent with that of the first three frames, and the background of each frame also shows consistency with the style (as shown in Fig 9). Through comparison, it is found that the proposed method not only maintains the alignment with the text, but also has obvious advantages in terms of overall artistic style, color, and texture presentation.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. Comparison of smoothing editing results of the stylized video generated by FateZero.

https://doi.org/10.1371/journal.pone.0321193.g008

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Comparison of video background texture stylization effect.

https://doi.org/10.1371/journal.pone.0321193.g009

Comparison with Video-P2P. Video-P2P [26] can achieve better application in generating new videos by changing the guide text and attention weighting, and has good performance in maintaining frame consistency. In this paper, Video-P2P is selected to compare with Slicedit [27], STEM [40], and Ours(TF). The inversion noise weight of Ours(TF) is set to 0.5.

As shown in Fig 10, the results of Video-P2P show many inconsistencies, such as inconsistent background and texture (as shown in the red boxes of Video-P2P), inconsistent structure and shape of rabbit head (as shown in the yellow boxes of Video-P2P), and inconsistent color and shape of rabbit feet (as shown in the blue boxes of Video-P2P). Inconsistent backgrounds and textures similarly appear in the STEM results. In contrast, Slicedit and the proposed method Ours(TF) achieve text alignment, however, the origami effect of Slicedit needs to be strengthened.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 10. Comparison of smoothing editing results of semantic style video generated by Video-P2P.

https://doi.org/10.1371/journal.pone.0321193.g010

Quantitative results

In this section, the training and training-free video smoothing methods proposed in this paper are quantitatively compared with the following T2V baselines, including Tune-A-Video [8], ControlVideo [9], Smooth Video [13], FateZero [23], Video-P2P [26], Slicedit [27], and STEM [40]. Figs 4, 5, and 7 show that the results of Slicedit are almost identical to the source video and are not aligned with the text. Therefore, Slicedit is excluded from these quantitative comparisons. Tables 1 and 2 display the results of the quantitative comparison. CLIP (T) represents the degree of alignment between the generated video and the text, so the higher the value, the better the alignment. CLIP (F) further indicates the frame consistency of the generated video, so higher value indicates the better smoothness. The purpose of optical flow metric is to compute the motion vector of each pixel between two frames using image information from succeeding frames in order to determine the video’s smoothness.

Download:

• PNG
  larger image
• TIFF
  original image

Table 1. Quantitative comparison with training baselines.

https://doi.org/10.1371/journal.pone.0321193.t001

Download:

• PNG
  larger image
• TIFF
  original image

Table 2. Quantitative comparison with training-free baselines.

https://doi.org/10.1371/journal.pone.0321193.t002

As shown in Table 1, through quantitative comparison, the proposed video smoothing editing method (Ours(T)) which is based on the training models, is significantly superior to the compared baselines in the comprehensive scores of the three metrics: text alignment CLIP (T), frame consistency CLIP (F), and Optical flow. In the experiments based on ControlVideo, the shadow of jeans is smoothed out due to the over-smoothing of Smooth Video, so the CLIP (T) and optical flow metrics of Smooth Video are improved, but the experimental results of the proposed method (Ours(T)) are more consistent with the background texture and light shadow of the source video.

Table 2 shows that the proposed video smoothing editing method (Ours(TF)), based on training-free models, significantly improves text alignment, frame consistency, and temporal consistency. It is worth noting that when the proposed method (Ours(TF)) is used to smooth the stylized video generated by FateZero, the art style texture effect improves, resulting in a higher CLIP (T); however, since the background of each frame also implements the artistic style, the CLIP(F) and optical flow metrics of the proposed method are relatively poor compared with the solid color background of FateZero and STEM.

The preceding studies demonstrate that the proposed methods have significant effects in text-to-video editing. We extensively evaluate the training and inference time of the training and training-free methods. For example, Tune-A-Video takes about 12 minutes to train on a 24-frame video with 500 epochs and requires 15.8G of memory resources. Compared with Tune-A-Video, the proposed method (Ours(T)) requires the same training time as Tune-A-Video, and occupies 18.8G of memory resources. It can be seen that the proposed method increases the memory resources by about 128M per frame while editing the training baseline, but does not add extra computation time. The improvement in text alignment, temporal consistency, and fidelity makes the additional memory resources worthwhile. Training-free method FateZero takes about 1 minute 32 seconds to infer 8 frames of video and requires 9.7 to 11.4G of memory resources. Our method (Ours(TF)) takes the same computational cost as FateZero. It can be seen that the computational resources required to edit a training-free model mainly depend on the baseline model. For various text inputs such as scene, dress, style, and material, compared with the state-of-the-art methods, the proposed method can generate consistent video content that conforms to the semantics of the text. For the additional guidance conditions, such as human pose, the proposed method can also achieve alignment with the source video and guide pose. The above stable generation performance further verifies the robustness of the proposed method.

Ablation study

We conduct ablation experiments by removing the forward noise constraint (Forward) and the backward noise constraint (Backward) from our frame work, respectively, as shown in Fig 11. Ablation experiments reveal that using the forward noise constraint can improve text alignment and local properties. For example, the material of the swan head and neck is improved, and the structure shape of the swan tail is improved. The use of the backward noise constraint also improves the text alignment, the material of the swan head and neck is improved, but the shape of the swan tail does not change significantly. By adding noise constraints in both directions at the same time (Double), the overall material and structure shape achieve more significant improvement, both text alignment and temporal consistency are enhanced.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 11. Ablation study. We ablate the following components: (Forward) noise constraint from the last frame to the first, (Backward) noise constraint from the first frame to the last.

https://doi.org/10.1371/journal.pone.0321193.g011