Work plan and enhancement / 工作计划和用户诉求

[zRzRzRzRzRzRzR]

Tasks that have been identified and scheduled:

• Fine-tuning support for Diffusers version models
• Adaptation for CPU / NPU inference frameworks (e.g., Huawei, Intel devices)
• ComfyUI adaptation work and plugin support

已经明确并排期的任务：

• Diffusers 版本模型微调支持
• CPU/NPU 推理框架适配（例如华为NPU、英特尔设备）
• ComfyUI 适配工作及插件支持

如果你有更多诉求，欢迎在这里提出

[zRzRzRzRzRzRzR]

#182 #191 #47 #84 have similar content, all looking forward to the open-source CogVideoX I2V model. We are conducting research and evaluation.

#111 #186 are similar, both expecting to provide fine-tuning work for VAE. We will try to place it in the fine-tuning version, and there is a probability that it can be adapted in diffusers fine-tuning, but it will consume relatively high resources

[rookiemann]

5b image to video please! I2V would be lovely!

[PR-Ryan]

The 3D VAE model consumes significantly more memory compared to diffusion models, which is severely limiting the batch size for fine-tuning. Any suggestions or optimizations to reduce memory usage would be greatly appreciated.

[zRzRzRzRzRzRzR]

The 3D VAE model consumes significantly more memory compared to diffusion models, which is severely limiting the batch size for fine-tuning. Any suggestions or optimizations to reduce memory usage would be greatly appreciated.

You make a very good point. We will work together with the Diffusers team to modify the fake quantization (fakecp) process in the VAE section to optimize it for lower memory usage. Please give us some time, as we will collaborate with the Diffusers team to develop a version of the model that is fine-tuned specifically for Diffusers, which is expected to save a significant amount of memory.

[KihongK]

First of all, thank you for your excellent work!

The dataset format used SAT way for fine tuning & full training be the same as the format that will be used for fine-tuning Diffusers version models?

+ wrong discord link

[zRzRzRzRzRzRzR]

We are currently completing several tasks

1. Adaptation work for the I2V model, expected to be open-sourced in September
   Detailed tutorial on model fine-tuning, expected to be completed in September

Work that has been completed

1. The model fine-tuned with SAT can be converted to a diffusers model and mounted directly. For specific usage, see here
2. The new Discord invitation link will be merged into the main branch today

[sincerity711]

When will vertical video generation be supported?

[zRzRzRzRzRzRzR]

When will vertical video generation be supported?

The current model cannot generate vertical videos, such as 480x720 resolution. We are working on fine-tuning to reach this capability, but it’s still in progress. Once we have any updates, we will share them as soon as possible.

[zRzRzRzRzRzRzR]

Two related issues working now:

1. Diffusers supports CogVideoX-I2V. This PR has been merged, but the patch has not yet been released.

2. Fine-tuning the Diffusers version of CogVideoX-2B T2V without using the SAT model, running directly under the Diffusers framework. It can run on a single A100 GPU.
   This PR is still under debugging. I am working with members of the Diffusers team to attempt fine-tuning the CogVideoX-5B and I2V models. We will provide a small dataset (a few dozen samples) for this PR, which is sufficient for LoRA fine-tuning CogVideoX.

Many thanks to @a-r-r-o-w for the help with these two tasks!

[JH-Xie]

when will CogVideoX-2B-I2V be released?

[SanGilbert]

1. Various resolution support. Maybe RoPE + resize data to random resolution will achieve this?
2. Control ability of model, more than just text prompt.

[Florenyci]

@zRzRzRzRzRzRzR Many thanks to you and the team! I know fine-tuning vae is not very useful, but I'm curious is there any way I can just fine-tuning decoder part?

[zRzRzRzRzRzRzR]

Our publicly available fine-tuning code is for the fine-tuning of the transformers part, not for vae. We indeed have not updated the training and fine-tuning parts of vae (because I have not received the corresponding permissions either).
Additionally, fine-tuning vae alone seems to have little significance for the overall model effect. If you want to try fine-tuning the diffusers model, all fine-tuning of the transformers module is already in dev, currently it is lora, and it is expected to implement SFT by early October.

[Florenyci]

Our publicly available fine-tuning code is for the fine-tuning of the transformers part, not for vae. We indeed have not updated the training and fine-tuning parts of vae (because I have not received the corresponding permissions either). Additionally, fine-tuning vae alone seems to have little significance for the overall model effect. If you want to try fine-tuning the diffusers model, all fine-tuning of the transformers module is already in dev, currently it is lora, and it is expected to implement SFT by early October.

@zRzRzRzRzRzRzR thank you but actually I'm asking how to fine-tuning VAE decoder, any advice?

[chenshuo20]

Hi @zRzRzRzRzRzRzR, what's your plan about diffuser I2V lora fine-tune code? Thanks!

[alfredplpl]

@zRzRzRzRzRzRzR

Thank you for your great works!

I would like to covert a full-finetuned 2b model weight in sat into a model weight in diffusers.
My full-finetuned model weight in sat is approx. 22GB.
Therefore, I do not convert the model by your conversion code: python ../tools/convert_weight_sat2hf.py.
This model weight may includes transformer and optimizer weight and so on.
Therefore, I tried to extract transformer from the model weight only.
However, something went on.

How can I do it?