-
Notifications
You must be signed in to change notification settings - Fork 22
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
add optional external post-processing toolchain, add brief chinese re…
…adme
- Loading branch information
Showing
8 changed files
with
340 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # meta | ||
| .vscode/ | ||
| __pycache__/ | ||
|
|
||
| # third party tools | ||
| tools/* | ||
| !tools/README.md | ||
| !tools/*.cmd | ||
| !tools/busybox.exe |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| # 提示词跃迁 | ||
|
|
||
| 在模型隐层旅行以制作伪动画，项目 AUTOMATIC1111/stable-diffusion-webui 的插件。 | ||
|
|
||
| ---- | ||
|
|
||
| 对语言理解模型 CLIP 的输出进行插值，从而实现多条提示词之间的语义过渡，产生看似连续的图像序列，或者说伪动画。😀 | ||
|
|
||
| ⚠ 我们成立了插件反馈 QQ 群: 616795645 (赤狐屿)，欢迎出建议、意见、报告bug等 (w | ||
|
|
||
| ℹ 实话不说，我想有可能通过这个来做ppt童话绘本<del>甚至本子</del>…… | ||
| ℹ 聪明的用法：先手工盲搜两张好看的图 (只有提示词差异)，然后再尝试在其间跃迁 :lolipop: | ||
|
|
||
|
|
||
| ### 使用方法 & 它如何工作 | ||
|
|
||
| - 在提示词/负向提示词框里输入**多行**文本，每一行被称作一个**阶段** | ||
| - 逐帧生成图像，在每个阶段内，所使用的提示词向量是经过插值运算的 | ||
| - 为了保证某种连续性，所有其他参数将被固定 | ||
| - 虽然所有图的主随机数种子将被统一固定，但你仍然可以启用 `subseed` 去增加随机性 | ||
| - 导出视频！ | ||
| - 使用额外的 [后处理流程](#post-processing-pipeline) 可以获得更好的画质和流畅度 👌 | ||
|
|
||
|
|
||
| ### 参数选项 | ||
|
|
||
| - 提示词: (多行文本) | ||
| - 反向提示词: (多行文本) | ||
| - 就是提示词和反向提示词的输入框，但是你必须输入多行文本，每一行是一个阶段 | ||
| - 如果提示词和反向提示词的阶段数量不一致，少的那一方会被重复到对齐多的一方 | ||
| - 插帧数/steps: (整数，或者逗号分隔的多个整数) | ||
| - 每个阶段之间插帧的数量 | ||
| - 若为单个整数，每个阶段使用相同的插帧数量 | ||
| - 若为西文逗号分隔的多个整数，每个阶段使用不同的插帧数量，比如有4个阶段则可给出3个独立步数：`12, 24, 36` | ||
| - 起源/genesis: (选项), 每张图像的内容先验 | ||
| - `固定/fixed`: 在 txt2img 流程中，始终从高斯噪声开始降噪；在 img2img 流程中，始终从给定的参考图开始降噪 | ||
| - `连续/successive`: 从上一帧的内容开始降噪 (这会导致 txt2img 流程从第二步开始强制转为 img2img 流程) | ||
| - `胚胎/embryo`: 从某个已部分降噪的公共先祖胚胎开始降噪，参考 [=> 原理](https://replicate.com/andreasjansson/stable-diffusion-animation#readme) | ||
| - (该功能为实验性质) 只支持两个阶段跃迁，并且不能为逆向提示词插值 :( | ||
| - 起源的额外参数 | ||
| - 降噪强度: (浮点数), 在 img2img 流程中所用的降噪强度 (仅对 `连续/successive` 模式) | ||
| - 胚胎步数: (整数或浮点数), 产生公共胚胎的预降噪步数 (仅对 `胚胎/embryo` 模式) | ||
| - 如果 >= 1，解释为采样步数 | ||
| - 如果 < 1，解释为占总采样步数的比例 | ||
| - 视频相关 | ||
| - 帧率/fps: (浮点数), 导出视频的帧率，设置为 `0` 将禁用导出 | ||
| - 文件格式/fmt: (选项), 导出视频的文件格式 | ||
| - 首尾填充/pad: (整数), 重复首尾帧 `N` 次以留出一段入场/退场时间 | ||
| - 帧选择器/pick: (切片器), 使用 [Python切片语法](https://www.pythoncentral.io/how-to-slice-listsarrays-and-tuples-in-python) 精心选择所需要导出的帧，注意切片发生在填充之前 (例如：设为 `::2` 将只使用偶数帧 , 设为 `:-1` 将去除最后一帧) | ||
| - 调试开关: (逻辑值) | ||
| - 是否在控制台显示详细日志 | ||
|
|
||
|
|
||
| ### 后处理流程 | ||
|
|
||
| 单凭CLIP模型自身能实现语义插值就已经到达能力天花板了，但我们距离高清丝滑的动画还差两步： **图像超分辨率** 和 **视频插帧**。 | ||
| ⚠ 多媒体数据的处理是非常消耗资源的，我们不能指望 webui 去做这件事。实际上，我们将其从宿主和插件中分离，出来作为一个可选的外部工具。 😃 | ||
|
|
||
| #### 安装依赖 | ||
|
|
||
| ⚪ 自动安装 | ||
|
|
||
| - 运行 `tools/install.cmd` | ||
| - 如果遇到诸如 `访问被拒绝` 之类的错误，多次运行直到提示 `Done!` 无错误退出 😂 | ||
| - 你将安装好 [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan), [RIFE](https://github.com/nihui/rife-ncnn-vulkan), [FFmpeg](https://ffmpeg.org/) 这三个组件在 [tools](tools) 目录下 | ||
|
|
||
| ⚪ 手动安装 | ||
|
|
||
| - 参照 [README.md](README.md#post-processing-pipeline) | ||
| - 我寻思你既然都想着手动安装了，也不至于不肯咬一口英文罢…… 🤔 | ||
|
|
||
| #### 运行任务 | ||
|
|
||
| - 检查 [postprocess.cmd](postprocess.cmd) 中的默认参数 | ||
| - 你有两种方式启动后处理任务 😃 | ||
| - 从命令行运行 `postprocess.cmd path/to/<image_folder>` | ||
| - 鼠标拖拽任意图片文件夹到 `postprocess.cmd` 的文件图标上然后释放 | ||
|
|
||
| ℹ 任务完成后，资源浏览器将被自动打开并定位到导出的 `synth.mp4` 文件~ | ||
|
|
||
|
|
||
| 插件直出和加入后处理对比 (配置为 `img2img-fixed-ddim`)： | ||
|
|
||
| | 插件直出 | 加入后处理 | | ||
| | :-: | :-: | | ||
| |  |  | | ||
|
|
||
|
|
||
| ---- | ||
| by Armit | ||
| 2023/01/20 |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| @REM Test script for post-process pipeline | ||
| @ECHO OFF | ||
| SETLOCAL | ||
|
|
||
| REM ================================================= | ||
| REM Real-ESRGAN model ckpt | ||
| SET RESR_MODEL=realesr-animevideov3 | ||
| REM image upscale rate, must choose from [2, 3, 4] | ||
| SET RESR_UPSCALE=2 | ||
| REM RIFE model ckpt | ||
| SET RIFE_MODEL=rife-v4 | ||
| REM interpolated frame count, set 0 to simply Nx2 | ||
| SET RIFE_INTERP=0 | ||
| REM rendered video fps, higher value requires more interpolations | ||
| SET FPS=20 | ||
| REM ================================================= | ||
|
|
||
|
|
||
| IF NOT EXIST "%~1" ( | ||
| ECHO Usage: %~nx0 path/to/^<image_folder^> [-k] | ||
| ECHO -k keep cache data for debug | ||
| PAUSE | ||
| GOTO :EOF | ||
| ) | ||
|
|
||
| SET RESR_HOME=%~dp0tools\realesrgan-ncnn-vulkan | ||
| SET RIFE_HOME=%~dp0tools\rife-ncnn-vulkan | ||
| SET FFMPEG_HOME=%~dp0tools\ffmpeg | ||
|
|
||
| SET RESR_BIN=realesrgan-ncnn-vulkan.exe | ||
| SET RIFE_BIN=rife-ncnn-vulkan.exe | ||
| SET FFMPEG_BIN=ffmpeg.exe | ||
|
|
||
| PATH %RESR_HOME%;%PATH% | ||
| PATH %RIFE_HOME%;%PATH% | ||
| PATH %FFMPEG_HOME%\bin;%FFMPEG_HOME%;%PATH% | ||
|
|
||
| SET IMAGE_FOLDER=%~1 | ||
| SET RESR_FOLDER="%IMAGE_FOLDER%\resr" | ||
| SET RIFE_FOLDER="%IMAGE_FOLDER%\rife" | ||
| SET OUT_FILE="%IMAGE_FOLDER%\synth.mp4" | ||
|
|
||
| ECHO ================================================== | ||
|
|
||
| ECHO [1/5] image super-resolution | ||
| IF EXIST %RESR_FOLDER% GOTO skip_resr | ||
| MKDIR %RESR_FOLDER% | ||
| %RESR_BIN% -v -s %RESR_UPSCALE% -n %RESR_MODEL% -i %IMAGE_FOLDER% -o %RESR_FOLDER% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| :skip_resr | ||
| ECHO ================================================== | ||
|
|
||
| ECHO [2/5] video frame-interpolation | ||
| IF EXIST %RIFE_FOLDER% GOTO skip_rife | ||
| MKDIR %RIFE_FOLDER% | ||
| SET NFRAMES=%RESR_FOLDER% | ||
|
|
||
| %RIFE_BIN% -v -n %RIFE_INTERP% -m %RIFE_MODEL% -i %RESR_FOLDER% -o %RIFE_FOLDER% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| :skip_rife | ||
| ECHO ================================================== | ||
|
|
||
| ECHO [3/5] render video | ||
| %FFMPEG_BIN% -y -framerate %FPS% -i %RIFE_FOLDER%\%%08d.png -crf 20 -c:v libx264 -pix_fmt yuv420p %OUT_FILE% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| ECHO ================================================== | ||
|
|
||
| IF NOT "%2"=="-k" ( | ||
| RMDIR /S /Q %RESR_FOLDER% | ||
| RMDIR /S /Q %RIFE_FOLDER% | ||
| ) | ||
|
|
||
| explorer.exe /e,/select,%OUT_FILE% | ||
|
|
||
| ECHO ^>^> Done! | ||
| ECHO. | ||
| GOTO :eof | ||
|
|
||
| :die | ||
| ECHO ^<^< errorlevel: %ERRORLEVEL% | ||
|
|
||
| :eof |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| @REM Auto download and setup post-process tools | ||
| @ECHO OFF | ||
| SETLOCAL | ||
|
|
||
| TITLE Install tools for post-process... | ||
|
|
||
| SET CURL_BIN=curl.exe -L -C - | ||
| SET UNZIP_BIN=busybox.exe unzip | ||
|
|
||
| SET RESR_URL=https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesrgan-ncnn-vulkan-20220424-windows.zip | ||
| SET RESR_ZIP=realesrgan-ncnn-vulkan.zip | ||
| SET RESR_DIR=realesrgan-ncnn-vulkan | ||
|
|
||
| SET RIFE_URL=https://github.com/nihui/rife-ncnn-vulkan/releases/download/20221029/rife-ncnn-vulkan-20221029-windows.zip | ||
| SET RIFE_ZIP=rife-ncnn-vulkan.zip | ||
| SET RIFE_DIR=rife-ncnn-vulkan | ||
| SET RIFE_RDIR=rife-ncnn-vulkan-20221029-windows | ||
|
|
||
| SET FFMPEG_URL=https://github.com/GyanD/codexffmpeg/releases/download/5.1.2/ffmpeg-5.1.2-full_build-shared.zip | ||
| SET FFMPEG_ZIP=ffmpeg.zip | ||
| SET FFMPEG_DIR=ffmpeg | ||
| SET FFMPEG_RDIR=ffmpeg-5.1.2-full_build-shared | ||
|
|
||
| SET DOWNLOAD_DIR=.download | ||
| IF NOT EXIST %~dp0%DOWNLOAD_DIR% MKDIR %~dp0%DOWNLOAD_DIR% | ||
| ATTRIB +H %~dp0%DOWNLOAD_DIR% | ||
|
|
||
| ECHO ================================================== | ||
|
|
||
| ECHO [1/3] install Real-ESRGAN | ||
| IF EXIST %~dp0%RESR_DIR% GOTO skip_resr | ||
| IF EXIST %DOWNLOAD_DIR%\%RESR_ZIP% GOTO skip_dl_resr | ||
| ECHO ^>^> download from %RESR_URL% | ||
| %CURL_BIN% %RESR_URL% -o %DOWNLOAD_DIR%\%RESR_ZIP% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| :skip_dl_resr | ||
| ECHO ^>^> uzip %RESR_ZIP% | ||
| MKDIR %~dp0%RESR_DIR% | ||
| %UNZIP_BIN% %DOWNLOAD_DIR%\%RESR_ZIP% -d %~dp0%RESR_DIR% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| :skip_resr | ||
| ECHO ================================================== | ||
|
|
||
| ECHO [2/3] install RIFE | ||
| IF EXIST %~dp0%RIFE_DIR% GOTO skip_rife | ||
| IF EXIST %DOWNLOAD_DIR%\%RIFE_ZIP% GOTO skip_dl_rife | ||
| ECHO ^>^> download from %RIFE_URL% | ||
| %CURL_BIN% %RIFE_URL% -o %DOWNLOAD_DIR%\%RIFE_ZIP% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| :skip_dl_rife | ||
| ECHO ^>^> uzip %RIFE_ZIP% | ||
| %UNZIP_BIN% %DOWNLOAD_DIR%\%RIFE_ZIP% -d %~dp0 | ||
| IF ERRORLEVEL 1 GOTO die | ||
| RENAME %~dp0%RIFE_RDIR% %RIFE_DIR% | ||
| :skip_rife | ||
| ECHO ================================================== | ||
|
|
||
| ECHO [3/3] install FFmpeg | ||
| IF EXIST %~dp0%FFMPEG_DIR% GOTO skip_ffmpeg | ||
| IF EXIST %DOWNLOAD_DIR%\%FFMPEG_ZIP% GOTO skip_dl_ffmpeg | ||
| ECHO ^>^> download from %FFMPEG_URL% | ||
| %CURL_BIN% %FFMPEG_URL% -o %DOWNLOAD_DIR%\%FFMPEG_ZIP% | ||
| IF ERRORLEVEL 1 GOTO die | ||
| :skip_dl_ffmpeg | ||
| ECHO ^>^> uzip %FFMPEG_ZIP% | ||
| %UNZIP_BIN% %DOWNLOAD_DIR%\%FFMPEG_ZIP% -d %~dp0 | ||
| IF ERRORLEVEL 1 GOTO die | ||
| RENAME %~dp0%FFMPEG_RDIR% %FFMPEG_DIR% | ||
| :skip_ffmpeg | ||
| ECHO ================================================== | ||
|
|
||
| ECHO ^>^> Done! | ||
| ECHO. | ||
|
|
||
| ATTRIB -H %~dp0%DOWNLOAD_DIR% | ||
| RMDIR /S /Q %~dp0%DOWNLOAD_DIR% | ||
|
|
||
| PAUSE | ||
| GOTO :eof | ||
|
|
||
| :die | ||
| ECHO ^<^< errorlevel: %ERRORLEVEL% | ||
|
|
||
| :eof |
Oops, something went wrong.