 |
|---|
- [2023.10.29] release Mask_Inpaint
Fooocus-ControlNet-SDXL is a ⭐free⭐ image generating software (based on Fooocus , ControlNet ,👉SDXL , IP-Adapter , etc.).
Fooocus-ControlNet-SDXL adds more control to the original Fooocus software.
| control | status 🚀 | show case |
|---|---|---|
| image prompt | ✅ provided by fooocus | ip |
| canny | ✅provided by fooocus | canny |
| cpds | ✅ provided by fooocus | |
| depth | ✅ provided by Fooocus-ControlNet-SDXL | depth |
| pose(body,hand,face) | ✅ provided by Fooocus-ControlNet-SDXL | pose |
| recolor | ✅ provided by Fooocus-ControlNet-SDXL | recolor |
| sketch | ✅ provided by Fooocus-ControlNet-SDXL | sketch |
| segmentation | 📍 todo | |
| pose(only body) | 📍 todo | |
| pose(only hand) | 📍 todo | |
| pose(only face) | 📍 todo | |
| pose(body+hand) | 📍 todo | |
| pose(body+face) | 📍 todo | |
| Scribble | 📍 todo | |
| Soft Edges | 📍 todo | |
| Linear | 📍 todo | |
| Anime Linear | 📍 todo | |
| Tile | 📍 todo | |
| relighing | 📍 todo | |
| mask InPaint | ✅ provided by Fooocus-ControlNet-SDXL | mask |
| newBackground | 📍 todo | |
| etc | 📍 todo |
- Fooocus is an excellent SDXL-based software, which provides excellent generation effects based on the simplicity of liking midjourney, while being free as stable diffusion.
- FooocusControl inherits the core design concepts of fooocus, in order to minimize the learning threshold, FooocusControl has the same UI interface as fooocus (only in the Input Image/Image Prompt/advance to add more options).
- FooocusControl does all the complicated stuff behind the scenes, such as model downloading, loading, registration, image preprocessing, etc. Users don't need to bother with any of this at all, you just need to check the desired image control method.
- Fooocus-ControlNet-SDXL simplifies the way fooocus integrates with controlnet by simply defining pre-processing and adding configuration files.
- If you are a developer with your own unique controlnet model , with Fooocus-ControlNet-SDXL , you can easily integrate it into fooocus .
- In addition to controlnet, FooocusControl plans to continue to integrate ip-adapter and other models to further provide users with more control methods.
- Free software usually encounters a lot of installation and use of the problem, such as 😞 network problems caused by the model file that can not be downloaded and updated 😞, 😞a variety of headaches gpu driver😞, 😞plug-ins lack of dependent libraries and other issues😞. These are very annoying invisible thresholds. fooocusControl is committed to solving these problems.
- FooocusControl has no external dependencies, all the dependencies are built into the software.
- FooocusControl provides both online(light) and offline(massive) download methods, the offline(massive) version will download all the models built into the software, which is very friendly to users who need to run offline, or those who have a bad network.
- Outside the windows platform, FooocusControl will try to use docker and other technologies to further simplify the user installation, to avoid various gpu and dependency problems to the greatest extent possible.
- FooocusControl will add multiple download sources for each model (coming soon) to prevent models from being downloaded or updated due to network problems.
- FooocusControl将会针对每一个模型添加多个下载源(即将上线)，防止因网络问题导致模型无法下载或者更新..
- windows平台上offline下载方式基本避免网络问题
- 中文翻译(即将上线)
- 代码码云镜像(https://gitee.com/fenneishi/Fooocus-ControlNet-SDXL)
Using a sketch image as a prompt input to👉👉👉generate an image incorporating sketch elements.Awesome!!! Can be used for anything now!
Using an image as a prompt input➕one-sentence description👉👉👉A perfect image.
Background image controls the backgroundposture image➕ controls the pose➕one-sentence description👉👉👉You can achieve any 3D result with the background and pose you desire.
Picture as a prompt, AI extracts 3D wireframe information from the image➕one-sentence description👉👉👉A picture that perfectly aligns with 3D wireframe information.
Picture as a prompt, AI extracts 3D information from the image ➕ one-sentence description👉👉👉A perfectly 3D-informed image.
:face
Using a facial pose from an image as a prompt to control input👉👉👉it generates an image with a specific facial pose consistent with it. This is perfect for making model images and design illustrations, it's simply unbeatable!
Using an image as a prompt input to👉👉👉re-color it.
Click here to the see
| version | Description | Suitable For | download(Unzip Password:ddert657) |
|---|---|---|---|
| online | update automatically, download model automatically when needed | for users with a good internet connection | >>> Click here to download from huggingface <<< |
| >>> Click here to download from BaiduNetDisk <<< | |||
| >>> Click here to download from GoogleDrive <<< | |||
| offline | without update, pre-download all models with the installation package | for users with a bad internet connection | >>> Click here to download from huggingface <<< (Uploading to Hugging Face always fails😞) |
| >>> Click here to download from BaiduNetDisk <<< | |||
| >>> Click here to download from GoogleDrive <<< |
PS : offline is the old version,please use online version
Q: Do I need to download the original fooocus software?
No, you don't need to. Fooocus-ControlNet-SDXL is a standalone software, not a fooocus plugin. Like most other software in the world, all you need to do is download (and unzip) -> launch it, there's nothing else required.Q: What is 'run_anime.bat' used for?
'run.bat' will enable the generic version of Fooocus-ControlNet-SDXL, while 'run_anime.bat' will start the animated version of Fooocus-ControlNet-SDXL. The animated version of Fooocus-ControlNet-SDXL doesn't have any magical spells inside; it simply changes some default configurations from the generic version. You can try launching both the generic and animated versions separately to see if there are any differences in the user interface.Q: What is 'run_realistic.bat' used for?
Realistic version of Fooocus-ControlNet-SDXLQ: How to speed up ?
In the first time you launch the software, it will automatically download models(only for online Version): 1. It will download [sd_xl_base_1.0_0.9vae.safetensors from here](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0_0.9vae.safetensors) as the file "Fooocus\models\checkpoints\sd_xl_base_1.0_0.9vae.safetensors". 2. It will download [sd_xl_refiner_1.0_0.9vae.safetensors from here](https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/resolve/main/sd_xl_refiner_1.0_0.9vae.safetensors) as the file "Fooocus\models\checkpoints\sd_xl_refiner_1.0_0.9vae.safetensors". 3. Note that if you use inpaint, at the first time you inpaint an image, it will download [Fooocus's own inpaint control model from here](https://huggingface.co/lllyasviel/fooocus_inpaint/resolve/main/inpaint.fooocus.patch) as the file "Fooocus\models\inpaint\inpaint.fooocus.patch" (the size of this file is 1.28GB).After Fooocus-ControlNet-SDXL 2.1.60(for convenience, the version of Fooocus-ControlNet-SDXL follows fooocus),
you will also have run_anime.bat and run_realistic.bat.
They are different model presets (and requires different models, but they will be automatically downloaded).
Check here for more details.
🚀If you already have these files, you can copy them to the above locations to speed up installation.🚀
Note that if you see "MetadataIncompleteBuffer" or "PytorchStreamReader", then your model files are corrupted. Please download models again.
Below is a test on a relatively low-end laptop with 16GB System RAM and 6GB VRAM (Nvidia 3060 laptop). The speed on this machine is about 1.35 seconds per iteration. Pretty impressive – nowadays laptops with 3060 are usually at very acceptable price.
For faster rendering speeds and reduced computer lag, consider buy more RAM. This is the most cost-effective solution, as RAM is significantly cheaper than graphics memory (VRAM). Cost-effective option suggestion: 🚀 32GB of RAM + 8GB of VRAM 🚀or 🚀48GB of RAM + 8GB of VRAM 🚀
Besides, recently many other software report that Nvidia driver above 532 is sometimes 10x slower than Nvidia driver 531. If your generation time is very long, consider
- 🚀download Nvidia Driver 531 Laptop
- 🚀or Nvidia Driver 531 Desktop.
Note that the minimal requirement is 4GB Nvidia GPU memory (4GB VRAM) and 8GB system memory (8GB RAM). This requires using Microsoft’s Virtual Swap technique, which is automatically enabled by your Windows installation in most cases, so you often do not need to do anything about it. However, if you are not sure, or if you manually turned it off (would anyone really do that?), or if you see any "RuntimeError: CPUAllocator", you can enable it here:
Click here to the see the image instruction.
And make sure that you have at least 40GB free space on each drive if you still see "RuntimeError: CPUAllocator" !
Please open an issue if you use similar devices but still cannot achieve acceptable performances.
Click here to the see
(Last tested - 2023 Oct 10)| Colab | Info |
|---|---|
 |
FooocusControl Official |
Note that this Colab will disable refiner by default because Colab free's resource is relatively limited.
Thanks to camenduru!
Click here to the see
If you want to use Anaconda/Miniconda, you cangit clone https://github.com/fenneishi/Fooocus-ControlNet-SDXL.git
cd Fooocus
conda env create -f environment.yaml
conda activate fooocusControl
pip install pygit2==1.12.2
Then download the models: download sd_xl_base_1.0_0.9vae.safetensors from here as the file "Fooocus\models\checkpoints\sd_xl_base_1.0_0.9vae.safetensors", and download sd_xl_refiner_1.0_0.9vae.safetensors from here as the file "Fooocus\models\checkpoints\sd_xl_refiner_1.0_0.9vae.safetensors". Or let Fooocus automatically download the models using the launcher:
conda activate fooocusControl
python entry_with_update.py
Or if you want to open a remote port, use
conda activate fooocusControl
python entry_with_update.py --listen
Use python entry_with_update.py --preset anime or python entry_with_update.py --preset realistic for Fooocus
Anime/Realistic Edition.
Click here to the see
Your Linux needs to have **Python 3.10** installed, and lets say your Python can be called with command **python3** with your venv system working, you cangit clone https://github.com/fenneishi/Fooocus-ControlNet-SDXL.git
cd Fooocus
python3 -m venv fooocus_control_env
source fooocus_control_env/bin/activate
pip install pygit2==1.12.2
See the above sections for model downloads. You can launch the software with:
source fooocus_control_env/bin/activate
python entry_with_update.py
Or if you want to open a remote port, use
source fooocus_control_env/bin/activate
python entry_with_update.py --listen
Use python entry_with_update.py --preset anime or python entry_with_update.py --preset realistic for Fooocus
Anime/Realistic Edition.
Click here to the see
If you know what you are doing, and your Linux already has **Python 3.10** installed, and your Python can be called with command **python3** (and Pip with **pip3**), you cangit clone https://github.com/fenneishi/Fooocus-ControlNet-SDXL.git
cd Fooocus
pip3 install pygit2==1.12.2
See the above sections for model downloads. You can launch the software with:
python3 entry_with_update.py
Or if you want to open a remote port, use
python3 entry_with_update.py --listen
Use python entry_with_update.py --preset anime or python entry_with_update.py --preset realistic for Fooocus
Anime/Realistic Edition.
Click here to the see
Same with the above instructions. You need to change torch to AMD versionpip uninstall torch torchvision torchaudio torchtext functorch xformers
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.6
AMD is not intensively tested, however. The AMD support is in beta.
Use python entry_with_update.py --preset anime or python entry_with_update.py --preset realistic for Fooocus
Anime/Realistic Edition.
Click here to the see
Same with Windows. Download the software, edit the content of `run.bat` as:.\python_embeded\python.exe -m pip uninstall torch torchvision torchaudio torchtext functorch xformers -y
.\python_embeded\python.exe -m pip install torch-directml
.\python_embeded\python.exe -s Fooocus\entry_with_update.py --directml
pause
Then run the run.bat.
AMD is not intensively tested, however. The AMD support is in beta.
Use python entry_with_update.py --preset anime or python entry_with_update.py --preset realistic for Fooocus
Anime/Realistic Edition.
Click here to the see
Mac is not intensively tested. Below is an unofficial guideline for using Mac. You can discuss problems [here](lllyasviel#129).You can install Fooocus on Apple Mac silicon (M1 or M2) with macOS 'Catalina' or a newer version. Fooocus runs on Apple silicon computers via PyTorch MPS device acceleration. Mac Silicon computers don't come with a dedicated graphics card, resulting in significantly longer image processing times compared to computers with dedicated graphics cards.
- Install the conda package manager and pytorch nightly. Read the Accelerated PyTorch training on Mac Apple Developer guide for instructions. Make sure pytorch recognizes your MPS device.
- Open the macOS Terminal app and clone this repository
with
git clone https://github.com/fenneishi/Fooocus-ControlNet-SDXL.git. - Change to the new Fooocus directory,
cd Fooocus. - Create a new conda environment,
conda env create -f environment.yaml. - Activate your new conda environment,
conda activate fooocusControl. - Install the pygit2,
pip install pygit2==1.12.2. - Install the packages required by Fooocus,
pip install -r requirements_versions.txt. - Launch Fooocus by running
python entry_with_update.py. The first time you run Fooocus, it will automatically download the Stable Diffusion SDXL models and will take a significant time, depending on your internet connection.
Use python entry_with_update.py --preset anime or python entry_with_update.py --preset realistic for Fooocus
Anime/Realistic Edition.
click here to show
After the first time you run Fooocus, a config file will be generated at Fooocus\user_path_config.txt. This file can
be edited for changing the model path. You can also change some parameters to turn Fooocus into "your Fooocus".
For
example "realisticStockPhoto_v10" is a pretty good model from CivitAI.
This model needs a special CFG=3.0 and probably works better with some specific styles. Below is an example config to
turn Fooocus into a "Fooocus Realistic Stock Photo Software":
Fooocus\user_path_config.txt:
{
"modelfile_path": "D:\\Fooocus\\models\\checkpoints",
"lorafile_path": "D:\\Fooocus\\models\\loras",
"vae_approx_path": "D:\\Fooocus\\models\\vae_approx",
"upscale_models_path": "D:\\Fooocus\\models\\upscale_models",
"inpaint_models_path": "D:\\Fooocus\\models\\inpaint",
"controlnet_models_path": "D:\\Fooocus\\models\\controlnet",
"clip_vision_models_path": "D:\\Fooocus\\models\\clip_vision",
"fooocus_expansion_path": "D:\\Fooocus\\models\\prompt_expansion\\fooocus_expansion",
"temp_outputs_path": "D:\\Fooocus\\outputs",
"default_model": "realisticStockPhoto_v10.safetensors",
"default_refiner": "",
"default_lora": "",
"default_lora_weight": 0.25,
"default_cfg_scale": 3.0,
"default_sampler": "dpmpp_2m",
"default_scheduler": "karras",
"default_negative_prompt": "low quality",
"default_positive_prompt": "",
"default_styles": [
"Fooocus V2",
"Default (Slightly Cinematic)",
"SAI Photographic"
]
}🔥About Fooocus(Thanks to lllyasviel great work! )🔥
click here to show
Fooocus is an image generating software (based on Gradio).
Fooocus is a rethinking of Stable Diffusion and Midjourney’s designs:
-
Learned from Stable Diffusion, the software is offline, open source, and free.
-
Learned from Midjourney, the manual tweaking is not needed, and users only need to focus on the prompts and images.
Fooocus has included and automated lots of inner optimizations and quality improvements. Users can forget
all those difficult technical parameters, and just enjoy the interaction between human and computer to "explore new
mediums of thought and expanding the imaginative powers of the human species" [1].
Fooocus has simplified the installation. Between pressing "download" and generating the first image, the number of needed mouse clicks is strictly limited to less than 3. Minimal GPU memory requirement is 4GB (Nvidia).
[1] David Holz, 2019.
Using Fooocus is as easy as (probably easier than) Midjourney – but this does not mean we lack functionality. Below are the details.
| Midjourney | Fooocus |
|---|---|
| High-quality text-to-image without needing much prompt engineering or parameter tuning. (Unknown method) |
High-quality text-to-image without needing much prompt engineering or parameter tuning. (Fooocus has offline GPT-2 based prompt processing engine and lots of sampling improvements so that results are always beautiful, no matter your prompt is as short as “house in garden” or as long as 1000 words) |
| V1 V2 V3 V4 | Input Image -> Upscale or Variation -> Vary (Subtle) / Vary (Strong) |
| U1 U2 U3 U4 | Input Image -> Upscale or Variation -> Upscale (1.5x) / Upscale (2x) |
| Inpaint / Up / Down / Left / Right (Pan) | Input Image -> Inpaint or Outpaint -> Inpaint / Up / Down / Left / Right (Fooocus uses its own inpaint algorithm and inpaint models so that results are more satisfying than all other software that uses standard SDXL inpaint method/model) |
| Image Prompt | Input Image -> Image Prompt (Fooocus uses its own image prompt algorithm so that result quality and prompt understanding are more satisfying than all other software that uses standard SDXL methods like standard IP-Adapters or Revisions) |
| --style | Advanced -> Style |
| --stylize | Advanced -> Advanced -> Guidance |
| --niji | Multiple launchers: "run.bat", "run_anime.bat", and "run_realistic.bat". Fooocus support SDXL models on Civitai (You can google search “Civitai” if you do not know about it) |
| --quality | Advanced -> Quality |
| --repeat | Advanced -> Image Number |
| Multi Prompts (::) | Just use multiple lines of prompts |
| Prompt Weights | You can use " I am (happy:1.5)". Fooocus uses A1111's reweighting algorithm so that results are better than ComfyUI if users directly copy prompts from Civitai. (Because if prompts are written in ComfyUI's reweighting, users are less likely to copy prompt texts as they prefer dragging files) To use embedding, you can use "(embedding:file_name:1.1)" |
| --no | Advanced -> Negative Prompt |
| --ar | Advanced -> Aspect Ratios |
We also have a few things borrowed from the best parts of LeonardoAI:
| LeonardoAI | Fooocus |
|---|---|
| Prompt Magic | Advanced -> Style -> Fooocus V2 |
| Advanced Sampler Parameters (like Contrast/Sharpness/etc) | Advanced -> Advanced -> Sampling Sharpness / etc |
| User-friendly ControlNets | Input Image -> Image Prompt -> Advanced |
Fooocus also developed many "fooocus-only" features for advanced users to get perfect results. Click here to browse the advanced features.
List of "Hidden" Tricks
Below things are already inside the software, and users do not need to do anything about these.
GPT2-based prompt expansion as a dynamic style "Fooocus V2". ( similar to Midjourney's hidden pre-processsing and "raw" mode, or the LeonardoAI's Prompt Magic).
- Native refiner swap inside one single k-sampler. The advantage is that now the refiner model can reuse the base model's momentum (or ODE's history parameters) collected from k-sampling to achieve more coherent sampling. In Automatic1111's high-res fix and ComfyUI's node system, the base model and refiner use two independent k-samplers, which means the momentum is largely wasted, and the sampling continuity is broken. Fooocus uses its own advanced k-diffusion sampling that ensures seamless, native, and continuous swap in a refiner setup. (Update Aug 13: Actually I discussed this with Automatic1111 several days ago and it seems that the “native refiner swap inside one single k-sampler” is merged into the dev branch of webui. Great!)
- Negative ADM guidance. Because the highest resolution level of XL Base does not have cross attentions, the positive and negative signals for XL's highest resolution level cannot receive enough contrasts during the CFG sampling, causing the results look a bit plastic or overly smooth in certain cases. Fortunately, since the XL's highest resolution level is still conditioned on image aspect ratios (ADM), we can modify the adm on the positive/negative side to compensate for the lack of CFG contrast in the highest resolution level. (Update Aug 16, the IOS App Drawing Things will support Negative ADM Guidance. Great!)
- We implemented a carefully tuned variation of the Section 5.1 of "Improving Sample Quality of Diffusion Models Using Self-Attention Guidance". The weight is set to very low, but this is Fooocus's final guarantee to make sure that the XL will never yield overly smooth or plastic appearance (examples here). This can almostly eliminate all cases that XL still occasionally produce overly smooth results even with negative ADM guidance. (Update 2023 Aug 18, the Gaussian kernel of SAG is changed to an anisotropic kernel for better structure preservation and fewer artifacts.)
- We modified the style templates a bit and added the "cinematic-default".
- We tested the "sd_xl_offset_example-lora_1.0.safetensors" and it seems that when the lora weight is below 0.5, the results are always better than XL without lora.
- The parameters of samplers are carefully tuned.
- Because XL uses positional encoding for generation resolution, images generated by several fixed resolutions look a bit better than that from arbitrary resolutions (because the positional encoding is not very good at handling int numbers that are unseen during training). This suggests that the resolutions in UI may be hard coded for best results.
- Separated prompts for two different text encoders seem unnecessary. Separated prompts for base model and refiner may work but the effects are random, and we refrain from implement this.
- DPM family seems well-suited for XL, since XL sometimes generates overly smooth texture but DPM family sometimes generate overly dense detail in texture. Their joint effect looks neutral and appealing to human perception.
- A carefully designed system for balancing multiple styles as well as prompt expansion.
- Using automatic1111's method to normalize prompt emphasizing. This significantly improve results when users directly copy prompts from civitai.
- The joint swap system of refiner now also support img2img and upscale in a seamless way.
- CFG Scale and TSNR correction (tuned for SDXL) when CFG is bigger than 10.
After the first time you run Fooocus, a config file will be generated at Fooocus\user_path_config.txt. This file can
be edited for changing the model path. You can also change some parameters to turn Fooocus into "your Fooocus".
For
example "realisticStockPhoto_v10" is a pretty good model from CivitAI.
This model needs a special CFG=3.0 and probably works better with some specific styles. Below is an example config to
turn Fooocus into a "Fooocus Realistic Stock Photo Software":
Fooocus\user_path_config.txt:
{
"modelfile_path": "D:\\Fooocus\\models\\checkpoints",
"lorafile_path": "D:\\Fooocus\\models\\loras",
"vae_approx_path": "D:\\Fooocus\\models\\vae_approx",
"upscale_models_path": "D:\\Fooocus\\models\\upscale_models",
"inpaint_models_path": "D:\\Fooocus\\models\\inpaint",
"controlnet_models_path": "D:\\Fooocus\\models\\controlnet",
"clip_vision_models_path": "D:\\Fooocus\\models\\clip_vision",
"fooocus_expansion_path": "D:\\Fooocus\\models\\prompt_expansion\\fooocus_expansion",
"temp_outputs_path": "D:\\Fooocus\\outputs",
"default_model": "realisticStockPhoto_v10.safetensors",
"default_refiner": "",
"default_lora": "",
"default_lora_weight": 0.25,
"default_cfg_scale": 3.0,
"default_sampler": "dpmpp_2m",
"default_scheduler": "karras",
"default_negative_prompt": "low quality",
"default_positive_prompt": "",
"default_styles": [
"Fooocus V2",
"Default (Slightly Cinematic)",
"SAI Photographic"
]
}Consider twice before you really change the config. If you find yourself breaking things, just
delete Fooocus\user_path_config.txt. Fooocus will go back to default.
A safter way is just to try "run_anime.bat" or "run_realistic.bat" - they should be already good enough for different tasks.
Click here to browse the advanced features.
Fooocus also has many community forks, just like SD-WebUI, for enthusiastic users who want to try!
| SD-WebUI's forks | Fooocus' forks |
|---|---|
| vladmandic/automatic anapnoe/stable-diffusion-webui-ux and so on ... |
runew0lf/RuinedFooocus MoonRide303/Fooocus-MRE and so on ... |
See also About Forking and Promotion of Forks.
Fooocus is powered by FCBH backend, which starts from an odd mixture of Automatic1111 and ComfyUI.
Special thanks to twri and 3Diva for creating additional SDXL styles available in Fooocus.
The log is here.