Fooocus 2.5.0: Enhance!

[mashb1t]

Fooocus 2.5.0: Enhance!

This release includes a feature requested multiple times by the community (e.g. in lllyasviel#3113, lllyasviel#3089, lllyasviel#3039, and a few more, also see lllyasviel#3122).

What does this feature do?

Enhance allows you to automatically upscale and/or improve parts of the picture based on either a prompt or an input image.
It is comparable to ADetailer (repository), but offers better and more flexible object detection and replacement with detection and replacement prompts instead of static detection models, each having ~140MB.

How do i use it?

Disclaimer

It is highly recommended to use performance Speed or Quality (no performance which loads a LoRA) as any inpaint engine uses a LoRA, which may not produce the best results when combined with other performance LoRAs.
Using inpaint mode Improve Detail (face, hand, eyes, etc.) does not set an inpaint engine, making it compatible with all performances. The documentation of inpaint modes can be found here: lllyasviel#414
All of this is also the case for normal inpainting without enhancements.

ControlNets (ImagePrompt, PyraCanny, CPDS, FaceSwap) are currently not supported for enhance steps but can be used for image generation used as basis for enhance.

With image generation

1. generate any image with the settings you'd like to use
2. if you're satisfied, disable random seed and open the enhance tab
   2.1. (optional) Enable and define order upscaling or variation (default disabled)
   2.2. Enable and configure any amount of other improvement steps.
   2.3. input detection prompt (what you want to detect in the image)
   2.4. input positive / negative prompt (what you want to replace the detected masks with, defaults to your normal prompts if not set)
3. generate image

Based on an existing image

Simply open Image Input and upload an image to the Enhance tab. Prompt processing will be skipped and only enhancement steps are processed. Follow steps 2+ above.
You may set --enable-auto-describe-image to automatically generate the prompt after image upload.

Examples

UI

Original (generated)	Enhance

Original (generated)	Upscale (before)	#1 Yellow Sundress	#2 Hands replacement

Upscaling or Variation

Before First Enhancement

• always use original prompts for upscaling or variation
• improved details when enhancing due to larger canvas size
• less context of image composition due to inpainting with 1024x1024
• recommended for enhancing smaller areas (details, faces, eyes)

After Last Enhancement

• may apply changes to the image which were already fixed by enhance
• recommended for enhancing large areas (backgrounds, clothes, whole persons)
• choose which prompt to use for upscaling or variation, use either original prompts or last filled enhancement prompts with non-empty value (increases subject recognition, prevents subject from being changed back to original prompt after enhancement. Use this when substantially changing the subject described in the original prompts)

Original (generated)	Upscale (before)	Enhance

Original (generated)	Enhance	Upscale (after)

Models

By default it uses the SAM (website, repository) masking model, backed by GroundingDINO (paper, diffusers docs), but offers support for all additional models currently supported by RemBG (repository). GroundingDINO + SAM do not use RemBG as handler, but have natively been implemented into focus for even better results and increased level control.

Currently supported models:

• SAM + GroundingDINO (default): A model combination which allows detection prompts and segmentation of objects in detected boxes in 3 variants (base (default), large, huge)
• u2net (download, source): A pre-trained model for general use cases.
• u2netp (download, source): A lightweight version of u2net model.
• u2net_human_seg (download, source): A pre-trained model for human segmentation.
• u2net_cloth_seg (download, source): A pre-trained model for Cloths Parsing from human portrait. Here clothes are parsed into 3 category: Upper body, Lower body and Full body.
• silueta (download, Model size reduced to 43Mb! and new Webapp xuebinqin/U-2-Net#295): Same as u2net but the size is reduced to 43Mb.
• isnet-general-use (download, source): A new pre-trained model for general use cases.
• isnet-anime (download, source): A high-accuracy segmentation for anime character.

Tech Debt / Code Improvements

While implementing the enhance feature, multiple methods have been introduced to make code reusable and allow for iterations.
The whole async_worker.py has been restructured and is now much more clear to read and easier to use.

Debugging

Please find debugging options in Developer Debug Mode > Inpaint:

• Debug Enhance Masks: outputs generated masks, default disabled
• Debug GroundingDINO: early return in the detection process, uses GroundingDINO boxes instead of refined SAM detections inside those boxes. This option can sometimes improve results when SAM isn't able to correctly detect inside GroundingDINO boxes. FYI: GroundingDINO supports detection prompts, SAM auto-detects objects without further promps inside of the boxes.
• GroundingDINO Box Erode or Dilate: increase/decrease the detected box area by GroundingDINO. This option can sometimes improve SAM detection when GroundingDINO accidentally cuts off elements (very rare)

GroundingDINO	GroundingDINO + SAM

[mashb1t]

Please let me know your opinion about the importance of optional next steps for you, see #40 (comment)

[Kaneki010]

there seems to be an issue , when using another model other than the default juggernaut one , it errors

[Parameters] Adaptive CFG = 7
[Parameters] CLIP Skip = 2
[Parameters] Sharpness = 2
[Parameters] ControlNet Softness = 0.25
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] Seed = 3190662481288638389
[Parameters] CFG = 7
[Fooocus] Loading control models ...
[Parameters] Sampler = dpmpp_2m_sde_gpu - karras
[Parameters] Steps = 30 - 15
[Fooocus] Initializing ...
[Fooocus] Loading models ...
Refiner unloaded.
Request to load LoRAs [] for model [/content/Fooocus/models/checkpoints/sdxxxl_v30.safetensors].
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 0.99 seconds
[Fooocus] Processing prompts ...
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] ((ohwx woman)), black sundress, park, sunset, full body shot, looking at viewer, epic cinematic composition, massive, artistic, grand elaborate amazing detail, sharp focus, surreal, fine, enhanced, perfect, colorful, symmetry, best, futuristic, illuminated, magic, atmosphere, vivid, beautiful, gorgeous, light, detailed, color, intricate, outstanding
[Fooocus] Preparing Fooocus text #2 ...
[Prompt Expansion] ((ohwx woman)), black sundress, park, sunset, full body shot, looking at viewer, detailed, intricate, elegant, highly colorful, vivid color, sacred light, sharp focus, majestic, cool, enhanced, loving, glowing, epic, singular, symmetry, determined, peaceful, focused, intense, extremely, beautiful, illuminated, attractive, designed, elite
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.14 seconds
[Fooocus] Encoding positive #2 ...
[Fooocus] Encoding negative #1 ...
[Fooocus] Encoding negative #2 ...
[Parameters] Denoising Strength = 1.0
[Parameters] Initial Latent shape: Image Space (896, 1152)
Preparation time: 3.26 seconds
Using karras scheduler.
[Fooocus] Preparing task 1/2 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.96 seconds
100% 30/30 [00:27<00:00,  1.07it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.17 seconds
[Fooocus] Saving image 1/2 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-06-27/log.html
Generating and saving time: 31.71 seconds
[Fooocus] Preparing task 2/2 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.17 seconds
100% 30/30 [00:26<00:00,  1.12it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.14 seconds
[Fooocus] Saving image 2/2 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-06-27/log.html
Generating and saving time: 29.68 seconds
[Fooocus] Processing enhance ...
[Fooocus] Downloading upscale models ...
[Fooocus] Upscaling image from (1152, 896) ...
Upscaling image with shape (896, 1152, 3) ...
Image upscaled.
[Fooocus] Saving image 1/2 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-06-27/log.html
Traceback (most recent call last):
  File "/content/Fooocus/modules/async_worker.py", line 1441, in worker
    handler(task)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/content/Fooocus/modules/async_worker.py", line 1328, in handler
    current_task_id, done_steps_inpainting, done_steps_upscaling, img, exception_result = enhance_upscale(
  File "/content/Fooocus/modules/async_worker.py", line 1041, in enhance_upscale
    current_progress, img, prompt, negative_prompt = process_enhance(
ValueError: not enough values to unpack (expected 4, got 2)
Total time: 82.00 seconds

(i'm using colab btw)

[mashb1t]

@Kaneki010 found the bug, will fix it today in a few hours. This is only an issue for direct return, which is x2 (fast).
https://github.com/mashb1t/Fooocus/blob/main/modules/async_worker.py#L986

[mashb1t]

fixed & pushed to main (also docker tag edge), also fixed another edge case when using an enhance image, enhance upscale fast and no other enhancement => fix for division by 0 due to all_steps = 0. Check out the latest main commits!

[Kaneki010]

thanks a lot ! , upscaling works flawlessly now , but unfortunately an error still occurs when trying to use the Enhance feature , the adetailer-like one
`[Parameters] Adaptive CFG = 7
[Parameters] CLIP Skip = 2
[Parameters] Sharpness = 2
[Parameters] ControlNet Softness = 0.25
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] Seed = 9101246591662288144
[Parameters] CFG = 4
[Fooocus] Loading control models ...
[Parameters] Sampler = dpmpp_2m_sde_gpu - karras
[Parameters] Steps = 30 - 15
[Fooocus] Initializing ...
[Fooocus] Loading models ...
Refiner unloaded.
[Fooocus] Processing prompts ...
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] ((ohwx woman)) in jungle, black sundress, candid photograph, smiling at viewer, thoughtful, focused, elegant, highly detailed, dramatic ambient, rich colors, sharp focus, perfect composition, beautiful intricate, innocent, novel, classic, best, modern, romantic, futuristic, surreal, glowing, light, strong, emotional, pretty, clear, artistic, color
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding negative #1 ...
[Parameters] Denoising Strength = 1.0
[Parameters] Initial Latent shape: Image Space (896, 1152)
Preparation time: 1.49 seconds
Using karras scheduler.
[Fooocus] Preparing task 1/1 ...
[Sampler] refiner_swap_method = joint
[Sampler] sigma_min = 0.0291671771556139, sigma_max = 14.614643096923828
Requested to load SDXL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 1.01 seconds
100% 30/30 [00:26<00:00, 1.13it/s]
Requested to load AutoencoderKL
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.29 seconds
[Fooocus] Saving image 1/1 to system ...
Image generated with private log at: /content/Fooocus/outputs/2024-06-27/log.html
Generating and saving time: 30.70 seconds
[Fooocus] Processing enhance ...
[Fooocus] Preparing enhancement 1/1 ...
[Enhance] Searching for "face"
Downloading: "https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth" to /content/Fooocus/models/inpaint/groundingdino_swint_ogc.pth

100% 662M/662M [00:06<00:00, 106MB/s]
/usr/local/lib/python3.10/dist-packages/torch/functional.py:512: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3587.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
final text_encoder_type: bert-base-uncased
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
tokenizer_config.json: 100% 48.0/48.0 [00:00<00:00, 240kB/s]
config.json: 100% 570/570 [00:00<00:00, 3.07MB/s]
vocab.txt: 100% 232k/232k [00:00<00:00, 1.85MB/s]
tokenizer.json: 100% 466k/466k [00:00<00:00, 32.0MB/s]
model.safetensors: 100% 440M/440M [00:01<00:00, 247MB/s]
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  Requested to load GroundingDINO
  Loading 1 new model
  [Fooocus Model Management] Moving model(s) has taken 0.47 seconds
  /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:884: FutureWarning: The device argument is deprecated and will be removed in v5 of Transformers.
  warnings.warn(
  /usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:464: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
  /usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
  Downloading: "https://huggingface.co/mashb1t/misc/resolve/main/sam_vit_b_01ec64.pth" to /content/Fooocus/models/sam/sam_vit_b_01ec64.pth

100% 358M/358M [00:09<00:00, 41.1MB/s]
Requested to load Sam
Loading 1 new model
[Fooocus Model Management] Moving model(s) has taken 0.43 seconds
[Enhance] 1 boxes detected
[Enhance] 1 segments detected in boxes
[Enhance] 1 segments applied to mask
[Fooocus] Downloading inpainter ...
Downloading: "https://huggingface.co/lllyasviel/fooocus_inpaint/resolve/main/fooocus_inpaint_head.pth" to /content/Fooocus/models/inpaint/fooocus_inpaint_head.pth

100% 51.4k/51.4k [00:00<00:00, 8.39MB/s]
Downloading: "https://huggingface.co/lllyasviel/fooocus_inpaint/resolve/main/inpaint_v26.fooocus.patch" to /content/Fooocus/models/inpaint/inpaint_v26.fooocus.patch

100% 1.23G/1.23G [00:04<00:00, 270MB/s]
[Fooocus] Preparing enhance prompts ...
[Fooocus] Loading models ...
Refiner unloaded.
Request to load LoRAs [('ortega_sdxxxl_dh128_v1.safetensors', 1.1), ('/content/Fooocus/models/inpaint/inpaint_v26.fooocus.patch', 1.0)] for model [/content/Fooocus/models/checkpoints/sdxxxl_v30.safetensors].
Loaded LoRA [/content/Fooocus/models/loras/ortega_sdxxxl_dh128_v1.safetensors] for UNet [/content/Fooocus/models/checkpoints/sdxxxl_v30.safetensors] with 788 keys at weight 1.1.
Loaded LoRA [/content/Fooocus/models/loras/ortega_sdxxxl_dh128_v1.safetensors] for CLIP [/content/Fooocus/models/checkpoints/sdxxxl_v30.safetensors] with 264 keys at weight 1.1.
Loaded LoRA [/content/Fooocus/models/inpaint/inpaint_v26.fooocus.patch] for UNet [/content/Fooocus/models/checkpoints/sdxxxl_v30.safetensors] with 960 keys at weight 1.0.
Requested to load SDXLClipModel
Requested to load GPT2LMHeadModel
Loading 2 new models
[Fooocus Model Management] Moving model(s) has taken 1.60 seconds
[Fooocus] Processing prompts ...
[Fooocus] Preparing Fooocus text #1 ...
[Prompt Expansion] ((ohwx woman)) detailed face, beautiful, highly intricate, elegant, dynamic light, designed, vivid colors, ambient background, sharp focus, inspired, transparent glossy full color, professional fine detail, extremely complex, appealing, stunning composition, creative, hopeful, innocent, symmetry, perfect, pretty, attractive, epic, romantic,, clear, brilliant, cinematic, bright
[Fooocus] Encoding positive #1 ...
[Fooocus Model Management] Moving model(s) has taken 0.12 seconds
[Fooocus] Encoding negative #1 ...
Upscaling image with shape (229, 229, 3) ...
Traceback (most recent call last):
File "/content/Fooocus/modules/patch.py", line 465, in loader
result = original_loader(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 997, in load
with _open_file_like(f, 'rb') as opened_file:
File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 444, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 425, in init
super().init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/content/Fooocus/models/upscale_models/fooocus_upscaler_s409985e5.bin'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/Fooocus/modules/async_worker.py", line 1443, in worker
handler(task)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/content/Fooocus/modules/async_worker.py", line 1389, in handler
current_progress, img, enhance_prompt_processed, enhance_negative_prompt_processed = process_enhance(
File "/content/Fooocus/modules/async_worker.py", line 1011, in process_enhance
denoising_strength, initial_latent, width, height, current_progress = apply_inpaint(
File "/content/Fooocus/modules/async_worker.py", line 486, in apply_inpaint
inpaint_worker.current_task = inpaint_worker.InpaintWorker(
File "/content/Fooocus/modules/inpaint_worker.py", line 162, in init
self.interested_image = perform_upscale(self.interested_image)
File "/content/Fooocus/modules/upscaler.py", line 21, in perform_upscale
sd = torch.load(model_filename)
File "/content/Fooocus/modules/patch.py", line 481, in loader
raise ValueError(exp)
ValueError: [Errno 2] No such file or directory: '/content/Fooocus/models/upscale_models/fooocus_upscaler_s409985e5.bin'

Total time: 74.28 seconds`

[mashb1t]

@Kaneki010 i'll add the download also there tomorrow. You can simply upscale once without enhance and get the model this way.

[mashb1t]

@Kaneki010 fixed in c0e556a, will be included in release 2.5.2 and has already been merged into main

[dezvin]

@ILikeToasters need your video tutorial)

[ILikeToasters]

Will be working on one soon. Still testing things out and of course want to wait until a little closer to being ready. There is a lot of stuff to digest in this for sure.

[ILikeToasters]

Video here. Will be doing a more detailed one soon. https://youtu.be/NiGbksd9B_Y

[mashb1t]

@ILikeToasters nice, thanks!

[dezvin]

@ILikeToasters Awesome! Thank you so much!

[melteemcaglar]

I'm new here and I'm using fooocus via pinokio. How to run mashb1t's fork with pinokio?