Some interesting things to try

[lllyasviel]

As you know, fooocus uses a very special method for SD1.5 as refiner swap. Since recent versions, the "switch at" can be set in advanced.

Playing with this is lots of fun. We are still experimenting this and will document it in a more formal way after it is ready.

I think I should put an early example

Below is "woman, street", seed 1234, run.bat default parameters (default sdxl + refiner)

then you download Realistic Vision V5.1 (it is a SD1.5) https://civitai.com/models/4201/realistic-vision-v51

Then you set it as refiner

Then set refiner switch to 0.4

Then you get this

just omg - what the h

I also tried dreamsharperv8 but Realistic Vision looks better for realistic images

Note that we are still experimenting this and the swapping algorithm may be changed in the future. (maybe the latent image saturation will be reviewed again, probably)

This swapping algorithm is only available in Fooocus.

[fenneishi]

I mean, the effect is so amazing,

[MoonRide303]

@lllyasviel Using similar technique for executing first few steps with SD-based models can also yield interesting results.

There are some interesting finetuned SD models available that do pretty well at resolutions close to 512x512, and they can be used to provide initial concept of the image (let's say first 10-30% of the steps), that then can be used as input for further SDXL (or SDXL base + SD refiner) processing - in general it would work like built-in img2img stage with high denoise value.

This way you can for example use output from SD like this:

and turn it into this (using original SDXL base and refiner):

I was playing with hybrid workflows like that when no ControlNets for SDXL were available - I was applying CN 1.1 to SD 1.5 at initial stage, and then processed it further through img2img with SDXL models.

But SDXL base model (or any finetuned variant of it) should probably stay in the flow, as it seems to be best choice for providing middle steps for resolutions close to 1024x1024. So let's say roughly start for SDXL base could be at 20%, and switch at around 70% of the steps. Optimal switch value might of course vary (depending on used sampler, scheduler, seed, prompt, and models) - your example here is good illustration of that :).

[Panagopulos]

Can I ask how do I achieve this which steps in terms of what to download and where to put it I have to do?

[MoonRide303]

Another interesting thing that might be worth trying is SSD-1B model (GitHub, Hugging Face). It's a distilled (about 4.5 GB) variant of SDXL with 1) some layers removed and 2) some additional training provided. According to authors it allows achieving results close to SDXL with about 60% speedup. Support for it was added in ComfyUI yesterday.

My initial observations:

• works as drop-in replacement for SDXL base (in current ComfyUI),
• it's noticeably faster (total time with refiner used goes down around 20%) and uses less VRAM,
• it provides different and less varied output (I tried generating some cars, and most of the time I was getting black color - even though I didn't specify any color in the prompt),
• it's hard to compare quality due to big difference in the output for given seed and prompt. Most of the time I liked output from original SDXL a bit more, but output from SSD-1B didn't look bad, either. More tests on bigger user base could help here.

It might be be useful alternative for people with weaker GPUs (provided for example via another preset in Fooocus).

[lllyasviel]

i refreshed that github bot and ssd1b should work out of box

[Remember2015]

so we can use SSD-1B in Fooocus now?

[MoonRide303]

@Remember2015 Yes, but there's one more side-effect I didn't previously notice - a lot of messages about some LoRA keys not being loaded (in both ComfyUI and Fooocus), like that:

Lora key not loaded:  diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_q.weight
Lora key not loaded:  diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_k.weight
Lora key not loaded:  diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_v.weight
Lora key not loaded:  diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_out.0.weight
Lora key not loaded:  diffusion_model.input_blocks.7.1.transformer_blocks.4.ff.net.0.proj.weight
Lora key not loaded:  diffusion_model.input_blocks.7.1.transformer_blocks.4.ff.net.2.weight

Probably it's caused by some layers being removed in SSD-1B compared to SDXL. LoRAs for SDXL seems to be somewhat working (output is changed when applying different LoRAs), but due to different number of layers this model might need own dedicated LoRAs.

[Remember2015]

After testing SSD-1B in lastest Fooocus, it works

[3Diva]

This is really quite a cool method to generate images!

SDXL only:

SDXL plus SD1.5 as a refiner:

I definitely increases the realism! I really like this workflow method!

[Panagopulos]

Can I ask how do I achieve this which steps in terms of what to download and where to put it I have to do?

[WTFox1]

Go to the link to download Realistic Vision V5.1 (it is a SD1.5) https://civitai.com/models/4201/realistic-vision-v51, and download the SAFETENSORS File. Drop that into your ...\Fooocus\models\checkpoints .

It should then become an option as a Refiner

[meagerfindings]

Probably a dumb question, but is this only for checkpoints. And not loras?

[3Diva]

@meagerfindings If I understand your question correctly, yes, you can only use SDXL loras in Fooocus - it doesn't read SD1.5 loras and they have no effect on the image.

[meagerfindings]

Ty for the answer! Sorry for not being specific there. I was mistakenly trying to put 1.5 loras into the checkpoints folder to use as diffusers. I see now it's just checkpoint 1.5s that work. Thank you for the clarification and guidance!

[subhroacharjee]

@lllyasviel can you drop the preset json file that you are using for the above image generation, as i am have been trying to do the same but i am getting this error

ValueError: Error while deserializing header: HeaderTooLarge
File corrupted: /content/Fooocus/models/checkpoints/realisticVisionV51_v51VAE.safetensors 

I have added my preset file below. When i am making the refiner realisticVisionV51_v%1VAE.safetensors as a default then i am getting this issue; in case i am not mentioning it as default and just adding it to checkpoint_downloads it's not showing the refiner in the dropdown.

my custom preset json
custom.json

[BabaSass]

I´m having the same issue as you.

And this is the view inside Fooocus, waiting for task to start

If anyone knows how to resolve this issue I would be eternally grateful!

[subhroacharjee]

hey @BabaSass so i resolved this issue, so it turns out all you have to do is add ?download=true to the url you are using in the checkpoint_downloads. Here is an example with 6VAE https://huggingface.co/SG161222/Realistic_Vision_V6.0_B1_noVAE/resolve/main/Realistic_Vision_V6.0_NV_B1.safetensors?download=true

[michaelcawcutt]

Has anybody gotten Realistic Vision v51 to work with FaceSwap imagePrompt?

I admit, the quality and realism of the output is impressive - however, it's not playing nice, nor taking the FaceSwap input.

Thoughts?

[subhroacharjee]

Yup i tried with Faceswap, like you said the realism is quite impressive but the output is much different than what base image was

[sdprompts]

Looks like the refiner totally ignores the prompt, I've had men turn into women, Tyra Banks turn asian, animals turn into people, etc. If you want a good laugh try using "Ape close up" as prompt.

[michaelcawcutt]

Who can I fund to update Fooocus FaceSwap to play well with Realism :D

[McxCZE]

I have troubles using faceswap when using this method to refine an image. It basically jumps in, and deletes the face, and swaps it alltogether. Any tips ?

[michaelcawcutt]

That's what we're all seeing - Realism is fantastic for creating intricate detailed outputs - but it ignores FaceSwap all together.

[crixaliss]

Hello, sorry for stupid questions.
How can I install this plug-in in fooocus 2.1.859
Thank you

[3Diva]

What plug-in do you mean? If you're wanting to know how to use Realistic Vision as a refiner within Fooocus you'll need to download the Realistic Vision model from CivitAi: https://civitai.com/models/4201/realistic-vision-v51 and install that into your Checkpoints folder in Fooocus. The checkpoints folder can be found in the folder you've installed Fooocus in and Fooocus >> models >> checkpoints. Once the Realistic Vision model is in your Checkpoints folder you should see it in the Model Tab and the drop-down list for the Refiner. You may need to restart Fooocus for it to show up in the refiner dropdown. 😊

[crixaliss]

What plug-in do you mean? If you're wanting to know how to use Realistic Vision as a refiner within Fooocus you'll need to download the Realistic Vision model from CivitAi: https://civitai.com/models/4201/realistic-vision-v51 and install that into your Checkpoints folder in Fooocus. The checkpoints folder can be found in the folder you've installed Fooocus in and Fooocus >> models >> checkpoints. Once the Realistic Vision model is in your Checkpoints folder you should see it in the Model Tab and the drop-down list for the Refiner. You may need to restart Fooocus for it to show up in the refiner dropdown. 😊

Thank you very much. This is what I want. 👍🤝.