Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve FLUX image-to-image (Trajectory Guidance) #6900

Merged
merged 17 commits into from
Sep 20, 2024

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Sep 20, 2024

Summary

This PR makes some improvements to the FLUX image-to-image and inpainting behaviours.

Changes:

  • Expand inpainting region at a cutoff timestep. This improves seam coherence around inpainting regions.
  • Add Trajectory Guidance to improve the ability to control how much an image gets modified during image-to-image/inpainting (see the code for a more technical explanation - it's well-documented).

trajectory_guidance_strength Usage

  • The trajectory_guidance_strength param has been added to the FLUX Denoise invocation.
  • trajectory_guidance_strength defaults to 0.0 and should be in the range [0, 1].
  • trajectory_guidance_strength = 0.0 has no effect on the denoising process.
  • trajectory_guidance_strength = 1.0 will guide strongly towards the original image.

FLUX image-to-image usage tips

  • As always, prompt matters a lot.
  • If you are trying to making minor perturbations to an image, use vanilla image-to-image by setting the denoising_start param.
  • If you are trying to make significant changes to an image, using trajectory guidance will give more control than using vanilla image-to-image. Set denoising_start=0.0 and adjust trajectory_guidance_strength to control the amount of change in the image.
  • The 'transition point' where the image changes the most as you adjust trajectory_guidance_strength or denoise_start varies depending on the noise. So, set a fixed noise seed, then tune those params.

QA Instructions

  • Vanilla image-to-image - No change in output
  • Vanilla inpainting - No change in output
  • Vanilla outpainting - No change in output
  • Trajectory Guidance image-to-image
    • TGS = 0.0 is identical to Vanilla case
    • TGS = 1.0 guides close to the original image
      • Not as close as I'd like, but it's not broken.
    • Smooth transition as TGS varies
    • Smoke test: TGS with denoise_start > 0.0
  • TG inpainting
    • TGS = 0.0 is identical to Vanilla case
    • TGS = 1.0 guides close to the original image
      • Not as close as I'd like, but it's not broken
    • Smooth transition as TGS varies
    • Smoke test: TGS with denoise_start > 0.0
  • TG outpainting
    • TGS = 0.0 is identical to Vanilla case
    • Smoke test TGS outpainting
  • Smoke test FLUX text-to-image
  • Preview images look ok for all of above.

Known issues (will be addressed in follow-up PRs)

  • The current TGS scale biases towards creating more change than desired in the image. More tuning of the TG change schedule is required.
  • TGS does not work very well for outpainting right now. This might be solvable, but more likely we'll just want to discourage it in the Linear UI.

Merge Plan

No special instructions.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files python-tests PRs that change python tests labels Sep 20, 2024
@github-actions github-actions bot added the frontend PRs that change frontend files label Sep 20, 2024
@github-actions github-actions bot added the services PRs that change app services label Sep 20, 2024
@RyanJDick RyanJDick merged commit eea20f1 into main Sep 20, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/flux-trajectory-guidance branch September 20, 2024 22:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests services PRs that change app services
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants