[Model Support] FLUX.1-dev #28

raoulritter · 2024-09-01T08:14:57Z

No description provided.

…plementation from flux in mlx

atiorh

Thanks @raoulritter! Did you test FLUX.1-schnell after your changes?

python/src/diffusionkit/mlx/model_io.py

python/src/diffusionkit/mlx/__init__.py

python/test-gen.py

atiorh · 2024-09-02T11:30:53Z

python/src/diffusionkit/mlx/test-conversion-mlx.py

@@ -0,0 +1,123 @@
+import mlx.core as mx


Could you please follow the UnitTest.TestCase usage and make this a simple unit test like this? Also, no need to upload to hub from within the test 👍

raoulritter · 2024-09-03T14:16:19Z

Hey @atiorh, I've made the changes as requested. Just unsure in regards to what you exactly expect from the test-conversion-mlx? Should it be moved to the tests or should it maybe even be deleted as it's not really needed.

Also checking the merge conflicts it appears these are related to the hosting of the checkpoint. Assuming your team will host these checkpoints yourself so won't resolve these for you. If there is anything else let me know.

atiorh · 2024-09-03T14:45:46Z

@raoulritter Thanks for the quick iteration! We will revise the testing files, download the ckpt and host it alongside schnell. cc: @arda-argmax

arda-argmax · 2024-09-04T18:43:18Z

Hey @raoulritter, I have encountered with problems while trying to run your code. I think you need to make your huggingface checkpoint repo public because I cannot access it. Also, it seems that your branch diverged from our main branch. Can you also rebase it? 🙏

This is the cli command I'm trying to run:

diffusionkit-cli --prompt "a photo of a cat" --output-path out.png --height 512 --width 512 --model-version FLUX.1-dev --shift 1

raoulritter · 2024-09-04T19:13:55Z

Hey @arda-argmax,

I think it would perhaps be better based on the conflicts if you guys fixed them according to how you want them instead of me rebasing.

I have encountered with problems while trying to run your code. I think you need to make your huggingface checkpoint repo public because I cannot access it`

I previously wrote the following

Also checking the merge conflicts it appears these are related to the hosting of the checkpoint. Assuming your team will host these checkpoints yourself so won't resolve these for you.

Let me know if you still want me to take a look but perhaps would make more sense from your side.

arda-argmax · 2024-09-04T23:04:49Z

I see, sorry I missed it. Can you also share how you run the dev model?

I'm getting this error when I'm trying to run your branch with the following command:

DiffusionKit % diffusionkit-cli --prompt "a photo of a cat" --output-path out.png --height 512 --width 512 --model-version FLUX.1-dev --shift 1

Error:

scikit-learn version 1.5.1 is not supported. Minimum required version: 0.17. Maximum required version: 1.1.2. Disabling scikit-learn conversion API.
Torch version 2.4.0 has not been tested with coremltools. You may run into unexpected errors. Torch 2.2.0 is the most recent version that has been tested.
WARNING:diffusionkit.mlx.scripts.generate_images:Disabling CFG for FLUX.1-schnell model.
/opt/miniconda3/envs/dkit-test/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
INFO:diffusionkit.mlx.scripts.generate_images:Output image resolution will be 512x512
INFO:diffusionkit.mlx:Pre text encoding peak memory: 0.0GB
INFO:diffusionkit.mlx:Pre text encoding active memory: 0.0GB
INFO:diffusionkit.mlx:Post text encoding peak memory: 2.761GB
INFO:diffusionkit.mlx:Post text encoding active memory: 0.478GB
INFO:diffusionkit.mlx:Text encoding time: 0.616s
INFO:diffusionkit.mlx:Pre denoise peak memory: 0.0GB
INFO:diffusionkit.mlx:Pre denoise active memory: 0.004GB
INFO:diffusionkit.mlx:Seed: 1725483544
  0%|                                                                                                                                                                  | 0/50 [00:00<?, ?it/s]INFO:diffusionkit.mlx.mmdit:Cached modulation_params for timesteps=array([1000, 980, 960, ..., 40, 20, 0], dtype=bfloat16)
INFO:diffusionkit.mlx.mmdit:Cached modulation_params will reduce peak memory by 6.5 GB
  0%|                                                                                                                                                                  | 0/50 [00:04<?, ?it/s]
Traceback (most recent call last):
  File "/opt/miniconda3/envs/dkit-test/bin/diffusionkit-cli", line 33, in <module>
    sys.exit(load_entry_point('diffusionkit', 'console_scripts', 'diffusionkit-cli')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/scripts/generate_images.py", line 166, in cli
    image, _ = sd.generate_image(
               ^^^^^^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/__init__.py", line 412, in generate_image
    latents, iter_time = self.denoise_latents(
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/__init__.py", line 284, in denoise_latents
    latent, iter_time = sample_euler(
                        ^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/__init__.py", line 767, in sample_euler
    denoised = model(x, timesteps[i], sigmas[i], **extra_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/__init__.py", line 703, in __call__
    mmdit_output = self.model.mmdit(**mmdit_input)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/mmdit.py", line 222, in __call__
    latent_image_embeddings, token_level_text_embeddings = block(
                                                           ^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/mmdit.py", line 573, in __call__
    image_intermediates = self.image_transformer_block.pre_sdpa(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/argmax-mac-studio/arda/test_repos/DiffusionKit/python/src/diffusionkit/mlx/mmdit.py", line 444, in pre_sdpa
    modulation_params = self._modulation_params[timestep.item()]

ValueError: [convert] Only length-1 arrays can be converted to Python scalars.

ZachNagengast

Thanks for contributing, this will be really cool! Tests still need some work to conform to the rest of the repo's style, but added some suggestions to use the argmax HF repo. If we want to follow up with tests in another PR, you may want to just confirm the pipeline is working with the new repo, and take out the current tests for later.

python/src/diffusionkit/mlx/config.py

python/src/diffusionkit/mlx/__init__.py

python/src/diffusionkit/mlx/mmdit.py

python/src/diffusionkit/mlx/model_io.py

ZachNagengast · 2024-09-06T21:07:09Z

Also updated the merge conflicts 👍 Looks like just a linter issue now

ZachNagengast · 2024-09-06T22:10:36Z

Added a couple changes and got it working locally, just need to fix/remove the tests 👍

diffusionkit-cli --prompt "detailed cinematic dof render of a \
detailed MacBook Pro on a wooden desk in a dim room with items \
around, messy dirty room. On the screen are the letters 'FLUX on \
DiffusionKit' glowing softly. High detail hard surface render" \
--height 768 \
--width 1360 \
--seed 1001 \
--step 50 \
--output ~/Desktop/flux_on_mac.png \
--model-version argmaxinc/mlx-FLUX.1-dev

arda-argmax

Everything seems to be working 👍 LGTM!
@raoulritter Thank you for your contribution and @ZachNagengast thank you for the pipeline fixes 🙏

…to raoulritter/main

mgierschdev · 2024-09-09T20:24:09Z

Trying to run from this branch and I am getting:
huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-66df5923-68833e15000b3b0d21d61def;e6a2d04b-f799-46e3-ab18-23cf484b0657)

Repository Not Found for url: https://huggingface.co/argmaxinc/mlx-FLUX.1-dev/resolve/main/flux1-dev.safetensors.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.

ZachNagengast · 2024-09-09T22:05:40Z

@mgierschdev Thanks for the report, there were some restrictions on the model repo that I've removed now, could you try it again and lmk how it goes? You may still need to accept the terms of the license from black forest labs to use it.

ZachNagengast · 2024-09-10T01:35:05Z

@raoulritter This is an awesome contribution, thank you!

raoulritter · 2024-09-10T06:50:07Z

Hi @ZachNagengast happy to help thanks for fixing the conflicts and nits. I would love to (help) add lora capabilities as well not sure what your team's plan is.

raoulritter added 4 commits August 22, 2024 12:50

Temp commit to show issue:

217a211

Temp commit to show issue. Need to add guidance_embed according to im…

19d9b6c

…plementation from flux in mlx

Working conversion unable to save properly

8f2c9a6

Working conversion no negative prompt

19d65df

atiorh reviewed Sep 2, 2024

View reviewed changes

atiorh changed the title ~~Working conversion without negative prompts~~ [Model Support] FLUX.1-dev Sep 2, 2024

raoulritter marked this pull request as ready for review September 3, 2024 14:03

Update according to review

dce4c87

ZachNagengast requested changes Sep 6, 2024

View reviewed changes

Merge branch 'main' into main

1508f9f

ZachNagengast and others added 5 commits September 6, 2024 14:24

Apply suggestions from code review for flux dev

1577412

Guidance variable fixes

b076287

Formatting

b329c27

Use proper constants

a3c108c

Update python/src/diffusionkit/mlx/config.py

0391f29

ZachNagengast requested a review from arda-argmax September 6, 2024 22:08

arda-argmax approved these changes Sep 6, 2024

View reviewed changes

ZachNagengast added 2 commits September 6, 2024 15:53

Update formatting

c5726c2

Merge branch 'main' of https://github.com/raoulritter/DiffusionKit in…

6541efb

…to raoulritter/main

arda-argmax added 2 commits September 9, 2024 16:20

update README.md

d0ee0fd

deleted test codes

5d55b07

arda-argmax added 2 commits September 9, 2024 16:42

update README

9c2289a

update version

5cdda4d

arda-argmax merged commit bfbdd0e into argmaxinc:main Sep 9, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Model Support] FLUX.1-dev #28

[Model Support] FLUX.1-dev #28

raoulritter commented Sep 1, 2024

atiorh left a comment

atiorh Sep 2, 2024

raoulritter commented Sep 3, 2024 •

edited

Loading

atiorh commented Sep 3, 2024

arda-argmax commented Sep 4, 2024 •

edited

Loading

raoulritter commented Sep 4, 2024

arda-argmax commented Sep 4, 2024 •

edited

Loading

ZachNagengast left a comment

ZachNagengast commented Sep 6, 2024

ZachNagengast commented Sep 6, 2024

arda-argmax left a comment

mgierschdev commented Sep 9, 2024

ZachNagengast commented Sep 9, 2024 •

edited

Loading

ZachNagengast commented Sep 10, 2024

raoulritter commented Sep 10, 2024

[Model Support] FLUX.1-dev #28

[Model Support] FLUX.1-dev #28

Conversation

raoulritter commented Sep 1, 2024

atiorh left a comment

Choose a reason for hiding this comment

atiorh Sep 2, 2024

Choose a reason for hiding this comment

raoulritter commented Sep 3, 2024 • edited Loading

atiorh commented Sep 3, 2024

arda-argmax commented Sep 4, 2024 • edited Loading

raoulritter commented Sep 4, 2024

arda-argmax commented Sep 4, 2024 • edited Loading

ZachNagengast left a comment

Choose a reason for hiding this comment

ZachNagengast commented Sep 6, 2024

ZachNagengast commented Sep 6, 2024

arda-argmax left a comment

Choose a reason for hiding this comment

mgierschdev commented Sep 9, 2024

ZachNagengast commented Sep 9, 2024 • edited Loading

ZachNagengast commented Sep 10, 2024

raoulritter commented Sep 10, 2024

raoulritter commented Sep 3, 2024 •

edited

Loading

arda-argmax commented Sep 4, 2024 •

edited

Loading

arda-argmax commented Sep 4, 2024 •

edited

Loading

ZachNagengast commented Sep 9, 2024 •

edited

Loading