-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Token Merging (ToMe) for Stable Diffusion #2940
Comments
Hi. This is actually on our radar. The Meta team might soon start working on this. |
This looks really cool! |
@dbolya has this: https://github.com/dbolya/tomesd#diffusers. This allows users to just do:
However, it needs
@patrickvonplaten WDYT? Cc: @dbolya |
Option 3 seems the best to me. I would caution against just copy-pasting the code, since I do plan to update |
Option 3.) is ok/good for me! |
Alright. I will start working on the doc soon. |
Closing this with #3208. |
Is ToMe + controlnet avaible for use? I have heared that ToMe will be less affective due to controlnet modifying SD‘s forward. |
ToMe for SD speeds up diffusion by merging redundant tokens. by @dbolya
Code: https://github.com/dbolya/tomesd
Paper: https://arxiv.org/abs/2303.17604
I conducted a simple generation speed benchmark on my end. I applied a patch to stable-diffusion-webui and used the best value from 4 runs via the API. For the baseline, I used xFormers, as it is commonly used in use cases seeking speed. xFormers is also enabled when using ToMe. I adopted the recommended quality value of 0.5 for ToMe's ratio. It was curious to see high-resolution images being adopted in the paper, but the method is more effective for high-resolution images. This could be a useful feature for high-resolution use cases.
The text was updated successfully, but these errors were encountered: