Offloading to cpu and disk #77
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
awq/models/base.py
Outdated
| model, | ||
| device_map=device_map, | ||
| offload_dir=offload_folder | ||
| ) |
There was a problem hiding this comment.
Why not just use dispatch_model() instead if it works?
There was a problem hiding this comment.
You're right. It works just fine. I've removed call to the simple_dispatch(). And fixed a minor bug in fused/attn.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Greetings.
This PR adds arguments
max_memoryandoffload_folderto thefrom_quantized()method.Additionally, for the offloading it relies on standard
accelerate.dispatch_model()instead ofsimple_dispatch()and makes patches to the fusion layers before the offloading to prevent the cases where different tensors end up on different devices. Especially llama normalization layer stuck onmetadevice.The code was tested on RTX3060 with this model - https://huggingface.co/TheBloke/Llama-2-70B-chat-AWQ
P.S. I've noticed a strange behavior with the
accelerate.infer_auto_device_map(). Returned device map marks more layers to be placed in VRAM then it's physically possible. I haven't seen this with GPTQ models. Also not tested with the official AWQ implementation. Maybe authors of this repo have some ideas.