[Quantization] Quanto quantizer #29023
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
cc @dacorvo |
|
Quick update, all tests are passing with the exception of safetensors tests. I've also implemented the quantization on the spot just like we do for bnb quantization. However, if the user wants to use cpu/disk offload, he will need to install the main branch of accelerate for now because of this PR. |
|
LGTM, mostly concerned with quanto specific function addition, that should leave in quanto.py not in the modeling (as much as possible of course). |
| if hf_quantizer is not None: | ||
| missing_keys = hf_quantizer.update_missing_keys(model, missing_keys, prefix) |
Co-authored-by: Arthur <[email protected]>
…formers into quanto_integration
|
Note to the core mainteners and especially @ArthurZucker . I reverted a lot of changes I did for the serialization. I decided to postpone the serialization feature by setting the I will do a follow up PR for the serialization since the PR to make it compatible with safetensors or Without serialization, this PR should still be good to merge ! |
* start integration * fix * add and debug tests * update tests * make pytorch serialization works * compatible with device_map and offload * fix tests * make style * add ref * guard against safetensors * add float8 and style * fix is_serializable * Fix shard_checkpoint compatibility with quanto * more tests * docs * adjust memory * better * style * pass tests * Update src/transformers/modeling_utils.py Co-authored-by: Younes Belkada <[email protected]> * add is_safe_serialization instead * Update src/transformers/quantizers/quantizer_quanto.py Co-authored-by: Younes Belkada <[email protected]> * add QbitsTensor tests * fix tests * simplify activation list * Update docs/source/en/quantization.md Co-authored-by: David Corvoysier <[email protected]> * better comment * Update tests/quantization/quanto_integration/test_quanto.py Co-authored-by: David Corvoysier <[email protected]> * Update tests/quantization/quanto_integration/test_quanto.py Co-authored-by: David Corvoysier <[email protected]> * find and fix edge case * Update docs/source/en/quantization.md Co-authored-by: Arthur <[email protected]> * pass weights_only_kwarg instead * fix shard_checkpoint loading * simplify update_missing_keys * Update tests/quantization/quanto_integration/test_quanto.py Co-authored-by: Arthur <[email protected]> * recursion to get all tensors * block serialization * skip serialization tests * fix * change by cuda:0 for now * fix regression * update device_map * fix doc * add noteboon * update torch_dtype * update doc * typo * typo * remove comm --------- Co-authored-by: Younes Belkada <[email protected]> Co-authored-by: David Corvoysier <[email protected]> Co-authored-by: Arthur <[email protected]> Co-authored-by: Younes Belkada <[email protected]>
What does this PR do ?
This PR adds the quantization methods from quanto library. We will support inference + model quantization if the user perform weights only quantization since we don't require a calibration dataset.
TODO: