Move quant API to quantization README #142
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
supriyar
reviewed
Apr 17, 2024
README.md
Outdated
| 4. Integration with other PyTorch native libraries like torchtune and ExecuTorch | ||
| 2. [Quantization algorithms](./torchao/quantization) such as dynamic quant, smoothquant, GPTQ that run on CPU/GPU and Mobile. | ||
| 3. [Sparsity algorithms](./torchao/sparsity) such as Wanda that help improve accuracy of sparse networks | ||
| 4. Integration with other PyTorch native libraries like [torchtune](https://github.com/pytorch/torchtune) and [ExecuTorch](https://github.com/pytorch/executorch) | ||
|
|
||
| ## Key Features | ||
| * Native PyTorch techniques, composable with torch.compile |
There was a problem hiding this comment.
can you link to the quantization readme from the main page?
supriyar
reviewed
Apr 17, 2024
| This technique works best when the torch._inductor.config.use_mixed_mm option is enabled. This avoids dequantizing the weight tensor before the matmul, instead fusing the dequantization into the matmul, thereby avoiding materialization of a large floating point weight tensor. | ||
|
|
||
|
|
||
| ## A16W4 WeightOnly Quantization |
There was a problem hiding this comment.
is it also possible to add an example on how to use GPTQ?
jerryzh168
commented
Apr 17, 2024
| 2. Quantization [algorithms](./torchao/quantization) such as dynamic quant, smoothquant, GPTQ that run on CPU/GPU and Mobile. | ||
| 3. Sparsity [algorithms](./torchao/sparsity) such as Wanda that help improve accuracy of sparse networks | ||
| 4. Integration with other PyTorch native libraries like torchtune and ExecuTorch | ||
| 2. [Quantization algorithms](./torchao/quantization) such as dynamic quant, smoothquant, GPTQ that run on CPU/GPU and Mobile. |
There was a problem hiding this comment.
@supriyar I was hoping that people can just find quantization README here, or do you feel we want to make it more explicit?
There was a problem hiding this comment.
I do agree that a getting started on the main page would be important to keep. So if GPTQ is the algorithm we feel most people would be interested in let's just show only that and then link to a broader set of algorithms as well
There was a problem hiding this comment.
we get some comment from torchtune that community has moved on to other techniques now, so I feel it's fine to keep it in the separate quantization page
jerryzh168
force-pushed
the
docs
branch
2 times, most recently
from
9346a94
to
64e0035
Compare
jerryzh168
commented
Apr 18, 2024
| 2. Quantization [algorithms](./torchao/quantization) such as dynamic quant, smoothquant, GPTQ that run on CPU/GPU and Mobile. | ||
| 3. Sparsity [algorithms](./torchao/sparsity) such as Wanda that help improve accuracy of sparse networks | ||
| 4. Integration with other PyTorch native libraries like torchtune and ExecuTorch | ||
| ## Our Goals |
There was a problem hiding this comment.
@supriyar I also restructured the README a bit, please take a look
jerryzh168
commented
Apr 18, 2024
| 2. While these techniques are designed to improve model performance, in some cases the opposite can occur. This is because quantization adds additional overhead to the model that is hopefully made up for by faster matmuls (dynamic quantization) or loading weights faster (weight-only quantization). If your matmuls are small enough or your non-quantized perf isn't bottlenecked by weight load time, these techniques may reduce performance. | ||
| 3. Use the PyTorch nightlies so you can leverage [tensor subclasses](https://pytorch.org/docs/stable/notes/extending.html#subclassing-torch-tensor) which is preferred over older module swap based methods because it doesn't modify the graph and is generally more composable and flexible. | ||
|
|
||
| ## Get Started |
There was a problem hiding this comment.
@msaroufim @supriyar I added a get started section here and linked to the API READEMEs
Summary: att Test Plan: / Reviewers: Subscribers: Tasks: Tags:
msaroufim
approved these changes
Apr 19, 2024
dbyoung18
pushed a commit
to dbyoung18/ao
that referenced
this pull request
Jul 31, 2024
Summary: att Test Plan: / Reviewers: Subscribers: Tasks: Tags:
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
att
Test Plan:
/
Reviewers:
Subscribers:
Tasks:
Tags: