Initial support for 8da4w QAT #138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
andrewor14
force-pushed
the
8da4w_qat
branch
2 times, most recently
from
5b1d404
to
044280a
Compare
cpuhrsch
approved these changes
Apr 16, 2024
cpuhrsch
reviewed
Apr 16, 2024
andrewor14
force-pushed
the
8da4w_qat
branch
6 times, most recently
from
653a07a
to
d5cd97f
Compare
jerryzh168
reviewed
Apr 16, 2024
| return (qmin, qmax) | ||
|
|
||
|
|
||
| def replace_linear_8da4w_qat( |
There was a problem hiding this comment.
this seems to be the same as replace_linear_8da4w, maybe we want to abstract out a helper function, to be less error prone
jerryzh168
approved these changes
Apr 16, 2024
andrewor14
force-pushed
the
8da4w_qat
branch
2 times, most recently
from
ae8bd77
to
83dd03a
Compare
Summary: This commit adds support for QAT, where linear layers are fake quantized with int8 per token dynamic activations (8da) and int4 grouped per channel weights (4w). This initial implementation uses the same module swap approach as 8da4w PTQ for simplicity and code reuse. In the future, we may wish to consider migrating both flows to use tensor subclasses for better composability with other PyTorch features. Test Plan: python test/quantization/test_qat.py -k test_fake_quantize_per_channel_group python test/quantization/test_qat.py -k test_fake_quantize_per_token python test/quantization/test_qat.py -k test_qat_8da4w_linear python test/quantization/test_qat.py -k test_qat_8da4w_quantizer Reviewers: jerryzh168, cpuhrsch, HDCharles Subscribers: jerryzh168, cpuhrsch, HDCharles, supriyar Tasks: #86
wat3rBro
reviewed
Apr 19, 2024
| @@ -1144,14 +1144,30 @@ def replace_linear_8da4w( | |||
| ), | |||
| ) | |||
| else: | |||
| replace_linear_8da4w( | |||
| _replace_linear_8da4w( | |||
| child, | |||
| groupsize, | |||
| padding_allowed, | |||
| precision, | |||
| scales_precision, | |||
| ) | |||
There was a problem hiding this comment.
Does it miss the linear_class? cc. @jerryzh168 @andrewor14
There was a problem hiding this comment.
I can fix it in fbcode first I guess
There was a problem hiding this comment.
oh my bad, let me submit a PR
dbyoung18
pushed a commit
to dbyoung18/ao
that referenced
this pull request
Jul 31, 2024
Summary: This commit adds support for QAT, where linear layers are fake quantized with int8 per token dynamic activations (8da) and int4 grouped per channel weights (4w). This initial implementation uses the same module swap approach as 8da4w PTQ for simplicity and code reuse. In the future, we may wish to consider migrating both flows to use tensor subclasses for better composability with other PyTorch features. Test Plan: python test/quantization/test_qat.py -k test_fake_quantize_per_channel_group python test/quantization/test_qat.py -k test_fake_quantize_per_token python test/quantization/test_qat.py -k test_qat_8da4w_linear python test/quantization/test_qat.py -k test_qat_8da4w_quantizer Reviewers: jerryzh168, cpuhrsch, HDCharles Subscribers: jerryzh168, cpuhrsch, HDCharles, supriyar Tasks: pytorch#86
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: This commit adds support for QAT, where linear layers are fake quantized with int8 per token dynamic activations (8da) and int4 grouped per channel weights (4w). This initial implementation uses the same module swap approach as 8da4w PTQ for simplicity and code reuse. In the future, we may wish to consider migrating both flows to use tensor subclasses for better composability with other PyTorch features.
Test Plan:
python test/quantization/test_qat.py -k test_fake_quantize_per_channel_group
python test/quantization/test_qat.py -k test_fake_quantize_per_token
python test/quantization/test_qat.py -k test_qat_8da4w_linear
python test/quantization/test_qat.py -k test_qat_8da4w_quantizer
Reviewers: jerryzh168, cpuhrsch, HDCharles
Subscribers: jerryzh168, cpuhrsch, HDCharles, supriyar
Tasks: #86