Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement batch size for speed test #26

Merged
merged 6 commits into from
Sep 5, 2023
Merged

Conversation

casper-hansen
Copy link
Owner

Added data parallelism in the speed test which is specified through the batch_size argument. However, you have to disable the fused_layers option currently as we work out the bug in #20.

python -m awq.entry --entry_type speed --model_path casperhansen/vicuna-7b-v1.5-awq --quant_file awq_model_w4_g128.pt --batch_size 2 --disable_fused_layers

@casper-hansen casper-hansen merged commit 85430dd into main Sep 5, 2023
@casper-hansen casper-hansen deleted the speed_batch_size branch September 6, 2023 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant