Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

4bit-QLora + Qwen2 72b + 16k cutoff_len #5798

Open
lmc8133 opened this issue Oct 23, 2024 · 4 comments
Open

4bit-QLora + Qwen2 72b + 16k cutoff_len #5798

lmc8133 opened this issue Oct 23, 2024 · 4 comments
Labels
pending This problem is yet to be addressed

Comments

@lmc8133
Copy link

lmc8133 commented Oct 23, 2024

How many gpus are needed to finetune? I have tried 16 PPUs (96GB each) but got CUDA OUT OF MEMROY
@github-actions github-actions bot added the pending This problem is yet to be addressed label Oct 23, 2024
@hiyouga
Copy link
Owner

hiyouga commented Oct 23, 2024

try --enable_liger_kernel and --use_unsloth_gc

@lmc8133
Copy link
Author

lmc8133 commented Oct 24, 2024

try --enable_liger_kernel and --use_unsloth_gc

image --use_unsloth_gc or --use_unsloth?

@hiyouga
Copy link
Owner

hiyouga commented Oct 24, 2024

use_unsloth_gc

@lmc8133
Copy link
Author

lmc8133 commented Oct 25, 2024

use_unsloth_gc

Thanks.

BTW, I have encounter an error : Triton Error [CUDA]: device kernel image is invalid when --enable_liger_kernel.

Here are some pkg info:
triton==3.1.0
transformers==4.44.2
torch=2.3.0
CUDA SDK == 12.3.2

Any suggestions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pending This problem is yet to be addressed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hiyouga @lmc8133