Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] We may need to remove max_memory arg #115

Closed
Qubitium opened this issue Jun 29, 2024 · 1 comment
Closed

[BUG] We may need to remove max_memory arg #115

Qubitium opened this issue Jun 29, 2024 · 1 comment
Assignees

Comments

@Qubitium
Copy link
Contributor

For very large models, multiple GPU may be needed for quantization but max_memory arg appears to be broken. Everything should be handled by accelerate and there should be no need for this arg. Investigate.

delete max_memory=max_memory can run.

Originally posted by @Xu-Chen in #48 (comment)

@CL-ModelCloud CL-ModelCloud self-assigned this Jul 1, 2024
@Qubitium
Copy link
Contributor Author

Qubitium commented Jul 1, 2024

We will remove this. Advanced users should just use and pass device_map fo accelerate. We should not be a arg modifying proxy for acclerate. Passing accelerate config args to accelerate is cleaner, more powerful, and we dont have to maintain compat.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants