You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
Having defaults high number of GPU layers doesn't always work. For instance big models can overfit the card and constrain the user to configure gpu_layers manually
Describe the solution you'd like
With libraries like https://github.com/gpustack/gguf-parser-go we could get along and identify beforeahead how much gpu vram could be used and adjust the default settings
Describe alternatives you've considered
Keep things as is
Additional context
The text was updated successfully, but these errors were encountered:
mudler
changed the title
automatically adjust default gpu_layers by available GPU memory
feat: automatically adjust default gpu_layers by available GPU memory
Sep 13, 2024
Is your feature request related to a problem? Please describe.
Having defaults high number of GPU layers doesn't always work. For instance big models can overfit the card and constrain the user to configure
gpu_layers
manuallyDescribe the solution you'd like
With libraries like https://github.com/gpustack/gguf-parser-go we could get along and identify beforeahead how much gpu vram could be used and adjust the default settings
Describe alternatives you've considered
Keep things as is
Additional context
The text was updated successfully, but these errors were encountered: