Is there any plan to support exllama?

[yarray]

Although llama.cpp can now support GPU via cublas, it seems that exllama runs times faster if with a good enough GPU (3090 as an example). Is there any plan to support exllama, or in general, other loaders to load LLM?

[tmm1]

@mudler could there theoretically be a grpc backend implemented in python that used exllama directly?

[mudler]

Yes, definetly - in the same way now python embeddings could be an option as well

[fblissjr]

If there's an modular way to bring in GPTQ models via exllama, autogptq (https://github.com/PanQiWei/AutoGPTQ/), I'd love to use LocalAI - have only used other options (like ooba's openai extension) because of the GPTQ support.

[mudler]

yes now it's possible - I'm converting this as an issue to track it!

[mudler]

created: #796

[mudler]

JFYI: coming soon #881