Add GPTQ via Transformers. [Basic]

[digisomni]

This enables support for GPTQ via Transformers. Seems the cleanest and most efficient way to do things.

Also updated format.sh to allow 'greater than or equal to' tool verisons.

Note: Perhaps the old way of quanting can be deprecated then as the maintainer recommends using AutoGPTQ. Alternatively, if there's a compelling reason to also have manual support, then the GPTQ module should be upgraded to use AutoGPTQ.

Closes #2215 #1745 #1671 #2375

Warning: This alters the package requirements for FastChat.

[digisomni]

Due to some bugs with functionality, I refactored the model_adapter quite a bit. Could use further refactoring I think but this is a start.

Please note this does affect almost all adapters, so proper code review + testing where possible should be employed.

[leonxia1018]

I tried your branch with Exllama enabled but get this error:

Really need Exllama kernel, as it much faster than the default kernel.

[digisomni]

Official documentation recommends it to be off. I am not entirely sure the source of why this happens, would need to investigate in order to sort out how / when it can be on.

[digisomni]

Relevant notes are here: https://huggingface.co/docs/transformers/main/main_classes/quantization

[digisomni]

Updated from master.

[digisomni]

Bump. :)

[merrymercy]

@digisomni Sorry for the delay. This is a big refactor. Please allow some time for me to review.
Could you rebase to the main following the recent updates (#2512, #2559 )?

[digisomni]

Should be good to go now.

[surak]

@merrymercy looks like @digisomni did their part to have it working. Can we merge it?

[digisomni]

I've rebased again. I recommend review + test + merging this sooner rather than later as people merge their PRs with adapter mods, it tends to muddy the waters as they're building on the old paradigm. This makes it harder to rebase the more time that elapses from the last rebase.

[digisomni]

I am closing this refactor in favor of using the vLLM worker. If at some point we need to push out features faster than vLLM, we can continue trying to maintain FastChat's native model worker, but for now it seems vLLM is going to be faster to the punch.

[merrymercy]

@digisomni I am sorry to hear that, but we have very limited bandwidth.
Currently, our focus is on SGLang worker #2928

You are welcome to try it as well. The default HF worker seems not well suited for high-performance deployment.

[surak]

@digisomni Really, try SGLang. I already moved as much as I can to it (except Mixtral, which I haven't got running yet) to SGLang, as it's so much better than VLLM and not even comparable to the original worker.

[merrymercy]

@surak could you submit the issues you got with SGLang worker?