load whisper in float16 or int8, no external dependencies required #1990
phineas-pta
started this conversation in
Show and tell
Replies: 1 comment
-
Great tip, I have tried this with different content (large-v2, qint8) and the quality has been essentially the same. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
advantages of quantization (float16 or int8):
usually, it requires external libraries like:
faster-whisper
,transformers
+bitsandbytes
,whisper.cpp
BUT recent
torch
already has quantization built-in, so no need for external librariescredit: https://github.com/MiscellaneousStuff/openai-whisper-cpu
in case u dont have enough RAM/VRAM: quantize sequentially
Beta Was this translation helpful? Give feedback.
All reactions