Quantized Model Performance

[Alumniminium]

The files were scored after removing all punctuation, [tokens], numbers from 0 to 9 rewritten to zero to nine, and everything lowercased.

Using the following python script WER + Levenshtein on the following (rap music) file The TV Screen.mp3 comparing to the manual transcript

Score: 6.51 WER: 0.02 Levenshtein: 13.00 - whisper-large-v0-q5_1.txt
Score: 6.51 WER: 0.02 Levenshtein: 13.00 - whisper-large-v2-q5_1.txt
Score: 7.51 WER: 0.02 Levenshtein: 15.00 - whisper-large-v0-q4_2.txt
Score: 7.51 WER: 0.02 Levenshtein: 15.00 - whisper-large-v2-q4_2.txt
Score: 12.52 WER: 0.04 Levenshtein: 25.00 - whisper-large-v0-q4_1.txt
Score: 12.52 WER: 0.04 Levenshtein: 25.00 - whisper-large-v2-q4_1.txt
Score: 13.52 WER: 0.04 Levenshtein: 27.00 - whisper-large-v0-q5_0.txt
Score: 13.52 WER: 0.04 Levenshtein: 27.00 - whisper-large-v2-q5_0.txt
Score: 14.52 WER: 0.04 Levenshtein: 29.00 - whisper-large-v0-q4_0.txt
Score: 14.52 WER: 0.04 Levenshtein: 29.00 - whisper-large-v2-q4_0.txt
Score: 18.03 WER: 0.05 Levenshtein: 36.00 - whisper-large-v1-q16.txt
Score: 18.03 WER: 0.05 Levenshtein: 36.00 - whisper-medium-q4_2.txt
Score: 18.53 WER: 0.05 Levenshtein: 37.00 - whisper-medium-q16.txt
Score: 19.03 WER: 0.05 Levenshtein: 38.00 - whisper-medium-q5_1.txt
Score: 20.53 WER: 0.06 Levenshtein: 41.00 - whisper-medium-q4_1.txt
Score: 21.53 WER: 0.06 Levenshtein: 43.00 - whisper-large-v1-q4_1.txt
Score: 21.53 WER: 0.06 Levenshtein: 43.00 - whisper-large-v1-q4_2.txt
Score: 21.53 WER: 0.06 Levenshtein: 43.00 - whisper-medium-en-q4_2.txt
Score: 22.03 WER: 0.06 Levenshtein: 44.00 - whisper-large-v0-q16.txt
Score: 23.03 WER: 0.06 Levenshtein: 46.00 - whisper-large-v1-q5_1.txt
Score: 24.03 WER: 0.07 Levenshtein: 48.00 - whisper-large-v1-q5_0.txt
Score: 24.03 WER: 0.07 Levenshtein: 48.00 - whisper-small-q16.txt
Score: 27.04 WER: 0.08 Levenshtein: 54.00 - whisper-large-v1-q4_0.txt
Score: 28.04 WER: 0.08 Levenshtein: 56.00 - whisper-small-en-q4_1.txt
Score: 28.04 WER: 0.08 Levenshtein: 56.00 - whisper-small-q5_1.txt
Score: 28.54 WER: 0.08 Levenshtein: 57.00 - whisper-small-q5_0.txt
Score: 31.04 WER: 0.09 Levenshtein: 62.00 - whisper-small-q4_2.txt
Score: 31.54 WER: 0.09 Levenshtein: 63.00 - whisper-small-en-q5_0.txt
Score: 32.55 WER: 0.09 Levenshtein: 65.00 - whisper-small-q4_1.txt
Score: 33.55 WER: 0.09 Levenshtein: 67.00 - whisper-medium-q4_0.txt
Score: 35.55 WER: 0.10 Levenshtein: 71.00 - whisper-small-en-q4_0.txt
Score: 42.06 WER: 0.12 Levenshtein: 84.00 - whisper-small-q4_0.txt
Score: 48.57 WER: 0.14 Levenshtein: 97.00 - whisper-medium-en-q4_1.txt
Score: 49.07 WER: 0.14 Levenshtein: 98.00 - whisper-small-en-q5_1.txt
Score: 51.07 WER: 0.14 Levenshtein: 102.00 - whisper-medium-en-q5_0.txt
Score: 51.57 WER: 0.14 Levenshtein: 103.00 - whisper-base-en-q5_1.txt
Score: 52.57 WER: 0.15 Levenshtein: 105.00 - whisper-medium-en-q16.txt
Score: 54.07 WER: 0.15 Levenshtein: 108.00 - whisper-base-en-q16.txt
Score: 54.08 WER: 0.15 Levenshtein: 108.00 - whisper-tiny-en-q16.txt
Score: 56.58 WER: 0.15 Levenshtein: 113.00 - whisper-base-q5_1.txt
Score: 58.08 WER: 0.16 Levenshtein: 116.00 - whisper-base-en-q5_0.txt
Score: 59.58 WER: 0.16 Levenshtein: 119.00 - whisper-base-en-q4_2.txt
Score: 60.08 WER: 0.16 Levenshtein: 120.00 - whisper-base-en-q4_1.txt
Score: 62.08 WER: 0.17 Levenshtein: 124.00 - whisper-base-q16.txt
Score: 63.59 WER: 0.17 Levenshtein: 127.00 - whisper-base-q4_1.txt
Score: 64.09 WER: 0.18 Levenshtein: 128.00 - whisper-tiny-en-q5_1.txt
Score: 67.09 WER: 0.18 Levenshtein: 134.00 - whisper-base-q4_2.txt
Score: 72.10 WER: 0.20 Levenshtein: 144.00 - whisper-base-q5_0.txt
Score: 76.10 WER: 0.21 Levenshtein: 152.00 - whisper-base-q4_0.txt
Score: 77.61 WER: 0.22 Levenshtein: 155.00 - whisper-tiny-q16.txt
Score: 82.62 WER: 0.23 Levenshtein: 165.00 - whisper-tiny-en-q4_2.txt
Score: 83.62 WER: 0.24 Levenshtein: 167.00 - whisper-tiny-q5_1.txt
Score: 87.62 WER: 0.25 Levenshtein: 175.00 - whisper-tiny-q5_0.txt
Score: 88.62 WER: 0.25 Levenshtein: 177.00 - whisper-tiny-en-q4_0.txt
Score: 102.14 WER: 0.29 Levenshtein: 204.00 - whisper-tiny-en-q4_1.txt
Score: 113.16 WER: 0.32 Levenshtein: 226.00 - whisper-small-en-q4_2.txt
Score: 123.67 WER: 0.35 Levenshtein: 247.00 - whisper-tiny-q4_1.txt
Score: 311.44 WER: 0.88 Levenshtein: 622.00 - whisper-tiny-en-q5_0.txt
Score: 355.00 WER: 1.00 Levenshtein: 709.00 - whisper-base-en-q4_0.txt
Score: 355.00 WER: 1.00 Levenshtein: 709.00 - whisper-medium-en-q4_0.txt
Score: 355.00 WER: 1.00 Levenshtein: 709.00 - whisper-medium-en-q5_1.txt
Score: 355.00 WER: 1.00 Levenshtein: 709.00 - whisper-small-en-q16.txt
Score: 355.00 WER: 1.00 Levenshtein: 709.00 - whisper-tiny-q4_0.txt
Score: 355.00 WER: 1.00 Levenshtein: 709.00 - whisper-tiny-q4_2.txt

The filename of the best transcript is: whisper-large-v0-q5_1.txt

The lowest ranking models just output [music] or music and nothing else. Is there a random seed? There's no parameter to pass to whisper to set a seed like there is with llama as far as i can tell.

George_W_Bush_Columbia_FINAL.ogg

Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v0-q16.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v0-q4_1.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v0-q5_0.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v0-q5_1.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v1-q5_1.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v2-q4_1.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v2-q5_0.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-large-v2-q5_1.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-medium-en-q4_1.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-medium-en-q4_2.txt
Score: 3.01 WER: 0.02 Levenshtein: 6.00 - whisper-medium-en-q5_1.txt
Score: 3.51 WER: 0.02 Levenshtein: 7.00 - whisper-medium-en-q5_0.txt
Score: 3.51 WER: 0.02 Levenshtein: 7.00 - whisper-medium-q5_1.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-large-v0-q4_0.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-large-v1-q16.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-large-v1-q4_1.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-large-v2-q4_0.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-medium-en-q16.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-medium-en-q4_0.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-small-q4_0.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-small-q4_1.txt
Score: 4.01 WER: 0.02 Levenshtein: 8.00 - whisper-small-q5_1.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-large-v0-q4_2.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-large-v1-q4_0.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-large-v1-q5_0.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-large-v2-q4_2.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-small-en-q16.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-small-en-q4_2.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-small-en-q5_0.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-small-en-q5_1.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-small-q16.txt
Score: 4.51 WER: 0.02 Levenshtein: 9.00 - whisper-small-q5_0.txt
Score: 5.01 WER: 0.03 Levenshtein: 10.00 - whisper-medium-q4_2.txt
Score: 5.01 WER: 0.03 Levenshtein: 10.00 - whisper-small-en-q4_1.txt
Score: 5.51 WER: 0.03 Levenshtein: 11.00 - whisper-base-q5_0.txt
Score: 5.51 WER: 0.03 Levenshtein: 11.00 - whisper-large-v1-q4_2.txt
Score: 5.51 WER: 0.03 Levenshtein: 11.00 - whisper-medium-q16.txt
Score: 5.51 WER: 0.03 Levenshtein: 11.00 - whisper-medium-q4_0.txt
Score: 5.51 WER: 0.03 Levenshtein: 11.00 - whisper-small-en-q4_0.txt
Score: 6.02 WER: 0.03 Levenshtein: 12.00 - whisper-base-en-q16.txt
Score: 6.02 WER: 0.03 Levenshtein: 12.00 - whisper-base-en-q4_2.txt
Score: 6.02 WER: 0.03 Levenshtein: 12.00 - whisper-base-q5_1.txt
Score: 6.02 WER: 0.03 Levenshtein: 12.00 - whisper-tiny-en-q4_1.txt
Score: 6.52 WER: 0.03 Levenshtein: 13.00 - whisper-base-q4_2.txt
Score: 6.52 WER: 0.03 Levenshtein: 13.00 - whisper-medium-q4_1.txt
Score: 7.02 WER: 0.04 Levenshtein: 14.00 - whisper-base-q16.txt
Score: 7.02 WER: 0.04 Levenshtein: 14.00 - whisper-tiny-en-q4_2.txt
Score: 7.02 WER: 0.04 Levenshtein: 14.00 - whisper-tiny-en-q5_1.txt
Score: 7.52 WER: 0.04 Levenshtein: 15.00 - whisper-base-en-q4_1.txt
Score: 7.52 WER: 0.04 Levenshtein: 15.00 - whisper-base-en-q5_0.txt
Score: 7.52 WER: 0.04 Levenshtein: 15.00 - whisper-base-q4_0.txt
Score: 7.52 WER: 0.04 Levenshtein: 15.00 - whisper-base-q4_1.txt
Score: 7.52 WER: 0.04 Levenshtein: 15.00 - whisper-tiny-q5_0.txt
Score: 8.02 WER: 0.04 Levenshtein: 16.00 - whisper-tiny-en-q16.txt
Score: 8.02 WER: 0.04 Levenshtein: 16.00 - whisper-tiny-q5_1.txt
Score: 9.02 WER: 0.05 Levenshtein: 18.00 - whisper-tiny-en-q5_0.txt
Score: 9.53 WER: 0.05 Levenshtein: 19.00 - whisper-tiny-en-q4_0.txt
Score: 9.53 WER: 0.05 Levenshtein: 19.00 - whisper-tiny-q16.txt
Score: 10.03 WER: 0.05 Levenshtein: 20.00 - whisper-small-q4_2.txt
Score: 10.53 WER: 0.06 Levenshtein: 21.00 - whisper-base-en-q5_1.txt
Score: 12.03 WER: 0.06 Levenshtein: 24.00 - whisper-tiny-q4_1.txt
Score: 13.54 WER: 0.07 Levenshtein: 27.00 - whisper-tiny-q4_2.txt
Score: 15.04 WER: 0.08 Levenshtein: 30.00 - whisper-base-en-q4_0.txt
Score: 16.54 WER: 0.09 Levenshtein: 33.00 - whisper-tiny-q4_0.txt

[Alumniminium]

CONTRIBUTE

I'm hosting an API with all models @ http://spch.her.st/ !

Use this Script to POST a file to the API
./eval-models.sh filename.mp3

Use this Script to measure the accuracy
python measure.py manual-transcript.txt

please keep the file duration below 5min, the API is not very fast, especially with the large models it takes quite a while. also the API only processes one file at a time

[StuartIanNaylor]

How does q8_0 stack up as finding it the fastest on my Arm Based RK3588?

I have always wondered about the smaller models as you traverse from large to tiny WER does increase drastically.
At the lower end of hardware for load have wondered if a specific LM Wav2Vec2.cpp might be a better for WER?
Also if you can find a dataset for short command type files such as 'turn on the light' or conversational AI ...
As WER increases considerabilly when there is little sentence context.

[ggerganov]

The lowest ranking models just output [music] or music and nothing else. Is there a random seed?

No, the model and the samplers are deterministic

What is large-v0?
There is no large-v2-q16?

[Alumniminium]

The lowest ranking models just output [music] or music and nothing else. Is there a random seed?

No, the model and the samplers are deterministic

What is large-v0? There is no large-v2-q16?

large-v0 is just the 'large' model, as for large-v2: https://huggingface.co/openai/whisper-large-v2

[Alumniminium]

I'm still seeing quite considerable usage on my API and the eval script. Please post your results here for others to see. Don't be greedy.

Otherwise I'll have to shut it down and you have to clone and deploy the repo yourself.

[genevera]

Hey @Alumniminium I'm using it rn; Trying out parallel requests to see if it can get done faster. Hopefully it isn't frying your server! Also, will large-v3 end up available? Thank you so much for making the api! Results forthcoming.

[Alumniminium]

Hi, this old api can only process one request at a time. Large v3 is not available on it. Im currently working on an updated version with GPU acceleration.

Sadly i cant find time for it right now due to my job keeping me busy