4x times slower inference time with a finetuned model

[Abner-01]

I've been finetuning some whisper models with the base tutorial of hugging face: https://huggingface.co/blog/fine-tune-whisper. As expected, I get the added_tokens.json, vocab.json, and merges.txt, but when I use the provided command of ct2, the performance is x4 slower than the base faster whisper model. If I remove the added_tokens.json file, the speed will remain slow, but not four times slower. What I'm missing? This is the ct2 transformation command : ct2-transformers-converter --model . --output_dir faster-whisper --copy_files added_tokens.json special_tokens_map.json tokenizer_config.json vocab.json --quantization float16

[trungkienbkhn]

Could you provide your fine-tuned model on HF hub ?
An example of fine-tuned model conversion:

ct2-transformers-converter --model sanchit-gandhi/distil-whisper-large-v3-de-kd --output_dir distil-whisper-large-v3-de-kd-ct2 --copy_files tokenizer.json preprocessor_config.json --quantization float16

Normally there is no need to copy added_tokens.json file, just tokenizer.json + preprocessor_config.json are enough.

[mudabek]

I am facing the same issue. When I just load original model it's fast. But fine-tuned model is not faster

import os
import time

os.environ["OMP_NUM_THREADS"] = "6"

import whisper
import faster_whisper

model = whisper.load_model("medium", device="cpu")

faster_model = faster_whisper.WhisperModel("faster-medium", device="cpu", compute_type="int8")


def run_openai_whisper():
    result = model.transcribe(
        "voice_note.wav",
        beam_size=1,
        word_timestamps=True,
        initial_prompt="Welcome to the software engineering courses channel.",
    )

def run_faster_whisper():
    segments, _ = faster_model.transcribe(
        "voice_note.wav",
        beam_size=1,
        word_timestamps=True,
        initial_prompt="Welcome to the software engineering courses channel.",
    )
    segments = list(segments)
    


run_openai_whisper()
start = time.time()
run_openai_whisper()
end = time.time()
print("openai-whisper: %f seconds" % (end - start))

run_faster_whisper()
start = time.time()
run_faster_whisper()
end = time.time()
print("faster-whisper: %f seconds" % (end - start))

[Herz3h]

I confirm this slow behavior on finetuned models, tested with already converted model: https://huggingface.co/bofenghuang/whisper-large-v3-french-distil-dec16