Hugging Face open_asr_leaderboard

[chainyo]

Hi @guillaumekln, I just submitted a PR to add ctranslate2 and faster-whisper to the Hugging Face open_asr_leaderboard benchmark: huggingface/open_asr_leaderboard#10

Let me know if some things could be improved for both scripts calc_rtf.py and run_eval.py.

[chainyo]

Thanks for moving this to a discussion. I always forget this GitHub feature! 🤗

[guillaumekln]

Thanks for sharing!

Here are some comments:

In calc_rtf.py:

• You don't iterate over the segments so the transcription is not actually running (cf. note in the README about the output being a generator).
• Not sure what audio_dict["raw"] contains and whether it is accepted by faster-whisper. Have you checked that? Alternatively you can pass the input file directly or decode it outside the loop with faster_whisper.decode_audio (no need for librosa in this case).
• cuda:0 is not a valid device string for faster-whisper/CTranslate2. You should remove :0.

In run_eval.py:

• I suggest calling .strip() on the final transcription output, just in case.

In requirements_ctranslate2.txt:

• You could remove ctranslate2>=3.17 and let faster-whisper pull the correct version.

[chainyo]

I have corrected everything you suggested above.

And yes librosa.load() is returning one np.ndarray: https://librosa.org/doc/latest/generated/librosa.load.html#librosa-load

I keep librosa to stay aligned with the other scripts as much as possible as suggested in their README.

[guillaumekln]

Also I'm not sure it's relevant to mention CTranslate2 since the library is not used directly. Maybe you could rename the top-level directory to faster_whisper and the requirements file to requirements_faster_whisper.txt?