Preprocessing audio for improved transcriptions (telephone stereo call 8k sr)

[Haxeebraja]

Are there any recommendation to preprocess an audio for improving transcriptions.
As sometimes there are missing parts in transcription and hallucination.

[EtienneAb3d]

@Haxeebraja
See this discussion:
openai/whisper#679
And this experimental code:
https://github.com/EtienneAb3d/WhisperHallu

[EtienneAb3d]

How to get timestamps along with transcriptions?

Add the flag addSRT, see here:
https://github.com/EtienneAb3d/WhisperHallu/blob/319dba323b30adda3b227da2d122e5263fd19e73/transcribeHallu.py#L158

[Haxeebraja]

if addSRT = True, what is the use of [line 274] as https://github.com/EtienneAb3d/WhisperHallu/blob/319dba323b30adda3b227da2d122e5263fd19e73/transcribeHallu.py#L276 calls the same function again.

when language is arabic (ar), I do not get timestamps as it sets mode =0.
https://github.com/EtienneAb3d/WhisperHallu/blob/319dba323b30adda3b227da2d122e5263fd19e73/transcribeHallu.py#L294

I tried the following modification but then get the result has hallucinations:
removed ar from above expression [line 294]
added OKW and KOW markers for arabic by copying/renaming es files.

[EtienneAb3d]

if addSRT = True, what is the use of [line 274] as https://github.com/EtienneAb3d/WhisperHallu/blob/319dba323b30adda3b227da2d122e5263fd19e73/transcribeHallu.py#L276 calls the same function again.

To avoid hallucinations, WhisperHallu is removing silence and noise parts. Thus, the timestamps are not accurate.

To get accurate timestamps without hallucination, WhisperHallu is processing the file twice, with and without cut, then I fuse the timestamp to the text using WhisperTimeSync:
https://github.com/EtienneAb3d/WhisperTimeSync

when language is arabic (ar), I do not get timestamps as it sets mode =0. https://github.com/EtienneAb3d/WhisperHallu/blob/319dba323b30adda3b227da2d122e5263fd19e73/transcribeHallu.py#L294

I tried the following modification but then get the result has hallucinations: removed ar from above expression [line 294] added OKW and KOW markers for arabic by copying/renaming es files.

This test is not removing timestamps, but the voice markers. I didn't succeed in making it well working for the AR language. You can't simply copy the ES files.

[Haxeebraja]

Whisper is also prone to giving different results on multiple runs. Is WhisperTimeSync able to run on 2 files with different transcriptions?

[EtienneAb3d]

@Haxeebraja
WhisperTimeSync is an aligner/synchronizer:

1. it find what words are more likely to match between two similar text with differences, then
2. produce an output with the timestamps of one file over the words of the other file.

So, yes, WhisperTimeSync is able to run with files with different transcriptions. It's its main goal.

[thibaultmol]

@guillaumekln could this be implemented to happen by default when you use faster-whisper?

[guillaumekln]

What are you referring to exactly?

[thibaultmol]

I meant: if I install faster-whisper. It would be nice if this feature/library was integrated with it. So that you can just use it with a - option