Possibility to show the best guess each x time or a "step" argument as the model transcribes?

[DE-ZIX]

First of all, thank you for this implementation. It's the most performant one I've found so far that doesn't hallucinates in most hardwares in general (whispercpp in my experience serves for no good in terms of performance outside of apple hardware -.-).

I am trying to do something similar as this: https://github.com/ggerganov/whisper.cpp/tree/v1.4.2/examples/stream

I already achieved a possibility of transcribing chunks of 5 seconds of audio while having another thread saving the audio in a buffer. Since it transcribes 5 seconds of audio in ~1.8 seconds, it works fine for a "realtime" transcription application. In the example shown in that video from whispercpp, it also process chunks of audio with a length of 5 seconds, but it returns the best guess every 500ms with the "--step" argument. It's nice as some sort of feedback to show what is going on as the model transcribes the audio.

Is there a possibility to add a feature like this for the transcribe or generate_segments methods, maybe with a "step" (or some other name) argument and if that argument is present it will return the best guess at that time?

[guillaumekln]

I don't think this should be implemented in model.transcribe.

As far as I know the function whisper_full in whisper.cpp basically corresponds to model.transcribe here. And looking at the stream.cpp example, we see that all logics related to the "step" is happening outside of this function. So similarly you should be able to implement this behavior on top of model.transcribe.

[DE-ZIX]

I see, I was thinking this could be a possibility. However do you have any idea on how I could retrieve the best guess from the transcribe method without modifying it or implementing something around it? If I understood that method correctly, it can't return anything until an entire segment is processed (assuming for example we only have one, which is the majority of the cases for short amounts of audio ~few seconds). But I may be wrong.

One possibilty I tried was instead of sending chunks of 5 seconds of audio send 500ms of audio everytime, but in terms of performance that does not work since the thread to transcribe audio starts to become slower than the one that records it, so it keeps accumulating audio in the buffer and the transcribing thread doesn't keep up.

[guillaumekln]

If I understood that method correctly, it can't return anything until an entire segment is processed (assuming for example we only have one, which is the majority of the cases for short amounts of audio ~few seconds).

That's true by default, but partial segments can be returned when using without_timestamps=True which seems to be set in this stream example:

https://github.com/ggerganov/whisper.cpp/blob/v1.4.2/examples/stream/stream.cpp#L138

One possibilty I tried was instead of sending chunks of 5 seconds of audio send 500ms of audio everytime, but in terms of performance that does not work since the thread to transcribe audio starts to become slower than the one that records it, so it keeps accumulating audio in the buffer and the transcribing thread doesn't keep up.

It seems to me that the stream example is running a transcription every 500 ms, and it prints a warning when it cannot keep up:

https://github.com/ggerganov/whisper.cpp/blob/v1.4.2/examples/stream/stream.cpp#L229-L244

However, it uses beam_size=1 which is faster than the default value of 5 here.

It also uses max_tokens = 32 by default which seems to limit the transcription to 32 tokens. Currently there is no public option to change that here but you can change a model attribute (see below).

So there are multiple options to change to get the behavior closer to this example. Note that I did not try anything, I'm just compiling what I understand from this stream example.

# "* 2" because the value is divided by 2 internally to account for a possible context.
model.max_length = 32 * 2

model.transcribe(audio, language="en", beam_size=1, condition_on_previous_text=False, without_timestamps=True)