Is it possible to convert to tensorrt?

[catalwaysright]

Since I noticed that it can be exported to onnx, I wonder if it supports tensorrt conversion.

[Y-T-G]

I was able to convert the encoder to TensorRT without any issues. But I have trouble converting the decoder to TensorRT.

[Y-T-G]

@Anjum48 There's another complicated way of doing the conversion using KV cache here. @pommedeterresautee explains how he did it fo T5.

[Y-T-G]

@catalwaysright I see.

I am looking forward to Kernl support for Whisper. ELS-RD/kernl#122

[Anjum48]

Why do you need to use a long audio for conversion? Wouldn't the result be the same?

The results are not the same. Steps to reproduce:

1. Create an ONNX model with the 4-sec sample clip.
2. Transcribe a clip longer than 4 seconds in the Gradio app and note the results
3. Create an ONNX model using an input clip > 30 seconds.
4. Repeat step 2

You'll see that the transcription from step 2 is truncated to the first few seconds, whereas step 4 has the full 30-second transcription. My suspicion is that the difference is caused by the while loop breaking as soon as there is silence/padding during graph creation, so the full clip is not processed.

[Y-T-G]

@Anjum48 I think the ONNX export script needs to be modified. It's exporting the whole Whisper model. You could try splitting it up to export the encoder and decoder part separately. That way the loop is not involved in the graph. cc: @FamousDirector

[FamousDirector]

Glad to see the FastWhisper repo is generating some discussion. Its implementation is certainly not production ready and should only serve as some general guidance.

To answer some questions:

• Yes, the decoder and encoder could be individually separated and converted, then used in tandem. Originally, I wanted it to be a single engine hence what I did. You'll likely want to compile the Python code or move it to C++ to maintain a higher speed. Also, may want to look at data on the GPU to reduce inter-device communication time.
• The original Whisper implementation also only does 30 sec at a time. See here: https://github.com/openai/whisper/blob/main/whisper/transcribe.py#L174 - you could implement something similar with a single engine of smaller length, feeding in the previous tokens, thus not needing as long as an input