return stride information in callback from speech recognition #605
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the whisper web repo, I noticed that time stamp generation was off while being streamed in - but correct after the transcription is complete. If you look at the demo video, and pause during generation, you will see a 1:02 timestamp for a 1 minute audio clip - which is overwritten upon completion to 0:57.
This is because timestamps are generated correctly when they are passed in as completed chunks, but are off when partial updates are streamed back from the model generation. The callback from completed chunks in
pipelines.js
contains stride information, but the callback during partial generation inmodels.js
does not, and stride information is needed to get correct timestamps from thedecode_asr
function intokenizers.js
.This is not a major issue right now, since the timestamps are only off by 5s, and the error is transient and corrects itself. However I want to make some improvements on whisper-web, like adding word level timestamps, making decoding more efficient, and adding streaming audio transcriptions from the microphone... and the timestamp discrepancies matter more in those code changes. I can do a workaround to solve this error in a fork of
whisper-web
, but making the change here is cleaner. If this code change is merged in, the timestamp issue can be mitigated by addinglast.stride = item.stride;
after line 134 inworker.js
inwhisper-web
- I can open a PR for that