Audio synchronization problems when converting from MP3 to WAV

[signsinthedust]

Whisper.cpp requires that audio files be in WAV format. I found, however, that the WAV files converted from MP3s are not synchronized with the MP3's audio. The WAV files seem to be anywhere between 6-8 seconds ahead of the audio of the corresponding MP3s.

Any suggestions on how to get around this?

[mrfragger]

most likely if the mp3 was cut at certain points and didn't reset the timestamps. So in other words if you have 5 segments from an original recording ffmpeg -i input.mp3 -c copy -f segment -segment_time 3:00 -reset_timestamps 1 test%02d.mp3
using the -reset_timestamps sets each segment to zero...that's my guess. Can you just re-encode the mp3s first and then convert them to wav? If so that's probably the best option.

[signsinthedust]

I am not sure, exactly, what you are recommending. My suspicion is that the problem is the sampling rate. The MP3s have a sampling rate of 44.1 kHz. Converting them to a 16 kHz WAV file, in addition to making them sound worse, desynchronizes the audio from the source MP3.

[signsinthedust]

This, by the way, is still an issue. Time stamps are simply not accurate.

[ulatekh]

If you're talking about the timestamps generated for subtitle formats such as .srt and .vtt, this is not specifically a whisper.cpp problem, it's a limitation with Whisper itself. Its timestamps are only accurate to the nearest whole second. There are derived projects, such as WhisperX, which use additional AI-driven models to try to narrow down the timestamp values.