HLS/TS multi-audio A/V sync

[rauldelperal]

Have you read the FAQ and checked for duplicate open issues?
Yes

What link can we use to reproduce this?
https://shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;uilang=es-ES;panel=CUSTOM%20CONTENT;build=uncompiled
https://shaka-player-demo.appspot.com/demo/#audiolang=es-ES;textlang=es-ES;uilang=es-ES;asset=https://directes-test.ccma.cat/live-origin/tv3-hls/master.m3u8;panel=CUSTOM%20CONTENT;build=uncompiled

What version of Shaka Player are you using?
v4.2.2-uncompiled

What browser and OS are you using?
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36"

What did you do?
We don´t have added any changes on our side. We use the same Encoder version.
Since you have added to your v4 releases the commit "36d0b54" there is an A/V sync error playing an HLS/TS Multi-audio Livestream through Shaka-Player v4.X.X. If you try with older versions, this error does not appears.

What did you expect to happen?
Shaka-player should be able to A/V sync these type of HLS/TS Live Streams like others players do.

What actually happened?
An a clear A/V sync error.

[joeyparrish]

Thank you for the report. We'll look into it!

[joeyparrish]

Possible duplicate of #4589

[JBurgosCires21]

Hi! Is there any update on this issue?

Regarding your last comment @joeyparrish, we do not think this is exactly a duplicate of that issue, because in our case there is no need of any ad break, the issue is always there. You can see we are emitting an HLS stream correctly formed in this URL, and the desynchronization can be seen in any version of shaka player that includes the commit 36d0b54.
You can check it directy here.

[theodab]

For whatever reason, it looks like some of the changes from 89409ce (the PR that should be responsible for A/V sync after 36d0b54) were undone. I believe there must have been a problem when merging with 47fa309, which also changes the logic behind appending buffers.
Because of that, we read EXT-X-PROGRAM-DATE-TIME from the manifest and store it internally, but never actually use it for anything besides logs.

[theodab]

Okay, I think I was wrong in my previous statement. I had forgotten about 314a987, which changed how EXT-X-PROGRAM-DATE-TIME was handled, so I had been looking in the wrong part of the code.

I did a git bisect, and I think the actual problem is #4338.
I did a few manual test runs, where I downgraded mux.js to v6.1.0 and looked for A/V sync issues. I don't... think I am seeing the issues anymore with mux.js downgraded, though I will admit that I am not that great at telling if someone's mouth matches up to the sound of their voice when they're speaking a language I don't.

That's where I'll have to sign off on this for now. I'll be going on break after today.

[joeyparrish]

Taking over for @theodab. Upgrading to P0, hoping to fix this by the end of the week (which is the end of the year given holidays).

[joeyparrish]

It's time-consuming to bisect this issue, because ad breaks tend not to have visible speech, and use voiceovers or on-screen titles instead. (Presumably because these are easier to localize and run across regions.)

[joeyparrish]

And some programs are dubbed foreign TV shows. 😢

I'm frequently unable to accurately judge lip sync on this content. But what I've seen so far is that I can't tell any difference between any combination of main, v4.0.0, mux.js v5.14, and mux.js v6.2. All appear off, but only very slightly (sub-second, can't judge how far).

So I'm analyzing the segments for now instead. I'm checking to see if the contents of the TS segments agree with the metadata in the HLS playlists.

Playlists all state that segments are exactly 6 seconds long, and PROGRAM-DATE-TIME is specified to the millisecond, but always ends with .000.

[joeyparrish]

Presentation timestamps in the media do not agree with PROGRAM-DATE-TIME metadata.

For example:

bitrate_3.m3u8:

#EXTM3U
#EXT-X-TARGETDURATION:6
#EXT-X-VERSION:4
#EXT-X-MEDIA-SEQUENCE:278479923
#EXT-X-PROGRAM-DATE-TIME:2022-12-12T19:12:17.000+0000
#EXTINF:6,
0/br3/221117115102/92827/geo2-br3_278479923.ts

Actual PTS: 7803148816

bitrate_7.m3u8:

#EXTM3U
#EXT-X-TARGETDURATION:6
#EXT-X-VERSION:4
#EXT-X-MEDIA-SEQUENCE:278479924
#EXT-X-PROGRAM-DATE-TIME:2022-12-12T19:12:22.000+0000
#EXTINF:6,
0/br7/221117115102/92827/geo2-br7_278479924.ts

Actual PTS: 7803561136

The PROGRAM-DATE-TIME metadata says these should be played at 5s apart. However, the timestamps are 4.58133 seconds apart. This would suggest desync of about 419 ms, which is roughly consistent with what I observed in the native-Catalan programs I was able to observe this morning.

7803561136 - 7803148816 = 412320
412320 / 90k = 4.58133

As far as I can tell, inaccuracy in the playlist's PROGRAM-DATE-TIME is to blame for this content being out of sync.

The reason v3 played this content is because we used a much higher-latency startup process to extract timestamps from media instead of MSE sequence mode, and we used that extracted timestamp information instead of PROGRAM-DATE-TIME to sync the content.

@rauldelperal, can you confirm my findings?

Lowering priority, tentatively marking this as bad content. I will prioritize #4589 now (previously suspected of being a duplicate), in case there is a separate AV sync issue there.

[github-actions]

Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including @shaka-bot reopen in a comment.