Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A/V sync problems with unaligned HLS streams #4308

Closed
julianaCastro opened this issue Jun 22, 2022 · 6 comments · Fixed by #4528
Closed

A/V sync problems with unaligned HLS streams #4308

julianaCastro opened this issue Jun 22, 2022 · 6 comments · Fixed by #4528
Assignees
@JulianDomingo
Labels
component: HLS The issue involves Apple's HLS manifest format priority: P0 Broken for everyone; no workaround; urgent status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Milestone
v4.3

Comments

@julianaCastro
Copy link

Have you read the FAQ and checked for duplicate open issues?
Yes

What version of Shaka Player are you using?
v4.1.1

Can you reproduce the issue with our latest release version?
Yes

Are you using the demo app or your own custom app?
Demo App

What browser and OS are you using?
GoogleChrome, Firefox, Ubuntu

I have a media content with video and audio encoded separately and different rates. During its streaming, sometimes when an adaptation event occurs and bitrate is changed , shaka is repeating the download and playback of the last video segment, with a different rate, that is leading video and audio out of sync.

I've experienced this behaviour on shaka player demo and on a custom application using v4.1.1 and v3.4.0 on chrome and firefox, and changing the bandwidth using the browser network tools. I also made changes in the main playlist to be similar to a hls main playlist example provided by apple at https://devstreaming-cdn.apple.com/videos/streaming/examples/img_bipbop_adv_example_ts/master.m3u8, but it didn't solve the problem.

Link to the main playlist of the encoded video:
https://storage.googleapis.com/akiry-public/video-and-audio-4s/upload_41af1db0-eb71-11ec-85da-710193d27b0f.m3u8

Screenshot of shaka player demo
Screenshot from 2022-06-20 21-56-47
@julianaCastro julianaCastro added the type: bug Something isn't working correctly label Jun 22, 2022
@github-actions github-actions bot added this to the v4.2 milestone Jun 22, 2022
@avelad avelad added the component: HLS The issue involves Apple's HLS manifest format label Jun 22, 2022
@avelad
Copy link
Member

avelad commented Jun 22, 2022

@joeyparrish can you review it? Thanks!

@joeyparrish
Copy link
Member

I can reproduce the issue. I'll take a closer look at the content. I suspect that this is a problem with your encoding, but I'll try to verify.

@joeyparrish joeyparrish added the priority: P1 Big impact or workaround impractical; resolve before feature release label Jun 27, 2022
@joeyparrish
Copy link
Member

Okay, I understand the problem now. This would occur with any content whose video segments or audio segments are not all aligned across playlists.

For example, the first four segment lengths across 3 different streams:

x 360p 720p 1080p
seg 0 4.000 4.066 4.000
seg 1 4.000 4.033 4.000
seg 2 4.000 3.966 4.000
seg 3 4.000 4.000 4.000

Switching between 360p and 1080p would be fine, but between 360p and 720p or 720p and 1080p would trigger the issue.

This appears to be a bug in StreamingEngine, which doesn't account for adaptation. On adaptation, just as on seek, we need to ask MediaSourceEngine to reset the timestamp offset, to ensure the new content is correctly placed on the timeline.

@joeyparrish joeyparrish self-assigned this Jun 27, 2022
@joeyparrish joeyparrish changed the title Shaka repeats download of segment on adaptation, video and audio go out of sync A/V sync problems with unaligned HLS streams Jun 27, 2022
@avelad avelad modified the milestones: v4.2, v4.3 Aug 17, 2022
@martinstark
Copy link
Contributor

We've had to turn off ad stitching and we think this issue is the root cause. Just wanted to add my voice to the issue, and it's pretty serious for us.

@joeyparrish joeyparrish added priority: P0 Broken for everyone; no workaround; urgent and removed priority: P1 Big impact or workaround impractical; resolve before feature release labels Sep 21, 2022
@joeyparrish
Copy link
Member

Thanks. I bumped the priority and tentatively assigned the issue to @JulianDomingo. I'll take it back if Julian doesn't have the bandwidth for it.

@martinstark
Copy link
Contributor

We could probably help with providing test streams if needed (or using yospace hls cmaf reference streams should work).

theodab added a commit to theodab/shaka-player that referenced this issue Sep 29, 2022
@theodab
fix(hls): Handle unaligned HLS streams on VOD
3d37535 
This reverts the HLS parser to using per-stream mediaSequenceToStartTime
values for VOD, which will allow us to play unaligned HLS streams on
that sort of content.

Issue shaka-project#4308
@theodab theodab mentioned this issue Sep 29, 2022
Merged
@JulianDomingo JulianDomingo mentioned this issue Sep 29, 2022
Merged
@avelad avelad linked a pull request Sep 30, 2022 that will close this issue
fix: Fix A/V sync in unaligned HLS VOD streams #4528
Merged
theodab added a commit that referenced this issue Oct 3, 2022
@theodab
fix(hls): Handle unaligned HLS streams on VOD
5349860 
This reverts the HLS parser to using per-stream mediaSequenceToStartTime
values for VOD, which will allow us to play unaligned HLS streams on
that sort of content.

Issue #4308
joeyparrish pushed a commit that referenced this issue Oct 3, 2022
@JulianDomingo
fix: Fix A/V sync in unaligned HLS VOD streams (#4528)
8c198df 
Fix is based on suggestions from @joeyparrish in
#4308 (comment).

During automatic adaptations, Shaka will now reset the timestamp offset
to ensure the newly active track is properly aligned in the
presentation.

Verified by disabling ABR and manually triggering variant switches
(`shaka.Player.selectVariantTrack()`) between seemingly problematic
combinations (e.g., `400k` bps => `6000k` bps stream). Behavior was
compared against production.

Since `adaptation` events are only triggered by ABR logic (and ABR was
disabled for manual testing), `selectVariantTrack()` logic was
temporarily changed from:

`this.switchVariant_(variant, /* fromAdaptation= */ false, clearBuffer,
safeMargin);`
=>
`this.switchVariant_(variant, /* fromAdaptation= */ true, clearBuffer,
safeMargin);`

to ensure the fixes proposed in this PR were taken into effect and being
used during manual testing. Tested content is from the reported bug,
located here:
#4308 (comment)

Closes #4308
JulianDomingo added a commit that referenced this issue Oct 6, 2022
@JulianDomingo
fix: Fix A/V sync in unaligned HLS VOD streams (#4528)
c7fae99 
Fix is based on suggestions from @joeyparrish in
#4308 (comment).

During automatic adaptations, Shaka will now reset the timestamp offset
to ensure the newly active track is properly aligned in the
presentation.

Verified by disabling ABR and manually triggering variant switches
(`shaka.Player.selectVariantTrack()`) between seemingly problematic
combinations (e.g., `400k` bps => `6000k` bps stream). Behavior was
compared against production.

Since `adaptation` events are only triggered by ABR logic (and ABR was
disabled for manual testing), `selectVariantTrack()` logic was
temporarily changed from:

`this.switchVariant_(variant, /* fromAdaptation= */ false, clearBuffer,
safeMargin);`
=>
`this.switchVariant_(variant, /* fromAdaptation= */ true, clearBuffer,
safeMargin);`

to ensure the fixes proposed in this PR were taken into effect and being
used during manual testing. Tested content is from the reported bug,
located here:
#4308 (comment)

Closes #4308
JulianDomingo added a commit that referenced this issue Oct 6, 2022
@JulianDomingo
fix: Fix A/V sync in unaligned HLS VOD streams (#4528)
8b17c7f 
Fix is based on suggestions from @joeyparrish in
#4308 (comment).

During automatic adaptations, Shaka will now reset the timestamp offset
to ensure the newly active track is properly aligned in the
presentation.

Verified by disabling ABR and manually triggering variant switches
(`shaka.Player.selectVariantTrack()`) between seemingly problematic
combinations (e.g., `400k` bps => `6000k` bps stream). Behavior was
compared against production.

Since `adaptation` events are only triggered by ABR logic (and ABR was
disabled for manual testing), `selectVariantTrack()` logic was
temporarily changed from:

`this.switchVariant_(variant, /* fromAdaptation= */ false, clearBuffer,
safeMargin);`
=>
`this.switchVariant_(variant, /* fromAdaptation= */ true, clearBuffer,
safeMargin);`

to ensure the fixes proposed in this PR were taken into effect and being
used during manual testing. Tested content is from the reported bug,
located here:
#4308 (comment)

Closes #4308
theodab added a commit that referenced this issue Oct 9, 2022
@theodab
feat(HLS): Lazy-load HLS media playlists (#4511)
b2f279d 
This changes the HLS parser so that the media playlists are only downloaded when the createSegmentIndex function for the associated stream is called.
Because there is some important information about HLS streams that is stored inside the media playlist, this also changes the player to call createSegmentIndex on the initial variant earlier in the load process, to make sure that information is available in time.
As of this change, we will now require HLS streams to be aligned (see #4308) for livestreams. VOD content can still
be unaligned.

Closes #1936
@fredrik-telia fredrik-telia mentioned this issue Oct 18, 2022
Closed
@github-actions github-actions bot added the status: archived Archived and locked; will not be updated label Dec 2, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@JulianDomingo JulianDomingo

Labels
component: HLS The issue involves Apple's HLS manifest format priority: P0 Broken for everyone; no workaround; urgent status: archived Archived and locked; will not be updated type: bug Something isn't working correctly
Projects
None yet
Milestone
v4.3
Development

Successfully merging a pull request may close this issue.

fix: Fix A/V sync in unaligned HLS VOD streams JulianDomingo/shaka-player
5 participants
@martinstark @joeyparrish @avelad @julianaCastro @JulianDomingo