Player stalls on stream with audio gaps

[erankor]

What version of Hls.js are you using?

demo v0.14.5

What browser and OS are you using?

Chrome 84.0.4147.89

Test stream:

http://lbd.kaltura.com:8001/audio-gaps/index-s1.m3u8

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client on https://hls-js.netlify.com/demo and not just on my page

• The issue occurs in the latest client on https://hls-js-dev.netlify.com/demo and not just on my page

• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Please provide clear steps to reproduce your problem
2. If the bug is intermittent, give a rough frequency

Play the test stream pasted above

Expected behavior

What you expected to happen
The stream has 20 sec of video+audio, 20 sec video-only, 12 sec video+audio, the expected behavior is that the video will play smoothly till the end, audio will of course become silent during the 20 sec video-only part.
This is the behavior seen when playing the stream on iPhone/Safari.

Actual behavior

What actually happened
The player stalls once it reaches the video-only part (slightly before 20 sec)

Console output

[log] > Loading 8 of [3 ,12],level 0, currentTime:2.005,bufferEnd:19.899
logger.js:40 [log] > main stream-controller: IDLE->FRAG_LOADING
logger.js:40 [log] > Loaded 8 of [3 ,12],level 0
logger.js:40 [log] > Parsing 8 of [3 ,12],level 0, cc 1
logger.js:40 [log] > main stream-controller: FRAG_LOADING->PARSING
logger.js:40 [log] > Parsed video,PTS:[25.633,31.633],DTS:[25.500/31.498],nb:90,dropped:0
logger.js:40 [log] > main stream-controller: PARSING->PARSED
logger.js:40 [log] > main buffered : [0.023,19.899]
logger.js:40 [log] > latency/loading/parsing/append/kbps:262/309/6/2/921
logger.js:40 [log] > main stream-controller: PARSED->IDLE
logger.js:40 [warn] > Playback stalling at @19.837612 due to low buffer (buffer=0.06186500000000095)
(anonymous) @ logger.js:40
_reportStall @ gap-controller.js:161
poll @ gap-controller.js:107
_checkBuffer @ stream-controller.js:1296
doTick @ stream-controller.js:136
tick @ task-loop.ts:111
setInterval (async)
setInterval @ task-loop.ts:73
startLoad @ stream-controller.js:60
(anonymous) @ hls.ts:301
startLoad @ hls.ts:300
onManifestParsed @ stream-controller.js:767
onEventGeneric @ event-handler.ts:82
onEvent @ event-handler.ts:69
emit @ index.js:203
trigger @ observer.ts:13
onManifestLoaded @ level-controller.js:150
onEventGeneric @ event-handler.ts:82
onEvent @ event-handler.ts:69
emit @ index.js:203
trigger @ observer.ts:13
_handleTrackOrLevelPlaylist @ playlist-loader.ts:382
loadsuccess @ playlist-loader.ts:265
readystatechange @ xhr-loader.js:118
XMLHttpRequest.send (async)
loadInternal @ xhr-loader.js:80
load @ xhr-loader.js:38
load @ playlist-loader.ts:235
onManifestLoading @ playlist-loader.ts:132
onEventGeneric @ event-handler.ts:82
onEvent @ event-handler.ts:69
emit @ index.js:203
trigger @ observer.ts:13
loadSource @ hls.ts:288
loadSelectedStream @ main.js:249
(anonymous) @ main.js:76
dispatch @ jquery.min.js:3
r.handle @ jquery.min.js:3
main.js:565 Error event: {type: "mediaError", details: "bufferStalledError", fatal: false, buffer: 0.06186500000000095}```

For media errors reported on Chrome browser, please also paste the output of chrome://media-internals

[robwalch]

It is expected behavior for the player to stall on buffer gaps (even if only momentarily before pushing through small gaps).

Is the actual expected behavior here that we fill those gaps with silence? Could you re-title this issue as "Audio gaps caused by segments missing audio filled with silence"?

[erankor]

Thanks @robwalch, yes, the expected behavior is that the player would be silent during this period (this is what iPhone does).

I'm not familiar with hls.js internals, but I guess there are 2 approaches to solving this -

1. Generating silence frames
2. Re-initializing the player in "video-only mode"
   Sounds to me like handle xhr timeout and i/o errors #2 may be a better solution, since it will probably solve the reverse case - audio continues while video stops (this scenario also works fine in iPhone - the video just freezes on the last frame).

Note that the m3u8 has EXT-X-DISCONTINUITY at the point where audio stops, this is required for smooth playback on iPhone - if I run a test live stream with audio gaps without discontinuity, the iPhone player will stall at the point where the audio is lost - it keeps loading more and more segments in the hope to find some audio in them. When I use discontinuity, it understands no audio is going to arrive, and just plays video-only. When the audio comes back, there is no need for discontinuity, it plays fine without it as well.

Regarding the title of the issue, maybe I should change it to 'audio/video gaps', because the behavior on iPhone is completely symmetrical for missing audio/video. If we can have a solution that solves both, that would be great.

One last note - when I ran my tests on iPhone, I also tested separate playlists for audio/video (using EXT-X-MEDIA). This changes things a bit, because instead of moving from video+audio segments to video-only segments, what happens is that some audio segments are just missing. I checked with Apple about this, and the recommendation is to mark the missing segments with EXT-X-GAP. According to them, all Apple operating systems release since late 2017 are "gap-aware" and will not even try to load these segments.
I tested such a stream on hls.js, and the behavior was the same as muxed segments - it tried to load the segments marked with EXT-X-GAP and stalled. I can provide a sample stream for that as well, if it helps.

[robwalch]

Hi @erankor,

hls.js can fill audio gaps within a segment that is supposed to contain audio, but it does not for video-only segments in this scenario (with discontinuity). It should - it needs to - to avoid stalling during playback. There's no option that I know of that will simply play over MSE source buffered AV where one of the tracks contains (large) gaps.

Thanks for the sample stream and detailed explanation. Have a look at the mp4-remuxer. That is where audio silence is inserted (or in this case, not being inserted).

[erankor]

"There's no option that I know of that will simply play over MSE source buffered AV where one of the tracks contains (large) gaps." - my thought was that maybe it can somehow reinitialize in 'video-only' mode, but I'm not familiar with MSE enough...
To put it differently, I'm assuming that when playing a video-only stream, you don't generate silence, you probably just initialize MSE only with the video. If the same can be done here, it may also be a solution.

Thanks!

[robwalch]

Hi @erankor,

While I'm no longer able to access the stream, if I recall, this kind of situation would require the use of GAP tags to be properly supported and to adhere to the HLS spec. If you'd like us to reopen, please provide another test stream and use case.

If you are interested in EXT-X-GAP tag support please +1 #2940.