HLS should explicitly signal a lack of captions

[djeikyb]

NOTE: This issue was transferred from Shaka Player to Shaka Packager. See #922 (comment) and #922 (comment) for the most relevant details.

---

---

Have you read the FAQ and checked for duplicate open issues?

Yes

What version of Shaka Player are you using?

3.0.10, but I checked each release back to 3.0.0 and see the same behaviour.

Can you reproduce the issue with our latest release version?

Yes

Can you reproduce the issue with the latest code from master?

Yes

Are you using the demo app or your own custom app?

Custom, following the shaka ui basic usage tutorial

If custom app, can you reproduce the issue using our demo app?

Not sure, I wasn't able to run the demo app

What browser and OS are you using?

macos 10.14.6, safari 14.0.3 (14610.4.3.1.7)

For embedded devices (smart TVs, etc.), what model and firmware version are you using?

n/a

What are the manifest and license server URIs?

https://github.com/djeikyb/shaka_player_spurious_caption_bug_report/blob/main/2MinTest/master.m3u8

What configuration are you using? What is the output of player.getConfiguration()?

{
  "drm": {
    "retryParameters": {
      "maxAttempts": 2,
      "baseDelay": 1000,
      "backoffFactor": 2,
      "fuzzFactor": 0.5,
      "timeout": 0
    },
    "servers": {},
    "clearKeys": {},
    "advanced": {},
    "delayLicenseRequestUntilPlayed": false,
    "logLicenseExchange": false,
    "updateExpirationTime": 1
  },
  "manifest": {
    "retryParameters": {
      "maxAttempts": 2,
      "baseDelay": 1000,
      "backoffFactor": 2,
      "fuzzFactor": 0.5,
      "timeout": 0
    },
    "availabilityWindowOverride": null,
    "disableAudio": false,
    "disableVideo": false,
    "disableText": false,
    "defaultPresentationDelay": 0,
    "dash": {
      "clockSyncUri": "",
      "ignoreDrmInfo": false,
      "xlinkFailGracefully": false,
      "ignoreMinBufferTime": false,
      "autoCorrectDrift": true,
      "initialSegmentLimit": 1000,
      "ignoreSuggestedPresentationDelay": false,
      "ignoreEmptyAdaptationSet": false
    },
    "hls": {
      "ignoreTextStreamFailures": false,
      "useFullSegmentsForStartTime": false
    }
  },
  "streaming": {
    "retryParameters": {
      "maxAttempts": 2,
      "baseDelay": 1000,
      "backoffFactor": 2,
      "fuzzFactor": 0.5,
      "timeout": 0
    },
    "rebufferingGoal": 2,
    "bufferingGoal": 10,
    "bufferBehind": 30,
    "ignoreTextStreamFailures": false,
    "alwaysStreamText": false,
    "startAtSegmentBoundary": false,
    "smallGapLimit": 0.5,
    "jumpLargeGaps": false,
    "durationBackoff": 1,
    "forceTransmuxTS": false,
    "safeSeekOffset": 5,
    "stallEnabled": true,
    "stallThreshold": 1,
    "stallSkip": 0.1,
    "useNativeHlsOnSafari": true,
    "inaccurateManifestTolerance": 2
  },
  "offline": {
    "usePersistentLicense": true
  },
  "abr": {
    "enabled": true,
    "defaultBandwidthEstimate": 1000000,
    "switchInterval": 8,
    "bandwidthUpgradeTarget": 0.85,
    "bandwidthDowngradeTarget": 0.95,
    "restrictions": {
      "minWidth": 0,
      "maxWidth": null,
      "minHeight": 0,
      "maxHeight": null,
      "minPixels": 0,
      "maxPixels": null,
      "minFrameRate": 0,
      "maxFrameRate": null,
      "minBandwidth": 0,
      "maxBandwidth": null
    }
  },
  "preferredAudioLanguage": "",
  "preferredTextLanguage": "",
  "preferredVariantRole": "",
  "preferredTextRole": "",
  "preferredAudioChannelCount": 2,
  "restrictions": {
    "minWidth": 0,
    "maxWidth": null,
    "minHeight": 0,
    "maxHeight": null,
    "minPixels": 0,
    "maxPixels": null,
    "minFrameRate": 0,
    "maxFrameRate": null,
    "minBandwidth": 0,
    "maxBandwidth": null
  },
  "playRangeStart": 0,
  "playRangeEnd": null
}

What did you do?

I made a repo that reproduces the bug:

https://github.com/djeikyb/shaka_player_spurious_caption_bug_report

1. Host the files there in a web server. I used nginx in docker, like:

docker run --rm -p 8080:80 --name web -v "$PWD":/usr/share/nginx/html nginx:1.19

2. Open index.html in macos safari, I used safari 14.0.3 (14610.4.3.1.7): http://localhost:8080

3. Look at the text tracks, there is a track labeled "Unrecognized ()".

But you can see that no text tracks are configured. None in the hls playlist, none in the player config.

What did you expect to happen?

I expected no text tracks

What actually happened?

I found and can select a track labeled "Unrecognized ()". This is the output of player.getTextTracks()

JSON.stringify(player.getTextTracks(), null, 2)

[
  {
    "id": 1,
    "active": false,
    "type": "text",
    "bandwidth": 0,
    "language": "",
    "label": "",
    "kind": "captions",
    "width": null,
    "height": null,
    "frameRate": null,
    "pixelAspectRatio": null,
    "mimeType": "application/cea-608",
    "codecs": null,
    "audioCodec": null,
    "videoCodec": null,
    "primary": false,
    "roles": [
      "captions"
    ],
    "audioRoles": null,
    "videoId": null,
    "audioId": null,
    "channelsCount": null,
    "audioSamplingRate": null,
    "audioBandwidth": null,
    "videoBandwidth": null,
    "originalVideoId": null,
    "originalAudioId": null,
    "originalTextId": ""
  }
]

I used shaka packager to create the hls playlist and segments.

[djeikyb]

I should mention, this is the simplest reduction of the bug I could come up with. I first encountered this with an hls playlist with multiple video and multiple audio tracks, and with subtitle text tracks as child elements of the video tag. I would see all the expected text tracks, and then also the extra caption track.

[ismena]

Hi @djeikyb
Does this only happen on Safari or other browsers too?
If you just load the manifest directly in Safari (with no Shaka - post the manifest link into Safari's website navigation field), do you see the fantom text track?

[TheModMaker]

This track is for a CEA track. I think we add that because we assume there is CEA data even if it isn't present. We should probably delay-add the track until we see some CEA data.

[djeikyb]

It's only in Safari, and yep I see this extra caption track shows up without shaka. It looks like Safari is intentionally adding this track. I found this from Apple:

https://developer.apple.com/library/archive/qa/qa1801/_index.html

As part of the new automatic media selection feature on iOS 7, the presence of closed captions and subtitles must be known prior to playback. Prior to iOS 7, there is no way to declare the presence of closed caption data in the playlist.

By default, the alternate track button (see Figure 1) will be displayed along with the various other player controls and will be set to the "Unknown CC" option

Apple's recommendation is to positively declare the lack of captions:

iOS 7 also supports a CLOSED-CAPTIONS attribute for the EXT-X-STREAM-INF tag. You can declare the absence of closed caption content in the playlist by specifying CLOSED-CAPTIONS=NONE on the EXT-X-STREAM-INF tag. The "Unknown CC" option as shown in Figure 1 will not be displayed in the selection when the absence of closed caption content is declared

They mention ios7 as the motivator for the behaviour, but it applies to Safari. I found that adding the cc=none attribute worked as documented. Our drm provider is using shaka packager, so I'm looking to see if there's a way shaka packager could add the attribute. A workaround we might use in the meantime is providing our own shaka.ui.TextSelection widget that filters out this caption.

[joeyparrish]

@djeikyb, do you have an issue filed already against Shaka Packager for this? If not, I believe I can transfer this issue to the other repo and edit the title to reflect this new information.

[djeikyb]

No I don't, if you feel it's better tracked there please feel free!

This track is for a CEA track. I think we add that because we assume there is CEA data even if it isn't present. We should probably delay-add the track until we see some CEA data.

I'm wondering if you want a separate issue for this idea?

[joeyparrish]

We should probably delay-add the track until we see some CEA data.

I'm wondering if you want a separate issue for this idea?

If the track was created by Safari in src= mode, then we aren't in control of that.

Separately, in MediaSource mode, which is used in all other browsers, we only look for and expose CEA tracks when they are positively signaled in the content. The idea to wait to create a track for CEA would only make sense if we changed that design and made it so that the player would look for CEA data in all content, even if it were not signaled in the content.

This could open up other problems, though. For example, if broadcast TV content has CEA embedded, but has been augmented with VTT or TTML for streaming, the content owner may prefer to have CEA ignored. Today, that can happen just by leaving out the signaling in the manifest/playlist.

[joeyparrish]

Alright. I've transferred the issue to Shaka Packager and renamed it.

@kqyang, @TheModMaker, it seems that HLS content should explicitly signal a lack of captions with CLOSED-CAPTIONS=NONE on the EXT-X-STREAM-INF tag. Otherwise, if there is no positive or negative information about captions, some versions of Safari will assume captions and show UI accordingly. It should be pretty trivial to have Packager always generate an explicit CLOSED-CAPTIONS attribute, right?

See #922 (comment) for more details.

[kqyang]

Correct. Shaka Packager does not have explicit support for CLOSED-CAPTIONS. I think we can tag EXT-X-STREAM-INF with CLOSED-CAPTIONS=NONE for all streams.

If it is necessary, i.e. if anybody wants the implicit CLOSED-CAPTIONS support, we can add a flag later to control the behavior.

It should just be one line change after
https://github.com/google/shaka-packager/blob/master/packager/hls/base/master_playlist.cc#L255:

    tag.AddString("CLOSED-CAPTIONS", "NONE");

There will be a few unit-tests and end-to-end tests need to be updated.

@djeikyb Will you be interested in making the change, validating the change and sending a PR to us?

[djeikyb]

@kqyang I don't think I'm a good candidate, never worked with c++ before. I took a look, but I'm having trouble establishing an environment that can build the project.

[joeyparrish]

Sounds quick enough, and I've done this before. I'll try.