What is the best to compose the react provides components to handle some "Playback routing logic"?

[pietrop]

Non urgent question - I wanted to get your take on this problem. To see what's a good solution, but also if there's anything here that might be useful for Vidstack.

Vidstack is organized in Providers (which I think makes sense) Audio, Video, HLS etc..

I am working on a player, where I am covering both use case of audio and video, as well as optimizing for when to use HLS and when to use HlsJs.

Question: What is the best way to compose the React components providers (Audio and Video) and playback technologies (Hls provider) to only use the once that are needed based on the file input?

I could do a bunch of if statements but was wondering if there's a cleaner way?

Below some more details on the input files we consider and how we currently handle choosing the playback logic.

Input

Our input can be a source string or a source object and we support the following media types:

HLS	application/x-mpegURL
MP4	video/mp4
WEBM	video/webm
MP3	audio/mpeg
OGG_VIDEO	video/ogg
OGG_AUDIO	audio/ogg
WAV	audio/wav

Playback logic

This is some of the logic we follow (from our docs)

If we don't have an hls file, let's use the HTML5 audio or Video element. If it's hls and we are on a device with native support (an apple device) then let's still use the HTML5 audio or Video element.
However if it's hls and it's not on device with native support, then we can check for support of the hlsJs library. If the hlsJs library is supported we can load it and use it. If not (unlikely) will have to throw an error.

Which translate into the following algorithm in pseudoscope

// Is the source input an HLS file (m3u8)?
    // No - 
        use HTML5 src tag 
    // Yes - 
        // Is HLS supported natively?
            use HTML5 src tag 

        // Is HLSJS Supported
            Use HLSJS lib with HTML5 src Tag

        // Not native and not HSLJS Support 
            Throw error?

Also in diagram form

flowchart TD
    A[Player] --> B{Do we have an hls playlist in the source?}
    B -- Yes --> C[Do we have native HLS support? Eg Apple]
    C -- Yes --> D{Is it video or audio?}
    C -- No --> H{Does it support HlsJs?}
    H -- No --> L[Error]
    H -- Yes --> M[HLSJs]
    M --> D
    B -- No ----> D
    D -- Audio --> F[HTML5 Audio]
    D -- Video --> G[HTML5 Video]

Loading

Hls & HlsJs

As a side note, for more context

Some more context from our docs on our take on HlsJs and playback logic mental model

HlsJs

All option, under the hood, use the HML5 video or audio element.

Even HlsJS lib, converts hlsJs into mp4 chunks and is used on the HTML5 Audio/video element.

Note that HLSJS lib provides external helper methods to determined if the browser has native support for HLS. As once initialized the library itself does not fallback to native hls by default.

However we have used our own method to detect if hlsJs is supported, so that we can avoid the initial load of the hlsJs library if it's not necessary.

Mental model, file formats and playback technologies

In this mental model, file formats are decoupled from presentation UI layer and playback technology but instead inform the playback technologies algorithm.

The playback technology algorithm is also decoupled from the UI layer and overall architecture. Having it as a separate functionality and not used as the basis to model the problem domain, helps to simplify the overall architecture.

At a higher level, this means that our architecture design can be less deterministic, by being less opinionated about file formats and browser specs, and more probabilistic by providing routing logic for media playback support and other functionalities.

There's a "chicken and egg" s 🐔🥚 problem with hls.

Where the HlsJs library is quiet a big package, and in order to tell if you need it you'd have to first load it regardless to use their method to tell if hlsJs is supported by the browser. So we have these two modules that work independently of HlsJs to tell if Hls is natively supported or as a fallback to tell if hlsJs is supported. (To support hlsJs the browser needs to support fragmented mp4, as far as I understand)

isHlsJsSupported module example

function getMediaSource() {
	return window.MediaSource || window.WebKitMediaSource;
}

function isMp4CodecSupported() {
	const mediaSource = getMediaSource();
	if (!mediaSource) {
		return false;
	}
	const isTypeSupported =
		mediaSource &&
		typeof mediaSource.isTypeSupported === "function" &&
		mediaSource.isTypeSupported('video/mp4; codecs="avc1.42E01E,mp4a.40.2"');
	return Boolean(isTypeSupported);
}
/**
 * @description HlsJs has a lib to check if it's supported
 * via `Hls.isSupported()` but it requires loading hte hlsJs library
 * with this helper function we can check if HlsJs is supported
 * without the hlsJs library. Which helps with bundle size and performance.
 * @see [hls.js/is-supported.ts at c35441b94f7ffef94e12ecf99aa7a5c61d5759a0 · video-dev/hls.js](https://github.com/video-dev/hls.js/blob/c35441b94f7ffef94e12ecf99aa7a5c61d5759a0/src/is-supported.ts)
 * @private
 * @returns {boolean}
 */
const isHlsJsSupported = () => {
	const isTypeSupported = isMp4CodecSupported();
	return isTypeSupported;
};

export default isHlsJsSupported;

isNativeHlsSupported module example

import canPlayHlsType from "./can-play-type.js";
import isAndroidChrome from "./is-android-chrome.js";
import isEdge from "./is-edge.js";
import isIE11 from "./is-ie11.js";
import getUserAgent from "./get-user-agent.js";

/**
 *
 * @todo establish if this logic can be simplified.
 * eg in hls Js lib they use the HTML5 video element `canPlay` with `application/vnd.apple.mpegurl`.
 * which could also check for `application/x-mpegURL`. Would that be sufficient, or do we need to run all these other checks?
 * @example <caption> hlsJs lib example </caption>
 *  // First check for native browser HLS support
 * if (video.canPlayType('application/vnd.apple.mpegurl')) {
 * // ...
 * @see [GitHub - video-dev/hls.js: HLS.js is a JavaScript library that plays HLS in browsers with support for MSE.](https://github.com/video-dev/hls.js/#alternative-setup)
 */

/**
 *
 * @description MS browsers' native HLS implementations do not support WebVTT over HLS
 * @private
 * @param {string} userAgent
 */
const supportsWebVTTOverHLS = (userAgent) => {
	return !isIE11(userAgent) && !isEdge(userAgent);
};

/**
 *
 *
 * @param {HTMLMediaElement} mediaElement
 * @returns {boolean}
 * @private
 * @example <caption>example user agent string input</caption>
 * 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'
 */
const isNativeHlsSupported = (mediaElement) => {
	const userAgent = getUserAgent();
	return (
		supportsWebVTTOverHLS(userAgent) &&
		!isAndroidChrome(userAgent) &&
		canPlayHlsType({ mediaElement: mediaElement })
	);
};

export default isNativeHlsSupported;

/**
 * @param {object} options
 * @param {HTMLMediaElement} [options.mediaElement]
 * @returns {boolean}
 * @private
 */
const canPlayHlsType = ({ mediaElement = document.createElement("video") }) => {
	return (
		typeof mediaElement.canPlayType === "function" &&
		(mediaElement.canPlayType("application/vnd.apple.mpegURL") !== "" ||
			mediaElement.canPlayType("application/x-mpegURL") !== "")
	);
};

export default canPlayHlsType;

and a bunch of helper functions

const getWindow = () => window;
const getNavigator = (win = window) => win.navigator;

const getUserAgent = (win = getWindow()) => {
	const navigator = getNavigator(win);
	// eslint-disable-next-line no-useless-escape
	return navigator.userAgent.toLowerCase().replace(/\[.*\]/, "");
};

const isAndroid = () => {
	const userAgent = getUserAgent();
	return userAgent.toLowerCase().includes("android");
};

const isChrome = () => {
	const userAgent = getUserAgent();
	const lowerCaseUserAgent = userAgent.toLowerCase();
	return lowerCaseUserAgent.includes("chrome");
};

/**
 *
 * @private
 * @param {string} userAgent
 */
const isAndroidChrome = (userAgent) => {
	return isAndroid(userAgent) && isChrome(userAgent);
};

const isEdge = () => {
	const userAgent = getUserAgent();
	return /edge\/(\d+)/i.test(userAgent);
};

const isIe11 = () => {
	const userAgent = getUserAgent();
	return /trident.+?rv[ :]11/i.test(userAgent);
};

[mihar-22]

Thanks for all this! This is an important problem to think about as it'll affect how we design providers for Vidstack Player.

Provider switching and HLS loading states are definitely tricky. Currently vds-hls will fallback to loading hls.js only if media loading is aborted and the current src is a HLS playlist (here). I need to test this behaviour again but this should mean native support is accounted for because it won't abort if supported. This way we don't need to do any fragile UA testing.

In the event of an abort, we run the same check for hls.js support via the implementation as the isHlsJsSupported function.

Diagram

stateDiagram
    [*] --> set_src

    set_src --> load_start:success
    set_src --> abort:error
    abort --> is_hls_js_supported
    is_hls_js_supported --> load_hls_js:yes
    is_hls_js_supported --> fire_hls_unsupported_event:no
    load_hls_js --> load_hls_playlist:success
    load_hls_js --> fire_hls_load_error_event:error

Loading

Question: What is the best way to compose the React components providers (Audio and Video) and playback technologies (Hls provider) to only use the once that are needed based on the file input?

At the moment we support video falling back to HLS. Our HLS provider is not that big and it won't immediately load hls.js as discussed above. Right now you could just test for audio and load vds-audio, otherwise vds-hls.

Possible Enhancement

It'd be really easy to add audio support to vds-hls provider as well. This way it can render <audio> element if it detects an audio src, otherwise continue the current video loading behaviour.

Invalid suggestion because the <audio> and <video> element exists in user code so Vidstack can dynamically switch it. But, it's probably not needed as I'm assuming the native <video> element support audio? If true, then vds-hls should handle audio, video, and HLS playlists. They're all tied together by the Html5MediaElement class.

Alternative

We could also just provide a utility function isAudioSrc function in Vidstack and document how to handle this case?

Thoughts? Are we missing something? What can do we better?

[pietrop]

Currently vds-hls will fallback to loading hls.js only if media loading is aborted and the current src is a HLS playlist (here). I

That's an interesting approach 💡 . I am assuming you are considering the abort time to be negligible in terms of user experience?

At the moment we support video falling back to HLS. Our HLS provider is not that big and it won't immediately load hls.js as discussed above. Right now you could just test for audio and load vds-audio, otherwise vds-hls.

That's cool, thanks, might try that later. And circle back.

[mihar-22]

That's an interesting approach 💡 . I am assuming you are considering the abort time to be negligible in terms of user experience?

Honestly, requires more testing. I think we should add a timeout via Promise.race if it exceeds a certain amount. I've read the spec but I think it's a browser implementation detail on how they actually handle resource selection/loading in terms of rejecting on first byte or not.

We could also make the loading strategy optional -> whether this abort strategy or the one you've outlined (we used exact same implementation previously).

[mihar-22]

Okay so in my testing it was always instant and here's why:

⌛ If candidate has a type attribute whose value, when parsed as a MIME type (including any codecs described by the codecs parameter, for types that define that parameter), represents a type that the user agent knows it cannot render, then end the synchronous section, and jump down to the failed with elements step below.

Basically, if the media source is set like so:

<video>
  <source src="..." type="application/x-mpegURL" />
</video>

It will abort synchronously (if not supported) as it runs through the resource selection algorithm. No network request should be made.

[pietrop]

oh, this is very cool!

So that's for HLS. And then how would you currently check if hlsJs is supported by the browser?

[mihar-22]

So that's for HLS. And then how would you currently check if hlsJs is supported by the browser?

This strategy will handle checking for native HLS support. To check if hls.js is supported as a fallback you'd use the same isHlsJsSupported function you shared above (coped from hls.js source).