Performant component to use GreenScreen effect for multiple videos at the same time

[mnlbox]

Hi @JonnyBurger,

I have multiple greenscreen videos in my Remotion project, as I checked the documentation I should use something like this:
https://www.remotion.dev/docs/video-manipulation#greenscreen-example

It's OK but then I found this comparison between <Video> and <OffthreadVideo> components and there was two points mentioned there as Cons for <Video> component:

⛔ Chrome may throttle video tags if the page is heavy.
⛔ If too many video tags get rendered simultaneously, a timeout may occur.

Now I'm asking to know which one is a better way to use for GreenScreen effect if I have multiple videos? Should I use <OffthreadVideo> for my GreenScreen effect?
Is there any easier and more performant way of using GreenScreen effect on Remotion?

[mnlbox]

Hi @JonnyBurger,

I see it seems in Remotion, transparent Apple ProRes is the official overlays format suggested to export overlay videos from Remotion and reuse in other softwares.

I wondered if it's also supported by default by Remotion or not?
I want to use third party greenscreen animations like this:
https://pixabay.com/videos/search/greenscreen/

Is there any other format supported by Remotion (for example: transparent Apple ProRes)? In this case I can convert my greenscreen animations one time to Apple ProRes and then use them normally in Remotion without any video manipulation filter to speed up Rendering as well.

[JonnyBurger]

You can use ProRes or transparent WebMs and probably other formats that are out there, if you use OffthreadVideo.

This might only work during rendering, since during preview we use the native browser element.

[mnlbox]

In case if it's interesting I created this GreenScreen component based on the doc example, I just made the video dimensions dynamic based on input video:

import {getVideoMetadata} from '@remotion/media-utils';
import {useCallback, useEffect, useRef, useState} from 'react';
import {
	AbsoluteFill,
	continueRender,
	delayRender,
	staticFile,
	Video,
} from 'remotion';

type Props = {
	src: string;
};
export const GreenScreen: React.FC<Props> = ({src}) => {
	const [handle] = useState(() =>
		delayRender('Calculate greenscreen video dimensions'),
	);
	const [dimensions, setDimensions] = useState<{
		width: number;
		height: number;
	} | null>(null);

	const video = useRef<HTMLVideoElement>(null);
	const canvas = useRef<HTMLCanvasElement>(null);
	useEffect(() => {
		const fetchGreenscreenVideoDimensions = async () => {
			const {width, height} = await getVideoMetadata(staticFile(src));
			setDimensions({width, height});
			continueRender(handle);
		};

		fetchGreenscreenVideoDimensions();
	}, []);

	const onVideoFrame = useCallback(
		(op: number) => {
			if (
				!canvas.current ||
				!video.current ||
				!video.current.requestVideoFrameCallback
			) {
				return;
			}

			const context = canvas.current.getContext('2d');

			if (!context || !dimensions) {
				return;
			}

			context.drawImage(
				video.current,
				0,
				0,
				dimensions.width,
				dimensions.height,
			);
			const imageFrame = context.getImageData(
				0,
				0,
				dimensions.width,
				dimensions.height,
			);
			const {length} = imageFrame.data;

			for (let i = 0; i < length; i += 4) {
				const red = imageFrame.data[i + 0];
				const green = imageFrame.data[i + 1];
				const blue = imageFrame.data[i + 2];
				if (green > 100 && red < 100 && blue < 100) {
					imageFrame.data[i + 3] = op * 255;
				}
			}

			context.putImageData(imageFrame, 0, 0);
		},
		[dimensions],
	);

	useEffect(() => {
		if (!video.current?.requestVideoFrameCallback) {
			return;
		}

		const {current} = video;
		let handle = 0;
		const callback = () => {
			onVideoFrame(0);
			handle = current.requestVideoFrameCallback(callback) as unknown as number;
		};

		callback();

		return () => {
			current.cancelVideoFrameCallback(handle);
		};
	}, [onVideoFrame]);

	if (dimensions === null) {
		return null; // Render nothing until dimensions is fetched
	}

	return (
		<AbsoluteFill>
			<AbsoluteFill>
				<Video ref={video} style={{opacity: 0}} src={staticFile(src)} />
			</AbsoluteFill>
			<AbsoluteFill>
				<canvas
					ref={canvas}
					width={dimensions.width}
					height={dimensions.height}
				/>
			</AbsoluteFill>
		</AbsoluteFill>
	);
};

I think maybe it can be handy to have this as an official filter and a new prop for the official <Video> component. Something like this:

<Video src={staticFile("video.webm")} filterrs={[RemotionFilters.GreenScreen]} />

I'm going to try this for my project with multiple GreenScreen videos and share my experience about performance here.

[JonnyBurger]

Nice, thanks for sharing your component!

[mnlbox]

I have another issue with this GreenScreen component mentioned above:
my video has a very green edges after using this GreenScreen effect. It's so clear after render, something like this:

Any idea for this? 🤔
Is there any way to blur the edges a bit?

[JonnyBurger]

Indeed, we only provided a simple algorithm to demonstrate the point. Probably you need a more sophisticated algorithm to get good results. If you google "JavaScript Greenscreen code" you find many examples. For example this one seems more sophisticated:

https://github.com/conradg/green-screen-js/blob/master/main.js