Safari leaks memory when videos are rapidly removed from the screen and re-added

[mmarekbb]

What happened and what did you expect to happen?

Safari has a problem with the HTMLVideoElement object that keeps leaking memory if there are many video elements that are being removed from the screen and re-attached back.

There is a similar bug reported in the WebKit bugzilla: https://bugs.webkit.org/show_bug.cgi?id=216820

At some point, Safari will show a message that the application is using significant memory. If it continues even further, Safari will crash.

Have you reviewed our existing documentation?

• Amazon Chime SDK for JavaScript GitHub issues
• Amazon Chime SDK React Components Library GitHub issues
• Storybook documentation
• README page
• How-to documentation
• React SDK meeting demo

Reproduction steps

1. Update the Meeting demo to render videos in smaller batches (such as with pagination).
2. Open Activity Monitor.
3. Join the demo with about 25 other video-enabled participants.
4. Cycle through the pages.
5. Notice that the memory keeps steadily rising without ever dropping back.

I'm able to confirm that this happens because RemoteVideo.tsx. If I comment the useEffect out so no video stream is attached, I cannot see any memory leaks.

Amazon Chime SDK React Components Library version

2.15.0

What browsers are you seeing the problem on?

MacOS Safari, iOS Safari

Browser version

14.1

Device Information

MacOS 11.6.4

Meeting and Attendee ID Information.

No response

Browser console logs

collaborate-ultra-log-1649082858254.log

Add any other context about the problem here.

Updating the RemoteVideo.tsx component's useEffect to this seems to help but it does not eliminate the memory leaks completely. Perhaps there's something else going on?

  React.useEffect(() => {
    if (!audioVideo || !videoEl.current) {
      return;
    }

    audioVideo.bindVideoElement(tileId, videoEl.current);
    const videoElement = videoEl.current;

    return () => {
      const tile = audioVideo.getVideoTile(tileId);

      if (tile) {
        audioVideo.unbindVideoElement(tileId);
      }

      if (videoElement) { // cleanup here
        videoElement.srcObject = null;
        videoElement.load();
      }
    };
  }, [audioVideo, tileId]);

[mmarekbb]

I found some additional details.

It seems that unbinding the video element (audioVideo.unbindVideoElement) does not work as expected here. VideoTile.disconnectVideoStreamFromVideoElement gets called with videoElement = null and no cleanup happens.

That is why me adding videoElement.srcObject = null helps to an extent. Calling the static method directly with the video element here (DefaultVideoTile.disconnectVideoStreamFromVideoElement(videoElement, false, false)) however causes another problem. The stream tracks are stopped but then never restarted, so if the same video is removed and re-added the tracks aren't there and the video doesn't start which is where the second memory leak problem is coming from (tracks not being stopped when video is unmounted) I believe.

[devalevenkatesh]

Hi @mmarekbb ,

Yup you are right that the unbindVideoElement will just call bindVideoElement with null and it is a no-op on stream cleaning and only the tileState is updated with boundVideoElement as null. I found a similar reported issue in JS SDK repo mentioning a similar problem, but in their case setting the srcObject to null solves the issue.

I am still checking on the RemoteVideo useEffect cleanup and will keep this issue updated.

Questions:

1. If we rapidly start-stop a single remote attendee video, do we run into this issue? Or it has to be 25+ tiles?
2. Is this macOS and Safari specific and also specific to a version or can be reproduced in newer versions as well?

[devalevenkatesh]

Created a demo using react component library which at an interval mounts and then un-mounts 25 RemoteVideo components tied to a single tileId.

Test Scenario
Mount un-mount 25 remote video tiles and check for memory increase in Safari Timelines from developer tools also compare the same with Chrome devtools.

1. Clear video srcObject tracks and then set the HTML video element’s srcObject to null.
2. Directly set HTML video element’s srcObject to null without clearing any source tracks.
3. Do nothing and keep as is the current code.

Clear video srcObject tracks and then set the srcObject to null	Directly set srcObject to null	Existing code

Observations

• From the graphs, it seems setting srcObject to null helps but clearing tracks and then setting it to null is the most memory efficient.
• Chrome does not seem to differ with any approaches.

Few next questions

1. My testing strategy may not be a good real world use-case hence wanted to understand your pagination flow and when RemoteVideo component mounts -> un-mounts?
2. If using chooseRemoteVideoSources, when the new remote video sources are selected, do you keep the RemoteVideo's from last render or you un-mount all the RemoteVideo's and re-mount the RemoteVideo's with the new tileId from current page’s attendeeIds?
3. If yes, does it mean that there is no stopping of video happening anywhere and only RemoteVideo component mount/un-mount?

[mmarekbb]

HI @devalevenkatesh, that is the same behavior I was observing. Setting srcObject helps a bit but the tracks are causing another memory leak.

Answers:

1. We are doing basically the same thing. Whenever the next page is loaded, we immediately unmount all the previous RemoteVideo components
2. We do not use chooseRemoteVideoSources - we only use the RemoteVideo component
3. We do not stop the video at all, we only unmount RemoteVideo. Attempting to stop the video tracks in the component's cleanup never starts the video again when rendering it another time though.

[devalevenkatesh]

Thanks for getting back @mmarekbb . For the source tracks cleanup, I have the following code which attaches the tracks back correctly after un-mount and re-mount, could you try if this helps?

return () => {
  const tile = audioVideo.getVideoTile(tileId);
  if (tile) {
    audioVideo.unbindVideoElement(tileId);
    // Added the cleanup.
    if (videoEl.current) {
      const mediaStream = videoEl.current.srcObject as MediaStream;
      const tracks = mediaStream.getTracks();
      for (const track of tracks) {
        track.stop();
        mediaStream.removeTrack(track);
      }
      videoEl.current.srcObject = null;
    }
  }
}

[devalevenkatesh]

Hi @mmarekbb ,

Checking if you found time to test above suggestion. I also tested just cleaning up tracks and not the srcObject and it does not help. So it seems doing above code in cleanup is the most effective solution yet.

Graph for cleaning up source tracks but not setting srcObject to null:

[mmarekbb]

Hi @devalevenkatesh, this unfortunately has the issue I was mentioning earlier. The tracks are stopped but then no videos start loading again. I'm testing this with 25 videos (which is the video limit) if that makes any difference. I'm also attaching the console logs, I'm getting a lot of bwe: cannot Upgrade: bitrateKbp: 1200 targetBitrateKbp: 300 but not sure if that is related.

collaborate-ultra-log-1651480608446.log

[devalevenkatesh]

Thanks for checking @mmarekbb . Well, I think it is hard to reproduce how the binding un-binding works when paginating unless we update the demo to do that. Could you please let me know:

1. Appreciate if you can help understand - how the pagination works? More importantly, how does your app select through new video streams. (You earlier mentioned that you do not use chooseRemoteVideoSources hence want to understand this in order to reproduce this in our demo or a sample mocking your implementation.
2. Do you use RemoteVideo component and pass a specific tileId to it or you just use RemoteVideos component? If yes, how are new tileIds been received.
3. You mentioned earlier, when you go to next page, the previous RemoteVideo components are un-mounted, so on the next page how are the video stream binded again? Are you using useRemoteVideoTileState to get all the tildeIds and then showing the RemoteVideo component with those specific tileIds?

Next Steps:

1. Based on above questions we have to build a sample pagination app to mock the memory leaking path to reproduce the issue.

[mmarekbb]

Thanks @devalevenkatesh. I'll try to boil our pagination component down as much as possible to share with you. For the time being, here is the answer to your questions (consolidated into one block :))

Our app allows users to choose to see fewer streams. By default, we display 25, but in cases where the screen estate is too small or the user wishes to use fewer, we limit it to say 6. We use the useRosterState() hook to grab the roster, sort it and then take the first 6 attendee IDs. We use useRemoteVideoTileState, specifically the attendeeIdToTileId map, to render a bunch of RemoteVideo components with their respective Tile IDs.

When switching to the next page, we take another 6 attendee IDs from the roster, unmount the previous 6 and using the same mechanism we render the new 6 <RemoteVideo> components. Stopping the media tracks however prevents the previous 6 videos from starting when going back to the previous page.

[mmarekbb]

@devalevenkatesh I can reproduce with this example of pagination:

import * as React from 'react';
import {
  useRemoteVideoTileState,
  useRosterState,
  useAudioVideo,
  useApplyVideoObjectFit,
  VideoTile
} from 'amazon-chime-sdk-component-library-react';


// Copied from the React SDK, added cleanup and replaced className by some simple styles for the example.
const RemoteVideo: React.FC<any> = ({
  name,
  tileId,
  ...rest
}) => {
  const audioVideo = useAudioVideo();
  const videoEl = React.useRef<HTMLVideoElement>(null);
  useApplyVideoObjectFit(videoEl);

  React.useEffect(() => {
    if (!audioVideo || !videoEl.current) {
      return;
    }

    audioVideo.bindVideoElement(tileId, videoEl.current);

    return () => {
      const tile = audioVideo.getVideoTile(tileId);
      if (tile) {
        audioVideo.unbindVideoElement(tileId);
        // Added the cleanup.
        if (videoEl.current) {
          const mediaStream = videoEl.current.srcObject as MediaStream;
          const tracks = mediaStream.getTracks();
          for (const track of tracks) {
            track.stop();
            mediaStream.removeTrack(track);
          }
          videoEl.current.srcObject = null;
        }
      }
    }
  }, [audioVideo, tileId]);

  return (
    <VideoTile
      {...rest}
      ref={videoEl}
      nameplate={name}
      style={{width: 300, height: 169}}
    />
  );
};

export const RemoteVideos = () => {
  const { attendeeIdToTileId } = useRemoteVideoTileState();
  const { roster } = useRosterState();
  const [startIndex, setStartIndex] = React.useState(0);

  const PER_PAGE = 3;
  const total = Object.keys(roster).length;
  const attendees = Object.values(roster).slice(startIndex, startIndex + PER_PAGE);

  console.log({ startIndex, attendees, total });

  return (
    <div style={{ width: 1000, height: 800 }}>
      {startIndex > 0 && (
        <button onClick={() => setStartIndex(Math.max(0, startIndex - PER_PAGE))}>Previous page</button>
      )}
      {startIndex < total && (
        <button onClick={() => setStartIndex(Math.min(total - 1, startIndex + PER_PAGE))}>Next page</button>
      )}

      <div style={{display: 'grid'}}>
        {attendees.map(attendee => (
          <RemoteVideo
            key={attendee.chimeAttendeeId}
            name={attendee.name}
            attendeeId={attendee.chimeAttendeeId}
            tileId={attendeeIdToTileId[attendee.chimeAttendeeId]}
          />
        ))}
      </div>
    </div>
  );
}

[devalevenkatesh]

Thanks @mmarekbb for providing the code snippet, I could repro the issue just had to clean up the local attendeeId showing from the remote attendee videos.

I found that the main reason for the video going black is due to the cleanup in useEffect that makes the media stream to go in-active. JS SDK keeps and uses the same MediaStream when binding to HTMLVideoElement and is the only one that is also maintained in the VideoTileState. When in cleanup of RemoteVideo useEffect if we stop the tracks then it results into this MediaStream going inactive (active flag on the media stream goes to false).

If we remove the tracks using mediaStream.removeTrack(track), then this results into a ended event on the media stream and the CreatePeerConnectionTask inside the JS SDK listens to this ended event and then sends video tile removal update which will remove the video tile itself and that is not intended.

Thus, due to this we cannot touch the MediaStream attached to the HTMLVideoElement as it is the same one that is referenced in VideoTileState when bounded using meetingSession.audioVideo.bindVideoElement API.

I have pushed the pagination code sample to this repo (slightly modified from your version so thanks for providing yours):
https://github.com/devalevenkatesh/test-amazon-chime-sdk-react-components/tree/main/pagination/without-VideoPriorityBasedPolicy

I also tried the pagination using VideoPriorityBasedPolicy where we can use chooseRemoteVideoSources API to choose specific remote videos for specific remote attendees, but I could not find a way to re-use RemoteVideo component as the component is tileId dependent. Hence, if remote video sources change, the attendeeId associated with the remote video will change and thus the tileId also changes which is attached to a remote video tile in JS SDK. Thus, this results into RemoteVideo being mounted and un-mounted. For now I think the only way to not leak is not un-mounting the RemoteVideo component and re-using it which is not possible as explained.

I have pushed the code sample to this repo:
https://github.com/devalevenkatesh/test-amazon-chime-sdk-react-components/tree/main/pagination/with-VideoPriorityBasedPolicy

The graph on the memory leak with this sample looks like below:
Check Page row under Categories in the bottom part of screenshots

safari-as-is-pagination	safari-srcobject-null-pagination	using chooseRemoteVideoSources but no useEffect cleaning

Also, cleaning up using videoEl.current.srcObject = null does not really work as in React 17 the ref videoEl.current is null in the useEffect return function. Reference. To address this the videoEl.current needs to be cached and then used in the cleanup like below.

const RemoteVideo: React.FC<any> = ({
  name,
  tileId,
  ...rest
}) => {
  const audioVideo = useAudioVideo();
  const videoEl = React.useRef<HTMLVideoElement>(null);
  useApplyVideoObjectFit(videoEl);

  React.useEffect(() => {
    ...
    // Store the reference and then use in cleanup.
    const videoElement = videoEl.current
    return () => {
      const tile = audioVideo.getVideoTile(tileId);
      if (tile) {
        audioVideo.unbindVideoElement(tileId);
        // Added the cleanup.
        if (videoElement) {
          videoElement.srcObject = null;
          videoElement.load();
        }
      }
    }
  }, [audioVideo, tileId]);

I will consult within team if I can do more but for now I am concluding that:

• Cleaning srcObject in RemoteVideo useEffect cleanup is the best approach we have to minimize the memory leak and we cannot cleanup the tracks as explained.

[devalevenkatesh]

@mmarekbb The memory leak fix for setting the videoElement.srcObject = null; is done in JS SDK itself as it is helpful for all. The PR linked above is merged and will be available in the next JS SDK release unless flagged by builders for any issue.

We have made provision in the API to optionally fallback to older behavior if new is not un-intended. Once JS SDK release is done, component library will take the dependency and release in its next version.

Thank you for your patience and all the inputs on this.

[mmarekbb]

@devalevenkatesh Thanks for the update. For the time being we're unsetting the srcObject in our codebase.

Should this ticket be closed? There still is a memory leak in Safari, although not as big as before.

[devalevenkatesh]

Yes, I think the major leak is the one we are addressing with the srcObject. I will keep this open until the releases happen and then close it.

[devalevenkatesh]

I am closing this issue since builders can install the JS SDK 3.1.0 or later on the latest component library 3.2.0. Newer release on component library is not needed. Thanks!