Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update unbindVideoElement #2217

Merged
merged 8 commits into from
May 12, 2022
Merged

Update unbindVideoElement #2217

merged 8 commits into from
May 12, 2022

Conversation

devalevenkatesh
Copy link
Contributor

@devalevenkatesh devalevenkatesh commented May 11, 2022

Issue #:

Libraries or apps depending on JS SDK can leak memory when the video element bound to a video tile is not cleaned correctly. Based on the above issues, the unbindVideoElement API is used for cleanup. For example, in Amazon Chime SDK react component library, as part of the cleanup function in React's useEffect (or similarly in various apps) we call unbindVideoElement for cleanup. But this does not really dissociate the media stream from the now un-bounded video element. Builders have to explicitly address this on their side. Builders expect that the unbindVideoElement should do the opposite of bindVideoElement as per #1076 .

In Safari specifically, memory leak is observed if the un-bounded video element's stream is not cleaned up.
Check below graph, the memory keeps increasing steadily in a pagination app where the video elements are binded and un-binded often on page changes.
srcObject-not-null

I have checked multiple approaches and found that with JS SDK v3, we can now disconnect the srcObject from the bounded video element as part of un-bind. Below graph shows the memory usage when srcObject is set to null:
srcObject-null

Note: Check the "page" memory category.

It is also recommended in WebRTC to set this and we could not do so in v2 as we were also cleaning the media stream tracks to avoid crashing in Safari 12.
https://webrtchacks.com/srcobject-intervention/

With V3, we no more clear tracks and hence can fix this. We are still looking for the impact and will gain some more info by sharing this PR with builders and see if there are any issues.

Since, this is a behavior change, added a cleanUpVideoElement optional parameter in unbindVideoElement API defaulting to true. If builders face any issue and this srcObject cleanup is un-intended, builders can call unbindVideoElement(tileId, false) to fallback to previous behavior.

Description of changes:

  • With V3, we do not cleanup stream tracks bound to a video element and hence leveraged that to cleanup the srcObject as part of unbindVideoElement API. Note that the DefaultVideoTile currently does that internally in disconnectVideoStreamFromVideoElement function hence used it.
  • This should now address UnbindVideoElement #627 (comment).
  • Fixes unit test in POSTLogger which was not reaching the 100% threshold.

Testing:

Can these tested using a demo application? Please provide reproducible step-by-step instructions.
Tested with custom pagination sample using React component library. Sample link

  1. Started the demo app.
  2. Join 6 attendees. (Join first from Chrome and FF)
  3. Load the demo in Safari, start recording the memory by going to "Timelines" in network tab.
  4. Join the meeting in Safari.
  5. Demo automatically changes through the pages so that multiple video elements get binded and un-binded to get the graph over 90 seconds.
  6. Compare with current graph and check the significant memory usage is reduced.

Tested the JS SDK video test app to specifically test the unbindVideoElement API.

  1. Run the video_test app under integration/js/app folder.
  2. Join a meeting.
  3. Click startLocalVideoTile.
  4. Click bindVideoElement setting tileId to 1 and video element id as video-0.
  5. Verify that the video shows up.
  6. Click stopLocalVideoTile.
  7. Verify the stream is now null.
  8. Click unbindVideoElement and see that the tile state shows boundVideoElement as null.
  9. Call the unbindVideoElement with cleanUpVideoElement true and false and verify that when false the un-bounded HTML video element still has the stream attached through srcObject. When true, the video element does not have the stream attached anymore.

Checklist:

  1. Have you successfully run npm run build:release locally?
    Yes

  2. Do you add, modify, or delete public API definitions? If yes, has that been reviewed and approved?
    Yes, modified unbindVideoElement API. The PR is for review.

  3. Do you change the wire protocol, e.g. the request method? If yes, has that been reviewed and approved?
    No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@devalevenkatesh devalevenkatesh changed the title Unbind video element Cleanup srcObject in unbindVideoElement May 11, 2022
@devalevenkatesh devalevenkatesh changed the title Cleanup srcObject in unbindVideoElement Update unbindVideoElement May 11, 2022
- Update the unbindVideoElement API to take `cleanUpVideoElement` as an
optional parameter with default set to true.
- When cleanUpVideoElement is true, bounded video element's srcObject is set to null. This helps
in memory leaks. The stream wont remain bound to the video element once
unbound.
- When cleanUpVideoElement is false, the stream remains bound to the video element once
unbound.
- Since, this is a change in behavior the control flag is added.
- Update documentation to match v3 for some methods.
- Add corresponding unit tests.
@devalevenkatesh devalevenkatesh force-pushed the unbindVideoElement branch 2 times, most recently from 5b8f1a8 to 336c3f7 Compare May 11, 2022 21:31
@devalevenkatesh devalevenkatesh marked this pull request as ready for review May 11, 2022 21:38
@devalevenkatesh devalevenkatesh requested a review from a team as a code owner May 11, 2022 21:38
- Update docs.
- Tested in video_test app.
- Updated unit tests.
guides/03_API_Overview.md Outdated Show resolved Hide resolved
src/videotilecontroller/VideoTileController.ts Outdated Show resolved Hide resolved
src/videotile/DefaultVideoTile.ts Outdated Show resolved Hide resolved
@devalevenkatesh devalevenkatesh merged commit b7d2c31 into main May 12, 2022
@devalevenkatesh devalevenkatesh deleted the unbindVideoElement branch May 12, 2022 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants