Add APIs for converting between native VideoFrameBuffers and their underlying CVPixelBuffer on macOS
#355
+133
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…n macOS co-authored-by: Mikayla <[email protected]>
co-authored-by: Mikayla <[email protected]>
13 tasks
maxbrunsfeld
force-pushed
the
native-video-frame-buffer
branch
3 times, most recently
from
fc222ea
to
0867d5b
Compare
maxbrunsfeld
force-pushed
the
native-video-frame-buffer
branch
from
0867d5b
to
04764ae
Compare
maxbrunsfeld
force-pushed
the
native-video-frame-buffer
branch
from
04764ae
to
9ee485e
Compare
theomonnom
approved these changes
Jun 26, 2024
|
Awesome, thanks! 🙌 |
SomeoneToIgnore
added a commit
to zed-industries/zed
that referenced
this pull request
Nov 15, 2024
See livekit/rust-sdks#355 Todo: * [x] make `call` / `live_kit_client` crates use the livekit rust sdk * [x] create a fake version of livekit rust API for integration tests * [x] capture local audio * [x] play remote audio * [x] capture local video tracks * [x] play remote video tracks * [x] tests passing * bugs * [x] deafening does not work (livekit/rust-sdks#359) * [x] mute and speaking status are not replicated properly: (livekit/rust-sdks#358) * [x] **linux** - crash due to symbol conflict between WebRTC's BoringSSL and libcurl's openssl (livekit/rust-sdks#89) * [x] **linux** - libwebrtc-sys adds undesired dependencies on `libGL` and `libXext` * [x] **windows** - linker error, maybe related to the C++ stdlib (livekit/rust-sdks#364) ``` libwebrtc_sys-54978c6ad5066a35.rlib(video_frame.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MT_StaticRelease' doesn't match value 'MD_DynamicRelease' in libtree_sitter_yaml-df6b0adf8f009e8f.rlib(2e40c9e35e9506f4-scanner.o) ``` * [x] audio problems Release Notes: - Switch from Swift to Rust LiveKit SDK 🦀 --------- Co-authored-by: Mikayla Maki <[email protected]> Co-authored-by: Conrad Irwin <[email protected]> Co-authored-by: Kirill Bulatov <[email protected]> Co-authored-by: Michael Sloan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #354 (this branch contains that change as well).
Background
In the Zed editor, we're switching over from using the LiveKit Swift SDK to this crate. The main motivation is that we want our LiveKit-based screen-sharing and audio features to work on Linux (and someday Windows), not just macOS.
In order to implement screen sharing efficiently (as it was before, when using the Swift SDK), we want to pass video frames directly from the
ScreenCaptureKitframework to this crate'sNativeVideoSource::capture_framemethod, without copying them into intermediate buffers. Similarly, when playing remote video tracks, we want to take decoded video frames directly from theNativeVideoStreamand use them as GPU textures without performing an extra copy.Change
To support both of these use cases, we have added two new macOS-specific APIs to the
video_frame::NativeBuffertype:from_cv_pixel_buffer, andget_cv_pixel_buffer. I think it makes sense to have this functionality, since AFAICT, the purpose of theNativevariant ofVideoFrameBufferis to adapt a platform-specific video frame representation into the WebRTC's logic.Details
CoreMediaframework, so we decided to leave the interface in terms of*mut c_void.get_cv_pixel_buffercan returnNULLif the native buffer is not actually backed by aCVPixelBuffer. Rather than implicitly perform some potentially expensive conversion implicitly, we decided to leave that to the caller. The only point of these method is to enable a fast path in which we minimize conversions.Let me know if there is a different solution to this problem that you'd prefer. Thanks!