Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Share events to Python without copying via arrow crate #228

Merged
merged 43 commits into from
Apr 18, 2023
Merged

Share events to Python without copying via arrow crate #228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
b766a44
use arrow for rust-python interface
haixuanTao Mar 14, 2023
2ca97fd
Remove lifetime from `Event` type
phil-opp Mar 21, 2023
1fab082
Use `arrow` crate to send events from Rust to Python without copying
phil-opp Mar 21, 2023
d8663b6
Update `Cargo.lock`
phil-opp Mar 21, 2023
0d0d663
Add TODO
phil-opp Mar 22, 2023
3496073
Return custom object from event stream to support both arrow and clas…
phil-opp Apr 4, 2023
fc71955
Name `eyre::Result` type explicitly
phil-opp Apr 4, 2023
d808ace
Add back support for sending `PyBytes` as output for backwards compat…
phil-opp Apr 4, 2023
983a26f
Box shared memory instance to reduce size of `Event` enum
phil-opp Apr 5, 2023
d60d0c8
Fix drop token handling when there are no local subscribers
phil-opp Apr 5, 2023
9030e72
Extract event stream loop into separate function
phil-opp Apr 5, 2023
453e03c
Warn if shared memory regions are closed before receiving drop tokens
phil-opp Apr 5, 2023
a661030
Break from drop token wait loop on timeout
phil-opp Apr 5, 2023
446fa7f
Wait for remaining drop tokens before breaking from event stream thread
phil-opp Apr 5, 2023
0ed4dcf
Handle drop tokens asynchronously instead of waiting for them
phil-opp Apr 5, 2023
5b76c16
Release Python GIL while waiting for events
phil-opp Apr 5, 2023
bb822a5
Move pending drop token handling to separate function
phil-opp Apr 5, 2023
59f4dde
Improve tracing output
phil-opp Apr 5, 2023
a19a746
Report remaining drop tokens before event stream thread finishes
phil-opp Apr 5, 2023
6688ae3
Close outputs directly on drop to notify subscribers immediately
phil-opp Apr 5, 2023
453498b
Fix: Only send each `InputClosed` event once
phil-opp Apr 5, 2023
e31c6a0
Improve output of object detection example node
phil-opp Apr 5, 2023
56e2126
Fix C dataflow on Windows: Link `oleaut32` library
phil-opp Apr 5, 2023
e413d38
Validate `arrow::ArrayData` creation
phil-opp Apr 5, 2023
ccdbada
Make `send_output` work with both `PyBytes` and arrow arrays
phil-opp Apr 5, 2023
79fd5e9
Rename arrow array access field: `data_arrow` -> `value`
phil-opp Apr 6, 2023
ef37caa
Add arrow support for Python operator inputs
phil-opp Apr 6, 2023
abd850c
Send drop tokens over separate channel
phil-opp Apr 6, 2023
a308c48
Refactor Rust node API to ensure proper stopping
phil-opp Apr 8, 2023
8dff241
Signal daemon when event stream is dropped
phil-opp Apr 8, 2023
e3d0e0c
Enable node tracing for dataflow example
phil-opp Apr 8, 2023
a453adf
Fix: `ReportDropTokens` message expects no reply when using TCP
phil-opp Apr 8, 2023
9d4909b
Adjust log level
phil-opp Apr 8, 2023
cc4acfe
Verify arrow data type in `send_output`
phil-opp Apr 12, 2023
235f6da
Support sending arrow data from operators
phil-opp Apr 12, 2023
c1544f8
Allow accessing data multiple times on Python event type
phil-opp Apr 14, 2023
3a35c22
Enable trace level logging no CI
phil-opp Apr 14, 2023
9644cb0
Merge branch 'main' into arrow-fixes
phil-opp Apr 15, 2023
3eef207
Return optional value for `__getitem__`
phil-opp Apr 15, 2023
d7cd370
Add `allocate_data_sample` and `send_output_sample` methods to Rust n…
phil-opp Apr 15, 2023
41e38f0
Remove largest data size from benchmark example to avoid CI issues
phil-opp Apr 15, 2023
1948a45
Copy outputs directly into shared memory in dora runtime
phil-opp Apr 15, 2023
8ec4523
Send empty map empty arrays to `None` when sending outputs
phil-opp Apr 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci-python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ on:
pull_request:
workflow_dispatch:

env:
RUST_LOG: trace

jobs:
examples:
name: "Python Examples"
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ on:
pull_request:
workflow_dispatch:

env:
RUST_LOG: trace

jobs:
test:
name: "Test"
Expand Down
Loading