Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for single operator nodes in YAML dataflow specification #50

Merged
merged 3 commits into from
Jul 27, 2022

Conversation

phil-opp
Copy link
Collaborator

Single operator nodes are specified through an operator field, in contrast to the operators list of normal runtime nodes. Since there is only a single operator on such nodes, specifying an additional operator ID is not necessary. If no operator ID is given, the coordinator will assign the ID op.

Other nodes can also reference outputs of single-operator nodes by only specifying the node ID, the operator ID is optional here as well. The coordinator will automatically resolve these links.

This feature is useful for Python operators because all operators on the same node use a single global interpreter lock (GIL). By using a separate node for each Python operator, they run in different processes without GIL contention.

@phil-opp phil-opp requested a review from haixuanTao July 26, 2022 14:05
binaries/runtime/src/operator/mod.rs Outdated Show resolved Hide resolved
binaries/runtime/src/operator/mod.rs Outdated Show resolved Hide resolved
binaries/runtime/src/operator/mod.rs Outdated Show resolved Hide resolved
@phil-opp phil-opp merged commit e98fc07 into main Jul 27, 2022
@phil-opp phil-opp deleted the python-operator-alias branch July 27, 2022 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants