Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Workflow service type and handler type #1506

Merged
merged 10 commits into from
May 15, 2024

Conversation

slinkydeveloper
Copy link
Contributor

@slinkydeveloper slinkydeveloper commented May 13, 2024

Fix #1505

This PR:

  • Adds the new service type and handler type, plus reorganizes a bit the internal representation of service types
  • Implements the run-once semantics of workflow methods
  • Allows the user to tweak the completion_retention_time of the workflow from the admin api

Also merge ServiceType and HandlerType in InvocationTargetType. This is what the ingress and invoker will use from now on.
@slinkydeveloper slinkydeveloper changed the title [WIP] Add Workflow service type and handler type Add Workflow service type and handler type May 14, 2024
Copy link
Contributor

@tillrohrmann tillrohrmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for creating this PR @slinkydeveloper. The changes look good to me. I guess the e2e are failing because of the changes to the admin API. +1 for merging after resolving my minor comments.

crates/admin/src/schema_registry/updater.rs Show resolved Hide resolved
crates/ingress-http/src/handler/service_handler.rs Outdated Show resolved Hide resolved
Comment on lines -124 to +118
oneof idempotency_key {
google.protobuf.Empty idempotency_key_none = 12;
string idempotency_key_value = 13;
}
optional string idempotency_key = 12;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are gonna have some nice merge conflicts with one of my PRs 😅

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😨

crates/storage-api/src/storage.rs Outdated Show resolved Hide resolved
Comment on lines 898 to 902
effects.unlock_service_id(
invocation_target
.as_keyed_service_id()
.expect("Workflow methods must have keyed service id"),
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a side note: Introducing special cases makes it a tad bit harder to follow what is happening in the state machine. For example, I was stumbling upon this call because I thought that we will unlock the service when the inbox gets popped and there are no more invocations. However, this is not happening because we are only doing it for exclusive virtual objects.

Copy link
Contributor Author

@slinkydeveloper slinkydeveloper May 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is not much i can act on for this comment i think. I hope once we get rid of the effect thing, it will be much easier to track all the state machine read/writes and all those special casings, because at least i'll be able to follow the full flow.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not necessarily the effects but the different cases that we handle slightly differently. If we could unify the cases, then things would become simpler, I believe.

Copy link
Contributor Author

@slinkydeveloper slinkydeveloper May 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we could unify the cases, then things would become simpler, I believe.

As much as I agree with you, right now I unfortunately don't see this happening. The different semantics we want to support are inherently different, and this will only get worse in future... I think we should instead strive to make the "capabilities" of the state machine more and more modular and mixable between each other, so that the state machine reasons more about "This invocation needs feature x, y, z" rather than "this invocation needs these behaviours because it's a workflow call". A bit in a rule engine fashion, if you get what i mean. Testability could also gain from this design approach a lot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as a side note, i do like a lot the approach of ECS engines, it's a bit of a mix of rule engine where the input is wildly dynamic. We don't necessarily need that, but still some rule engine of some sort could be interesting here...

@slinkydeveloper slinkydeveloper merged commit ab3c57e into restatedev:main May 15, 2024
4 checks passed
@slinkydeveloper slinkydeveloper deleted the issues/1505 branch May 15, 2024 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement Workflow service and handler type
2 participants