Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Venice Push Status System store needs to have a stronger guarantee since it is a critical piece in Push Job #657

Open
2 of 10 tasks
ZacAttack opened this issue Sep 21, 2023 · 1 comment · May be fixed by #1079
Open
2 of 10 tasks
Labels
enhancement New feature or request

Comments

@ZacAttack
Copy link
Contributor

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Feature Request Proposal

Right now, Venice Push Status System Store creation seems to be a best effort in prod and some stores have it and some stores are not.

But polling DaVinci Push Status System Store is in the critical path of push job offline status polling and if push status system store is not there, the push job will never succeed.

One idea is to have a monitoring service in Controller to periodically check the healthiness of the push status system store for each Venice store and fix it automatically.

Motivation

What is the use case for this feature?

This is to improve write path availability.

Details

No response

What component(s) does this bug affect?

  • Controller: This is the control-plane for Venice. Used to create/update/query stores and their metadata.
  • Router: This is the stateless query-routing layer for serving read requests.
  • Server: This is the component that persists all the store data.
  • VenicePushJob: This is the component that pushes derived data from Hadoop to Venice backend.
  • VenicePulsarSink: This is a Sink connector for Apache Pulsar that pushes data from Pulsar into Venice.
  • Thin Client: This is a stateless client users use to query Venice Router for reading store data.
  • Fast Client: This is a stateful client users use to query Venice Server for reading store data.
  • Da Vinci Client: This is an embedded, stateful client that materializes store data locally.
  • Samza: This is the library users use to make nearline updates to store data.
  • Admin Tool: This is the stand-alone client used for ad-hoc operations on Venice.
@ZacAttack ZacAttack added the enhancement New feature or request label Sep 21, 2023
@ZacAttack
Copy link
Contributor Author

In the controller on addVersion, check to make sure the push status system store for the store exists, and if it doesn't exist, try to create it, and if that fails, fail the addVersion call in the controller to abort the push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant