Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[controller][compat] Controller part change for supporting separate real-time topic functionality for hybrid stores. #1172

Merged
merged 4 commits into from
Sep 25, 2024

Conversation

haoxu07
Copy link
Contributor

@haoxu07 haoxu07 commented Sep 12, 2024

Summary, imperative, start upper case, don't end with a period

This change add store version level config separateRealTimeTopicEnabled to allow creating separate real-time topic for isolate bulk real-time traffic in short time range from incremental push. There is cluster level config for enabling this config for all newly converted hybrid store with incremental push enabled.

  1. Creation: it will be automatically created by checking whether incremental push and separateRealTimeTopicEnabled is enabled when hybrid enabled store version is added and normal real-time topic is created.
  2. Deletion: separate real time topic will be cleaned up when normal real-time topic is deleted due to all hybrid store version has been deleted.

Added Integration test to see if incremental push job would send expected traffic to the separate real-time topic after new store is turned this feature.

PS: skip the flaky test: testDaVinciMemoryLimitShouldFailLargeDataPushAndResumeHybridStore.

How was this PR tested?

Does this PR introduce any user-facing changes?

  • No. You can skip the rest of this section.
  • Yes. Make sure to explain your proposed changes and call out the behavior change.

@haoxu07 haoxu07 marked this pull request as ready for review September 13, 2024 17:40
Copy link
Contributor

@sixpluszero sixpluszero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Hao for the change. I take one pass, overall it looks good, I left a few small comments for clarification.

@haoxu07 haoxu07 changed the title [controller] Controller part change for supporting separate real-time topic functionality for hybrid stores. [controller][compat] Controller part change for supporting separate real-time topic functionality for hybrid stores. Sep 18, 2024
@haoxu07 haoxu07 force-pushed the separateTopicController branch 2 times, most recently from 2afaf9e to 69426a9 Compare September 20, 2024 00:04
@haoxu07 haoxu07 force-pushed the separateTopicController branch 2 times, most recently from 09ea3a0 to 8449734 Compare September 24, 2024 21:15
Copy link
Contributor

@sixpluszero sixpluszero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the change! Let's get the unrelated flaky test passed (should address that separately outside this PR scope) and we can get this merged.

@haoxu07 haoxu07 merged commit 939d655 into linkedin:main Sep 25, 2024
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants