Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCS][Storage unification 3/n] Deprecate MemoryInternalKV, use StoreClientInternalKV<MemoryStoreClient> instead #24211

Merged
merged 5 commits into from
Apr 28, 2022

Conversation

scv119
Copy link
Contributor

@scv119 scv119 commented Apr 26, 2022

Why are these changes needed?

In this PR, we deprecated MemoryInternalKV. Instead, we use a wrapper that wraps around MemoryStoreClient. This PR is build upon #23754 and #23725

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@scv119 scv119 changed the title [GCS][Storage unification 3/n] Use storage client by default [GCS][Storage unification 3/n] Use storage client by default, deprecate RedisInternalKV and MemoryInternalKV Apr 26, 2022
@scv119 scv119 changed the title [GCS][Storage unification 3/n] Use storage client by default, deprecate RedisInternalKV and MemoryInternalKV [WIP][GCS][Storage unification 3/n] Use storage client by default, deprecate RedisInternalKV and MemoryInternalKV Apr 26, 2022
@scv119 scv119 changed the title [WIP][GCS][Storage unification 3/n] Use storage client by default, deprecate RedisInternalKV and MemoryInternalKV [GCS][Storage unification 3/n] Deprecate MemoryInternalKV, use StoreClientInternalKV<MemoryStoreClient> instead Apr 26, 2022
Comment on lines 447 to 452
if (storage_type_ == "redis") {
instance = std::make_unique<RedisInternalKV>(GetRedisClientOptions());
} else if (storage_type_ == "memory") {
instance = std::make_unique<MemoryInternalKV>(main_service_);
instance = std::make_unique<StoreClientInternalKV>(
std::make_unique<InMemoryStoreClient>(main_service_));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just use store clients here? Also for redis.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think this will mostly impact Ant. cc @raulchen to see if we should turn it on by default.

Copy link
Contributor

@fishbone fishbone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remind me why we can't build table on top of internal kv?

@fishbone fishbone added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Apr 27, 2022
@scv119
Copy link
Contributor Author

scv119 commented Apr 27, 2022

Could you remind me why we can't build table on top of internal kv?

Yeah I think it's because of AsyncBatchDelete, AsyncMultiGet and GetNextJobID are not easy to implement with InternalKV. If you add them the two interfaces are almost identical.

@kfstorm kfstorm assigned WangTaoTheTonic and unassigned kfstorm Apr 28, 2022
@fishbone
Copy link
Contributor

My feeling is that we should hide the underline store from internal kv and it should be built upon StoreClient directly.

In the end, the store client will be used by table and kv. We can use memory or Redis as the backend of the store client.

@scv119 scv119 added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Apr 28, 2022
@scv119 scv119 merged commit 3bc6902 into ray-project:master Apr 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants