Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Set optional full-scan for deletion #4189

Merged

Conversation

phil-park
Copy link
Contributor

What this PR does / why we need it:

In this link (#3857), I added function to perform a full-scan and delete data to save Redis memory. However, full-scan in Redis does not work beautifully, takes too long at production scale, and often fails in CI that performs feast-apply. So, I think it would be a good idea to add the 'full_scan_for_deletion' option in redis so that the user can select and delete it.

Which issue(s) this PR fixes:

Fixes

@phil-park phil-park requested a review from DvirDukhan as a code owner May 9, 2024 07:48
@tokoko
Copy link
Collaborator

tokoko commented May 9, 2024

I think this might be valuable in the interim, although you might want to chime in the discussion here as well #4133. The implicit consensus from there is to add a feast cleanup command that will delete all redundant keys from online store (keys that belong to deleted tables and also keys containing feature values that have expired according to ttl).


for table in tables_to_delete:
self.delete_table(config, table)
if online_store_config.full_scan_for_deletion:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the old implementation deletes tables by default.
should full_scan_for_deletion's default value be True?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you said, I set default to True for compatibility with older implementations.
If compatibility is not enforced, I think False would be better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@breno-costa Changed to default True to ensure backward compatibility

Copy link
Collaborator

@tokoko tokoko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@franciscojavierarceo franciscojavierarceo merged commit b9cadd5 into feast-dev:master May 27, 2024
17 checks passed
franciscojavierarceo pushed a commit that referenced this pull request Jun 18, 2024
# [0.39.0](v0.38.0...v0.39.0) (2024-06-18)

### Bug Fixes

* Feast UI importlib change ([#4248](#4248)) ([5d486b8](5d486b8))
* Feature server no_feature_log argument error ([#4255](#4255)) ([15524ce](15524ce))
* Feature UI Server image won't start in an OpenShift cluster ([#4250](#4250)) ([4891f76](4891f76))
* Handles null values in data during GO Feature retrieval ([#4274](#4274)) ([c491e57](c491e57))
* Make Java gRPC client use timeouts as expected ([#4237](#4237)) ([f5a37c1](f5a37c1))
* Remove self assignment code line. ([#4238](#4238)) ([e514f66](e514f66))
* Set default values for feature_store.serve() function ([#4225](#4225)) ([fa74438](fa74438))

### Features

* Add online_read_async for dynamodb ([#4244](#4244)) ([b5ef384](b5ef384))
* Add the ability to list objects by `tags` ([#4246](#4246)) ([fbf92da](fbf92da))
* Added deadline to gRPC Java client ([#4217](#4217)) ([ff429c9](ff429c9))
* Adding vector search for sqlite ([#4176](#4176)) ([2478831](2478831))
* Change get_online_features signature, move online retrieval functions to utils ([#4278](#4278)) ([7287662](7287662))
* Feature/adding remote online store ([#4226](#4226)) ([9454d7c](9454d7c))
* List all feature views ([#4256](#4256)) ([36a574d](36a574d))
* Make RegistryServer writable ([#4231](#4231)) ([79e1143](79e1143))
* Remote offline Store  ([#4262](#4262)) ([28a3d24](28a3d24))
* Set optional full-scan for deletion ([#4189](#4189)) ([b9cadd5](b9cadd5))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants