feat: Add Rockset as an OnlineStore #3405

danielin917 · 2022-12-19T23:10:16Z

This commit adds Rockset as a contributed online store. It implements the Update, Teardown, Read and Write apis using the Rockset python client.

Signed-off-by: Daniel Lin [email protected]

What this PR does / why we need it:
This PR adds Rockset as an implemented OnlineFeatureStore

Which issue(s) this PR fixes:

Fixes #
N/A

danielin917 · 2022-12-19T23:41:46Z

Hello! Integrations tests have all passed for the online store. Did not add a local universal test to the Makefile since there is no docker image to use.

danielin917 · 2022-12-19T23:42:02Z

/assign @mavysavydav

danielin917 · 2023-01-10T21:41:11Z

👋 Just checking in to see if there is anything I should do to make this easier to review

mavysavydav · 2023-01-11T18:10:32Z

hi! Do you know who might have context on these changes who may be better suited to review? I also recently stepped down from my role as a maintainer so am not as involved in reviewing these for the time being. Apologies for the delay! If there's nobody you have in mind who might be a good fit for reviewing, I can pass this on to someone else in the maintainer team.

danielin917 · 2023-01-11T18:47:52Z

Not sure who the best person is, I think @felixwang9817 mentioned he was interested but not sure who else could review

mavysavydav · 2023-01-11T21:15:14Z

@felixwang9817 -- were you interested in taking a look at this or do you know who might be?

sdk/python/requirements/py3.9-ci-requirements.txt

felixwang9817

looks good overall! just a few nits

docs/SUMMARY.md

docs/reference/online-stores/README.md

docs/reference/online-stores/rockset.md

sdk/python/requirements/py3.8-ci-requirements.txt

setup.py

felixwang9817 · 2023-01-16T01:25:37Z

hey @danielin917 thanks for the PR and sorry for the delay on the review - I'm happy to help you get this PR merged

I left a few nits above, and I think you'll want to rebase to ensure you can pass the latest tests!

danielin917 · 2023-01-18T21:45:54Z

Thanks for all the comments! Reran make test-python-integration and don't see any Rockset errors

felixwang9817

/lgtm

felixwang9817 · 2023-01-19T02:06:33Z

@danielin917 I stamped but looks like there's actually a few outstanding errors that need to be addressed

danielin917 · 2023-01-24T18:42:01Z

ERROR: Cannot uninstall types-protobuf 4.21.0.3, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps types-protobuf==4.21.0.3'.

Not sure if there is a way to just rerun this test and this goes away or might need to play with dependencies more.

felixwang9817 · 2023-01-24T19:06:53Z

ERROR: Cannot uninstall types-protobuf 4.21.0.3, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps types-protobuf==4.21.0.3'.

Not sure if there is a way to just rerun this test and this goes away or might need to play with dependancies more.

@danielin917 looks like it's probably just a dependencies issue - I'll look into it today/tmrw!

felixwang9817 · 2023-01-25T08:26:49Z

setup.py

@@ -151,7 +156,7 @@
        "minio==7.1.0",
        "mock==2.0.0",
        "moto<4",
-        "mypy>=0.931",
+        "mypy>=0.931, <=0.981",


@danielin917 why is this necessary?

I was seeing a lot of unrelated lint errors before from different files. But actually now after removing the cap lint passed so not sure what changed

felixwang9817 · 2023-01-25T08:27:19Z

setup.py

@@ -174,7 +179,7 @@
        "assertpy==1.1",
        "pip-tools",
        "pybindgen",
-        "types-protobuf",
+        "types-protobuf<4",


@danielin917 why is this necessary? the error message in the failing unit tests is ERROR: Cannot uninstall types-protobuf 4.21.0.3, RECORD file not found. You might be able to recover from this via: 'pip install --force-reinstall --no-deps types-protobuf==4.21.0.3'., so this line seems very suspicious to me

Hey yea, mentioned above that I pinned this type because the linter was producing a lot of errors using the new version Library stubs not installed for "google.*". One thing that might be causing this is that we have:
"mypy-protobuf==3.1",

danielin917 · 2023-01-25T18:22:39Z

Not sure what happened between earlier testing and now but after removing cap on mypy and types-proto and this time I saw no lint errors locally

danielin917 · 2023-01-26T22:39:32Z

~~Ugh, I am going to open a separate commit for upgrading mypy and deal with all of the linting issues there to unblock this diff as is instead of playing around with package dependencies~~

Looks like these were recently pinned? make lint-python works locally after rebase

This commit adds Rockset as a contributed online store. It implements the Update, Teardown, Read and Write apis using the Rockset pythonn client. Signed-off-by: Daniel Lin <[email protected]>

danielin917 · 2023-02-08T17:43:47Z

Sorry to keep pinging just want to test the workflow after the rebase

felixwang9817

/lgtm

feast-ci-bot · 2023-02-09T07:19:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danielin917, felixwang9817

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [felixwang9817]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

felixwang9817 · 2023-02-09T07:45:30Z

@danielin917 we finally got this merged in!! thanks a ton for the contribution, and for your patience throughout the review process - I know we ran into a bunch of issues throughout 😅

danielin917 · 2023-02-09T18:36:10Z

Ahh sweet! Thanks @felixwang9817 and @adchia for the assist!

# [0.30.0](v0.29.0...v0.30.0) (2023-03-17) ### Bug Fixes * Add description attribute to the Field.from_proto method ([#3469](#3469)) ([473f8d9](473f8d9)) * Add filesystem kwargs when read prev_table on FileRetrievalJob (… ([#3491](#3491)) ([dca4745](dca4745)), closes [#3490](#3490) * Feature view `entities` from_proto type ([#3524](#3524)) ([57bbb61](57bbb61)) * Fix missing requests requirement after GCP requirement removed. Make BigQuerySource not require gcp extra ([2c85421](2c85421)) * Fix SQL Registry cache miss ([#3482](#3482)) ([3249b97](3249b97)) * Fixed path inside quickstart notebook ([#3456](#3456)) ([66edc32](66edc32)) * Improve BQ point-in-time joining scalability ([#3429](#3429)) ([ff66784](ff66784)) * Pin typeguard to 2.13.3 which is what we are currently using. ([#3542](#3542)) ([61f6fb0](61f6fb0)) * Protobuf lower bound to 3.20 to alert that Feast is incompatible with tensorflow ([#3476](#3476)) ([9ca59e3](9ca59e3)) * Spark kafka processor sorting ([#3479](#3479)) ([f2cbf43](f2cbf43)) * UI working behind base url ([#3514](#3514)) ([9a3fd98](9a3fd98)) * Update go dependencies ([#3512](#3512)) ([bada97c](bada97c)) ### Features * Add Rockset as an OnlineStore ([#3405](#3405)) ([fd91cda](fd91cda)) * Add Snowflake Registry ([#3363](#3363)) ([ec1e61d](ec1e61d)) * Adding query timeout to `to_df` and `to_arrow` retrieval methods ([#3505](#3505)) ([bab6644](bab6644)) * adds k8s config options to Bytewax materialization engine ([#3518](#3518)) ([1883f55](1883f55))

# [0.30.0](v0.29.0...v0.30.0) (2023-03-24) ### Bug Fixes * Add description attribute to the Field.from_proto method ([#3469](#3469)) ([473f8d9](473f8d9)) * Add filesystem kwargs when read prev_table on FileRetrievalJob (… ([#3491](#3491)) ([dca4745](dca4745)), closes [#3490](#3490) * Bytewax image pull secret config ([#3547](#3547)) ([d2d13b1](d2d13b1)) * Clean up Rockset Online Store for use ([#3549](#3549)) ([a76c6d0](a76c6d0)) * Feature view `entities` from_proto type ([#3524](#3524)) ([57bbb61](57bbb61)) * Fix missing requests requirement after GCP requirement removed. Make BigQuerySource not require gcp extra ([2c85421](2c85421)) * Fix SQL Registry cache miss ([#3482](#3482)) ([3249b97](3249b97)) * Fixed path inside quickstart notebook ([#3456](#3456)) ([66edc32](66edc32)) * Improve BQ point-in-time joining scalability ([#3429](#3429)) ([ff66784](ff66784)) * Pin typeguard to 2.13.3 which is what we are currently using. ([#3542](#3542)) ([61f6fb0](61f6fb0)) * Protobuf lower bound to 3.20 to alert that Feast is incompatible with tensorflow ([#3476](#3476)) ([9ca59e3](9ca59e3)) * Spark kafka processor sorting ([#3479](#3479)) ([f2cbf43](f2cbf43)) * UI working behind base url ([#3514](#3514)) ([9a3fd98](9a3fd98)) * Update go dependencies ([#3512](#3512)) ([bada97c](bada97c)) ### Features * Add Rockset as an OnlineStore ([#3405](#3405)) ([fd91cda](fd91cda)) * Add Snowflake Registry ([#3363](#3363)) ([ec1e61d](ec1e61d)) * Added SnowflakeConnection caching ([#3531](#3531)) ([f9f8df2](f9f8df2)) * Adding query timeout to `to_df` and `to_arrow` retrieval methods ([#3505](#3505)) ([bab6644](bab6644)) * adds k8s config options to Bytewax materialization engine ([#3518](#3518)) ([1883f55](1883f55))

feast-ci-bot added the size/XXL label Dec 19, 2022

danielin917 marked this pull request as draft December 19, 2022 23:29

feast-ci-bot added the do-not-merge/work-in-progress label Dec 19, 2022

danielin917 force-pushed the add_rockset_as_online_store branch from ac2da51 to 2b6c591 Compare December 19, 2022 23:34

danielin917 marked this pull request as ready for review December 19, 2022 23:35

feast-ci-bot removed the do-not-merge/work-in-progress label Dec 19, 2022

feast-ci-bot assigned mavysavydav Dec 19, 2022

adchia reviewed Jan 12, 2023

View reviewed changes

sdk/python/requirements/py3.9-ci-requirements.txt Outdated Show resolved Hide resolved

felixwang9817 reviewed Jan 16, 2023

View reviewed changes

danielin917 force-pushed the add_rockset_as_online_store branch from 2b6c591 to 4756df4 Compare January 18, 2023 21:44

danielin917 requested review from felixwang9817 and adchia and removed request for felixwang9817 January 18, 2023 21:45

danielin917 force-pushed the add_rockset_as_online_store branch from 4756df4 to af7b49a Compare January 18, 2023 21:46

danielin917 requested review from felixwang9817 and removed request for adchia January 18, 2023 23:26

felixwang9817 approved these changes Jan 19, 2023

View reviewed changes

feast-ci-bot assigned felixwang9817 Jan 19, 2023

feast-ci-bot added lgtm approved labels Jan 19, 2023

danielin917 force-pushed the add_rockset_as_online_store branch from af7b49a to 490d723 Compare January 22, 2023 17:33

feast-ci-bot added the lgtm label Jan 24, 2023

felixwang9817 reviewed Jan 25, 2023

View reviewed changes

danielin917 force-pushed the add_rockset_as_online_store branch from f93458b to 3b66a2e Compare January 25, 2023 18:21

feast-ci-bot removed the lgtm label Jan 25, 2023

danielin917 force-pushed the add_rockset_as_online_store branch 2 times, most recently from 8bb7be7 to 5134577 Compare January 25, 2023 22:44

feat: Add Rockset as an OnlineStore

a799056

This commit adds Rockset as a contributed online store. It implements the Update, Teardown, Read and Write apis using the Rockset pythonn client. Signed-off-by: Daniel Lin <[email protected]>

danielin917 force-pushed the add_rockset_as_online_store branch from 5134577 to a799056 Compare February 4, 2023 03:27

feast-ci-bot added size/XL and removed size/XXL labels Feb 4, 2023

danielin917 requested review from felixwang9817 and adchia and removed request for felixwang9817 and adchia February 7, 2023 21:10

felixwang9817 approved these changes Feb 9, 2023

View reviewed changes

feast-ci-bot added the lgtm label Feb 9, 2023

feast-ci-bot merged commit fd91cda into feast-dev:master Feb 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Rockset as an OnlineStore #3405

feat: Add Rockset as an OnlineStore #3405

danielin917 commented Dec 19, 2022 •

edited

Loading

danielin917 commented Dec 19, 2022

danielin917 commented Dec 19, 2022

danielin917 commented Jan 10, 2023

mavysavydav commented Jan 11, 2023

danielin917 commented Jan 11, 2023

mavysavydav commented Jan 11, 2023

felixwang9817 left a comment

felixwang9817 commented Jan 16, 2023

danielin917 commented Jan 18, 2023 •

edited

Loading

felixwang9817 left a comment

felixwang9817 commented Jan 19, 2023

danielin917 commented Jan 24, 2023 •

edited

Loading

felixwang9817 commented Jan 24, 2023

felixwang9817 Jan 25, 2023

danielin917 Jan 25, 2023

felixwang9817 Jan 25, 2023

danielin917 Jan 25, 2023

danielin917 commented Jan 25, 2023

danielin917 commented Jan 26, 2023 •

edited

Loading

danielin917 commented Feb 8, 2023

felixwang9817 left a comment

feast-ci-bot commented Feb 9, 2023

felixwang9817 commented Feb 9, 2023

danielin917 commented Feb 9, 2023

feat: Add Rockset as an OnlineStore #3405

feat: Add Rockset as an OnlineStore #3405

Conversation

danielin917 commented Dec 19, 2022 • edited Loading

danielin917 commented Dec 19, 2022

danielin917 commented Dec 19, 2022

danielin917 commented Jan 10, 2023

mavysavydav commented Jan 11, 2023

danielin917 commented Jan 11, 2023

mavysavydav commented Jan 11, 2023

felixwang9817 left a comment

Choose a reason for hiding this comment

felixwang9817 commented Jan 16, 2023

danielin917 commented Jan 18, 2023 • edited Loading

felixwang9817 left a comment

Choose a reason for hiding this comment

felixwang9817 commented Jan 19, 2023

danielin917 commented Jan 24, 2023 • edited Loading

felixwang9817 commented Jan 24, 2023

felixwang9817 Jan 25, 2023

Choose a reason for hiding this comment

danielin917 Jan 25, 2023

Choose a reason for hiding this comment

felixwang9817 Jan 25, 2023

Choose a reason for hiding this comment

danielin917 Jan 25, 2023

Choose a reason for hiding this comment

danielin917 commented Jan 25, 2023

danielin917 commented Jan 26, 2023 • edited Loading

danielin917 commented Feb 8, 2023

felixwang9817 left a comment

Choose a reason for hiding this comment

feast-ci-bot commented Feb 9, 2023

felixwang9817 commented Feb 9, 2023

danielin917 commented Feb 9, 2023

danielin917 commented Dec 19, 2022 •

edited

Loading

danielin917 commented Jan 18, 2023 •

edited

Loading

danielin917 commented Jan 24, 2023 •

edited

Loading

danielin917 commented Jan 26, 2023 •

edited

Loading