feat: Decouple transformation types from ODFVs #3949

tokoko · 2024-02-12T05:37:51Z

What this PR does / why we need it:
This PR decouples transformation logic and default engine from on demand feature views. It's a first step to eventually resolve - #3945. It introduces a new class OnDemandPandasTransformation and refactors pandas-related code from OnDemandFeatureView. This is to be followed by other types of transformations like OnDemandSqlTransformation and OnDemandSubstraitTransformation.

PR technically breaks OnDemandFeatureView "public" api, as the following code will no longer work:

OnDemandFeatureView(
        name="my-on-demand-feature-view",
        sources=sources,
        schema=[
            Field(name="output1", dtype=Float32),
            Field(name="output2", dtype=Float32),
        ],
        udf=udf1,
        udf_string="udf1 source code",
    )

Instead now you would do:

OnDemandFeatureView(
        name="my-on-demand-feature-view",
        sources=sources,
        schema=[
            Field(name="output1", dtype=Float32),
            Field(name="output2", dtype=Float32),
        ],
        transformation=OnDemandPandasTransformation(
            udf=udf1, udf_string="udf1 source code"
        ),
    )

Having said that, I think that's fine as except for internal tests probably noone creates on demand feature views like this. They instead would use a decorator like shown in docs which remains unchanged. Plus, on demand feature views are considered an alpha feature.

Signed-off-by: tokoko <[email protected]>

HaoXuAI · 2024-02-12T07:45:12Z

protos/feast/core/OnDemandFeatureView.proto

@@ -48,7 +48,9 @@ message OnDemandFeatureViewSpec {
    // Map of sources for this feature view.
    map<string, OnDemandSource> sources = 4;

-    UserDefinedFunction user_defined_function = 5;


This won't be backwards compatible right?

Pretty sure it is. The binary format will be the same and there would have been no way for multiple fields to have been set with previous version, as it's just a single field. Googled a bit just now.. the very last sentence in this blog post seems to also indicate this should be fine.

HaoXuAI · 2024-02-13T19:48:31Z

sdk/python/feast/on_demand_feature_view.py

@@ -51,8 +48,7 @@ class OnDemandFeatureView(BaseFeatureView):
            sources with type FeatureViewProjection.
        source_request_sources: A map from input source names to the actual input
            sources with type RequestSource.
-        udf: The user defined transformation function, which must take pandas dataframes


can we still keep the udf and add deprecated to make it still backward compatible? I like the new field transformation but just in case some teams still use udf in the code base

Sure, that's a good idea

Cool. nit, maybe keep the udf in the comment as well.

It's in the comment for the init method. Do you mean the comment that lists the attributes? it's no longer an attribute, so probably shouldn't be there. It's being converted to transformation in init method and discarded.

ah that make sense

… compatibility Signed-off-by: tokoko <[email protected]>

HaoXuAI

LGTM

Signed-off-by: tokoko <[email protected]>

tokoko · 2024-02-19T19:15:11Z

@sudohainguyen this is no longer breaking. I added udf and udf_string parameters back in per Hao's suggestion.

tokoko · 2024-02-20T07:50:39Z

@sudohainguyen @HaoXuAI any idea why integration tests haven't run for this?

HaoXuAI · 2024-02-23T04:44:21Z

@sudohainguyen @HaoXuAI any idea why integration tests haven't run for this?

I don't know what's going on but i re-run the job hope that works.

sudohainguyen · 2024-02-23T06:20:12Z

@tokoko re-add label helps 🤣

tokoko · 2024-02-23T09:12:10Z

@HaoXuAI Can we merge this before other PRs get in the way? 😄

HaoXuAI · 2024-02-24T08:23:05Z

@HaoXuAI Can we merge this before other PRs get in the way? 😄

Just merged. great job!

franciscojavierarceo · 2024-03-09T18:46:22Z

oh @HaoXuAI @tokoko i have all sorts of opinion here sad i missed this

franciscojavierarceo · 2024-03-09T18:46:38Z

I was just looking to merge the python native transformations we made

* decouple transformation from odfvs Signed-off-by: tokoko <[email protected]> * OnDemandFeatureView: keep udf and udf_string parameters for backwards compatibility Signed-off-by: tokoko <[email protected]> * fix linting issues Signed-off-by: tokoko <[email protected]> * remove unused import in registry protos Signed-off-by: tokoko <[email protected]> --------- Signed-off-by: tokoko <[email protected]>

# [0.36.0](v0.35.0...v0.36.0) (2024-04-16) ### Bug Fixes * Add __eq__, __hash__ to SparkSource for correct comparison ([#4028](#4028)) ([e703b40](e703b40)) * Add conn.commit() to Postgresonline_write_batch.online_write_batch ([#3904](#3904)) ([7d75fc5](7d75fc5)) * Add missing __init__.py to embedded_go ([#4051](#4051)) ([6bb4c73](6bb4c73)) * Add missing init files in infra utils ([#4067](#4067)) ([54910a1](54910a1)) * Added registryPath parameter documentation in WebUI reference ([#3983](#3983)) ([5e0af8f](5e0af8f)), closes [#3974](#3974) [#3974](#3974) * Adding missing init files in materialization modules ([#4052](#4052)) ([df05253](df05253)) * Allow trancated timestamps when converting ([#3861](#3861)) ([bdd7dfb](bdd7dfb)) * Azure blob storage support in Java feature server ([#2319](#2319)) ([#4014](#4014)) ([b9aabbd](b9aabbd)) * Bugfix for grabbing historical data from Snowflake with array type features. ([#3964](#3964)) ([1cc94f2](1cc94f2)) * Bytewax materialization engine fails when loading feature_store.yaml ([#3912](#3912)) ([987f0fd](987f0fd)) * CI unittest warnings ([#4006](#4006)) ([0441b8b](0441b8b)) * Correct the returning class proto type of StreamFeatureView to StreamFeatureViewProto instead of FeatureViewProto. ([#3843](#3843)) ([86d6221](86d6221)) * Create index only if not exists during MySQL online store update ([#3905](#3905)) ([2f99a61](2f99a61)) * Disable minio tests in workflows on master and nightly ([#4072](#4072)) ([c06dda8](c06dda8)) * Disable the Feast Usage feature by default. ([#4090](#4090)) ([b5a7013](b5a7013)) * Dump repo_config by alias ([#4063](#4063)) ([e4bef67](e4bef67)) * Extend SQL registry config with a sqlalchemy_config_kwargs key ([#3997](#3997)) ([21931d5](21931d5)) * Feature Server image startup in OpenShift clusters ([#4096](#4096)) ([9efb243](9efb243)) * Fix copy method for StreamFeatureView ([#3951](#3951)) ([cf06704](cf06704)) * Fix for materializing entityless feature views in Snowflake ([#3961](#3961)) ([1e64c77](1e64c77)) * Fix type mapping spark ([#4071](#4071)) ([3afa78e](3afa78e)) * Fix typo as the cli does not support shortcut-f option. ([#3954](#3954)) ([dd79dbb](dd79dbb)) * Get container host addresses from testcontainers ([#3946](#3946)) ([2cf1a0f](2cf1a0f)) * Handle ComplexFeastType to None comparison ([#3876](#3876)) ([fa8492d](fa8492d)) * Hashlib md5 errors in FIPS for python 3.9+ ([#4019](#4019)) ([6d9156b](6d9156b)) * Making the query_timeout variable as optional int because upstream is considered to be optional ([#4092](#4092)) ([fd5b620](fd5b620)) * Move gRPC dependencies to an extra ([#3900](#3900)) ([f93c5fd](f93c5fd)) * Prevent spamming pull busybox from dockerhub ([#3923](#3923)) ([7153cad](7153cad)) * Quickstart notebook example ([#3976](#3976)) ([b023aa5](b023aa5)) * Raise error when not able read of file source spark source ([#4005](#4005)) ([34cabfb](34cabfb)) * remove not use input parameter in spark source ([#3980](#3980)) ([7c90882](7c90882)) * Remove parentheses in pull_latest_from_table_or_query ([#4026](#4026)) ([dc4671e](dc4671e)) * Remove proto-plus imports ([#4044](#4044)) ([ad8f572](ad8f572)) * Remove unnecessary dependency on mysqlclient ([#3925](#3925)) ([f494f02](f494f02)) * Restore label check for all actions using pull_request_target ([#3978](#3978)) ([591ba4e](591ba4e)) * Revert mypy config ([#3952](#3952)) ([6b8e96c](6b8e96c)) * Rewrite Spark materialization engine to use mapInPandas ([#3936](#3936)) ([dbb59ba](dbb59ba)) * Run feature server w/o gunicorn on windows ([#4024](#4024)) ([584e9b1](584e9b1)) * SqlRegistry _apply_object update statement ([#4042](#4042)) ([ef62def](ef62def)) * Substrait ODFVs for online ([#4064](#4064)) ([26391b0](26391b0)) * Swap security label check on the PR title validation job to explicit permissions instead ([#3987](#3987)) ([f604af9](f604af9)) * Transformation server doesn't generate files from proto ([#3902](#3902)) ([d3a2a45](d3a2a45)) * Trino as an OfflineStore Access Denied when BasicAuthenticaion ([#3898](#3898)) ([49d2988](49d2988)) * Trying to import pyspark lazily to avoid the dependency on the library ([#4091](#4091)) ([a05cdbc](a05cdbc)) * Typo Correction in Feast UI Readme ([#3939](#3939)) ([c16e5af](c16e5af)) * Update actions/setup-python from v3 to v4 ([#4003](#4003)) ([ee4c4f1](ee4c4f1)) * Update typeguard version to >=4.0.0 ([#3837](#3837)) ([dd96150](dd96150)) * Upgrade sqlalchemy from 1.x to 2.x regarding PVE-2022-51668. ([#4065](#4065)) ([ec4c15c](ec4c15c)) * Use CopyFrom() instead of __deepycopy__() for creating a copy of protobuf object. ([#3999](#3999)) ([5561b30](5561b30)) * Using version args to install the correct feast version ([#3953](#3953)) ([b83a702](b83a702)) * Verify the existence of Registry tables in snowflake before calling CREATE sql command. Allow read-only user to call feast apply. ([#3851](#3851)) ([9a3590e](9a3590e)) ### Features * Add duckdb offline store ([#3981](#3981)) ([161547b](161547b)) * Add Entity df in format of a Spark Dataframe instead of just pd.DataFrame or string for SparkOfflineStore ([#3988](#3988)) ([43b2c28](43b2c28)) * Add gRPC Registry Server ([#3924](#3924)) ([373e624](373e624)) * Add local tests for s3 registry using minio ([#4029](#4029)) ([d82d1ec](d82d1ec)) * Add python bytes to array type conversion support proto ([#3874](#3874)) ([8688acd](8688acd)) * Add python client for remote registry server ([#3941](#3941)) ([42a7b81](42a7b81)) * Add Substrait-based ODFV transformation ([#3969](#3969)) ([9e58bd4](9e58bd4)) * Add support for arrays in snowflake ([#3769](#3769)) ([8d6bec8](8d6bec8)) * Added delete_table to redis online store ([#3857](#3857)) ([03dae13](03dae13)) * Adding support for Native Python feature transformations for ODFVs ([#4045](#4045)) ([73bc853](73bc853)) * Bumping requirements ([#4079](#4079)) ([1943056](1943056)) * Decouple transformation types from ODFVs ([#3949](#3949)) ([0a9fae8](0a9fae8)) * Dropping Python 3.8 from local integration tests and integration tests ([#3994](#3994)) ([817995c](817995c)) * Dropping python 3.8 requirements files from the project. ([#4021](#4021)) ([f09c612](f09c612)) * Dropping the support for python 3.8 version from feast ([#4010](#4010)) ([a0f7472](a0f7472)) * Dropping unit tests for Python 3.8 ([#3989](#3989)) ([60f24f9](60f24f9)) * Enable Arrow-based columnar data transfers ([#3996](#3996)) ([d8d7567](d8d7567)) * Enable Vector database and retrieve_online_documents API ([#4061](#4061)) ([ec19036](ec19036)) * Kubernetes materialization engine written based on bytewax ([#4087](#4087)) ([7617bdb](7617bdb)) * Lint with ruff ([#4043](#4043)) ([7f1557b](7f1557b)) * Make arrow primary interchange for offline ODFV execution ([#4083](#4083)) ([9ed0a09](9ed0a09)) * Pandas v2 compatibility ([#3957](#3957)) ([64459ad](64459ad)) * Pull duckdb from contribs, add to CI ([#4059](#4059)) ([318a2b8](318a2b8)) * Refactor ODFV schema inference ([#4076](#4076)) ([c50a9ff](c50a9ff)) * Refactor registry caching logic into a separate class ([#3943](#3943)) ([924f944](924f944)) * Rename OnDemandTransformations to Transformations ([#4038](#4038)) ([9b98eaf](9b98eaf)) * Revert updating dependencies so that feast can be run on 3.11. ([#3968](#3968)) ([d3c68fb](d3c68fb)), closes [#3958](#3958) * Rewrite ibis point-in-time-join w/o feast abstractions ([#4023](#4023)) ([3980e0c](3980e0c)) * Support s3gov schema by snowflake offline store during materialization ([#3891](#3891)) ([ea8ad17](ea8ad17)) * Update odfv test ([#4054](#4054)) ([afd52b8](afd52b8)) * Update pyproject.toml to use Python 3.9 as default ([#4011](#4011)) ([277b891](277b891)) * Update the Pydantic from v1 to v2 ([#3948](#3948)) ([ec11a7c](ec11a7c)) * Updating dependencies so that feast can be run on 3.11. ([#3958](#3958)) ([59639db](59639db)) * Updating protos to separate transformation ([#4018](#4018)) ([c58ef74](c58ef74)) ### Reverts * Reverting bumping requirements ([#4081](#4081)) ([1ba65b4](1ba65b4)), closes [#4079](#4079) * Verify the existence of Registry tables in snowflake… ([#3907](#3907)) ([c0d358a](c0d358a)), closes [#3851](#3851)

decouple transformation from odfvs

5a5278d

Signed-off-by: tokoko <[email protected]>

HaoXuAI reviewed Feb 12, 2024

View reviewed changes

HaoXuAI reviewed Feb 13, 2024

View reviewed changes

OnDemandFeatureView: keep udf and udf_string parameters for backwards…

f0180e6

… compatibility Signed-off-by: tokoko <[email protected]>

HaoXuAI approved these changes Feb 15, 2024

View reviewed changes

HaoXuAI added ok-to-test approved labels Feb 15, 2024

tokoko added 3 commits February 17, 2024 23:08

Merge branch 'master' into odfv-pluggable

b3221fb

fix linting issues

9b936f1

Signed-off-by: tokoko <[email protected]>

remove unused import in registry protos

2f0c67c

Signed-off-by: tokoko <[email protected]>

sudohainguyen added the compat/breaking Breaking user-facing change label Feb 19, 2024

sudohainguyen removed the compat/breaking Breaking user-facing change label Feb 19, 2024

sudohainguyen added ok-to-test and removed approved ok-to-test labels Feb 23, 2024

sudohainguyen added the approved label Feb 23, 2024

HaoXuAI merged commit 0a9fae8 into feast-dev:master Feb 24, 2024
43 checks passed

tokoko deleted the odfv-pluggable branch February 24, 2024 12:02

franciscojavierarceo mentioned this pull request Mar 25, 2024

feat: Rename OnDemandTransformations to Transformations #4038

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Decouple transformation types from ODFVs #3949

feat: Decouple transformation types from ODFVs #3949

tokoko commented Feb 12, 2024 •

edited

Loading

HaoXuAI Feb 12, 2024

tokoko Feb 12, 2024

HaoXuAI Feb 13, 2024

tokoko Feb 13, 2024

tokoko Feb 13, 2024

HaoXuAI Feb 14, 2024

tokoko Feb 14, 2024

HaoXuAI Feb 15, 2024

HaoXuAI left a comment

tokoko commented Feb 19, 2024

tokoko commented Feb 20, 2024

HaoXuAI commented Feb 23, 2024

sudohainguyen commented Feb 23, 2024

tokoko commented Feb 23, 2024

HaoXuAI commented Feb 24, 2024

franciscojavierarceo commented Mar 9, 2024

franciscojavierarceo commented Mar 9, 2024

feat: Decouple transformation types from ODFVs #3949

feat: Decouple transformation types from ODFVs #3949

Conversation

tokoko commented Feb 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HaoXuAI left a comment

Choose a reason for hiding this comment

tokoko commented Feb 19, 2024

tokoko commented Feb 20, 2024

HaoXuAI commented Feb 23, 2024

sudohainguyen commented Feb 23, 2024

tokoko commented Feb 23, 2024

HaoXuAI commented Feb 24, 2024

franciscojavierarceo commented Mar 9, 2024

franciscojavierarceo commented Mar 9, 2024

tokoko commented Feb 12, 2024 •

edited

Loading