Pydantic 2.0 support

[mjurkus]

Is your feature request related to a problem? Please describe.
I need to install feast with Pydantic 2.0 support, but it's pinned to pydantic>=1,<2

Describe the solution you'd like
Update dependency to pydantic>=2

Pydantic v2 offers improved performance (2x-50x faster), which can contribute to the overall efficiency of Feast.

Updating Pydantic to version 2 will be a valuable enhancement to the feast, and I am willing to contribute to this update if needed. I look forward to hearing your thoughts on this proposal and am open to discussing it further.

[shuchu]

Here is a list of Python files that need to do transformation :

(.venv) ubuntu➜~/feast/sdk(master)» grep --include=\*.py -rnw ./python -e "pydantic"                                                        [21:32:27]
./python/feast/feature_server.py:14:from pydantic import BaseModel
./python/feast/infra/offline_stores/redshift.py:22:from pydantic import StrictStr, root_validator
./python/feast/infra/offline_stores/redshift.py:23:from pydantic.typing import Literal
./python/feast/infra/offline_stores/file.py:13:from pydantic.typing import Literal
./python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:14:from pydantic import StrictStr
./python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py:22:from pydantic.typing import Literal
./python/feast/infra/offline_stores/contrib/trino_offline_store/trino.py:8:from pydantic import Field, FilePath, SecretStr, StrictBool, StrictStr, root_validator
./python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py:13:from pydantic.types import StrictStr
./python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py:14:from pydantic.typing import Literal
./python/feast/infra/offline_stores/contrib/athena_offline_store/athena.py:20:from pydantic import StrictStr
./python/feast/infra/offline_stores/contrib/athena_offline_store/athena.py:21:from pydantic.typing import Literal
./python/feast/infra/offline_stores/bigquery.py:22:from pydantic import ConstrainedStr, StrictStr, validator
./python/feast/infra/offline_stores/bigquery.py:23:from pydantic.typing import Literal
./python/feast/infra/offline_stores/snowflake.py:25:from pydantic import Field, StrictStr
./python/feast/infra/offline_stores/snowflake.py:26:from pydantic.typing import Literal
./python/feast/infra/utils/postgres/postgres_config.py:4:from pydantic import StrictStr
./python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py:13:from pydantic import StrictStr
./python/feast/infra/materialization/aws_lambda/lambda_engine.py:10:from pydantic import StrictStr
./python/feast/infra/materialization/snowflake_engine.py:10:from pydantic import Field, StrictStr
./python/feast/infra/registry/sql.py:9:from pydantic import StrictStr
./python/feast/infra/registry/snowflake.py:10:from pydantic import Field, StrictStr
./python/feast/infra/registry/snowflake.py:11:from pydantic.schema import Literal
./python/feast/infra/online_stores/contrib/cassandra_online_store/cassandra_online_store.py:36:from pydantic import StrictFloat, StrictInt, StrictStr
./python/feast/infra/online_stores/contrib/cassandra_online_store/cassandra_online_store.py:37:from pydantic.typing import Literal
./python/feast/infra/online_stores/contrib/hazelcast_online_store/hazelcast_online_store.py:29:from pydantic import StrictStr
./python/feast/infra/online_stores/contrib/mysql_online_store/mysql.py:8:from pydantic import StrictStr
./python/feast/infra/online_stores/contrib/hbase_online_store/hbase.py:8:from pydantic import StrictStr
./python/feast/infra/online_stores/contrib/hbase_online_store/hbase.py:9:from pydantic.typing import Literal
./python/feast/infra/online_stores/contrib/postgres.py:12:from pydantic.schema import Literal
./python/feast/infra/online_stores/datastore.py:22:from pydantic import PositiveInt, StrictStr
./python/feast/infra/online_stores/datastore.py:23:from pydantic.typing import Literal
./python/feast/infra/online_stores/sqlite.py:21:from pydantic import StrictStr
./python/feast/infra/online_stores/sqlite.py:22:from pydantic.schema import Literal
./python/feast/infra/online_stores/dynamodb.py:19:from pydantic import StrictBool, StrictStr
./python/feast/infra/online_stores/dynamodb.py:20:from pydantic.typing import Literal, Union
./python/feast/infra/online_stores/bigtable.py:10:from pydantic import StrictStr
./python/feast/infra/online_stores/bigtable.py:11:from pydantic.typing import Literal
./python/feast/infra/online_stores/redis.py:32:from pydantic import StrictStr
./python/feast/infra/online_stores/redis.py:33:from pydantic.typing import Literal
./python/feast/infra/online_stores/snowflake.py:8:from pydantic import Field, StrictStr
./python/feast/infra/online_stores/snowflake.py:9:from pydantic.schema import Literal
./python/feast/infra/feature_servers/base_config.py:3:from pydantic import StrictBool, StrictInt
./python/feast/infra/feature_servers/aws_lambda/config.py:1:from pydantic import StrictBool, StrictStr
./python/feast/infra/feature_servers/aws_lambda/config.py:2:from pydantic.typing import Literal
./python/feast/infra/feature_servers/local_process/config.py:1:from pydantic.typing import Literal
./python/feast/infra/feature_servers/gcp_cloudrun/config.py:1:from pydantic import StrictBool
./python/feast/infra/feature_servers/gcp_cloudrun/config.py:2:from pydantic.typing import Literal
./python/feast/repo_config.py:8:from pydantic import (
./python/feast/repo_config.py:17:from pydantic.error_wrappers import ErrorWrapper
./python/feast/repo_config.py:18:from pydantic.typing import Dict, Optional
./python/feast/repo_config.py:313:        # considered tech debt until we can implement 

pydantic/pydantic#619 or a more

[shuchu]

A single PR to modify all these files (and with proper testing) seems a little bit scary to me. Any suggestions?

One possible suggestion from Pydantic's doc (https://docs.pydantic.dev/latest/migration/) is:

1. upgrade the pydantic version from v1 to v2.
2. use a different import to keep using variables from v1, for example:
   from pydantic.v1 import BaseModel
3. incrementally upgrade the code base and projects.

[mjurkus]

The migration mostly relates to imports and validation function annotations so it's not that scary.
What is more worrying is that it also requires updating great-expectations to 0.18.1 and fastapi to 0.103.0 .

[mjurkus]

To start the migration, this needs to be fixed and released pydantic/pydantic#7873

[shuchu]

I'll keep an eye on the Pydantic's bug you mentioned. Once they fixed it, maybe we can discuss the upgrading of Pydantic from v1 to v2. At the same time, let me check the "great-expectations" and "fastapi" upgrades.

1. fastapi v.0.104.1 requires pydantic >= 2.0.0
2. great-expectations v0.18.1 requires pydantic>=1.9.2

[shuchu]

I'm going to work on this issue.

[TomSteenbergen]

Is there any update on Pydantic 2.0 support?

[shuchu]

sorry for the late. I'm still working on it. please wait for few days :)

[shuchu]

Just a progress update:
1, (solved) I have a branch in my own account with Pydantic version equal to the latest one (v2.6). Right now, only one unit test failure left.

= short test summary info =
FAILED sdk/python/tests/unit/cli/test_cli.py::test_3rd_party_registry_store_with_fs_yaml_override_by_env_var - AssertionError: Expected <1> to be equal to <0>, but was not.
= 1 failed, 240 passed, 25 warnings in 51.35s =

2, (solved) There is a "mypy" issue here as I also need to update the version to "mypy~=1.1.1" (as required by Pydantic). it will create 120+ mypy lint check errors that need to be fixed. (It will be a tedious grammar change but I just need more time :))

3, Hope there will be not too many failures while doing the integration test. //pray

[shuchu]

My friends, the PR is here: #3942

I humbly ask you guys to review my PR if you have time, since this PR changes many files from the codebase.

[shuchu]

Folks, the Pydantic V2 is in the master branch. I configured the setup.py as "pydantic>=2.0.0". Especially want to thank Harry @sudohainguyen for spending time to review it!

Enjoy!