Fides Pydantic V2 Upgrade #5020
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r occurred while loading CORS domains: Cannot add middleware after an application has started" - Pin fideslang, bump Pydantic to 2.7.1, bump FastAPI to 0.89.1, and temporarily add bump-pydantic to add in upgrade - This caused us to have to update typing_extensions and fastapi-pagination as well. Also necessitated installing new requirements pydnatic-settings. - FidesKey.validate has been replaced with validate_fides_key for validating FidesKeys outside of a model - conlist max_items/min_items -> max_length/min_length - root_validators -> model_validators, largely using mode=before as classmethods, some as after validators - Need to define __pydantic_init_subclass__ instead of init_subclass - Constrainedstr -> constr - EmailStr takes no arguments - Coercing string ports into integer ports since Pydantic longer does this for us and I don't want this to be a breaking change - Advanced settings was being overridden by a non-annotated attribute - smart_union supported by default now, var has been removed - ConnectorTempltae.validate_config has config param name that conflicts - SettingsSourceCallable is updated to PydanticBaseSettingsSource and has been moved to pydnatic-settings - Field validators should be accessing values through info.data - Deleted _FUNCS.clear under the assumption that allow_resuse behavior is no longer necessary - URLOrigin custom type can no longer be defined as a subclass. Redefine with URL Origin - PostgresDSR user argument -> username
…he application has already started. - Replace fast api's app.on_event(startup) with a lifespan that defines logic before the app starts up. - Instead, load cors middleware early, not using ConfigProxy. This is temporary, and I will address this more later. - This gets the server up and running and the shell is loaded.
…p-pydantic on the src directory - Update custom types from being defined as sub classes with __get_validators__ class methods to Annotated resources. - Replaced conlist(T, *args) with Annotated[List[T], Field(*args)]. - Add default of None to Optional fields to keep them Optional - Swap class Config for model_config - Swap vaidators for field validators (pre=True -> mode=before) - Field validators that need access to "values" should be refactored to be "before" validators accessing info.data
…, not an AnyUrl, with the trailing slash removed. Our CORS urls can't have trailing slashes.
…"before", so they are always SaaSRequests. - If I'm using the before model validator for SaaSRequests, sometimes "values" are a list, but other times they are a dict. This surfaced in the Salesforce config where read requests are a list of SaaS Requests.
…PrivacyRequestResponse. When creating a `PrivacyRequestCreate` object, custom_privacy_request_fields are now being coerced so the value is a CustomPrivacyRequestField. Will need a follow-up to ensure this still gets saved properly.
… are moving them from the meta object to top-level attributes for convenience. Converting this back into a literal to/from to prevent a ValidationError.
…rly compare with the collections coming out of the database.
…lack of type coercion. Rather than push this to customers updating existing rollbar config, updating internally.
…ds optional. - The validate_details_validator field_validator's second param is a ValidationInfo object not a dictionary. - Use the EdgeDirection value
…at's coming out of the database with the pydantic model. In Pydantic V2, models are no longer equal to the dicts containing their data.
…s() doesn't include extra fields so Identity.labeled_dict is throwing out custom identities.
…t converted into a private attribute which can't be accessed as intended in the validators.
…ting the schemas for the manual webhooks.
…ore validator - so I'm updating to use the validator in "after" mode.
…m fields in the SaaSSchemaFactory. - Also, need to use is_required() function on the FieldInfo object. - Update some error messages where the default text thrown by pydantic has changed slightly, but the intent is the same.
…e values. - Adjust the error message in a test - the specifics of the error thrown by pydantic have changed but the essence is the same.
# Conflicts: # src/fides/api/api/v1/endpoints/privacy_request_endpoints.py # src/fides/api/schemas/messaging/messaging.py # src/fides/api/schemas/privacy_request.py
…ng thrown by pydantic have changed but the essence is the same.
…Converters, FieldAddresses, and CollectionAddresses. These are not Pydantic types.
…ject fields using duck-typing behavior. Nested object fields were not being included with json() when serializing a collection. > "Duck-typing serialization is the behavior of serializing an object based on the fields present in the object itself, rather than the fields present in the schema of the object."
- Fides keys can no longer be integers - Models are no longer equal to the dicts containing their data - Pydantic error message has changed - Newly added connection type needs to have _required_components be a ClassVar
- Saas secrets not making it into the database, being thrown out in serialization because they don't match fields on the generic SaaS Schema. Using duck typing serialization by adding serialize_as_any=True - Allow extra fields to come through on the SaaS schema so the secrets can come in through - Don't save secrets to the database that were not officially set to match original behavior with exclude_unset=True - New ValiationError has the input_value which is a SaaS Schema which isn't json serializable. Wrapping it in a jsonable encoder - Use json_schema_extra to add extra data to the json schema since arbitrary arguments are no longer supported - parse_obj -> model_validate - Model validators with mode="after" are a lot easy to work with - before validators - they are passed raw input, so it might be. adict, it could be an instance of the model itself - - Fix incorrect advanced settings type I introduced which is causing cookie_ids to get dropped - Behavior change: fields passed in for secrets that are not defined on the schema are no longer saved to the db - Adjust tests that were looking at error messages thrown by pydantic where the generated error messages have changed
…s arbitrary keyword arguments. This was preventing secrets from being masked.
…"null" as acceptable options to many of the fields generated from the pydantic schemas.
# Conflicts: # CHANGELOG.md
clients/admin-ui/src/types/api/models/ConnectionConfigurationResponse.ts
Show resolved
Hide resolved
pattisdr
removed
the
do not merge
|
Test summaryRun details
This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Cloud |
Roger-Ethyca
pushed a commit
that referenced
this pull request
Aug 20, 2024
Pydantic v1 -> Pydantic v2 upgrade Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: eastandwestwind <[email protected]>
|
@pattisdr bravo! |
|
Thank you @ThomasLaPiana!! 🙌 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #PROD-2263
❗ Dependent on ethyca/fideslang#11
ℹ️ Replaces #4442
Description Of Changes
Pydantic V2/Fast API Upgrade
Code Changes
model_config=ConfigDict(...)dict()->model_dump(), often with mode="json" since model_dump() is not always serializable as json, depending on model contentsparse_obj()->model_validate()from_orm()->model_validate()root_validators->model_validators. mode="after" is often the better way to go because before validators can be called in surprising places, where sometimes the incoming data is a dict, or other times, the schema itself__init_subclass__->__pydantic_init_subclass____fields__->model_fields()customise_sources->customise_setting_sourcesupdate_forward_refs->model_rebuildvalidate_default=Trueon the field directlyConstrainedStrnow need to be defined using StringConstraintsNoneadded to keep Optional fields optional@app.on_event("startup")is deprecated, using Lifespan insteadNoValidationSchemaso it can still be used as a Pydantic schema that skips validation allowing a schema to be used for docs-onlyjson_schema_extraSteps to Confirm
This is something that really needs a full regression. These are features that needed more attention:
Pre-Merge Checklist
CHANGELOG.mdmaindowngrade()migration is correct and works