Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Synapse does not take advantage of Pydantic 2.0 #15858

Open
c7hm4r opened this issue Jul 1, 2023 · 12 comments · May be fixed by #15979
Open

Synapse does not take advantage of Pydantic 2.0 #15858

c7hm4r opened this issue Jul 1, 2023 · 12 comments · May be fixed by #15979
Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.

Comments

@c7hm4r
Copy link

c7hm4r commented Jul 1, 2023

Description

After upgrading the PIP packages, Synapse failed to launch.

Steps to reproduce

Homeserver

another homeserver

Synapse Version

1.87.0rc1

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Single process

Platform

Raspberry Pi OS (Debian)

Configuration

No response

Relevant log output

/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/_internal/_config.py:257: UserWarning: Valid config keys have changed in V2:                                                            
* 'allow_mutation' has been removed                                                
  warnings.warn(message, UserWarning)                                              
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main             
    return _run_code(code, main_globals, None,                                     
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                        
    exec(code, run_globals)                                                        
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py", line 37, in <module>                                                                                         
    from synapse.app import _base                                                  
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/_base.py", line 68, in <module>                                                                                              
    from synapse.handlers.auth import load_legacy_password_auth_providers          
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/auth.py", line 56, in <module>                                                                                          
    from synapse.api.ratelimiting import Ratelimiter                              
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/api/ratelimiting.py", line 21, in <module>                                                                                       
    from synapse.storage.databases.main import DataStore                           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/__init__.py", line 34, in <module>                                                                                       
    from synapse.storage.databases import Databases                                
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 20, in <module>                                                                             
    from synapse.storage.databases.main.events import PersistEventsStore           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/__init__.py", line 27, in <module>                                                                        
    from synapse.storage.databases.main.stats import UserSortOrder                 
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 42, in <module>                                                                           
    from synapse.storage.databases.main.events_worker import InvalidEventError     
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_worker.py", line 62, in <module>
    from synapse.replication.tcp.streams import BackfillStream, UnPartialStatedEventStream
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/__init__.py", line 27, in <module>
    from synapse.replication.tcp.streams._base import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/_base.py", line 32, in <module>
    from synapse.replication.http.streams import ReplicationGetStreamUpdates
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/__init__.py", line 18, in <module>
    from synapse.replication.http import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/account_data.py", line 21, in <module>
    from synapse.replication.http._base import ReplicationEndpoint
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/_base.py", line 31, in <module>
    from synapse.http.servlet import parse_json_object_from_request
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/servlet.py", line 32, in <module>
    from pydantic import BaseModel, MissingError, PydanticValueError, ValidationError
ImportError: cannot import name 'MissingError' from 'pydantic' (/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/__init__.py)
systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

Anything else that would be useful to know?

pydantic 2.0 was released yesterday

@c7hm4r
Copy link
Author

c7hm4r commented Jul 1, 2023

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

@DMRobertson
Copy link
Contributor

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

Suggest we specify this in pyproject.toml, or change

synapse/pyproject.toml

Lines 209 to 210 in 07fd6d8

# This is the most recent version of Pydantic with available on common distros.
pydantic = ">=1.7.4"

to ^=1.7.4.

Not sure why I didn't do this when starting to use Pydantic. I think because generally speaking it makes packagers' lives easier if we don't impose upper bounds?

@DMRobertson DMRobertson added X-Release-Blocker Must be resolved before making a release X-Regression Something broke which worked on a previous release labels Jul 2, 2023
@PaarthShah
Copy link
Contributor

PaarthShah commented Jul 2, 2023

Just encountered this, and in other projects as well on Friday morning. I agree with pinning to <2.0.0 as a workaround; hoping to make a PR for this shortly.

@PaarthShah
Copy link
Contributor

PaarthShah commented Jul 2, 2023

Just created: #15862

arkamar added a commit to arkamar/gentoo that referenced this issue Jul 5, 2023
Pillow 10 removes some deprecated constatns that are used in synapse,
the issue is already fixed in upstream, see [1,2].
Moreover, synapse is incompatible with pydantic 2.0 [3].

[1] matrix-org/synapse#15873
[2] matrix-org/synapse#15876
[3] matrix-org/synapse#15858

Closes: https://bugs.gentoo.org/909644
Signed-off-by: Petr Vaněk <[email protected]>
@H-Shay
Copy link
Contributor

H-Shay commented Jul 5, 2023

Closing this as it looks like it is resolved by #15862

@H-Shay H-Shay closed this as completed Jul 5, 2023
gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Jul 12, 2023
Pillow 10 removes some deprecated constatns that are used in synapse,
the issue is already fixed in upstream, see [1,2].
Moreover, synapse is incompatible with pydantic 2.0 [3].

[1] matrix-org/synapse#15873
[2] matrix-org/synapse#15876
[3] matrix-org/synapse#15858

Closes: https://bugs.gentoo.org/909644
Signed-off-by: Petr Vaněk <[email protected]>
Signed-off-by: Sam James <[email protected]>
@dvzrv
Copy link

dvzrv commented Jul 23, 2023

Hi! We're currently blocked by matrix-synapse in our pydantic 2 rebuild for Arch Linux. Please reopen this ticket so that discussion around how/when/if to switch to pydantic2 can happen.

The change in #15862 is merely a workaround and not the solution to this ticket (which would be "be compatible with pydantic >= 2").

Thanks! :)

dvzrv added a commit to dvzrv/synapse that referenced this issue Jul 23, 2023
Adapt codebase to use pydantic >= 2 models and functionalities.

Remove unneeded checks from `scripts-dev/check_pydantic_models.py`,
since pydantic can now be used in a strict mode which will prevent the
type coercion:
https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode

Closes matrix-org#15858
dvzrv added a commit to dvzrv/synapse that referenced this issue Jul 23, 2023
Adapt codebase to use pydantic >= 2 models and functionalities.

Remove unneeded checks from `scripts-dev/check_pydantic_models.py`,
since pydantic can now be used in a strict mode which will prevent the
type coercion:
https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode

Closes matrix-org#15858

Signed-off-by: David Runge <[email protected]>
@dvzrv dvzrv linked a pull request Jul 23, 2023 that will close this issue
4 tasks
dvzrv added a commit to dvzrv/synapse that referenced this issue Jul 24, 2023
Adapt codebase to use pydantic >= 2 models and functionalities.

Remove unneeded checks from `scripts-dev/check_pydantic_models.py`,
since pydantic can now be used in a strict mode which will prevent the
type coercion:
https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode

Closes matrix-org#15858

Signed-off-by: David Runge <[email protected]>
@clokep
Copy link
Member

clokep commented Jul 24, 2023

Re-opening per the above -- pinning to <= 2.0 isn't supporting Pydantic 2.0.

@clokep clokep reopened this Jul 24, 2023
dvzrv added a commit to dvzrv/synapse that referenced this issue Jul 24, 2023
Adapt codebase to use pydantic >= 2 models and functionalities.

Remove unneeded checks from `scripts-dev/check_pydantic_models.py`,
since pydantic can now be used in a strict mode which will prevent the
type coercion:
https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode

Closes matrix-org#15858

Signed-off-by: David Runge <[email protected]>
dvzrv added a commit to dvzrv/synapse that referenced this issue Jul 24, 2023
Adapt codebase to use pydantic >= 2 models and functionalities.

Remove unneeded checks from `scripts-dev/check_pydantic_models.py`,
since pydantic can now be used in a strict mode which will prevent the
type coercion:
https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode

Closes matrix-org#15858

Signed-off-by: David Runge <[email protected]>
@MatMaul MatMaul removed X-Release-Blocker Must be resolved before making a release X-Regression Something broke which worked on a previous release labels Jul 25, 2023
@clokep clokep added S-Minor Blocks non-critical functionality, workarounds exist. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. O-Occasional Affects or can be seen by some users regularly or most users rarely labels Jul 25, 2023
@reivilibre
Copy link
Contributor

Pydantic V2 includes V1 built-in to help migration.

We could use the following all over the place as a transitional measure.

try:
    from pydantic import v1 as pydantic
except ImportError:
    import pydantic

Given that V1 and V2 use the same Python package name (⇒ you can't have them both installed at the same time), we sadly can expect some trouble with distro packages that would make switching to using V2 proper difficult for some time.

@dvzrv
Copy link

dvzrv commented Aug 13, 2023

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

I can earliest look at my open PR again after 2023-08-21. If someone manages to fix this earlier and create a new release, that would be much appreciated, as synapse is currently the only package blocking us from moving to pydantic v2 on Arch Linux: https://archlinux.org/todo/python-pydantic-2/

yingziwu added a commit to yingziwu/synapse that referenced this issue Aug 21, 2023
Please note that this will be the last release of Synapse that is compatible with
Python 3.7 and earlier.
This is due to Python 3.7 now having reached End of Life; see our [deprecation policy](https://matrix-org.github.io/synapse/v1.87/deprecation_policy.html)
for more details.

- Pin `pydantic` to `^1.7.4` to avoid backwards-incompatible API changes from the 2.0.0 release.
  Resolves matrix-org#15858.
  Contributed by @PaarthShah. ([\matrix-org#15862](matrix-org#15862))

- Split out 2022 changes from the changelog so the rendered version in GitHub doesn't timeout as much. ([\matrix-org#15846](matrix-org#15846))

- Improve `/messages` response time by avoiding backfill when we already have messages to return. ([\matrix-org#15737](matrix-org#15737))
- Add spam checker module API for logins. ([\matrix-org#15838](matrix-org#15838))

- Fix a long-standing bug where media files were served in an unsafe manner. Contributed by @joshqou. ([\matrix-org#15680](matrix-org#15680))
- Avoid invalidating a cache that was just prefilled. ([\matrix-org#15758](matrix-org#15758))
- Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15770](matrix-org#15770))
- Fix joining rooms through aliases where the alias server isn't a real homeserver. Contributed by @tulir @ Beeper. ([\matrix-org#15776](matrix-org#15776))
- Fix a bug in push rules handling leading to an invalid (per spec) `is_user_mention` rule sent to clients. Also fix wrong rule names for `is_user_mention` and `is_room_mention`. ([\matrix-org#15781](matrix-org#15781))
- Fix a bug introduced in 1.57.0 where the wrong table would be locked on updating database rows when using SQLite as the database backend. ([\matrix-org#15788](matrix-org#15788))
- Fix Sytest environmental variable evaluation in CI. ([\matrix-org#15804](matrix-org#15804))
- Fix forgotten rooms missing from initial sync after rejoining them. Contributed by Nico from Famedly. ([\matrix-org#15815](matrix-org#15815))
- Fix sqlite `user_filters` upgrade introduced in v1.86.0. ([\matrix-org#15817](matrix-org#15817))

- Document `looping_call()` functionality that will wait for the given function to finish before scheduling another. ([\matrix-org#15772](matrix-org#15772))
- Fix a typo in the [Admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html). ([\matrix-org#15805](matrix-org#15805))
- Fix typo in MSC number in faster remote room join architecture doc. ([\matrix-org#15812](matrix-org#15812))

- Remove experimental [MSC2716](matrix-org/matrix-spec-proposals#2716) implementation to incrementally import history into existing rooms. ([\matrix-org#15748](matrix-org#15748))

- Replace `EventContext` fields `prev_group` and `delta_ids` with field `state_group_deltas`. ([\matrix-org#15233](matrix-org#15233))
- Regularly try to send transactions to other servers after they failed instead of waiting for a new event to be available before trying. ([\matrix-org#15743](matrix-org#15743))
- Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15755](matrix-org#15755))
- Allow for the configuration of max request retries and min/max retry delays in the matrix federation client. ([\matrix-org#15783](matrix-org#15783))
- Switch from `matrix://` to `matrix-federation://` scheme for internal Synapse routing of outbound federation traffic. ([\matrix-org#15806](matrix-org#15806))
- Fix harmless exceptions being printed when running the port DB script. ([\matrix-org#15814](matrix-org#15814))

* Bump attrs from 22.2.0 to 23.1.0. ([\matrix-org#15801](matrix-org#15801))
* Bump cryptography from 40.0.2 to 41.0.1. ([\matrix-org#15800](matrix-org#15800))
* Bump ijson from 3.2.0.post0 to 3.2.1. ([\matrix-org#15802](matrix-org#15802))
* Bump phonenumbers from 8.13.13 to 8.13.14. ([\matrix-org#15798](matrix-org#15798))
* Bump ruff from 0.0.265 to 0.0.272. ([\matrix-org#15799](matrix-org#15799))
* Bump ruff from 0.0.272 to 0.0.275. ([\matrix-org#15833](matrix-org#15833))
* Bump serde_json from 1.0.96 to 1.0.97. ([\matrix-org#15797](matrix-org#15797))
* Bump serde_json from 1.0.97 to 1.0.99. ([\matrix-org#15832](matrix-org#15832))
* Bump towncrier from 22.12.0 to 23.6.0. ([\matrix-org#15831](matrix-org#15831))
* Bump types-opentracing from 2.4.10.4 to 2.4.10.5. ([\matrix-org#15830](matrix-org#15830))
* Bump types-setuptools from 67.8.0.0 to 68.0.0.0. ([\matrix-org#15835](matrix-org#15835))
dvzrv added a commit to dvzrv/synapse that referenced this issue Sep 10, 2023
Adapt codebase to use pydantic >= 2 models and functionalities.

Remove unneeded checks from `scripts-dev/check_pydantic_models.py`,
since pydantic can now be used in a strict mode which will prevent the
type coercion:
https://docs.pydantic.dev/2.0/usage/strict_mode/#type-coercions-in-strict-mode

Closes matrix-org#15858

Signed-off-by: David Runge <[email protected]>
@gotmax23
Copy link
Contributor

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

The pydantic.v1 module is simply a copy of the old pydantic v1 code, so there should not be breakages there.

@gotmax23
Copy link
Contributor

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

@clokep clokep changed the title Synapse is incompatible with Pydantic 2.0 Synapse does not take advantage of Pydantic 2.0 Oct 2, 2023
@clokep
Copy link
Member

clokep commented Oct 2, 2023

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

I re-titled this since Synapse is now compatible, but not taking advantage of pydantic v2.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants