Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed presence results not returning offline users on initial sync #17231

Merged
merged 2 commits into from
Jul 23, 2024
Merged

Fixed presence results not returning offline users on initial sync #17231

merged 2 commits into from
Jul 23, 2024

Conversation

Michael-Hollister
Copy link
Contributor

@Michael-Hollister Michael-Hollister commented May 24, 2024 •
edited
Loading

This is to address an issue in which m.presence results on initial sync are not returning entries of users who are currently offline. I believe this was intended behavior as stated in #1535, but since this was 8+ years ago, I wanted to check if that is still the case. If this is still desired behavior, then feel free to close the PR.

However, I think this change is useful for applications that use the presence system for tracking user profile information/updates (e.g. #16992 or for profile status messages).

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct
    (run the linters)

@Michael-Hollister Michael-Hollister requested a review from a team as a code owner May 24, 2024 16:58
@reivilibre
Copy link
Contributor

Just to clarify, after this change, all users that you share a room with, even if offline, will be added to the initial sync response?
I think this will be quite expensive and therefore doesn't sound that desirable.

It sounds like you are actually more interested in the status messages of these users; is it perhaps wiser to only include offline users if they actually have a status message?

@Michael-Hollister
Copy link
Contributor Author

Just to clarify, after this change, all users that you share a room with, even if offline, will be added to the initial sync response?

Yes that is correct.

I think this will be quite expensive and therefore doesn't sound that desirable.

Actually, I believe there should be no performance difference with my change since we already fetch presence results from all users you share a room with and then throw away the results:

  1. Synapse calls _generate_sync_entry_for_presence if presence is enabled (sync.py, 2259)
  2. We then call presence_source.get_new_events. Note that since this is for initial sync, from_key=None (sync.py, 2287-2291)
  3. In get_new_events (presence.py, 1766), There are only 3 areas where include_offline is checked:
    a. First one on line 1808, this code block is triggered for non-initial sync cases
    b. On line 1842 (call to self._filter_all_presence_updates_for_user), this is triggered only when the user registers custom presence routers that specifies all users should be tracked. Users by default don't use such router.
    c. Finally on line 1911, we filter offline results if include_offline=False (which is what Syanpse did before this change).

Happy to be corrected if I'm missing something, but my understanding currently is that there should be no performance impact with the change.

It sounds like you are actually more interested in the status messages of these users; is it perhaps wiser to only include offline users if they actually have a status message?

Not quite for status messages (although its helpful for this if I wanted to track that) but for displayname and avatar_url fields. I'm using the presence system for tracking user profile updates instead of using room state events. These fields are not currently in Synapse although the spec states these should be included in m.presence events (https://spec.matrix.org/v1.10/client-server-api/#mpresence). I do have PR open for that however should the team wish to merge it: #16992.

If I need to fix errors in the sytest test suite due to this change, let me know (since I'm not sure when the CI would pickup the docker image with the fixed test cases).

@reivilibre
Copy link
Contributor

Happy to be corrected if I'm missing something, but my understanding currently is that there should be no performance impact with the change.

The main concern I have, is that this will bloat the response of /sync, particularly initial /sync, which can already notoriously be very large.
This then takes finite time to arrive at the client and then clients have to then process this response — not saying we should make excuses for poor client performance but it does concern me a bit.

In effect, the whole idea of this also kind of contradicts the idea of lazy-loading room memberships, if you can appreciate what I mean.

@Michael-Hollister
Copy link
Contributor Author

Michael-Hollister commented May 30, 2024 •
edited
Loading

The main concern I have, is that this will bloat the response of /sync, particularly initial /sync, which can already notoriously be very large. This then takes finite time to arrive at the client and then clients have to then process this response — not saying we should make excuses for poor client performance but it does concern me a bit.

Fair point regarding client-side performance. For response bloat, we already include offline presence status for /sync with a since token. The additional response bloat would only affect initial /sync.

In effect, the whole idea of this also kind of contradicts the idea of lazy-loading room memberships, if you can appreciate what I mean.

Makes sense using lazy-loading when clients are using room memberships primarily for tracking profile updates. For my client application's use-case though, we have disabled room member changes on profile updates (MSC4069), hence the use of the presence system for profile update tracking.

To address client-side performance concerns on initial /sync, do you think making this behavior of including offline presence results a server-side configurable option or adding a new sync filter field would be of interest?

@reivilibre
Copy link
Contributor

reivilibre commented Jul 8, 2024 •
edited
Loading

To address client-side performance concerns on initial /sync, do you think making this behavior of including offline presence results a server-side configurable option or adding a new sync filter field would be of interest?

Hi, sorry for letting this sit around so long. I've talked to the others; we've said that the performance impact of having this enabled by default worries us too much. What you want to do doesn't sound unreasonable but sounds like, to do it properly, would need to 'fix the profile system', which is a bigger task than what we can take on right now. As a pragmatic solution given the small size of the change here, we'd be happy to accept a server-side configurable option for this.

@anoadragon453 anoadragon453 removed the request for review from a team July 9, 2024 12:45
@Michael-Hollister
Copy link
Contributor Author

No problem, and thanks for getting back. I've updated the PR to make the change enabled via a config option.

@reivilibre reivilibre requested a review from a team July 22, 2024 14:55
reivilibre
reivilibre approved these changes Jul 23, 2024
View reviewed changes
Copy link
Contributor

@reivilibre reivilibre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for changing that, sorry for this getting so delayed!

@github-actions github-actions bot deployed to PR Documentation Preview July 23, 2024 09:33 Active
@reivilibre reivilibre enabled auto-merge (squash) July 23, 2024 09:35
@reivilibre reivilibre merged commit a9ee832 into element-hq:develop Jul 23, 2024
41 checks passed
@Michael-Hollister Michael-Hollister mentioned this pull request Jul 29, 2024
Merged
yingziwu added a commit to yingziwu/synapse that referenced this pull request Aug 1, 2024
@yingziwu
Merge tag 'v1.112.0' into develop
6cd05d2 
This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](GHSA-c8m8-j448-xjx7).

Note that this security fix is also available as **Synapse 1.111.1**, which does not include the rest of the changes in Synapse 1.112.0.

This issue means that, if multiple HTTP requests are pipelined in the same TCP connection, Synapse can send responses to the wrong HTTP request.
If a reverse proxy was configured to use HTTP pipelining, this could result in responses being sent to the wrong user, severely harming confidentiality.

With that said, despite being a high severity issue, **we consider it unlikely that Synapse installations will be affected**.
The use of HTTP pipelining in this fashion would cause worse performance for clients (request-response latencies would be increased as users' responses would be artificially blocked behind other users' slow requests). Further, Nginx and Haproxy, two common reverse proxies, do not appear to support configuring their upstreams to use HTTP pipelining and thus would not be affected. For both of these reasons, we consider it unlikely that a Synapse deployment would be set up in such a configuration.

Despite that, we cannot rule out that some installations may exist with this unusual setup and so we are releasing this security update today.

**pip users:** Note that by default, upgrading Synapse using pip will not automatically upgrade Twisted. **Please manually install the new version of Twisted** using `pip install Twisted==24.7.0rc1`. Note also that even the `--upgrade-strategy=eager` flag to `pip install -U matrix-synapse` will not upgrade Twisted to a patched version because it is only a release candidate at this time.

- Upgrade locked dependency on Twisted to 24.7.0rc1. ([\#17502](element-hq/synapse#17502))

Please note that this release candidate does not include the security dependency update
included in version 1.111.1 as this version was released before 1.111.1.
The same security fix can be found in the full release of 1.112.0.

- Add to-device extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17416](element-hq/synapse#17416))
- Populate `name`/`avatar` fields in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17418](element-hq/synapse#17418))
- Populate `heroes` and room summary fields (`joined_count`, `invited_count`) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17419](element-hq/synapse#17419))
- Populate `is_dm` room field in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17429](element-hq/synapse#17429))
- Add room subscriptions to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17432](element-hq/synapse#17432))
- Prepare for authenticated media freeze. ([\#17433](element-hq/synapse#17433))
- Add E2EE extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17454](element-hq/synapse#17454))

- Add configurable option to always include offline users in presence sync results. Contributed by @Michael-Hollister. ([\#17231](element-hq/synapse#17231))
- Fix bug in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint when using room type filters and the user has one or more remote invites. ([\#17434](element-hq/synapse#17434))
- Order `heroes` by `stream_ordering` as the Matrix specification states (applies to `/sync`). ([\#17435](element-hq/synapse#17435))
- Fix rare bug where `/sync` would break for a user when using workers with multiple stream writers. ([\#17438](element-hq/synapse#17438))

- Update the readme image to have a white background, so that it is readable in dark mode. ([\#17387](element-hq/synapse#17387))
- Add Red Hat Enterprise Linux and Rocky Linux 8 and 9 installation instructions. ([\#17423](element-hq/synapse#17423))
- Improve documentation for the [`default_power_level_content_override`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#default_power_level_content_override) config option. ([\#17451](element-hq/synapse#17451))

- Make sure we always use the right logic for enabling the media repo. ([\#17424](element-hq/synapse#17424))
- Fix argument documentation for method `RateLimiter.record_action`. ([\#17426](element-hq/synapse#17426))
- Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. ([\#17428](element-hq/synapse#17428))
- Limit concurrent remote downloads to 6 per IP address, and decrement remote downloads without a content-length from the ratelimiter after the download is complete. ([\#17439](element-hq/synapse#17439))
- Remove unnecessary call to resume producing in fake channel. ([\#17449](element-hq/synapse#17449))
- Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to bump room when it is created. ([\#17453](element-hq/synapse#17453))
- Speed up generating sliding sync responses. ([\#17458](element-hq/synapse#17458))
- Add cache to `get_rooms_for_local_user_where_membership_is` to speed up sliding sync. ([\#17460](element-hq/synapse#17460))
- Speed up fetching room keys from backup. ([\#17461](element-hq/synapse#17461))
- Speed up sorting of the room list in sliding sync. ([\#17468](element-hq/synapse#17468))
- Implement handling of `$ME` as a state key in sliding sync. ([\#17469](element-hq/synapse#17469))

* Bump bytes from 1.6.0 to 1.6.1. ([\#17441](element-hq/synapse#17441))
* Bump hiredis from 2.3.2 to 3.0.0. ([\#17464](element-hq/synapse#17464))
* Bump jsonschema from 4.22.0 to 4.23.0. ([\#17444](element-hq/synapse#17444))
* Bump matrix-org/done-action from 2 to 3. ([\#17440](element-hq/synapse#17440))
* Bump mypy from 1.9.0 to 1.10.1. ([\#17445](element-hq/synapse#17445))
* Bump pyopenssl from 24.1.0 to 24.2.1. ([\#17465](element-hq/synapse#17465))
* Bump ruff from 0.5.0 to 0.5.4. ([\#17466](element-hq/synapse#17466))
* Bump sentry-sdk from 2.6.0 to 2.8.0. ([\#17456](element-hq/synapse#17456))
* Bump sentry-sdk from 2.8.0 to 2.10.0. ([\#17467](element-hq/synapse#17467))
* Bump setuptools from 67.6.0 to 70.0.0. ([\#17448](element-hq/synapse#17448))
* Bump twine from 5.1.0 to 5.1.1. ([\#17443](element-hq/synapse#17443))
* Bump types-jsonschema from 4.22.0.20240610 to 4.23.0.20240712. ([\#17446](element-hq/synapse#17446))
* Bump ulid from 1.1.2 to 1.1.3. ([\#17442](element-hq/synapse#17442))
* Bump zipp from 3.15.0 to 3.19.1. ([\#17427](element-hq/synapse#17427))
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Aug 12, 2024
@Midar
Update chat/matrix-synapse to 1.112.0
399488c 
# Synapse 1.112.0 (2024-07-30)

This security release is to update our locked dependency on Twisted to 24.7.0rc1, which includes a security fix for [CVE-2024-41671 / GHSA-c8m8-j448-xjx7: Disordered HTTP pipeline response in twisted.web, again](GHSA-c8m8-j448-xjx7).

Note that this security fix is also available as **Synapse 1.111.1**, which does not include the rest of the changes in Synapse 1.112.0.

This issue means that, if multiple HTTP requests are pipelined in the same TCP connection, Synapse can send responses to the wrong HTTP request.
If a reverse proxy was configured to use HTTP pipelining, this could result in responses being sent to the wrong user, severely harming confidentiality.

With that said, despite being a high severity issue, **we consider it unlikely that Synapse installations will be affected**.
The use of HTTP pipelining in this fashion would cause worse performance for clients (request-response latencies would be increased as users' responses would be artificially blocked behind other users' slow requests). Further, Nginx and Haproxy, two common reverse proxies, do not appear to support configuring their upstreams to use HTTP pipelining and thus would not be affected. For both of these reasons, we consider it unlikely that a Synapse deployment would be set up in such a configuration.

Despite that, we cannot rule out that some installations may exist with this unusual setup and so we are releasing this security update today.

**pip users:** Note that by default, upgrading Synapse using pip will not automatically upgrade Twisted. **Please manually install the new version of Twisted** using `pip install Twisted==24.7.0rc1`. Note also that even the `--upgrade-strategy=eager` flag to `pip install -U matrix-synapse` will not upgrade Twisted to a patched version because it is only a release candidate at this time.

### Internal Changes

- Upgrade locked dependency on Twisted to 24.7.0rc1. ([\#17502](element-hq/synapse#17502))


# Synapse 1.112.0rc1 (2024-07-23)

Please note that this release candidate does not include the security dependency update
included in version 1.111.1 as this version was released before 1.111.1.
The same security fix can be found in the full release of 1.112.0.

### Features

- Add to-device extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17416](element-hq/synapse#17416))
- Populate `name`/`avatar` fields in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17418](element-hq/synapse#17418))
- Populate `heroes` and room summary fields (`joined_count`, `invited_count`) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17419](element-hq/synapse#17419))
- Populate `is_dm` room field in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17429](element-hq/synapse#17429))
- Add room subscriptions to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17432](element-hq/synapse#17432))
- Prepare for authenticated media freeze. ([\#17433](element-hq/synapse#17433))
- Add E2EE extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17454](element-hq/synapse#17454))

### Bugfixes

- Add configurable option to always include offline users in presence sync results. Contributed by @Michael-Hollister. ([\#17231](element-hq/synapse#17231))
- Fix bug in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint when using room type filters and the user has one or more remote invites. ([\#17434](element-hq/synapse#17434))
- Order `heroes` by `stream_ordering` as the Matrix specification states (applies to `/sync`). ([\#17435](element-hq/synapse#17435))
- Fix rare bug where `/sync` would break for a user when using workers with multiple stream writers. ([\#17438](element-hq/synapse#17438))

### Improved Documentation

- Update the readme image to have a white background, so that it is readable in dark mode. ([\#17387](element-hq/synapse#17387))
- Add Red Hat Enterprise Linux and Rocky Linux 8 and 9 installation instructions. ([\#17423](element-hq/synapse#17423))
- Improve documentation for the [`default_power_level_content_override`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#default_power_level_content_override) config option. ([\#17451](element-hq/synapse#17451))

### Internal Changes

- Make sure we always use the right logic for enabling the media repo. ([\#17424](element-hq/synapse#17424))
- Fix argument documentation for method `RateLimiter.record_action`. ([\#17426](element-hq/synapse#17426))
- Reduce volume of 'Waiting for current token' logs, which were introduced in v1.109.0. ([\#17428](element-hq/synapse#17428))
- Limit concurrent remote downloads to 6 per IP address, and decrement remote downloads without a content-length from the ratelimiter after the download is complete. ([\#17439](element-hq/synapse#17439))
- Remove unnecessary call to resume producing in fake channel. ([\#17449](element-hq/synapse#17449))
- Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to bump room when it is created. ([\#17453](element-hq/synapse#17453))
- Speed up generating sliding sync responses. ([\#17458](element-hq/synapse#17458))
- Add cache to `get_rooms_for_local_user_where_membership_is` to speed up sliding sync. ([\#17460](element-hq/synapse#17460))
- Speed up fetching room keys from backup. ([\#17461](element-hq/synapse#17461))
- Speed up sorting of the room list in sliding sync. ([\#17468](element-hq/synapse#17468))
- Implement handling of `$ME` as a state key in sliding sync. ([\#17469](element-hq/synapse#17469))



### Updates to locked dependencies

* Bump bytes from 1.6.0 to 1.6.1. ([\#17441](element-hq/synapse#17441))
* Bump hiredis from 2.3.2 to 3.0.0. ([\#17464](element-hq/synapse#17464))
* Bump jsonschema from 4.22.0 to 4.23.0. ([\#17444](element-hq/synapse#17444))
* Bump matrix-org/done-action from 2 to 3. ([\#17440](element-hq/synapse#17440))
* Bump mypy from 1.9.0 to 1.10.1. ([\#17445](element-hq/synapse#17445))
* Bump pyopenssl from 24.1.0 to 24.2.1. ([\#17465](element-hq/synapse#17465))
* Bump ruff from 0.5.0 to 0.5.4. ([\#17466](element-hq/synapse#17466))
* Bump sentry-sdk from 2.6.0 to 2.8.0. ([\#17456](element-hq/synapse#17456))
* Bump sentry-sdk from 2.8.0 to 2.10.0. ([\#17467](element-hq/synapse#17467))
* Bump setuptools from 67.6.0 to 70.0.0. ([\#17448](element-hq/synapse#17448))
* Bump twine from 5.1.0 to 5.1.1. ([\#17443](element-hq/synapse#17443))
* Bump types-jsonschema from 4.22.0.20240610 to 4.23.0.20240712. ([\#17446](element-hq/synapse#17446))
* Bump ulid from 1.1.2 to 1.1.3. ([\#17442](element-hq/synapse#17442))
* Bump zipp from 3.15.0 to 3.19.1. ([\#17427](element-hq/synapse#17427))

# Synapse 1.111.0 (2024-07-16)

No significant changes since 1.111.0rc2.




# Synapse 1.111.0rc2 (2024-07-10)

### Bugfixes

- Fix bug where using `synapse.app.media_repository` worker configuration would break the new media endpoints. ([\#17420](element-hq/synapse#17420))

### Improved Documentation

- Document the new federation media worker endpoints in the [upgrade notes](https://element-hq.github.io/synapse/v1.111/upgrade.html) and [worker docs](https://element-hq.github.io/synapse/v1.111/workers.html). ([\#17421](element-hq/synapse#17421))

### Internal Changes

- Route authenticated federation media requests to media repository workers in Complement tests. ([\#17422](element-hq/synapse#17422))




# Synapse 1.111.0rc1 (2024-07-09)

### Features

- Add `rooms` data to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17320](element-hq/synapse#17320))
- Add `room_types`/`not_room_types` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17337](element-hq/synapse#17337))
- Return "required state" in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17342](element-hq/synapse#17342))
- Support [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) by adding [`_matrix/client/v1/media/download`](https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediadownloadservernamemediaid) endpoint. ([\#17365](element-hq/synapse#17365))
- Support [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md)
  by adding [`_matrix/client/v1/media/thumbnail`](https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediathumbnailservernamemediaid), [`_matrix/federation/v1/media/thumbnail`](https://spec.matrix.org/v1.11/server-server-api/#get_matrixfederationv1mediathumbnailmediaid) endpoints and stabilizing the
  remaining [`_matrix/client/v1/media`](https://spec.matrix.org/v1.11/client-server-api/#get_matrixclientv1mediaconfig) endpoints. ([\#17388](element-hq/synapse#17388))
- Add `rooms.bump_stamp` for easier client-side sorting in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17395](element-hq/synapse#17395))
- Forget all of a user's rooms upon deactivation, preventing local room purges from being blocked on deactivated users. ([\#17400](element-hq/synapse#17400))
- Declare support for [Matrix 1.11](https://matrix.org/blog/2024/06/20/matrix-v1.11-release/). ([\#17403](element-hq/synapse#17403))
- [MSC3861](matrix-org/matrix-spec-proposals#3861): allow overriding the introspection endpoint. ([\#17406](element-hq/synapse#17406))

### Bugfixes

- Fix rare race which caused no new to-device messages to be received from remote server. ([\#17362](element-hq/synapse#17362))
- Fix bug in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint when using an old database. ([\#17398](element-hq/synapse#17398))

### Improved Documentation

- Clarify that `url_preview_url_blacklist` is a usability feature. ([\#17356](element-hq/synapse#17356))
- Fix broken links in README. ([\#17379](element-hq/synapse#17379))
- Clarify that changelog content *and file extension* need to match in order for entries to merge. ([\#17399](element-hq/synapse#17399))

### Internal Changes

- Make the release script create a release branch for Complement as well. ([\#17318](element-hq/synapse#17318))
- Fix uploading packages to PyPi. ([\#17363](element-hq/synapse#17363))
- Add CI check for the README. ([\#17367](element-hq/synapse#17367))
- Fix linting errors from new `ruff` version. ([\#17381](element-hq/synapse#17381), [\#17411](element-hq/synapse#17411))
- Fix building debian packages on non-clean checkouts. ([\#17390](element-hq/synapse#17390))
- Finish up work to allow per-user feature flags. ([\#17392](element-hq/synapse#17392), [\#17410](element-hq/synapse#17410))
- Allow enabling sliding sync per-user. ([\#17393](element-hq/synapse#17393))



### Updates to locked dependencies

* Bump certifi from 2023.7.22 to 2024.7.4. ([\#17404](element-hq/synapse#17404))
* Bump cryptography from 42.0.7 to 42.0.8. ([\#17382](element-hq/synapse#17382))
* Bump ijson from 3.2.3 to 3.3.0. ([\#17413](element-hq/synapse#17413))
* Bump log from 0.4.21 to 0.4.22. ([\#17384](element-hq/synapse#17384))
* Bump mypy-zope from 1.0.4 to 1.0.5. ([\#17414](element-hq/synapse#17414))
* Bump pillow from 10.3.0 to 10.4.0. ([\#17412](element-hq/synapse#17412))
* Bump pydantic from 2.7.1 to 2.8.2. ([\#17415](element-hq/synapse#17415))
* Bump ruff from 0.3.7 to 0.5.0. ([\#17381](element-hq/synapse#17381))
* Bump serde from 1.0.203 to 1.0.204. ([\#17409](element-hq/synapse#17409))
* Bump serde_json from 1.0.117 to 1.0.120. ([\#17385](element-hq/synapse#17385), [\#17408](element-hq/synapse#17408))
* Bump types-setuptools from 69.5.0.20240423 to 70.1.0.20240627. ([\#17380](element-hq/synapse#17380))

# Synapse 1.110.0 (2024-07-03)

No significant changes since 1.110.0rc3.




# Synapse 1.110.0rc3 (2024-07-02)

### Bugfixes

- Fix bug where `/sync` requests could get blocked indefinitely after an upgrade from Synapse versions before v1.109.0. ([\#17386](element-hq/synapse#17386), [\#17391](element-hq/synapse#17391))

### Internal Changes

- Limit size of presence EDUs to 50 entries. ([\#17371](element-hq/synapse#17371))
- Fix building debian package for debian sid. ([\#17389](element-hq/synapse#17389))




# Synapse 1.110.0rc2 (2024-06-26)

### Internal Changes

- Fix uploading packages to PyPi. ([\#17363](element-hq/synapse#17363))




# Synapse 1.110.0rc1 (2024-06-26)

### Features

- Add initial implementation of an experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17187](element-hq/synapse#17187))
- Add experimental support for [MSC3823](matrix-org/matrix-spec-proposals#3823) - Account suspension. ([\#17255](element-hq/synapse#17255))
- Improve ratelimiting in Synapse. ([\#17256](element-hq/synapse#17256))
- Add support for the unstable [MSC4151](matrix-org/matrix-spec-proposals#4151) report room API. ([\#17270](element-hq/synapse#17270), [\#17296](element-hq/synapse#17296))
- Filter for public and empty rooms added to Admin-API [List Room API](https://element-hq.github.io/synapse/latest/admin_api/rooms.html#list-room-api). ([\#17276](element-hq/synapse#17276))
- Add `is_dm` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17277](element-hq/synapse#17277))
- Add `is_encrypted` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17281](element-hq/synapse#17281))
- Include user membership in events served to clients, per [MSC4115](matrix-org/matrix-spec-proposals#4115). ([\#17282](element-hq/synapse#17282))
- Do not require user-interactive authentication for uploading cross-signing keys for the first time, per [MSC3967](matrix-org/matrix-spec-proposals#3967). ([\#17284](element-hq/synapse#17284))
- Add `stream_ordering` sort to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17293](element-hq/synapse#17293))
- `register_new_matrix_user` now supports a --password-file flag, which
  is useful for scripting. ([\#17294](element-hq/synapse#17294))
- `register_new_matrix_user` now supports a --exists-ok flag to allow registration of users that already exist in the database.
  This is useful for scripts that bootstrap user accounts with initial passwords. ([\#17304](element-hq/synapse#17304))
- Add support for via query parameter from [MSC4156](matrix-org/matrix-spec-proposals#4156). ([\#17322](element-hq/synapse#17322))
- Add `is_invite` filtering to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17335](element-hq/synapse#17335))
- Support [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) by adding a federation /download endpoint. ([\#17350](element-hq/synapse#17350))

### Bugfixes

- Fix searching for users with their exact localpart whose ID includes a hyphen. ([\#17254](element-hq/synapse#17254))
- Fix wrong retention policy being used when filtering events. ([\#17272](element-hq/synapse#17272))
- Fix bug where OTKs were not always included in `/sync` response when using workers. ([\#17275](element-hq/synapse#17275))
- Fix a long-standing bug where an invalid 'from' parameter to [`/notifications`](https://spec.matrix.org/v1.10/client-server-api/#get_matrixclientv3notifications) would result in an Internal Server Error. ([\#17283](element-hq/synapse#17283))
- Fix edge case in `/sync` returning the wrong the state when using sharded event persisters. ([\#17295](element-hq/synapse#17295))
- Add initial implementation of an experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17301](element-hq/synapse#17301))
- Fix email notification subject when invited to a space. ([\#17336](element-hq/synapse#17336))

### Improved Documentation

- Add missing quotes for example for `exclude_rooms_from_sync`. ([\#17308](element-hq/synapse#17308))
- Update header in the README to visually fix the the auto-generated table of contents. ([\#17329](element-hq/synapse#17329))
- Fix stale references to the Foundation's Security Disclosure Policy. ([\#17341](element-hq/synapse#17341))
- Add default values for `rc_invites.per_issuer` to docs. ([\#17347](element-hq/synapse#17347))
- Fix an error in the docs for `search_all_users` parameter under `user_directory`. ([\#17348](element-hq/synapse#17348))

### Internal Changes

- Remove unused `expire_access_token` option in the Synapse Docker config file. Contributed by @AaronDewes. ([\#17198](element-hq/synapse#17198))
- Use fully-qualified `PersistedEventPosition` when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation. ([\#17265](element-hq/synapse#17265))
- Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. ([\#17266](element-hq/synapse#17266))
- Handle OTK uploads off master. ([\#17271](element-hq/synapse#17271))
- Don't try and resync devices for remote users whose servers are marked as down. ([\#17273](element-hq/synapse#17273))
- Re-organize Pydantic models and types used in handlers. ([\#17279](element-hq/synapse#17279))
- Expose the worker instance that persisted the event on `event.internal_metadata.instance_name`. ([\#17300](element-hq/synapse#17300))
- Update the README with Element branding, improve headers and fix the #synapse:matrix.org support room link rendering. ([\#17324](element-hq/synapse#17324))
- Change path of the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync implementation to `/org.matrix.simplified_msc3575/sync` since our simplified API is slightly incompatible with what's in the current MSC. ([\#17331](element-hq/synapse#17331))
- Handle device lists notifications for large accounts more efficiently in worker mode. ([\#17333](element-hq/synapse#17333), [\#17358](element-hq/synapse#17358))
- Do not block event sending/receiving while calculating large event auth chains. ([\#17338](element-hq/synapse#17338))
- Tidy up `parse_integer` docs and call sites to reflect the fact that they require non-negative integers by default, and bring `parse_integer_from_args` default in alignment. Contributed by Denis Kasak (@dkasak). ([\#17339](element-hq/synapse#17339))



### Updates to locked dependencies

* Bump authlib from 1.3.0 to 1.3.1. ([\#17343](element-hq/synapse#17343))
* Bump dawidd6/action-download-artifact from 3.1.4 to 5. ([\#17289](element-hq/synapse#17289))
* Bump dawidd6/action-download-artifact from 5 to 6. ([\#17313](element-hq/synapse#17313))
* Bump docker/build-push-action from 5 to 6. ([\#17312](element-hq/synapse#17312))
* Bump jinja2 from 3.1.3 to 3.1.4. ([\#17287](element-hq/synapse#17287))
* Bump lazy_static from 1.4.0 to 1.5.0. ([\#17355](element-hq/synapse#17355))
* Bump msgpack from 1.0.7 to 1.0.8. ([\#17317](element-hq/synapse#17317))
* Bump netaddr from 1.2.1 to 1.3.0. ([\#17353](element-hq/synapse#17353))
* Bump packaging from 24.0 to 24.1. ([\#17352](element-hq/synapse#17352))
* Bump phonenumbers from 8.13.37 to 8.13.39. ([\#17315](element-hq/synapse#17315))
* Bump regex from 1.10.4 to 1.10.5. ([\#17290](element-hq/synapse#17290))
* Bump requests from 2.31.0 to 2.32.2. ([\#17345](element-hq/synapse#17345))
* Bump sentry-sdk from 2.1.1 to 2.3.1. ([\#17263](element-hq/synapse#17263))
* Bump sentry-sdk from 2.3.1 to 2.6.0. ([\#17351](element-hq/synapse#17351))
* Bump tornado from 6.4 to 6.4.1. ([\#17344](element-hq/synapse#17344))
* Bump mypy from 1.8.0 to 1.9.0. ([\#17297](element-hq/synapse#17297))
* Bump types-jsonschema from 4.21.0.20240311 to 4.22.0.20240610. ([\#17288](element-hq/synapse#17288))
* Bump types-netaddr from 1.2.0.20240219 to 1.3.0.20240530. ([\#17314](element-hq/synapse#17314))
* Bump types-pillow from 10.2.0.20240423 to 10.2.0.20240520. ([\#17285](element-hq/synapse#17285))
* Bump types-pyyaml from 6.0.12.12 to 6.0.12.20240311. ([\#17316](element-hq/synapse#17316))
* Bump typing-extensions from 4.11.0 to 4.12.2. ([\#17354](element-hq/synapse#17354))
* Bump urllib3 from 2.0.7 to 2.2.2. ([\#17346](element-hq/synapse#17346))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reivilibre reivilibre reivilibre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Michael-Hollister @reivilibre