-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Conversation
We might have events in the staging area if the service was restarted while there were unhandled events in the staging area. Fixes #10295
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
gneerally fine, a few comments
_INBOUND_EVENT_HANDLING_LOCK_NAME, room_id | ||
) | ||
if lock: | ||
self._process_incoming_pdus_in_room_inner( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably worth logging something here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the common case is that we get the lock, so I wonder if it'd be better to log when we don't acquire the lock? (From reducing our log lines perspective?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've done that for now, but I'm not particularly happy about the wording.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Also note we log handling received PDU
when we come round to actually handling an event)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hrmmm. given this should only happen once (per room with staged events) at startup - I don't feel like the logging would be too onerous?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I completely misread which line this was talking about. Sorry and fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good other than the logging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Synapse 1.38.0rc1 (2021-07-06) ============================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. Features -------- - Implement refresh tokens as specified by [MSC2918](matrix-org/matrix-spec-proposals#2918). ([\#9450](#9450)) - Add support for evicting cache entries based on last access time. ([\#10205](#10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](#10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](#10225), [\#10243](#10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](#10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](#10263)) - Add metrics for new inbound federation staging area. ([\#10284](#10284)) - Add script to print information about recently registered users. ([\#10290](#10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](#10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](#10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](#10264), [\#10267](#10267), [\#10282](#10282), [\#10286](#10286), [\#10291](#10291), [\#10314](#10314), [\#10326](#10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](#10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](#10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](#10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](#10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](#10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](#10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](#10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](#10114)) - Add type hints to the federation servlets. ([\#10213](#10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](#10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](#10239)) - Fix type hints for computing auth events. ([\#10253](#10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](#10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](#10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](#10292))
Synapse 1.38.0 (2021-07-13) =========================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. No significant changes since 1.38.0rc3. Synapse 1.38.0rc3 (2021-07-13) ============================== Internal Changes ---------------- - Build the Debian packages in CI. ([\matrix-org#10247](matrix-org#10247), [\matrix-org#10379](matrix-org#10379)) Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\matrix-org#10336](matrix-org#10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\matrix-org#10287](matrix-org#10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\matrix-org#10331](matrix-org#10331)) Synapse 1.38.0rc1 (2021-07-06) ============================== Features -------- - Implement refresh tokens as specified by [MSC2918](matrix-org/matrix-spec-proposals#2918). ([\matrix-org#9450](matrix-org#9450)) - Add support for evicting cache entries based on last access time. ([\matrix-org#10205](matrix-org#10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\matrix-org#10214](matrix-org#10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\matrix-org#10225](matrix-org#10225), [\matrix-org#10243](matrix-org#10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\matrix-org#10261](matrix-org#10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\matrix-org#10263](matrix-org#10263)) - Add metrics for new inbound federation staging area. ([\matrix-org#10284](matrix-org#10284)) - Add script to print information about recently registered users. ([\matrix-org#10290](matrix-org#10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\matrix-org#10223](matrix-org#10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\matrix-org#10252](matrix-org#10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\matrix-org#10264](matrix-org#10264), [\matrix-org#10267](matrix-org#10267), [\matrix-org#10282](matrix-org#10282), [\matrix-org#10286](matrix-org#10286), [\matrix-org#10291](matrix-org#10291), [\matrix-org#10314](matrix-org#10314), [\matrix-org#10326](matrix-org#10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\matrix-org#10279](matrix-org#10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\matrix-org#10303](matrix-org#10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\matrix-org#10166](matrix-org#10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\matrix-org#10242](matrix-org#10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\matrix-org#10258](matrix-org#10258)) - Fix homeserver config option name in presence router documentation. ([\matrix-org#10288](matrix-org#10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\matrix-org#10302](matrix-org#10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\matrix-org#10114](matrix-org#10114)) - Add type hints to the federation servlets. ([\matrix-org#10213](matrix-org#10213)) - Improve the reliability of auto-joining remote rooms. ([\matrix-org#10237](matrix-org#10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\matrix-org#10239](matrix-org#10239)) - Fix type hints for computing auth events. ([\matrix-org#10253](matrix-org#10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\matrix-org#10256](matrix-org#10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\matrix-org#10268](matrix-org#10268)) - Re-enable a SyTest after it has been fixed. ([\matrix-org#10292](matrix-org#10292))
Synapse 1.38.0 (2021-07-13) =========================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. No significant changes since 1.38.0rc3. Synapse 1.38.0rc3 (2021-07-13) ============================== Internal Changes ---------------- - Build the Debian packages in CI. ([\#10247](matrix-org/synapse#10247), [\#10379](matrix-org/synapse#10379)) Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](matrix-org/synapse#10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](matrix-org/synapse#10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](matrix-org/synapse#10331)) Synapse 1.38.0rc1 (2021-07-06) ============================== Features -------- - Implement refresh tokens as specified by [MSC2918](matrix-org/matrix-spec-proposals#2918). ([\#9450](matrix-org/synapse#9450)) - Add support for evicting cache entries based on last access time. ([\#10205](matrix-org/synapse#10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](matrix-org/synapse#10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](matrix-org/synapse#10225), [\#10243](matrix-org/synapse#10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](matrix-org/synapse#10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](matrix-org/synapse#10263)) - Add metrics for new inbound federation staging area. ([\#10284](matrix-org/synapse#10284)) - Add script to print information about recently registered users. ([\#10290](matrix-org/synapse#10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](matrix-org/synapse#10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](matrix-org/synapse#10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](matrix-org/synapse#10264), [\#10267](matrix-org/synapse#10267), [\#10282](matrix-org/synapse#10282), [\#10286](matrix-org/synapse#10286), [\#10291](matrix-org/synapse#10291), [\#10314](matrix-org/synapse#10314), [\#10326](matrix-org/synapse#10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](matrix-org/synapse#10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](matrix-org/synapse#10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](matrix-org/synapse#10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](matrix-org/synapse#10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](matrix-org/synapse#10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](matrix-org/synapse#10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](matrix-org/synapse#10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](matrix-org/synapse#10114)) - Add type hints to the federation servlets. ([\#10213](matrix-org/synapse#10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](matrix-org/synapse#10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](matrix-org/synapse#10239)) - Fix type hints for computing auth events. ([\#10253](matrix-org/synapse#10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](matrix-org/synapse#10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](matrix-org/synapse#10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](matrix-org/synapse#10292))
Synapse 1.38.0 (2021-07-13) =========================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. No significant changes since 1.38.0rc3. Synapse 1.38.0rc3 (2021-07-13) ============================== Internal Changes ---------------- - Build the Debian packages in CI. ([\#10247](matrix-org/synapse#10247), [\#10379](matrix-org/synapse#10379)) Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](matrix-org/synapse#10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](matrix-org/synapse#10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](matrix-org/synapse#10331)) Synapse 1.38.0rc1 (2021-07-06) ============================== Features -------- - Implement refresh tokens as specified by [MSC2918](matrix-org/matrix-spec-proposals#2918). ([\#9450](matrix-org/synapse#9450)) - Add support for evicting cache entries based on last access time. ([\#10205](matrix-org/synapse#10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](matrix-org/synapse#10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](matrix-org/synapse#10225), [\#10243](matrix-org/synapse#10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](matrix-org/synapse#10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](matrix-org/synapse#10263)) - Add metrics for new inbound federation staging area. ([\#10284](matrix-org/synapse#10284)) - Add script to print information about recently registered users. ([\#10290](matrix-org/synapse#10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](matrix-org/synapse#10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](matrix-org/synapse#10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](matrix-org/synapse#10264), [\#10267](matrix-org/synapse#10267), [\#10282](matrix-org/synapse#10282), [\#10286](matrix-org/synapse#10286), [\#10291](matrix-org/synapse#10291), [\#10314](matrix-org/synapse#10314), [\#10326](matrix-org/synapse#10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](matrix-org/synapse#10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](matrix-org/synapse#10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](matrix-org/synapse#10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](matrix-org/synapse#10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](matrix-org/synapse#10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](matrix-org/synapse#10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](matrix-org/synapse#10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](matrix-org/synapse#10114)) - Add type hints to the federation servlets. ([\#10213](matrix-org/synapse#10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](matrix-org/synapse#10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](matrix-org/synapse#10239)) - Fix type hints for computing auth events. ([\#10253](matrix-org/synapse#10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](matrix-org/synapse#10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](matrix-org/synapse#10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](matrix-org/synapse#10292))
Synapse 1.38.0 (2021-07-13) =========================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. No significant changes since 1.38.0rc3. Synapse 1.38.0rc3 (2021-07-13) ============================== Internal Changes ---------------- - Build the Debian packages in CI. ([\matrix-org#10247](matrix-org#10247), [\matrix-org#10379](matrix-org#10379)) Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\matrix-org#10336](matrix-org#10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\matrix-org#10287](matrix-org#10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\matrix-org#10331](matrix-org#10331)) Synapse 1.38.0rc1 (2021-07-06) ============================== Features -------- - Implement refresh tokens as specified by [MSC2918](matrix-org/matrix-spec-proposals#2918). ([\matrix-org#9450](matrix-org#9450)) - Add support for evicting cache entries based on last access time. ([\matrix-org#10205](matrix-org#10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\matrix-org#10214](matrix-org#10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\matrix-org#10225](matrix-org#10225), [\matrix-org#10243](matrix-org#10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\matrix-org#10261](matrix-org#10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\matrix-org#10263](matrix-org#10263)) - Add metrics for new inbound federation staging area. ([\matrix-org#10284](matrix-org#10284)) - Add script to print information about recently registered users. ([\matrix-org#10290](matrix-org#10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\matrix-org#10223](matrix-org#10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\matrix-org#10252](matrix-org#10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\matrix-org#10264](matrix-org#10264), [\matrix-org#10267](matrix-org#10267), [\matrix-org#10282](matrix-org#10282), [\matrix-org#10286](matrix-org#10286), [\matrix-org#10291](matrix-org#10291), [\matrix-org#10314](matrix-org#10314), [\matrix-org#10326](matrix-org#10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\matrix-org#10279](matrix-org#10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\matrix-org#10303](matrix-org#10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\matrix-org#10166](matrix-org#10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\matrix-org#10242](matrix-org#10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\matrix-org#10258](matrix-org#10258)) - Fix homeserver config option name in presence router documentation. ([\matrix-org#10288](matrix-org#10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\matrix-org#10302](matrix-org#10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\matrix-org#10114](matrix-org#10114)) - Add type hints to the federation servlets. ([\matrix-org#10213](matrix-org#10213)) - Improve the reliability of auto-joining remote rooms. ([\matrix-org#10237](matrix-org#10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\matrix-org#10239](matrix-org#10239)) - Fix type hints for computing auth events. ([\matrix-org#10253](matrix-org#10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\matrix-org#10256](matrix-org#10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\matrix-org#10268](matrix-org#10268)) - Re-enable a SyTest after it has been fixed. ([\matrix-org#10292](matrix-org#10292))
We might have events in the staging area if the service was restarted while there were unhandled events in the staging area.
Fixes #10295
Currently this works by starting up a background task when we see an incoming transaction. The reason for doing it that way is that we don't know what know which workers are handling the incoming federation traffic. The downside to this is that there will be a delay in beginning to process old events until we see a new incoming transaction.
An alternative would be to always handle on e.g. the worker configured to run background tasks?