Add type hints to logging/context.py #6309

neiljp · 2019-10-31T12:17:29Z

This is a preliminary exploration of adding fuller type hints to a file that seems to be checked by mypy in CI - not all are? I can expand this PR to include more of logging/ if this kind of approach is deemed reasonable.

I'm not running a full dev environment right now due to lack of space/time, but have a matching mypy installed. I have one outstanding type hint complaint by mypy from my own changes, and one that appears to be in upstream.

I've split the flake8 changes into a separate commit, for now, as I'm not sure if they're all necessary or desirable - partly based on my experience with type hint style in Zulip. See the comments in the commit - basically re spacing and quotes.

Most changes in the first commit should be straightforward, but I'm unsure regarding whether:

extra assert statements are acceptable; in Zulip they're used to help with typing
the removal of the self.previous_context = None line (currently commented); it being None complicates the typing and appears a bit strange in any case

This does not currently fully add type hints to every function, but covers a good proportion.

auscompgeek · 2019-10-31T12:35:48Z

re: spacing around type hints for default args - this is a PEP 8 recommendation.

re: forward declaration within the same class - from __future__ import annotations isn't used (that's new in Python 3.7) so annotations need to be valid expressions at runtime.

neiljp · 2019-10-31T13:43:52Z

re: spacing around type hints for default args - this is a PEP 8 recommendation.

Agreed; perhaps this was in flux when Zulip started using it or it was deemed too space-filling, though I know Zulip doesn't fully match PEP8 in other ways too.

re: forward declaration within the same class - from __future__ import annotations isn't used (that's new in Python 3.7) so annotations need to be valid expressions at runtime.

Right, so to support Python 3.5 forward declarations are required.

anoadragon453 · 2019-10-31T14:43:47Z

changelog.d/6309.misc

@@ -0,0 +1 @@
+Add type hints to logging/context.py.


Suggested change

Add type hints to logging/context.py.

Add type hints to `logging/context.py`.

neiljp · 2019-11-25T01:48:59Z

@hawkowl Any preliminary feedback on this would be useful :)

anoadragon453

A couple small bits but otherwise looks generally sane.

anoadragon453 · 2019-12-12T11:50:37Z

synapse/logging/context.py


 from twisted.internet import defer, threads

+if False:
+    from synapse.logging.scopecontextmanager import _LogContextScope


I'm missing the point of this?

This avoids the import at runtime, but type checkers become aware of the name. In sufficiently recent versions this uses typing.TYPE_CHECKING, so I've changed the commit to use this.

anoadragon453 · 2019-12-12T12:06:49Z

synapse/logging/context.py

        self.defaults = defaults

-    def filter(self, record):
+    def filter(self, record) -> bool:  # FIXME: Use Literal[True] ?


Literal[True] works.

I've not made this change yet, as only python 3.8 has Literal present, unless the project is willing to add a dependency upon typing_extensions.

fwiw, we already have a dependency on typing-extensions (currently on v3.7.4)

anoadragon453 · 2019-12-12T12:25:32Z

synapse/logging/context.py

@@ -317,7 +333,7 @@ def __exit__(self, type, value, traceback):
                logger.warning(
                    "Expected logging context %s but found %s", self, current
                )
-        self.previous_context = None
+        # self.previous_context = None  # FIXME If this *can* be removed, this simplifies the typing...


As far as I can tell, this isn't used for anything, so feel free.

Signed-off-by: neiljp (Neil Pilgrim) <[email protected]>

neiljp · 2020-02-04T00:07:02Z

@anoadragon453 Updated; should have covered all bases now.

neiljp · 2020-03-07T02:53:37Z

@anoadragon453 Just checking in, in case you're waiting on me here.

richvdh

sorry @neiljp I think this fell off the radar a bit. It looks great to me - thank you very much!

fixed

@devec0

Synapse 1.12.0 (2020-03-23) =========================== No significant changes since 1.12.0rc1. Debian packages and Docker images are rebuilt using the latest versions of dependency libraries, including Twisted 20.3.0. **Please see security advisory below**. Security advisory ----------------- Synapse may be vulnerable to request-smuggling attacks when it is used with a reverse-proxy. The vulnerabilties are fixed in Twisted 20.3.0, and are described in [CVE-2020-10108](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10108) and [CVE-2020-10109](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10109). For a good introduction to this class of request-smuggling attacks, see https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn. We are not aware of these vulnerabilities being exploited in the wild, and do not believe that they are exploitable with current versions of any reverse proxies. Nevertheless, we recommend that all Synapse administrators ensure that they have the latest versions of the Twisted library to ensure that their installation remains secure. * Administrators using the [`matrix.org` Docker image](https://hub.docker.com/r/matrixdotorg/synapse/) or the [Debian/Ubuntu packages from `matrix.org`](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#matrixorg-packages) should ensure that they have version 1.12.0 installed: these images include Twisted 20.3.0. * Administrators who have [installed Synapse from source](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source) should upgrade Twisted within their virtualenv by running: ```sh <path_to_virtualenv>/bin/pip install 'Twisted>=20.3.0' ``` * Administrators who have installed Synapse from distribution packages should consult the information from their distributions. The `matrix.org` Synapse instance was not vulnerable to these vulnerabilities. Advance notice of change to the default `git` branch for Synapse ---------------------------------------------------------------- Currently, the default `git` branch for Synapse is `master`, which tracks the latest release. After the release of Synapse 1.13.0, we intend to change this default to `develop`, which is the development tip. This is more consistent with common practice and modern `git` usage. Although we try to keep `develop` in a stable state, there may be occasions where regressions creep in. Developers and distributors who have scripts which run builds using the default branch of `Synapse` should therefore consider pinning their scripts to `master`. Synapse 1.12.0rc1 (2020-03-19) ============================== Features -------- - Changes related to room alias management ([MSC2432](matrix-org/matrix-spec-proposals#2432)): - Publishing/removing a room from the room directory now requires the user to have a power level capable of modifying the canonical alias, instead of the room aliases. ([\#6965](#6965)) - Validate the `alt_aliases` property of canonical alias events. ([\#6971](#6971)) - Users with a power level sufficient to modify the canonical alias of a room can now delete room aliases. ([\#6986](#6986)) - Implement updated authorization rules and redaction rules for aliases events, from [MSC2261](matrix-org/matrix-spec-proposals#2261) and [MSC2432](matrix-org/matrix-spec-proposals#2432). ([\#7037](#7037)) - Stop sending m.room.aliases events during room creation and upgrade. ([\#6941](#6941)) - Synapse no longer uses room alias events to calculate room names for push notifications. ([\#6966](#6966)) - The room list endpoint no longer returns a list of aliases. ([\#6970](#6970)) - Remove special handling of aliases events from [MSC2260](matrix-org/matrix-spec-proposals#2260) added in v1.10.0rc1. ([\#7034](#7034)) - Expose the `synctl`, `hash_password` and `generate_config` commands in the snapcraft package. Contributed by @devec0. ([\#6315](#6315)) - Check that server_name is correctly set before running database updates. ([\#6982](#6982)) - Break down monthly active users by `appservice_id` and emit via Prometheus. ([\#7030](#7030)) - Render a configurable and comprehensible error page if something goes wrong during the SAML2 authentication process. ([\#7058](#7058), [\#7067](#7067)) - Add an optional parameter to control whether other sessions are logged out when a user's password is modified. ([\#7085](#7085)) - Add prometheus metrics for the number of active pushers. ([\#7103](#7103), [\#7106](#7106)) - Improve performance when making HTTPS requests to sygnal, sydent, etc, by sharing the SSL context object between connections. ([\#7094](#7094)) Bugfixes -------- - When a user's profile is updated via the admin API, also generate a displayname/avatar update for that user in each room. ([\#6572](#6572)) - Fix a couple of bugs in email configuration handling. ([\#6962](#6962)) - Fix an issue affecting worker-based deployments where replication would stop working, necessitating a full restart, after joining a large room. ([\#6967](#6967)) - Fix `duplicate key` error which was logged when rejoining a room over federation. ([\#6968](#6968)) - Prevent user from setting 'deactivated' to anything other than a bool on the v2 PUT /users Admin API. ([\#6990](#6990)) - Fix py35-old CI by using native tox package. ([\#7018](#7018)) - Fix a bug causing `org.matrix.dummy_event` to be included in responses from `/sync`. ([\#7035](#7035)) - Fix a bug that renders UTF-8 text files incorrectly when loaded from media. Contributed by @TheStranjer. ([\#7044](#7044)) - Fix a bug that would cause Synapse to respond with an error about event visibility if a client tried to request the state of a room at a given token. ([\#7066](#7066)) - Repair a data-corruption issue which was introduced in Synapse 1.10, and fixed in Synapse 1.11, and which could cause `/sync` to return with 404 errors about missing events and unknown rooms. ([\#7070](#7070)) - Fix a bug causing account validity renewal emails to be sent even if the feature is turned off in some cases. ([\#7074](#7074)) Improved Documentation ---------------------- - Updated CentOS8 install instructions. Contributed by Richard Kellner. ([\#6925](#6925)) - Fix `POSTGRES_INITDB_ARGS` in the `contrib/docker/docker-compose.yml` example docker-compose configuration. ([\#6984](#6984)) - Change date in [INSTALL.md](./INSTALL.md#tls-certificates) for last date of getting TLS certificates to November 2019. ([\#7015](#7015)) - Document that the fallback auth endpoints must be routed to the same worker node as the register endpoints. ([\#7048](#7048)) Deprecations and Removals ------------------------- - Remove the unused query_auth federation endpoint per [MSC2451](matrix-org/matrix-spec-proposals#2451). ([\#7026](#7026)) Internal Changes ---------------- - Add type hints to `logging/context.py`. ([\#6309](#6309)) - Add some clarifications to `README.md` in the database schema directory. ([\#6615](#6615)) - Refactoring work in preparation for changing the event redaction algorithm. ([\#6874](#6874), [\#6875](#6875), [\#6983](#6983), [\#7003](#7003)) - Improve performance of v2 state resolution for large rooms. ([\#6952](#6952), [\#7095](#7095)) - Reduce time spent doing GC, by freezing objects on startup. ([\#6953](#6953)) - Minor perfermance fixes to `get_auth_chain_ids`. ([\#6954](#6954)) - Don't record remote cross-signing keys in the `devices` table. ([\#6956](#6956)) - Use flake8-comprehensions to enforce good hygiene of list/set/dict comprehensions. ([\#6957](#6957)) - Merge worker apps together. ([\#6964](#6964), [\#7002](#7002), [\#7055](#7055), [\#7104](#7104)) - Remove redundant `store_room` call from `FederationHandler._process_received_pdu`. ([\#6979](#6979)) - Update warning for incorrect database collation/ctype to include link to documentation. ([\#6985](#6985)) - Add some type annotations to the database storage classes. ([\#6987](#6987)) - Port `synapse.handlers.presence` to async/await. ([\#6991](#6991), [\#7019](#7019)) - Add some type annotations to the federation base & client classes. ([\#6995](#6995)) - Port `synapse.rest.keys` to async/await. ([\#7020](#7020)) - Add a type check to `is_verified` when processing room keys. ([\#7045](#7045)) - Add type annotations and comments to the auth handler. ([\#7063](#7063))

* Add type hints to logging/context.py Signed-off-by: neiljp (Neil Pilgrim) <[email protected]>

* commit '2bff4457d': Add type hints to logging/context.py (#6309)

neiljp marked this pull request as ready for review October 31, 2019 12:19

neiljp force-pushed the 2019-10-typehints-logging-context branch from 4d0a726 to 8672cd6 Compare October 31, 2019 12:20

richvdh requested a review from a team October 31, 2019 12:35

neiljp force-pushed the 2019-10-typehints-logging-context branch 3 times, most recently from 9178dd8 to f4e520c Compare October 31, 2019 14:29

anoadragon453 assigned hawkowl Oct 31, 2019

anoadragon453 reviewed Oct 31, 2019

View reviewed changes

anoadragon453 previously requested changes Dec 12, 2019

View reviewed changes

anoadragon453 self-assigned this Dec 12, 2019

neiljp force-pushed the 2019-10-typehints-logging-context branch 3 times, most recently from 7c3491e to ba1adfe Compare February 3, 2020 19:08

neiljp added 2 commits February 3, 2020 11:29

Add type hints to logging/context.py

9177961

Signed-off-by: neiljp (Neil Pilgrim) <[email protected]>

Add changelog entry.

c9afed2

Signed-off-by: neiljp (Neil Pilgrim) <[email protected]>

neiljp force-pushed the 2019-10-typehints-logging-context branch from ba1adfe to c9afed2 Compare February 3, 2020 19:29

richvdh self-requested a review March 7, 2020 06:56

richvdh approved these changes Mar 7, 2020

View reviewed changes

richvdh changed the title ~~WIP Add type hints to logging/context.py~~ Add type hints to logging/context.py Mar 7, 2020

richvdh merged commit 2bff445 into matrix-org:develop Mar 7, 2020

babolivier mentioned this pull request Mar 10, 2020

SAML2: render a comprehensible error page if something goes wrong #7058

Merged

phil-flex pushed a commit to phil-flex/synapse that referenced this pull request Apr 15, 2020

Add type hints to logging/context.py (matrix-org#6309)

08d51b0

* Add type hints to logging/context.py Signed-off-by: neiljp (Neil Pilgrim) <[email protected]>

babolivier pushed a commit that referenced this pull request Sep 1, 2021

Add type hints to logging/context.py (#6309)

942f048

* commit '2bff4457d': Add type hints to logging/context.py (#6309)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add type hints to logging/context.py #6309

Add type hints to logging/context.py #6309

neiljp commented Oct 31, 2019

auscompgeek commented Oct 31, 2019

neiljp commented Oct 31, 2019

anoadragon453 Oct 31, 2019

neiljp commented Nov 25, 2019

anoadragon453 left a comment

anoadragon453 Dec 12, 2019

neiljp Feb 2, 2020

anoadragon453 Dec 12, 2019

neiljp Feb 2, 2020

richvdh Feb 3, 2020

anoadragon453 Dec 12, 2019

neiljp commented Feb 4, 2020

neiljp commented Mar 7, 2020

richvdh left a comment

	Add type hints to logging/context.py.
	Add type hints to `logging/context.py`.

Add type hints to logging/context.py #6309

Add type hints to logging/context.py #6309

Conversation

neiljp commented Oct 31, 2019

auscompgeek commented Oct 31, 2019

neiljp commented Oct 31, 2019

anoadragon453 Oct 31, 2019

Choose a reason for hiding this comment

neiljp commented Nov 25, 2019

anoadragon453 left a comment

Choose a reason for hiding this comment

anoadragon453 Dec 12, 2019

Choose a reason for hiding this comment

neiljp Feb 2, 2020

Choose a reason for hiding this comment

anoadragon453 Dec 12, 2019

Choose a reason for hiding this comment

neiljp Feb 2, 2020

Choose a reason for hiding this comment

richvdh Feb 3, 2020

Choose a reason for hiding this comment

anoadragon453 Dec 12, 2019

Choose a reason for hiding this comment

neiljp commented Feb 4, 2020

neiljp commented Mar 7, 2020

richvdh left a comment

Choose a reason for hiding this comment