Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add some clarifications to README.md in the database schema directory #6615

Merged
merged 1 commit into from
Feb 20, 2020

Conversation

rubo77
Copy link
Contributor

@rubo77 rubo77 commented Jan 2, 2020

Please someone explain, what the two folders 16 and 54 are about and why the dump is used for later, and why it is important.

I plan to add some information, how to add the foreign key CONSTRAINT like visualized in #934 like

ALTER TABLE event_forward_extremities ADD CONSTRAINT room_id_fk FOREIGN KEY (room_id) REFERENCES rooms (room_id);
ALTER TABLE room_account_data         ADD CONSTRAINT room_id_fk FOREIGN KEY (room_id) REFERENCES rooms (room_id);
ALTER TABLE room_account_data         ADD CONSTRAINT user_id_fk FOREIGN KEY (user_id) REFERENCES users (name);
ALTER TABLE room_memberships          ADD CONSTRAINT room_id_fk FOREIGN KEY (room_id) REFERENCES rooms (room_id);
ALTER TABLE room_memberships          ADD CONSTRAINT user_id_fk FOREIGN KEY (user_id) REFERENCES users (name);
ALTER TABLE room_stats_state          ADD CONSTRAINT room_id_fk FOREIGN KEY (room_id) REFERENCES rooms (room_id);
ALTER TABLE state_groups              ADD CONSTRAINT room_id_fk FOREIGN KEY (room_id) REFERENCES rooms (room_id);

Flooring keys would be a great way to maintain data integrity, especially as a safeguard against orphaned records.

Maybe we can cleanup script that removes all orphaned entries and then adds all foreign keys to the PostgreSQL Database, or is there already such a script out in the wild somewhere?

@richvdh
Copy link
Member

richvdh commented Jan 2, 2020

I plan to add some information, how to add the foreign key CONSTRAINT

please don't do this. It will lead to people having different database schemas to what we test and develop against, which will be a support nightmare.

I'm not totally against having foreign-key constraints in the database, but if we do it, we should do it for everyone via delta files, not by manual hackery. In any case, please come and discuss why we should do this in #synapse-dev first, once people are back from the holiday period.

@rubo77 rubo77 force-pushed the patch-10 branch 2 times, most recently from e8b5416 to e68992e Compare January 3, 2020 15:37
Copy link
Contributor Author

@rubo77 rubo77 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the readme can be merged now

@richvdh richvdh self-assigned this Feb 6, 2020
@richvdh richvdh self-requested a review February 6, 2020 17:54
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rubo77 I appreciate the contribution, and please don't take this the wrong way, but you might consider leaving documentation updates to people whose first language is English. I'm not sure that this is actually adding much, and I feel like I've had to rewrite most of your contributions anyway?

./scripts-dev/make_full_schema.sh -p postgres_username -o output_dir/
```
There are two folders with full-schema snapshots `16` is a snapshot from 2015,
for historical reference, and one folder with most recent database-schema.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for historical reference, and one folder with most recent database-schema.
for historical reference. `54` is currently the most recent full schema snapshot.

Copy link
Contributor Author

@rubo77 rubo77 Feb 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to "The other contains the most recent full
schema snapshot." so we wouldn't have to change the Readme every time, we update the schema

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but this will be wrong next time we create a full schema... 🤷‍♂️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this wron the next time? "the other" is always the latest, isn't it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, but then there will be three folders.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the point is: we're going to have to update this paragraph next time we create a full schema anyway, so we may as well be clear which folders we are talking about.

changelog.d/6615.misc Outdated Show resolved Hide resolved
@rubo77 rubo77 changed the title [RFC] Explain more in database structure README.txt Add some clarifications to README.md in the database schema directory Feb 20, 2020
@richvdh richvdh merged commit 4fb5f4d into matrix-org:develop Feb 20, 2020
@rubo77 rubo77 deleted the patch-10 branch February 20, 2020 11:01
richvdh added a commit that referenced this pull request Mar 23, 2020
Synapse 1.12.0 (2020-03-23)
===========================

No significant changes since 1.12.0rc1.

Debian packages and Docker images are rebuilt using the latest versions of
dependency libraries, including Twisted 20.3.0. **Please see security advisory
below**.

Security advisory
-----------------

Synapse may be vulnerable to request-smuggling attacks when it is used with a
reverse-proxy. The vulnerabilties are fixed in Twisted 20.3.0, and are
described in
[CVE-2020-10108](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10108)
and
[CVE-2020-10109](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10109).
For a good introduction to this class of request-smuggling attacks, see
https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn.

We are not aware of these vulnerabilities being exploited in the wild, and
do not believe that they are exploitable with current versions of any reverse
proxies. Nevertheless, we recommend that all Synapse administrators ensure that
they have the latest versions of the Twisted library to ensure that their
installation remains secure.

* Administrators using the [`matrix.org` Docker
  image](https://hub.docker.com/r/matrixdotorg/synapse/) or the [Debian/Ubuntu
  packages from
  `matrix.org`](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#matrixorg-packages)
  should ensure that they have version 1.12.0 installed: these images include
  Twisted 20.3.0.
* Administrators who have [installed Synapse from
  source](https://github.com/matrix-org/synapse/blob/master/INSTALL.md#installing-from-source)
  should upgrade Twisted within their virtualenv by running:
  ```sh
  <path_to_virtualenv>/bin/pip install 'Twisted>=20.3.0'
  ```
* Administrators who have installed Synapse from distribution packages should
  consult the information from their distributions.

The `matrix.org` Synapse instance was not vulnerable to these vulnerabilities.

Advance notice of change to the default `git` branch for Synapse
----------------------------------------------------------------

Currently, the default `git` branch for Synapse is `master`, which tracks the
latest release.

After the release of Synapse 1.13.0, we intend to change this default to
`develop`, which is the development tip. This is more consistent with common
practice and modern `git` usage.

Although we try to keep `develop` in a stable state, there may be occasions
where regressions creep in. Developers and distributors who have scripts which
run builds using the default branch of `Synapse` should therefore consider
pinning their scripts to `master`.

Synapse 1.12.0rc1 (2020-03-19)
==============================

Features
--------

- Changes related to room alias management ([MSC2432](matrix-org/matrix-spec-proposals#2432)):
  - Publishing/removing a room from the room directory now requires the user to have a power level capable of modifying the canonical alias, instead of the room aliases. ([\#6965](#6965))
  - Validate the `alt_aliases` property of canonical alias events. ([\#6971](#6971))
  - Users with a power level sufficient to modify the canonical alias of a room can now delete room aliases. ([\#6986](#6986))
  - Implement updated authorization rules and redaction rules for aliases events, from [MSC2261](matrix-org/matrix-spec-proposals#2261) and [MSC2432](matrix-org/matrix-spec-proposals#2432). ([\#7037](#7037))
  - Stop sending m.room.aliases events during room creation and upgrade. ([\#6941](#6941))
  - Synapse no longer uses room alias events to calculate room names for push notifications. ([\#6966](#6966))
  - The room list endpoint no longer returns a list of aliases. ([\#6970](#6970))
  - Remove special handling of aliases events from [MSC2260](matrix-org/matrix-spec-proposals#2260) added in v1.10.0rc1. ([\#7034](#7034))
- Expose the `synctl`, `hash_password` and `generate_config` commands in the snapcraft package. Contributed by @devec0. ([\#6315](#6315))
- Check that server_name is correctly set before running database updates. ([\#6982](#6982))
- Break down monthly active users by `appservice_id` and emit via Prometheus. ([\#7030](#7030))
- Render a configurable and comprehensible error page if something goes wrong during the SAML2 authentication process. ([\#7058](#7058), [\#7067](#7067))
- Add an optional parameter to control whether other sessions are logged out when a user's password is modified. ([\#7085](#7085))
- Add prometheus metrics for the number of active pushers. ([\#7103](#7103), [\#7106](#7106))
- Improve performance when making HTTPS requests to sygnal, sydent, etc, by sharing the SSL context object between connections. ([\#7094](#7094))

Bugfixes
--------

- When a user's profile is updated via the admin API, also generate a displayname/avatar update for that user in each room. ([\#6572](#6572))
- Fix a couple of bugs in email configuration handling. ([\#6962](#6962))
- Fix an issue affecting worker-based deployments where replication would stop working, necessitating a full restart, after joining a large room. ([\#6967](#6967))
- Fix `duplicate key` error which was logged when rejoining a room over federation. ([\#6968](#6968))
- Prevent user from setting 'deactivated' to anything other than a bool on the v2 PUT /users Admin API. ([\#6990](#6990))
- Fix py35-old CI by using native tox package. ([\#7018](#7018))
- Fix a bug causing `org.matrix.dummy_event` to be included in responses from `/sync`. ([\#7035](#7035))
- Fix a bug that renders UTF-8 text files incorrectly when loaded from media. Contributed by @TheStranjer. ([\#7044](#7044))
- Fix a bug that would cause Synapse to respond with an error about event visibility if a client tried to request the state of a room at a given token. ([\#7066](#7066))
- Repair a data-corruption issue which was introduced in Synapse 1.10, and fixed in Synapse 1.11, and which could cause `/sync` to return with 404 errors about missing events and unknown rooms. ([\#7070](#7070))
- Fix a bug causing account validity renewal emails to be sent even if the feature is turned off in some cases. ([\#7074](#7074))

Improved Documentation
----------------------

- Updated CentOS8 install instructions. Contributed by Richard Kellner. ([\#6925](#6925))
- Fix `POSTGRES_INITDB_ARGS` in the `contrib/docker/docker-compose.yml` example docker-compose configuration. ([\#6984](#6984))
- Change date in [INSTALL.md](./INSTALL.md#tls-certificates) for last date of getting TLS certificates to November 2019. ([\#7015](#7015))
- Document that the fallback auth endpoints must be routed to the same worker node as the register endpoints. ([\#7048](#7048))

Deprecations and Removals
-------------------------

- Remove the unused query_auth federation endpoint per [MSC2451](matrix-org/matrix-spec-proposals#2451). ([\#7026](#7026))

Internal Changes
----------------

- Add type hints to `logging/context.py`. ([\#6309](#6309))
- Add some clarifications to `README.md` in the database schema directory. ([\#6615](#6615))
- Refactoring work in preparation for changing the event redaction algorithm. ([\#6874](#6874), [\#6875](#6875), [\#6983](#6983), [\#7003](#7003))
- Improve performance of v2 state resolution for large rooms. ([\#6952](#6952), [\#7095](#7095))
- Reduce time spent doing GC, by freezing objects on startup. ([\#6953](#6953))
- Minor perfermance fixes to `get_auth_chain_ids`. ([\#6954](#6954))
- Don't record remote cross-signing keys in the `devices` table. ([\#6956](#6956))
- Use flake8-comprehensions to enforce good hygiene of list/set/dict comprehensions. ([\#6957](#6957))
- Merge worker apps together. ([\#6964](#6964), [\#7002](#7002), [\#7055](#7055), [\#7104](#7104))
- Remove redundant `store_room` call from `FederationHandler._process_received_pdu`. ([\#6979](#6979))
- Update warning for incorrect database collation/ctype to include link to documentation. ([\#6985](#6985))
- Add some type annotations to the database storage classes. ([\#6987](#6987))
- Port `synapse.handlers.presence` to async/await. ([\#6991](#6991), [\#7019](#7019))
- Add some type annotations to the federation base & client classes. ([\#6995](#6995))
- Port `synapse.rest.keys` to async/await. ([\#7020](#7020))
- Add a type check to `is_verified` when processing room keys. ([\#7045](#7045))
- Add type annotations and comments to the auth handler. ([\#7063](#7063))
phil-flex pushed a commit to phil-flex/synapse that referenced this pull request Apr 15, 2020
babolivier pushed a commit that referenced this pull request Sep 1, 2021
#6615)

* commit '4fb5f4d0c':
  Add some clarifications to README.md in the database schema directory. (#6615)
  Minor perf fixes to `get_auth_chain_ids`.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants