sqlite3.OperationalError: no such table: event_txn_id_device_id when upgrading from Docker v1.89.0 to v1.92.2

[Anthial]

Description

Messages cannot be sent. When attempting to send a message the server logs a database error.

Steps to reproduce

Note, I do not currently have the capabilities to do this fresh, but assume a normal upgrade path from somewhere around v1.70.

1. Run Synapse Docker v1.89.0 with sqlite3
2. Upgrade to v1.92.2 via docker compose pull
3. Messages can no longer be sent

Homeserver

the.eldritch.gift

Synapse Version

v1.92.2

Installation Method

Docker (matrixdotorg/synapse)

Database

SQLite (Docker Default)

Workers

Single process

Platform

Distro: Pop OS 22.04
Hardware:

• CPU: Intel(R) Celeron(R) N4505 @ 2.00GHz
• 16GB DDR4 RAM
• 500GB Storage

Configuration

No response

Relevant log output

docker-synapse-1  | 2023-09-17 12:52:14,536 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 0/5
docker-synapse-1  | 2023-09-17 12:52:14,537 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 1/5
docker-synapse-1  | 2023-09-17 12:52:14,537 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 2/5
docker-synapse-1  | 2023-09-17 12:52:14,538 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 3/5
docker-synapse-1  | 2023-09-17 12:52:14,538 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 4/5
docker-synapse-1  | 2023-09-17 12:52:14,539 - synapse.storage.txn - 780 - WARNING - PUT-287 - [TXN OPERROR] {get_event_id_from_transaction_id_and_device_id-575} no such table: event_txn_id_device_id 5/5
docker-synapse-1  | 2023-09-17 12:52:14,539 - synapse.http.server - 140 - ERROR - PUT-287 - Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x7f35bb9654d0 method='PUT' uri='/_matrix/client/v3/rooms/!bMPYjpwMVArbXrMnrM%3Athe.eldritch.gift/send/m.room.encrypted/m1694955131659.2' clientproto='HTTP/1.1' site='8008'>
docker-synapse-1  | Traceback (most recent call last):
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
docker-synapse-1  |     result = context.run(
docker-synapse-1  |              ^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
docker-synapse-1  |     return g.throw(self.type, self.value, self.tb)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 366, in _do
docker-synapse-1  |     ) = await self.event_creation_handler.create_and_send_nonmember_event(
docker-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1012, in create_and_send_nonmember_event
docker-synapse-1  |     event = await self.get_event_from_transaction(
docker-synapse-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 931, in get_event_from_transaction
docker-synapse-1  |     existing_event_id = await self.get_event_id_from_transaction(
docker-synapse-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 901, in get_event_id_from_transaction
docker-synapse-1  |     await self.store.get_event_id_from_transaction_id_and_device_id(
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/events_worker.py", line 2031, in get_event_id_from_transaction_id_and_device_id
docker-synapse-1  |     return await self.db_pool.simple_select_one_onecol(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1709, in simple_select_one_onecol
docker-synapse-1  |     return await self.runInteraction(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 945, in runInteraction
docker-synapse-1  |     return await delay_cancellation(_runInteraction())
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
docker-synapse-1  |     result = context.run(
docker-synapse-1  |              ^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
docker-synapse-1  |     return g.throw(self.type, self.value, self.tb)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 911, in _runInteraction
docker-synapse-1  |     result = await self.runWithConnection(
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1040, in runWithConnection
docker-synapse-1  |     return await make_deferred_yieldable(
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
docker-synapse-1  |     result = inContext.theWork()  # type: ignore[attr-defined]
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
docker-synapse-1  |     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
docker-synapse-1  |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
docker-synapse-1  |     return self.currentContext().callWithContext(ctx, func, *args, **kw)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
docker-synapse-1  |     return func(*args, **kw)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
docker-synapse-1  |     result = func(conn, *args, **kw)
docker-synapse-1  |              ^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1033, in inner_func
docker-synapse-1  |     return func(db_conn, *args, **kwargs)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 773, in new_transaction
docker-synapse-1  |     r = func(cursor, *args, **kwargs)
docker-synapse-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1752, in simple_select_one_onecol_txn
docker-synapse-1  |     ret = cls.simple_select_onecol_txn(
docker-synapse-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1775, in simple_select_onecol_txn
docker-synapse-1  |     txn.execute(sql, list(keyvalues.values()))
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 432, in execute
docker-synapse-1  |     self._do_execute(self.txn.execute, sql, parameters)
docker-synapse-1  |   File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 484, in _do_execute
docker-synapse-1  |     return func(sql, *args, **kwargs)
docker-synapse-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-synapse-1  | sqlite3.OperationalError: no such table: event_txn_id_device_id

Anything else that would be useful to know?

I do believe this was also the case for Synapse v1.90.0

[michaelschefczyk]

With postgresql, I did end up in a similar situation. I had to revert to 1.89.0, as I do not have docker images for the versions in between available. Before failing, the database was updated, so I was required to restore the previous version from a backup/snapshot.

[clokep]

Table was added in #15318, back in Synapse 1.83.0.

[michaelschefczyk]

Thank you very much @clokep.

Is there a practical way to add missing tables to the database? Would my homeserver otherwise be doomed and lost so that I would have to restart from zero under a different name - or use signal only instead of two alternatives?

[clokep]

Is there a practical way to add missing tables to the database? Would my homeserver otherwise be doomed and lost so that I would have to restart from zero under a different name - or use signal only instead of two alternatives?

I'm not sure how it missed the update to add it, but I haven't actually done any investigation.

Can you run SELECT * FROM applied_schema_deltas; and paste the results here?

[clokep]

It looks like Synapse 1.83.0 did the following operations:

• Bumped the minimum schema compat version from 73 -> 74.
• Bumped the schema version from 74 -> 76.

I'm suspicious that bumping the schema version twice isn't safe as we have CI checks which compare if develop is +/- of the PR version.

---

@Anthial The SQL in #16335 (comment) would be interesting to see still.

[Anthial]

Yes! I'm on it! Just trying to figure out the best way of accessing the SQLite database! I figure I might just back it up once more, and use a DB browser to find the required information! (There seems to be a lack of documentation on how to access the SQLite database when utilising Docker)

[Anthial]

Something like?

Version | file
73 | 73/01event_failed_pull_attempts.sql
-- | --
73 | 73/02add_pusher_enabled.sql
73 | 73/02room_id_indexes_for_purging.sql
73 | 73/03pusher_device_id.sql
73 | 73/03users_approved_column.sql
73 | 73/04partial_join_details.sql
73 | 73/04pending_device_list_updates.sql
73 | 73/05old_push_actions.sql.sqlite
73 | 73/06thread_notifications_thread_id_idx.sql
73 | 73/08thread_receipts_non_null.sql.sqlite
73 | 73/09partial_joined_via_destination.sql
73 | 73/09threads_table.sql
73 | 73/10_update_sqlite_fts4_tokenizer.py
73 | 73/10login_tokens.sql
73 | 73/12refactor_device_list_outbound_pokes.sql
73 | 73/13add_device_lists_index.sql
73 | 73/20_un_partial_stated_room_stream.sql
73 | 73/22_rebuild_user_dir_stats.sql
73 | 73/22_un_partial_stated_event_stream.sql
73 | 73/23_fix_thread_index.sql
73 | 73/24_events_jump_to_date_index.sql
73 | 73/25drop_presence.sql
74 | 74/01_user_directory_stale_remote_users.sql
74 | 74/01membership_tables_event_stream_ordering.sql
74 | 74/02_set_device_id_for_pushers_bg_update.sql
74 | 74/02membership_tables_event_stream_ordering_triggers.py
76 | 76/01_add_profiles_full_user_id_column.sql
76 | 76/02_add_user_filters_full_user_id_column.sql
76 | 76/03_per_user_experimental_features.sql
76 | 76/04_add_room_forgetter.sql
77 | 77/03bg_populate_full_user_id_profiles.sql
77 | 77/04bg_populate_full_user_id_user_filters.sql
77 | 77/05thread_notifications_backfill.sql
77 | 77/06thread_notifications_not_null.sql.sqlite
77 | 77/14bg_indices_event_stream_ordering.sql
78 | 78/01_validate_and_update_profiles.py
78 | 78/02_validate_and_update_user_filters.py
78 | 78/03_remove_unused_indexes_user_filters.py
78 | 78/04_add_full_user_id_index_user_filters.py
78 | 78/03event_extremities_constraints.py
79 | 79/03_read_write_locks_triggers.sql.sqlite
79 | 79/04_mitigate_stream_ordering_update_race.py
79 | 79/05_read_write_locks_triggers.sql.sqlite
80 | 80/01_users_alter_locked.sql
80 | 80/02_scheduled_tasks.sql

[clokep]

Thanks! It looks like you're missing 3 deltas:

• 74/03_membership_tables_event_stream_ordering.sql.sqlite
• 74/04_membership_tables_event_stream_ordering_triggers.py
• 74/05_events_txn_id_device_id.sql

[Anthial]

Very strange! I wonder how that could have happened!

[Anthial]

Is there anyway to apply these deltas to the database or is it permanently broken?

[clokep]

Is there anyway to apply these deltas to the database or is it permanently broken?

You could try to apply them by hand, in order. (And hope no invalid data has occurred in the meantime.) I'd love to know how you database ended up in this state though.

[clokep]

I'm suspicious that bumping the schema version twice isn't safe as we have CI checks which compare if develop is +/- of the PR version.

#11559 seems to describe the concern I have.

[Anthial]

Alright! I'm going to try to run each of these commands (from each of these deltas, including one more, since I also seem to be missing 03_room_membership_index.sql)! Wish me luck! 😁

I really hope this is just, like, a one or two-time occurrence and that most folks manage to avoid this kind of weirdness!

[Anthial]

SQL manually is always a little bit scary BUT IT WORKS! Thank you so much @clokep! You are the best! Feel free to ask if you want to see any kind of logs, and I'll try to get them for you!

[clokep]

Glad that it worked! I'll poke the team and see if anyone has ideas (there's 1 or 2 things I want to try also.)

[erikjohnston]

We should be correctly handling jumps in schema versions:

synapse/synapse/storage/prepare_database.py

Line 424 in 2763c49

for v in range(start_ver, SCHEMA_VERSION + 1):

There was multiple upgrades and downgrades of schema version in v1.80.0..v1.83.0. @Anthial I assume you weren't running against the develop branch (or prereleases?) at any point?

Otherwise, the schema versions seem sane between v1.70.0 and v1.90.0:

This shows the schema version at each release, and the new files for each version.

v1.70.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/06thread_notifications_thread_id_idx.sql | 23 +++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/08thread_receipts_non_null.sql.postgres  | 23 +++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/08thread_receipts_non_null.sql.sqlite    | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/09partial_joined_via_destination.sql     | 18 ++++++++++++++++++
 synapse/storage/schema/main/delta/73/09threads_table.sql                      | 30 ++++++++++++++++++++++++++++++
 5 files changed, 170 insertions(+)

v1.71.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/10_update_sqlite_fts4_tokenizer.py | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/10login_tokens.sql                 | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+)

v1.72.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/11event_search_room_id_n_distinct.sql.postgres | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

v1.73.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/12refactor_device_list_outbound_pokes.sql | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/13add_device_lists_index.sql              | 20 ++++++++++++++++++++
 2 files changed, 73 insertions(+)

v1.74.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/20_un_partial_stated_room_stream.sql              | 32 ++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/21_un_partial_stated_room_stream_seq.sql.postgres | 20 ++++++++++++++++++++
 synapse/storage/schema/main/delta/73/22_rebuild_user_dir_stats.sql                     | 29 +++++++++++++++++++++++++++++
 3 files changed, 81 insertions(+)

v1.75.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/22_un_partial_stated_event_stream.sql             | 34 ++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/23_fix_thread_index.sql                           | 33 +++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/73/23_un_partial_stated_room_stream_seq.sql.postgres | 20 ++++++++++++++++++++
 3 files changed, 87 insertions(+)

v1.76.0: New Schema Version: 73
 synapse/storage/schema/main/delta/73/24_events_jump_to_date_index.sql | 17 +++++++++++++++++
 synapse/storage/schema/main/delta/73/25drop_presence.sql              | 17 +++++++++++++++++
 2 files changed, 34 insertions(+)

v1.77.0: New Schema Version: 74

v1.78.0: New Schema Version: 74

v1.79.0: New Schema Version: 74

v1.80.0: New Schema Version: 74
 synapse/storage/schema/main/delta/74/01_user_directory_stale_remote_users.sql | 39 +++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/90COMMENTS_destinations.sql.postgres     | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)

v1.81.0: New Schema Version: 74
 synapse/storage/schema/main/delta/74/02_set_device_id_for_pushers_bg_update.sql | 19 +++++++++++++++++++
 synapse/storage/schema/main/delta/74/03_room_membership_index.sql               | 19 +++++++++++++++++++
 2 files changed, 38 insertions(+)

v1.82.0: New Schema Version: 74
 synapse/storage/schema/main/delta/74/04_delete_e2e_backup_keys_for_deactivated_users.sql | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

v1.83.0: New Schema Version: 76
 synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.postgres | 29 +++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.sqlite   | 23 +++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/04_membership_tables_event_stream_ordering_triggers.py  | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/05_events_txn_id_device_id.sql                          | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/76/01_add_profiles_full_user_id_column.sql                 | 20 ++++++++++++++++++++
 synapse/storage/schema/main/delta/76/02_add_user_filters_full_user_id_column.sql             | 20 ++++++++++++++++++++
 synapse/storage/schema/main/delta/76/03_per_user_experimental_features.sql                   | 27 +++++++++++++++++++++++++++
 7 files changed, 251 insertions(+)

v1.84.0: New Schema Version: 76
 synapse/storage/schema/main/delta/76/04_add_room_forgetter.sql | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

v1.85.0: New Schema Version: 77
 synapse/storage/schema/main/delta/77/01_add_profiles_not_valid_check.sql.postgres                            |  16 +++++++++++++++
 synapse/storage/schema/main/delta/77/02_add_user_filters_not_valid_check.sql.postgres                        |  16 +++++++++++++++
 synapse/storage/schema/main/delta/77/03bg_populate_full_user_id_profiles.sql                                 |  16 +++++++++++++++
 synapse/storage/schema/main/delta/77/04bg_populate_full_user_id_user_filters.sql                             |  16 +++++++++++++++
 synapse/storage/schema/main/delta/77/05thread_notifications_backfill.sql                                     |  28 ++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/77/06thread_notifications_not_null.sql.sqlite                              | 102 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/77/06thread_notifications_not_null_event_push_actions.sql.postgres         |  27 +++++++++++++++++++++++++
 synapse/storage/schema/main/delta/77/06thread_notifications_not_null_event_push_actions_staging.sql.postgres |  27 +++++++++++++++++++++++++
 synapse/storage/schema/main/delta/77/06thread_notifications_not_null_event_push_summary.sql.postgres         |  29 +++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/77/14bg_indices_event_stream_ordering.sql                                  |  20 +++++++++++++++++++
 10 files changed, 297 insertions(+)

v1.86.0: New Schema Version: 78
 synapse/storage/schema/main/delta/78/01_validate_and_update_profiles.py     | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/78/02_validate_and_update_user_filters.py | 95 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 187 insertions(+)

v1.87.0: New Schema Version: 78
 synapse/storage/schema/main/delta/78/03_remove_unused_indexes_user_filters.py  | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/78/04_add_full_user_id_index_user_filters.py | 25 +++++++++++++++++++++++++
 2 files changed, 90 insertions(+)

v1.88.0: New Schema Version: 78
 synapse/storage/schema/main/delta/78/03event_extremities_constraints.py        |  57 ++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/78/04_read_write_locks_triggers.sql.postgres | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/78/04_read_write_locks_triggers.sql.sqlite   | 119 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 328 insertions(+)

v1.89.0: New Schema Version: 79
 synapse/storage/schema/main/delta/79/04_mitigate_stream_ordering_update_race.py | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/79/05_read_write_locks_triggers.sql.postgres  | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/79/05_read_write_locks_triggers.sql.sqlite    | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 204 insertions(+)

v1.90.0: New Schema Version: 80

Command: for i in $( seq 70 90 ); do echo -n "v1.$i.0: New Schema Version: "; git checkout -q "v1.$i.0"; python -c "from __init__ import SCHEMA_VERSION; print(SCHEMA_VERSION)"; git diff --stat --diff-filter=A v1.$(($i-1)).0..v1.$i.0 .; echo ""; git checkout -q ~/git/synapse; done

The changes to the schemas (only showing addition/deletions) in the v1.80 to v1.83 range are:

v1.80.0rc2: Schema: 75. Commit: e6af49fbea Reintroduce membership tables event stream ordering (#15128) — 2023-03-24 11:44:01 +0000
 synapse/storage/schema/main/delta/74/01membership_tables_event_stream_ordering.sql         | 20 ++++++++++++++++++++
 synapse/storage/schema/main/delta/74/02membership_tables_event_stream_ordering_triggers.py | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 99 insertions(+)

v1.80.0rc2: Schema: 75. Commit: 5b70f240cf Make cleaning up pushers depend on the device_id instead of the token_id (#15280) — 2023-03-24 11:09:39 -0400
 synapse/storage/schema/main/delta/74/02_set_device_id_for_pushers_bg_update.sql | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

v1.80.0: Schema: 74. Commit: 5350b5d04d Revert "Reintroduce membership tables event stream ordering (#15128)" (#15347) — 2023-03-29 13:24:28 +0100
 synapse/storage/schema/main/delta/74/01membership_tables_event_stream_ordering.sql         | 20 --------------------
 synapse/storage/schema/main/delta/74/02membership_tables_event_stream_ordering_triggers.py | 79 -------------------------------------------------------------------------------
 2 files changed, 99 deletions(-)

v1.80.0: Schema: 74. Commit: 2a234b788e Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350) — 2023-03-30 15:11:31 -0400
 synapse/storage/schema/main/delta/74/02thread_notifications_backfill.sql          | 28 ++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.postgres | 23 +++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.sqlite   | 99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 150 insertions(+)

v1.80.0: Schema: 74. Commit: 72d2ceaa9a Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350)" — 2023-03-31 12:10:10 +0100
 synapse/storage/schema/main/delta/74/02thread_notifications_backfill.sql          | 28 ----------------------------
 synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.postgres | 23 -----------------------
 synapse/storage/schema/main/delta/74/03thread_notifications_not_null.sql.sqlite   | 99 ---------------------------------------------------------------------------------------------------
 3 files changed, 150 deletions(-)

v1.80.0: Schema: 74. Commit: 79d2e2e79c Speed up membership queries for users with forgotten rooms (#15385) — 2023-04-04 14:11:34 +0100
 synapse/storage/schema/main/delta/74/03_room_membership_index.sql | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

v1.81.0rc1: Schema: 74. Commit: 6b23d74ad1 Delete server-side backup keys when deactivating an account. (#15181) — 2023-04-04 20:16:08 +0000
 synapse/storage/schema/main/delta/74/04_delete_e2e_backup_keys_for_deactivated_users.sql | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

v1.82.0rc1: Schema: 74. Commit: 8b3a502996 Experimental support for MSC3970: per-device transaction IDs (#15318) — 2023-04-25 09:37:09 +0100
 synapse/storage/schema/main/delta/74/05_events_txn_id_device_id.sql | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

v1.82.0rc1: Schema: 75. Commit: c55293c230 Re re introduce membership tables event stream ordering (#15356) — 2023-04-25 09:44:29 +0100
 synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.postgres | 29 +++++++++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/03_membership_tables_event_stream_ordering.sql.sqlite   | 23 +++++++++++++++++++++++
 synapse/storage/schema/main/delta/74/04_membership_tables_event_stream_ordering_triggers.py  | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 131 insertions(+)

v1.82.0: Schema: 76. Commit: 301b4156d5 Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458) — 2023-04-26 16:03:26 -0700
 synapse/storage/schema/main/delta/76/01_add_profiles_full_user_id_column.sql     | 20 ++++++++++++++++++++
 synapse/storage/schema/main/delta/76/02_add_user_filters_full_user_id_column.sql | 20 ++++++++++++++++++++
 2 files changed, 40 insertions(+)

v1.82.0: Schema: 76. Commit: 89f6fb0d5a Add an admin API endpoint to support per-user feature flags (#15344) — 2023-04-28 11:33:45 -0700
 synapse/storage/schema/main/delta/76/03_per_user_experimental_features.sql | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

Command: for c in e6af49fbea 5b70f240cf 5350b5d04d 2a234b788e 72d2ceaa9a 79d2e2e79c 6b23d74ad1 8b3a502996 c55293c230 301b4156d5 89f6fb0d5a; do git checkout -q $c; VERSION=$(git describe --tags --abbrev=0); SCHEMA=$( python -c "from __init__ import SCHEMA_VERSION; print(SCHEMA_VERSION)" ); date=$(git show -s --format=%ci); oneline=$(git show -s --oneline); echo "$VERSION: Schema: $SCHEMA. Commit: $oneline — $date"; git diff --stat --diff-filter=AD HEAD^ .; echo ""; git checkout -q ~/git/synapse; done

[Anthial]

As far as I am aware, I did not run the develop branch or any of the prereleases!

[erikjohnston]

I can't see how this can happen without running the develop branch somehow, though seems like it did. I'm wondering if there was some weird quirk in the setup here? Given no one else is complaining about similar problems, and I don't see a way of progressing this issue, I'm going to close it.

If anyone else sees this please shout!