On latest develop, new events don't come down sync anymore

[babolivier]

When running the develop branch on abolivier.bzh, new events are processed and store in the database but don't come down sync anymore. On top of that, I'm seeing a bunch of

2020-09-16 11:59:16,431 - synapse.metrics.background_process_metrics - 210 - ERROR - user_directory.notify_new_event-5- Background process 'user_directory.notify_new_event' threw an exception
Traceback (most recent call last):
  File "/home/synapse/synapse/synapse/metrics/background_process_metrics.py", line 205, in run
    result = await result
  File "/home/synapse/synapse/synapse/handlers/user_directory.py", line 103, in process
    await self._unsafe_process()
  File "/home/synapse/synapse/synapse/handlers/user_directory.py", line 150, in _unsafe_process
    self.pos, room_max_stream_ordering
  File "/home/synapse/synapse/synapse/storage/databases/main/state_deltas.py", line 54, in get_current_state_deltas
    assert prev_stream_id <= max_stream_id
AssertionError
2020-09-16 11:59:16,432 - synapse.metrics.background_process_metrics - 210 - ERROR - stats.notify_new_event-5- Background process 'stats.notify_new_event' threw an exception
Traceback (most recent call last):
  File "/home/synapse/synapse/synapse/metrics/background_process_metrics.py", line 205, in run
    result = await result
  File "/home/synapse/synapse/synapse/handlers/stats.py", line 69, in process
    await self._unsafe_process()
  File "/home/synapse/synapse/synapse/handlers/stats.py", line 94, in _unsafe_process
    self.pos, room_max_stream_ordering
  File "/home/synapse/synapse/synapse/storage/databases/main/state_deltas.py", line 54, in get_current_state_deltas
    assert prev_stream_id <= max_stream_id
AssertionError

on startup.

FWIW I don't see any error in the logs when processing the event or when syncing.

It doesn't look like my HS has this issue with the v1.20.0 release branch.

[erikjohnston]

This happens due to:

1. We merged a PR that changed stream IDs to be generated by a sequence
2. We reverted that PR
3. We re-merged that PR, without recreating the sequence.

This means that between steps 2 and 3 the server was assigning new stream IDs without updating the sequence, so when the PR was merged again the sequence was out of sync.

One fix is to check on startup that the sequence is above the current max position.

[erikjohnston]

This should be "fixed" by: #8402, in the sense it'll error and give you a SQL statement to run