MSC1763 retention purge_jobs fail to run when run_background_tasks_on is configured to be a synapse.app.generic_worker

[bradtgmurray]

Description

m.room.retention state events are correctly filtering out events in the endpoints, but the purge jobs aren't correctly running and the events are still in the database.

Seeing this error log which points pretty clearly at the issue
{"log":"[purge] failed","namespace":"synapse.handlers.pagination","level":"ERROR","time":1661238458.09,"request":"_purge_history-0","server_name":"beeper.com","exc_type":"AttributeError","exc_value":"'GenericWorkerSlavedStore' object has no attribute 'purge_history'"}

Can we just add PurgeEventsStore to GenericWorkerSlavedStore or can only the main process purge events? 🤔

Steps to reproduce

Relevant config:

run_background_tasks_on: synapse-background-worker-0

retention:
  enabled: true

so it's just the default daily job.

synapse-background-worker-0 is the hostname of a synapse.app.generic_worker process configured to run background jobs.

Homeserver

beeper.com

Synapse Version

1.64.0 (Beeper fork)

Installation Method

No response

Platform

Running in containers on Kubernetes.

Relevant log output

{"log":"[purge] failed","namespace":"synapse.handlers.pagination","level":"ERROR","time":1661238458.33,"request":"_purge_history-7","server_name":"beeper.com","exc_type":"AttributeError","exc_value":"'GenericWorkerSlavedStore' object has no attribute 'purge_history'"}
{"log":"[purge] Attempting to purge messages in room !XPxwZVXpPnNWmrJQwB:beeper.com","namespace":"synapse.handlers.pagination","level":"INFO","time":1661238458.31,"request":"purge_history_for_rooms_in_range-0","server_name":"beeper.com"}

### Anything else that would be useful to know?

_No response_

[DMRobertson]

Can we just add PurgeEventsStore to GenericWorkerSlavedStore or can only the main process purge events?

Difficult to say. There's wording here which makes it sound like there should only be one purge running at a time across the entire application (all workers).

synapse/synapse/storage/databases/main/purge_events.py

Lines 35 to 36 in 2ee0b6e

	Note that only a single purge can occur at once, this is guaranteed via
	a higher level (in the PaginationHandler).

Naively: I'd be scared of deleting on events from workers that aren't event persisters. Can't give you an authoritative answer, though; any ideas from @matrix-org/synapse-core?

[erikjohnston]

I think the quick option here is to change the purge background job to ignore run_background_tasks_on config option and to always run on master (which is what it will do by default)

[clokep]

Seems like #8513 might have regressed this?

[clokep]

This seems to be a duplicate of #9927!