Plugin log entries

[Twixes]

Changes

This is a proposed the accepted and awaiting complete implementation in this PR (and a complimentary one in https://github.com/PostHog/plugin-server) data model to resolve PostHog/plugin-server#72.
The unit of plugin logging is a single plugin log entry (PluginLogEntry model for Postgres and a schema-wise drop-in equivalent plugin_log_entries topic → kafka_plugin_log_entries table → plugin_log_entries table setup in Kafka+ClickHouse).

This looks like some complexity at first, but it's straightforward compared to fighting Postgres's scale limitations:

• For FOSS users, the solution is the simplest possible plain INSERTs/SELECTs on posthog_pluginlogentry, and we know they won't have problems with scale.
• For Cloud/EE, we need to support LARGE scale with minimum maintenance and high data integrity. For this part Kafka+ClickHouse is unbeatable within our existing infrastructure. And also very simple "produce to topic on every console method call in plugin server, SELECT relevant entries from table in Django server".

Kafka+ClickHouse parts adapted from session recording events handling, which is similarly simple (though actually logs are far better for this infrastructure than session recording, which is too storage space-intensive).

[mariusandra]

I think we can go in this direction. The only thing we'll definitely need to add then is some decent TTL restrictions in clickhouse (30 days?) and a script to delete old rows from postgres, like we have now for session recordings. The size of this table should probably also be added to the system status page.

[mariusandra]

Also, into the models, I'd add another field for "instance_id" - a random ID per rebooted plugin server. This will help read the logs if there are multiple plugins talking at the same time.

[Twixes]

Great, I'll add the rest of this solution after putting #3486 up for review.

[Twixes]

Added support for system (plugin server) logs too (PostHog/plugin-server#333):

[Twixes]

This is ready. @macobo After the migration is ran, the logs won't be produced yet – they have to be enabled with env var ENABLE_PERSISTENT_CONSOLE in the plugins task def. After that we'll be able to remove that env var and enable this feature by default in another PR.

[mariusandra]

@macobo @fuziontech can we do anything from our side to help move this along? Doing things manually once would also be fine perhaps.... though, yeah, VPCs...

[macobo]

Sorry for the delays, have had my own fires to fight. Will get this in soon! :)