Rework console extension to send data to Kafka>ClickHouse or Postgres

[Twixes]

Changes

Plugin server changes for PostHog/posthog#3482.

Checklist

• Jest tests

[yakkomajuri]

Sincerely naive question, but how come not just use Postgres for this given we still store data there for both EE and FOSS? Is it a matter of future-proofing and keeping as much of the new additions specific to one db?

[Twixes]

Given the potentially high data volume nature of this feature, ClickHouse is definitely more future-proof and only marginally more complicated. @yakkomajuri

[yakkomajuri]

Tests are still expecting stuff like console.log to be called. Even though we're piping logs somewhere else, we could still call the native console methods, no?

Either way, if the answer is no, I guess tests still have to be updated

[Twixes]

Yep, adjusted for that @yakkomajuri. FYI in-plugin console method calls are now status'd to stdout in dev mode (DEBUG truthy or NODE_ENV is dev).

[Twixes]

PR reviewable.
This functionality (including its tests) is completely disabled by default currently though, configurable with setting ENABLE_PERSISTENT_CONSOLE. That's because we should first merge this to make PostHog/posthog#3482 work end-to-end, then merge PostHog/posthog#3482, and then remove ENABLE_PERSISTENT_CONSOLE when the functionality can be put into production (mind that PostHog Cloud will require a migration to be manually initiated for this to work, Infra teams knows about this).

Top green check is the end goal state - persistent console fully tested and enabled by default. That ran with PostHog/posthog#3482 as the main repo base though, so until that is merged into main repo master, the version merged will have some if (!server.ENABLE_PERSISTENT_CONSOLE) return conditions – bottom green check.

[mariusandra]

LGTM. Merge whenever convenient.