Backups error

[davordragic]

Bug description

Added clickhouse backups, but backup container returns errors
Code: 60. DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table system.backup_actions doesn't exist. (UNKNOWN_TABLE)

Expected behavior

backups should work

Actual behavior

Backup container returns error missing table

How to reproduce

1. Add backup.enabled: true
2. wait for cron

Environment

updated posthog and helm chart

• Deployment platform (AWS/Azure/DigitalOcean/GCP/...): GCP
• k8s version (server version from kubectl version): v1.23.11
• Chart version/commit (helm list -n posthog): posthog-29.0.11
• Posthog version: 1.41.4

Additional context

clickhouse:
....
backup:
enabled: true
env:
- name: BACKUPS_TO_KEEP_LOCAL
value: "10"
- name: BACKUPS_TO_KEEP_REMOTE
value: "30"
- name: REMOTE_STORAGE
value: "gcs"
- name: GCS_CREDENTIALS_JSON_ENCODED
value: "hidden"
- name: GCS_BUCKET
value: "posthog-k8s-backup"
- name: GCS_PATH
value: ""

[davordragic]

Found some ideas in backup repo, but this helm chart already defines CLICKHOUSE_HOST
Suggested solution is to set for backup clickhouse_host=127.0.0.1
I might "override" with clickhouse.backup.env, but which will stick, one in my values.yaml or from template?

• name: CLICKHOUSE_HOST
  value: {{ template "posthog.clickhouse.fullname" . }}

{{/*
Return clickhouse fullname
*/}}
{{- define "posthog.clickhouse.fullname" -}}
{{- if .Values.clickhouse.fullnameOverride -}}
{{- .Values.clickhouse.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" "clickhouse" .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}

Full error:

2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,"(query: SELECT status,error FROM system.backup_actions WHERE command='create chi-posthog-posthog-0-0-full-2022-12-06-00-00-07')",2022-12-06T00:00:08.010052183Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Code: 60. DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table system.backup_actions doesn't exist. (UNKNOWN_TABLE),2022-12-06T00:00:08.010049713Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Received exception from server (version 22.3.13):,2022-12-06T00:00:08.010044266Z
2022-12-06T00:00:13.397274353Z,k8s_container,INFO,"SELECT status,error FROM system.backup_actions WHERE command='create chi-posthog-posthog-0-0-full-2022-12-06-00-00-07'",
2022-12-06T00:00:13.397274353Z,k8s_container,INFO,error create full-2022-12-06-00-00-07 on chi-posthog-posthog-0-0,2022-12-06T00:00:07.987132452Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,(query: SELECT status FROM system.backup_actions WHERE command='create chi-posthog-posthog-0-0-full-2022-12-06-00-00-07'),2022-12-06T00:00:07.983222708Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Code: 60. DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table system.backup_actions doesn't exist. (UNKNOWN_TABLE),2022-12-06T00:00:07.983220114Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Received exception from server (version 22.3.13):,2022-12-06T00:00:07.983211982Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,(query: SELECT status FROM system.backup_actions WHERE command='create chi-posthog-posthog-0-0-full-2022-12-06-00-00-07'),2022-12-06T00:00:07.956158568Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Code: 60. DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table system.backup_actions doesn't exist. (UNKNOWN_TABLE),2022-12-06T00:00:07.956155400Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Received exception from server (version 22.3.13):,2022-12-06T00:00:07.956141033Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,(query: INSERT INTO system.backup_actions(command) VALUES('create chi-posthog-posthog-0-0-full-2022-12-06-00-00-07')),2022-12-06T00:00:07.928767960Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Code: 60. DB::Exception: Received from chi-posthog-posthog-0-0:9000. DB::Exception: Table system.backup_actions doesn't exist. (UNKNOWN_TABLE),2022-12-06T00:00:07.928764402Z
2022-12-06T00:00:13.393621223Z,k8s_container,ERROR,Received exception from server (version 22.3.13):,2022-12-06T00:00:07.928755742Z
2022-12-06T00:00:13.397274353Z,k8s_container,INFO,INSERT INTO system.backup_actions(command) VALUES('create chi-posthog-posthog-0-0-full-2022-12-06-00-00-07'),2022-12-06T00:00:07.927824877Z
2022-12-06T00:00:13.397274353Z,k8s_container,INFO,create full-2022-12-06-00-00-07 on chi-posthog-posthog-0-0,2022-12-06T00:00:07.904611353Z