Failed to take the backup -- Error: envconfig.Process: assigning CLICKHOUSE_CLICKHOUSE_PORT to Port: converting

[kishorer00]

Hi Alex,

Unable to list /create Clickhouse backups,

Below are details for your reference.

Backup tool version : 1.2.2

Running on Container image: On top of AWS EKS

Clickhouse version : x86_64 image)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Error Message : -

List :-

Server-1# ./clickhouse-backup -c config.yml list

2022/02/11 08:50:34 error envconfig.Process: assigning __CLICKHOUSE_PORT to Port: converting 'tcp://10.43.234.97:8123' to type uint. details: strconv.ParseUint: parsing "tcp://10.43.234.97:8123": invalid syntax

Create backup:-

Server-1# ./clickhouse-backup -c config.yml create 'backup-1'

2022/02/11 08:50:34 error envconfig.Process: assigning __CLICKHOUSE_PORT to Port: converting 'tcp://10.43.234.97:8123' to type uint. details: strconv.ParseUint: parsing "tcp://10.43.234.97:8123": invalid syntax
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I found a similar issue: #165 - Fixed

Using the latest backup tool version : [1.2.2] & default config.yml file - Not sure why the issue occures?

Please check to help me.

[Slach]

clickhouse-backup use TCP native protocol, you should use 9000 port instead of 8123
Moreover, you should understand clickhouse-backup should direct access to clickhouse-server data volumes to allow upload/download, it means you should run clickhouse-backup container in the same Pod with clickhouse-server
look to https://github.com/AlexAkulov/clickhouse-backup/blob/master/Examples.md#how-to-use-clickhouse-backup-in-kubernetes for details

[kishorer00]

Hi Slach,

Using port 9000 only for TCP communication.

+++++++++++++++++++

• name: tcp
  nodePort: 30040 (port exported to outside)
  port: 9000
  protocol: TCP
  targetPort: 9000
  +++++++++++++++

Is there way to check the error with debug option?

How can we test TCP communicating through port 9000. ?

[Slach]

What the part of ports specification you shared?

could you provide your kubernetes manifests related to clickhouse-backup?

Do you use clickhouse-server managed with clickhouse-operator or something other way to manage clickhouse?

[kishorer00]

HI Slach ,

Thanks for checking on this ...

--> Running Clickhouse on Kubernetes Statefullset. Not using Operator.

YAML used for deployment:-

---
apiVersion: v1
kind: Service
metadata:
  name: clickhouse-test
  namespace: clickhouse-test
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: clickhouse-test
  ports:
  - name: http
    port: 8123
    targetPort: 8123
  - name: tcp
    port: 9000
    targetPort: 9000
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: clickhouse-test
  labels:
    app: clickhouse-test
spec:
  serviceName: clickhouse-test
  selector:
    matchLabels:
      app: clickhouse-test
  template:
    metadata:
      labels:
        app: clickhouse-test
    spec:
      containers:
      - name: clickhouse-test
        image: docker pull yandex/clickhouse-server:latest
        imagePullPolicy: Always
        ports:
        - name: http
          containerPort: 8123
        - name: tcp
          containerPort: 9000
        volumeMounts:
        - name: backuptest
          mountPath: /var/lib/clickhouse

• In addition to that , I have updated the configs from ClusterIP to NodePort.

• Also, using ingress

      #kubectl get svc -n test 
      Name 		                TYPE 		EXTERNAL-IP		PORT(S)
      clickhouse-test 		NodePort	        10.20.71.232	8123:30278/TCP,9000:30450/TCP
  

[Slach]

You shall add second container to your kind: StatefulSet with image: alexakulov/clickhouse-backup:latest
you also shall mount /var/lib/clickhouse inside this container

[kishorer00]

Hi Slach,

Got help from one of the Developer and was able to figure out the issue.

The Environment variable set on Clickhouse Server was wrong .

+++++++++++++++++++++++++++++++++
Server-1#env
CLICKHOUSE_PORT=tcp://172.20.73.248:8123
+++++++++++++++++++++++++++++++++

After removing the variable, the Backup code is using default port 9000 - Able to work on both Backups and Restore.

Thanks for your support !!!!. Slach .

[1Const1]

for those who using Altinity clickhouse operator (even last version) recommend to add to ENV for container backup

              command:
                - bash
                - -xc
                - "/bin/clickhouse-backup server"
               env:
                  - name: CLICKHOUSE_PORT
                     value: "9000"       

because something wrong with env in k8s and env from clean docker container (image docker official clickhouse)
CLICKHOUSE_PORT=9000

become looks like in operator container CLICKHOUSE_PORT=tcp://172.20.73.248:8123 in k8s

i have no idea why

[Slach]

always better to add env explicitly to container to avoid conflicts with other container in the same pod
it not depends on clickhouse-operator

but 9000 is default value

@1Const1 which image did you use for clickhouse-backup container?

Where did you see CLICKHOUSE_PORT=tcp://172.20.73.248:8123 ? in clickhouse-backup container without env section?

[1Const1]

always better to add env explicitly to container to avoid conflicts with other container in the same pod it not depends on clickhouse-operator

but 9000 is default value

@1Const1 which image did you use for clickhouse-backup container?

Where did you see CLICKHOUSE_PORT=tcp://172.20.73.248:8123 ? in clickhouse-backup container without env section?

for backup container
altinity/clickhouse-backup:2.4.34

for clickhouse
yandex/clickhouse-server:24.2.2.71

i'm not sure what gives this effect because if i created deployment by myself inside pod env i will see correct env val CLICKHOUSE_PORT=9000

but inside clickhouse-backup or clickhouse containers by operator i will se something like this CLICKHOUSE_PORT=tcp://172.20.73.248:8123 but in my case only subnet will be internal like 10.250.78.56

[Slach]

@1Const1

for clickhouse yandex/clickhouse-server:24.2.2.71

https://hub.docker.com/r/yandex/clickhouse-server/tags?page=1&name=24.2
do you mean clickhouse/clickhouse-server:24.2.2.71?

could you share?

kubectl get chi --all-namespaces