ERROR 1064 (HY000): Access denied for user 'root' (using password: NO) || BE constantly has restart after a few hours #275

jimenez-f · 2023-10-04T17:57:57Z

I have an issue with Starrocks Helmchart v1.8.3 when I set a password in user root via mysql-client the BE apparently lost the connection via password and constantly has restart the pods of BE with this message:

ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:47 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:49 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:51 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:53 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:55 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:57 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:37:59 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:38:02 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:38:04 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:38:06 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:38:08 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:38:10 America 2023] Add myself (kube-starrocks-be-1.kube-starrocks-be-search.starrocks.svc.cluster.local:9050) into FE ...ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)ERROR 1064 (HY000): Access denied for user 'root' (using password: NO)[Wed Oct 4 17:38:10 America 2023] Time out, abort!
I asked about this issue into the starrocks channel and I followed the solution that gave me: https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/change_root_password_howto.md but the issue persist.

Could you please help me with this issue?

Starrocks Helmchart Version: 1.8.3
Tanzu Kubernetes Version: 1.21

The text was updated successfully, but these errors were encountered:

yandongxiao · 2023-10-05T02:22:21Z

Can you post the contents of values.yaml?

The document https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/change_root_password_howto.md describes how to set MYSQL_PWD in StarRocksCluster CRD. If you deploy StarRocks with Helm, you should set it in values.yaml.

yandongxiao · 2023-10-05T02:27:48Z

If you install StarRocks with kube-starrocks chart, you can follow:

starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/values.yaml

Line 163 in 5a7c3af

feEnvVars: []

This describes how to set FE environment.

starrocks-kubernetes-operator/helm-charts/charts/kube-starrocks/values.yaml

Line 440 in 5a7c3af

beEnvVars: []

This describes how to set FE environment.

jimenez-f · 2023-10-05T13:46:51Z

Thanks a lot for you answer.

I've tried this configuration and it is the one that is giving me this type of problem. It is the same as the documentation I have passed previously.

yandongxiao · 2023-10-05T13:54:15Z

Please also check that the MYSQL_PWD environment in BE pod is set.

jimenez-f · 2023-10-05T14:48:22Z

I've seen MYSQL_PWD environmentin FE and BE is set with this command: kubectl exec <podName> -- sh -c 'echo $MYSQL_PWD' but the issue persist.

Currently I've executed a deploy from scratch with InitPassword and works well but the issue now it's that refused the external connection via msql client

jimenez-f · 2023-10-06T21:06:17Z

Do you have any idea about the last issue?

Currently I've executed a deploy from scratch with InitPassword and works well but the issue now it's that refused the external connection via msql client

kevincai · 2023-10-07T01:53:48Z

one thing to confirm first, if you create a brand-new cluster with no init password, (leave the root password empty), will you be able to connect to the sr cluster from your local vms?

jimenez-f · 2023-10-09T14:42:29Z

This is affirmative, the problem occurs when we assign the root password via mysql client. After a few hours the BE cannot connect with the FE. This is the first problem I reported above following this documentation: https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/change_root_password_howto.md

The second problem is: with InitPassword the BE and FE work well, but it does not allow you to connect at any time through mysql client and neither does any other connection using Apache services.

Extra details: we are working on Tanzu Kubernetes Server Version: v1.21.6+vmware.1

kevincai · 2023-10-11T03:19:17Z

would you please try following steps to narrow down the issue

get a brand-new env, (that is: clean all previous installation, crd/helmcharts)
follow this doc to do a fresh install with initial password: https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/initialize_root_password_howto.md
check if the cluster can be deployed successful (both FE and BE)
manual delete one of the BE pod, check if it can be started and running correctly (in ready status)
check if can access it with mysql-client to the FE service from outside the outside kubernetes env.

jimenez-f · 2023-10-11T19:41:07Z

I've made a new deployment in a different cluster of Kubernetes but the same version: Tanzu Kubernetes Server Version: v1.21.6+vmware.1

The test was successful and also I've tested the expose via configmap of nginx ingress controller and works well.

Maybe the issue persist in the other cluster. I'll try to replicate this.

Many thanks.

jimenez-f · 2023-10-12T14:53:51Z

I've tested the two ways to apply the credential password for root.The first way to apply the password root works well, but is a insecurity manner to set a password.

The second way doesn't work. I guess that the issue is into charts/starrocks/templates/ini-pwd/

job.yaml
configmap.yaml

in the last one it's possible that the bash script doesn't apply in the correct way and generating this issue.Please let me know if you found a solution for the second way.

kevincai · 2023-10-12T23:17:57Z

by saying first and second method, can you elaborate exactly what they are, and possibly doc link? and how did you do that for the second one which it doesn't work?

jimenez-f · 2023-10-17T17:22:36Z

I followed this guide:

https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/initialize_root_password_howto.md

I did it the same as in documentation.

kevincai · 2023-10-17T23:10:53Z

It only works when doing helm install in the first place, helm upgrade won't trigger the password change.

jimenez-f · 2023-10-18T13:24:02Z

Yes, we know that it only works on the first installation.

What I mean is that I delete the deployment and launch it again from scratch. The second way using a kubernetes secret, although starrocks with FE and BE work well, it does not allow us to access it from the outside, that is the problem, and I have tested this difference in a new kubernetes cluster many times, which I believe that the issue must be in the mount of the script that is made in the configmap. Since the first way always works well and the second does not.

{{- if .Values.initPassword.enabled }}
{{- if not .Values.initPassword.passwordSecret }}
apiVersion: v1
kind: Secret
metadata:
name: {{ template "starrockscluster.initpassword.secret.name"
namespace: {{ template "starrockscluster.namespace" . }}
data:
password: {{ .Values.initPassword.password | b64enc }}

{{- end }}
apiVersion: v1
kind: ConfigMap
metadata:
name: init-pwd-shell
namespace: {{ template "starrockscluster.namespace" . }}
data:
fe_initpwd.sh: |-
#!/bin/bash

# ensure `mysql` command can be ended with 30 seconds
# Change the root password during initial installation
init_pwd_sql="SET PASSWORD = PASSWORD('$INIT_PWD');"
success=`mysql --connect-timeout 30 -h $1 -P $2 -u root --s
if [ $? -ne 0 ] ; then
  echo $success
  errcode=`echo $success | awk -F " " '{print $2}'`
  echo "error code: $errcode"

  # Password error, believed to have been changed, exiting 
  if [[ $errcode = '1045' || $errcode = '1064' ]] ; then
    echo "Password error, believed to have been changed, ex
    exit 0
  fi

  # Other unsuccessful errors, abnormal exit
  echo "Other unsuccessful errors, abnormal exit"
  exit 1

fi

echo "Successfully modified password"
exit 0

kevincai · 2023-10-18T13:30:28Z

can you do a kubectl get job starrocks-initpwd -o yaml, and paste the output here?

jimenez-f · 2023-10-18T15:53:33Z

`
apiVersion: batch/v1
kind: Job
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"batch/v1","kind":"Job","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"starrocks"},"name":"starrocks-initpwd","namespace":"starrocks"},"spec":{"backoffLimit":10,"template":{"spec":{"containers":[{"args":["/opt/starrocks/fe_initpwd.sh","kube-starrocks-fe-0.kube-starrocks-fe-search","9030"],"command":["/bin/bash"],"env":[{"name":"INIT_PWD","valueFrom":{"secretKeyRef":{"key":"password","name":"kube-starrocks-credential"}}}],"image":"starrocks/fe-ubuntu:3.1-latest","imagePullPolicy":"Always","name":"kube-starrocks-initpwd","volumeMounts":[{"mountPath":"/opt/starrocks/fe_initpwd.sh","name":"fe-initpwd-shell","subPath":"fe_initpwd.sh"}]}],"restartPolicy":"OnFailure","volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"fe_initpwd.sh","path":"fe_initpwd.sh"}],"name":"init-pwd-shell","optional":false},"name":"fe-initpwd-shell"}]}}}}
creationTimestamp: "2023-10-18T15:50:42Z"
labels:
argocd.argoproj.io/instance: starrocks
managedFields:

apiVersion: batch/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:argocd.argoproj.io/instance: {}
f:spec:
f:backoffLimit: {}
f:completions: {}
f:parallelism: {}
f:template:
f:spec:
f:containers:
k:{"name":"kube-starrocks-initpwd"}:
.: {}
f:args: {}
f:command: {}
f:env:
.: {}
k:{"name":"INIT_PWD"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:secretKeyRef:
.: {}
f:key: {}
f:name: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/opt/starrocks/fe_initpwd.sh"}:
.: {}
f:mountPath: {}
f:name: {}
f:subPath: {}
f:dnsPolicy: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
f:volumes:
.: {}
k:{"name":"fe-initpwd-shell"}:
.: {}
f:configMap:
.: {}
f:defaultMode: {}
f:items: {}
f:name: {}
f:optional: {}
f:name: {}
manager: argocd-controller
operation: Update
time: "2023-10-18T15:50:42Z"
apiVersion: batch/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:active: {}
f:startTime: {}
manager: kube-controller-manager
operation: Update
time: "2023-10-18T15:50:43Z"
name: starrocks-initpwd
namespace: starrocks
resourceVersion: "4128667"
uid: a6d347d3-b3ff-491c-99fd-a7f6995a109e
spec:
backoffLimit: 10
completions: 1
parallelism: 1
selector:
matchLabels:
controller-uid: a6d347d3-b3ff-491c-99fd-a7f6995a109e
template:
metadata:
creationTimestamp: null
labels:
controller-uid: a6d347d3-b3ff-491c-99fd-a7f6995a109e
job-name: starrocks-initpwd
spec:
containers:
- args:
  - /opt/starrocks/fe_initpwd.sh
  - kube-starrocks-fe-0.kube-starrocks-fe-search
  - "9030"
    command:
  - /bin/bash
    env:
  - name: INIT_PWD
    valueFrom:
    secretKeyRef:
    key: password
    name: kube-starrocks-credential
    image: starrocks/fe-ubuntu:3.1-latest
    imagePullPolicy: Always
    name: kube-starrocks-initpwd
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
  - mountPath: /opt/starrocks/fe_initpwd.sh
    name: fe-initpwd-shell
    subPath: fe_initpwd.sh
    dnsPolicy: ClusterFirst
    restartPolicy: OnFailure
    schedulerName: default-scheduler
    securityContext: {}
    terminationGracePeriodSeconds: 30
    volumes:
- configMap:
  defaultMode: 420
  items:
  - key: fe_initpwd.sh
    path: fe_initpwd.sh
    name: init-pwd-shell
    optional: false
    name: fe-initpwd-shell
    status:
    active: 1
    startTime: "2023-10-18T15:50:42Z"
    `

kevincai · 2023-10-18T23:25:52Z

what's your mysql client version?

cheny1994 · 2023-10-21T07:35:35Z

same problem,After I deployed the fe and cn nodes and set the root user password, the cn node will lose the connection.

adhasahar97 · 2023-10-27T08:14:18Z

containers: args: /opt/starrocks/fe_initpwd.sh kube-starrocks-fe-0.kube-starrocks-fe-search

The kube-starrocks-fe-0.kube-starrocks-fe-search might be the problem.

My FE hostname is starrocks-sc-cluster-fe-search.starrocks

kevincai · 2023-10-27T08:17:02Z

can you describe the statefulset the fe belongs to?

@adhasahar97

adhasahar97 · 2023-10-27T08:20:20Z

apiVersion: apps/v1 kind: StatefulSet metadata: name: starrocks-sc-cluster-fe namespace: starrocks uid: 5de25963-fd43-4ec1-8b6d-8faea9a54ea3 resourceVersion: '32071531' generation: 1 creationTimestamp: '2023-10-27T07:52:16Z' labels: app.kubernetes.io/component: fe app.starrocks.ownerreference/name: starrocks-sc-cluster annotations: app.starrocks.components/hash: '821581395' ownerReferences: - apiVersion: starrocks.com/v1 kind: StarRocksCluster name: starrocks-sc-cluster uid: 75f3f695-fc47-4f3a-843f-5d164e8c32e4 controller: true blockOwnerDeletion: true managedFields: - manager: sroperator operation: Update apiVersion: apps/v1 time: '2023-10-27T07:52:16Z' fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:app.starrocks.components/hash: {} f:labels: .: {} f:app.kubernetes.io/component: {} f:app.starrocks.ownerreference/name: {} f:ownerReferences: .: {} k:{"uid":"75f3f695-fc47-4f3a-843f-5d164e8c32e4"}: {} f:spec: f:persistentVolumeClaimRetentionPolicy: .: {} f:whenDeleted: {} f:whenScaled: {} f:podManagementPolicy: {} f:replicas: {} f:revisionHistoryLimit: {} f:selector: {} f:serviceName: {} f:template: f:metadata: f:annotations: .: {} f:app.starrocks.io/fe-config-hash: {} f:labels: .: {} f:app.kubernetes.io/component: {} f:app.starrocks.ownerreference/name: {} f:name: {} f:namespace: {} f:spec: f:automountServiceAccountToken: {} f:containers: k:{"name":"fe"}: .: {} f:args: {} f:command: {} f:env: .: {} k:{"name":"COMPONENT_NAME"}: .: {} f:name: {} f:value: {} k:{"name":"CONFIGMAP_MOUNT_PATH"}: .: {} f:name: {} f:value: {} k:{"name":"FE_SERVICE_NAME"}: .: {} f:name: {} f:value: {} k:{"name":"HOST_IP"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} k:{"name":"HOST_TYPE"}: .: {} f:name: {} f:value: {} k:{"name":"MYSQL_PWD"}: .: {} f:name: {} f:valueFrom: .: {} f:secretKeyRef: {} k:{"name":"POD_IP"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} k:{"name":"POD_NAME"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} k:{"name":"POD_NAMESPACE"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} k:{"name":"TZ"}: .: {} f:name: {} f:value: {} f:image: {} f:imagePullPolicy: {} f:lifecycle: .: {} f:preStop: .: {} f:exec: .: {} f:command: {} f:livenessProbe: .: {} f:failureThreshold: {} f:httpGet: .: {} f:path: {} f:port: {} f:scheme: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:name: {} f:ports: .: {} k:{"containerPort":8030,"protocol":"TCP"}: .: {} f:containerPort: {} f:name: {} f:protocol: {} k:{"containerPort":9020,"protocol":"TCP"}: .: {} f:containerPort: {} f:name: {} f:protocol: {} k:{"containerPort":9030,"protocol":"TCP"}: .: {} f:containerPort: {} f:name: {} f:protocol: {} f:readinessProbe: .: {} f:failureThreshold: {} f:httpGet: .: {} f:path: {} f:port: {} f:scheme: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:resources: .: {} f:limits: .: {} f:cpu: {} f:memory: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:allowPrivilegeEscalation: {} f:readOnlyRootFilesystem: {} f:startupProbe: .: {} f:failureThreshold: {} f:httpGet: .: {} f:path: {} f:port: {} f:scheme: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/etc/starrocks/fe/conf"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/opt/starrocks/fe/log"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/opt/starrocks/fe/meta"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:nodeSelector: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: .: {} f:fsGroupChangePolicy: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"fe-log"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"fe-meta"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"kube-starrocks-fe-cm"}: .: {} f:configMap: .: {} f:defaultMode: {} f:name: {} f:name: {} f:updateStrategy: f:rollingUpdate: .: {} f:partition: {} f:type: {} - manager: kube-controller-manager operation: Update apiVersion: apps/v1 time: '2023-10-27T07:54:39Z' fieldsType: FieldsV1 fieldsV1: f:status: f:availableReplicas: {} f:collisionCount: {} f:currentReplicas: {} f:currentRevision: {} f:observedGeneration: {} f:readyReplicas: {} f:replicas: {} f:updateRevision: {} f:updatedReplicas: {} subresource: status selfLink: /apis/apps/v1/namespaces/starrocks/statefulsets/starrocks-sc-cluster-fe status: observedGeneration: 1 replicas: 1 readyReplicas: 1 currentReplicas: 1 updatedReplicas: 1 currentRevision: starrocks-sc-cluster-fe-775dbf9b58 updateRevision: starrocks-sc-cluster-fe-775dbf9b58 collisionCount: 0 availableReplicas: 1 spec: replicas: 1 selector: matchLabels: app.kubernetes.io/component: fe app.starrocks.ownerreference/name: starrocks-sc-cluster-fe template: metadata: name: starrocks-sc-cluster-fe namespace: starrocks creationTimestamp: null labels: app.kubernetes.io/component: fe app.starrocks.ownerreference/name: starrocks-sc-cluster-fe annotations: app.starrocks.io/fe-config-hash: f924ded4 spec: volumes: - name: fe-meta emptyDir: {} - name: fe-log emptyDir: {} - name: kube-starrocks-fe-cm configMap: name: kube-starrocks-fe-cm defaultMode: 420 containers: - name: fe image: starrocks/fe-ubuntu:3.1-latest command: - /opt/starrocks/fe_entrypoint.sh args: - $(FE_SERVICE_NAME) ports: - name: http-port containerPort: 8030 protocol: TCP - name: rpc-port containerPort: 9020 protocol: TCP - name: query-port containerPort: 9030 protocol: TCP env: - name: TZ value: Asia/Kuala_Lumpur - name: MYSQL_PWD valueFrom: secretKeyRef: name: kube-starrocks-credential key: password - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.podIP - name: HOST_IP valueFrom: fieldRef: apiVersion: v1 fieldPath: status.hostIP - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: HOST_TYPE value: FQDN - name: COMPONENT_NAME value: fe - name: FE_SERVICE_NAME value: starrocks-sc-cluster-fe-service.starrocks - name: CONFIGMAP_MOUNT_PATH value: /etc/starrocks/fe/conf resources: limits: cpu: '32' memory: 16Gi requests: cpu: '4' memory: 4Gi volumeMounts: - name: fe-meta mountPath: /opt/starrocks/fe/meta - name: fe-log mountPath: /opt/starrocks/fe/log - name: kube-starrocks-fe-cm mountPath: /etc/starrocks/fe/conf livenessProbe: httpGet: path: /api/health port: 8030 scheme: HTTP timeoutSeconds: 1 periodSeconds: 5 successThreshold: 1 failureThreshold: 3 readinessProbe: httpGet: path: /api/health port: 8030 scheme: HTTP timeoutSeconds: 1 periodSeconds: 5 successThreshold: 1 failureThreshold: 3 startupProbe: httpGet: path: /api/health port: 8030 scheme: HTTP timeoutSeconds: 1 periodSeconds: 5 successThreshold: 1 failureThreshold: 60 lifecycle: preStop: exec: command: - /opt/starrocks/fe_prestop.sh terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: IfNotPresent securityContext: readOnlyRootFilesystem: false allowPrivilegeEscalation: false restartPolicy: Always terminationGracePeriodSeconds: 120 dnsPolicy: ClusterFirst nodeSelector: starrocks: owned automountServiceAccountToken: false securityContext: fsGroupChangePolicy: OnRootMismatch schedulerName: default-scheduler serviceName: starrocks-sc-cluster-fe-search podManagementPolicy: Parallel updateStrategy: type: RollingUpdate rollingUpdate: partition: 0 revisionHistoryLimit: 10 persistentVolumeClaimRetentionPolicy: whenDeleted: Retain whenScaled: Retain

kevincai · 2023-10-27T08:29:42Z

interesting, can you paste the result of kubectl get service -n starrocks

and also your cr yaml

adhasahar97 · 2023-10-27T08:33:11Z

I think I found the issue. When the value starrocks.starrocksCluster.name is set, the service hostname will follows while the hostname inside the initpwd still using the chart name. @kevincai

kevincai · 2023-10-27T08:38:13Z

@yandongxiao can you take a look?

adhasahar97 · 2023-10-27T08:38:45Z

If starrocks.initPassword.enabled is true, and the value of starrocks.starrocksCluster.name is set, it will cause an issue.

kevincai · 2023-10-27T08:39:35Z

the job will never finished?

adhasahar97 · 2023-10-27T08:43:47Z

the job will never finished?

Correct. And the BE can't join the FE.

kevincai · 2023-10-27T08:46:30Z

will have a quick hotfix for this.

kevincai · 2023-10-27T08:46:50Z

great thx! @adhasahar97

kevincai · 2023-10-27T09:03:24Z

If starrocks.initPassword.enabled is true, and the value of starrocks.starrocksCluster.name is set, it will cause an issue.

@jimenez-f can you confirm if this matches your case?

yandongxiao · 2023-10-27T09:55:07Z

#292 @adhasahar97 @kevincai

The issue is caused by mixing the usage of kube-starrocks.name and starrockscluster.name.

yandongxiao mentioned this issue Oct 27, 2023

[Bugfix] remove chart function kube-starrocks.name #292

Merged

kevincai closed this as completed in #292 Oct 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR 1064 (HY000): Access denied for user 'root' (using password: NO) || BE constantly has restart after a few hours #275

ERROR 1064 (HY000): Access denied for user 'root' (using password: NO) || BE constantly has restart after a few hours #275

jimenez-f commented Oct 4, 2023

yandongxiao commented Oct 5, 2023

yandongxiao commented Oct 5, 2023

jimenez-f commented Oct 5, 2023

yandongxiao commented Oct 5, 2023

jimenez-f commented Oct 5, 2023

jimenez-f commented Oct 6, 2023

kevincai commented Oct 7, 2023

jimenez-f commented Oct 9, 2023 •

edited

Loading

kevincai commented Oct 11, 2023 •

edited

Loading

jimenez-f commented Oct 11, 2023

jimenez-f commented Oct 12, 2023 •

edited

Loading

kevincai commented Oct 12, 2023 •

edited

Loading

jimenez-f commented Oct 17, 2023

kevincai commented Oct 17, 2023

jimenez-f commented Oct 18, 2023 •

edited

Loading

kevincai commented Oct 18, 2023

jimenez-f commented Oct 18, 2023

kevincai commented Oct 18, 2023

cheny1994 commented Oct 21, 2023

adhasahar97 commented Oct 27, 2023 •

edited

Loading

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

kevincai commented Oct 27, 2023

kevincai commented Oct 27, 2023

yandongxiao commented Oct 27, 2023

ERROR 1064 (HY000): Access denied for user 'root' (using password: NO) || BE constantly has restart after a few hours #275

ERROR 1064 (HY000): Access denied for user 'root' (using password: NO) || BE constantly has restart after a few hours #275

Comments

jimenez-f commented Oct 4, 2023

yandongxiao commented Oct 5, 2023

yandongxiao commented Oct 5, 2023

jimenez-f commented Oct 5, 2023

yandongxiao commented Oct 5, 2023

jimenez-f commented Oct 5, 2023

jimenez-f commented Oct 6, 2023

kevincai commented Oct 7, 2023

jimenez-f commented Oct 9, 2023 • edited Loading

kevincai commented Oct 11, 2023 • edited Loading

jimenez-f commented Oct 11, 2023

jimenez-f commented Oct 12, 2023 • edited Loading

kevincai commented Oct 12, 2023 • edited Loading

jimenez-f commented Oct 17, 2023

kevincai commented Oct 17, 2023

jimenez-f commented Oct 18, 2023 • edited Loading

kevincai commented Oct 18, 2023

jimenez-f commented Oct 18, 2023

kevincai commented Oct 18, 2023

cheny1994 commented Oct 21, 2023

adhasahar97 commented Oct 27, 2023 • edited Loading

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

adhasahar97 commented Oct 27, 2023

kevincai commented Oct 27, 2023

kevincai commented Oct 27, 2023

kevincai commented Oct 27, 2023

yandongxiao commented Oct 27, 2023

jimenez-f commented Oct 9, 2023 •

edited

Loading

kevincai commented Oct 11, 2023 •

edited

Loading

jimenez-f commented Oct 12, 2023 •

edited

Loading

kevincai commented Oct 12, 2023 •

edited

Loading

jimenez-f commented Oct 18, 2023 •

edited

Loading

adhasahar97 commented Oct 27, 2023 •

edited

Loading