Missing secret-key config value validation

[gustavosr98]

It is not clear from the charm docs that the secret-key needs to be at least 8 char long
It is also not very verbose from Juju POV what is actually happening on the charm

Ideally I would expect to have some config value validation on the charm to set it maybe on blocked state but avoid having the actual service down

Reproduce

juju config minio secret-key=minio

Logs

I cannot access MinIO website

$ juju status | grep minio
minio                      res:oci-image@1755999    waiting      1  minio                    ckf-1.7/stable  186  10.152.183.165  no       
mlflow-minio               res:oci-image@1755999    active       1  minio                    ckf-1.7/edge    186  10.152.183.108  no       
minio/0*                      error     idle   10.1.149.251  9000/TCP,9001/TCP  crash loop backoff: back-off 2m40s restarting failed container=minio pod=minio-0_kubeflow(1980c8fe-8cb3-4099-b9eb-2c6...
mlflow-minio/0*               active    idle   10.1.150.41   9000/TCP,9001/TCP

$ microk8s.kubectl get pods -n kubeflow | grep minio
minio-operator-0                                1/1     Running            0              40d
mlflow-minio-operator-0                         1/1     Running            0              67m
mlflow-minio-0                                  1/1     Running            0              66m
minio-0                                         0/1     CrashLoopBackOff   5 (105s ago)   6m24s

$ microk8s.kubectl logs -n kubeflow minio-0
Defaulted container "minio" out of: minio, juju-pod-init (init)
ERROR Unable to validate credentials inherited from the shell environment: Invalid credentials
      > Please provide correct credentials
      HINT:
        Access key length should be at least 3, and secret key length at least 8 characters

[lucabello]

@orfeas-k I just spent a few hours having to debug this with trial-and-error, because juju debug-log doesn't help at all :) Could you add some check in the charm, and set it to Blocked with a nice message?

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6057.

This message was autogenerated

[ShrishtiKarkera]

The bug still persists. I tried to configure the secret-key as minio and it changed to error state without any warning.
I was able to verify the reason of the error in the logs as it had a hint there.

Before configuring minio secret-key -

After configuring minio secret-key - juju config minio secret-key=minio

Destroying the model and redeploying the bundle seems to be the only way out. I tried changing the minio secret-key to minio1234 but it doesn't budge. I also tried to restart the node by deleting the minio-0 pod but it returns back to the error state.

These are some logs and metrics that might help:
kubectl logs -n kubeflow minio-0

juju config minio

kubectl get po -n kubeflow | grep minio

kubectl describe po -n kubeflow minio-0

shrishtikarkera@rag-demo-jh:~$ kubectl describe po -n kubeflow minio-0
Name:             minio-0
Namespace:        kubeflow
Priority:         0
Service Account:  default
Node:             juju-df992b-0/10.128.0.17
Start Time:       Thu, 24 Oct 2024 17:26:26 +0000
Labels:           app.kubernetes.io/name=minio
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=minio-6c98d6979f
                  statefulset.kubernetes.io/pod-name=minio-0
Annotations:      apparmor.security.beta.kubernetes.io/pod: runtime/default
                  charm.juju.is/modified-version: 0
                  cni.projectcalico.org/containerID: 0edba5c721d19cab0f453b3b50bcd1b4c558f1ea99da427c976854d3079777a5
                  cni.projectcalico.org/podIP: 10.1.209.188/32
                  cni.projectcalico.org/podIPs: 10.1.209.188/32
                  controller.juju.is/id: b5d69926-8e8a-4363-8b01-49608c270755
                  model.juju.is/id: e60163c9-3b29-4c02-8129-918cf0fbec30
                  seccomp.security.beta.kubernetes.io/pod: docker/default
                  unit.juju.is/id: minio/0
Status:           Running
IP:               10.1.209.188
IPs:
  IP:           10.1.209.188
Controlled By:  StatefulSet/minio
Init Containers:
  juju-pod-init:
    Container ID:  containerd://59b404417f50ebd8d4dfe780b3a77279a680f37a3776d5df50d53ba34e5356c1
    Image:         docker.io/jujusolutions/jujud-operator:3.5.4
    Image ID:      docker.io/jujusolutions/jujud-operator@sha256:c00558be1d56a960451686327a422aaa766b1024730126600b5d79ad9ea10b84
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      export JUJU_DATA_DIR=/var/lib/juju
      export JUJU_TOOLS_DIR=$JUJU_DATA_DIR/tools
      
      mkdir -p $JUJU_TOOLS_DIR
      cp /opt/jujud $JUJU_TOOLS_DIR/jujud
      
      initCmd=$($JUJU_TOOLS_DIR/jujud help commands | grep caas-unit-init)
      if test -n "$initCmd"; then
      exec $JUJU_TOOLS_DIR/jujud caas-unit-init --debug --wait;
      else
      exit 0
      fi
      
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 24 Oct 2024 17:26:28 +0000
      Finished:     Thu, 24 Oct 2024 17:26:41 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/juju from juju-data-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6glbp (ro)
Containers:
  minio:
    Container ID:  containerd://1000e74102d4c18c555f77bbb5e2ce010a24891e68b1c24b5e7b889a998e1474
    Image:         registry.jujucharms.com/charm/81j63o4a2ldarn1umc22iyjz1q9l9g0sx5b8j/oci-image@sha256:220b31a68d3264f53a746a364207f28868887a7c62c61cc650fd52d8e557641a
    Image ID:      registry.jujucharms.com/charm/81j63o4a2ldarn1umc22iyjz1q9l9g0sx5b8j/oci-image@sha256:220b31a68d3264f53a746a364207f28868887a7c62c61cc650fd52d8e557641a
    Ports:         9000/TCP, 9001/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      server
      /data
      --certs-dir
      /minio/.minio/certs
      --console-address
      :9001
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 24 Oct 2024 17:37:33 +0000
      Finished:     Thu, 24 Oct 2024 17:37:33 +0000
    Ready:          False
    Restart Count:  7
    Environment Variables from:
      minio-secret  Secret  Optional: false
    Environment:
      MINIO_PROMETHEUS_AUTH_TYPE:  public
      configmap-hash:              654cf2f1d31af8f2f86f275ea9f423a05743a81a2bfdfd055048c1cad270e388
    Mounts:
      /data from minio-data-75d42bd0 (rw)
      /minio/.minio/certs/CAs from ssl-ca (rw)
      /usr/bin/juju-exec from juju-data-dir (rw,path="tools/jujud")
      /var/lib/juju from juju-data-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6glbp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  minio-data-75d42bd0:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  minio-data-75d42bd0-minio-0
    ReadOnly:   false
  juju-data-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  ssl-ca:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-6glbp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/arch=amd64
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Warning  BackOff  4m36s (x46 over 14m)  kubelet  Back-off restarting failed container minio in pod minio-0_kubeflow(7808ee0c-01f4-42f8-8dae-adfb8354568e)

[orfeas-k]

Hey @ShrishtiKarkera, I think this should be due to not having backported this fix. I 'll check with the team and get back to you.