Secret not getting created with sumologic-fluentd Chart

[EppO]

Hi,

There is something fishy going on when trying to use argo-cd with sumologic-fluentd chart.
The secret never gets created by argo-cd.

$ argocd app create sumologic --repo https://github.com/helm/charts.git  --path stable/sumologic-fluentd  --dest-server https://kubernetes.default.svc  -p sumologic.collectorUrl="blabla" -p rbac.create=true --dest-namespace default
$ argocd app manifests sumoglogic
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: sumologic-fluentd
    app.kubernetes.io/instance: sumologic
    chart: sumologic-fluentd-0.11.0
    heritage: Tiller
    release: sumologic
  name: sumologic-sumologic-fluentd

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: sumologic-fluentd
    app.kubernetes.io/instance: sumologic
    chart: sumologic-fluentd-0.11.0
    heritage: Tiller
    release: sumologic
  name: sumologic-sumologic-fluentd
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - list
  - watch

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app: sumologic-fluentd
    app.kubernetes.io/instance: sumologic
    chart: sumologic-fluentd-0.11.0
    heritage: Tiller
    release: sumologic
  name: sumologic-sumologic-fluentd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: sumologic-sumologic-fluentd
subjects:
- kind: ServiceAccount
  name: sumologic-sumologic-fluentd
  namespace: default

---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    app: sumologic-fluentd
    app.kubernetes.io/instance: sumologic
    chart: sumologic-fluentd-0.11.0
    heritage: Tiller
    release: sumologic
  name: sumologic-sumologic-fluentd
spec:
  template:
    metadata:
      labels:
        app: sumologic-fluentd
        release: sumologic
      name: sumologic-sumologic-fluentd
    spec:
      containers:
      - env:
        - name: COLLECTOR_URL
          valueFrom:
            secretKeyRef:
              key: collector-url
              name: sumologic-sumologic-fluentd
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: FLUENTD_SOURCE
          value: file
        - name: LOG_FORMAT
          value: json
        - name: KUBERNETES_META
          value: "true"
        - name: TIME_KEY
          value: <nil>
        - name: ADD_TIMESTAMP
          value: "true"
        - name: ADD_TIME
          value: "true"
        - name: ADD_STREAM
          value: "true"
        - name: VERIFY_SSL
          value: "true"
        - name: READ_FROM_HEAD
          value: "true"
        image: sumologic/fluentd-kubernetes-sumologic:v2.1.0
        imagePullPolicy: IfNotPresent
        name: sumologic-sumologic-fluentd
        resources:
          limits:
            cpu: 256m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
        volumeMounts:
        - mountPath: /mnt/log/
          name: host-logs
          readOnly: true
        - mountPath: /var/log/
          name: host-logs
          readOnly: true
        - mountPath: /var/lib/docker/
          name: docker-logs
          readOnly: true
        - mountPath: /mnt/pos/
          name: pos-files
      serviceAccountName: sumologic-sumologic-fluentd
      volumes:
      - emptyDir: {}
        name: pos-files
      - hostPath:
          path: /var/log/
        name: host-logs
      - hostPath:
          path: /var/lib/docker
        name: docker-logs
  updateStrategy:
    type: OnDelete

I can see the override parameters but secret is not listed as "Missing"

Name:               sumologic
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://<argocd-ip>/applications/sumologic
Repo:               https://github.com/helm/charts.git
Target:
Path:               stable/sumologic-fluentd
Sync Policy:        <none>
Sync Status:        OutOfSync from  (7255244)
Health Status:      Missing

NAME                    VALUE
sumologic.collectorUrl  blabla
rbac.create             true

GROUP                      KIND                NAMESPACE  NAME                         STATUS     HEALTH
                           ServiceAccount      default    sumologic-sumologic-fluentd  OutOfSync  Missing
rbac.authorization.k8s.io  ClusterRole         default    sumologic-sumologic-fluentd  OutOfSync  Missing
rbac.authorization.k8s.io  ClusterRoleBinding  default    sumologic-sumologic-fluentd  OutOfSync  Missing
extensions                 DaemonSet           default    sumologic-sumologic-fluentd  OutOfSync  Missing

Even after a sync, secret doesn't show up and sumologic pods crash because the secret is missing.

When using helm to install the chart, the secret gets created correctly though.

$ helm install -n sumologic-fluentd --set sumologic.collectorUrl="blabla" --set rbac.create=true stable/sumologic-fluentd
$ kubectl get secrets
NAME                                              TYPE                                  DATA   AGE
default-token-mn1x3                               kubernetes.io/service-account-token   3      16m
sumologic-fluentd-sumologic-fluentd               Opaque                                1      16m
sumologic-fluentd-sumologic-fluentd-token-wx5ad   kubernetes.io/service-account-token   3      16m
$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
sumologic-fluentd-sumologic-fluentd-4v7hj   1/1     Running   0          16m
sumologic-fluentd-sumologic-fluentd-jwt3d   1/1     Running   0          16m
sumologic-fluentd-sumologic-fluentd-s2dfk   1/1     Running   0          16m

Am I missing something here? Or is it a bug?

[jessesuen]

Seems like a bug.

Was service account role for Argo CD application-controller given permissions to get/list/watch secrets? This is the default installation behavior, but I'm wondering if it's possible for secret access to have been prohibited.

[jessesuen]

I found the issue. I'm not sure why, but this chart makes the secret a helm hook for some reason.

apiVersion: v1
kind: Secret
metadata:
  name: release-name-sumologic-fluentd
  labels:
    app: sumologic-fluentd
    chart: "sumologic-fluentd-0.12.0"
    release: "release-name"
    heritage: "Tiller"
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": "before-hook-creation"
type: Opaque
data:
  collector-url: "YmxhYmxh"

Argo CD does not support helm hooks at the moment (issue #355), partly because we do not have a concept of a "pre-install"

[jessesuen]

Will duplicate this to #355

[jessesuen]

The workaround specifically for this chart, is to use the sumologic.collectorUrlExistingSecret parameter and reference a secret that is created out-of-band from this chart.

[jessesuen]

Submitted upstream PR to see if maintainers are willing to treat the secret as just a normal resource: helm/charts#11375

[EppO]

TIL about Helm hooks! Thanks for the quick answer, indeed I was using the existing secret as a workaround for that limitation. Will follow #355 for the full implementation of Helm hooks.