Run Kubernetes conformance test #1385

jefflill · 2022-01-05T00:14:45Z

We need to run the compliance test so we know where we stand.

https://www.cncf.io/certification/software-conformance/
https://github.com/cncf/k8s-conformance/blob/master/instructions.md

FYI: Looks like we need to join CNCF to be certificated:

https://github.com/cncf/k8s-conformance/blob/master/faq.md#what-is-the-cost-of-certification

jefflill · 2022-02-02T18:03:27Z

Single node cluster:

12GB RAM
32GB disk

I thought I'd go ahead and run the conformance test this morning while I work in neon-desktop and it looks like bad things are happening. I downloaded and extracted sonobuoy as described in the instructions above and then ran it like:

λ sonobuoy run --mode=certified-conformance
I0202 09:18:54.593052   42404 request.go:668] Waited for 1.1841515s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.2:6442/apis/storage.k8s.io/v1beta1?timeout=32s
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy namespace= resource=namespaces
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy namespace=sonobuoy resource=pods
time="2022-02-02T09:19:03-08:00" level=info msg="create request issued" name=sonobuoy-aggregator namespace=sonobuoy resource=services

sonobuoy printed out the lines above and exited. Next, I ran this:

neon get pods -A --watch

and the cluster is in chaos:

neon-storage   openebs-localpv-provisioner-5c78777b68-rdhvs           0/1     Pending       0          0s
neon-storage   openebs-localpv-provisioner-5c78777b68-rdhvs           0/1     Pending       0          0s
neon-storage   openebs-ndm-operator-5bc8b6574b-r4988                  0/1     Evicted       0          3s
neon-storage   openebs-ndm-operator-5bc8b6574b-4cfkc                  0/1     Pending       0          0s
neon-storage   openebs-ndm-operator-5bc8b6574b-4cfkc                  0/1     Pending       0          0s
neon-system    neon-cluster-operator-8675c4c57f-6frb6                 0/2     Evicted       0          4s
neon-system    neon-cluster-operator-8675c4c57f-dkwrz                 0/2     Pending       0          0s
neon-system    neon-cluster-operator-8675c4c57f-dkwrz                 0/2     Pending       0          0s
neon-storage   openebs-snapshot-operator-88df5cbc-h25vb               0/2     Evicted       0          3s
neon-storage   openebs-snapshot-operator-88df5cbc-42xm8               0/2     Pending       0          0s
neon-storage   openebs-snapshot-operator-88df5cbc-42xm8               0/2     Pending       0          0s
neon-storage   openebs-provisioner-85b79b6db5-sfkn4                   0/1     Evicted       0          3s
neon-storage   openebs-provisioner-85b79b6db5-cfwsj                   0/1     Pending       0          0s
neon-storage   openebs-provisioner-85b79b6db5-cfwsj                   0/1     Pending       0          0s
neon-storage   openebs-apiserver-767dcd948c-xtwnc                     0/1     Evicted       0          4s
neon-storage   openebs-apiserver-767dcd948c-hzwnq                     0/1     Pending       0          0s
neon-storage   openebs-apiserver-767dcd948c-hzwnq                     0/1     Pending       0          0s
neon-storage   openebs-ndm-operator-5bc8b6574b-4cfkc                  0/1     Evicted       0          3s
neon-storage   openebs-ndm-operator-5bc8b6574b-bv75p                  0/1     Pending       0          0s
neon-storage   openebs-ndm-operator-5bc8b6574b-bv75p                  0/1     Pending       0          0s
neon-storage   openebs-provisioner-85b79b6db5-cfwsj                   0/1     Evicted       0          2s
neon-storage   openebs-provisioner-85b79b6db5-gk465                   0/1     Pending       0          0s
neon-storage   openebs-provisioner-85b79b6db5-gk465                   0/1     Pending       0          0s
neon-system    neon-cluster-operator-8675c4c57f-dkwrz                 0/2     Evicted       0          3s
neon-system    neon-cluster-operator-8675c4c57f-nn7nj                 0/2     Pending       0          0s
neon-system    neon-cluster-operator-8675c4c57f-nn7nj                 0/2     Pending       0          0s
neon-storage   openebs-localpv-provisioner-5c78777b68-rdhvs           0/1     Evicted       0          4s
neon-storage   openebs-localpv-provisioner-5c78777b68-2k2bx           0/1     Pending       0          0s
neon-storage   openebs-localpv-provisioner-5c78777b68-2k2bx           0/1     Pending       0          0s
neon-storage   openebs-snapshot-operator-88df5cbc-42xm8               0/2     Evicted       0          4s
neon-storage   openebs-snapshot-operator-88df5cbc-4lb2c               0/2     Pending       0          0s
neon-storage   openebs-snapshot-operator-88df5cbc-4lb2c               0/2     Pending       0          0s
neon-storage   openebs-admission-server-74b979df68-22b46              0/1     Evicted       0          6s
neon-storage   openebs-admission-server-74b979df68-pql5j              0/1     Pending       0          0s
neon-storage   openebs-admission-server-74b979df68-pql5j              0/1     Pending       0          0s
neon-storage   openebs-apiserver-767dcd948c-hzwnq                     0/1     Evicted       0          3s
neon-storage   openebs-apiserver-767dcd948c-ml6vn                     0/1     Pending       0          0s
neon-storage   openebs-apiserver-767dcd948c-ml6vn                     0/1     Pending       0          0s
neon-storage   openebs-ndm-operator-5bc8b6574b-bv75p                  0/1     Evicted       0          4s
neon-storage   openebs-ndm-operator-5bc8b6574b-844dr                  0/1     Pending       0          0s
neon-storage   openebs-ndm-operator-5bc8b6574b-844dr                  0/1     Pending       0          0s
neon-storage   openebs-provisioner-85b79b6db5-gk465                   0/1     Evicted       0          3s
neon-storage   openebs-provisioner-85b79b6db5-424l7                   0/1     Pending       0          0s
neon-storage   openebs-provisioner-85b79b6db5-424l7                   0/1     Pending       0          0s
neon-system    neon-cluster-operator-8675c4c57f-nn7nj                 0/2     Evicted       0          3s
neon-system    neon-cluster-operator-8675c4c57f-l8frq                 0/2     Pending       0          0s
neon-system    neon-cluster-operator-8675c4c57f-l8frq                 0/2     Pending       0          1s
neon-storage   openebs-localpv-provisioner-5c78777b68-2k2bx           0/1     Evicted       0          4s
neon-storage   openebs-localpv-provisioner-5c78777b68-264wn           0/1     Pending       0          0s
neon-storage   openebs-localpv-provisioner-5c78777b68-264wn           0/1     Pending       0          0s
...

and it also looks like the sonobuoy pod is not starting due to disk pressure:

λ neon describe pod/sonobuoy
Name:                 sonobuoy
Namespace:            sonobuoy
Priority:             3000
Priority Class Name:  user-medium
Node:                 <none>
Labels:               component=sonobuoy
                      sonobuoy-component=aggregator
                      tier=analysis
Annotations:          <none>
Status:               Pending
IP:
IPs:                  <none>
Containers:
  kube-sonobuoy:
    Image:      sonobuoy/sonobuoy:v0.56.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /sonobuoy
    Args:
      aggregator
      --no-exit
      --level=info
      -v=4
      --alsologtostderr
    Environment:
      SONOBUOY_ADVERTISE_IP:   (v1:status.podIP)
    Mounts:
      /etc/sonobuoy from sonobuoy-config-volume (rw)
      /plugins.d from sonobuoy-plugins-volume (rw)
      /tmp/sonobuoy from output-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8xdmb (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  sonobuoy-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sonobuoy-config-cm
    Optional:  false
  sonobuoy-plugins-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sonobuoy-plugins-cm
    Optional:  false
  output-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-8xdmb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 kubernetes.io/e2e-evict-taint-key op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 30s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 30s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  4m43s (x124 over 24m)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.

neon get node master-0 -o yaml is reporting significant problems. Here are the node conditions:

I SSH-ed into the node and see that we have nearly 4GB RAM available and we're at 85% disk full, which will trigger the disk pressure status. I bunch of pods are evicted now, so there may have been less RAM available when sonobuoy started.

The sonobuoy pod spec in the cluster reports that it cannot be scheduled due to low RAM. I suspect that even though there's enough RAM now, the scheduler is trying to reschedule the higher priority pods first and can't due to the disk pressure.

jefflill · 2022-02-02T18:05:35Z

We should increase the disk pressure trigger to 95% and see if that helps:

https://github.com/nforgeio/neonCLOUD/issues/241

We may also need to modify Kublet to prevent eviction of high priority pods in certain circumstances like single node clusters.

jefflill · 2022-02-02T19:13:51Z

I tried this again with a much larger VM for the single node cluster:

32GB RAM
128GB disk

NOTE: Sonobouy requires that docker be configured as a default registry, You can configure this in your cluster definition or via a ContainerRegistry resource to an existing cluster:

apiVersion: neonkube.io/v1alpha1
kind: ContainerRegistry
metadata:
  name: docker.io
spec:
  prefix: docker.io
  searchOrder: 1
  insecure: false
  blocked: false
  location: docker.io

or via:

neon apply -f "%NC_ROOT%\devops\jefflill\docker-registry.yaml"

Sonobuoy is running much better on the bigger VM!

jefflill · 2022-02-03T02:03:16Z

The test run completed with 4 failures.

Here's the log file:

logs.txt

There are several warnings that here that probably don't count as compliance failures but we should address these anyway:

W0202 21:15:06.790127       1 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0202 21:15:06.813646       1 warnings.go:70] v1 ComponentStatus is deprecated in v1.19+
W0202 21:15:07.339368       1 warnings.go:70] openebs.io/v1alpha1 CStorPool is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorPoolInstance
W0202 21:15:07.344256       1 warnings.go:70] openebs.io/v1alpha1 StoragePool is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to openebs.io/v1alpha1 JivaVolumePolicy
W0202 21:15:07.363819       1 warnings.go:70] openebs.io/v1alpha1 StoragePoolClaim is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorPoolCluster
W0202 21:15:08.063079       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:08.071700       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:08.081002       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:09Z" level=info msg="Running ns query (kube-node-lease)"
W0202 21:15:11.268975       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:11.452550       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:11.457181       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:13Z" level=info msg="Running ns query (kube-public)"
W0202 21:15:14.631959       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:14.765140       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:14.798848       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:16Z" level=info msg="Running ns query (kube-system)"
W0202 21:15:18.006367       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:18.131919       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:18.179799       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:19Z" level=info msg="Running ns query (neon-ingress)"
W0202 21:15:21.365419       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:21.498328       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:21.531558       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:23Z" level=info msg="Running ns query (neon-monitor)"
W0202 21:15:24.731682       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:24.865369       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:24.899401       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:26Z" level=info msg="Running ns query (neon-storage)"
W0202 21:15:28.108681       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:28.231775       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:28.297189       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:29Z" level=info msg="Running ns query (neon-system)"
W0202 21:15:31.464496       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:31.637342       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:31.650633       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups
time="2022-02-02T21:15:33Z" level=info msg="Running ns query (sonobuoy)"
W0202 21:15:34.831954       1 warnings.go:70] openebs.io/v1alpha1 CStorRestore is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorRestore
W0202 21:15:34.965644       1 warnings.go:70] openebs.io/v1alpha1 CStorCompletedBackup is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorCompletedBackup
W0202 21:15:34.998368       1 warnings.go:70] openebs.io/v1alpha1 CStorBackups is deprecated; see https://github.com/openebs/upgrade/blob/HEAD/README.md for instructions to migrate to cstor.openebs.io/v1 CStorBackups

Here are the entire results (the relevant sub-directory is .\plugins\e2e\results\global):

results.zip

Here are the failures:

• Failure [38.772 seconds]
[sig-auth] ServiceAccounts
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/framework.go:23
  ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] [It]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:630

  Feb  2 19:31:47.115: Unexpected error:
      <*errors.errorString | 0xc0012600d0>: {
          s: "pod \"oidc-discovery-validator\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.1.20 PodIP:10.254.36.139 PodIPs:[{IP:10.254.36.139}] StartTime:2022-02-02 19:31:08 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:oidc-discovery-validator State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-02-02 19:31:14 +0000 UTC,FinishedAt:2022-02-02 19:31:14 +0000 UTC,ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:k8s.gcr.io/e2e-test-images/agnhost:2.32 ImageID:k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c Started:0xc00239160e}] QOSClass:BestEffort EphemeralContainerStatuses:[]}",
      }
      pod "oidc-discovery-validator" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.1.20 PodIP:10.254.36.139 PodIPs:[{IP:10.254.36.139}] StartTime:2022-02-02 19:31:08 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:oidc-discovery-validator State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-02-02 19:31:14 +0000 UTC,FinishedAt:2022-02-02 19:31:14 +0000 UTC,ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:k8s.gcr.io/e2e-test-images/agnhost:2.32 ImageID:k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c Started:0xc00239160e}] QOSClass:BestEffort EphemeralContainerStatuses:[]}
  occurred

  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/service_accounts.go:789
------------------------------
{"msg":"FAILED [sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","total":339,"completed":77,"skipped":1296,"failed":2,"failures":["[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]","[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]"]}

• Failure [38.772 seconds]
[sig-auth] ServiceAccounts
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/framework.go:23
  ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] [It]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:630

  Feb  2 19:31:47.115: Unexpected error:
      <*errors.errorString | 0xc0012600d0>: {
          s: "pod \"oidc-discovery-validator\" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.1.20 PodIP:10.254.36.139 PodIPs:[{IP:10.254.36.139}] StartTime:2022-02-02 19:31:08 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:oidc-discovery-validator State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-02-02 19:31:14 +0000 UTC,FinishedAt:2022-02-02 19:31:14 +0000 UTC,ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:k8s.gcr.io/e2e-test-images/agnhost:2.32 ImageID:k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c Started:0xc00239160e}] QOSClass:BestEffort EphemeralContainerStatuses:[]}",
      }
      pod "oidc-discovery-validator" failed with status: {Phase:Failed Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [oidc-discovery-validator]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2022-02-02 19:31:08 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.0.1.20 PodIP:10.254.36.139 PodIPs:[{IP:10.254.36.139}] StartTime:2022-02-02 19:31:08 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:oidc-discovery-validator State:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message:,StartedAt:2022-02-02 19:31:14 +0000 UTC,FinishedAt:2022-02-02 19:31:14 +0000 UTC,ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c,}} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:k8s.gcr.io/e2e-test-images/agnhost:2.32 ImageID:k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 ContainerID:cri-o://566c0ddb9ca7f0ca646695790e7cbdbebe53569aa02bd8538cee6cadcd03ab7c Started:0xc00239160e}] QOSClass:BestEffort EphemeralContainerStatuses:[]}
  occurred

  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/auth/service_accounts.go:789
------------------------------
{"msg":"FAILED [sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","total":339,"completed":77,"skipped":1296,"failed":2,"failures":["[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]","[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]"]}

• Failure [0.888 seconds]
[sig-apps] Daemon set [Serial]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/framework.go:23
  should rollback without unnecessary restarts [Conformance] [It]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:630

  Feb  2 19:50:03.641: Conformance test suite needs a cluster with at least 2 nodes.
  Expected
      <int>: 1
  to be >
      <int>: 1

  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apps/daemon_set.go:424
------------------------------
{"msg":"FAILED [sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]","total":339,"completed":139,"skipped":2348,"failed":3,"failures":["[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]","[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]"]}

• Failure [61.477 seconds]
[sig-scheduling] SchedulerPreemption [Serial]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/framework.go:40
  validates lower priority pod preemption by critical pod [Conformance] [It]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:630

  Feb  2 21:05:51.785: We need at least two pods to be created but all nodes are already heavily utilized, so preemption tests cannot be run

  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113
------------------------------
{"msg":"FAILED [sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance]","total":339,"completed":313,"skipped":5133,"failed":4,"failures":["[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]","[sig-auth] ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance]","[sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance]","[sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance]"]}

• Failure [61.627 seconds]
[sig-scheduling] SchedulerPreemption [Serial]
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduling/framework.go:40
  validates basic preemption works [Conformance] [It]
  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:630

  Feb  2 19:26:34.254: We need at least two pods to be created but all nodes are already heavily utilized, so preemption tests cannot be run

  /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/onsi/ginkgo/internal/leafnodes/runner.go:113
------------------------------
{"msg":"FAILED [sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]","total":339,"completed":63,"skipped":905,"failed":1,"failures":["[sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance]"]}

jefflill · 2022-02-03T02:09:48Z

Here's a summary of the failures:

ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer

The issue above is probably the only one that matters. The three tests below require at least 2 cluster node and I only had one:

Daemonset should rollback without unnecessary restarts
SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod
SchedulerPreemption [Serial] validates basic preemption works

jefflill · 2022-02-03T02:19:10Z

Here's a Kubernetes issue discussing the ServiceAccountIssuerDiscovery problem.

kubernetes/kubernetes#99480

...and here's some documentation:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

jefflill · 2022-02-03T06:20:02Z

So it looks like the problem is the lack of the https:// scheme prefix for this option in the API server's static pod manifest:

--service-account-issuer=https://kubernetes.default.svc

This looks like a Kubernetes problem because the static pod files configured by the kubeadm tool.

jefflill · 2022-02-03T20:00:33Z

neonKUBE IS COMPLANT!!!

λ sonobuoy status
         PLUGIN     STATUS   RESULT   COUNT               PROGRESS
            e2e   complete   passed       1   339/339 (0 failures)
   systemd-logs   complete   passed       3

Sonobuoy has completed. Use `sonobuoy retrieve` to get results.

jefflill changed the title ~~Run Kubernetes compliance test~~ Run Kubernetes conformance test Feb 2, 2022

jefflill assigned jefflill and marcusbooyah Feb 2, 2022

jefflill closed this as completed Feb 3, 2022

jefflill added a commit that referenced this issue Feb 3, 2022

#1385: Kubernetes conformance test PASSES!!!

2252e5b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run Kubernetes conformance test #1385

Run Kubernetes conformance test #1385

jefflill commented Jan 5, 2022 •

edited

Loading

jefflill commented Feb 2, 2022 •

edited

Loading

jefflill commented Feb 2, 2022 •

edited

Loading

jefflill commented Feb 2, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022

Run Kubernetes conformance test #1385

Run Kubernetes conformance test #1385

Comments

jefflill commented Jan 5, 2022 • edited Loading

jefflill commented Feb 2, 2022 • edited Loading

jefflill commented Feb 2, 2022 • edited Loading

jefflill commented Feb 2, 2022 • edited Loading

jefflill commented Feb 3, 2022 • edited Loading

jefflill commented Feb 3, 2022 • edited Loading

jefflill commented Feb 3, 2022 • edited Loading

jefflill commented Feb 3, 2022 • edited Loading

jefflill commented Feb 3, 2022

jefflill commented Jan 5, 2022 •

edited

Loading

jefflill commented Feb 2, 2022 •

edited

Loading

jefflill commented Feb 2, 2022 •

edited

Loading

jefflill commented Feb 2, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading

jefflill commented Feb 3, 2022 •

edited

Loading