Collect logging for exited containers

[adriansuarez]

For any containers with non-0 restart counts, collect logging for exited containers using kubectl logs ... --previous.

[adriansuarez]

FYI @sivanov-nuodb, I created the branch asuarez/test-minikube-2.5.0 which includes this change and also reverts the Control Plane used in Minikube to 2.5.0. You should be able to run the pipeline on that branch to reproduce the webhook issue.

The root cause seems to be that the container exits unexpectedly and takes a long time to acquire the lease on restart. Not sure what is causing the crashes on 2.5.0 or whether it is still relevant in newer versions.

[sivanov-nuodb]

Thanks for addign this!

[sivanov-nuodb]

FYI @sivanov-nuodb, I created the branch asuarez/test-minikube-2.5.0 which includes this change and also reverts the Control Plane used in Minikube to 2.5.0. You should be able to run the pipeline on that branch to reproduce the webhook issue.

The root cause seems to be that the container exits unexpectedly and takes a long time to acquire the lease on restart. Not sure what is causing the crashes on 2.5.0 or whether it is still relevant in newer versions.

Thanks for creating the branch so that we can investigate further!

The webhook server is started immediately and does not wait for the leader assignment. I agree that the webhook error (connection refused) is due to the operator container being restarted.

The operator will fail if the VolumeSnapshot CRD is not installed in the cluster but the embedded backup plugin is enabled. In CP 2.6.0, the operator Helm chart will automatically disable the backup plugin (fixed by you in https://github.com/nuodb/nuodb-control-plane/commit/9759ef911e4163ee2d48a3cd376d1a3a86487107) which is why the issue is not exposed in CP 2.7.0.

[adriansuarez]

Thanks for creating the branch so that we can investigate further!

The webhook server is started immediately and does not wait for the leader assignment. I agree that the webhook error (connection refused) is due to the operator container being restarted.

The operator will fail if the VolumeSnapshot CRD is not installed in the cluster but the embedded backup plugin is enabled. In CP 2.6.0, the operator Helm chart will automatically disable the backup plugin (fixed by you in nuodb/nuodb-control-plane@9759ef9) which is why the issue is not exposed in CP 2.7.0.

Okay, so it is no longer an issue.

I will keep using 2.6.1 rather than explicitly disabling the backup manager, since it does give us more coverage of past product versions. Currently we have coverage of 2.5.0 via KWOK, and both the KWOK and Minikube variants exercise 2.7.0 (latest).