add an st2canary pod that validates st2.packs.volumes

[cognifloyd]

Closes #321

We often get questions about getting the volumes working, but the errors are very hard to debug by the time st2 is running. So, I'm trying to prevent people from installing this chart if their values for volumes do not lead to a valid setup. And, if someone tries to upgrade from an install w/o volumes enabled to an install with it enabled, then I want the upgrade to fail early if the volumes settings are not working.

There are several sources for volumes issues:

• the values are wrong
• the k8s cluster is configured incorrectly
• the underlying storage is configured incorrectly

I really want a way to push people to figure that out before the chart finishes installing something that is broken.

A key component of this is that it runs BEFORE the chart gets installed or updated. Especially with production installs, volumes issues should be found and fixed as early as possible, which is before the rest of the components get installed or upgraded.

[cognifloyd]

@armab I want to get this in the next release as it should help isolate issues with st2.packs.volumes config and make it easier to identify and help people with that.

[arm4b]

Considering what this Pod does, verifying write filesystem access, can we move the logic to the original tests dir, maybe even possible under the BATS tests?

It can be the same pod
https://github.com/StackStorm/stackstorm-k8s/blob/master/templates/tests/st2tests-pod.yaml
or perhaps a new one dedicated to volume testing?

And so it could be run with the same testing mechanism helm test exactly when it's needed for troubleshooting, instead of running on every Helm deployment which feels a bit off for production.

[cognifloyd]

I've never thought of using helm test as a debugging tool. Our current test script is designed for running in CI. It would have to be a lot more robust if we want people to run it on their customized st2 clusters. For example, assuming that logging in with the st2admin user/pass is often not valid (eg w/ ldap).

[cognifloyd]

@mamercad What do you think of this pod?

We often get questions about getting the volumes working, but the errors are very hard to debug by the time st2 is running. So, I'm trying to prevent people from installing this chart if their values for volumes do not lead to a valid setup. And, if someone tries to upgrade from an install w/o volumes enabled to an install with it enabled, then I want the upgrade to fail early if the volumes settings are not working.

There are several sources for volumes issues:

• the values are wrong
• the k8s cluster is configured incorrectly
• the underlying storage is configured incorrectly

I really want a way to push people to figure that out before the chart finishes installing something that is broken.

A key component of this is that it runs BEFORE the chart gets installed or updated. helm test can only run AFTER the chart is installed, so it does not meet my requirements (plus that has all of the CI assumptions baked in, so it is not useful for end users to debug with). I suppose we could have a values flag to opt-out of running this pod, but I want to make sure people run it before installing the chart.

@armab doesn't like the idea of running this with production clusters, but that's explicitly one of my goals. Especially with production installs, volumes issues should be found and fixed as early as possible, which is before the rest of the components get installed or upgraded.

Thoughts?

[mamercad]

So, I'm trying to prevent people from installing this chart if their values for volumes do not lead to a valid setup. And, if someone tries to upgrade from an install w/o volumes enabled to an install with it enabled, then I want the upgrade to fail early if the volumes settings are not working.

I'm very much a fan of this.

[mamercad]

@armab doesn't like the idea of running this with production clusters, but that's explicitly one of my goals. Especially with production installs, volumes issues should be found and fixed as early as possible, which is before the rest of the components get installed or upgraded.

I'm not overly familiar with StackStorm HA at this point in time (I'm not running it in production yet), but, I will say that as an operator of other Kubernetes applications, I'd personally rather have an application never get off of the ground (versus being broken while getting off of the ground) if that's a possibility. Adding simple RW testing to a Pod whose purpose and name implies that it's for testing (heh, maybe it should be renamed to st2canary) seems like a fine idea to me.

In short, I'm in the "fail as early as possible" camp as well.

[cognifloyd]

Ooh. I like the idea of calling it st2canary. Very nice. I'm going to use that.

[arm4b]

Left a few more comments.

Can we make it a Job, same as others that we already have and call more explicitly what it does like st2-verify-pack-write-access (or somewhat close to that)?

The jobs we already have:
https://github.com/StackStorm/stackstorm-k8s/blob/master/templates/jobs.yaml
and as I remember if a job like register-content failed, the entire helm deployment will report a failure too. So logically that's following what you want to do.

That would be a bit more consistent with the existing mechanics.

[cognifloyd]

Yeah, that makes sense. I'll work on:

• turning it into a job
• adding more detailed error messaging with:
  • a link to the docs
  • what to do in case of failure

[arm4b]

Thanks for moving it to the jobs, looks more consistent.

Left a few more comments and observations.

[CLAassistant]

All committers have signed the CLA.

[arm4b]

Thanks a lot! 👍