Split CI pipelines

[jolheiser]

• This PR attempts to split our various DB tests into separate pipelines.
• It splits up some of the extra feature-related tests rather than having most of them in the MySQL test.
• It disables the race detector for some of the pipelines as well, as it can cause slower runs and is mostly redundant when the pipelines just swap DBs.
• It builds without SQLite support for any of the non-SQLite pipelines.
• It moves the e2e test to using SQLite rather than PG (partially because I moved the minio tests to PG and that mucked up the test config, and partially because it avoids another running service)
• It splits up the go mod download task in the Makefile from the tool installation, as the tools are only needed in the compliance pipeline. (Arguably even some of the tools aren't needed there, but that could be a follow-up PR)
• SQLite is now the only arm64 pipeline, moving PG back to amd64 which can leverage autoscaler

Should resolve #22010 - one thing that wasn't changed here but is mentioned in that issue, unit tests are needed in the same pipeline as an integration test in order to form a complete coverage report (at least as far as I could tell), so for now it remains in a pipeline with a DB integration test.

Please let me know if I've inadvertently changed something that was how it was on purpose.

---

I will say sometimes it's hard to pin down the average time, as a pipeline could be waiting for a runner for X minutes and that brings the total up by X minutes as well, but overall this does seem to be faster on average.

[delvh]

Apart from my comments below, the CI seems to agree with you that you improved its performance.

[delvh]

?

[jolheiser]

This was set twice in the same step, see two lines below.

[silverwind]

pull: always should only be set for the first occurence of an image in the whole file. @jolheiser can you check that this is still true?

[jolheiser]

I can double-check, sure.
Since some of these can now run in parallel, it might be a bit off. I'll see what comes up.

[silverwind]

Yeah I guess for the sake of performance, we can skip double pull in parallel runs. The CI runs often enough to keep images up to date. Even better would be an outside job on the server that updates images once a day or so.

[silverwind]

I think this whole image pulling business is something that drone or each server could do itself. Even if you make a cron pipeline that does the pulling, it's not guaranteed to execute on all daemons in a distributed setup. What I do on my drone machines is this in a system cron once per day:

docker images --format "{{.Repository}}:{{.Tag}}" | grep -v '<none>' | sort | xargs -L1 docker pull
docker system prune -f

[jolheiser]

I'm not sure if this current PR would hit dockerhub rate limits as-is, but in general I think image pulling should be cleaned up.

I'm just not sure if it should be done before or after this PR.

Another possibility is to more fully qualify the tag used, such that we pin the version more specifically.
A pro would be needing to pull less frequently, a con being we would need to manually update CI more often to update image versions, and I'm not sure if all images would play nice with that.

[silverwind]

I don't think exact tags is really maintainable without automation that also continuously updates the versions as some images publish very frequently.

[techknowlogick]

We use a combination of a static runner, and autoscaler that will create/destroy runners as needed.

[silverwind]

We use a combination of a static runner, and autoscaler that will create/destroy runners as needed.

Ok, sounds like we are not really in control of the docker image storage in such a setup, right?

[delvh]

The other images will probably be outdated too, but I don't think this should be the scope of this PR (including my earlier comment regarding the postgres version 😅 )