Run tests as separate jobs

[nkaretnikov]

Description

This pull request:

• splits up tests into multiple workflows
• splits up integration tests into multiple jobs
• fixes some flaky tests.

Pull request checklist

• Did you test this change locally?
• Did you update the documentation (if required)?
• Did you add/update relevant tests for this change (if required)?

Additional information

• Now, if integration tests fail, you don't need to wait for unit tests to finish before restarting them, and vice versa.
• Integration tests are split into multiple jobs, so it's easier to restart each separately (there's less chance of a job failing because it involves less tests now).

Also see #803.

How to test

[netlify]

✅ Deploy Preview for conda-store canceled.

Name	Link
🔨 Latest commit	e8159cb
🔍 Latest deploy log	https://app.netlify.com/sites/conda-store/deploys/663189452f9a990008d97c75

[peytondmurray]

Just a few comments, otherwise this looks like it will really help the CI be more useful. Thanks for this 🚀

[trallard]

We should perhaps rename it to build_docker.yaml as it is not testing the Docker image per se

[trallard]

Suggested change

	name: "Test build Docker image"
	name: "Build and push Docker image"

[nkaretnikov]

Had a similar idea as well, but decided against it to avoid confusion. We already have a job called build_and_push_docker_image that's part of release.yaml. This job was part of tests.yaml, so I prefixed it with "Test" to see which is which. AFAICT, it also doesn't push anything because of push: false, so looks like the intention here is to just test whether a docker container can be built.

In the interest of time, I'm going to merge this as is since it's already approved, but this can always be resolved in followup PRs.

[trallard]

But it is not testing the docker image just building it the fact that we have a step/job named build and push is inconsequential to having a workflow named build or build and push.

[nkaretnikov]

Yeah, I think a better name might be "Build Docker image", but we already have a file called build_docker_image.yaml with the name "Publish Dev Docker Images".

I think my preferred solution would be making all these consistent, so that the name matches the filename and there are no clashes between files. Something like this (untested):

• "Publish dev Docker image" -> publish_dev_docker_image.yaml (renamed from "Publish Dev Docker Images")
• "Build Docker image" -> build_docker_image.yaml (the workflow discussed in this thread)

@peytondmurray Hey Peyton, I'm currently on PTO, would you be comfortable taking over this in a separate PR? The task would be coming up with a consistent scheme for names and filenames of docker workflows.