Run more comprehensive tests with a daily CI test suite

[leseb]

Currently, Rook tests only stable versions of Ceph, which means no testing is being done for the up-coming Ceph version. Developers have to locally build and test the next Ceph version. For this, no coordination is done so it's really up to the devs to do some tests when they think about it. That's far from ideal. Fortunately, so far, nothing major araised and transitions have been smooth. Although, I truly believe that some changes could have happened earlier in the cycle.

They are numerous reasons to be as close as possible from Ceph master:

• be aware of any breaking changes coming from Ceph that might affect Rook, be more responsive
• testing upgrades more frequently
• implement improvements in Rook coming from newest Ceph features

One downside to that is that the CI will be "Red" more frequently because Ceph is broken upstream. So that will hold some of Rook's work.

This obviously raises another issue that is closely connected to #2632 which is how do we decide which stable version of Ceph is tested as part of a given stable version of Rook. Typically, in projects that are consuming Ceph we try to pin a stable version of that project (e,g: ceph-ansible) with one stable version of Ceph, even though in practice we can deploy more versions of Ceph. Since Rook claims its support for multiple versions of Ceph, the testing matrix (CI) will considerably grow but we can keep things simple and for a given stable branch of Rook always test the latest stable version of Ceph.

Additionally, I don't know if Rook's CI has nightly jobs to test stable branches but I believe that's something nice to have to guarantee its stability.

In the end, this will result in a huge refactor of the actual CI but that's worth it.

[travisn]

These suggestions are the perfect scenario for a nightly build. Today we only have builds for PRs, master, and release branches. All of the CI today is focused around what we are supporting in the releases. Already the test matrix is growing and we need to refactor this so we get better test coverage without increasing the time to run the typical CI with every PR. I could see the nightly build being designed to do the following:

1. New test suites would be implemented to increase the test coverage of different supported versions of Ceph, upgrades, other comprehensive tests scenarios.
2. The same tests from the regular CI could run against the latest master.
   1. The base image for the operator would be the latest image from master.
3. Stress/longhaul tests would be run regularly (although perhaps on a weekly basis).

These would be best as independent CI runs so that we could have different levels of priority for investigation. Failures for supported versions (1st bullet) would be urgent, failures against the latest master (2nd bullet) would not be urgent, and stress failures somewhere in between.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[travisn]

With the github actions we are already able to more easily add new tests and expand test coverage.