Create a nightly pipeline for GA/EUS releases

[tkoscieln]

Separate RHEL GA/EUS releases testing to nightly pipeline from the pipeline ran on every PR.

This pull request includes:

• adequate testing for the new functionality or fixed issue

[tkoscieln]

Sample succesfull run of the GA/EUS Nightly pipeline (debug):
https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1258487788

[jrusz]

Can you please rename it to just GA pipeline? The nightly pipeline is called that way because it's testing nightly composes while this is supposed to be testing GA composes which is something else, nightly GA doesn't make sense. So please also update all instances of "ga" and "nightly" used together to just "ga".

The rules don't seem to work, I see many jobs on GA runners on the regular PR pipeline.

[atodorov]

Minor comments

[jrusz]

I'm thinking that it would be enough to just call it GA not GA/EUS because there is also ELS and idk what else and it's basically all a subset of GA. Every released RHEL version no matter the support level can still be considered GA. What do you think @atodorov ?

Also the rules don't seem to work, I see many GA jobs in the PR pipeline. The way these rules work by default is that whenever it matches the conditions it will schedule the job. Since you've added these new rules it will enabled to run it in the scheduled GA pipeline but there is nothing preventing it from running on PRs for that I think you'll need to modify the upstream rules to exclude these jobs

[jrusz]

So what seems to work is if you take your exclude rule and add it to the upstream rules and build rules for example:

.build_rules:
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "false" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $SKIP_CI == "true" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'
      when: manual

.upstream_rules_all:
  rules:
    - if: '$CI_PIPELINE_SOURCE != "schedule" && $RUNNER !~ /[\S]+rhel-[\S]+-(?:(?:ga)|(?:eus))[\S]+/'

There is also the upstream rule for x86_64 which will require a little bit different regex and then there are a few jobs which have their own rules defined like aws.sh and some regression jobs. Use https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/ci/lint to check out if it works, just paste the whole CI config there, tick the checkbox to Simulate a pipeline created for the default branch and then just search for any -ga jobs that are still there.

[atodorov]

I'm thinking that it would be enough to just call it GA not GA/EUS because there is also ELS and idk what else and it's basically all a subset of GA. Every released RHEL version no matter the support level can still be considered GA. What do you think @atodorov ?

+1 for the proposed simplified naming. Let's use what you propose for now and update it later if need be.

[jrusz]

I've added some comments in the code. One of the main issues I see right now is that I don't think it will run on main after a PR is merged. It's defined as one of the goals in the Jira ticket, we just want to remove these jobs from running on each PR.

@thozza I wanted to ask you if you agree with this change. The goal here is to not run tests on GA releases on PRs but instead run them only on main and in a schedule similar to current nightly pipelines. This will make PR testing faster with minimum impact to quality if and an issue does occur on some of the GA releases we'll see it reported on main and in Slack along with nightly pipelines.

[thozza]

@thozza I wanted to ask you if you agree with this change. The goal here is to not run tests on GA releases on PRs but instead run them only on main and in a schedule similar to current nightly pipelines. This will make PR testing faster with minimum impact to quality if and an issue does occur on some of the GA releases we'll see it reported on main and in Slack along with nightly pipelines.

We should keep testing on PRs at least for GA releases used in our SaaS deployment. Otherwise, this LGTM.

@ondrejbudai @croissanne WDYT?

[thozza]

So, for the SaaS version, we need to keep testing the latest RHEL-9 GA release (9.4).

WRT the specific test cases, the following ones are relevant for the Saas:

• api.sh
• koji.sh
• multi-tenancy.sh
• regression-old-worker-new-composer.sh

@ondrejbudai @croissanne feel free to add / correct me.

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days.

[atodorov]

@tkoscieln can you execute the new so called GA pipeline and post a link here so we can compare the results ?

[tkoscieln]

I have updated the pipeline to re-add the tests specified by @thozza earlier.
How the pipeline looked before: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1347409921
How it looks now after I re-added some of the test cases: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1347426312
How the "GA" pipeline looks now:
https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1348619307

I have left the re-added test which now run in the PR pipeline again in the GA pipeline as well. What do you think @atodorov @jrusz ? Do we want to test them in both pipelines or just in the PR one?

[atodorov]

How the pipeline looked before:

https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1348643668 (from pull #4232) - 217 test jobs

How it looks now after I re-added some of the test cases: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1347426312

^^^ 271 test jobs - quite a lot more actually.

• Packer job is missing, see comment above. I'm not quite certain if it needs to be executed on PRs but the original conditions indicate so and you are making it the opposite.

• Not sure about Container either, need decision there.

The rest of the jobs in the list are the same, need to examine a bit better whether or not the expected runners are used.

How the "GA" pipeline looks now: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1348619307

What is the expected result here? That test jobs are executed only on GA runners and not on nightly runners, isn't it ?

[tkoscieln]

How the pipeline looked before:

https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1348643668 (from pull #4232) - 217 test jobs

How it looks now after I re-added some of the test cases: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1347426312

^^^ 271 test jobs - quite a lot more actually.

This number includes some cancelled jobs, so I checked and it is actually lower than 217.

• Packer job is missing, see comment above. I'm not quite certain if it needs to be executed on PRs but the original conditions indicate so and you are making it the opposite.
• Not sure about Container either, need decision there.

The rest of the jobs in the list are the same, need to examine a bit better whether or not the expected runners are used.

I left Packer and Container jobs as they were not specified to be re-added, but can add them back as well.

How the "GA" pipeline looks now: https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1348619307

What is the expected result here? That test jobs are executed only on GA runners and not on nightly runners, isn't it ?

Yes, it should contain jobs run on GA runners only, with some of the tests cases being extracted from the PR pipeline entirely to be run only in this GA pipeline.

[atodorov]

How the pipeline looked before:
^^^ 271 test jobs - quite a lot more actually.

This number includes some cancelled jobs, so I checked and it is actually lower than 217.

Can you git push this PR again so we can see the real number of jobs being scheduled.

I left Packer and Container jobs as they were not specified to be re-added, but can add them back as well.

Container job should be executed on PRs to detect changes which break container builds

from templates/packer/README.md:

This directory contains a packer configuration for building osbuild-composer
worker AMIs based on RHEL.

So it looks like this job, Packer, needs to be executed on pull requests as well, seems important to me.

[tkoscieln]

Ok, will add back the Packer and Container jobs and in the process trigger a new pipeline

[tkoscieln]

@atodorov it seems that with this optimalization we are down to 136 jobs in the PR pipeline from 217 including the Packer and Container jobs.

[atodorov]

LGTM

[jrusz]

Looks good, just some cosmetic changes requested.

[tkoscieln]

Pipelines results:
GA - pass - https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1373902418
PR - fail due to the API test (unrelated to these changes) - https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1373777282
NIGHTLY - pass - https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/pipelines/1373777209