Decouple regression test module from Fabric8 Maven Plugin

[rohanKanojia]

These regression tests were basically written to test build and deployment, redeployment aspects of fabric8 maven plugin. It sounds to have some overlap with integration tests in general for f8m-p. These were written to test overall integration of fabric8 maven plugin with fabric8io. These booster projects are some getting started applications which are on the Openshift.io page. Each test basically clones the repository from Github and test all the deployment/redeployment scenarios of fabric8 maven plugin.

But as pointed out by @rhuss in this pull request:

If I understand right, the regression tests are using various other booster projects which are in turn using f-m-p. This is a circular dependency which must not be there. It's not problem if f-m-p would have external test projects which are under full control of f-m-p and are created only for the purpose of testing f-m-p. However, testing other projects which can be developed independently must not be the scope of tests within f-m-p as they can break the f-m-p build any time (and without even being f-m-p fault). Also, f-m-p must not depend on any project using f-m-p, if you want to keep it as an independent product.

If anything changes in these booster projects, something breaks in these regression tests and whole CI starts breaking for no reason. It has happened two times already. So, It's not a good idea to keep them in fabric8 maven plugin source code. I have the following doubts:

• So even if we want to move this module outside fabric8 maven plugin, where should we keep it? Shall we keep it in fabric8io Github org? Or maybe the org which is maintaining those booster projects?

• How shall they be run? I can't think of any way of integrating them again with CI running on pull requests. Or maybe we can have some job which would run these tests on a daily/weekly basis triggering email alarms if something breaks.

[rohanKanojia]

@hrishin @rhuss Please share your thoughts on this.

[rhuss]

So even if we want to move this module outside fabric8 maven plugin, where should we keep it? Shall we keep it in fabric8io Github org? Or maybe the org which is maintaining those booster projects?

This really depends how you see this tests, but from my feeling they are closer to the boosters than fmp. I.e. the tests ensure that the boosters are not broken, which can happen when either fmp or the booster changes.

How shall they be run? I can't think of any way of integrating them again with CI running on pull requests. Or maybe we can have some job which would run these tests on a daily/weekly basis triggering email alarms if something breaks.

Ideally the test should run as soon as either fmp or the booster changes (i.e. when merging but also for the PRs). Not sure how is this is to achieve. Probably not with circleci, which also triggers build on changes on the repo with the change. But with a Jenkins CI job this should be possible.

The second best alternative would be to run it periodically and report it either to both groups (fmp and booster) when something fails, or only to the booster team, which then, after some analyses, would ping the fmp team if they identified an issue with fmp resource generation.

[hrishin]

@rhuss @rohanKanojia has summarized most of the aspects about regression tests.

Step 1:
To begin with, we can move the regression test module to the other/own repo in fabric8/right org and implement the scheduled test that could invoke tests periodically. In this way, we can achieve the decoupling objective without interrupting PR flows of F8MP and booster. Let's seek feedback from this test setup and see if it's sufficient to serve the testing purpose.

Step2:
If step1 is not sufficient then in subsequent steps, we can invoke tests on PR/Merge Github events from F8MP and eventually booster application repo. Where in PR chain from FMP/Booster repo to regression test repo could trigger a test job. Though need to evaluate but fabric8 Jenkins CI/CD has a bunch of utilities to achieve such kind of work.

cc: @piyush1594

[rhuss]

+1 for Step 1, guess this will be already a big step in the right direction.

Step 2 would be a nice automation enhancement which can be done then if required.

[rohanKanojia]

For step 1, Travis provides builds at regular intervals independently of whether any commits pushed to the repository. We're using travis for current setup, we can configure it to run periodically .

[rohanKanojia]

So I've raised a pull request for decoupling regression tests from fmp:-

I've also decoupled regression-tests from fabric8-maven-plugin into a separate repository which can be found here: https://github.com/rohanKanojia/fabric8-maven-plugin-rt

You can find it's build history here: https://travis-ci.org/rohanKanojia/fabric8-maven-plugin-rt/builds . We can configure Travis to run periodic builds against latest fmp master and email results to required stakeholders.

[rhuss]

@rohanKanojia awesome, very cool !

[piyush-garg]

@rohanKanojia Thanks for the PR.

I have two points in mind right now

1. It should be in fabric8io org I think. Not somewhere else.
2. I think we should run it for every PR on FMP so that the author of the PR should get an idea. Running periodic builds on master should not give any feedback to the author and will provide the feedback to stakeholders later that one of the last few PR's you have merged has some problem and is breaking. I think the feedback should be to the author that he/she can fix it in the same PR rather than stakeholders fixing later.

Correct me if I am wrong. Thanks @hrishin @rohanKanojia

[hrishin]

It should be in fabric8io org I think. Not somewhere else.

yes, appears like.

I think the feedback should be to the author that he/she can fix it in the same PR rather than stakeholders fixing later.

I see a good point here. We don't want to get rid of PR builds really, however, let's see how periodic builds work for regression tests and as mentioned will introduce it if required. Otherwise unit tests and integration tests are taking care of essential things.

[rhuss]

Cross-triggering of test across repos might be possible, but reporting back on the original PR as a GitHub status test is probably not that easy. We would have to write a step updating a status check via the GitHub API (including auth).

Also, how would we be able to run the integration tests over there when f-m-p is not pushed to a Maven repo because it is build because of a PR

Let's do it step-wise:

• Move the ITs into a separate repository
• The IT build needs to access the latest f-m-p source code for running the tests. That's a bit tricky.

One solution could be:

• For the build is on master for f-m-p we add an extra final step:
  • Update a file in the IT repo with the latest commit id of master (either directly via GitHub API & curl or by temporarily checking the IT repo out locally and git commit & push)
  • When this file gets committed and pushed and it is different, this will trigger a Travis/Circle CI build in the IT repo automatically (no need for a scheduled test)
  • The IT Test run will checkout exactly this version from the updated commit id, builds fabric8-maven-plugin locally and then runs the IT tests.
  • In case of an error the test should create an issue in fabric8-maven-plugin with the log and problem.

How does this sound ?

[rohanKanojia]

I'm building a SNAPSHOT image of fmp before tests execution:
https://github.com/rohanKanojia/fabric8-maven-plugin-rt/blob/master/src/main/resources/scripts/prepare-test-environment.sh#L62

[rhuss]

Yes, I've seen that. But still, you would need to trigger a scheduled build with Travis, which is probably not so easy (or even impossible). So might be better to communicate the concrete commit ID directly via a file which then automatically would trigger a test run.

And we could add a final step to create an issue in f-m-p repo if the build fails. Better than an E-Mail as somebody has to take care about.

[lordofthejars]

@rhuss @rohanKanojia We discussed something similar in Arquillian community, not the same but similar.

In our case what we wanted to do is that when a new release of arquillian core was done, Arquillian projects that depend on core, could be automatically tested with this new version.

With Travis it is possible to trigger any other build using the Rest API they are offering, even there are people who share a shell script that does it.

https://docs.travis-ci.com/user/triggering-builds/
https://github.com/stephanmg/travis-dependent-builds

So if you want you can skip the creation of the file and just trigger a build directly from Travis pipeline.