Integration tests stub #2951

rhcarvalho · 2016-12-08T12:51:30Z

In the past few weeks, @brenton @sosiouxme and I have been working on several pre-flight checks: playbook(s) that verify certain bits of the environment that folks might run before an installation, upgrade or to debug an existing cluster.

We've been also experimenting with an additional approach to integration testing of such playbooks.

I'd like to start bringing our work gradually to make it easier to review, and get better feedback.

I've cherry-picked commits from our work-branch related to the integration tests, for I think this is an easier piece to grasp and part of the foundation of our tests.
More tests (here is just a demo of the concept) and the playbooks will come in separate PRs.

rhcarvalho · 2016-12-08T12:52:38Z

@abutcher @detiber PTAL

@brenton and I would love your feedback!

rhcarvalho · 2016-12-08T12:54:01Z

.travis.yml

@@ -11,3 +11,4 @@ script:
  # TODO(rhcarvalho): check syntax of other important entrypoint playbooks
  - ansible-playbook --syntax-check playbooks/byo/config.yml
  - cd utils && make ci
+  - make test-integration


We have already some integration with vagrant-openshift to run these tests in our Jenkins, but before we transition it to point to openshift/openshift-ansible, Travis can demo a test run.

If there are tests that we can run under travis-ci, then I think we should. We should only need to run tests on our internal jenkins if they require access to internal content/resources.

make test-integration is running from the utils directory, probably want to do something like the following instead:

- ansible-playbook --syntax-check playbooks/byo/config.yml - pushd utils - make ci - popd - make test-integration

Problem solved.

rhcarvalho · 2016-12-08T12:55:24Z

utils/setup.cfg

-tests=../,../roles/openshift_master_facts/test/,test/
+tests = ../test/unit,
+        .,
+        ../roles/openshift_master_facts/test,


Could have been a separate PR... I realized we were missing 2 tests via cd utils; make ci.

rhcarvalho · 2016-12-08T13:13:06Z

Makefile

@@ -0,0 +1,7 @@
+.PHONY: test
+test:
+	nosetests


@detiber @tbielawa when we move utils/Makefile into the top-level, we can have make test be the one to give very quick feedback during development, while keeping the make ci target doing all the things it does today (setup venv, unit tests, flake8, pylint).

I would probably argue that we should be running unit-tests and flake8 for make test, since both ideally will give quick feedback.

For now we're still using (cd utils; make ...); let's move utils/Makefile -> Makefile in a separate PR and we can have a more focused discussion of what runs when.

detiber · 2016-12-08T22:38:02Z

I'll take a closer look at this tonight/tomorrow, but I wanted to share this article I came across the other day talking about testing with Ansible: https://medium.com/@sebinjohn/testing-ansible-roles-using-molecule-and-vagrant-a15c22af23ab#.8yu427q47

detiber · 2016-12-09T01:59:25Z

.travis.yml

@@ -11,3 +11,4 @@ script:
  # TODO(rhcarvalho): check syntax of other important entrypoint playbooks
  - ansible-playbook --syntax-check playbooks/byo/config.yml
  - cd utils && make ci
+  - make test-integration


If there are tests that we can run under travis-ci, then I think we should. We should only need to run tests on our internal jenkins if they require access to internal content/resources.

detiber · 2016-12-09T02:00:28Z

Makefile

@@ -0,0 +1,7 @@
+.PHONY: test
+test:
+	nosetests


I would probably argue that we should be running unit-tests and flake8 for make test, since both ideally will give quick feedback.

detiber · 2016-12-09T02:09:41Z

.travis.yml

@@ -11,3 +11,4 @@ script:
  # TODO(rhcarvalho): check syntax of other important entrypoint playbooks
  - ansible-playbook --syntax-check playbooks/byo/config.yml
  - cd utils && make ci
+  - make test-integration


make test-integration is running from the utils directory, probably want to do something like the following instead:

- ansible-playbook --syntax-check playbooks/byo/config.yml - pushd utils - make ci - popd - make test-integration

detiber · 2016-12-09T02:11:46Z

setup.cfg

+[nosetests]
+tests = test/unit,
+        utils,
+        roles/openshift_master_facts/test,


should roles/openshift_master_facts/test/* be moved to roles/openshift_master_facts/test/unit/ as well?

Possibly not. I moved the Python files under test/ just to have a better organization and reuse the test/ top-level dir.

@detiber, I would even consider moving test/unit/modify_yaml_tests.py -> library/test_modify_yaml.py, so that test files live close to what they test.

consider moving test/unit/modify_yaml_tests.py -> library/test_modify_yaml.py, so that test files live close to what they test

@detiber what do you think about that idea?

@rhcarvalho I'm not sure how well that would work, since then the test would be in the Ansible module lookup path.

detiber · 2016-12-09T02:23:54Z

test/integration/preflight/preflight_test.go

@@ -0,0 +1,120 @@
+package preflight


Can't say I'm a huge fan of using golang for testing here, seems like it is going to be difficult to get external contributors to participate if the tests are written in golang.

I'd much prefer that we move towards a framework like https://molecule.readthedocs.io instead, since it also allows for much more comprehensive testing against multiple machines, etc.

Can't say I'm a huge fan of using golang for testing here, seems like it is going to be difficult to get external contributors to participate if the tests are written in golang.

Why do you think so?

I'd much prefer that we move towards a framework like https://molecule.readthedocs.io

Thanks for the link. I see how it could be useful for us in end-to-end testing of the many playbooks in openshift-ansible (installation, upgrades, one-off actions). IIUC it's a YAML-centric framework for a particular flow of testing playbooks? I see it can do check syntax, convergence, idempotence, verify final state.

For testing the preflight checks, what we need is a way to run openshift-ansible someplaybook.yml and check the exit code and output. Each test playbook will setup a dynamic inventory composed of one or more containers, and then will include another playbook from the preflight checks, to ensure that the check work correctly under certain environments (e.g. system with missing subscriptions, or incompatible versions of packages).

We wanted something that could give us fast feedback to iterate on the checks. Thus, we use Docker images instead of VMs. Running the tests in parallel seemed desirable, and Go's testing let us do that without much hassle.

Open to suggestions.

Can't say I'm a huge fan of using golang for testing here, seems like it is going to be difficult to get external contributors to participate if the tests are written in golang.

Why do you think so?

Most external contributors will not be familiar with golang, If they are already advanced Ansible users, then we can generally assume they are familiar with Python.

I'd much prefer that we move towards a framework like https://molecule.readthedocs.io

Thanks for the link. I see how it could be useful for us in end-to-end testing of the many playbooks in openshift-ansible (installation, upgrades, one-off actions). IIUC it's a YAML-centric framework for a particular flow of testing playbooks? I see it can do check syntax, convergence, idempotence, verify final state.

The verify final state is the one that allows for arbitrary validations to be run.

For testing the preflight checks, what we need is a way to run openshift-ansible someplaybook.yml and check the exit code and output. Each test playbook will setup a dynamic inventory composed of one or more containers, and then will include another playbook from the preflight checks, to ensure that the check work correctly under certain environments (e.g. system with missing subscriptions, or incompatible versions of packages).

I'll play around with it some, I'm pretty sure this use case is baked in, even if the docs are lacking in how to do it.

We wanted something that could give us fast feedback to iterate on the checks. Thus, we use Docker images instead of VMs. Running the tests in parallel seemed desirable, and Go's testing let us do that without much hassle.

Molecule is able to use Docker as a backend as well. I'll have to play around with the parallelization, but I think we can accomplish something similar.

rhcarvalho · 2017-03-07T16:05:24Z

.travis.yml

 script:
-  - tox
+  # REVIEW(rhcarvalho): if our Travis account doesn't support more than one
+  # build at a time, there is no point in parallelizing unit/integration tests.


Our account can do 2 concurrent builds across all openshift/* repos.

To make room for integration tests.

rhcarvalho · 2017-03-08T13:06:07Z

Updated to use tox and have some examples of what tests could look like:

automate setting up containers for a dynamic test inventory
run a playbook
make assertions about exit code and output

detiber · 2017-03-09T17:23:36Z

tox.ini

+    # Unset GOPATH because tests use relative imports. This should be removed if
+    # we require openshift-ansible to live in a Go work space and use absolute
+    # imports in tests (desirable).
+    integration: env -u GOPATH go test -v ./test/integration/...


This doesn't seem like it should be needed: http://tox.readthedocs.io/en/latest/config.html#confval-passenv=SPACE-SEPARATED-GLOBNAMES

Hmm, I failed to see how that would be used?

Did you mean something like

[testenv:integration] setenv= GOPATH=

?

But then, that looks like a lot more to unset an envvar than using env -u?

I'm unsetting GOPATH just to not require the code to be within GOPATH, and use relative imports. Relative imports are notably a bad idea in Go, I just wanted to experiment and see how it would compare. This basically affects CI in that we need to decide if the checkout path is within a Go work space or not. And we need to decide between Travis and Jenkins...

If I'm reading the tox docs correctly, the GOPATH env var shouldn't be available in the test environment with the default config. Outside of a handful of whitelisted vars, you should have to specify which env cars you want to make available using passenv or by explicitly defining them with setenv.

Ah, okay, now I understand.
I'll update it.

openshift-bot · 2017-04-16T00:23:12Z

OpenShift Ansible Action Required: Pull request cannot be automatically merged, please rebase your branch from latest HEAD and push again

rhcarvalho · 2017-05-05T11:13:57Z

This was merged through #3816.

rhcarvalho requested review from abutcher and detiber December 8, 2016 12:51

rhcarvalho commented Dec 8, 2016

View reviewed changes

detiber reviewed Dec 9, 2016

View reviewed changes

rhcarvalho force-pushed the integration-stub branch from 4f9fd2f to 1ac275e Compare December 9, 2016 08:53

rhcarvalho mentioned this pull request Feb 14, 2017

Use tox and molecule for Ansible testing #2998

Closed

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 4, 2017

rhcarvalho force-pushed the integration-stub branch from 1ac275e to d87361b Compare March 7, 2017 13:52

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 7, 2017

rhcarvalho force-pushed the integration-stub branch from d87361b to 94596cd Compare March 7, 2017 16:04

rhcarvalho commented Mar 7, 2017

View reviewed changes

rhcarvalho added 2 commits March 8, 2017 14:02

Move Python unit tests to subdirectory

45708d8

To make room for integration tests.

Add stub of preflight integration tests

5d846c2

rhcarvalho force-pushed the integration-stub branch from 94596cd to 5d846c2 Compare March 8, 2017 13:02

detiber reviewed Mar 9, 2017

View reviewed changes

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 16, 2017

rhcarvalho closed this May 5, 2017

rhcarvalho deleted the integration-stub branch May 5, 2017 11:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration tests stub #2951

Integration tests stub #2951

rhcarvalho commented Dec 8, 2016

rhcarvalho commented Dec 8, 2016

rhcarvalho Dec 8, 2016

detiber Dec 9, 2016

detiber Dec 9, 2016

rhcarvalho Dec 9, 2016

rhcarvalho Dec 8, 2016

rhcarvalho Dec 8, 2016

detiber Dec 9, 2016

rhcarvalho Dec 9, 2016

detiber commented Dec 8, 2016

detiber Dec 9, 2016

detiber Dec 9, 2016

detiber Dec 9, 2016

detiber Dec 9, 2016

rhcarvalho Dec 9, 2016

rhcarvalho Dec 9, 2016

detiber Dec 9, 2016

detiber Dec 9, 2016

rhcarvalho Dec 9, 2016

detiber Dec 9, 2016

rhcarvalho Mar 7, 2017

rhcarvalho commented Mar 8, 2017

detiber Mar 9, 2017

rhcarvalho Mar 10, 2017

detiber Mar 10, 2017

rhcarvalho Mar 10, 2017

openshift-bot commented Apr 16, 2017

rhcarvalho commented May 5, 2017

Integration tests stub #2951

Integration tests stub #2951

Conversation

rhcarvalho commented Dec 8, 2016

rhcarvalho commented Dec 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

detiber commented Dec 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhcarvalho commented Mar 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-bot commented Apr 16, 2017

rhcarvalho commented May 5, 2017