Add helper to get hosts on metricbeat integration tests #11925

jsoriano · 2019-04-24T15:47:38Z

Add a helper to automatically obtain the host of a service in metricbeat
system tests so tests don't depend on environment variables or
current running docker services. This is needed to allow in the future
to run multiple versions of the same service, is also helpful when
running tests locally.

Environment variables with the format SERVICE_HOST, being SERVICE
the uppercased name of the compose service can be used to override the
automatic value. By now OVERRIDE_HOST needs to be set to for these
variables to work in the python-based tests.

NO_COMPOSE environment variable can be used now to avoid stopping
the running containers too.

Add ports to all services in docker compose files, required for the
helper.

Part of #7957 for python-based integration tests.

Add a helper to automatically obtain the host of a service in metricbeat integration tests so tests don't depend on environment variables or current running docker services. This is needed to allow in the future to run multiple versions of the same service, is also helpful when running tests locally. Environment variables with the format SERVICE_HOST, being SERVICE the uppercased name of the compose service can be used to override the automatic value. `NO_COMPOSE` environment variable can be used now to avoid stopping the running containers too. Add ports to all services in docker compose files, required for the helper. Part of elastic#7957 for python-based integration tests.

jsoriano · 2019-04-24T16:27:41Z

jenkins, test this again please

…used

exekias

I like this change, it should make tests easier!

I wonder if we can detect ports as exposed in the Dockerfile, without explicitly exposing them in docker-compose.yml, or maybe that's too hidden

exekias · 2019-04-25T13:28:48Z

metricbeat/module/kafka/_meta/Dockerfile

 ENV KAFKA_LOGS_DIR="/kafka-logs"
 ENV _JAVA_OPTIONS "-Djava.net.preferIPv4Stack=true"
 ENV TERM=linux

-RUN apt-get update && apt-get install -y curl openjdk-8-jre-headless netcat dnsutils
+RUN apk add -u bash


changes to this file and run.sh probably belong to a different PR?

Changes here could probably be reverted, yes, I think they only reduced the size of the image.

Changes in run.sh are needed, kafka announces the addresses clients should use, current container only works if tests can connect to "kafka:9092". After this change services can be listening on random addresses and ports, so we need some plumbing to let kafka know what address it should announce.
But said that, not all changes are needed now, I got all changes done in #7957, that are intended also to be able to have multiple versions of the same service running at the same time, this is not needed till I merge the Go-part of this PR.

I will reduce the changes in these files to the ones really needed by now.

exekias · 2019-04-25T13:30:25Z

metricbeat/tests/system/test_jolokia.py

@@ -50,5 +50,4 @@ def test_jmx(self, mbean):
        assert evt["jolokia"]["test"]["gc"]["collection_count"] >= 0

    def get_hosts(self):
-        return [os.getenv('JOLOKIA_HOST', 'localhost') + ':' +
-                os.getenv('JOLOKIA_PORT', '8778')]
+        return [self.compose_host()]


I see this pattern a lot, you could also replace get_hosts() call with self.compose_host() and remove get_hosts all together??

Good suggestion, thanks! It should be possible, I will do it and override only when needed.

jsoriano · 2019-04-25T14:12:50Z

I wonder if we can detect ports as exposed in the Dockerfile, without explicitly exposing them in docker-compose.yml, or maybe that's too hidden

I tried, but I didn't manage to do it, it seems that docker compose only expose ports defined in the docker-compose.yml file. If there is a way to make it expose the ports defined in the Dockerfile we could consider doing it, yes.
In any case it is in my plans to move all the testing code of each module to the module itself, including the docker compose definitions. In many cases we are only using the Dockerfile to define the healthcheck and this can be done in the docker-compose.yml now, so many Dockerfiles might disappear in the future.

jsoriano · 2019-04-25T17:13:25Z

I have seen Kafka tests to fail a couple of times in travis, I am taking a look to this.

jsoriano · 2019-04-26T08:46:16Z

Ok, it fails because it is not ready yet to have multiple docker compose files in the same run, I will try to refactor Kafka docker files to don't need to bring here more changes from #7957

I don't know though why it doesn't fail on Jenkins.

…-python

jsoriano · 2019-04-26T16:39:42Z

Issues fixed by reverting more uneeded changes for kafka, this would be ready for another review.

exekias

Great improvement!

jsoriano added module Metricbeat Metricbeat :Testing Team:Integrations Label for the Integrations team labels Apr 24, 2019

jsoriano self-assigned this Apr 24, 2019

jsoriano force-pushed the compose-test-runner-python branch 2 times, most recently from 88cda5b to ede58db Compare April 24, 2019 16:03

jsoriano force-pushed the compose-test-runner-python branch from f4e0192 to 386432a Compare April 24, 2019 18:12

jsoriano added the [zube]: In Progress label Apr 24, 2019

jsoriano force-pushed the compose-test-runner-python branch from 4929f5c to 37dc4eb Compare April 25, 2019 08:24

Fix kafka python tests

16ddc0d

jsoriano force-pushed the compose-test-runner-python branch 2 times, most recently from dfcd3e2 to 0071b37 Compare April 25, 2019 09:44

jsoriano added 3 commits April 25, 2019 12:20

Fix MariaDB versions

a509fec

Fix advertised host for kafka integration tests in Go

e1005c9

Use private host in Linux so it works when host network is not being …

d1c93d0

…used

jsoriano force-pushed the compose-test-runner-python branch from 0071b37 to 3c411cf Compare April 25, 2019 10:55

Require a second variable to override hosts

cbed1fd

jsoriano force-pushed the compose-test-runner-python branch from 3c411cf to cbed1fd Compare April 25, 2019 11:01

jsoriano marked this pull request as ready for review April 25, 2019 12:19

jsoriano requested review from a team as code owners April 25, 2019 12:19

Don't change MariaDB versions in this branch

fcc2b56

jsoriano force-pushed the compose-test-runner-python branch from 8825fa9 to fcc2b56 Compare April 25, 2019 12:29

jsoriano added [zube]: In Review and removed [zube]: In Progress labels Apr 25, 2019

exekias reviewed Apr 25, 2019

View reviewed changes

jsoriano added 2 commits April 25, 2019 16:25

Implement get_hosts in the base class

cdd0574

Revert unneeded changes to Kafka docker images

a32c640

jsoriano force-pushed the compose-test-runner-python branch from b525479 to a32c640 Compare April 25, 2019 15:52

jsoriano added [zube]: In Progress in progress Pull request is currently in progress. and removed [zube]: In Review labels Apr 26, 2019

jsoriano added 2 commits April 26, 2019 16:18

Revert more unneeded changes for kafka module

dbe0610

Merge remote-tracking branch 'origin/master' into compose-test-runner…

0085449

…-python

jsoriano added [zube]: In Review review and removed [zube]: In Progress in progress Pull request is currently in progress. review labels Apr 26, 2019

zube bot added the review label Apr 26, 2019

exekias approved these changes Apr 29, 2019

View reviewed changes

jsoriano merged commit 7b28038 into elastic:master Apr 29, 2019

jsoriano deleted the compose-test-runner-python branch April 29, 2019 12:57

zube bot added [zube]: Done and removed [zube]: In Review labels Apr 29, 2019

alvarolobato removed the [zube]: Done label Apr 29, 2019

jsoriano mentioned this pull request Jul 15, 2019

Improvements in metricbeat modules integration tests #12909

Closed

6 tasks

jsoriano mentioned this pull request Jul 24, 2019

Add helper to get hosts on Go Metricbeat integration tests #13055

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add helper to get hosts on metricbeat integration tests #11925

Add helper to get hosts on metricbeat integration tests #11925

jsoriano commented Apr 24, 2019 •

edited

Loading

jsoriano commented Apr 24, 2019

exekias left a comment

exekias Apr 25, 2019

jsoriano Apr 25, 2019 •

edited

Loading

exekias Apr 25, 2019

jsoriano Apr 25, 2019

jsoriano commented Apr 25, 2019

jsoriano commented Apr 25, 2019

jsoriano commented Apr 26, 2019

jsoriano commented Apr 26, 2019

exekias left a comment

Add helper to get hosts on metricbeat integration tests #11925

Add helper to get hosts on metricbeat integration tests #11925

Conversation

jsoriano commented Apr 24, 2019 • edited Loading

jsoriano commented Apr 24, 2019

exekias left a comment

Choose a reason for hiding this comment

exekias Apr 25, 2019

Choose a reason for hiding this comment

jsoriano Apr 25, 2019 • edited Loading

Choose a reason for hiding this comment

exekias Apr 25, 2019

Choose a reason for hiding this comment

jsoriano Apr 25, 2019

Choose a reason for hiding this comment

jsoriano commented Apr 25, 2019

jsoriano commented Apr 25, 2019

jsoriano commented Apr 26, 2019

jsoriano commented Apr 26, 2019

exekias left a comment

Choose a reason for hiding this comment

jsoriano commented Apr 24, 2019 •

edited

Loading

jsoriano Apr 25, 2019 •

edited

Loading