From 33ecc2447cc6bc5fe5f1f7f870189edfeb49b13d Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Wed, 26 Jan 2022 16:54:41 +0900 Subject: [PATCH 1/9] Updated dev guide instructions to work with master filebeat fails (hangs) on 8.0 due to `Elasticsearch is too old.` so updating to 8.1 as well as fixing up expectations around authentication and ssl. --- docs/devguide/modules-dev-guide.asciidoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/devguide/modules-dev-guide.asciidoc b/docs/devguide/modules-dev-guide.asciidoc index 03b3179c596..70e0b8ef5af 100644 --- a/docs/devguide/modules-dev-guide.asciidoc +++ b/docs/devguide/modules-dev-guide.asciidoc @@ -485,8 +485,10 @@ run. In order to test the filesets with the sample logs and/or generate the expected output one should run the tests locally for a specific module, using the following procedure under Filebeat directory: -. Run an Elasticsearch instance locally using docker: `docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT` +. Run an Elasticsearch instance locally using docker: `docker run --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "ELASTIC_PASSWORD=changeme" -e "discovery.type=single-node" -e "xpack.security.http.ssl.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.1.0-SNAPSHOT` +. Create an "admin" user on that Elasticsearch instance: `curl -u elastic:changeme http://localhost:9200/_security/user/admin -X POST -H 'Content-Type: application/json' -d '{"password": "changeme", "roles": ["superuser"]}'` . Create python env: `make python-env` . Source python env: `source ./build/python-env/bin/activate` . Create the testing binary: `make filebeat.test` -. Run the test, ie: `GENERATE=1 INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=nginx pytest tests/system/test_modules.py` +. Run the test, ie: `GENERATE=1 INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=nginx ES_PASS=changeme pytest tests/system/test_modules.py -v --full-trace` +. Remove the `GENERATE=1` once you have the expected output, so you can modify it to meet ingest expectations. From 93c0f569a693db86c25be8a1abc1d5ca3d41e48c Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Mon, 31 Jan 2022 14:52:27 +0900 Subject: [PATCH 2/9] Add a note about log file tailing --- docs/devguide/modules-dev-guide.asciidoc | 45 +++++++++++++++++++++--- 1 file changed, 40 insertions(+), 5 deletions(-) diff --git a/docs/devguide/modules-dev-guide.asciidoc b/docs/devguide/modules-dev-guide.asciidoc index 70e0b8ef5af..2afcb418612 100644 --- a/docs/devguide/modules-dev-guide.asciidoc +++ b/docs/devguide/modules-dev-guide.asciidoc @@ -485,10 +485,45 @@ run. In order to test the filesets with the sample logs and/or generate the expected output one should run the tests locally for a specific module, using the following procedure under Filebeat directory: -. Run an Elasticsearch instance locally using docker: `docker run --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "ELASTIC_PASSWORD=changeme" -e "discovery.type=single-node" -e "xpack.security.http.ssl.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.1.0-SNAPSHOT` -. Create an "admin" user on that Elasticsearch instance: `curl -u elastic:changeme http://localhost:9200/_security/user/admin -X POST -H 'Content-Type: application/json' -d '{"password": "changeme", "roles": ["superuser"]}'` +. Start an Elasticsearch instance locally. For example, using Docker: ++ +[source,bash] +---- +docker run \ + --name elasticsearch \ + -p 9200:9200 -p 9300:9300 \ + -e "xpack.security.http.ssl.enabled=false" -e "ELASTIC_PASSWORD=changeme" \ + -e "discovery.type=single-node" \ + --pull allways --rm --detach \ + docker.elastic.co/elasticsearch/elasticsearch:master-SNAPSHOT +---- +. Create an "admin" user on that Elasticsearch instance: ++ +[source,bash] +---- +curl -u elastic:changeme \ + http://localhost:9200/_security/user/admin \ + -X POST -H 'Content-Type: application/json' \ + -d '{"password": "changeme", "roles": ["superuser"]}' +---- +. Create the testing binary: `make filebeat.test` +. Update fields yaml: `make update` . Create python env: `make python-env` . Source python env: `source ./build/python-env/bin/activate` -. Create the testing binary: `make filebeat.test` -. Run the test, ie: `GENERATE=1 INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=nginx ES_PASS=changeme pytest tests/system/test_modules.py -v --full-trace` -. Remove the `GENERATE=1` once you have the expected output, so you can modify it to meet ingest expectations. +. Run a test, for example to check nginx access log parsing: ++ +[source,bash] +---- +INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false ES_PASS=changeme \ +TESTING_FILEBEAT_MODULES=nginx \ +pytest tests/system/test_modules.py -v --full-trace +---- +. Add and remove option env vars as required. Here are some useful ones: +* `TESTING_FILEBEAT_MODULES`: comma separated list of modules to test. +* `TESTING_FILEBEAT_FILESETS`: comma separated list of filesets to test. +* `TESTING_FILEBEAT_FILEPATTERN`: glob pattern for log files within the fileset to test. +* `GENERATE`: if set to 1, the expected documents will be generated. + +The filebeat logs are writen to the `build` directory. It may be useful to tail them in another terminal using `tail -F build/system-tests/run/test_modules.Test.*/output.log`. + +For example if there's a syntax error in an ingest pipeline, the test will probably just hang. The filebeat log output will contain the error message from elasticsearch. From 8cba85c137c8da6a5450b879d5ffb7c20fa9768f Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Mon, 31 Jan 2022 14:56:47 +0900 Subject: [PATCH 3/9] Prefix packetbeat testing to avoid conflict with beats testing page --- docs/devguide/new_protocol.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/devguide/new_protocol.asciidoc b/docs/devguide/new_protocol.asciidoc index c653ff86316..f48152a5b3b 100644 --- a/docs/devguide/new_protocol.asciidoc +++ b/docs/devguide/new_protocol.asciidoc @@ -95,7 +95,7 @@ We are working on updating this section. While you're waiting for updates, you might want to try out the TCP protocol generator at https://github.com/elastic/beats/tree/master/packetbeat/scripts/tcp-protocol. -[[testing]] +[[protocol-testing]] === Testing -We are working on updating this section. +We are working on updating this section. From 8bdc6d2d38c8762582dd27be66dc3053b5e61346 Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Mon, 31 Jan 2022 15:01:37 +0900 Subject: [PATCH 4/9] Linking the filebeat module guide to the test docs --- docs/devguide/testing.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/devguide/testing.asciidoc b/docs/devguide/testing.asciidoc index 75a476d5c5a..e907e9f3b14 100644 --- a/docs/devguide/testing.asciidoc +++ b/docs/devguide/testing.asciidoc @@ -34,6 +34,8 @@ To run the tests without the services running locally, the command `make system- All Python tests are under `tests/system` directory. +Filebeat's module python tests have additional documentation found in the <> guide. + ==== Test commands This is a quick summary of the available test commands: From 5d502c13bc142a73840c714915f8ad44d203eb57 Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Tue, 1 Feb 2022 11:22:44 +0900 Subject: [PATCH 5/9] Fix trailing space --- docs/devguide/contributing.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/devguide/contributing.asciidoc b/docs/devguide/contributing.asciidoc index 435eb10416b..ae9e4a1ee8e 100644 --- a/docs/devguide/contributing.asciidoc +++ b/docs/devguide/contributing.asciidoc @@ -83,7 +83,7 @@ and documentation. The primary command used for this is: -------------------------------------------------------------------------------- make update -------------------------------------------------------------------------------- -Each Beat has its own `update` target, as well as a master `update` in the repository root. +Each Beat has its own `update` target, as well as a master `update` in the repository root. If a PR adds or removes a dependency, run `make update` in the root `beats` directory. Another command properly formats go source files and adds a copyright header: From 0f177e28c59c2c9ec8f0c402cb2ae01524a90982 Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Tue, 1 Feb 2022 11:23:27 +0900 Subject: [PATCH 6/9] Include testing doc in index --- docs/devguide/index.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/devguide/index.asciidoc b/docs/devguide/index.asciidoc index 6810240121c..2f9af959ce1 100644 --- a/docs/devguide/index.asciidoc +++ b/docs/devguide/index.asciidoc @@ -19,6 +19,8 @@ include::./pull-request-guidelines.asciidoc[] include::./contributing.asciidoc[] +include::./testing.asciidoc[] + include::{libbeat-dir}/communitybeats.asciidoc[] include::./fields-yml.asciidoc[] @@ -36,5 +38,3 @@ include::./metricbeat-devguide.asciidoc[] include::./modules-dev-guide.asciidoc[] include::./migrate-dashboards.asciidoc[] - - From ff386c6726869a1170d2ea4f7901e8877ea840d7 Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Tue, 1 Feb 2022 11:23:48 +0900 Subject: [PATCH 7/9] Fix protocol testing link reference --- docs/devguide/new_protocol.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/devguide/new_protocol.asciidoc b/docs/devguide/new_protocol.asciidoc index f48152a5b3b..defd50c0bc3 100644 --- a/docs/devguide/new_protocol.asciidoc +++ b/docs/devguide/new_protocol.asciidoc @@ -5,7 +5,7 @@ The following topics describe how to add a new protocol to Packetbeat: * <> * <> -* <> +* <> [[getting-ready-new-protocol]] === Getting Ready From c9dff1b3dbf0b16f4bacaf628c896677877050f5 Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Tue, 1 Feb 2022 11:24:02 +0900 Subject: [PATCH 8/9] Cleaner link wording --- docs/devguide/testing.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/devguide/testing.asciidoc b/docs/devguide/testing.asciidoc index e907e9f3b14..4f8a138beb3 100644 --- a/docs/devguide/testing.asciidoc +++ b/docs/devguide/testing.asciidoc @@ -34,7 +34,7 @@ To run the tests without the services running locally, the command `make system- All Python tests are under `tests/system` directory. -Filebeat's module python tests have additional documentation found in the <> guide. +Filebeat's module python tests have additional documentation found in the <> guide. ==== Test commands From 88ba36ac7bfd9b487ec803f676f848b4f6bd6550 Mon Sep 17 00:00:00 2001 From: Mat Schaffer Date: Tue, 1 Feb 2022 11:45:47 +0900 Subject: [PATCH 9/9] Document TESTING_FILEBEAT_ALLOW_OLDER --- docs/devguide/modules-dev-guide.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/devguide/modules-dev-guide.asciidoc b/docs/devguide/modules-dev-guide.asciidoc index 2afcb418612..ee264b15741 100644 --- a/docs/devguide/modules-dev-guide.asciidoc +++ b/docs/devguide/modules-dev-guide.asciidoc @@ -519,6 +519,7 @@ TESTING_FILEBEAT_MODULES=nginx \ pytest tests/system/test_modules.py -v --full-trace ---- . Add and remove option env vars as required. Here are some useful ones: +* `TESTING_FILEBEAT_ALLOW_OLDER`: if set to 1, allow connecting older versions of Elasticsearch * `TESTING_FILEBEAT_MODULES`: comma separated list of modules to test. * `TESTING_FILEBEAT_FILESETS`: comma separated list of filesets to test. * `TESTING_FILEBEAT_FILEPATTERN`: glob pattern for log files within the fileset to test.