Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the Docker image to be run with a random user id #12905

Merged
merged 4 commits into from
May 19, 2020

Conversation

barkbay
Copy link
Contributor

@barkbay barkbay commented Jul 15, 2019

On secured Kubernetes environments (not only Openshift) the user ID used to run a container can't be known in advance. Consequently the APM server container can't be started on these environments because it expects to run with the user 1000 or 0.

This PR brings some compatibility with such environments, based on the fact that on secured Kubernetes clusters and on Openshift the only thing you know is that the user is always a member of the root group.

You can find more details here: https://docs.openshift.com/container-platform/3.11/creating_images/guidelines.html#openshift-specific-guidelines

See elastic/apm-server#2325 and #12686

How to test

  • Do it with auditbeat (that uses the root user by default), and with some other beat like metricbeat or filebeat (that use a non-root user by default):
    • Build the docker image for the beat with PLATFORMS=linux/amd64 mage package.
    • Run it in docker and/or kubernetes with some basic configuration or with some of the ones in deploy directory.
    • Check that it is working correctly, without BEAT_STRICT_PERMS.
  • Check in some scenario that beats can write their state to data directories mounted from the host.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@elasticmachine
Copy link
Collaborator

elasticmachine commented May 8, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 0
Passed 7660
Skipped 1231
Total 8891

Steps errors

Expand to view the steps failures

  • Name: Make -C libbeat testsuite
    • Description: make -C libbeat testsuite

    • Result: FAILURE

    • Duration: 25 min 28 sec

    • Start Time: 2020-05-14T10:48:41.047+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-14T11:33:22.882Z] + FILE=heartbeat/build/coverage/full.cov
[2020-05-14T11:33:22.882Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-05-14T11:33:22.882Z] + FILE=libbeat/build/coverage/full.cov
[2020-05-14T11:33:22.882Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-05-14T11:33:22.882Z] + FILE=metricbeat/build/coverage/full.cov
[2020-05-14T11:33:22.882Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-05-14T11:33:22.882Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-14T11:33:22.882Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-14T11:33:22.882Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-14T11:33:22.882Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-14T11:33:22.882Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-14T11:33:22.882Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-14T11:33:23.426Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats
[2020-05-14T11:33:23.739Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-14T11:33:23.753Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Lint
[2020-05-14T11:33:23.834Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-05-14T11:33:23.915Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Winlogbeat-oss
[2020-05-14T11:33:23.996Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-05-14T11:33:24.079Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Dockerlogbeat
[2020-05-14T11:33:24.166Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-05-14T11:33:24.247Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-14T11:33:24.345Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Journalbeat-oss
[2020-05-14T11:33:24.439Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-05-14T11:33:24.523Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-05-14T11:33:24.614Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-05-14T11:33:24.708Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-14T11:33:24.784Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-14T11:33:24.861Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Heartbeat-oss
[2020-05-14T11:33:24.941Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-05-14T11:33:25.026Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-05-14T11:33:25.106Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Libbeat-x-pack
[2020-05-14T11:33:25.189Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-05-14T11:33:25.292Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-Windows
[2020-05-14T11:33:25.387Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-05-14T11:33:25.477Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-Linux
[2020-05-14T11:33:25.557Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Packetbeat-oss
[2020-05-14T11:33:25.645Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-14T11:33:25.733Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-05-14T11:33:25.816Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-05-14T11:33:25.894Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-05-14T11:33:25.985Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-x-pack
[2020-05-14T11:33:26.088Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-Mac-OS-X
[2020-05-14T11:33:26.181Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Libbeat-oss
[2020-05-14T11:33:26.327Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-oss
[2020-05-14T11:33:26.404Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Functionbeat-Windows
[2020-05-14T11:33:26.504Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Heartbeat-Windows
[2020-05-14T11:33:26.612Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-14T11:33:26.707Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-14T11:33:26.806Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-05-14T11:33:26.897Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-05-14T11:33:26.995Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-Windows
[2020-05-14T11:33:27.112Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-14T11:33:27.528Z] + cat
[2020-05-14T11:33:27.528Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-14T11:33:27.528Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-14T11:33:34.144Z] runbld>>> runbld started
[2020-05-14T11:33:34.144Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-14T11:33:36.079Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-12905' in order of occurrence in the config (last value wins).
[2020-05-14T11:33:37.466Z] runbld>>> Debug logging enabled.
[2020-05-14T11:33:37.466Z] runbld>>> Storing result
[2020-05-14T11:33:37.729Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-14T11:33:37.729Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200514113337-93C3D959
[2020-05-14T11:33:37.729Z] runbld>>> Adding system facts.
[2020-05-14T11:33:38.678Z] runbld>>> Adding vcs info for the latest commit:  3344bdebfb2067028bab568ec02dfa839a812761
[2020-05-14T11:33:38.678Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-14T11:33:38.678Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-14T11:33:38.678Z] Processing JUnit reports with runbld...
[2020-05-14T11:33:38.678Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-14T11:33:39.254Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-14T11:33:39.254Z] runbld>>> DURATION: 16ms
[2020-05-14T11:33:39.254Z] runbld>>> STDOUT: 40 bytes
[2020-05-14T11:33:39.254Z] runbld>>> STDERR: 49 bytes
[2020-05-14T11:33:39.254Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-14T11:33:39.254Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats
[2020-05-14T11:33:40.647Z] runbld>>> Storing build metadata: 
[2020-05-14T11:33:40.647Z] runbld>>> Adding test report.
[2020-05-14T11:33:40.647Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats
[2020-05-14T11:33:42.048Z] runbld>>> Found 102 test output files
[2020-05-14T11:33:42.624Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-05-14T11:33:42.624Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-05-14T11:33:42.624Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-05-14T11:33:42.624Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-05-14T11:33:42.624Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-05-14T11:33:42.624Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-05-14T11:33:43.575Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-05-14T11:33:43.575Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-05-14T11:33:43.838Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 8741 Skipped: 1015
[2020-05-14T11:33:44.109Z] runbld>>> Storing result
[2020-05-14T11:33:44.109Z] runbld>>> FAILURES: 0
[2020-05-14T11:33:44.370Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-14T11:33:44.370Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200514113337-93C3D959
[2020-05-14T11:33:44.370Z] runbld>>> Email notification disabled by environment variable.
[2020-05-14T11:33:44.370Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-14T11:33:49.950Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905
[2020-05-14T11:33:50.079Z] [INFO] getVaultSecret: Getting secrets
[2020-05-14T11:33:50.146Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-14T11:33:50.986Z] + chmod 755 generate-build-data.sh
[2020-05-14T11:33:50.986Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/runs/135 FAILURE 4455646
[2020-05-14T11:33:51.537Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/runs/135/steps/?limit=10000 -o steps-info.json
[2020-05-14T11:33:54.021Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/runs/135/tests/?status=FAILED -o tests-errors.json

@exekias
Copy link
Contributor

exekias commented May 11, 2020

It seems we missed this one, sorry about that.

This looks good to me, wdyt @jsoriano?

jenkins, test this please

@jsoriano
Copy link
Member

Umm, this branch will need an update with master so we can run the packaging tests. I have tried it locally and some checks fail:

>> Testing package contents
--- FAIL: TestDocker (3.94s)
    --- FAIL: TestDocker/auditbeat-oss-8.0.0-linux-amd64.docker.tar.gz_config_file_permissions (0.00s)
        package_test.go:206: file usr/share/auditbeat/auditbeat.yml has wrong permissions: expected=-rw-r----- actual=-rw-rw----

Is it needed to give write permissions to the group?

@jsoriano
Copy link
Member

Is it needed to give write permissions to the group?

Answering myself, this seems to be the recommendation from Openshift, so I am ok with the change, but we should update the checks.

@barkbay
Copy link
Contributor Author

barkbay commented May 13, 2020

Hi, thanks for the feedback.
I'll update this PR shortly.

@jsoriano
Copy link
Member

run beats-ci/package

@jsoriano
Copy link
Member

Packaging tests failed for filebeat, something else seems to be needed.
https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-package-linux/67/console

@barkbay
Copy link
Contributor Author

barkbay commented May 18, 2020

I will update the PR to fix that.
In the meantime I'm building the APM Server from this branch and I'm doing some tests on Openshift to confirm that it is also fixing the root issue.

@barkbay
Copy link
Contributor Author

barkbay commented May 18, 2020

Filebeat oss Linux failed with following message:

[2020-05-18T08:44:50.063Z] ======================================================================
[2020-05-18T08:44:50.063Z] ERROR: Test reload same config with same file but different config. Makes sure reloading also works on conflicts.
[2020-05-18T08:44:50.063Z] ----------------------------------------------------------------------
[2020-05-18T08:44:50.063Z] Traceback (most recent call last):
[2020-05-18T08:44:50.063Z]   File "/go/src/github.com/elastic/beats/filebeat/tests/system/test_reload_inputs.py", line 284, in test_reload_same_config
[2020-05-18T08:44:50.063Z]     max_timeout=15)
[2020-05-18T08:44:50.063Z]   File "/go/src/github.com/elastic/beats/filebeat/tests/system/../../../libbeat/tests/system/beat/beat.py", line 365, in wait_until
[2020-05-18T08:44:50.063Z]     "Waited {} seconds.".format(max_timeout))
[2020-05-18T08:44:50.063Z] beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 15 seconds.

Not sure if it is related to the PR.

@elasticmachine
Copy link
Collaborator

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 1
Passed 9210
Skipped 1542
Total 10753

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat oss / test_reload_same_config – test_reload_inputs.Test

    • Status: FAILED
    • Age: 1
    • Duration: 16.271
    • Error Details: Timeout waiting for 'cond' to be true. Waited 15 seconds.

Steps errors

Expand to view the steps failures

  • Name: Make -C filebeat testsuite

    • Description: make -C filebeat testsuite

    • Result: FAILURE

    • Duration: 28 min 22 sec

    • Start Time: 2020-05-18T08:16:29.075+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Result: FAILURE

    • Duration: 2 min 22 sec

    • Start Time: 2020-05-18T08:27:16.863+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Result: FAILURE

    • Duration: 0 min 10 sec

    • Start Time: 2020-05-18T08:50:47.451+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-05-18T09:09:49.347Z] + FILE=packetbeat/build/coverage/full.cov
[2020-05-18T09:09:49.347Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-05-18T09:09:49.347Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-05-18T09:09:49.347Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-05-18T09:09:49.347Z] + FILE=journalbeat/build/coverage/full.cov
[2020-05-18T09:09:49.347Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-05-18T09:09:49.928Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats
[2020-05-18T09:09:50.250Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-05-18T09:09:50.266Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Lint
[2020-05-18T09:09:50.390Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-05-18T09:09:50.482Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Winlogbeat-oss
[2020-05-18T09:09:50.586Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-05-18T09:09:50.683Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Dockerlogbeat
[2020-05-18T09:09:50.790Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Journalbeat-oss
[2020-05-18T09:09:50.909Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-05-18T09:09:51.085Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-05-18T09:09:51.228Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-05-18T09:09:51.339Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-05-18T09:09:51.419Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-05-18T09:09:51.506Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-05-18T09:09:51.587Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Heartbeat-oss
[2020-05-18T09:09:51.677Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-05-18T09:09:51.769Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-05-18T09:09:51.855Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-05-18T09:09:51.944Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-05-18T09:09:52.019Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Libbeat-x-pack
[2020-05-18T09:09:52.232Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-Linux
[2020-05-18T09:09:52.308Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-Windows
[2020-05-18T09:09:52.378Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-05-18T09:09:52.476Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-05-18T09:09:52.556Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Packetbeat-oss
[2020-05-18T09:09:52.635Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-05-18T09:09:52.710Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-05-18T09:09:52.775Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-Windows
[2020-05-18T09:09:52.857Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-05-18T09:09:52.944Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-05-18T09:09:53.031Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-05-18T09:09:53.118Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-x-pack
[2020-05-18T09:09:53.192Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-Mac-OS-X
[2020-05-18T09:09:53.274Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Functionbeat-Windows
[2020-05-18T09:09:53.360Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Filebeat-oss
[2020-05-18T09:09:53.446Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-05-18T09:09:53.538Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Libbeat-oss
[2020-05-18T09:09:53.612Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-05-18T09:09:53.680Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-05-18T09:09:53.751Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Heartbeat-Windows
[2020-05-18T09:09:53.831Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-05-18T09:09:53.897Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-05-18T09:09:53.964Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-05-18T09:09:54.033Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-05-18T09:09:54.101Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Auditbeat-Windows
[2020-05-18T09:09:54.176Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-05-18T09:09:54.532Z] + cat
[2020-05-18T09:09:54.532Z] + /usr/local/bin/runbld ./runbld-script
[2020-05-18T09:09:54.532Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-05-18T09:10:01.130Z] runbld>>> runbld started
[2020-05-18T09:10:01.130Z] runbld>>> 1.6.11/a66728ff8f4356963772e6e6d2069392fa06acbe
[2020-05-18T09:10:02.520Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-12905' in order of occurrence in the config (last value wins).
[2020-05-18T09:10:03.910Z] runbld>>> Debug logging enabled.
[2020-05-18T09:10:03.910Z] runbld>>> Storing result
[2020-05-18T09:10:04.173Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-05-18T09:10:04.173Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200518091003-69C16C3C
[2020-05-18T09:10:04.173Z] runbld>>> Adding system facts.
[2020-05-18T09:10:05.125Z] runbld>>> Adding vcs info for the latest commit:  e249113f93056ddf42a3e57500b8339e6b35033f
[2020-05-18T09:10:05.125Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-05-18T09:10:05.125Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-05-18T09:10:05.125Z] Processing JUnit reports with runbld...
[2020-05-18T09:10:05.125Z] + echo 'Processing JUnit reports with runbld...'
[2020-05-18T09:10:05.705Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-05-18T09:10:05.705Z] runbld>>> DURATION: 28ms
[2020-05-18T09:10:05.705Z] runbld>>> STDOUT: 40 bytes
[2020-05-18T09:10:05.705Z] runbld>>> STDERR: 49 bytes
[2020-05-18T09:10:05.705Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-05-18T09:10:05.705Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats
[2020-05-18T09:10:07.093Z] runbld>>> Storing build metadata: 
[2020-05-18T09:10:07.093Z] runbld>>> Adding test report.
[2020-05-18T09:10:07.093Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats
[2020-05-18T09:10:08.040Z] runbld>>> Found 111 test output files
[2020-05-18T09:10:08.613Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-05-18T09:10:08.613Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-05-18T09:10:08.613Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-05-18T09:10:08.613Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-05-18T09:10:08.881Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-05-18T09:10:08.881Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-cloudfoundry.xml
[2020-05-18T09:10:09.837Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-05-18T09:10:09.837Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-05-18T09:10:10.414Z] runbld>>> Test output logs contained: Errors: 1 Failures: 0 Tests: 10603 Skipped: 1304
[2020-05-18T09:10:10.414Z] runbld>>> Storing result
[2020-05-18T09:10:10.414Z] runbld>>> FAILURES: 1
[2020-05-18T09:10:10.987Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-05-18T09:10:10.987Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200518091003-69C16C3C
[2020-05-18T09:10:10.987Z] runbld>>> Email notification disabled by environment variable.
[2020-05-18T09:10:10.987Z] runbld>>> Slack notification disabled by environment variable.
[2020-05-18T09:10:16.918Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-12905
[2020-05-18T09:10:17.024Z] [INFO] getVaultSecret: Getting secrets
[2020-05-18T09:10:17.076Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-05-18T09:10:17.892Z] + chmod 755 generate-build-data.sh
[2020-05-18T09:10:17.892Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/runs/137 FAILURE 4617192
[2020-05-18T09:10:18.443Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/runs/137/steps/?limit=10000 -o steps-info.json
[2020-05-18T09:10:19.354Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-12905/runs/137/tests/?status=FAILED -o tests-errors.json

@jsoriano
Copy link
Member

run beats-ci/package

@jsoriano
Copy link
Member

Thanks @barkbay for doing some additional tests. The failure in Filebeat OSS build is not related.

I have started another packaging build, if this goes well I think this can be merged.

@jsoriano jsoriano removed the needs_backport PR is waiting to be backported to other branches. label May 19, 2020
jsoriano pushed a commit to jsoriano/beats that referenced this pull request May 19, 2020
Modify docker images so files required by beats are owned
by group root, this follows Openshifts recommendations to
run containerized applications with custom user ids.

(cherry picked from commit 9dbdc15)

Co-Authored-By: Michael Morello <[email protected]>
jsoriano added a commit that referenced this pull request May 19, 2020
jsoriano added a commit that referenced this pull request May 19, 2020
Modify docker images so files required by beats are owned
by group root, this follows Openshifts recommendations to
run containerized applications with custom user ids.

(cherry picked from commit 9dbdc15)

Co-authored-by: Michael Morello <[email protected]>
jsoriano added a commit to jsoriano/beats that referenced this pull request May 31, 2020
jsoriano added a commit that referenced this pull request Jun 2, 2020
jsoriano added a commit to jsoriano/beats that referenced this pull request Jun 2, 2020
jsoriano added a commit to jsoriano/beats that referenced this pull request Jun 2, 2020
Apply the ownership changes of elastic#12905, without applying the permission
changes, so it still satisfies strict perms checks.
v1v added a commit to v1v/beats that referenced this pull request Jun 2, 2020
…-stage-level

* upstream/master:
  [CI] Fix permissions should not fail (elastic#18899)
  Revert "Allow the Docker image to be run with a random user id (elastic#12905)" (elastic#18872)
  Add new fields to HAProxy module of Metricbeat (elastic#18523)
  Avoid duplicate names in dynamic_templates (elastic#18849)
jsoriano added a commit that referenced this pull request Jun 3, 2020
…ith a random user id (#12905)" (#18902)

This reverts commits 9dbdc15 and 3eac5f7.

(cherry picked from commit 7927da7)
jsoriano added a commit that referenced this pull request Jul 1, 2020
Prepare docker images to be run with arbitrary user ids. Following common practices
and recommendations, files that need to be read by Beats have now read permissions
for the group and belong to the root group. Also, the user included in the docker image
is added to the root group so it can read these files when run on docker with default
user and privileges.

Some changes are also added to Kubernetes reference manifests to help running beats
with arbitrary user ids, though this is not completely supported and it requires additional
setup.

Co-authored-by: Michael Morello <[email protected]>
jsoriano added a commit to jsoriano/beats that referenced this pull request Jul 1, 2020
elastic#18873)

Prepare docker images to be run with arbitrary user ids. Following common practices
and recommendations, files that need to be read by Beats have now read permissions
for the group and belong to the root group. Also, the user included in the docker image
is added to the root group so it can read these files when run on docker with default
user and privileges.

Some changes are also added to Kubernetes reference manifests to help running beats
with arbitrary user ids, though this is not completely supported and it requires additional
setup.

Co-authored-by: Michael Morello <[email protected]>
(cherry picked from commit 3ff02cb)
jsoriano added a commit that referenced this pull request Jul 2, 2020
…) (#19555)

Prepare docker images to be run with arbitrary user ids. Following common practices
and recommendations, files that need to be read by Beats have now read permissions
for the group and belong to the root group. Also, the user included in the docker image
is added to the root group so it can read these files when run on docker with default
user and privileges.

Some changes are also added to Kubernetes reference manifests to help running beats
with arbitrary user ids, though this is not completely supported and it requires additional
setup.

(cherry picked from commit 3ff02cb)

Co-authored-by: Michael Morello <[email protected]>
@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Jul 14, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
elastic#18873)

Prepare docker images to be run with arbitrary user ids. Following common practices
and recommendations, files that need to be read by Beats have now read permissions
for the group and belong to the root group. Also, the user included in the docker image
is added to the root group so it can read these files when run on docker with default
user and privileges.

Some changes are also added to Kubernetes reference manifests to help running beats
with arbitrary user ids, though this is not completely supported and it requires additional
setup.

Co-authored-by: Michael Morello <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Packaging review Team:Integrations Label for the Integrations team Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants