Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Use libbeat logger #19211

Merged
merged 25 commits into from
Jul 1, 2020

Conversation

michalpristas
Copy link
Contributor

@michalpristas michalpristas commented Jun 16, 2020

What does this PR do?

Changes our dependency on ecs logger and uses libbeat logger by default.
By default we log to file now, so users can share logs easier.

Why is it important?

ECS compliant logging.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Fixes: #15129

@michalpristas michalpristas added review needs_backport PR is waiting to be backported to other branches. Team:Ingest Management Ingest Management:beta1 Group issues for ingest management beta1 labels Jun 16, 2020
@michalpristas michalpristas self-assigned this Jun 16, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 16, 2020
@michalpristas michalpristas changed the title Agent liblog [Ingest Manager] Use libbeat logger Jun 16, 2020
@michalpristas michalpristas requested a review from ph June 16, 2020 07:03
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 16, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-07-01T10:39:58.820+0000

  • Duration: 123 min 1 sec

Test stats 🧪

Test Results
Failed 0
Passed 6435
Skipped 1053
Total 7488

Steps errors

Expand to view the steps failures

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-01T11:38:34.207+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-01T12:23:35.160+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-01T12:31:55.670+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-01T12:12:05.369+0000

    • log

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Duration: 2 min 22 sec

    • Start Time: 2020-07-01T12:10:02.380+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-01T12:35:07.408Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-01T12:35:07.408Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-01T12:35:07.408Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-01T12:35:07.408Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-01T12:42:29.253Z] Cancelling nested steps due to timeout
[2020-07-01T12:42:29.552Z] Failed in branch Elastic Agent Mac OS X
[2020-07-01T12:42:29.553Z] Failed in branch Filebeat Mac OS X
[2020-07-01T12:42:29.553Z] Failed in branch Filebeat x-pack Mac OS X
[2020-07-01T12:42:29.554Z] Failed in branch Auditbeat oss Mac OS X
[2020-07-01T12:42:29.555Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-07-01T12:42:29.558Z] Failed in branch Metricbeat Mac OS X
[2020-07-01T12:42:29.559Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-07-01T12:42:29.630Z] Stage "Heartbeat Windows" skipped due to earlier failure(s)
[2020-07-01T12:42:29.633Z] Stage "Functionbeat Windows" skipped due to earlier failure(s)
[2020-07-01T12:42:29.635Z] Stage "Generators Beat Mac OS X" skipped due to earlier failure(s)
[2020-07-01T12:42:29.965Z] Failed in branch Heartbeat
[2020-07-01T12:42:29.966Z] Failed in branch Functionbeat
[2020-07-01T12:42:29.968Z] Failed in branch Generators
[2020-07-01T12:42:30.213Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats
[2020-07-01T12:42:30.643Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-01T12:42:30.660Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Lint
[2020-07-01T12:42:30.778Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-01T12:42:30.894Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-01T12:42:31.007Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-01T12:42:31.209Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-01T12:42:31.335Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-01T12:42:31.471Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-01T12:42:31.600Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-01T12:42:31.752Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-01T12:42:31.880Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-01T12:42:31.990Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-01T12:42:32.110Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-01T12:42:32.225Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-01T12:42:32.374Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-01T12:42:32.587Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-01T12:42:32.682Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-01T12:42:32.795Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-01T12:42:32.944Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-01T12:42:33.131Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-01T12:42:33.247Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-01T12:42:33.346Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Packetbeat-oss
[2020-07-01T12:42:33.438Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-01T12:42:33.519Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-01T12:42:33.616Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-01T12:42:33.790Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-01T12:42:33.890Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-01T12:42:33.987Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Filebeat-oss
[2020-07-01T12:42:34.109Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-01T12:42:34.219Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-01T12:42:34.332Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Libbeat-oss
[2020-07-01T12:42:34.471Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-01T12:42:34.611Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-07-01T12:42:34.738Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-07-01T12:42:35.134Z] + cat
[2020-07-01T12:42:35.134Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-01T12:42:35.134Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-01T12:42:41.727Z] runbld>>> runbld started
[2020-07-01T12:42:41.727Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-01T12:42:44.280Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19211' in order of occurrence in the config (last value wins).
[2020-07-01T12:42:45.234Z] runbld>>> Debug logging enabled.
[2020-07-01T12:42:45.234Z] runbld>>> Storing result
[2020-07-01T12:42:45.234Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-01T12:42:45.234Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701124244-3794D75A
[2020-07-01T12:42:45.234Z] runbld>>> Adding system facts.
[2020-07-01T12:42:46.179Z] runbld>>> Adding vcs info for the latest commit:  fa41b951e26e4bf34659d82aeaf6d33f22503dd9
[2020-07-01T12:42:46.179Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-01T12:42:46.179Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-01T12:42:46.439Z] Processing JUnit reports with runbld...
[2020-07-01T12:42:46.439Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-01T12:42:46.700Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-01T12:42:46.700Z] runbld>>> DURATION: 36ms
[2020-07-01T12:42:46.700Z] runbld>>> STDOUT: 40 bytes
[2020-07-01T12:42:46.700Z] runbld>>> STDERR: 49 bytes
[2020-07-01T12:42:46.700Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-01T12:42:46.700Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats
[2020-07-01T12:42:48.083Z] runbld>>> Storing build metadata: 
[2020-07-01T12:42:48.083Z] runbld>>> Adding test report.
[2020-07-01T12:42:48.083Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats
[2020-07-01T12:42:49.038Z] runbld>>> Found 95 test output files
[2020-07-01T12:42:49.882Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-01T12:42:49.882Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-01T12:42:50.156Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-01T12:42:50.156Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-01T12:42:50.156Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-01T12:42:50.156Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-01T12:42:50.156Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-01T12:42:51.104Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 7338 Skipped: 849
[2020-07-01T12:42:51.104Z] runbld>>> Storing result
[2020-07-01T12:42:51.104Z] runbld>>> FAILURES: 0
[2020-07-01T12:42:51.365Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-01T12:42:51.365Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200701124244-3794D75A
[2020-07-01T12:42:51.625Z] runbld>>> Email notification disabled by environment variable.
[2020-07-01T12:42:51.625Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-01T12:42:57.991Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19211
[2020-07-01T12:42:58.418Z] [INFO] getVaultSecret: Getting secrets
[2020-07-01T12:42:58.683Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-01T12:43:00.423Z] + chmod 755 generate-build-data.sh
[2020-07-01T12:43:00.423Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19211/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19211/runs/31 ABORTED 7381307
[2020-07-01T12:43:00.423Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19211/runs/31/steps/?limit=10000 -o steps-info.json
[2020-07-01T12:43:04.582Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19211/runs/31/tests/?status=FAILED -o tests-errors.json

go.sum Show resolved Hide resolved
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

type loggingLevel logp.Level

var loggingLevelMap = map[string]loggingLevel{
"trace": loggingLevel(logp.DebugLevel), // For backward compatibility
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trace was only for us? correct? we can remove that.

# Sets output of logging. Available options are `console`, `file`.
# File output is located at `data/logs/elastic-agent`
# logging.output: console

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what i dont like in to_xxx configuration options is that it evokes that you can set true to more than one and it will work, having a single option with a single value makes it clear there's no way of having 2 outputs of logging.
i will introduce some more options or i can even make it 1;1 and let libbeat logger handle that without any verification from our side.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not like it either but I am into moving responsibility out if possible, @urso is there a plan to change the logging options?

@ph
Copy link
Contributor

ph commented Jun 17, 2020

@michalpristas Few changes I see.

  1. Could we use the same options defined in the https://github.com/elastic/beats/blob/master/libbeat/_meta/config/logging.reference.yml.tmpl
    I see we have logging.output.

  2. We should move to "info" soon.

  3. Can you create an issue to move to go template to generate the configuration file Build config files with Go text/template #18148

@michalpristas
Copy link
Contributor Author

@ph issue created here: #19269

JSON bool `config:"json"` // Write logs as JSON.
Level Level `config:"level"` // Logging level (error, warning, info, debug).
Selectors []string `config:"selectors"` // Selectors for debug level logging.
ECSEnabled bool `config:"ecs"` // Adds minimal ECS information using ECS conformant keys to every log line
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this on by default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michalpristas michalpristas merged commit 2a1b4f4 into elastic:master Jul 1, 2020
michalpristas added a commit to michalpristas/beats that referenced this pull request Jul 1, 2020
michalpristas added a commit that referenced this pull request Jul 2, 2020
[Ingest Manager] Use libbeat logger (#19211)
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ingest Management:beta1 Group issues for ingest management beta1 needs_backport PR is waiting to be backported to other branches. review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Agent] Refactor: Export our log using libbeat logging library.;
4 participants