Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ensure ES index names are lowercase #4295

Merged
merged 4 commits into from
Oct 12, 2020
Merged

Conversation

simitt
Copy link
Contributor

@simitt simitt commented Oct 8, 2020

Motivation/summary

A regression was introduced when pulling in libbeat changes from elastic/beats#18854. Libbeat no longer guarantees lower case index names for all outputs, but rather moved the implementation to the standard index manager. APM Server implements its own index manager but did not apply changes when upgrading to libbeat changes.

This PR fixes the regression by ensuring index names are lowercased when derived from the event, and from the meta alias or index information.

Checklist

- [ ] I have signed the Contributor License Agreement.

I have considered changes for:
- [ ] documentation
- [ ] logging (add log lines, choose appropriate log selector, etc.)
- [ ] metrics and monitoring (create issue for Kibana team to add metrics to visualizations, e.g. Kibana#44001)

How to test these changes

  • configure customized index names in apm-server.yml containing upper cases
  • ingest events and ensure created indices are lowercased

Related issues

fixes #4294

Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes elastic#4294
@apmmachine
Copy link
Contributor

apmmachine commented Oct 8, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #4295 updated]

  • Start Time: 2020-10-12T06:54:09.012+0000

  • Duration: 44 min 2 sec

Test stats 🧪

Test Results
Failed 1
Passed 4277
Skipped 145
Total 4423

Test errors 1

Expand to view the tests failures

  • Name: Build and Test / APM Integration Tests / test_concurrent_req_java_spring – tests.agent.test_java

    • Age: 1
    • Duration: 77.53
    • Error Details: AssertionError: queried for [('processor.event', 'transaction'), ('service.name', ['springapp'])], expected 2000, got 1999

Steps errors 2

Expand to view the steps failures

  • Name: Compress

    • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage

    • Duration: 0 min 0 sec

    • Start Time: 2020-10-12T07:10:19.078+0000

    • log

  • Name: Compress

    • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests

    • Duration: 0 min 0 sec

    • Start Time: 2020-10-12T07:21:48.277+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-10-12T07:20:24.131Z] go: downloading github.com/golang/protobuf v1.3.4
[2020-10-12T07:20:24.131Z] go: downloading google.golang.org/genproto v0.0.0-20200218151345-dad8c97a84f5
[2020-10-12T07:20:24.131Z] go: downloading github.com/gogo/googleapis v1.0.1-0.20180501115203-b23578765ee5
[2020-10-12T07:20:24.131Z] go: downloading github.com/opentracing/opentracing-go v1.1.0
[2020-10-12T07:20:24.390Z] go: downloading golang.org/x/text v0.3.2
[2020-10-12T07:21:45.912Z] apm-server_kibana_1 is up-to-date
[2020-10-12T07:21:45.912Z] apm-server_elasticsearch_1 is up-to-date
[2020-10-12T07:21:45.912Z] === RUN   TestTransactionAggregation
[2020-10-12T07:21:45.912Z] 2020/10/12 07:20:51 Building apm-server...
[2020-10-12T07:21:45.912Z] 2020/10/12 07:20:53 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4295/src/github.com/elastic/apm-server/apm-server
[2020-10-12T07:21:45.912Z] --- PASS: TestTransactionAggregation (4.93s)
[2020-10-12T07:21:45.912Z] === RUN   TestTransactionAggregationShutdown
[2020-10-12T07:21:45.912Z] --- PASS: TestTransactionAggregationShutdown (2.56s)
[2020-10-12T07:21:45.912Z] === RUN   TestServiceDestinationAggregation
[2020-10-12T07:21:45.912Z] --- PASS: TestServiceDestinationAggregation (2.72s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPIKeyCreate
[2020-10-12T07:21:45.912Z] --- PASS: TestAPIKeyCreate (2.83s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPIKeyCreateExpiration
[2020-10-12T07:21:45.912Z] --- PASS: TestAPIKeyCreateExpiration (2.01s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPIKeyInvalidateName
[2020-10-12T07:21:45.912Z] --- PASS: TestAPIKeyInvalidateName (3.04s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPIKeyInvalidateID
[2020-10-12T07:21:45.912Z] --- PASS: TestAPIKeyInvalidateID (2.02s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerEnvironment
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerEnvironment/container
[2020-10-12T07:21:45.912Z] === PAUSE TestAPMServerEnvironment/container
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerEnvironment/systemd
[2020-10-12T07:21:45.912Z] === PAUSE TestAPMServerEnvironment/systemd
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerEnvironment/macos_service
[2020-10-12T07:21:45.912Z] === PAUSE TestAPMServerEnvironment/macos_service
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerEnvironment/windows_service
[2020-10-12T07:21:45.912Z] === PAUSE TestAPMServerEnvironment/windows_service
[2020-10-12T07:21:45.912Z] === CONT  TestAPMServerEnvironment/container
[2020-10-12T07:21:45.912Z] === CONT  TestAPMServerEnvironment/macos_service
[2020-10-12T07:21:45.912Z] === CONT  TestAPMServerEnvironment/windows_service
[2020-10-12T07:21:45.912Z] === CONT  TestAPMServerEnvironment/systemd
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServerEnvironment (0.00s)
[2020-10-12T07:21:45.912Z]     --- PASS: TestAPMServerEnvironment/container (0.49s)
[2020-10-12T07:21:45.912Z]     --- PASS: TestAPMServerEnvironment/macos_service (0.54s)
[2020-10-12T07:21:45.912Z]     --- PASS: TestAPMServerEnvironment/systemd (0.54s)
[2020-10-12T07:21:45.912Z]     --- PASS: TestAPMServerEnvironment/windows_service (0.59s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerInstrumentation
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServerInstrumentation (2.69s)
[2020-10-12T07:21:45.912Z] === RUN   TestJaegerGRPC
[2020-10-12T07:21:45.912Z] --- PASS: TestJaegerGRPC (2.93s)
[2020-10-12T07:21:45.912Z] === RUN   TestJaegerGRPCSampling
[2020-10-12T07:21:45.912Z] --- PASS: TestJaegerGRPCSampling (2.36s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerRequestLoggingValid
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServerRequestLoggingValid (0.31s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerMonitoring
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServerMonitoring (2.28s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerMonitoringBuiltinUser
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServerMonitoringBuiltinUser (2.00s)
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServerOnboarding
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServerOnboarding (2.62s)
[2020-10-12T07:21:45.912Z] === RUN   TestRUMXForwardedFor
[2020-10-12T07:21:45.912Z] --- PASS: TestRUMXForwardedFor (2.45s)
[2020-10-12T07:21:45.912Z] === RUN   TestKeepUnsampled
[2020-10-12T07:21:45.912Z] === RUN   TestKeepUnsampled/false
[2020-10-12T07:21:45.912Z] === RUN   TestKeepUnsampled/true
[2020-10-12T07:21:45.912Z] --- PASS: TestKeepUnsampled (4.76s)
[2020-10-12T07:21:45.912Z]     --- PASS: TestKeepUnsampled/false (2.33s)
[2020-10-12T07:21:45.912Z]     --- PASS: TestKeepUnsampled/true (2.43s)
[2020-10-12T07:21:45.912Z] PASS
[2020-10-12T07:21:45.912Z] ok  	github.com/elastic/apm-server/systemtest	44.151s
[2020-10-12T07:21:45.912Z] === RUN   TestAPMServer
[2020-10-12T07:21:45.912Z] 2020/10/12 07:20:50 Building apm-server...
[2020-10-12T07:21:45.912Z] 2020/10/12 07:20:52 Built /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4295/src/github.com/elastic/apm-server/apm-server
[2020-10-12T07:21:45.912Z] --- PASS: TestAPMServer (5.06s)
[2020-10-12T07:21:45.912Z] === RUN   TestUnstartedAPMServer
[2020-10-12T07:21:45.912Z] --- PASS: TestUnstartedAPMServer (0.00s)
[2020-10-12T07:21:45.912Z] === RUN   TestExpvar
[2020-10-12T07:21:45.912Z] --- PASS: TestExpvar (0.24s)
[2020-10-12T07:21:45.912Z] PASS
[2020-10-12T07:21:45.912Z] ok  	github.com/elastic/apm-server/systemtest/apmservertest	5.309s
[2020-10-12T07:21:45.913Z] ?   	github.com/elastic/apm-server/systemtest/estest	[no test files]
[2020-10-12T07:21:45.913Z] + cleanup
[2020-10-12T07:21:45.913Z] + rm -rf /tmp/tmp.uHV4IKRxHG
[2020-10-12T07:21:45.913Z] + .ci/scripts/docker-get-logs.sh
[2020-10-12T07:21:46.984Z] Post stage
[2020-10-12T07:21:46.993Z] Running in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4295/src/github.com/elastic/apm-server
[2020-10-12T07:21:47.017Z] Archiving artifacts
[2020-10-12T07:21:47.251Z] Recording test results
[2020-10-12T07:21:47.923Z] [WARN] tar: pathPrefix parameter is deprecated.
[2020-10-12T07:21:48.248Z] + tar --version
[2020-10-12T07:21:48.568Z] + tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
[2020-10-12T07:21:48.568Z] tar: system-tests: Cannot stat: No such file or directory
[2020-10-12T07:21:48.568Z] tar: Exiting with failure status due to previous errors
[2020-10-12T07:21:48.581Z] [INFO] system-tests-linux-files.tgz was not compressed or archived : script returned exit code 2
[2020-10-12T07:38:04.912Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/master/10915/display/redirect
[2020-10-12T07:38:07.389Z] Copied 51 artifacts from "APM Integration Test MBP Selector » master" build number 10915
[2020-10-12T07:38:08.549Z] Post stage
[2020-10-12T07:38:08.561Z] Recording test results
[2020-10-12T07:38:09.835Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4295
[2020-10-12T07:38:09.904Z] [INFO] getVaultSecret: Getting secrets
[2020-10-12T07:38:10.126Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-12T07:38:10.839Z] + chmod 755 generate-build-data.sh
[2020-10-12T07:38:10.839Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4295/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4295/runs/6 UNSTABLE 2641567
[2020-10-12T07:38:11.390Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4295/runs/6/steps/?limit=10000 -o steps-info.json
[2020-10-12T07:38:11.941Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4295/runs/6/tests/?status=FAILED -o tests-errors.json

@codecov-io
Copy link

Codecov Report

Merging #4295 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #4295   +/-   ##
=======================================
  Coverage   79.73%   79.73%           
=======================================
  Files         161      161           
  Lines        8984     8984           
=======================================
  Hits         7163     7163           
  Misses       1821     1821           

@simitt simitt merged commit 73e94cb into elastic:master Oct 12, 2020
simitt added a commit to simitt/apm-server that referenced this pull request Oct 12, 2020
Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes elastic#4294
simitt added a commit to simitt/apm-server that referenced this pull request Oct 12, 2020
Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes elastic#4294
simitt added a commit to simitt/apm-server that referenced this pull request Oct 12, 2020
Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes elastic#4294
simitt added a commit that referenced this pull request Oct 12, 2020
Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes #4294
simitt added a commit that referenced this pull request Oct 12, 2020
Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes #4294
simitt added a commit that referenced this pull request Oct 12, 2020
Ensure customized index names are always lowercase,
fixing regression introduced with https://github.com/elastic/beats/pull/18854/files

fixes #4294
@simitt simitt deleted the fix-index-names branch October 19, 2020 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ensure ES index names are lowercase
4 participants