Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for receiving traces over OTLP/gRPC #4677

Merged
merged 4 commits into from
Feb 6, 2021
Merged

Conversation

axw
Copy link
Member

@axw axw commented Feb 3, 2021

Motivation/summary

Add the gRPC service for receiving OTLP traces, and plumb it through to processor/otel. Once this lands we can follow up with:

  • adding support for metrics
  • translating OpenTelemetry semantic conventions to our model (i.e. what's in elasticexporter in opentelemetry-collector-contrib)
  • (later) adding support for logs

We use an interceptor for ensuring clients are authorised. Auth works by setting the "Authorization" header (gRPC request metadata) just like in HTTP requests.

Checklist

  • Update CHANGELOG.asciidoc
    - [ ] Documentation has been updated (I will create a separate issue for this)

How to test these changes

  1. Run APM Server
  2. Run opentelemetry-collector with the "otlp" exporter, pointed at APM Server, using the standard port (8200). It will be necessary to set insecure: true in the exporter config, unless APM Server is configured to use TLS.
  3. Send some OpenTelemetry data to opentelemetry-collector, and ensure that it shows up in the APM app.
  4. Configure APM Server with a secret token.
  5. Run step 3 again, observe a failure due to invalid auth.
  6. Add the secret token to the otlp exporter config, like headers: {'Authorization': 'Bearer secret_token'}, restart opentelemetry-collector
  7. Run step 3 again, ensure the data shows up in the APM app again.

Related issues

#4503

@apmmachine
Copy link
Contributor

apmmachine commented Feb 3, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #4677 updated

    • Start Time: 2021-02-06T00:16:13.159+0000
  • Duration: 42 min 53 sec

  • Commit: c3892c5

Test stats 🧪

Test Results
Failed 0
Passed 4625
Skipped 124
Total 4749

Steps errors 5

Expand to view the steps failures

Run Window tests
  • Took 11 min 51 sec . View more details on here
Compress
  • Took 0 min 0 sec . View more details on here
  • Description: tar --exclude=coverage-files.tgz -czf coverage-files.tgz coverage
Run Linux tests
  • Took 22 min 26 sec . View more details on here
  • Description: ./.ci/scripts/linux-test.sh
Compress
  • Took 0 min 0 sec . View more details on here
  • Description: tar --exclude=system-tests-linux-files.tgz -czf system-tests-linux-files.tgz system-tests
Test Sync
  • Took 4 min 47 sec . View more details on here
  • Description: ./.ci/scripts/sync.sh

Log output

Expand to view the last 100 lines of log output

[2021-02-06T00:50:02.485Z] === SKIP: x-pack/apm-server/sampling/pubsub TestElasticsearchIntegration_SubscribeSampledTraceIDs (0.00s)
[2021-02-06T00:50:02.485Z]     pubsub_integration_test.go:229: Skipping integration test, export INTEGRATION_TESTS=1 to run
[2021-02-06T00:50:02.485Z] 
[2021-02-06T00:50:02.485Z] DONE 1404 tests, 4 skipped in 132.454s
[2021-02-06T00:50:06.758Z] >> go test: Unit Test Passed
[2021-02-06T00:50:06.758Z] System testing 
[2021-02-06T00:50:13.341Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/apmpackage/cmd/gen-package
[2021-02-06T00:50:13.341Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/beater/beatertest
[2021-02-06T00:50:13.341Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/cmd/pprofessor
[2021-02-06T00:50:13.341Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/elasticsearch/estest
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/model/modeldecoder/generator
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/model/modeldecoder/generator/cmd
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/model/modeldecoder/generator/generatortest
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/model/modeldecoder/modeldecodertest
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/processor/stream/package_tests
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/sourcemap/test
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/tests
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/tests/loader
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/tests/system/jaegergrpc
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/cmd
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/include
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/sampling
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/sampling/pubsub
[2021-02-06T00:50:13.342Z] warning: no packages being tested depend on matches for pattern github.com/elastic/apm-server/x-pack/apm-server/sampling/pubsub/pubsubtest
[2021-02-06T00:50:13.342Z] 
[2021-02-06T00:50:14.281Z] Running python tests
[2021-02-06T00:50:20.870Z] 2021/02/06 00:50:20 exec: go list -m
[2021-02-06T00:50:21.440Z] >> python test: Unit Testing
[2021-02-06T00:50:24.744Z] ============================= test session starts =============================
[2021-02-06T00:50:24.744Z] platform win32 -- Python 3.8.1, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
[2021-02-06T00:50:24.744Z] rootdir: C:\Users\jenkins\workspace\pm-server_apm-server-mbp_PR-4677\src\github.com\elastic\apm-server
[2021-02-06T00:50:24.744Z] plugins: rerunfailures-9.0, timeout-1.3.4
[2021-02-06T00:50:24.744Z] timeout: 90.0s
[2021-02-06T00:50:24.744Z] timeout method: thread
[2021-02-06T00:50:24.744Z] timeout func_only: False
[2021-02-06T00:50:24.744Z] collected 142 items
[2021-02-06T00:50:24.744Z] 
[2021-02-06T00:50:24.744Z] tests\system\test_apikey_cmd.py sssss                                    [  3%]
[2021-02-06T00:50:34.733Z] tests\system\test_auth.py ..sssssssss                                    [ 11%]
[2021-02-06T00:50:34.733Z] tests\system\test_export.py sss                                          [ 13%]
[2021-02-06T00:50:34.733Z] tests\system\test_integration.py ssssssssssssssssssssssss                [ 30%]
[2021-02-06T00:50:34.733Z] tests\system\test_integration_acm.py sssssssss                           [ 36%]
[2021-02-06T00:50:34.733Z] tests\system\test_integration_sourcemap.py ssssssssssssss                [ 46%]
[2021-02-06T00:50:34.733Z] tests\system\test_jaeger.py sssss                                        [ 50%]
[2021-02-06T00:50:34.733Z] tests\system\test_pipelines.py sssssss                                   [ 54%]
[2021-02-06T00:52:26.180Z] tests\system\test_requests.py ...........................                [ 73%]
[2021-02-06T00:52:26.180Z] tests\system\test_setup_index_management.py sssssssssssssss              [ 84%]
[2021-02-06T00:52:26.180Z] tests\system\test_tls.py ssssssssssssssssssssss                          [100%]
[2021-02-06T00:52:26.180Z] 
[2021-02-06T00:52:26.180Z] ============================== warnings summary ===============================
[2021-02-06T00:52:26.180Z] c:\users\jenkin~1.pac\appdata\local\temp\python-env\build\ve\windows\lib\site-packages\_pytest\junitxml.py:446
[2021-02-06T00:52:26.181Z]   c:\users\jenkin~1.pac\appdata\local\temp\python-env\build\ve\windows\lib\site-packages\_pytest\junitxml.py:446: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0. See:
[2021-02-06T00:52:26.181Z]     https://docs.pytest.org/en/stable/deprecations.html#junit-family-default-value-change-to-xunit2
[2021-02-06T00:52:26.181Z]   for more information.
[2021-02-06T00:52:26.181Z]     _issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)
[2021-02-06T00:52:26.181Z] 
[2021-02-06T00:52:26.181Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-02-06T00:52:26.181Z] - generated xml file: C:\Users\jenkins\workspace\pm-server_apm-server-mbp_PR-4677\src\github.com\elastic\apm-server\build\TEST-python-unit.xml -
[2021-02-06T00:52:26.181Z] ============================ slowest 20 durations =============================
[2021-02-06T00:52:26.181Z] 8.09s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit_small_hit
[2021-02-06T00:52:26.181Z] 7.67s call     tests/system/test_requests.py::CorsTest::test_ok
[2021-02-06T00:52:26.181Z] 7.66s call     tests/system/test_requests.py::ClientSideTest::test_ok
[2021-02-06T00:52:26.181Z] 5.26s call     tests/system/test_auth.py::TestAccessDefault::test_full_access
[2021-02-06T00:52:26.181Z] 4.19s call     tests/system/test_auth.py::TestAccessWithSecretToken::test_backend_intake
[2021-02-06T00:52:26.181Z] 4.16s call     tests/system/test_requests.py::CorsTest::test_preflight_bad_headers
[2021-02-06T00:52:26.181Z] 4.11s call     tests/system/test_requests.py::RateLimitTest::test_multiple_ips_rate_limit
[2021-02-06T00:52:26.181Z] 4.10s call     tests/system/test_requests.py::RateLimitTest::test_multiple_ips_rate_limit_hit
[2021-02-06T00:52:26.181Z] 4.10s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit
[2021-02-06T00:52:26.181Z] 4.09s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit_hit
[2021-02-06T00:52:26.181Z] 3.59s call     tests/system/test_requests.py::RateLimitTest::test_rate_limit_only_metadata
[2021-02-06T00:52:26.181Z] 3.18s call     tests/system/test_requests.py::Test::test_deflate
[2021-02-06T00:52:26.181Z] 3.17s call     tests/system/test_requests.py::Test::test_ok_verbose
[2021-02-06T00:52:26.181Z] 3.17s call     tests/system/test_requests.py::Test::test_deflate_error
[2021-02-06T00:52:26.181Z] 3.16s call     tests/system/test_requests.py::Test::test_validation_fail
[2021-02-06T00:52:26.181Z] 3.16s call     tests/system/test_requests.py::Test::test_expvar_default
[2021-02-06T00:52:26.181Z] 3.16s call     tests/system/test_requests.py::Test::test_healthcheck
[2021-02-06T00:52:26.181Z] 3.15s call     tests/system/test_requests.py::CorsTest::test_preflight
[2021-02-06T00:52:26.181Z] 3.15s call     tests/system/test_requests.py::Test::test_empty
[2021-02-06T00:52:26.181Z] 3.15s call     tests/system/test_requests.py::Test::test_bad_json
[2021-02-06T00:52:26.181Z] =========== 29 passed, 113 skipped, 1 warning in 114.17s (0:01:54) ============
[2021-02-06T00:52:26.181Z] >> python test: Unit Testing Complete
[2021-02-06T00:52:27.145Z] Post stage
[2021-02-06T00:52:27.170Z] Recording test results
[2021-02-06T00:52:27.996Z] [Checks API] No suitable checks publisher found.
[2021-02-06T00:58:01.383Z] [INFO] For detailed information see: https://apm-ci.elastic.co/job/apm-integration-tests-selector-mbp/job/master/13750/display/redirect
[2021-02-06T00:58:03.551Z] Copied 25 artifacts from "APM Integration Test MBP Selector » master" build number 13750
[2021-02-06T00:58:04.547Z] Post stage
[2021-02-06T00:58:04.557Z] Recording test results
[2021-02-06T00:58:05.190Z] [Checks API] No suitable checks publisher found.
[2021-02-06T00:58:05.578Z] Running on Jenkins in /var/lib/jenkins/workspace/pm-server_apm-server-mbp_PR-4677
[2021-02-06T00:58:05.642Z] [INFO] getVaultSecret: Getting secrets
[2021-02-06T00:58:05.810Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-06T00:58:06.528Z] + chmod 755 generate-build-data.sh
[2021-02-06T00:58:06.528Z] + ./generate-build-data.sh https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4677/ https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4677/runs/8 FAILURE 2513108
[2021-02-06T00:58:07.079Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4677/runs/8/steps/?limit=10000 -o steps-info.json
[2021-02-06T00:58:07.630Z] INFO: curl https://apm-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/apm-server/apm-server-mbp/PR-4677/runs/8/tests/?status=FAILED -o tests-errors.json

Add the gRPC service for receiving OTLP traces,
and plumb it through to processor/otel.

Auth works by setting the "Authorization" header
(gRPC request metadata) like in HTTP requests.
@axw axw marked this pull request as ready for review February 3, 2021 12:16
@axw axw requested a review from a team February 3, 2021 12:16
@axw
Copy link
Member Author

axw commented Feb 4, 2021

jenkins run the tests please

@bmorelli25
Copy link
Member

bmorelli25 commented Feb 5, 2021

How are you running opentelemetry-collector? I'm trying to run the docker demo and can't get it to work 😬 (almost definitely user-error and not because something is wrong with the PR).

Edit: Never mind! @simitt hooked me up with a little sample app 🎉

Screen Shot 2021-02-05 at 10 24 25 AM

Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I teseted it through as described - works as expected

@simitt
Copy link
Contributor

simitt commented Feb 5, 2021

@bmorelli25 I run a small local app in go, basically following https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp#example-package-Insecure.

@axw
Copy link
Member Author

axw commented Feb 6, 2021

Test failure is unrelated, see #4692

@axw axw merged commit 6cb1f41 into elastic:master Feb 6, 2021
@axw axw deleted the native-otlp branch February 6, 2021 01:25
axw added a commit to axw/apm-server that referenced this pull request Feb 18, 2021
Add the gRPC service for receiving OTLP traces,
and plumb it through to processor/otel.

Auth works by setting the "Authorization" header
(gRPC request metadata) like in HTTP requests.
# Conflicts:
#	NOTICE.txt
#	changelogs/head.asciidoc
axw added a commit that referenced this pull request Feb 18, 2021
Add the gRPC service for receiving OTLP traces,
and plumb it through to processor/otel.

Auth works by setting the "Authorization" header
(gRPC request metadata) like in HTTP requests.
# Conflicts:
#	NOTICE.txt
#	changelogs/head.asciidoc
@cyrille-leclerc
Copy link
Contributor

cyrille-leclerc commented Feb 27, 2021

I have successfully tested traces and metrics with

  • OpenTelemetry Collector 0.21
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: '127.0.0.1:4317'
  hostmetrics:
    collection_interval: 1m
    scrapers:
      # cpu not supported on MacOS
      # cpu:
      load:
      memory:
processors:
  batch: null
exporters:
  logging:
    loglevel: info
  prometheus:
    endpoint: 0.0.0.0:8889
  otlp/elastic:
    endpoint: "localhost:8200"
    insecure: true
    headers:
      Authorization: "Bearer my_secret_token"
service:
  pipelines:
    metrics:
      receivers:
        - otlp
        - hostmetrics
      exporters:
        - prometheus
        - otlp/elastic
    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - otlp/elastic
  • Java Agent opentelemetry-javaagent-all-0.16.1.jar
export OTEL_RESOURCE_ATTRIBUTES=service.name=checkoutService,service.namespace=com-shoppingcart,service.version=1.0-SNAPSHOT,deployment.environment=staging
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer my_secret_token"

java -javaagent:$PRGDIR/../.otel/opentelemetry-javaagent-all-0.16.1.jar \
     -Dio.opentelemetry.auto.slf4j.simpleLogger.defaultLogLevel=info \
     -classpath target/dependency/*:target/classes/ com.mycompany.checkout.CheckoutServiceServer

@axw
Copy link
Member Author

axw commented Mar 2, 2021

Thanks for testing it out @cyrille-leclerc!

@simitt
Copy link
Contributor

simitt commented Mar 4, 2021

Tested with BC 2 - Worked with the elastic and the otel exporter.

@patpatpat123
Copy link

Hello team,

In this page, as well as in the official APM doc: https://www.elastic.co/guide/en/apm/guide/current/open-telemetry-direct.html#instrument-apps-otel

We see this example being used every time: com.mycompany.checkout.CheckoutServiceServer

Is it possible to make this example com.mycompany.checkout.CheckoutServiceServer available please?

Thank you

@axw
Copy link
Member Author

axw commented Mar 27, 2023

@patpatpat123 I don't know if there even is any such code, or if it was just a made up example class name. If you're looking for a demo application to instrument with OpenTelemetry, then you should consider using OpenTelemetry Demo: https://github.com/open-telemetry/opentelemetry-demo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants