Add OTel Agent e2e pipeline tests #29091

liustanley · 2024-09-05T18:56:32Z

What does this PR do?

This PR adds e2e pipeline tests for the OTel Agent and refactors the file structure of new-e2e/tests/otel according to our testing plan.

These tests use three different collector config files:

complete.yml for a complete config file to use with the converter disabled
minimal.yml for a minimal config file to test that the converter adds components as expected
sampling.yml for adding the probabilistic sampler and verifying APM stats still work

The following test cases are implemented in utils/pipelines_utils.go:

Verify fakeintake receives scraped prometheus metrics: collector metrics, trace agent metrics, and agent metrics
Verify fakeintake receives telemetry from telemetrygen
- Resource attributes are tagged as expected
- Telemetry attributes are tagged as expected
- Infraattributesprocessor adds kubernetes tags as expected
- Service, env, and version tags appear as expected
- Host appears as expected and consistent across all signals
- Trace metrics are received by fakeintake
Verify trace metrics work with sampling as expected

Motivation

Additional Notes

The non-pipeline tests have been moved to utils/config_utils.go, and will be expanded on in a future PR for adding extensive config tests.

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

pr-commenter · 2024-09-05T19:28:24Z

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=44814819 --os-family=ubuntu

Note: This applies to commit 8ff38a1

pr-commenter · 2024-09-05T19:50:14Z

Regression Detector

Regression Detector Results

Run ID: a4b1711e-c63c-4ddb-a1e0-a56c1279b841 Metrics dashboard Target profiles

Baseline: 60d4126
Comparison: 8ff38a1

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	basic_py_check	% cpu utilization	+0.80	[-2.02, +3.62]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+0.17	[-0.59, +0.93]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.00, +0.00]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.06	[-0.11, -0.00]	1	Logs
➖	idle	memory utilization	-0.19	[-0.24, -0.15]	1	Logs
➖	file_tree	memory utilization	-0.34	[-0.46, -0.23]	1	Logs
➖	otel_to_otel_logs	ingress throughput	-0.93	[-1.75, -0.11]	1	Logs
➖	pycheck_lots_of_tags	% cpu utilization	-2.90	[-5.38, -0.41]	1	Logs

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
❌	idle	memory_usage	9/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

agent-platform-auto-pr · 2024-09-10T14:50:34Z

[Fast Unit Tests Report]

On pipeline 44814819 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

jackgopack4 · 2024-09-11T18:55:23Z

test/new-e2e/tests/otel/otel-agent/complete_test.go

+package otelagent
+
+import (
+	_ "embed"


asking out of ignorance, why is this added as "_"?

We're only using this package to embed yaml files into strings (//go:embed config/complete.yml), but aren't using any of its functions so it needs to be a blank import: https://pkg.go.dev/embed#hdr-Strings_and_Bytes.

dineshg13

We also need to test infra attributes processor with different cardinalities

test/new-e2e/tests/otel/otel-agent/config/sampling.yml

test/new-e2e/tests/otel/utils/config_utils.go

dineshg13 · 2024-09-11T19:02:40Z

test/new-e2e/tests/otel/utils/pipelines_utils.go

+		// Verify container tags from infraattributes processor
+		assert.NotNil(s.T(), ctags["kube_container_name"])
+		assert.NotNil(s.T(), ctags["kube_namespace"])
+		assert.NotNil(s.T(), ctags["pod_name"])
+		assert.Equal(s.T(), sp.Meta["k8s.container.name"], ctags["kube_container_name"])
+		assert.Equal(s.T(), sp.Meta["k8s.namespace.name"], ctags["kube_namespace"])
+		assert.Equal(s.T(), sp.Meta["k8s.pod.name"], ctags["pod_name"])


We set these on SDK, can we test specific ones that infra attributes processor adds ?

These are all the tags added by the infra attributes processor, if we want the infra attributes processor to add more tags we need to set more Kubernetes attributes on the SDK

tags:"custom.attribute:true" tags:"deployment.environment:e2e" tags:"host.name:kind-control-plane" tags:"k8s.container.name:telemetrygen-job" tags:"k8s.namespace.name:datadog" tags:"k8s.node.name:kind-control-plane" tags:"k8s.pod.name:telemetrygen-job-metrics-fv6tn" tags:"service.name:telemetrygen-job" tags:"service.version:1.0" tags:"env:e2e" tags:"service:telemetrygen-job" tags:"version:1.0" tags:"kube_container_name:telemetrygen-job" tags:"kube_namespace:datadog" tags:"pod_name:telemetrygen-job-metrics-fv6tn"

liustanley added 2 commits September 5, 2024 14:56

New tests

9d091f3

Logs fix

f9ad75e

github-actions bot added the team/opentelemetry OpenTelemetry team label Sep 5, 2024

liustanley added 3 commits September 5, 2024 14:57

Revert

3dd909c

Move

3d9bba6

Add test back

6488a48

liustanley mentioned this pull request Sep 5, 2024

Assign mapped hostname from OTLP logs in logs agent pipeline #28870

Merged

Merge branch 'main' into stanley.liu/otel-e2e-pipelines

2477d1f

liustanley added 3 commits September 10, 2024 14:38

Refactor test structure

1ef009d

Fix tabs

af8fd85

Fix lint

e907b76

liustanley changed the title ~~WIP Stanley.liu/otel e2e pipelines~~ Add OTel Agent e2e pipeline tests Sep 10, 2024

Clean up code

7304255

liustanley added changelog/no-changelog qa/no-code-change Skip QA week as there is no code change in Agent code labels Sep 10, 2024

Add prometheus test

393ca55

liustanley marked this pull request as ready for review September 10, 2024 20:47

liustanley requested a review from a team as a code owner September 10, 2024 20:47

liustanley requested a review from songy23 September 10, 2024 20:47

Fix package comment

9e8cf2e

liustanley requested a review from dineshg13 September 10, 2024 20:49

Fix config

1a0cc2a

liustanley added this to the 7.59.0 milestone Sep 10, 2024

Fix helm values

6cdef1b

jackgopack4 reviewed Sep 11, 2024

View reviewed changes

dineshg13 reviewed Sep 11, 2024

View reviewed changes

liustanley added 5 commits September 12, 2024 11:22

Move batch

1245477

Merge branch 'main' into stanley.liu/otel-e2e-pipelines

727cfe8

Add temp tests

cfc84e7

Fix config

c559ae6

Remove resources

8ff38a1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OTel Agent e2e pipeline tests #29091

Add OTel Agent e2e pipeline tests #29091

liustanley commented Sep 5, 2024 •

edited

Loading

pr-commenter bot commented Sep 5, 2024 •

edited

Loading

pr-commenter bot commented Sep 5, 2024 •

edited

Loading

Fine details of change detection per experiment

Explanation

agent-platform-auto-pr bot commented Sep 10, 2024 •

edited

Loading

jackgopack4 Sep 11, 2024

liustanley Sep 11, 2024

dineshg13 left a comment •

edited

Loading

dineshg13 Sep 11, 2024

liustanley Sep 12, 2024

Add OTel Agent e2e pipeline tests #29091

Are you sure you want to change the base?

Add OTel Agent e2e pipeline tests #29091

Conversation

liustanley commented Sep 5, 2024 • edited Loading

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

pr-commenter bot commented Sep 5, 2024 • edited Loading

Test changes on VM

pr-commenter bot commented Sep 5, 2024 • edited Loading

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Bounds Checks

Explanation

agent-platform-auto-pr bot commented Sep 10, 2024 • edited Loading

jackgopack4 Sep 11, 2024

Choose a reason for hiding this comment

liustanley Sep 11, 2024

Choose a reason for hiding this comment

dineshg13 left a comment • edited Loading

Choose a reason for hiding this comment

dineshg13 Sep 11, 2024

Choose a reason for hiding this comment

liustanley Sep 12, 2024

Choose a reason for hiding this comment

liustanley commented Sep 5, 2024 •

edited

Loading

pr-commenter bot commented Sep 5, 2024 •

edited

Loading

pr-commenter bot commented Sep 5, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Sep 10, 2024 •

edited

Loading

dineshg13 left a comment •

edited

Loading