x-pack/metricbeat AWS tests failure

[oakrizan]

Flaky Test

• Test Name: x-pack/metricbeat/module/aws/sqs TestFetch
• Link: Link to file/line number in github.
• Branch: main
• Artifact Link: https://buildkite.com/elastic/beats-xpack-metricbeat/builds/5164#0190b59b-0ca1-4371-bc12-819a10ada908/655-1025
• Notes: The tests were successful last week

Stack Trace

=== Failed
--
  | === FAIL: x-pack/metricbeat/module/aws/sqs TestFetch (17.64s)
  | sqs_integration_test.go:28:
  | Error Trace:	/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/sqs/sqs_integration_test.go:28
  | Error:      	Should NOT be empty, but was []
  | Test:       	TestFetch

To execute AWS tests on CI aws label should added to the PR

[dliappis]

FYI @elastic/obs-ds-hosted-services

[dliappis]

Another occurrence in:

• 8.15: https://buildkite.com/elastic/beats-aws-tests/builds/7#0190b668-7c28-4025-a25a-0f52da04aa32/584-968
• 8.14: https://buildkite.com/elastic/beats-aws-tests/builds/5#0190b668-98b2-4b7c-beb6-493ebf51a261/579-903

[zmoog]

Hey @dliappis, we're checking the tests. @kaiyan-sheng thinks we may need to update the latency settings.

[kaiyan-sheng]

I created a SQS queue at 9:35 AM but the first cloudwatch metric didn't show up till 9:41 AM. With 5min latency configured, unless we wait for a long time before checking for events, we have a high chance not seeing any data point.

[zmoog]

We may need to fetch the metrics more than once for latency. Testing in progress.

[zmoog]

@dliappis, the PR #40251 is in review.

We added a PeriodicReportingFetchV2Error function that tries to fetch the metrics multiple times until it gets values or the timeout expires. The (non-periodic) ReportingFetchV2Error only fetches the metric values once, resulting in much higher chances of not getting any metric value.

I tried running the tests five times with five successful executions.

There are other aspects we want to investigate and improve, but I hope these changes will remove the flakiness.

[dliappis]

@dliappis, the PR #40251 is in review.

Thank you @zmoog for the quick actions here! Looking forward to getting the PR merged/resolve the issue asap as pretty much all branches now, most of the time, are hitting this issue (example: https://buildkite.com/elastic/beats-pipeline-scheduler/builds/89#0190bdfe-8bed-442a-9f72-5b7074694c12, see also here).

[zmoog]

I'm closing this since the issue seems to be resolved. Feel free to reopen it in case the problem shows up again.