[Metricbeat] azure: move event report into loop validDim loop

[ClumsyPotato]

What does this PR do?

Moved the report.Event() function into the loop that iterates over all grouped Dimension in order to report all events instead of just a single one

Why is it important?

Trying to fix the issue in #27226
If a azure metric endpoint with multiple dimensions is pulled only one dimension is send back to elastic

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #27226

Also relates to this
https://support.elastic.co/cases/5008X000023agMSQAY

###Backport
Pls backport this to version 7.x.x

[cla-checker-service]

💚 CLA has been signed

[mergify]

This pull request does not have a backport label. Could you fix it @ClumsyPotato? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-11T20:29:17.507+0000

• Duration: 90 min 40 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2774
Skipped	240
Total	3014

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ClumsyPotato]

/test

[ClumsyPotato]

/test

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[kaiyan-sheng]

@ClumsyPotato Thank you so much for your contribution!!

[kaiyan-sheng]

/test

[ClumsyPotato]

@kaiyan-sheng Glad i can help. I am open for suggestion and changes :)

Can you also make sure that this fix gets backportet to version 7.x? That would be awsome :3

[kaiyan-sheng]

@ClumsyPotato Sorry for the delay on reviewing this PR! Overall it looks good, one nit is seems like there are several lines of code are repeated:

event, metricList = createEvent(timestamp, defaultMetric, resource, groupTimeValues)
if client.Config.AddCloudMetadata {
	vm = client.GetVMForMetaData(&resource, groupTimeValues)
	addCloudVMMetadata(&event, vm, resource.Subscription)
}
if client.Config.DefaultResourceType == "" {
	event.ModuleFields.Put("metrics", metricList)
} else {
	for key, metric := range metricList {
		event.MetricSetFields.Put(key, metric)
	}
}

Could you refactor this a bit please? Also it would be good to run TestData to regenerate data.json sample file to show that multiple dimensions are all collected. If you can add some test maybe in data_test.go, that would be great!
Thanks!!

[Patrick-Eichhorn]

/test

[ClumsyPotato]

@kaiyan-sheng thanks for the input. We just moved the whole block into its own function. Hope thats fine now^^
About The integration tests...I did some changes to the test and tried to run the tests with mage IntegTest and GoIntegTest but it did not make any change to data.json
Any idea what could be the problem?

[kaiyan-sheng]

@ClumsyPotato Thanks for the change! What I usually do to regenerate data.json file is: (using compute VM as an example) in azure/compute_vm/compute_vm_integration_test.go, remove //go:build integration && azure and // +build integration,azure. Then add all the environment variables required for testing: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET and etc. Then change the DataFlag to true under metricbeat/mb/testing/flags/flags.go. After that you should be able to see updated data.json when running the TestData function.

This is probably not the right way to run the test but it works 😂

[endorama]

Should be possible to run TestData tests responsible for updating sample data by running, from within the metricset folder:

$ go test --tags integration,azure -data -run "TestData"

Note the -data flag and the tags (integrations is required, azure is required for this particular metricset). I've used this command for the GCP module successfully. -run "TestData" may be omitted, as it's only used to target data tests only.

A note about tags: the -data flag is a custom flag added by metricbeat framework. I will not be present in case tags do not match, as the relevant code will not be run and silently skipped (without the tag the test file is ignored by go compiler so the framework doesn't load). This may happen if there are different tags in the build tags of the metricset under test (i.e. the GCP billing module has also billing tag).

I don't have an Azure environment setup to test this PR right now, but if you can try this command and confirm it works I'll add it to the developer documentation :)

[kaiyan-sheng]

/test

[ClumsyPotato]

Hey @shelly-yao @endorama
I added a test that makes sure that a certain dimension is always present in the metricset.
Also running the test with the labels added like @endorama describe also works when running inside the monitor directory

[kaiyan-sheng]

/test

[Patrick-Eichhorn]

Hello @kaiyan-sheng,
I saw that there are backport labels for 7.17.0 and 8.0.0. Will be the bug-fix merged in older versions too? For example 7.13.X and above?

[kaiyan-sheng]

Hey @Patrick-Eichhorn, sorry we won't be able to backport it to 7.13.x because we are not planning to release any new patch release for 7.13 anymore.