azure logs: add ECS mapping for event.duration

[zmoog]

Proposed commit message

Convert the event.duration field value to the long type.

Users reported mapping exceptions due to event.duration string values causing field mapping as keyword instead of long. See #10848 to learn more.

Elasticsearch maps a field as a keyword if it has a string value. This happens even on stack versions 8.13+ because ecs@mappings does not perform type coercion.

By converting the event.duration field values to the long type, we ensure Elasticsearch uses the expected ECS field mapping as long.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Related issues

• Relates Verify mapping problems after migrating to ecs@mappings #10848

[elasticmachine]

🚀 Benchmarks report

Package azure 👍(5) 💚(4) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
signinlogs	2450.98	2016.13	-434.85 (-17.74%)	💔
firewall_logs	1694.92	1398.6	-296.32 (-17.48%)	💔

To see the full report comment with /test benchmark fullreport

[efd6]

Suggest also

diff --git a/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml b/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
index e5e7c52bbd..fcdc5b2eb3 100644
--- a/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/default.yml
@@ -72,10 +72,23 @@ processors:
       field: azure.activitylogs.durationMs
       target_field: event.duration
       ignore_missing: true
+  - convert:
+      field: event.duration
+      tag: convert_event_duration
+      type: long
+      ignore_missing: true
+      on_failure:
+        - remove:
+            field: event.duration
+        - append:
+            field: error.message
+            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
   - script:
       lang: painless
-      source: if (ctx.event.duration!= null) {ctx.event.duration = ctx.event.duration
-        * params.param_nano;}
+      source: >
+        if (ctx.event.duration != null) {
+          ctx.event.duration = ctx.event.duration * params.param_nano;
+        }
       params:
         param_nano: 1000000
       ignore_failure: true

and changelog/manifest updates.

[zmoog]

Hey @efd6, thanks for suggesting the convert processor.

I was looking for a temporarily quick fix for the mapping, adding back fields/ecs.yml file, but converting it solves the problem. I can remove the fields/ecs.yml and only leverage ecs@mappings.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
93.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[zmoog]

/test

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 456a3e2

Failed CI Steps

• Check integrations azure

History

• 💔 Build #16041 failed 456a3e2
• 💚 Build #15984 succeeded 1ecc6e6
• 💚 Build #15817 succeeded 03d8ef2
• 💚 Build #15801 succeeded 48a8cd2

cc @zmoog

[ishleenk17]

Can't we adding back the mapping to ecs.yml instead of doing this ?

[efd6]

I agree, we have seen that the conversion is not effective in communicating that the type should be a long.