[Fleet] Make stream id unique in agent policy #82447

nchaulet · 2020-11-03T15:52:40Z

Description

Resolve #81617 elastic/beats#22054

This PR make stream id unique, this allow to have multiple time the same integrations in an agent policy.

What changed?

the stream id is not generated client side anymore
when we persist a package policy (create|bulkCreate|update) we generated ids for streams
the stream id is suffixed the package policy id.
For input validate client side we do not use id anymore but dataset

I confirmed with @michalpristas there is not limit on the id size.

Example config

After

Before

How to test this?

Create a config
add multiple time the same integration (custom log is easy to read and configure)
Copy a policy should work too

…-81617-unique-stream-id

elasticmachine · 2020-11-03T22:25:56Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…-81617-unique-stream-id

ruflin

What happens in the following scenario:

Integration has 3 inputs and it is saved
I update 1 input and save the integration

Will all ids stay the same, will it regenerate all 3 or just one?

This is relevant because on the Agent / Beat side AFAIK the hash of the input is used to decide if an input is restarted or not. So if all ids keep changing all the time, it means for example log inputs are constantly stopped and started again.

nchaulet · 2020-11-06T12:58:53Z

@ruflin The ids will stay the same, the uuid part of the id is created when you create the package policy for the first time.

ruflin · 2020-11-06T14:43:14Z

Great. So even if an input is changed, it keeps the id! That might even be beneficial to track issues.

nchaulet · 2020-11-06T17:09:23Z

@elasticmachine merge upstream

jen-huang

LGTM, tested locally

jen-huang · 2020-11-06T21:01:16Z

x-pack/plugins/ingest_manager/server/types/models/package_policy.ts

@@ -54,7 +54,7 @@ const PackagePolicyBaseSchema = {
      ),
      streams: schema.arrayOf(
        schema.object({
-          id: schema.string(),
+          id: schema.maybe(schema.string()), // BWC < 7.11


can we add this to https://github.com/elastic/integrations-dev/issues/334?

nchaulet · 2020-11-08T21:12:10Z

@elasticmachine merge upstream

kibanamachine · 2020-11-08T22:56:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 042a4a9

Metrics [docs]

async chunks size

id	before	after	diff
`ingestManager`	1.2MB	1.2MB	+34.0B

page load bundle size

id	before	after	diff
`ingestManager`	386.4KB	386.3KB	-78.0B

History

💚 Build #86466 succeeded 9f6c97a
💚 Build #85832 succeeded 755e88d
💚 Build #85680 succeeded aa333fb
💔 Build #85553 failed c50837e
💔 Build #85545 failed 62c4091

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (68 commits) [Fleet] Make stream id unique in agent policy (elastic#82447) skip flaky suite (elastic#82915) skip flaky suite (elastic#75794) Copy `dateAsStringRt` to observability plugin (elastic#82839) [Maps] rename connected_components/map folder to mb_map (elastic#82897) [Security Solution] Fix EventsViewer DnD cypress tests (elastic#82619) [Security Solution] Adds logging and performance fan out API for threat/Indicator matching (elastic#82546) Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' (elastic#79056) [APM] Adds new configuration 'xpack.apm.maxServiceEnvironments' (elastic#82090) Move single use function in line (elastic#82885) [ML] Add unsigned_long support to data frame analytics and anomaly detection (elastic#82636) Add flot_chart dependency from shared_deps to Shareable Runtime (elastic#81649) [Security Solution][Detections] - Auto refresh all rules/monitoring tables (elastic#82062) [APM] Fix apm e2e runner script commands (elastic#82798) [Ingest Manager] Move cache functions to from registry to archive (elastic#82871) Update webpack-dev-server and webpack-cli (elastic#82844) [Uptime] Migrate to new es client (elastic#82003) Move parseAndVerify* functions to validation.ts (elastic#82845) Remove yeoman & yo (elastic#82825) [Canvas] Fix elements not being updated properly when filter is changed on workpad (elastic#81863) ...

…e-details-overlay * 'master' of github.com:elastic/kibana: (201 commits) Added `defaultActionMessage` to index threshold alert UI type definition (elastic#80936) [ILM] Migrate Delete phase and name field to Form Lib (elastic#82834) skip flaky suite (elastic#57426) [Alerting] adds an Run When field in the alert flyout to assign the action to an Action Group (elastic#82472) [APM] Expose APM event client as part of plugin contract (elastic#82724) [Logs UI] Fix errors during navigation (elastic#78319) Enable send to background in TSVB (elastic#82835) SavedObjects search_dsl: add match_phrase_prefix clauses when using prefix search (elastic#82693) [Ingest Manager] Unify install* under installPackage (elastic#82916) [Fleet] Make stream id unique in agent policy (elastic#82447) skip flaky suite (elastic#82915) skip flaky suite (elastic#75794) Copy `dateAsStringRt` to observability plugin (elastic#82839) [Maps] rename connected_components/map folder to mb_map (elastic#82897) [Security Solution] Fix EventsViewer DnD cypress tests (elastic#82619) [Security Solution] Adds logging and performance fan out API for threat/Indicator matching (elastic#82546) Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' (elastic#79056) [APM] Adds new configuration 'xpack.apm.maxServiceEnvironments' (elastic#82090) Move single use function in line (elastic#82885) [ML] Add unsigned_long support to data frame analytics and anomaly detection (elastic#82636) ...

nchaulet self-assigned this Nov 3, 2020

[Fleet] Make stream id unique in agent policy

c50837e

nchaulet force-pushed the feature-fleet-81617-unique-stream-id branch from 62c4091 to c50837e Compare November 3, 2020 16:18

nchaulet added 2 commits November 3, 2020 16:36

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

62eef03

…-81617-unique-stream-id

Fix tests

aa333fb

nchaulet added Team:Fleet Team label for Observability Data Collection Fleet team v7.11.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Nov 3, 2020

nchaulet marked this pull request as ready for review November 3, 2020 22:25

nchaulet requested a review from a team November 3, 2020 22:25

nchaulet requested a review from jen-huang November 3, 2020 22:34

Merge branch 'master' of github.com:elastic/kibana into feature-fleet…

755e88d

…-81617-unique-stream-id

nchaulet requested a review from ruflin November 5, 2020 18:59

ruflin reviewed Nov 6, 2020

View reviewed changes

nchaulet requested a review from a team November 6, 2020 15:24

Merge branch 'master' into feature-fleet-81617-unique-stream-id

9f6c97a

jen-huang approved these changes Nov 6, 2020

View reviewed changes

Merge branch 'master' into feature-fleet-81617-unique-stream-id

042a4a9

nchaulet merged commit 8560b2d into elastic:master Nov 8, 2020

nchaulet deleted the feature-fleet-81617-unique-stream-id branch November 8, 2020 23:51

nchaulet mentioned this pull request Nov 8, 2020

[7.x] [Fleet] Make stream id unique in agent policy (#82447) #82923

Merged

nchaulet added a commit to nchaulet/kibana that referenced this pull request Nov 8, 2020

[Fleet] Make stream id unique in agent policy (elastic#82447)

103b2e8

nchaulet added a commit that referenced this pull request Nov 9, 2020

[Fleet] Make stream id unique in agent policy (#82447) (#82923)

2d41717

andrewkroh mentioned this pull request Jun 9, 2021

[Elastic Agent] Filebeat registry collisions when an integration is used twice elastic/beats#22054

Closed

cmacknz mentioned this pull request Jun 15, 2022

Update API protocol elastic/elastic-agent-shipper#56

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Make stream id unique in agent policy #82447

[Fleet] Make stream id unique in agent policy #82447

nchaulet commented Nov 3, 2020 •

edited

Loading

elasticmachine commented Nov 3, 2020

ruflin left a comment

nchaulet commented Nov 6, 2020

ruflin commented Nov 6, 2020

nchaulet commented Nov 6, 2020

jen-huang left a comment

jen-huang Nov 6, 2020

nchaulet commented Nov 8, 2020

kibanamachine commented Nov 8, 2020

[Fleet] Make stream id unique in agent policy #82447

[Fleet] Make stream id unique in agent policy #82447

Conversation

nchaulet commented Nov 3, 2020 • edited Loading

Description

Example config

After

Before

How to test this?

elasticmachine commented Nov 3, 2020

ruflin left a comment

Choose a reason for hiding this comment

nchaulet commented Nov 6, 2020

ruflin commented Nov 6, 2020

nchaulet commented Nov 6, 2020

jen-huang left a comment

Choose a reason for hiding this comment

jen-huang Nov 6, 2020

Choose a reason for hiding this comment

nchaulet commented Nov 8, 2020

kibanamachine commented Nov 8, 2020

💚 Build Succeeded

Metrics [docs]

async chunks size

page load bundle size

History

nchaulet commented Nov 3, 2020 •

edited

Loading