Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Make agent Policy input id unique #83101

Closed

Conversation

nchaulet
Copy link
Member

@nchaulet nchaulet commented Nov 10, 2020

Summary

Resolve #82426

In the effort to have better agent observability and status per input, we need to have unique id per input. (this will probably help with the future Fleet server data model too)

This PR change input ids to be unique.

Details

Input ids are now build like this ${input.type}-${packagePolicyId}

This give the following config for example

Screen Shot 2020-11-11 at 10 35 51 AM

This a breaking change as packagePolicyId is no longer equal to inputId

@kevinlog This require change in the endpoint app

@nchaulet nchaulet added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes 7.11.0 labels Nov 10, 2020
@nchaulet nchaulet self-assigned this Nov 10, 2020
@jen-huang jen-huang added v7.11.0 and removed 7.11.0 labels Nov 10, 2020
@kibanamachine
Copy link
Contributor

kibanamachine commented Nov 11, 2020

💔 Build Failed

Failed CI Steps


Test Failures

X-Pack Endpoint Functional Tests.x-pack/test/security_solution_endpoint/apps/endpoint/policy_details·ts.endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 1 times on tracked branches: https://dryrun

[00:00:00]       │
[00:00:00]         └-: endpoint
[00:00:00]           └-> "before all" hook
[00:00:00]           └-> "before all" hook
[00:00:00]             │ proc [kibana]   log   [16:46:43.177] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:46:43.181] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39930, url.original: /search?package=endpoint&internal=true&experimental=true
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39934, url.original: /search?package=system&internal=true&experimental=true
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ proc [kibana]   log   [16:46:43.204] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:46:43.205] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39950, url.original: /search?package=system&internal=true&experimental=true
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39948, url.original: /search?package=endpoint&internal=true&experimental=true
[00:00:00]             │ proc [kibana]   log   [16:46:43.222] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:46:43.223] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39956, url.original: /package/system/0.7.0
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39958, url.original: /package/endpoint/0.16.0-dev.4
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39966, url.original: /package/system/0.7.0/
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39964, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ proc [kibana]   log   [16:46:43.316] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39970, url.original: /epr/system/system-0.7.0.zip
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:00]             │ proc [kibana]   log   [16:46:43.404] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39974, url.original: /epr/endpoint/endpoint-0.16.0-dev.4.zip
[00:00:00]             │ proc [kibana]   log   [16:46:43.475] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39978, url.original: /package/endpoint/0.16.0-dev.4
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39982, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:00]             │ proc [kibana]   log   [16:46:43.541] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39986, url.original: /package/system/0.7.0
[00:00:00]             │ info [docker:registry] 2020/11/11 16:46:43 source.ip: 172.17.0.1:39990, url.original: /package/system/0.7.0/
[00:00:01]             │ proc [kibana]   log   [16:46:44.182] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:01]             │ proc [kibana]   log   [16:46:44.186] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40000, url.original: /search?experimental=true
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:39998, url.original: /search?experimental=true
[00:00:01]             │ proc [kibana]   log   [16:46:44.219] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:01]             │ proc [kibana]   log   [16:46:44.220] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:01]             │ proc [kibana]   log   [16:46:44.222] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:01]             │ proc [kibana]   log   [16:46:44.222] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40014, url.original: /package/system/0.7.0
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40016, url.original: /package/system/0.7.0
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40012, url.original: /package/endpoint/0.16.0-dev.4
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40008, url.original: /package/endpoint/0.16.0-dev.4
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40026, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40032, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40030, url.original: /package/system/0.7.0/
[00:00:01]             │ info [docker:registry] 2020/11/11 16:46:44 source.ip: 172.17.0.1:40028, url.original: /package/system/0.7.0/
[00:00:01]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:02]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:02]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:02]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:04]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-metadata-current] for index patterns [metrics-endpoint.metadata_current_*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.security-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.network-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.library-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [metrics-endpoint.policy-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.alerts-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [metrics-endpoint.metadata-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.file-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.registry-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [metrics-endpoint.metrics-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.process-mappings]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
[00:00:07]             │ proc [kibana]   log   [16:46:50.225] [info][ingestManager][plugins] Found previous transform references:
[00:00:07]             │ proc [kibana]  []
[00:00:07]             │ proc [kibana]   log   [16:46:50.226] [info][ingestManager][plugins] Deleting currently installed transform ids 
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
[00:00:07]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
[00:00:07]             │ proc [kibana]   log   [16:46:50.663] [info][ingestManager][plugins] Found previous transform references:
[00:00:07]             │ proc [kibana]  []
[00:00:07]             │ proc [kibana]   log   [16:46:50.664] [info][ingestManager][plugins] Deleting currently installed transform ids 
[00:00:08]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.transform-internal-005] creating index, cause [auto(bulk api)], templates [.transform-internal-005], shards [1]/[1]
[00:00:08]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] updating number_of_replicas to [0] for indices [.transform-internal-005]
[00:00:08]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.transform-notifications-000002] creating index, cause [auto(bulk api)], templates [.transform-notifications-000002], shards [1]/[1]
[00:00:08]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] updating number_of_replicas to [0] for indices [.transform-notifications-000002]
[00:00:08]             │ info [o.e.x.t.t.p.SchemaUtil] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] Failed to deduce mapping for [agent.id], fall back to keyword. Create the destination index with complete mappings first to avoid deducing the mappings
[00:00:08]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [metrics-endpoint.metadata_current_default] creating index, cause [api], templates [metrics-metadata-current], shards [1]/[1]
[00:00:08]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] updating number_of_replicas to [0] for indices [metrics-endpoint.metadata_current_default]
[00:00:08]             │ info [o.e.x.t.t.TransformTask] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [endpoint.metadata_current-default-0.16.0-dev.4] updating state for transform to [{"task_state":"started","indexer_state":"stopped","checkpoint":0,"should_stop_at_checkpoint":false}].
[00:00:08]             │ info [o.e.x.t.t.TransformPersistentTasksExecutor] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [endpoint.metadata_current-default-0.16.0-dev.4] successfully completed and scheduled task in node operation
[00:00:09]             │ proc [kibana]   log   [16:46:52.373] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:09]             │ proc [kibana]   log   [16:46:52.374] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40100, url.original: /search?package=system&internal=true&experimental=true
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40098, url.original: /package/system/0.7.0
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40104, url.original: /package/system/0.7.0/
[00:00:09]             │ proc [kibana]   log   [16:46:52.467] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:09]             │ proc [kibana]   log   [16:46:52.468] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40110, url.original: /package/system/0.7.0
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40112, url.original: /search?package=system&internal=true&experimental=true
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40116, url.original: /package/system/0.7.0/
[00:00:09]             │ proc [kibana]   log   [16:46:52.555] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40120, url.original: /package/system/0.7.0
[00:00:09]             │ info [docker:registry] 2020/11/11 16:46:52 source.ip: 172.17.0.1:40124, url.original: /package/system/0.7.0/
[00:00:09]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:11]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] added role [fleet_enroll]
[00:00:11]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] added user [fleet_enroll]
[00:00:13]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:14]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/7F88046BSd6hM38lAWzXFA] update_mapping [_doc]
[00:00:45]           └-: When on the Endpoint Policy Details Page
[00:00:45]             └-> "before all" hook
[00:01:13]             └-: and the save button is clicked
[00:01:13]               └-> "before all" hook
[00:01:13]               └-> should display success toast on successful save
[00:01:13]                 └-> "before each" hook: global before each
[00:01:13]                 └-> "before each" hook
[00:01:17]                   │ info Using cached retrieval of endpoint package
[00:01:18]                   │ proc [kibana]   log   [16:48:01.266] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:18]                   │ proc [kibana]   log   [16:48:01.268] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:18]                   │ info [docker:registry] 2020/11/11 16:48:01 source.ip: 172.17.0.1:41526, url.original: /search?package=endpoint&internal=true&experimental=true
[00:01:18]                   │ info [docker:registry] 2020/11/11 16:48:01 source.ip: 172.17.0.1:41524, url.original: /package/endpoint/0.16.0-dev.4
[00:01:18]                   │ info [docker:registry] 2020/11/11 16:48:01 source.ip: 172.17.0.1:41530, url.original: /package/endpoint/0.16.0-dev.4/
[00:01:18]                   │ proc [kibana]   log   [16:48:01.410] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:18]                   │ info [docker:registry] 2020/11/11 16:48:01 source.ip: 172.17.0.1:41534, url.original: /package/endpoint/0.16.0-dev.4
[00:01:18]                   │ info [docker:registry] 2020/11/11 16:48:01 source.ip: 172.17.0.1:41538, url.original: /package/endpoint/0.16.0-dev.4/
[00:01:20]                   │ debg navigateToActualUrl http://localhost:61121/app/security/administration/policy/52f6942d-509d-4012-bd03-40449296c2c8
[00:01:20]                   │ debg browser[INFO] http://localhost:61121/app/security/administration/policy/52f6942d-509d-4012-bd03-40449296c2c8?_t=1605113283943 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:01:20]                   │
[00:01:20]                   │ debg browser[INFO] http://localhost:61121/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:01:20]                   │ debg currentUrl = http://localhost:61121/app/security/administration/policy/52f6942d-509d-4012-bd03-40449296c2c8
[00:01:20]                   │          appUrl = http://localhost:61121/app/security/administration/policy/52f6942d-509d-4012-bd03-40449296c2c8
[00:01:20]                   │ debg TestSubjects.find(kibanaChrome)
[00:01:20]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:01:22]                   │ debg isGlobalLoadingIndicatorVisible
[00:01:22]                   │ debg TestSubjects.exists(globalLoadingIndicator)
[00:01:22]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:01:22]                   │ proc [kibana]   log   [16:48:05.564] [error][data][elasticsearch] [index_not_found_exception]: no such index [filebeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:48:05.564] [error][data][elasticsearch] [index_not_found_exception]: no such index [apm-*-transaction*]
[00:01:22]                   │ proc [kibana]   log   [16:48:05.565] [error][data][elasticsearch] [index_not_found_exception]: no such index [auditbeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:48:05.566] [error][data][elasticsearch] [index_not_found_exception]: no such index [packetbeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:48:05.566] [error][data][elasticsearch] [index_not_found_exception]: no such index [winlogbeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:48:05.567] [error][data][elasticsearch] [index_not_found_exception]: no such index [endgame-*]
[00:01:22]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:01:22]                   │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:01:22]                   │ proc [kibana]   log   [16:48:05.751] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:22]                   │ info [docker:registry] 2020/11/11 16:48:05 source.ip: 172.17.0.1:41558, url.original: /search?package=endpoint&internal=true&experimental=true
[00:01:23]                 │ debg Find.byXPath('//input[@data-test-subj='policyWindowsEvent_dns']/../label') with timeout=10000
[00:01:23]                 │ debg TestSubjects.exists(policyDetailsPage)
[00:01:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsPage"]') with timeout=120000
[00:01:23]                 │ debg TestSubjects.exists(policyDetailsPage)
[00:01:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsPage"]') with timeout=120000
[00:01:23]                 │ debg TestSubjects.find(policyDetailsSaveButton)
[00:01:23]                 │ debg Find.findByCssSelector('[data-test-subj="policyDetailsSaveButton"]') with timeout=10000
[00:01:24]                 │ debg TestSubjects.exists(policyDetailsConfirmModal)
[00:01:24]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsConfirmModal"]') with timeout=120000
[00:01:24]                 │ debg Clicking modal confirm
[00:01:24]                 │ debg TestSubjects.exists(confirmModalTitleText)
[00:01:24]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="confirmModalTitleText"]') with timeout=2500
[00:01:24]                 │ debg TestSubjects.click(confirmModalConfirmButton)
[00:01:24]                 │ debg Find.clickByCssSelector('[data-test-subj="confirmModalConfirmButton"]') with timeout=10000
[00:01:24]                 │ debg Find.findByCssSelector('[data-test-subj="confirmModalConfirmButton"]') with timeout=10000
[00:01:24]                 │ debg TestSubjects.exists(confirmModalTitleText)
[00:01:24]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="confirmModalTitleText"]') with timeout=2500
[00:01:26]                 │ERROR browser[SEVERE] http://localhost:61121/api/fleet/package_policies/52f6942d-509d-4012-bd03-40449296c2c8 - Failed to load resource: the server responded with a status of 400 (Bad Request)
[00:01:26]                 │ debg --- retry.tryForTime error: [data-test-subj="confirmModalTitleText"] is not displayed
[00:01:27]                 │ debg TestSubjects.exists(policyDetailsSuccessMessage)
[00:01:27]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsSuccessMessage"]') with timeout=120000
[00:01:29]                 │ debg --- retry.tryForTime error: [data-test-subj="policyDetailsSuccessMessage"] is not displayed
[00:01:32]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:35]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:38]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:41]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:44]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:47]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:50]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:54]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:57]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:00]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:03]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:06]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:09]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:12]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:15]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:18]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:21]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:24]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:27]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:30]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:33]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:36]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:39]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:42]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:45]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:48]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:51]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:54]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:57]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:00]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:03]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:06]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:09]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:12]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:15]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:18]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:21]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:24]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:27]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:28]                 │ info Taking screenshot "/dev/shm/workspace/parallel/12/kibana/x-pack/test/functional/screenshots/failure/endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save.png"
[00:03:28]                 │ info Current URL is: http://localhost:61121/app/security/administration/policy/52f6942d-509d-4012-bd03-40449296c2c8
[00:03:28]                 │ info Saving page source to: /dev/shm/workspace/parallel/12/kibana/x-pack/test/functional/failure_debug/html/endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save.html
[00:03:28]                 └- ✖ fail: endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save
[00:03:28]                 │      Error: expected testSubject(policyDetailsSuccessMessage) to exist
[00:03:28]                 │       at TestSubjects.existOrFail (/dev/shm/workspace/parallel/12/kibana/test/functional/services/common/test_subjects.ts:62:15)
[00:03:28]                 │ 
[00:03:28]                 │ 

Stack Trace

Error: expected testSubject(policyDetailsSuccessMessage) to exist
    at TestSubjects.existOrFail (/dev/shm/workspace/parallel/12/kibana/test/functional/services/common/test_subjects.ts:62:15)

X-Pack Endpoint Functional Tests.x-pack/test/security_solution_endpoint/apps/endpoint/policy_details·ts.endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: endpoint
[00:00:00]           └-> "before all" hook
[00:00:00]           └-> "before all" hook
[00:00:00]             │ proc [kibana]   log   [16:03:02.151] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.155] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50054, url.original: /search?package=endpoint&internal=true&experimental=true
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50058, url.original: /search?package=system&internal=true&experimental=true
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [logs-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ proc [kibana]   log   [16:03:02.178] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.179] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50068, url.original: /search?package=system&internal=true&experimental=true
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50070, url.original: /search?package=endpoint&internal=true&experimental=true
[00:00:00]             │ proc [kibana]   log   [16:03:02.201] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.202] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50078, url.original: /package/system/0.7.0
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50076, url.original: /package/endpoint/0.16.0-dev.4
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50084, url.original: /package/system/0.7.0/
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50086, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:00]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [metrics-index_pattern_placeholder] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]             │ proc [kibana]   log   [16:03:02.297] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50090, url.original: /epr/system/system-0.7.0.zip
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:00]             │ proc [kibana]   log   [16:03:02.350] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50094, url.original: /epr/endpoint/endpoint-0.16.0-dev.4.zip
[00:00:00]             │ proc [kibana]   log   [16:03:02.415] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50098, url.original: /package/endpoint/0.16.0-dev.4
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50102, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:00]             │ proc [kibana]   log   [16:03:02.503] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50106, url.original: /package/system/0.7.0
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50110, url.original: /package/system/0.7.0/
[00:00:00]             │ proc [kibana]   log   [16:03:02.624] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.628] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50118, url.original: /search?experimental=true
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50120, url.original: /search?experimental=true
[00:00:00]             │ proc [kibana]   log   [16:03:02.650] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.651] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.652] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ proc [kibana]   log   [16:03:02.653] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50134, url.original: /package/system/0.7.0
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50130, url.original: /package/endpoint/0.16.0-dev.4
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50132, url.original: /package/system/0.7.0
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50136, url.original: /package/endpoint/0.16.0-dev.4
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50146, url.original: /package/system/0.7.0/
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50148, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50150, url.original: /package/endpoint/0.16.0-dev.4/
[00:00:00]             │ info [docker:registry] 2020/11/11 16:03:02 source.ip: 172.17.0.1:50152, url.original: /package/system/0.7.0/
[00:00:00]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:01]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:01]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:01]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:04]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-metadata-current] for index patterns [metrics-endpoint.metadata_current_*]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.alerts-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.file-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.security-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.network-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.library-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.process-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [metrics-endpoint.policy-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [metrics-endpoint.metadata-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [metrics-endpoint.metrics-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding component template [logs-endpoint.events.registry-mappings]
[00:00:05]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.diskio] for index patterns [metrics-system.diskio-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.core] for index patterns [metrics-system.core-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-system.auth] for index patterns [logs-system.auth-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.cpu] for index patterns [metrics-system.cpu-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.filesystem] for index patterns [metrics-system.filesystem-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.load] for index patterns [metrics-system.load-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.fsstat] for index patterns [metrics-system.fsstat-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.network] for index patterns [metrics-system.network-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.memory] for index patterns [metrics-system.memory-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.process] for index patterns [metrics-system.process-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.socket_summary] for index patterns [metrics-system.socket_summary-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.process_summary] for index patterns [metrics-system.process_summary-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-system.uptime] for index patterns [metrics-system.uptime-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-system.syslog] for index patterns [logs-system.syslog-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.alerts] for index patterns [logs-endpoint.alerts-*]
[00:00:06]             │ proc [kibana]   log   [16:03:08.669] [info][ingestManager][plugins] Found previous transform references:
[00:00:06]             │ proc [kibana]  []
[00:00:06]             │ proc [kibana]   log   [16:03:08.669] [info][ingestManager][plugins] Deleting currently installed transform ids 
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.file] for index patterns [logs-endpoint.events.file-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.security] for index patterns [logs-endpoint.events.security-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.network] for index patterns [logs-endpoint.events.network-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.library] for index patterns [logs-endpoint.events.library-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.process] for index patterns [logs-endpoint.events.process-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-endpoint.policy] for index patterns [metrics-endpoint.policy-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-endpoint.metadata] for index patterns [metrics-endpoint.metadata-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [metrics-endpoint.metrics] for index patterns [metrics-endpoint.metrics-*]
[00:00:06]             │ info [o.e.c.m.MetadataIndexTemplateService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] adding index template [logs-endpoint.events.registry] for index patterns [logs-endpoint.events.registry-*]
[00:00:06]             │ proc [kibana]   log   [16:03:09.106] [info][ingestManager][plugins] Found previous transform references:
[00:00:06]             │ proc [kibana]  []
[00:00:06]             │ proc [kibana]   log   [16:03:09.107] [info][ingestManager][plugins] Deleting currently installed transform ids 
[00:00:07]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.transform-internal-005] creating index, cause [auto(bulk api)], templates [.transform-internal-005], shards [1]/[1]
[00:00:07]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] updating number_of_replicas to [0] for indices [.transform-internal-005]
[00:00:07]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.transform-notifications-000002] creating index, cause [auto(bulk api)], templates [.transform-notifications-000002], shards [1]/[1]
[00:00:07]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] updating number_of_replicas to [0] for indices [.transform-notifications-000002]
[00:00:07]             │ info [o.e.x.t.t.p.SchemaUtil] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] Failed to deduce mapping for [agent.id], fall back to keyword. Create the destination index with complete mappings first to avoid deducing the mappings
[00:00:07]             │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [metrics-endpoint.metadata_current_default] creating index, cause [api], templates [metrics-metadata-current], shards [1]/[1]
[00:00:07]             │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] updating number_of_replicas to [0] for indices [metrics-endpoint.metadata_current_default]
[00:00:07]             │ info [o.e.x.t.t.TransformTask] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [endpoint.metadata_current-default-0.16.0-dev.4] updating state for transform to [{"task_state":"started","indexer_state":"stopped","checkpoint":0,"should_stop_at_checkpoint":false}].
[00:00:07]             │ info [o.e.x.t.t.TransformPersistentTasksExecutor] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [endpoint.metadata_current-default-0.16.0-dev.4] successfully completed and scheduled task in node operation
[00:00:08]             │ proc [kibana]   log   [16:03:10.805] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:08]             │ proc [kibana]   log   [16:03:10.807] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:10 source.ip: 172.17.0.1:50446, url.original: /search?package=system&internal=true&experimental=true
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:10 source.ip: 172.17.0.1:50444, url.original: /package/system/0.7.0
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:10 source.ip: 172.17.0.1:50450, url.original: /package/system/0.7.0/
[00:00:08]             │ proc [kibana]   log   [16:03:10.904] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:08]             │ proc [kibana]   log   [16:03:10.905] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:10 source.ip: 172.17.0.1:50460, url.original: /search?package=system&internal=true&experimental=true
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:10 source.ip: 172.17.0.1:50458, url.original: /package/system/0.7.0
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:10 source.ip: 172.17.0.1:50464, url.original: /package/system/0.7.0/
[00:00:08]             │ proc [kibana]   log   [16:03:11.000] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:11 source.ip: 172.17.0.1:50476, url.original: /package/system/0.7.0
[00:00:08]             │ info [docker:registry] 2020/11/11 16:03:11 source.ip: 172.17.0.1:50480, url.original: /package/system/0.7.0/
[00:00:09]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:10]             │ info [o.e.x.s.a.r.TransportPutRoleAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] added role [fleet_enroll]
[00:00:10]             │ info [o.e.x.s.a.u.TransportPutUserAction] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] added user [fleet_enroll]
[00:00:12]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:12]             │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1605106243706819857] [.kibana_1/63-qyccfQzSV7we3bTUGFA] update_mapping [_doc]
[00:00:45]           └-: When on the Endpoint Policy Details Page
[00:00:45]             └-> "before all" hook
[00:01:13]             └-: and the save button is clicked
[00:01:13]               └-> "before all" hook
[00:01:13]               └-> should display success toast on successful save
[00:01:13]                 └-> "before each" hook: global before each
[00:01:13]                 └-> "before each" hook
[00:01:17]                   │ info Using cached retrieval of endpoint package
[00:01:17]                   │ proc [kibana]   log   [16:04:19.691] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:17]                   │ proc [kibana]   log   [16:04:19.692] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:17]                   │ info [docker:registry] 2020/11/11 16:04:19 source.ip: 172.17.0.1:53386, url.original: /search?package=endpoint&internal=true&experimental=true
[00:01:17]                   │ info [docker:registry] 2020/11/11 16:04:19 source.ip: 172.17.0.1:53384, url.original: /package/endpoint/0.16.0-dev.4
[00:01:17]                   │ info [docker:registry] 2020/11/11 16:04:19 source.ip: 172.17.0.1:53390, url.original: /package/endpoint/0.16.0-dev.4/
[00:01:17]                   │ proc [kibana]   log   [16:04:19.829] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:17]                   │ info [docker:registry] 2020/11/11 16:04:19 source.ip: 172.17.0.1:53394, url.original: /package/endpoint/0.16.0-dev.4
[00:01:17]                   │ info [docker:registry] 2020/11/11 16:04:19 source.ip: 172.17.0.1:53398, url.original: /package/endpoint/0.16.0-dev.4/
[00:01:20]                   │ debg navigateToActualUrl http://localhost:61121/app/security/administration/policy/66923630-0a02-4b8d-aa73-975e41bd26ea
[00:01:20]                   │ debg browser[INFO] http://localhost:61121/app/security/administration/policy/66923630-0a02-4b8d-aa73-975e41bd26ea?_t=1605110662367 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:01:20]                   │
[00:01:20]                   │ debg browser[INFO] http://localhost:61121/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:01:20]                   │ debg currentUrl = http://localhost:61121/app/security/administration/policy/66923630-0a02-4b8d-aa73-975e41bd26ea
[00:01:20]                   │          appUrl = http://localhost:61121/app/security/administration/policy/66923630-0a02-4b8d-aa73-975e41bd26ea
[00:01:20]                   │ debg TestSubjects.find(kibanaChrome)
[00:01:20]                   │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:01:21]                   │ debg isGlobalLoadingIndicatorVisible
[00:01:21]                   │ debg TestSubjects.exists(globalLoadingIndicator)
[00:01:21]                   │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:01:22]                   │ proc [kibana]   log   [16:04:24.143] [error][data][elasticsearch] [index_not_found_exception]: no such index [apm-*-transaction*]
[00:01:22]                   │ proc [kibana]   log   [16:04:24.144] [error][data][elasticsearch] [index_not_found_exception]: no such index [packetbeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:04:24.145] [error][data][elasticsearch] [index_not_found_exception]: no such index [winlogbeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:04:24.146] [error][data][elasticsearch] [index_not_found_exception]: no such index [auditbeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:04:24.146] [error][data][elasticsearch] [index_not_found_exception]: no such index [filebeat-*]
[00:01:22]                   │ proc [kibana]   log   [16:04:24.147] [error][data][elasticsearch] [index_not_found_exception]: no such index [endgame-*]
[00:01:22]                   │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:01:22]                   │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:01:22]                   │ proc [kibana]   log   [16:04:24.410] [info][ingestManager][plugins] Custom registry url is an experimental feature and is unsupported.
[00:01:22]                   │ info [docker:registry] 2020/11/11 16:04:24 source.ip: 172.17.0.1:53818, url.original: /search?package=endpoint&internal=true&experimental=true
[00:01:23]                 │ debg Find.byXPath('//input[@data-test-subj='policyWindowsEvent_dns']/../label') with timeout=10000
[00:01:23]                 │ debg TestSubjects.exists(policyDetailsPage)
[00:01:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsPage"]') with timeout=120000
[00:01:23]                 │ debg TestSubjects.exists(policyDetailsPage)
[00:01:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsPage"]') with timeout=120000
[00:01:23]                 │ debg TestSubjects.find(policyDetailsSaveButton)
[00:01:23]                 │ debg Find.findByCssSelector('[data-test-subj="policyDetailsSaveButton"]') with timeout=10000
[00:01:23]                 │ debg TestSubjects.exists(policyDetailsConfirmModal)
[00:01:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsConfirmModal"]') with timeout=120000
[00:01:23]                 │ debg Clicking modal confirm
[00:01:23]                 │ debg TestSubjects.exists(confirmModalTitleText)
[00:01:23]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="confirmModalTitleText"]') with timeout=2500
[00:01:23]                 │ debg TestSubjects.click(confirmModalConfirmButton)
[00:01:23]                 │ debg Find.clickByCssSelector('[data-test-subj="confirmModalConfirmButton"]') with timeout=10000
[00:01:23]                 │ debg Find.findByCssSelector('[data-test-subj="confirmModalConfirmButton"]') with timeout=10000
[00:01:24]                 │ debg TestSubjects.exists(confirmModalTitleText)
[00:01:24]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="confirmModalTitleText"]') with timeout=2500
[00:01:26]                 │ERROR browser[SEVERE] http://localhost:61121/api/fleet/package_policies/66923630-0a02-4b8d-aa73-975e41bd26ea - Failed to load resource: the server responded with a status of 400 (Bad Request)
[00:01:26]                 │ debg --- retry.tryForTime error: [data-test-subj="confirmModalTitleText"] is not displayed
[00:01:27]                 │ debg TestSubjects.exists(policyDetailsSuccessMessage)
[00:01:27]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="policyDetailsSuccessMessage"]') with timeout=120000
[00:01:29]                 │ debg --- retry.tryForTime error: [data-test-subj="policyDetailsSuccessMessage"] is not displayed
[00:01:32]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:35]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:38]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:41]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:44]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:47]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:50]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:53]                 │ debg --- retry.tryForTime failed again with the same message...
[00:01:56]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:00]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:03]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:06]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:09]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:12]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:15]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:18]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:21]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:24]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:27]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:30]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:33]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:36]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:39]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:42]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:45]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:48]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:51]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:54]                 │ debg --- retry.tryForTime failed again with the same message...
[00:02:57]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:00]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:03]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:06]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:09]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:12]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:15]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:18]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:21]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:24]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:27]                 │ debg --- retry.tryForTime failed again with the same message...
[00:03:28]                 │ info Taking screenshot "/dev/shm/workspace/parallel/12/kibana/x-pack/test/functional/screenshots/failure/endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save.png"
[00:03:28]                 │ info Current URL is: http://localhost:61121/app/security/administration/policy/66923630-0a02-4b8d-aa73-975e41bd26ea
[00:03:28]                 │ info Saving page source to: /dev/shm/workspace/parallel/12/kibana/x-pack/test/functional/failure_debug/html/endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save.html
[00:03:28]                 └- ✖ fail: endpoint When on the Endpoint Policy Details Page and the save button is clicked should display success toast on successful save
[00:03:28]                 │      Error: expected testSubject(policyDetailsSuccessMessage) to exist
[00:03:28]                 │       at TestSubjects.existOrFail (/dev/shm/workspace/parallel/12/kibana/test/functional/services/common/test_subjects.ts:62:15)
[00:03:28]                 │ 
[00:03:28]                 │ 

Stack Trace

Error: expected testSubject(policyDetailsSuccessMessage) to exist
    at TestSubjects.existOrFail (/dev/shm/workspace/parallel/12/kibana/test/functional/services/common/test_subjects.ts:62:15)

Metrics [docs]

Distributable file count

id before after diff
default 42765 42766 +1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
ingestManager 386.2KB 386.2KB +10.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id before after diff
fleet-agents 25 23 -2
ingest-outputs 11 10 -1
ingest-package-policies 35 31 -4
lens 7 6 -1
total -8

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@paul-tavares
Copy link
Contributor

This change will have impacts to Endpoint in Kibana and possibly also the actual Endpoint (maybe not).
This Package Policy ID is echo'ed back to Elastic Search by the Endpoint as part of the Policy Response message as well as the Endpoint metadata message. In Kibana, we use this ID to present links to the user that allows them to view/edit the Policy as well as jump over to Fleet views.

I don't believe Endpoint uses the ID for any processing, but @ferullo can confirm.

There may also be some impacts to our APIs which use the Fleet services to query for data - @pzl and/or @nnamdifrankie can confirm.

@nchaulet :
A question I have is: I thought the UUID is already unique, so why do we need to prepend the package name?

Any chance this could be done by having the Elastic Agent instead build the id based on {type}-{id}, so that we can minimize the impact? Or introduce a new field to store this new value?


cc:/ @efreeti , @kevinlog , @parkiino

@ferullo
Copy link
Contributor

ferullo commented Nov 16, 2020

@paul-tavares Assuming you are referring to the configuration item inputs[0].id that Endpoint echos back as Endpoint.policy.applied.id, yes Endpoint doesn't do any processing on this field. It's an opaque string to Endpoint.

@nchaulet
Copy link
Member Author

@paul-tavares

A question I have is: I thought the UUID is already unique, so why do we need to prepend the package name?

The uuid is unique for a package policy (by integration) but a package policy could have multiple inputs (most of integration have two one for log, one for metrics), so we need to differentiate them

@paul-tavares
Copy link
Contributor

Thanks @nchaulet . I see what you mean (for endpoint, we currently don't support multiples - only one).

@paul-tavares
Copy link
Contributor

I'm thinking a little more about this - @nchaulet do you happen to have a full Agent Policy YAML with Endpoint that you can post here?

@nchaulet
Copy link
Member Author

Yes sure

id: 3e5698a0-28e0-11eb-b346-3d2b3bc4c0e1
revision: 3
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
agent:
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
inputs:
  - id: logfile-73c9941e-a46f-4f88-954d-32b8543823f4
    name: system-1
    revision: 2
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 0.9.1
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.auth
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              fields:
                ecs.version: 1.5.0
              target: ''
      - id: logfile-system.syslog-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.syslog
          type: logs
        exclude_files:
          - .gz$
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
          - add_fields:
              fields:
                ecs.version: 1.5.0
              target: ''
  - id: system/metrics-73c9941e-a46f-4f88-954d-32b8543823f4
    name: system-1
    revision: 2
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 0.9.1
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.cpu-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.cpu
          type: metrics
        period: 10s
        cpu.metrics:
          - percentages
          - normalized_percentages
        metricsets:
          - cpu
      - id: system/metrics-system.diskio-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.diskio
          type: metrics
        period: 10s
        diskio.include_devices: null
        metricsets:
          - diskio
      - id: system/metrics-system.filesystem-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.filesystem
          type: metrics
        period: 1m
        metricsets:
          - filesystem
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.fsstat-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.fsstat
          type: metrics
        period: 1m
        metricsets:
          - fsstat
        processors:
          - drop_event.when.regexp:
              system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.load-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.load
          type: metrics
        period: 10s
        metricsets:
          - load
      - id: system/metrics-system.memory-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.memory
          type: metrics
        period: 10s
        metricsets:
          - memory
      - id: system/metrics-system.network-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.network
          type: metrics
        period: 10s
        network.interfaces: null
        metricsets:
          - network
      - id: system/metrics-system.process-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.process
          type: metrics
        process.include_top_n.by_memory: 5
        period: 10s
        processes:
          - .*
        process.include_top_n.by_cpu: 5
        process.cgroups.enabled: false
        process.cmdline.cache.enabled: true
        metricsets:
          - process
        process.include_cpu_ticks: false
      - id: >-
          system/metrics-system.process_summary-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.process_summary
          type: metrics
        period: 10s
        metricsets:
          - process_summary
      - id: >-
          system/metrics-system.socket_summary-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.socket_summary
          type: metrics
        period: 10s
        metricsets:
          - socket_summary
      - id: system/metrics-system.uptime-73c9941e-a46f-4f88-954d-32b8543823f4
        data_stream:
          dataset: system.uptime
          type: metrics
        period: 10s
        metricsets:
          - uptime
  - id: endpoint-743cc0db-072c-4b57-9eb3-de66092f33c1
    name: endpoint
    revision: 1
    type: endpoint
    use_output: default
    meta:
      package:
        name: endpoint
        version: 0.16.1
    data_stream:
      namespace: default
    artifact_manifest:
      manifest_version: 1.0.0
      schema_version: v1
      artifacts:
        endpoint-exceptionlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/endpoint/artifacts/download/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-exceptionlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/endpoint/artifacts/download/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-macos-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/endpoint/artifacts/download/endpoint-trustlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-windows-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/endpoint/artifacts/download/endpoint-trustlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
        endpoint-trustlist-linux-v1:
          encryption_algorithm: none
          decoded_sha256: d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          decoded_size: 14
          encoded_sha256: f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda
          encoded_size: 22
          relative_url: >-
            /api/endpoint/artifacts/download/endpoint-trustlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658
          compression_algorithm: zlib
    policy:
      windows:
        events:
          dll_and_driver_load: true
          dns: true
          file: true
          network: true
          process: true
          registry: true
          security: true
        malware:
          mode: prevent
        popup:
          malware:
            enabled: true
            message: ''
        logging:
          file: info
        antivirus_registration:
          enabled: false
      mac:
        events:
          process: true
          file: true
          network: true
        malware:
          mode: prevent
        popup:
          malware:
            enabled: true
            message: ''
        logging:
          file: info
      linux:
        events:
          process: true
          file: true
          network: true
        logging:
          file: info
    streams: []
fleet:
  kibana:
    protocol: http
    hosts:
      - 'localhost:5601'

@paul-tavares
Copy link
Contributor

Thanks @nchaulet . I reconfirmed our impacts in Kibana that I previously highlighted

@nchaulet
Copy link
Member Author

@paul-tavares do you see a way we can fix endpoint, and have unique input ids?

@kevinlog
Copy link
Contributor

@paul-tavares @nchaulet , some thoughts here. I'm seeing a few paths forward.

  1. Keep the ID's appended with the input type.
    With the current approach, we would need to adjust our APIs in order to access the package policies to take the endpoint- prefix into account. Seemingly, we'd need to update our API calls and ensure we strip the endpoint prefix out.

  2. It looks like the type used in the prefix is accessible in the YAML next to the ID. Could the statuses feature use the type + id to assign the correct status by looking at both fields instead of prepending the UUID?

  3. Alternatively, we could strip the type prefix out before sending it down to the Endpoint. That way we'd get the UUID without endpoint echoed back. Not sure if this is a good idea - also it seems like a special case for Endpoint in Agent which is undesirable.

FYI @nnamdifrankie @pzl @parkiino

@nchaulet
Copy link
Member Author

@kevinlog I am not sure keeping the prefix + id is completely future proof,

I am wondering if the better solution here is to allow you to search a package policy for a input Id when you need it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Fleet] Make Agent Policy inputs ids unique
7 participants