diff --git a/docs/epm/index.asciidoc b/docs/epm/index.asciidoc new file mode 100644 index 00000000000000..46d45b85690e30 --- /dev/null +++ b/docs/epm/index.asciidoc @@ -0,0 +1,146 @@ +[role="xpack"] +[[epm]] +== Elastic Package Manager + +These are the docs for the Elastic Package Manager (EPM). + + +=== Configuration + +The Elastic Package Manager by default access `epr.elastic.co` to retrieve the package. The url can be configured with: + +``` +xpack.epm.registryUrl: 'http://localhost:8080' +``` + +=== API + +The Package Manager offers an API. Here an example on how they can be used. + +List installed packages: + +``` +curl localhost:5601/api/ingest_manager/epm/packages +``` + +Install a package: + +``` +curl -X POST localhost:5601/api/ingest_manager/epm/packages/iptables-1.0.4 +``` + +Delete a package: + +``` +curl -X DELETE localhost:5601/api/ingest_manager/epm/packages/iptables-1.0.4 +``` + +=== Definitions + +This section is to define terms used across ingest management. + +==== Elastic Agent +A single, unified agent that users can deploy to hosts or containers. It controls which data is collected from the host or containers and where the data is sent. It will run Beats, Endpoint or other monitoring programs as needed. It can operate standalone or pull a configuration policy from Fleet. + +==== Namespace +A user-specified string that will be used to part of the index name in Elasticsearch. It helps users identify logs coming from a specific environment (like prod or test), an application, or other identifiers. + +==== Package + +A package contains all the assets for the Elastic Stack. A more detailed definition of a package can be found under https://github.com/elastic/package-registry . + + +== Indexing Strategy + +Ingest Management enforces an indexing strategy to allow the system to automically detect indices and run queries on it. In short the indexing strategy looks as following: + +``` +{type}-{dataset}-{namespace} +``` + +The `{type}` can be `logs` or `metrics`. The `{namespace}` is the part where the user can use free form. The only two requirement are that it has only characters allowed in an Elasticsearch index name and does NOT contain a `-`. The `dataset` is defined by the data that is indexed. The same requirements as for the namespace apply. It is expected that the fields for type, namespace and dataset are part of each event and are constant keywords. + +Note: More `{type}`s might be added in the future like `apm` and `endpoint`. + +This indexing strategy has a few advantages: + +* Each index contains only the fields which are relevant for the dataset. This leads to more dense indices and better field completion. +* ILM policies can be applied per namespace per dataset. +* Rollups can be specified per namespace per dataset. +* Having the namespace user configurable makes setting security permissions possible. +* Having a global metrics and logs template, allows to create new indices on demand which still follow the convention. This is common in the case of k8s as an example. +* Constant keywords allow to narrow down the indices we need to access for querying very efficiently. This is especially relevant in environments which a large number of indices or with indices on slower nodes. + +=== Ingest Pipeline + +The ingest pipelines for a specific dataset will have the following naming scheme: + +``` +{type}-{dataset}-{package.version} +``` + +As an example, the ingest pipeline for the Nginx access logs is called `logs-nginx.access-3.4.1`. The same ingest pipeline is used for all namespaces. It is possible that a dataset has multiple ingest pipelines in which case a suffix is added to the name. + +The version is included in each pipeline to allow upgrades. The pipeline itself is listed in the index template and is automatically applied at ingest time. + +=== Templates & ILM Policies + +To make the above strategy possible, alias templates are required. For each type there is a basic alias template with a default ILM policy. These default templates apply to all indices which follow the indexing strategy and do not have a more specific dataset alias template. + +The `metrics` and `logs` alias template contain all the basic fields from ECS. + +Each type template contains an ILM policy. Modifying this default ILM policy will affect all data covered by the default templates. + +The templates for a dataset are called as following: + +``` +{type}-{dataset} +``` + +The pattern used inside the index template is `{type}-{dataset}-*` to match all namespaces. + +=== Defaults + +If the Elastic Agent is used to ingest data and only the type is specified, `default` for the namespace is used and `generic` for the dataset. + +=== Data filtering + +Filtering for data in queries for example in visualizations or dashboards should always be done on the constant keyword fields. Visualizations needing data for the nginx.access dataset should query on `type:logs AND dataset:nginx.access`. As these are constant keywords the prefiltering is very efficient. + +=== Security permissions + +Security permissions can be set on different levels. To set special permissions for the access on the prod namespace an index pattern as below can be used: + +``` +/(logs|metrics)-[^-]+-prod-$/ +``` + +To set specific permissions on the logs index, the following can be used: + +``` +/^(logs|metrics)-.*/ +``` + +Todo: The above queries need to be tested. + + + +== Package Manager + +=== Package Upgrades + +When upgrading a package between a bugfix or a minor version, no breaking changes should happen. Upgrading a package has the following effect: + +* Removal of existing dashboards +* Installation of new dashboards +* Write new ingest pipelines with the version +* Write new Elasticsearch alias templates +* Trigger a rollover for all the affected indices + +The new ingest pipeline is expected to still work with the data coming from older configurations. In most cases this means some of the fields can be missing. For this to work, each event must contain the version of config / package it is coming from to make such a decision. + +In case of a breaking change in the data structure, the new ingest pipeline is also expected to deal with this change. In case there are breaking changes which cannot be dealt with in an ingest pipeline, a new package has to be created. + +Each package lists its minimal required agent version. In case there are agents enrolled with an older version, the user is notified to upgrade these agents as otherwise the new configs cannot be rolled out. + + diff --git a/package.json b/package.json index 5bb69f03945c91..d4f19eecc7527f 100644 --- a/package.json +++ b/package.json @@ -135,6 +135,7 @@ "@kbn/test-subj-selector": "0.2.1", "@kbn/ui-framework": "1.0.0", "@kbn/ui-shared-deps": "1.0.0", + "@types/tar": "^4.0.3", "JSONStream": "1.3.5", "abortcontroller-polyfill": "^1.4.0", "angular": "^1.7.9", @@ -449,6 +450,7 @@ "listr": "^0.14.1", "load-grunt-config": "^3.0.1", "mocha": "^6.2.2", + "mock-http-server": "1.3.0", "multistream": "^2.1.1", "murmurhash3js": "3.0.1", "mutation-observer": "^1.0.3", diff --git a/src/test_utils/kbn_server.ts b/src/test_utils/kbn_server.ts index 0ed33928ff63ce..f6140ea57ed057 100644 --- a/src/test_utils/kbn_server.ts +++ b/src/test_utils/kbn_server.ts @@ -249,7 +249,7 @@ export function createTestServers({ return { startES: async () => { - await es.start(); + await es.start(get(settings, 'es.esArgs', [])); if (['gold', 'trial'].includes(license)) { await setupUsers({ diff --git a/x-pack/legacy/plugins/ingest_manager/index.ts b/x-pack/legacy/plugins/ingest_manager/index.ts index 7ed5599b234a34..47c6478f66471d 100644 --- a/x-pack/legacy/plugins/ingest_manager/index.ts +++ b/x-pack/legacy/plugins/ingest_manager/index.ts @@ -9,6 +9,7 @@ import { OUTPUT_SAVED_OBJECT_TYPE, AGENT_CONFIG_SAVED_OBJECT_TYPE, DATASOURCE_SAVED_OBJECT_TYPE, + PACKAGES_SAVED_OBJECT_TYPE, } from '../../../plugins/ingest_manager/server'; // TODO https://github.com/elastic/kibana/issues/46373 @@ -34,6 +35,10 @@ export function ingestManager(kibana: any) { isNamespaceAgnostic: true, // indexPattern: INDEX_NAMES.INGEST, }, + [PACKAGES_SAVED_OBJECT_TYPE]: { + isNamespaceAgnostic: true, + // indexPattern: INDEX_NAMES.INGEST, + }, }, mappings: savedObjectMappings, }, diff --git a/x-pack/package.json b/x-pack/package.json index 1acaf4964a14d3..d94a9aee062a86 100644 --- a/x-pack/package.json +++ b/x-pack/package.json @@ -33,7 +33,7 @@ "@kbn/plugin-helpers": "9.0.2", "@kbn/test": "1.0.0", "@kbn/utility-types": "1.0.0", - "@mattapperson/slapshot": "1.4.0", + "@mattapperson/slapshot": "1.4.3", "@storybook/addon-actions": "^5.2.6", "@storybook/addon-console": "^1.2.1", "@storybook/addon-knobs": "^5.2.6", @@ -69,6 +69,7 @@ "@types/history": "^4.7.3", "@types/jest": "24.0.19", "@types/joi": "^13.4.2", + "@types/js-search": "^1.4.0", "@types/js-yaml": "^3.11.1", "@types/jsdom": "^12.2.4", "@types/json-stable-stringify": "^1.0.32", @@ -258,6 +259,7 @@ "isbinaryfile": "4.0.2", "joi": "^13.5.2", "jquery": "^3.4.1", + "js-search": "^1.4.3", "js-yaml": "3.13.1", "json-stable-stringify": "^1.0.1", "jsonwebtoken": "^8.5.1", diff --git a/x-pack/plugins/ingest_manager/README.md b/x-pack/plugins/ingest_manager/README.md index 60c2a457a28064..241138880780f5 100644 --- a/x-pack/plugins/ingest_manager/README.md +++ b/x-pack/plugins/ingest_manager/README.md @@ -1,20 +1,81 @@ # Ingest Manager +## Plugin + - No features enabled by default. See the TypeScript type for the [the available plugin configuration options](https://github.com/elastic/kibana/blob/feature-ingest/x-pack/plugins/ingest_manager/common/types/index.ts#L9-L19) + - Setting `xpack.ingestManager.enabled=true` is required to enable the plugin. It adds the `DATASOURCE_API_ROUTES` and `AGENT_CONFIG_API_ROUTES` values in [`common/constants/routes.ts`](./common/constants/routes.ts) + - Adding `--xpack.ingestManager.epm.enabled=true` will add the EPM API & UI + - Adding `--xpack.ingestManager.fleet.enabled=true` will add the Fleet API & UI + - [code for adding the routes](https://github.com/elastic/kibana/blob/1f27d349533b1c2865c10c45b2cf705d7416fb36/x-pack/plugins/ingest_manager/server/plugin.ts#L115-L133) + - [Integration tests](server/integration_tests/router.test.ts) + - Both EPM and Fleet require `ingestManager` be enabled. They are not standalone features. -## Getting started -See the Kibana docs for [how to set up your dev environment](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#setting-up-your-development-environment), [run Elasticsearch](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-elasticsearch), and [start Kibana](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-kibana). +## Development -One common workflow is: +### Getting started +See the Kibana docs for [how to set up your dev environment](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#setting-up-your-development-environment), [run Elasticsearch](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-elasticsearch), and [start Kibana](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-kibana) - 1. `yarn es snapshot` - 1. In another shell: `yarn start --xpack.ingestManager.enabled=true` (or set in `config.yml`) -## HTTP API - 1. Nothing by default. If `xpack.ingestManager.enabled=true`, it adds the `DATASOURCE_API_ROUTES` and `AGENT_CONFIG_API_ROUTES` values in [`common/constants/routes.ts`](./common/constants/routes.ts) - 1. [Integration tests](../../test/api_integration/apis/ingest_manager/endpoints.ts) - 1. In later versions the EPM and Fleet routes will be added when their flags are enabled. See the [currently disabled logic to add those routes](https://github.com/jfsiii/kibana/blob/feature-ingest-manager/x-pack/plugins/ingest_manager/server/plugin.ts#L86-L90). +One common development workflow is: + - Start Elasticsearch in one shell + ``` + yarn es snapshot -E xpack.security.authc.api_key.enabled=true + ``` + - Start Kibana in another shell + ``` + yarn start --xpack.ingestManager.enabled=true --xpack.ingestManager.epm.enabled=true --xpack.ingestManager.fleet.enabled=true + ``` -## Plugin architecture -Follows the `common`, `server`, `public` structure from the [Architecture Style Guide -](https://github.com/elastic/kibana/blob/master/style_guides/architecture_style_guide.md#file-and-folder-structure). +This plugin follows the `common`, `server`, `public` structure from the [Architecture Style Guide +](https://github.com/elastic/kibana/blob/master/style_guides/architecture_style_guide.md#file-and-folder-structure). We also follow the pattern of developing feature branches under your personal fork of Kibana. -We use New Platform approach (structure, APIs, etc) where possible. There's a `kibana.json` manifest, and the server uses the `server/{index,plugin}.ts` approach from [`MIGRATION.md`](https://github.com/elastic/kibana/blob/master/src/core/MIGRATION.md#architecture). \ No newline at end of file +### API Tests +#### Ingest & Fleet + 1. In one terminal, change to the `x-pack` directory and start the test server with + ``` + node scripts/functional_tests_server.js --config test/api_integration/config.ts + ``` + + 1. in a second terminal, run the tests from the Kibana root directory with + ``` + node scripts/functional_test_runner.js --config x-pack/test/api_integration/config.ts + ``` +#### EPM + 1. In one terminal, change to the `x-pack` directory and start the test server with + ``` + node scripts/functional_tests_server.js --config test/epm_api_integration/config.ts + ``` + + 1. in a second terminal, run the tests from the Kibana root directory with + ``` + node scripts/functional_test_runner.js --config x-pack/test/epm_api_integration/config.ts + ``` + + ### Staying up-to-date with `master` + While we're developing in the `feature-ingest` feature branch, here's is more information on keeping up to date with upstream kibana. + +
+ merge upstream master into feature-ingest + +```bash +## checkout feature branch to your fork +git checkout -B feature-ingest origin/feature-ingest + +## make sure your feature branch is current with upstream feature branch +git pull upstream feature-ingest + +## pull in changes from upstream master +git pull upstream master + +## push changes to your remote +git push origin + +# /!\ Open a DRAFT PR /!\ +# Normal PRs will re-notify authors of commits already merged +# Draft PR will trigger CI run. Once CI is green ... +# /!\ DO NOT USE THE GITHUB UI TO MERGE THE PR /!\ + +## push your changes to upstream feature branch from the terminal; not GitHub UI +git push upstream +``` +
+ +See https://github.com/elastic/kibana/pull/37950 for an example. diff --git a/x-pack/plugins/ingest_manager/common/constants/agent.ts b/x-pack/plugins/ingest_manager/common/constants/agent.ts new file mode 100644 index 00000000000000..fe6f7f57e28996 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/constants/agent.ts @@ -0,0 +1,16 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export const AGENT_SAVED_OBJECT_TYPE = 'agents'; + +export const AGENT_EVENT_SAVED_OBJECT_TYPE = 'agent_events'; + +export const AGENT_TYPE_PERMANENT = 'PERMANENT'; +export const AGENT_TYPE_EPHEMERAL = 'EPHEMERAL'; +export const AGENT_TYPE_TEMPORARY = 'TEMPORARY'; + +export const AGENT_POLLING_THRESHOLD_MS = 30000; +export const AGENT_POLLING_INTERVAL = 1000; diff --git a/x-pack/plugins/ingest_manager/common/constants/agent_config.ts b/x-pack/plugins/ingest_manager/common/constants/agent_config.ts index d0854d6ffeec7d..337022e5522786 100644 --- a/x-pack/plugins/ingest_manager/common/constants/agent_config.ts +++ b/x-pack/plugins/ingest_manager/common/constants/agent_config.ts @@ -3,16 +3,17 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { AgentConfigStatus } from '../types'; +import { AgentConfigStatus, DefaultPackages } from '../types'; export const AGENT_CONFIG_SAVED_OBJECT_TYPE = 'agent_configs'; -export const DEFAULT_AGENT_CONFIG_ID = 'default'; - export const DEFAULT_AGENT_CONFIG = { name: 'Default config', namespace: 'default', description: 'Default agent configuration created by Kibana', status: AgentConfigStatus.Active, datasources: [], + is_default: true, }; + +export const DEFAULT_AGENT_CONFIGS_PACKAGES = [DefaultPackages.system]; diff --git a/x-pack/plugins/ingest_manager/common/constants/enrollment_api_key.ts b/x-pack/plugins/ingest_manager/common/constants/enrollment_api_key.ts new file mode 100644 index 00000000000000..f4a4bcde2f393d --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/constants/enrollment_api_key.ts @@ -0,0 +1,7 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export const ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE = 'enrollment_api_keys'; diff --git a/x-pack/plugins/ingest_manager/common/constants/epm.ts b/x-pack/plugins/ingest_manager/common/constants/epm.ts new file mode 100644 index 00000000000000..eb72c28e7bf399 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/constants/epm.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export const PACKAGES_SAVED_OBJECT_TYPE = 'epm-package'; +export const INDEX_PATTERN_SAVED_OBJECT_TYPE = 'index-pattern'; diff --git a/x-pack/plugins/ingest_manager/common/constants/index.ts b/x-pack/plugins/ingest_manager/common/constants/index.ts index aa3b204be48890..45d315e6d56649 100644 --- a/x-pack/plugins/ingest_manager/common/constants/index.ts +++ b/x-pack/plugins/ingest_manager/common/constants/index.ts @@ -6,6 +6,9 @@ export * from './plugin'; export * from './routes'; +export * from './agent'; export * from './agent_config'; export * from './datasource'; +export * from './epm'; export * from './output'; +export * from './enrollment_api_key'; diff --git a/x-pack/plugins/ingest_manager/common/constants/output.ts b/x-pack/plugins/ingest_manager/common/constants/output.ts index e0262d0ca811ca..6060a2b63fc8ea 100644 --- a/x-pack/plugins/ingest_manager/common/constants/output.ts +++ b/x-pack/plugins/ingest_manager/common/constants/output.ts @@ -7,12 +7,10 @@ import { OutputType } from '../types'; export const OUTPUT_SAVED_OBJECT_TYPE = 'outputs'; -export const DEFAULT_OUTPUT_ID = 'default'; - export const DEFAULT_OUTPUT = { - name: DEFAULT_OUTPUT_ID, + name: 'default', + is_default: true, type: OutputType.Elasticsearch, hosts: [''], - ingest_pipeline: DEFAULT_OUTPUT_ID, api_key: '', }; diff --git a/x-pack/plugins/ingest_manager/common/constants/plugin.ts b/x-pack/plugins/ingest_manager/common/constants/plugin.ts index 7922e6cadfa288..c2390bb433953d 100644 --- a/x-pack/plugins/ingest_manager/common/constants/plugin.ts +++ b/x-pack/plugins/ingest_manager/common/constants/plugin.ts @@ -3,5 +3,4 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ - export const PLUGIN_ID = 'ingestManager'; diff --git a/x-pack/plugins/ingest_manager/common/constants/routes.ts b/x-pack/plugins/ingest_manager/common/constants/routes.ts index efd6ef17ba05b0..1dc98f9bc89476 100644 --- a/x-pack/plugins/ingest_manager/common/constants/routes.ts +++ b/x-pack/plugins/ingest_manager/common/constants/routes.ts @@ -11,11 +11,14 @@ export const EPM_API_ROOT = `${API_ROOT}/epm`; export const FLEET_API_ROOT = `${API_ROOT}/fleet`; // EPM API routes +const EPM_PACKAGES_MANY = `${EPM_API_ROOT}/packages`; +const EPM_PACKAGES_ONE = `${EPM_PACKAGES_MANY}/{pkgkey}`; export const EPM_API_ROUTES = { - LIST_PATTERN: `${EPM_API_ROOT}/list`, - INFO_PATTERN: `${EPM_API_ROOT}/package/{pkgkey}`, - INSTALL_PATTERN: `${EPM_API_ROOT}/install/{pkgkey}`, - DELETE_PATTERN: `${EPM_API_ROOT}/delete/{pkgkey}`, + LIST_PATTERN: EPM_PACKAGES_MANY, + INFO_PATTERN: EPM_PACKAGES_ONE, + INSTALL_PATTERN: EPM_PACKAGES_ONE, + DELETE_PATTERN: EPM_PACKAGES_ONE, + FILEPATH_PATTERN: `${EPM_PACKAGES_ONE}/{filePath*}`, CATEGORIES_PATTERN: `${EPM_API_ROOT}/categories`, }; @@ -35,6 +38,28 @@ export const AGENT_CONFIG_API_ROUTES = { CREATE_PATTERN: `${AGENT_CONFIG_API_ROOT}`, UPDATE_PATTERN: `${AGENT_CONFIG_API_ROOT}/{agentConfigId}`, DELETE_PATTERN: `${AGENT_CONFIG_API_ROOT}/delete`, + FULL_INFO_PATTERN: `${AGENT_CONFIG_API_ROOT}/{agentConfigId}/full`, +}; + +// Agent API routes +export const AGENT_API_ROUTES = { + LIST_PATTERN: `${FLEET_API_ROOT}/agents`, + INFO_PATTERN: `${FLEET_API_ROOT}/agents/{agentId}`, + UPDATE_PATTERN: `${FLEET_API_ROOT}/agents/{agentId}`, + DELETE_PATTERN: `${FLEET_API_ROOT}/agents/{agentId}`, + EVENTS_PATTERN: `${FLEET_API_ROOT}/agents/{agentId}/events`, + CHECKIN_PATTERN: `${FLEET_API_ROOT}/agents/{agentId}/checkin`, + ACKS_PATTERN: `${FLEET_API_ROOT}/agents/{agentId}/acks`, + ENROLL_PATTERN: `${FLEET_API_ROOT}/agents/enroll`, + UNENROLL_PATTERN: `${FLEET_API_ROOT}/agents/unenroll`, + STATUS_PATTERN: `${FLEET_API_ROOT}/agent-status`, +}; + +export const ENROLLMENT_API_KEY_ROUTES = { + CREATE_PATTERN: `${FLEET_API_ROOT}/enrollment-api-keys`, + LIST_PATTERN: `${FLEET_API_ROOT}/enrollment-api-keys`, + INFO_PATTERN: `${FLEET_API_ROOT}/enrollment-api-keys/{keyId}`, + DELETE_PATTERN: `${FLEET_API_ROOT}/enrollment-api-keys/{keyId}`, }; // Fleet setup API routes @@ -42,3 +67,7 @@ export const FLEET_SETUP_API_ROUTES = { INFO_PATTERN: `${FLEET_API_ROOT}/setup`, CREATE_PATTERN: `${FLEET_API_ROOT}/setup`, }; + +export const SETUP_API_ROUTE = `${API_ROOT}/setup`; + +export const INSTALL_SCRIPT_API_ROUTES = `${FLEET_API_ROOT}/install/{osType}`; diff --git a/x-pack/plugins/ingest_manager/common/services/agent_status.ts b/x-pack/plugins/ingest_manager/common/services/agent_status.ts new file mode 100644 index 00000000000000..7bbac55f119370 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/services/agent_status.ts @@ -0,0 +1,27 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + AGENT_TYPE_TEMPORARY, + AGENT_POLLING_THRESHOLD_MS, + AGENT_TYPE_PERMANENT, +} from '../constants'; + +export function buildKueryForOnlineAgents() { + return `agents.last_checkin >= now-${(3 * AGENT_POLLING_THRESHOLD_MS) / 1000}s`; +} + +export function buildKueryForOfflineAgents() { + return `agents.type:${AGENT_TYPE_TEMPORARY} AND agents.last_checkin < now-${(3 * + AGENT_POLLING_THRESHOLD_MS) / + 1000}s`; +} + +export function buildKueryForErrorAgents() { + return `agents.type:${AGENT_TYPE_PERMANENT} AND agents.last_checkin < now-${(4 * + AGENT_POLLING_THRESHOLD_MS) / + 1000}s`; +} diff --git a/x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.test.ts b/x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.test.ts new file mode 100644 index 00000000000000..9201cdcb6bbac6 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.test.ts @@ -0,0 +1,115 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { NewDatasource } from '../types'; +import { storedDatasourceToAgentDatasource } from './datasource_to_agent_datasource'; + +describe('Ingest Manager - storedDatasourceToAgentDatasource', () => { + const mockDatasource: NewDatasource = { + name: 'mock-datasource', + description: '', + config_id: '', + enabled: true, + output_id: '', + namespace: 'default', + inputs: [], + }; + + const mockInput = { + type: 'test-logs', + enabled: true, + streams: [ + { + id: 'test-logs-foo', + enabled: true, + dataset: 'foo', + config: { fooVar: 'foo-value', fooVar2: [1, 2] }, + }, + { + id: 'test-logs-bar', + enabled: false, + dataset: 'bar', + config: { barVar: 'bar-value', barVar2: [1, 2] }, + }, + ], + }; + + it('returns agent datasource config for datasource with no inputs', () => { + expect(storedDatasourceToAgentDatasource(mockDatasource)).toEqual({ + id: 'mock-datasource', + namespace: 'default', + enabled: true, + use_output: 'default', + inputs: [], + }); + + expect( + storedDatasourceToAgentDatasource({ + ...mockDatasource, + package: { + name: 'mock-package', + title: 'Mock package', + version: '0.0.0', + }, + }) + ).toEqual({ + id: 'mock-datasource', + namespace: 'default', + enabled: true, + use_output: 'default', + package: { + name: 'mock-package', + version: '0.0.0', + }, + inputs: [], + }); + }); + + it('returns agent datasource config with flattened stream configs', () => { + expect(storedDatasourceToAgentDatasource({ ...mockDatasource, inputs: [mockInput] })).toEqual({ + id: 'mock-datasource', + namespace: 'default', + enabled: true, + use_output: 'default', + inputs: [ + { + type: 'test-logs', + enabled: true, + streams: [ + { + id: 'test-logs-foo', + enabled: true, + dataset: 'foo', + fooVar: 'foo-value', + fooVar2: [1, 2], + }, + { + id: 'test-logs-bar', + enabled: false, + dataset: 'bar', + barVar: 'bar-value', + barVar2: [1, 2], + }, + ], + }, + ], + }); + }); + + it('returns agent datasource config without disabled inputs', () => { + expect( + storedDatasourceToAgentDatasource({ + ...mockDatasource, + inputs: [{ ...mockInput, enabled: false }], + }) + ).toEqual({ + id: 'mock-datasource', + namespace: 'default', + enabled: true, + use_output: 'default', + inputs: [], + }); + }); +}); diff --git a/x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.ts b/x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.ts new file mode 100644 index 00000000000000..57627fa60fe43a --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.ts @@ -0,0 +1,51 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { Datasource, NewDatasource, FullAgentConfigDatasource } from '../types'; +import { DEFAULT_OUTPUT } from '../constants'; + +export const storedDatasourceToAgentDatasource = ( + datasource: Datasource | NewDatasource +): FullAgentConfigDatasource => { + const { name, namespace, enabled, package: pkg, inputs } = datasource; + const fullDatasource: FullAgentConfigDatasource = { + id: name, + namespace, + enabled, + use_output: DEFAULT_OUTPUT.name, // TODO: hardcoded to default output for now + inputs: inputs + .filter(input => input.enabled) + .map(input => ({ + ...input, + streams: input.streams.map(stream => { + if (stream.config) { + const fullStream = { + ...stream, + ...Object.entries(stream.config).reduce((acc, [configName, configValue]) => { + if (configValue !== undefined) { + acc[configName] = configValue; + } + return acc; + }, {} as { [key: string]: any }), + }; + delete fullStream.config; + return fullStream; + } else { + const fullStream = { ...stream }; + return fullStream; + } + }), + })), + }; + + if (pkg) { + fullDatasource.package = { + name: pkg.name, + version: pkg.version, + }; + } + + return fullDatasource; +}; diff --git a/x-pack/plugins/ingest_manager/common/services/index.ts b/x-pack/plugins/ingest_manager/common/services/index.ts index 1b3ae4706e3a74..7d1013cf1feb6f 100644 --- a/x-pack/plugins/ingest_manager/common/services/index.ts +++ b/x-pack/plugins/ingest_manager/common/services/index.ts @@ -3,4 +3,9 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ +import * as AgentStatusKueryHelper from './agent_status'; + export * from './routes'; +export { packageToConfigDatasourceInputs } from './package_to_config'; +export { storedDatasourceToAgentDatasource } from './datasource_to_agent_datasource'; +export { AgentStatusKueryHelper }; diff --git a/x-pack/plugins/ingest_manager/common/services/package_to_config.test.ts b/x-pack/plugins/ingest_manager/common/services/package_to_config.test.ts new file mode 100644 index 00000000000000..a4a2eb6001495e --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/services/package_to_config.test.ts @@ -0,0 +1,252 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { PackageInfo, InstallationStatus } from '../types'; +import { packageToConfigDatasourceInputs } from './package_to_config'; + +describe('Ingest Manager - packageToConfigDatasourceInputs', () => { + const mockPackage: PackageInfo = { + name: 'mock-package', + title: 'Mock package', + version: '0.0.0', + description: 'description', + type: 'mock', + categories: [], + requirement: { kibana: { versions: '' }, elasticsearch: { versions: '' } }, + format_version: '', + download: '', + path: '', + assets: { + kibana: { + dashboard: [], + visualization: [], + search: [], + 'index-pattern': [], + }, + }, + status: InstallationStatus.notInstalled, + }; + + it('returns empty array for packages with no datasources', () => { + expect(packageToConfigDatasourceInputs(mockPackage)).toEqual([]); + expect(packageToConfigDatasourceInputs({ ...mockPackage, datasources: [] })).toEqual([]); + }); + + it('returns empty array for packages a datasource but no inputs', () => { + expect( + packageToConfigDatasourceInputs(({ + ...mockPackage, + datasources: [{ inputs: [] }], + } as unknown) as PackageInfo) + ).toEqual([]); + }); + + it('returns inputs with no streams for packages with no streams', () => { + expect( + packageToConfigDatasourceInputs(({ + ...mockPackage, + datasources: [{ inputs: [{ type: 'foo' }] }], + } as unknown) as PackageInfo) + ).toEqual([{ type: 'foo', enabled: true, streams: [] }]); + expect( + packageToConfigDatasourceInputs(({ + ...mockPackage, + datasources: [{ inputs: [{ type: 'foo' }, { type: 'bar' }] }], + } as unknown) as PackageInfo) + ).toEqual([ + { type: 'foo', enabled: true, streams: [] }, + { type: 'bar', enabled: true, streams: [] }, + ]); + }); + + it('returns inputs with streams for packages with streams', () => { + expect( + packageToConfigDatasourceInputs(({ + ...mockPackage, + datasources: [ + { + inputs: [ + { type: 'foo', streams: [{ dataset: 'foo' }] }, + { type: 'bar', streams: [{ dataset: 'bar' }, { dataset: 'bar2' }] }, + ], + }, + ], + } as unknown) as PackageInfo) + ).toEqual([ + { + type: 'foo', + enabled: true, + streams: [{ id: 'foo-foo', enabled: true, dataset: 'foo', config: {} }], + }, + { + type: 'bar', + enabled: true, + streams: [ + { id: 'bar-bar', enabled: true, dataset: 'bar', config: {} }, + { id: 'bar-bar2', enabled: true, dataset: 'bar2', config: {} }, + ], + }, + ]); + }); + + it('returns inputs with streams configurations for packages with stream vars', () => { + expect( + packageToConfigDatasourceInputs(({ + ...mockPackage, + datasources: [ + { + inputs: [ + { + type: 'foo', + streams: [ + { dataset: 'foo', vars: [{ default: 'foo-var-value', name: 'var-name' }] }, + ], + }, + { + type: 'bar', + streams: [ + { dataset: 'bar', vars: [{ default: 'bar-var-value', name: 'var-name' }] }, + { dataset: 'bar2', vars: [{ default: 'bar2-var-value', name: 'var-name' }] }, + ], + }, + ], + }, + ], + } as unknown) as PackageInfo) + ).toEqual([ + { + type: 'foo', + enabled: true, + streams: [ + { id: 'foo-foo', enabled: true, dataset: 'foo', config: { 'var-name': 'foo-var-value' } }, + ], + }, + { + type: 'bar', + enabled: true, + streams: [ + { id: 'bar-bar', enabled: true, dataset: 'bar', config: { 'var-name': 'bar-var-value' } }, + { + id: 'bar-bar2', + enabled: true, + dataset: 'bar2', + config: { 'var-name': 'bar2-var-value' }, + }, + ], + }, + ]); + }); + + it('returns inputs with streams configurations for packages with stream and input vars', () => { + expect( + packageToConfigDatasourceInputs(({ + ...mockPackage, + datasources: [ + { + inputs: [ + { + type: 'foo', + vars: [ + { default: 'foo-input-var-value', name: 'foo-input-var-name' }, + { default: 'foo-input2-var-value', name: 'foo-input2-var-name' }, + { name: 'foo-input3-var-name' }, + ], + streams: [ + { dataset: 'foo', vars: [{ default: 'foo-var-value', name: 'var-name' }] }, + ], + }, + { + type: 'bar', + vars: [ + { default: ['value1', 'value2'], name: 'bar-input-var-name' }, + { default: 123456, name: 'bar-input2-var-name' }, + ], + streams: [ + { dataset: 'bar', vars: [{ default: 'bar-var-value', name: 'var-name' }] }, + { dataset: 'bar2', vars: [{ default: 'bar2-var-value', name: 'var-name' }] }, + ], + }, + { + type: 'with-disabled-streams', + streams: [ + { + dataset: 'disabled', + enabled: false, + vars: [{ multi: true, name: 'var-name' }], + }, + { dataset: 'disabled2', enabled: false }, + ], + }, + ], + }, + ], + } as unknown) as PackageInfo) + ).toEqual([ + { + type: 'foo', + enabled: true, + streams: [ + { + id: 'foo-foo', + enabled: true, + dataset: 'foo', + config: { + 'var-name': 'foo-var-value', + 'foo-input-var-name': 'foo-input-var-value', + 'foo-input2-var-name': 'foo-input2-var-value', + 'foo-input3-var-name': undefined, + }, + }, + ], + }, + { + type: 'bar', + enabled: true, + streams: [ + { + id: 'bar-bar', + enabled: true, + dataset: 'bar', + config: { + 'var-name': 'bar-var-value', + 'bar-input-var-name': ['value1', 'value2'], + 'bar-input2-var-name': 123456, + }, + }, + { + id: 'bar-bar2', + enabled: true, + dataset: 'bar2', + config: { + 'var-name': 'bar2-var-value', + 'bar-input-var-name': ['value1', 'value2'], + 'bar-input2-var-name': 123456, + }, + }, + ], + }, + { + type: 'with-disabled-streams', + enabled: false, + streams: [ + { + id: 'with-disabled-streams-disabled', + enabled: false, + dataset: 'disabled', + config: { + 'var-name': [], + }, + }, + { + id: 'with-disabled-streams-disabled2', + enabled: false, + dataset: 'disabled2', + config: {}, + }, + ], + }, + ]); + }); +}); diff --git a/x-pack/plugins/ingest_manager/common/services/package_to_config.ts b/x-pack/plugins/ingest_manager/common/services/package_to_config.ts new file mode 100644 index 00000000000000..311a0a0fceddd3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/services/package_to_config.ts @@ -0,0 +1,69 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { + PackageInfo, + RegistryDatasource, + RegistryVarsEntry, + Datasource, + DatasourceInput, + DatasourceInputStream, +} from '../types'; + +/* + * This service creates a datasource inputs definition from defaults provided in package info + */ +export const packageToConfigDatasourceInputs = (packageInfo: PackageInfo): Datasource['inputs'] => { + const inputs: Datasource['inputs'] = []; + + // Assume package will only ever ship one datasource for now + const packageDatasource: RegistryDatasource | null = + packageInfo.datasources && packageInfo.datasources[0] ? packageInfo.datasources[0] : null; + + // Create datasource input property + if (packageDatasource?.inputs?.length) { + // Map each package datasource input to agent config datasource input + packageDatasource.inputs.forEach(packageInput => { + // Map each package input stream into datasource input stream + const streams: DatasourceInputStream[] = packageInput.streams + ? packageInput.streams.map(packageStream => { + // Copy input vars into each stream's vars + const streamVars: RegistryVarsEntry[] = [ + ...(packageInput.vars || []), + ...(packageStream.vars || []), + ]; + const streamConfig = {}; + const streamVarsReducer = ( + configObject: DatasourceInputStream['config'], + streamVar: RegistryVarsEntry + ): DatasourceInputStream['config'] => { + if (!streamVar.default && streamVar.multi) { + configObject![streamVar.name] = []; + } else { + configObject![streamVar.name] = streamVar.default; + } + return configObject; + }; + return { + id: `${packageInput.type}-${packageStream.dataset}`, + enabled: packageStream.enabled === false ? false : true, + dataset: packageStream.dataset, + config: streamVars.reduce(streamVarsReducer, streamConfig), + }; + }) + : []; + + const input: DatasourceInput = { + type: packageInput.type, + enabled: streams.length ? !!streams.find(stream => stream.enabled) : true, + streams, + }; + + inputs.push(input); + }); + } + + return inputs; +}; diff --git a/x-pack/plugins/ingest_manager/common/services/routes.ts b/x-pack/plugins/ingest_manager/common/services/routes.ts index bcd1646fe1f0cf..7ad3944096a5f0 100644 --- a/x-pack/plugins/ingest_manager/common/services/routes.ts +++ b/x-pack/plugins/ingest_manager/common/services/routes.ts @@ -8,6 +8,10 @@ import { EPM_API_ROUTES, DATASOURCE_API_ROUTES, AGENT_CONFIG_API_ROUTES, + FLEET_SETUP_API_ROUTES, + AGENT_API_ROUTES, + ENROLLMENT_API_KEY_ROUTES, + SETUP_API_ROUTE, } from '../constants'; export const epmRouteService = { @@ -24,7 +28,7 @@ export const epmRouteService = { }, getFilePath: (filePath: string) => { - return `${EPM_API_ROOT}${filePath}`; + return `${EPM_API_ROOT}${filePath.replace('/package', '/packages')}`; }, getInstallPath: (pkgkey: string) => { @@ -75,3 +79,29 @@ export const agentConfigRouteService = { return AGENT_CONFIG_API_ROUTES.DELETE_PATTERN; }, }; + +export const fleetSetupRouteService = { + getFleetSetupPath: () => FLEET_SETUP_API_ROUTES.INFO_PATTERN, + postFleetSetupPath: () => FLEET_SETUP_API_ROUTES.CREATE_PATTERN, +}; + +export const agentRouteService = { + getInfoPath: (agentId: string) => AGENT_API_ROUTES.INFO_PATTERN.replace('{agentId}', agentId), + getUpdatePath: (agentId: string) => AGENT_API_ROUTES.UPDATE_PATTERN.replace('{agentId}', agentId), + getEventsPath: (agentId: string) => AGENT_API_ROUTES.EVENTS_PATTERN.replace('{agentId}', agentId), + getUnenrollPath: () => AGENT_API_ROUTES.UNENROLL_PATTERN, + getListPath: () => AGENT_API_ROUTES.LIST_PATTERN, + getStatusPath: () => AGENT_API_ROUTES.STATUS_PATTERN, +}; + +export const enrollmentAPIKeyRouteService = { + getListPath: () => ENROLLMENT_API_KEY_ROUTES.LIST_PATTERN, + getCreatePath: () => ENROLLMENT_API_KEY_ROUTES.CREATE_PATTERN, + getInfoPath: (keyId: string) => ENROLLMENT_API_KEY_ROUTES.INFO_PATTERN.replace('{keyId}', keyId), + getDeletePath: (keyId: string) => + ENROLLMENT_API_KEY_ROUTES.DELETE_PATTERN.replace('{keyId}', keyId), +}; + +export const setupRouteService = { + getSetupPath: () => SETUP_API_ROUTE, +}; diff --git a/x-pack/plugins/ingest_manager/common/types/index.ts b/x-pack/plugins/ingest_manager/common/types/index.ts index 4abb1b659f0367..42f7a9333118e5 100644 --- a/x-pack/plugins/ingest_manager/common/types/index.ts +++ b/x-pack/plugins/ingest_manager/common/types/index.ts @@ -3,7 +3,6 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ - export * from './models'; export * from './rest_spec'; @@ -16,5 +15,20 @@ export interface IngestManagerConfigType { fleet: { enabled: boolean; defaultOutputHost: string; + kibana: { + host?: string; + ca_sha256?: string; + }; + elasticsearch: { + host?: string; + ca_sha256?: string; + }; }; } + +// Calling Object.entries(PackagesGroupedByStatus) gave `status: string` +// which causes a "string is not assignable to type InstallationStatus` error +// see https://github.com/Microsoft/TypeScript/issues/20322 +// and https://github.com/Microsoft/TypeScript/pull/12253#issuecomment-263132208 +// and https://github.com/Microsoft/TypeScript/issues/21826#issuecomment-479851685 +export const entries = Object.entries as (o: T) => Array<[keyof T, T[keyof T]]>; diff --git a/x-pack/plugins/ingest_manager/common/types/models/agent.ts b/x-pack/plugins/ingest_manager/common/types/models/agent.ts new file mode 100644 index 00000000000000..a0575c71d3aba1 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/models/agent.ts @@ -0,0 +1,77 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectAttributes } from '../../../../../../src/core/public'; +import { AGENT_TYPE_EPHEMERAL, AGENT_TYPE_PERMANENT, AGENT_TYPE_TEMPORARY } from '../../constants'; + +export type AgentType = + | typeof AGENT_TYPE_EPHEMERAL + | typeof AGENT_TYPE_PERMANENT + | typeof AGENT_TYPE_TEMPORARY; + +export type AgentStatus = 'offline' | 'error' | 'online' | 'inactive' | 'warning'; + +export interface AgentAction extends SavedObjectAttributes { + type: 'CONFIG_CHANGE' | 'DATA_DUMP' | 'RESUME' | 'PAUSE'; + id: string; + created_at: string; + data?: string; + sent_at?: string; +} + +export interface AgentEvent { + type: 'STATE' | 'ERROR' | 'ACTION_RESULT' | 'ACTION'; + subtype: // State + | 'RUNNING' + | 'STARTING' + | 'IN_PROGRESS' + | 'CONFIG' + | 'FAILED' + | 'STOPPING' + | 'STOPPED' + // Action results + | 'DATA_DUMP' + // Actions + | 'ACKNOWLEDGED' + | 'UNKNOWN'; + timestamp: string; + message: string; + payload?: any; + agent_id: string; + action_id?: string; + config_id?: string; + stream_id?: string; +} + +export interface AgentEventSOAttributes extends AgentEvent, SavedObjectAttributes {} + +interface AgentBase { + type: AgentType; + active: boolean; + enrolled_at: string; + shared_id?: string; + access_api_key_id?: string; + default_api_key?: string; + config_id?: string; + last_checkin?: string; + config_updated_at?: string; + actions: AgentAction[]; +} + +export interface Agent extends AgentBase { + id: string; + current_error_events: AgentEvent[]; + user_provided_metadata: Record; + local_metadata: Record; + access_api_key?: string; + status?: string; +} + +export interface AgentSOAttributes extends AgentBase, SavedObjectAttributes { + user_provided_metadata: string; + local_metadata: string; + current_error_events?: string; +} diff --git a/x-pack/plugins/ingest_manager/common/types/models/agent_config.ts b/x-pack/plugins/ingest_manager/common/types/models/agent_config.ts index 1cc8b32afe3c12..c63e496273adac 100644 --- a/x-pack/plugins/ingest_manager/common/types/models/agent_config.ts +++ b/x-pack/plugins/ingest_manager/common/types/models/agent_config.ts @@ -4,29 +4,58 @@ * you may not use this file except in compliance with the Elastic License. */ -import { DatasourceSchema } from './datasource'; +import { SavedObjectAttributes } from '../../../../../../src/core/public'; +import { + Datasource, + DatasourcePackage, + DatasourceInput, + DatasourceInputStream, +} from './datasource'; +import { Output } from './output'; export enum AgentConfigStatus { Active = 'active', Inactive = 'inactive', } -interface AgentConfigBaseSchema { +export interface NewAgentConfig { name: string; - namespace: string; + namespace?: string; description?: string; + is_default?: boolean; } -export type NewAgentConfigSchema = AgentConfigBaseSchema; - -export type AgentConfigSchema = AgentConfigBaseSchema & { +export interface AgentConfig extends NewAgentConfig, SavedObjectAttributes { id: string; status: AgentConfigStatus; - datasources: Array; + datasources: string[] | Datasource[]; updated_on: string; updated_by: string; -}; + revision: number; +} -export type NewAgentConfig = NewAgentConfigSchema; +export type FullAgentConfigDatasource = Pick & { + id: string; + package?: Pick; + use_output: string; + inputs: Array< + Omit & { + streams: Array< + Omit & { + [key: string]: any; + } + >; + } + >; +}; -export type AgentConfig = AgentConfigSchema; +export interface FullAgentConfig { + id: string; + outputs: { + [key: string]: Pick & { + [key: string]: any; + }; + }; + datasources: FullAgentConfigDatasource[]; + revision?: number; +} diff --git a/x-pack/plugins/ingest_manager/common/types/models/datasource.ts b/x-pack/plugins/ingest_manager/common/types/models/datasource.ts index f28037845c7f7f..3503bbdcd40e31 100644 --- a/x-pack/plugins/ingest_manager/common/types/models/datasource.ts +++ b/x-pack/plugins/ingest_manager/common/types/models/datasource.ts @@ -4,40 +4,39 @@ * you may not use this file except in compliance with the Elastic License. */ -interface DatasourceBaseSchema { +export interface DatasourcePackage { name: string; - namespace: string; - read_alias: string; - agent_config_id: string; - package: { - assets: Array<{ - id: string; - type: string; - }>; - description: string; - name: string; - title: string; - version: string; - }; - streams: Array<{ - config: Record; - input: { - type: string; - config: Record; - fields: Array>; - ilm_policy: string; - index_template: string; - ingest_pipelines: string[]; - }; - output_id: string; - processors: string[]; - }>; + title: string; + version: string; } -export type NewDatasourceSchema = DatasourceBaseSchema; +export interface DatasourceInputStream { + id: string; + enabled: boolean; + dataset: string; + processors?: string[]; + config?: Record; +} -export type DatasourceSchema = DatasourceBaseSchema & { id: string }; +export interface DatasourceInput { + type: string; + enabled: boolean; + processors?: string[]; + streams: DatasourceInputStream[]; +} -export type NewDatasource = NewDatasourceSchema; +export interface NewDatasource { + name: string; + description?: string; + namespace?: string; + config_id: string; + enabled: boolean; + package?: DatasourcePackage; + output_id: string; + inputs: DatasourceInput[]; +} -export type Datasource = DatasourceSchema; +export type Datasource = NewDatasource & { + id: string; + revision: number; +}; diff --git a/x-pack/plugins/ingest_manager/common/types/models/enrollment_api_key.ts b/x-pack/plugins/ingest_manager/common/types/models/enrollment_api_key.ts new file mode 100644 index 00000000000000..35cb851a729333 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/models/enrollment_api_key.ts @@ -0,0 +1,23 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { SavedObjectAttributes } from '../../../../../../src/core/public'; + +export interface EnrollmentAPIKey { + id: string; + api_key_id: string; + api_key: string; + name?: string; + active: boolean; + config_id?: string; +} + +export interface EnrollmentAPIKeySOAttributes extends SavedObjectAttributes { + api_key_id: string; + api_key: string; + name?: string; + active: boolean; + config_id?: string; +} diff --git a/x-pack/plugins/ingest_manager/common/types/models/epm.ts b/x-pack/plugins/ingest_manager/common/types/models/epm.ts new file mode 100644 index 00000000000000..a1a39444c3b50c --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/models/epm.ts @@ -0,0 +1,265 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +// Follow pattern from https://github.com/elastic/kibana/pull/52447 +// TODO: Update when https://github.com/elastic/kibana/issues/53021 is closed +import { + SavedObject, + SavedObjectAttributes, + SavedObjectReference, +} from '../../../../../../src/core/public'; + +export enum InstallationStatus { + installed = 'installed', + notInstalled = 'not_installed', +} +export enum InstallStatus { + installed = 'installed', + notInstalled = 'not_installed', + installing = 'installing', + uninstalling = 'uninstalling', +} + +export type DetailViewPanelName = 'overview' | 'data-sources'; +export type ServiceName = 'kibana' | 'elasticsearch'; +export type AssetType = KibanaAssetType | ElasticsearchAssetType | AgentAssetType; + +export enum KibanaAssetType { + dashboard = 'dashboard', + visualization = 'visualization', + search = 'search', + indexPattern = 'index-pattern', +} + +export enum ElasticsearchAssetType { + ingestPipeline = 'ingest-pipeline', + indexTemplate = 'index-template', + ilmPolicy = 'ilm-policy', +} + +export enum AgentAssetType { + input = 'input', +} + +// from /package/{name} +// type Package struct at https://github.com/elastic/package-registry/blob/master/util/package.go +// https://github.com/elastic/package-registry/blob/master/docs/api/package.json +export interface RegistryPackage { + name: string; + title?: string; + version: string; + readme?: string; + description: string; + type: string; + categories: string[]; + requirement: RequirementsByServiceName; + screenshots?: RegistryImage[]; + icons?: RegistryImage[]; + assets?: string[]; + internal?: boolean; + format_version: string; + datasets?: Dataset[]; + datasources?: RegistryDatasource[]; + download: string; + path: string; +} + +interface RegistryImage { + // https://github.com/elastic/package-registry/blob/master/util/package.go#L74 + // says src is potentially missing but I couldn't find any examples + // it seems like src should be required. How can you have an image with no reference to the content? + src: string; + title?: string; + size?: string; + type?: string; +} +export interface RegistryDatasource { + name: string; + title: string; + description: string; + inputs: RegistryInput[]; +} + +export interface RegistryInput { + type: string; + title: string; + description?: string; + vars?: RegistryVarsEntry[]; + streams: RegistryStream[]; +} + +export interface RegistryStream { + input: string; + dataset: string; + title: string; + description?: string; + enabled?: boolean; + vars?: RegistryVarsEntry[]; +} + +export type RequirementVersion = string; +export type RequirementVersionRange = string; +export interface ServiceRequirements { + versions: RequirementVersionRange; +} + +// Registry's response types +// from /search +// https://github.com/elastic/package-registry/blob/master/docs/api/search.json +export type RegistrySearchResults = RegistrySearchResult[]; +// from getPackageOutput at https://github.com/elastic/package-registry/blob/master/search.go +export type RegistrySearchResult = Pick< + RegistryPackage, + | 'name' + | 'title' + | 'version' + | 'description' + | 'type' + | 'icons' + | 'internal' + | 'download' + | 'path' + | 'datasets' + | 'datasources' +>; + +export type ScreenshotItem = RegistryImage; + +// from /categories +// https://github.com/elastic/package-registry/blob/master/docs/api/categories.json +export type CategorySummaryList = CategorySummaryItem[]; +export type CategoryId = string; +export interface CategorySummaryItem { + id: CategoryId; + title: string; + count: number; +} + +export type RequirementsByServiceName = Record; +export interface AssetParts { + pkgkey: string; + dataset?: string; + service: ServiceName; + type: AssetType; + file: string; +} +export type AssetTypeToParts = KibanaAssetTypeToParts & ElasticsearchAssetTypeToParts; +export type AssetsGroupedByServiceByType = Record< + Extract, + KibanaAssetTypeToParts +>; +// & Record, ElasticsearchAssetTypeToParts>; + +export type KibanaAssetParts = AssetParts & { + service: Extract; + type: KibanaAssetType; +}; + +export type ElasticsearchAssetParts = AssetParts & { + service: Extract; + type: ElasticsearchAssetType; +}; + +export type KibanaAssetTypeToParts = Record; +export type ElasticsearchAssetTypeToParts = Record< + ElasticsearchAssetType, + ElasticsearchAssetParts[] +>; + +export interface Dataset { + title: string; + path: string; + id: string; + release: string; + ingest_pipeline: string; + vars?: RegistryVarsEntry[]; + type: string; + streams?: RegistryStream[]; + package: string; +} + +// EPR types this as `[]map[string]interface{}` +// which means the official/possible type is Record +// but we effectively only see this shape +export interface RegistryVarsEntry { + name: string; + title?: string; + description?: string; + type: string; + required?: boolean; + multi?: boolean; + default?: string | string[]; + os?: { + [key: string]: { + default: string | string[]; + }; + }; +} + +// some properties are optional in Registry responses but required in EPM +// internal until we need them +interface PackageAdditions { + title: string; + assets: AssetsGroupedByServiceByType; +} + +// Managers public HTTP response types +export type PackageList = PackageListItem[]; + +export type PackageListItem = Installable; +export type PackagesGroupedByStatus = Record; +export type PackageInfo = Installable< + // remove the properties we'll be altering/replacing from the base type + Omit & + // now add our replacement definitions + PackageAdditions +>; + +export interface Installation extends SavedObjectAttributes { + installed: AssetReference[]; + name: string; + version: string; +} + +export type Installable = Installed | NotInstalled; + +export type Installed = T & { + status: InstallationStatus.installed; + savedObject: SavedObject; +}; + +export type NotInstalled = T & { + status: InstallationStatus.notInstalled; +}; + +export type AssetReference = Pick & { + type: AssetType | IngestAssetType; +}; + +/** + * Types of assets which can be installed/removed + */ +export enum IngestAssetType { + DataFrameTransform = 'data-frame-transform', + IlmPolicy = 'ilm-policy', + IndexTemplate = 'index-template', + IngestPipeline = 'ingest-pipeline', + MlJob = 'ml-job', + RollupJob = 'rollup-job', +} + +export enum DefaultPackages { + base = 'base', + system = 'system', +} + +export interface IndexTemplate { + order: number; + index_patterns: string[]; + settings: any; + mappings: object; + aliases: object; +} diff --git a/x-pack/plugins/ingest_manager/common/types/models/index.ts b/x-pack/plugins/ingest_manager/common/types/models/index.ts index 959dfe1d937b96..579b510e52daae 100644 --- a/x-pack/plugins/ingest_manager/common/types/models/index.ts +++ b/x-pack/plugins/ingest_manager/common/types/models/index.ts @@ -3,6 +3,10 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ + +export * from './agent'; export * from './agent_config'; export * from './datasource'; export * from './output'; +export * from './epm'; +export * from './enrollment_api_key'; diff --git a/x-pack/plugins/ingest_manager/common/types/models/output.ts b/x-pack/plugins/ingest_manager/common/types/models/output.ts index 5f96fe33b5e167..cedf5e81f3cb66 100644 --- a/x-pack/plugins/ingest_manager/common/types/models/output.ts +++ b/x-pack/plugins/ingest_manager/common/types/models/output.ts @@ -8,26 +8,18 @@ export enum OutputType { Elasticsearch = 'elasticsearch', } -interface OutputBaseSchema { +export interface NewOutput { + is_default: boolean; name: string; type: OutputType; - username?: string; - password?: string; - index_name?: string; - ingest_pipeline?: string; hosts?: string[]; + ca_sha256?: string; api_key?: string; admin_username?: string; admin_password?: string; config?: Record; } -export type NewOutputSchema = OutputBaseSchema; - -export type OutputSchema = OutputBaseSchema & { +export type Output = NewOutput & { id: string; }; - -export type NewOutput = NewOutputSchema; - -export type Output = OutputSchema; diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/agent.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/agent.ts new file mode 100644 index 00000000000000..af919d973b7d9c --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/agent.ts @@ -0,0 +1,142 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Agent, AgentAction, AgentEvent, AgentStatus, AgentType } from '../models'; + +export interface GetAgentsRequest { + query: { + page: number; + perPage: number; + kuery?: string; + showInactive: boolean; + }; +} + +export interface GetAgentsResponse { + list: Agent[]; + total: number; + page: number; + perPage: number; + success: boolean; +} + +export interface GetOneAgentRequest { + params: { + agentId: string; + }; +} + +export interface GetOneAgentResponse { + item: Agent; + success: boolean; +} + +export interface PostAgentCheckinRequest { + params: { + agentId: string; + }; + body: { + local_metadata?: Record; + events?: AgentEvent[]; + }; +} + +export interface PostAgentCheckinResponse { + action: string; + success: boolean; + actions: AgentAction[]; +} + +export interface PostAgentEnrollRequest { + body: { + type: AgentType; + shared_id?: string; + metadata: { + local: Record; + user_provided: Record; + }; + }; +} + +export interface PostAgentEnrollResponse { + action: string; + success: boolean; + item: Agent & { status: AgentStatus }; +} + +export interface PostAgentAcksRequest { + body: { + action_ids: string[]; + }; + params: { + agentId: string; + }; +} + +export interface PostAgentUnenrollRequest { + body: { kuery: string } | { ids: string[] }; +} + +export interface PostAgentUnenrollResponse { + results: Array<{ + success: boolean; + error?: any; + id: string; + action: string; + }>; + success: boolean; +} + +export interface GetOneAgentEventsRequest { + params: { + agentId: string; + }; + query: { + page: number; + perPage: number; + kuery?: string; + }; +} + +export interface GetOneAgentEventsResponse { + list: AgentEvent[]; + total: number; + page: number; + perPage: number; + success: boolean; +} + +export interface DeleteAgentRequest { + params: { + agentId: string; + }; +} + +export interface UpdateAgentRequest { + params: { + agentId: string; + }; + body: { + user_provided_metadata: Record; + }; +} + +export interface GetAgentStatusRequest { + query: { + configId: string; + }; +} + +export interface GetAgentStatusResponse { + success: boolean; + results: { + events: number; + total: number; + online: number; + error: number; + offline: number; + }; +} diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/agent_config.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/agent_config.ts index 5d281b03260db2..89d548d11dadb6 100644 --- a/x-pack/plugins/ingest_manager/common/types/rest_spec/agent_config.ts +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/agent_config.ts @@ -3,22 +3,24 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { AgentConfig, NewAgentConfigSchema } from '../models'; -import { ListWithKuerySchema } from './common'; +import { AgentConfig, NewAgentConfig, FullAgentConfig } from '../models'; +import { ListWithKuery } from './common'; -export interface GetAgentConfigsRequestSchema { - query: ListWithKuerySchema; +export interface GetAgentConfigsRequest { + query: ListWithKuery; } +export type GetAgentConfigsResponseItem = AgentConfig & { agents?: number }; + export interface GetAgentConfigsResponse { - items: AgentConfig[]; + items: GetAgentConfigsResponseItem[]; total: number; page: number; perPage: number; success: boolean; } -export interface GetOneAgentConfigRequestSchema { +export interface GetOneAgentConfigRequest { params: { agentConfigId: string; }; @@ -29,8 +31,8 @@ export interface GetOneAgentConfigResponse { success: boolean; } -export interface CreateAgentConfigRequestSchema { - body: NewAgentConfigSchema; +export interface CreateAgentConfigRequest { + body: NewAgentConfig; } export interface CreateAgentConfigResponse { @@ -38,8 +40,8 @@ export interface CreateAgentConfigResponse { success: boolean; } -export type UpdateAgentConfigRequestSchema = GetOneAgentConfigRequestSchema & { - body: NewAgentConfigSchema; +export type UpdateAgentConfigRequest = GetOneAgentConfigRequest & { + body: NewAgentConfig; }; export interface UpdateAgentConfigResponse { @@ -47,7 +49,7 @@ export interface UpdateAgentConfigResponse { success: boolean; } -export interface DeleteAgentConfigsRequestSchema { +export interface DeleteAgentConfigsRequest { body: { agentConfigIds: string[]; }; @@ -57,3 +59,14 @@ export type DeleteAgentConfigsResponse = Array<{ id: string; success: boolean; }>; + +export interface GetFullAgentConfigRequest { + params: { + agentConfigId: string; + }; +} + +export interface GetFullAgentConfigResponse { + item: FullAgentConfig; + success: boolean; +} diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/common.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/common.ts index d247933d4011f8..c52471ccfb4f58 100644 --- a/x-pack/plugins/ingest_manager/common/types/rest_spec/common.ts +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/common.ts @@ -4,10 +4,8 @@ * you may not use this file except in compliance with the Elastic License. */ -export interface ListWithKuerySchema { +export interface ListWithKuery { page: number; perPage: number; kuery?: string; } - -export type ListWithKuery = ListWithKuerySchema; diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/datasource.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/datasource.ts index 78859f20080053..f630602503f0af 100644 --- a/x-pack/plugins/ingest_manager/common/types/rest_spec/datasource.ts +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/datasource.ts @@ -3,28 +3,33 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { NewDatasourceSchema } from '../models'; -import { ListWithKuerySchema } from './common'; +import { Datasource, NewDatasource } from '../models'; +import { ListWithKuery } from './common'; -export interface GetDatasourcesRequestSchema { - query: ListWithKuerySchema; +export interface GetDatasourcesRequest { + query: ListWithKuery; } -export interface GetOneDatasourceRequestSchema { +export interface GetOneDatasourceRequest { params: { datasourceId: string; }; } -export interface CreateDatasourceRequestSchema { - body: NewDatasourceSchema; +export interface CreateDatasourceRequest { + body: NewDatasource; } -export type UpdateDatasourceRequestSchema = GetOneDatasourceRequestSchema & { - body: NewDatasourceSchema; +export interface CreateDatasourceResponse { + item: Datasource; + success: boolean; +} + +export type UpdateDatasourceRequest = GetOneDatasourceRequest & { + body: NewDatasource; }; -export interface DeleteDatasourcesRequestSchema { +export interface DeleteDatasourcesRequest { body: { datasourceIds: string[]; }; diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/enrollment_api_key.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/enrollment_api_key.ts new file mode 100644 index 00000000000000..851e6571c0dd2a --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/enrollment_api_key.ts @@ -0,0 +1,59 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EnrollmentAPIKey } from '../models'; + +export interface GetEnrollmentAPIKeysRequest { + query: { + page: number; + perPage: number; + kuery?: string; + }; +} + +export interface GetEnrollmentAPIKeysResponse { + list: EnrollmentAPIKey[]; + total: number; + page: number; + perPage: number; + success: boolean; +} + +export interface GetOneEnrollmentAPIKeyRequest { + params: { + keyId: string; + }; +} + +export interface GetOneEnrollmentAPIKeyResponse { + item: EnrollmentAPIKey; + success: boolean; +} + +export interface DeleteEnrollmentAPIKeyRequest { + params: { + keyId: string; + }; +} + +export interface DeleteEnrollmentAPIKeyResponse { + action: string; + success: boolean; +} + +export interface PostEnrollmentAPIKeyRequest { + body: { + name?: string; + config_id: string; + expiration?: string; + }; +} + +export interface PostEnrollmentAPIKeyResponse { + action: string; + item: EnrollmentAPIKey; + success: boolean; +} diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/epm.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/epm.ts new file mode 100644 index 00000000000000..5ac7fe9e2779b5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/epm.ts @@ -0,0 +1,75 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + AssetReference, + CategorySummaryList, + Installable, + RegistryPackage, + PackageInfo, +} from '../models/epm'; + +export interface GetCategoriesResponse { + response: CategorySummaryList; + success: boolean; +} +export interface GetPackagesRequest { + query: { + category?: string; + }; +} + +export interface GetPackagesResponse { + response: Array< + Installable< + Pick< + RegistryPackage, + 'name' | 'title' | 'version' | 'description' | 'type' | 'icons' | 'download' | 'path' + > + > + >; + success: boolean; +} + +export interface GetFileRequest { + params: { + pkgkey: string; + filePath: string; + }; +} + +export interface GetInfoRequest { + params: { + pkgkey: string; + }; +} + +export interface GetInfoResponse { + response: PackageInfo; + success: boolean; +} + +export interface InstallPackageRequest { + params: { + pkgkey: string; + }; +} + +export interface InstallPackageResponse { + response: AssetReference[]; + success: boolean; +} + +export interface DeletePackageRequest { + params: { + pkgkey: string; + }; +} + +export interface DeletePackageResponse { + response: AssetReference[]; + success: boolean; +} diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/fleet_setup.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/fleet_setup.ts index 926021baab0ef7..2eda4f187dafa1 100644 --- a/x-pack/plugins/ingest_manager/common/types/rest_spec/fleet_setup.ts +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/fleet_setup.ts @@ -5,9 +5,9 @@ */ // eslint-disable-next-line @typescript-eslint/no-empty-interface -export interface GetFleetSetupRequestSchema {} +export interface GetFleetSetupRequest {} -export interface CreateFleetSetupRequestSchema { +export interface CreateFleetSetupRequest { body: { admin_username: string; admin_password: string; diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/index.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/index.ts index 7d0d7e67f2db01..abe1bc8e3eddba 100644 --- a/x-pack/plugins/ingest_manager/common/types/rest_spec/index.ts +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/index.ts @@ -5,5 +5,9 @@ */ export * from './common'; export * from './datasource'; +export * from './agent'; export * from './agent_config'; export * from './fleet_setup'; +export * from './epm'; +export * from './enrollment_api_key'; +export * from './install_script'; diff --git a/x-pack/plugins/ingest_manager/common/types/rest_spec/install_script.ts b/x-pack/plugins/ingest_manager/common/types/rest_spec/install_script.ts new file mode 100644 index 00000000000000..0b0b0945d652e3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/common/types/rest_spec/install_script.ts @@ -0,0 +1,11 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export interface InstallScriptRequest { + params: { + osType: 'macos'; + }; +} diff --git a/x-pack/plugins/ingest_manager/dev_docs/actions_and_events.md b/x-pack/plugins/ingest_manager/dev_docs/actions_and_events.md new file mode 100644 index 00000000000000..b41cdc221c51a3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/actions_and_events.md @@ -0,0 +1,47 @@ +## Agent Fleet: actions protocol + +Agent is using `actions` and `events` to comunicate with fleet during checkin. + +## Actions + +Action are returned to the agent during the checkin [see](./api/agents_checkin) +Agent should aknowledge actions they received using `POST /agents/{agentId}/acks` API. + +### POLICY_CHANGE + +This action is send when a new policy is available, the policy is available under the `data` field. + +```js +{ + "type": "POLICY_CHANGE", + "id": "action_id_1", + "data": { + "policy": { + "id": "config_id", + "outputs": { + "default": { + "api_key": "slfhsdlfhjjkshfkjh:sdfsdfsdfsdf", + "id": "default", + "name": "Default", + "type": "elasticsearch", + "hosts": ["https://localhost:9200"], + } + }, + "streams": [ + { + "metricsets": [ + "container", + "cpu" + ], + "id": "string", + "type": "etc", + "output": { + "use_output": "default" + } + } + ] + } + } + }] +} +``` diff --git a/x-pack/plugins/ingest_manager/dev_docs/api/agents_acks.md b/x-pack/plugins/ingest_manager/dev_docs/api/agents_acks.md new file mode 100644 index 00000000000000..d5bf4d2c2e2369 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/api/agents_acks.md @@ -0,0 +1,37 @@ +# Fleet agent acks API + +Agent acks +Acknowledge actions received during checkin + +## Request + +`POST /api/ingest_manager/fleet/agents/{agentId}/acks` + +## Headers + +- `Authorization` (Required, string) A valid fleet access api key.. + +## Request body + +- `action_ids` (Required, array) An array of action id that the agent received. + +## Response code + +- `200` Indicates a successful call. + +## Example + +```js +POST /api/ingest_manager/fleet/agents/a4937110-e53e-11e9-934f-47a8e38a522c/acks +Authorization: ApiKey VALID_ACCESS_API_KEY +{ + "action_ids": ["action-1", "action-2"] +} +``` + +```js +{ + "action": "acks", + "success": true, +} +``` diff --git a/x-pack/plugins/ingest_manager/dev_docs/api/agents_checkin.md b/x-pack/plugins/ingest_manager/dev_docs/api/agents_checkin.md new file mode 100644 index 00000000000000..aa3e4b1335ecd9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/api/agents_checkin.md @@ -0,0 +1,47 @@ +# Fleet agent checkin API + +Agent checkin +Report current state of a Fleet agent. + +## Request + +`POST /api/ingest_manager/fleet/agents/{agentId}/checkin` + +## Headers + +- `Authorization` (Required, string) A valid fleet access api key.. + +## Request body + +- `events` (Required, array) An array of events with the properties `type`, `subtype`, `message`, `timestamp`, `payload`, and `agent_id`. + +- `local_metadata` (Optional, object) An object that contains the local metadata for an agent. The metadata is a dictionary of strings (example: `{ "os": "macos" }`). + +## Response code + +- `200` Indicates a successful call. + +## Example + +```js +POST /api/ingest_manager/fleet/agents/a4937110-e53e-11e9-934f-47a8e38a522c/checkin +Authorization: ApiKey VALID_ACCESS_API_KEY +{ + "events": [{ + "type": "STATE", + "subtype": "STARTING", + "message": "state changed from STOPPED to STARTING", + "timestamp": "2019-10-01T13:42:54.323Z", + "payload": {}, + "agent_id": "a4937110-e53e-11e9-934f-47a8e38a522c" + }] +} +``` + +```js +{ + "action": "checkin", + "success": true, + "actions": [] +} +``` diff --git a/x-pack/plugins/ingest_manager/dev_docs/api/agents_enroll.md b/x-pack/plugins/ingest_manager/dev_docs/api/agents_enroll.md new file mode 100644 index 00000000000000..304ce733b7dcd2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/api/agents_enroll.md @@ -0,0 +1,79 @@ +# Enroll Fleet agent API + +Enroll agent + +## Request + +`POST /api/ingest_manager/fleet/agents/enroll` + +## Headers + +- `Authorization` (Required, string) a valid enrollemnt api key. + +## Request body + +- `type` (Required, string) Agent type should be one of `EPHEMERAL`, `TEMPORARY`, `PERMANENT` +- `shared_id` (Optional, string) An ID for the agent. +- `metadata` (Optional, object) Objects with `local` and `user_provided` properties that contain the metadata for an agent. The metadata is a dictionary of strings (example: `"local": { "os": "macos" }`). + +## Response code + +`200` Indicates a successful call. +`400` For an invalid request. +`401` For an invalid api key. + +## Example + +```js +POST /api/ingest_manager/fleet/agents/enroll +Authorization: ApiKey VALID_API_KEY +{ + "type": "PERMANENT", + "metadata": { + "local": { "os": "macos"}, + "userProvided": { "region": "us-east"} + } +} +``` + +The API returns the following: + +```js +{ + "action": "created", + "success": true, + "item": { + "id": "a4937110-e53e-11e9-934f-47a8e38a522c", + "active": true, + "config_id": "default", + "type": "PERMANENT", + "enrolled_at": "2019-10-02T18:01:22.337Z", + "user_provided_metadata": {}, + "local_metadata": {}, + "actions": [], + "access_api_key": "ACCESS_API_KEY" + } +} +``` + +## Expected errors + +The API will return a response with a `401` status code and an error if the enrollment apiKey is invalid like this: + +```js +{ + "statusCode": 401, + "error": "Unauthorized", + "message": "Enrollment apiKey is not valid: Enrollement api key does not exists or is not active" +} +``` + +The API will return a response with a `400` status code and an error if you enroll an agent with the same `shared_id` than an already active agent: + +```js +{ + "statusCode": 400, + "error": "BadRequest", + "message": "Impossible to enroll an already active agent" +} +``` diff --git a/x-pack/plugins/ingest_manager/dev_docs/api/agents_list.md b/x-pack/plugins/ingest_manager/dev_docs/api/agents_list.md new file mode 100644 index 00000000000000..38f80a8bdc0222 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/api/agents_list.md @@ -0,0 +1,22 @@ +# Fleet agent listing API + +## Request + +`GET /api/ingest_manager/fleet/agents` + +## Query + +- `showInactive` (Optional, boolean) Show inactive agents (default to false) +- `kuery` (Optional, string) Filter using kibana query language +- `page` (Optional, number) +- `perPage` (Optional, number) + +## Response code + +- `200` Indicates a successful call. + +## Example + +```js +GET /api/ingest_manager/fleet/agents?kuery=agents.last_checkin:2019-10-01T13:42:54.323Z +``` diff --git a/x-pack/plugins/ingest_manager/dev_docs/api/agents_unenroll.md b/x-pack/plugins/ingest_manager/dev_docs/api/agents_unenroll.md new file mode 100644 index 00000000000000..13b0aadd7689d6 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/api/agents_unenroll.md @@ -0,0 +1,40 @@ +# Enroll Fleet agent API + +Unenroll an agent + +## Request + +`POST /api/ingest_manager/fleet/agents/unenroll` + +## Request body + +- `ids` (Optional, string) An list of agent id to unenroll. +- `kuery` (Optional, string) a kibana query to search for agent to unenroll. + +> Note: one and only of this keys should be present: + +## Response code + +`200` Indicates a successful call. + +## Example + +```js +POST /api/ingest_manager/fleet/agents/enroll +{ + "ids": ['agent1'], +} +``` + +The API returns the following: + +```js +{ + "results": [{ + "success":true, + "id":"agent1", + "action":"unenrolled" + }], + "success":true +} +``` diff --git a/x-pack/plugins/ingest_manager/dev_docs/api_keys.md b/x-pack/plugins/ingest_manager/dev_docs/api_keys.md new file mode 100644 index 00000000000000..95d7ba19635315 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/api_keys.md @@ -0,0 +1,15 @@ +# Fleet tokens + +Fleet uses 3 types of API Keys: + +1. Enrollment API Keys - A long lived token with optional rules around assignment of policy when enrolling. It is used to enroll N agents. + +2. Access API Keys - Generated during enrollment and hidden from the user. This token is used to communicate with Kibana and is unique to each agent. This allows a single agent to be revoked without affecting other agents or their data ingestion ability. + +3. Output API Keys - This is used by the agent to ship data to ES. At the moment this is one token per unique output cluster per policy due to the scale needed from ES tokens not currently being supported. Once ES can accept the levels of scale needed, we would like to move to one token per agent. + +### FAQ + +- Can't we work on solving some of these issues and thus make this even easier? + +Yes, and we plan to. This is the first phase of how this will work, and we plan to reduce complexity over time. Because we have automated most of the complexity, all the user will notice is shorter and shorter tokens. diff --git a/x-pack/plugins/ingest_manager/dev_docs/fleet_agent_communication.md b/x-pack/plugins/ingest_manager/dev_docs/fleet_agent_communication.md new file mode 100644 index 00000000000000..8430983dc4e1dc --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/fleet_agent_communication.md @@ -0,0 +1,32 @@ +# Fleet <> Agent communication protocal + +1. Makes request to the [`agent/enroll` endpoint](/docs/api/fleet.asciidoc) using the [enrollment API key](api_keys.md) as a barrier token, the policy ID being enrolled to, and the type of the agent. + +2. Fleet verifies the Enrollment API key is valid. And returns back a unique [access API key](api_keys.md). + +This Auth API key is created to work only for the assigned policy. +The auth API key is assigned to the combination of agent and policy, and the policy can be swapped out dynamically without creating a new auth API key. + +3. The agent now "checks in" with Fleet. + +The agent uses the access token to post its current event queue to [`agent/checkin`](/docs/api/fleet.asciidoc). The endpoint will return the agent's assigned policy and an array of actions for the agent or its software to run. +The agent continues posting events and receiving updated policy changes every 30 sec or via polling settings in the policy. + +4. The agent takes the returned policy and array of actions and first reloads any policy changes. It then runs any/all actions starting at index 0. + +### If an agent / host is compromised + +1. The user via the UI or API invalidates an agent's auth API key in Fleet by "unenrolling" an agent. + +2. At the time of the agent's next checkin, auth will fail resulting in a 403 error. + +3. The agent will stop polling and delete the locally cached policy. + +4. It is **/strongly/** recommended that if an agent is compromised, the outputs used on the given agent delete their ES access tokens, and regenerate them. + +To re-enable the agent, it must be re-enrolled. Permanent and temporary agents maintain state in Fleet. If one is re-enrolled a new auth token is generated and the agent is able to resume as it was. If this is not desired, the agent will be listed in a disabled state (`active: false`) and from the details screen it can be deleted. + +### If an enrollment token is compromised + +Fleet only supports a single active enrollment token at a time. If one becomes compromised, it is canceled and regenerated. +The singular enrollment token helps to reduce complexity, and also helps to reenforce to users that this token is an "admin" token in that it has a great deal of power, thus should be kept secret/safe. diff --git a/x-pack/plugins/ingest_manager/dev_docs/fleet_agents_interactions_detailed.md b/x-pack/plugins/ingest_manager/dev_docs/fleet_agents_interactions_detailed.md new file mode 100644 index 00000000000000..ac7005063da9de --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/fleet_agents_interactions_detailed.md @@ -0,0 +1,49 @@ +# Fleet <-> Agent Interactions + +## Agent enrollment and checkin + +Fleet workflow: + +- an agent enroll to fleet using an enrollmentAPiKey +- Every n seconds agent is polling the checkin API to send events and check for new configuration + +### Agent enrollment + +An agent must enroll using the REST Api provided by fleet. +When an agent enroll Fleet: + +- verify the API Key is a valid ES API key +- retrieve the Saved Object (SO) associated to this api key id (this SO contains the configuration|policy id) +- create an ES ApiKey unique to the agent for accessing kibana during checkin +- create an ES ApiKey per output to send logs and metrics to the output +- Save the new agent in a SO with keys encrypted inside the agent SO object + +![](schema/agent_enroll.png) + +### Agent checkin + +Agent are going to poll the checkin API to send events and check for new configration. To checkin agent are going to use the REST Api provided by fleet. + +When an agent checkin fleet: + +- verify the access API Key is a valid ES API key +- retrieve the agent (SO associated to this api key id) +- Insert events SO +- If the Agent configuration has been updated since last checkin + - generate the agent config + - Create the missing API key for agent -> ES communication +- Save the new agent (with last checkin date) in a SavedObject with keys encrypted inside the agent + +![](schema/agent_checkin.png) + +### Agent acknowledgement + +This is really similar to the checkin (same auth mecanism) and it's used for agent to acknowlege action received during checkin. + +An agent can acknowledge one or multiple actions by calling `POST /api/ingest_manager/fleet/agents/{agentId}/acks` + +## Other interactions + +### Agent Configuration update + +When a configuration is updated, every SO agent running this configuration is updated with a timestamp of the latest config. diff --git a/x-pack/plugins/ingest_manager/dev_docs/schema/agent_checkin.mml b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_checkin.mml new file mode 100644 index 00000000000000..a5332c50ab9472 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_checkin.mml @@ -0,0 +1,37 @@ +sequenceDiagram + participant Agent + participant Fleet API + participant SavedObjects + participant Elasticsearch + + Agent->>Fleet API: checkin(
accessAPIKey, events, updatedMetadata
) + rect rgba(191, 223, 255, .2) + Note over Fleet API,Elasticsearch: Authenticate the agent + Fleet API->>Elasticsearch: GET /_security/privileges + Fleet API->>SavedObjects: getAgent(apiKeyId) + Note right of SavedObjects: encrypted SO here + end + + alt If configuration updated since last checkin + Fleet API->>SavedObjects: getAgentConfiguration(configId) + + opt If there is not API Key for default output + Fleet API->>Elasticsearch: createAgentESApiKey() + end + end + + rect rgba(191, 223, 255, .2) + Note over Fleet API,Elasticsearch: Process agent events
(going to move to the agent directly) + Fleet API->>SavedObjects: createAgentEvents(events) + end + + + + rect rgba(191, 223, 255, .2) + Note over Fleet API,Elasticsearch: Update agent + Fleet API->>SavedObjects: updateAgent(metadata, checkinAt) + + end + + + Fleet API->>Agent: actions|agent config diff --git a/x-pack/plugins/ingest_manager/dev_docs/schema/agent_checkin.png b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_checkin.png new file mode 100644 index 00000000000000..cfb068b509b3a3 Binary files /dev/null and b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_checkin.png differ diff --git a/x-pack/plugins/ingest_manager/dev_docs/schema/agent_enroll.mml b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_enroll.mml new file mode 100644 index 00000000000000..8b8acbdcc68f26 --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_enroll.mml @@ -0,0 +1,17 @@ +sequenceDiagram + participant Agent + participant Fleet API + participant SavedObjects + participant Elasticsearch + + Agent->>Fleet API: enroll(enrollmentAPIKey) + + rect rgba(191, 223, 255, 0.2) + Note over Fleet API,Elasticsearch: Verify API Key + Fleet API->>Elasticsearch: GET /\_security/privileges + Fleet API->>SavedObjects: getEnrollmentApiKey(apiKeyId) + end + + + + Fleet API->>Agent: success diff --git a/x-pack/plugins/ingest_manager/dev_docs/schema/agent_enroll.png b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_enroll.png new file mode 100644 index 00000000000000..9bc0cdabb4aed3 Binary files /dev/null and b/x-pack/plugins/ingest_manager/dev_docs/schema/agent_enroll.png differ diff --git a/x-pack/plugins/ingest_manager/dev_docs/schema/saved_objects.mml b/x-pack/plugins/ingest_manager/dev_docs/schema/saved_objects.mml new file mode 100644 index 00000000000000..bb6dbc1037201e --- /dev/null +++ b/x-pack/plugins/ingest_manager/dev_docs/schema/saved_objects.mml @@ -0,0 +1,82 @@ +classDiagram + agent_configs "1" -- "*" datasources + agent_configs "1" -- "*" enrollment_api_keys + agent_configs "1" -- "*" agents : is used + agent_configs "*" -- "*" outputs + agents "1" -- "*" agent_events + agents "1" -- "*" agent_events + package "1" -- "*" datasources + + class package { + installed + } + + class agents { + status + access_api_key_id + config_id + last_checkin + local_metadata + user_provided_metadata + actions // Encrypted contains new agent config + default_api_key // Encrypted + + } + + class agent_events { + type + subtype + agent_id + action_id + config_id + stream_id + timestamp + message + payload + data + } + + class agent_configs { + datasources // datasource ids + name + namespace + description + status + } + + class datasources { + name + namespace + config_id + enabled + package + output_id + // Inputs + inputs.type + inputs.enabled + inputs.processors + // Inputs streams + inputs.streams.id + inputs.streams.enabled + inputs.streams.dataset + inputs.streams.processors + inputs.streams.config + } + + class enrollment_api_keys { + config_id + api_key_id + api_key // Encrypted + } + + + class outputs { + id + hosts + ca_sha256 + config + // Encrypted - user to create API keys + admin_username + admin_password + } + diff --git a/x-pack/plugins/ingest_manager/dev_docs/schema/saved_objects.png b/x-pack/plugins/ingest_manager/dev_docs/schema/saved_objects.png new file mode 100644 index 00000000000000..dcfc6d33eae99d Binary files /dev/null and b/x-pack/plugins/ingest_manager/dev_docs/schema/saved_objects.png differ diff --git a/x-pack/plugins/ingest_manager/package.json b/x-pack/plugins/ingest_manager/package.json new file mode 100644 index 00000000000000..56b7df6b56852b --- /dev/null +++ b/x-pack/plugins/ingest_manager/package.json @@ -0,0 +1,11 @@ +{ + "author": "Elastic", + "name": "ingest-manager", + "version": "8.0.0", + "private": true, + "license": "Elastic-License", + "dependencies": { + "react-markdown": "^4.2.2" + } + } + \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/error.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/error.tsx new file mode 100644 index 00000000000000..b9659016fec09c --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/error.tsx @@ -0,0 +1,18 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { EuiCallOut } from '@elastic/eui'; + +export const Error: React.FunctionComponent<{ + title: JSX.Element; + error: Error | string; +}> = ({ title, error }) => { + return ( + +

{typeof error === 'string' ? error : error.message}

+ + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/header.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/header.tsx index 0936b5dcfed109..c87a77320d3f7a 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/header.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/header.tsx @@ -18,6 +18,8 @@ const Wrapper = styled.div` margin-left: auto; margin-right: auto; padding-top: ${props => props.theme.eui.paddingSizes.xl}; + padding-left: ${props => props.theme.eui.paddingSizes.m}; + padding-right: ${props => props.theme.eui.paddingSizes.m}; `; const Tabs = styled(EuiTabs)` @@ -36,7 +38,7 @@ export interface HeaderProps { export const Header: React.FC = ({ leftColumn, rightColumn, tabs }) => ( - + {leftColumn ? {leftColumn} : null} {rightColumn ? {rightColumn} : null} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/index.ts index b6bb29462c5693..5551bff2c8bdeb 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/index.ts @@ -4,4 +4,5 @@ * you may not use this file except in compliance with the Elastic License. */ export { Loading } from './loading'; +export { Error } from './error'; export { Header, HeaderProps } from './header'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/search_bar.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/search_bar.tsx new file mode 100644 index 00000000000000..41c24dadba0682 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/components/search_bar.tsx @@ -0,0 +1,162 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useState, useEffect } from 'react'; +import { IFieldType } from 'src/plugins/data/public'; +// @ts-ignore +import { EuiSuggest, EuiSuggestItemProps } from '@elastic/eui'; +import { useDebounce } from '../hooks'; +import { useStartDeps } from '../hooks/use_deps'; +import { INDEX_NAME } from '../constants'; + +const DEBOUNCE_SEARCH_MS = 150; +const HIDDEN_FIELDS = ['agents.actions']; + +interface Suggestion { + label: string; + description: string; + value: string; + type: { + color: string; + iconType: string; + }; + start: number; + end: number; +} + +interface Props { + value: string; + fieldPrefix: string; + onChange: (newValue: string) => void; +} + +export const SearchBar: React.FunctionComponent = ({ value, fieldPrefix, onChange }) => { + const { suggestions } = useSuggestions(fieldPrefix, value); + + // TODO fix type when correctly typed in EUI + const onAutocompleteClick = (suggestion: any) => { + onChange( + [value.slice(0, suggestion.start), suggestion.value, value.slice(suggestion.end, -1)].join('') + ); + }; + // TODO fix type when correctly typed in EUI + const onChangeSearch = (e: any) => { + onChange(e.value); + }; + + return ( + { + return { + ...suggestion, + // For type + onClick: () => {}, + }; + })} + /> + ); +}; + +function transformSuggestionType(type: string): { iconType: string; color: string } { + switch (type) { + case 'field': + return { iconType: 'kqlField', color: 'tint4' }; + case 'value': + return { iconType: 'kqlValue', color: 'tint0' }; + case 'conjunction': + return { iconType: 'kqlSelector', color: 'tint3' }; + case 'operator': + return { iconType: 'kqlOperand', color: 'tint1' }; + default: + return { iconType: 'kqlOther', color: 'tint1' }; + } +} + +function useSuggestions(fieldPrefix: string, search: string) { + const { data } = useStartDeps(); + + const debouncedSearch = useDebounce(search, DEBOUNCE_SEARCH_MS); + const [suggestions, setSuggestions] = useState([]); + + const fetchSuggestions = async () => { + try { + const res = (await data.indexPatterns.getFieldsForWildcard({ + pattern: INDEX_NAME, + })) as IFieldType[]; + + if (!data || !data.autocomplete) { + throw new Error('Missing data plugin'); + } + const query = debouncedSearch || ''; + // @ts-ignore + const esSuggestions = ( + await data.autocomplete.getQuerySuggestions({ + language: 'kuery', + indexPatterns: [ + { + title: INDEX_NAME, + fields: res, + }, + ], + boolFilter: [], + query, + selectionStart: query.length, + selectionEnd: query.length, + }) + ) + .filter(suggestion => { + if (suggestion.type === 'conjunction') { + return true; + } + if (suggestion.type === 'value') { + return true; + } + if (suggestion.type === 'operator') { + return true; + } + + if (fieldPrefix && suggestion.text.startsWith(fieldPrefix)) { + for (const hiddenField of HIDDEN_FIELDS) { + if (suggestion.text.startsWith(hiddenField)) { + return false; + } + } + return true; + } + + return false; + }) + .map((suggestion: any) => ({ + label: suggestion.text, + description: suggestion.description || '', + type: transformSuggestionType(suggestion.type), + start: suggestion.start, + end: suggestion.end, + value: suggestion.text, + })); + + setSuggestions(esSuggestions); + } catch (err) { + setSuggestions([]); + } + }; + + useEffect(() => { + fetchSuggestions(); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [debouncedSearch]); + + return { + suggestions, + }; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/constants/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/constants/index.ts index 1af39a60455e02..1ac5bef629fde1 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/constants/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/constants/index.ts @@ -3,16 +3,15 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ - -export { - PLUGIN_ID, - EPM_API_ROUTES, - DEFAULT_AGENT_CONFIG_ID, - AGENT_CONFIG_SAVED_OBJECT_TYPE, -} from '../../../../common'; +export { PLUGIN_ID, EPM_API_ROUTES, AGENT_CONFIG_SAVED_OBJECT_TYPE } from '../../../../common'; export const BASE_PATH = '/app/ingestManager'; export const EPM_PATH = '/epm'; +export const EPM_DETAIL_VIEW_PATH = `${EPM_PATH}/detail/:pkgkey/:panel?`; export const AGENT_CONFIG_PATH = '/configs'; export const AGENT_CONFIG_DETAILS_PATH = `${AGENT_CONFIG_PATH}/`; export const FLEET_PATH = '/fleet'; +export const FLEET_AGENTS_PATH = `${FLEET_PATH}/agents`; +export const FLEET_AGENT_DETAIL_PATH = `${FLEET_AGENTS_PATH}/`; + +export const INDEX_NAME = '.kibana'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/index.ts index a224b599c13af0..5e0695bd3e305a 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/index.ts @@ -4,10 +4,13 @@ * you may not use this file except in compliance with the Elastic License. */ +export { useCapabilities } from './use_capabilities'; export { useCore, CoreContext } from './use_core'; export { useConfig, ConfigContext } from './use_config'; -export { useDeps, DepsContext } from './use_deps'; +export { useSetupDeps, useStartDeps, DepsContext } from './use_deps'; export { useLink } from './use_link'; -export { usePagination } from './use_pagination'; +export { usePagination, Pagination } from './use_pagination'; export { useDebounce } from './use_debounce'; export * from './use_request'; +export * from './use_input'; +export * from './use_url_params'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_capabilities.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_capabilities.ts new file mode 100644 index 00000000000000..0a16c4a62a7d1f --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_capabilities.ts @@ -0,0 +1,12 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { useCore } from './'; + +export function useCapabilities() { + const core = useCore(); + return core.application.capabilities.ingestManager; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_deps.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_deps.ts index a2e2f278930e3f..25e4ee8fca43ca 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_deps.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_deps.ts @@ -5,14 +5,25 @@ */ import React, { useContext } from 'react'; -import { IngestManagerSetupDeps } from '../../../plugin'; +import { IngestManagerSetupDeps, IngestManagerStartDeps } from '../../../plugin'; -export const DepsContext = React.createContext(null); +export const DepsContext = React.createContext<{ + setup: IngestManagerSetupDeps; + start: IngestManagerStartDeps; +} | null>(null); -export function useDeps() { +export function useSetupDeps() { const deps = useContext(DepsContext); if (deps === null) { throw new Error('DepsContext not initialized'); } - return deps; + return deps.setup; +} + +export function useStartDeps() { + const deps = useContext(DepsContext); + if (deps === null) { + throw new Error('StartDepsContext not initialized'); + } + return deps.start; } diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_input.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_input.ts new file mode 100644 index 00000000000000..4aa0ad7155d2f5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_input.ts @@ -0,0 +1,24 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React from 'react'; + +export function useInput(defaultValue = '') { + const [value, setValue] = React.useState(defaultValue); + + return { + value, + props: { + onChange: (e: React.ChangeEvent) => { + setValue(e.target.value); + }, + value, + }, + clear: () => { + setValue(''); + }, + }; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_pagination.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_pagination.tsx index ae0352a33b2ff2..699bba3c62f97a 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_pagination.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_pagination.tsx @@ -20,6 +20,6 @@ export function usePagination() { return { pagination, setPagination, - pageSizeOptions: 20, + pageSizeOptions: [5, 20, 50], }; } diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agent_config.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agent_config.ts index 389909e1d99ef4..c2981aee42ad37 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agent_config.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agent_config.ts @@ -3,18 +3,17 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ - import { HttpFetchQuery } from 'kibana/public'; import { useRequest, sendRequest } from './use_request'; import { agentConfigRouteService } from '../../services'; import { GetAgentConfigsResponse, GetOneAgentConfigResponse, - CreateAgentConfigRequestSchema, + CreateAgentConfigRequest, CreateAgentConfigResponse, - UpdateAgentConfigRequestSchema, + UpdateAgentConfigRequest, UpdateAgentConfigResponse, - DeleteAgentConfigsRequestSchema, + DeleteAgentConfigsRequest, DeleteAgentConfigsResponse, } from '../../types'; @@ -33,7 +32,14 @@ export const useGetOneAgentConfig = (agentConfigId: string) => { }); }; -export const sendCreateAgentConfig = (body: CreateAgentConfigRequestSchema['body']) => { +export const sendGetOneAgentConfig = (agentConfigId: string) => { + return sendRequest({ + path: agentConfigRouteService.getInfoPath(agentConfigId), + method: 'get', + }); +}; + +export const sendCreateAgentConfig = (body: CreateAgentConfigRequest['body']) => { return sendRequest({ path: agentConfigRouteService.getCreatePath(), method: 'post', @@ -43,7 +49,7 @@ export const sendCreateAgentConfig = (body: CreateAgentConfigRequestSchema['body export const sendUpdateAgentConfig = ( agentConfigId: string, - body: UpdateAgentConfigRequestSchema['body'] + body: UpdateAgentConfigRequest['body'] ) => { return sendRequest({ path: agentConfigRouteService.getUpdatePath(agentConfigId), @@ -52,7 +58,7 @@ export const sendUpdateAgentConfig = ( }); }; -export const sendDeleteAgentConfigs = (body: DeleteAgentConfigsRequestSchema['body']) => { +export const sendDeleteAgentConfigs = (body: DeleteAgentConfigsRequest['body']) => { return sendRequest({ path: agentConfigRouteService.getDeletePath(), method: 'post', diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agents.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agents.ts new file mode 100644 index 00000000000000..f08b950e71ea8c --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/agents.ts @@ -0,0 +1,61 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { useRequest, UseRequestConfig, sendRequest } from './use_request'; +import { agentRouteService } from '../../services'; +import { + GetOneAgentResponse, + GetOneAgentEventsResponse, + GetOneAgentEventsRequest, + GetAgentsRequest, + GetAgentsResponse, + GetAgentStatusRequest, + GetAgentStatusResponse, +} from '../../types'; + +type RequestOptions = Pick, 'pollIntervalMs'>; + +export function useGetOneAgent(agentId: string, options?: RequestOptions) { + return useRequest({ + path: agentRouteService.getInfoPath(agentId), + method: 'get', + ...options, + }); +} + +export function useGetOneAgentEvents( + agentId: string, + query: GetOneAgentEventsRequest['query'], + options?: RequestOptions +) { + return useRequest({ + path: agentRouteService.getEventsPath(agentId), + method: 'get', + query, + ...options, + }); +} + +export function useGetAgents(query: GetAgentsRequest['query'], options?: RequestOptions) { + return useRequest({ + method: 'get', + path: agentRouteService.getListPath(), + query, + ...options, + }); +} + +export function sendGetAgentStatus( + query: GetAgentStatusRequest['query'], + options?: RequestOptions +) { + return sendRequest({ + method: 'get', + path: agentRouteService.getStatusPath(), + query, + ...options, + }); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/datasource.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/datasource.ts new file mode 100644 index 00000000000000..60fbb9f0d2afa5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/datasource.ts @@ -0,0 +1,16 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { sendRequest } from './use_request'; +import { datasourceRouteService } from '../../services'; +import { CreateDatasourceRequest, CreateDatasourceResponse } from '../../types'; + +export const sendCreateDatasource = (body: CreateDatasourceRequest['body']) => { + return sendRequest({ + path: datasourceRouteService.getCreatePath(), + method: 'post', + body: JSON.stringify(body), + }); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/enrollment_api_keys.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/enrollment_api_keys.ts new file mode 100644 index 00000000000000..2640f36423a0c3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/enrollment_api_keys.ts @@ -0,0 +1,27 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { useRequest, UseRequestConfig } from './use_request'; +import { enrollmentAPIKeyRouteService } from '../../services'; +import { GetOneEnrollmentAPIKeyResponse, GetEnrollmentAPIKeysResponse } from '../../types'; + +type RequestOptions = Pick, 'pollIntervalMs'>; + +export function useGetOneEnrollmentAPIKey(keyId: string, options?: RequestOptions) { + return useRequest({ + method: 'get', + path: enrollmentAPIKeyRouteService.getInfoPath(keyId), + ...options, + }); +} + +export function useGetEnrollmentAPIKeys(options?: RequestOptions) { + return useRequest({ + method: 'get', + path: enrollmentAPIKeyRouteService.getListPath(), + ...options, + }); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/epm.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/epm.ts new file mode 100644 index 00000000000000..02865ffe6fb1ab --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/epm.ts @@ -0,0 +1,66 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { HttpFetchQuery } from 'kibana/public'; +import { useRequest, sendRequest } from './use_request'; +import { epmRouteService } from '../../services'; +import { + GetCategoriesResponse, + GetPackagesResponse, + GetInfoResponse, + InstallPackageResponse, + DeletePackageResponse, +} from '../../types'; + +export const useGetCategories = () => { + return useRequest({ + path: epmRouteService.getCategoriesPath(), + method: 'get', + }); +}; + +export const useGetPackages = (query: HttpFetchQuery = {}) => { + return useRequest({ + path: epmRouteService.getListPath(), + method: 'get', + query, + }); +}; + +export const useGetPackageInfoByKey = (pkgkey: string) => { + return useRequest({ + path: epmRouteService.getInfoPath(pkgkey), + method: 'get', + }); +}; + +export const sendGetPackageInfoByKey = (pkgkey: string) => { + return sendRequest({ + path: epmRouteService.getInfoPath(pkgkey), + method: 'get', + }); +}; + +export const sendGetFileByPath = (filePath: string) => { + return sendRequest({ + path: epmRouteService.getFilePath(filePath), + method: 'get', + }); +}; + +export const sendInstallPackage = (pkgkey: string) => { + return sendRequest({ + path: epmRouteService.getInstallPath(pkgkey), + method: 'post', + }); +}; + +export const sendRemovePackage = (pkgkey: string) => { + return sendRequest({ + path: epmRouteService.getRemovePath(pkgkey), + method: 'delete', + }); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/index.ts index 68d67080d90bae..5014049407e65d 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/index.ts @@ -3,5 +3,9 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -export { setHttpClient, sendRequest } from './use_request'; +export { setHttpClient, sendRequest, useRequest } from './use_request'; export * from './agent_config'; +export * from './datasource'; +export * from './agents'; +export * from './enrollment_api_keys'; +export * from './epm'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/setup.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/setup.ts new file mode 100644 index 00000000000000..04fdf9f66948f0 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/setup.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { sendRequest } from './use_request'; +import { setupRouteService } from '../../services'; + +export const sendSetup = () => { + return sendRequest({ + path: setupRouteService.getSetupPath(), + method: 'post', + }); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/use_request.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/use_request.ts index 12b4d0bdf7df6a..4b434bd1a149e2 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/use_request.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_request/use_request.ts @@ -7,13 +7,15 @@ import { HttpSetup } from 'kibana/public'; import { SendRequestConfig, SendRequestResponse, - UseRequestConfig, + UseRequestConfig as _UseRequestConfig, sendRequest as _sendRequest, useRequest as _useRequest, } from '../../../../../../../../src/plugins/es_ui_shared/public'; let httpClient: HttpSetup; +export type UseRequestConfig = _UseRequestConfig; + export const setHttpClient = (client: HttpSetup) => { httpClient = client; }; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_url_params.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_url_params.ts new file mode 100644 index 00000000000000..817d2dad88f0a4 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/hooks/use_url_params.ts @@ -0,0 +1,28 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { useLocation } from 'react-router-dom'; +import { parse, stringify } from 'query-string'; +import { useCallback, useEffect, useState } from 'react'; + +/** + * Parses `search` params and returns an object with them along with a `toUrlParams` function + * that allows being able to retrieve a stringified version of an object (default is the + * `urlParams` that was parsed) for use in the url. + * Object will be recreated every time `search` changes. + */ +export function useUrlParams() { + const { search } = useLocation(); + const [urlParams, setUrlParams] = useState(() => parse(search)); + const toUrlParams = useCallback((params = urlParams) => stringify(params), [urlParams]); + useEffect(() => { + setUrlParams(parse(search)); + }, [search]); + return { + urlParams, + toUrlParams, + }; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/index.tsx index 935eb42d0347e8..9a85358a2a69c3 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/index.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/index.tsx @@ -3,18 +3,26 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import React from 'react'; +import React, { useEffect, useState } from 'react'; import ReactDOM from 'react-dom'; import { useObservable } from 'react-use'; import { HashRouter as Router, Redirect, Switch, Route, RouteProps } from 'react-router-dom'; -import { CoreStart, AppMountParameters } from 'kibana/public'; +import { FormattedMessage } from '@kbn/i18n/react'; import { EuiErrorBoundary } from '@elastic/eui'; +import { CoreStart, AppMountParameters } from 'kibana/public'; import { EuiThemeProvider } from '../../../../../legacy/common/eui_styled_components'; -import { IngestManagerSetupDeps, IngestManagerConfigType } from '../../plugin'; +import { + IngestManagerSetupDeps, + IngestManagerConfigType, + IngestManagerStartDeps, +} from '../../plugin'; import { EPM_PATH, FLEET_PATH, AGENT_CONFIG_PATH } from './constants'; -import { DefaultLayout } from './layouts'; +import { DefaultLayout, WithoutHeaderLayout } from './layouts'; +import { Loading, Error } from './components'; import { IngestManagerOverview, EPMApp, AgentConfigApp, FleetApp } from './sections'; import { CoreContext, DepsContext, ConfigContext, setHttpClient, useConfig } from './hooks'; +import { PackageInstallProvider } from './sections/epm/hooks'; +import { sendSetup } from './hooks/use_request/setup'; export interface ProtectedRouteProps extends RouteProps { isAllowed?: boolean; @@ -32,6 +40,50 @@ export const ProtectedRoute: React.FunctionComponent = ({ const IngestManagerRoutes = ({ ...rest }) => { const { epm, fleet } = useConfig(); + const [isInitialized, setIsInitialized] = useState(false); + const [initializationError, setInitializationError] = useState(null); + + useEffect(() => { + (async () => { + setIsInitialized(false); + setInitializationError(null); + try { + const res = await sendSetup(); + if (res.error) { + setInitializationError(res.error); + } + } catch (err) { + setInitializationError(err); + } + setIsInitialized(true); + })(); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, []); + + if (!isInitialized || initializationError) { + return ( + + + + {initializationError ? ( + + } + error={initializationError} + /> + ) : ( + + )} + + + + ); + } + return ( @@ -66,22 +118,26 @@ const IngestManagerRoutes = ({ ...rest }) => { const IngestManagerApp = ({ basepath, coreStart, - deps, + setupDeps, + startDeps, config, }: { basepath: string; coreStart: CoreStart; - deps: IngestManagerSetupDeps; + setupDeps: IngestManagerSetupDeps; + startDeps: IngestManagerStartDeps; config: IngestManagerConfigType; }) => { const isDarkMode = useObservable(coreStart.uiSettings.get$('theme:darkMode')); return ( - + - + + + @@ -93,12 +149,19 @@ const IngestManagerApp = ({ export function renderApp( coreStart: CoreStart, { element, appBasePath }: AppMountParameters, - deps: IngestManagerSetupDeps, + setupDeps: IngestManagerSetupDeps, + startDeps: IngestManagerStartDeps, config: IngestManagerConfigType ) { setHttpClient(coreStart.http); ReactDOM.render( - , + , element ); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/default.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/default.tsx index f99d1bfe500268..8ec2d2ec03b358 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/default.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/default.tsx @@ -12,7 +12,7 @@ import { useLink, useConfig } from '../hooks'; import { EPM_PATH, FLEET_PATH, AGENT_CONFIG_PATH } from '../constants'; interface Props { - section: Section; + section?: Section; children?: React.ReactNode; } @@ -43,7 +43,7 @@ export const DefaultLayout: React.FunctionComponent = ({ section, childre - + = ({ children, ...rest }) => ( +export const WithHeaderLayout: React.FC = ({ restrictWidth, children, ...rest }) => (
- + {children} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/without_header.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/without_header.tsx new file mode 100644 index 00000000000000..cad98c5a0a7e1d --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/layouts/without_header.tsx @@ -0,0 +1,28 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { Fragment } from 'react'; +import styled from 'styled-components'; +import { EuiPage, EuiPageBody, EuiSpacer } from '@elastic/eui'; + +const Page = styled(EuiPage)` + background: ${props => props.theme.eui.euiColorEmptyShade}; +`; + +interface Props { + restrictWidth?: number; + children?: React.ReactNode; +} + +export const WithoutHeaderLayout: React.FC = ({ restrictWidth, children }) => ( + + + + + {children} + + + +); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_delete_provider.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_delete_provider.tsx index 6f51415a562a3d..b18349e0787666 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_delete_provider.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_delete_provider.tsx @@ -119,10 +119,10 @@ export const AgentConfigDeleteProvider: React.FunctionComponent = ({ chil } setIsLoadingAgentsCount(true); const { data } = await sendRequest<{ total: number }>({ - path: `/api/fleet/agents`, + path: `/api/ingest_manager/fleet/agents`, method: 'get', query: { - kuery: `agents.policy_id : (${agentConfigsToCheck.join(' or ')})`, + kuery: `agents.config_id : (${agentConfigsToCheck.join(' or ')})`, }, }); setAgentsCount(data?.total || 0); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_form.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_form.tsx index 5a25dc8bc92b5e..c1c9ce507c92c0 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_form.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/config_form.tsx @@ -4,15 +4,36 @@ * you may not use this file except in compliance with the Elastic License. */ -import React, { useState } from 'react'; -import { EuiFieldText, EuiForm, EuiFormRow } from '@elastic/eui'; +import React, { useMemo, useState } from 'react'; +import { + EuiAccordion, + EuiFieldText, + EuiFlexGroup, + EuiFlexItem, + EuiForm, + EuiFormRow, + EuiHorizontalRule, + EuiSpacer, + EuiSwitch, + EuiText, + EuiComboBox, + EuiIconTip, +} from '@elastic/eui'; import { FormattedMessage } from '@kbn/i18n/react'; +import { i18n } from '@kbn/i18n'; +import styled from 'styled-components'; import { NewAgentConfig } from '../../../types'; interface ValidationResults { [key: string]: JSX.Element[]; } +const StyledEuiAccordion = styled(EuiAccordion)` + .ingest-active-button { + color: ${props => props.theme.eui.euiColorPrimary}}; + } +`; + export const agentConfigFormValidation = ( agentConfig: Partial ): ValidationResults => { @@ -42,55 +63,185 @@ export const AgentConfigForm: React.FunctionComponent = ({ validation, }) => { const [touchedFields, setTouchedFields] = useState<{ [key: string]: boolean }>({}); - const fields: Array<{ name: 'name' | 'description' | 'namespace'; label: JSX.Element }> = [ - { - name: 'name', - label: ( - - ), - }, - { - name: 'description', - label: ( - - ), - }, - { - name: 'namespace', - label: ( - - ), - }, - ]; + const [showNamespace, setShowNamespace] = useState(false); + const fields: Array<{ + name: 'name' | 'description' | 'namespace'; + label: JSX.Element; + placeholder: string; + }> = useMemo(() => { + return [ + { + name: 'name', + label: ( + + ), + placeholder: i18n.translate('xpack.ingestManager.agentConfigForm.nameFieldPlaceholder', { + defaultMessage: 'Choose a name', + }), + }, + { + name: 'description', + label: ( + + ), + placeholder: i18n.translate( + 'xpack.ingestManager.agentConfigForm.descriptionFieldPlaceholder', + { + defaultMessage: 'How will this configuration be used?', + } + ), + }, + ]; + }, []); return ( - {fields.map(({ name, label }) => { + {fields.map(({ name, label, placeholder }) => { return ( updateAgentConfig({ [name]: e.target.value })} isInvalid={Boolean(touchedFields[name] && validation[name])} onBlur={() => setTouchedFields({ ...touchedFields, [name]: true })} + placeholder={placeholder} /> ); })} + + + + } + > + + {' '} + + + } + checked={true} + onChange={() => { + // FIXME: enable collection of system metrics - see: https://github.com/elastic/kibana/issues/59564 + }} + /> + + + + + } + buttonClassName="ingest-active-button" + > + + + + +

+ +

+ + + + + + + + + } + checked={showNamespace} + onChange={() => { + setShowNamespace(!showNamespace); + if (showNamespace) { + updateAgentConfig({ namespace: '' }); + } + }} + /> + {showNamespace && ( + <> + + + { + updateAgentConfig({ namespace: value }); + }} + onChange={selectedOptions => { + updateAgentConfig({ + namespace: (selectedOptions.length ? selectedOptions[0] : '') as string, + }); + }} + isInvalid={Boolean(touchedFields.namespace && validation.namespace)} + onBlur={() => setTouchedFields({ ...touchedFields, namespace: true })} + /> + + + )} + + + ); }; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/index.ts index d838221cd844ec..a0fdc656dd7ed1 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/index.ts @@ -6,3 +6,4 @@ export { AgentConfigForm, agentConfigFormValidation } from './config_form'; export { AgentConfigDeleteProvider } from './config_delete_provider'; +export { LinkedAgentCount } from './linked_agent_count'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/linked_agent_count.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/linked_agent_count.tsx new file mode 100644 index 00000000000000..ec66108c60f680 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/components/linked_agent_count.tsx @@ -0,0 +1,31 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { memo } from 'react'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { EuiLink } from '@elastic/eui'; +import { useLink } from '../../../hooks'; +import { FLEET_AGENTS_PATH } from '../../../constants'; + +export const LinkedAgentCount = memo<{ count: number; agentConfigId: string }>( + ({ count, agentConfigId }) => { + const FLEET_URI = useLink(FLEET_AGENTS_PATH); + const displayValue = ( + + ); + return count > 0 ? ( + + {displayValue} + + ) : ( + displayValue + ); + } +); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_config.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_config.tsx new file mode 100644 index 00000000000000..39f2f048ab88d0 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_config.tsx @@ -0,0 +1,136 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState, Fragment } from 'react'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiFlexGrid, + EuiFlexGroup, + EuiFlexItem, + EuiText, + EuiSpacer, + EuiButtonEmpty, + EuiTitle, +} from '@elastic/eui'; +import { DatasourceInput, RegistryVarsEntry } from '../../../../types'; +import { DatasourceInputVarField } from './datasource_input_var_field'; + +export const DatasourceInputConfig: React.FunctionComponent<{ + packageInputVars?: RegistryVarsEntry[]; + datasourceInput: DatasourceInput; + updateDatasourceInput: (updatedInput: Partial) => void; +}> = ({ packageInputVars, datasourceInput, updateDatasourceInput }) => { + // Showing advanced options toggle state + const [isShowingAdvanced, setIsShowingAdvanced] = useState(false); + + const requiredVars: RegistryVarsEntry[] = []; + const advancedVars: RegistryVarsEntry[] = []; + + if (packageInputVars) { + packageInputVars.forEach(varDef => { + if (varDef.required && !varDef.default) { + requiredVars.push(varDef); + } else { + advancedVars.push(varDef); + } + }); + } + + return ( + + + +

+ +

+ + + +

+ +

+ + + + + {requiredVars.map(varDef => { + const varName = varDef.name; + const value = datasourceInput.streams[0].config![varName]; + return ( + + { + updateDatasourceInput({ + streams: datasourceInput.streams.map(stream => ({ + ...stream, + config: { + ...stream.config, + [varName]: newValue, + }, + })), + }); + }} + /> + + ); + })} + {advancedVars.length ? ( + + + {/* Wrapper div to prevent button from going full width */} +
+ setIsShowingAdvanced(!isShowingAdvanced)} + flush="left" + > + + +
+ + {isShowingAdvanced + ? advancedVars.map(varDef => { + const varName = varDef.name; + const value = datasourceInput.streams[0].config![varName]; + return ( + + { + updateDatasourceInput({ + streams: datasourceInput.streams.map(stream => ({ + ...stream, + config: { + ...stream.config, + [varName]: newValue, + }, + })), + }); + }} + /> + + ); + }) + : null} + + ) : null} + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_panel.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_panel.tsx new file mode 100644 index 00000000000000..74b08f48df12de --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_panel.tsx @@ -0,0 +1,178 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState, Fragment } from 'react'; +import styled from 'styled-components'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiPanel, + EuiFlexGroup, + EuiFlexItem, + EuiSwitch, + EuiText, + EuiTextColor, + EuiButtonIcon, + EuiHorizontalRule, + EuiSpacer, +} from '@elastic/eui'; +import { DatasourceInput, DatasourceInputStream, RegistryInput } from '../../../../types'; +import { DatasourceInputConfig } from './datasource_input_config'; +import { DatasourceInputStreamConfig } from './datasource_input_stream_config'; + +const FlushHorizontalRule = styled(EuiHorizontalRule)` + margin-left: -${props => props.theme.eui.paddingSizes.m}; + margin-right: -${props => props.theme.eui.paddingSizes.m}; + width: auto; +`; + +export const DatasourceInputPanel: React.FunctionComponent<{ + packageInput: RegistryInput; + datasourceInput: DatasourceInput; + updateDatasourceInput: (updatedInput: Partial) => void; +}> = ({ packageInput, datasourceInput, updateDatasourceInput }) => { + // Showing streams toggle state + const [isShowingStreams, setIsShowingStreams] = useState(false); + + return ( + + {/* Header / input-level toggle */} + + + +

{packageInput.title || packageInput.type}

+ + } + checked={datasourceInput.enabled} + onChange={e => { + const enabled = e.target.checked; + updateDatasourceInput({ + enabled, + streams: datasourceInput.streams.map(stream => ({ + ...stream, + enabled, + })), + }); + }} + /> + + + + + + + + {datasourceInput.streams.filter(stream => stream.enabled).length} + + + ), + total: packageInput.streams.length, + }} + /> + + + + setIsShowingStreams(!isShowingStreams)} + color="text" + aria-label={ + isShowingStreams + ? i18n.translate( + 'xpack.ingestManager.createDatasource.stepConfigure.hideStreamsAriaLabel', + { + defaultMessage: 'Hide {type} streams', + values: { + type: packageInput.type, + }, + } + ) + : i18n.translate( + 'xpack.ingestManager.createDatasource.stepConfigure.showStreamsAriaLabel', + { + defaultMessage: 'Show {type} streams', + values: { + type: packageInput.type, + }, + } + ) + } + /> + + + + + + {/* Header rule break */} + {isShowingStreams ? : null} + + {/* Input level configuration */} + {isShowingStreams && packageInput.vars && packageInput.vars.length ? ( + + + + + ) : null} + + {/* Per-stream configuration */} + {isShowingStreams ? ( + + {packageInput.streams.map(packageInputStream => { + const datasourceInputStream = datasourceInput.streams.find( + stream => stream.dataset === packageInputStream.dataset + ); + return datasourceInputStream ? ( + + ) => { + const indexOfUpdatedStream = datasourceInput.streams.findIndex( + stream => stream.dataset === packageInputStream.dataset + ); + const newStreams = [...datasourceInput.streams]; + newStreams[indexOfUpdatedStream] = { + ...newStreams[indexOfUpdatedStream], + ...updatedStream, + }; + + const updatedInput: Partial = { + streams: newStreams, + }; + + // Update input enabled state if needed + if (!datasourceInput.enabled && updatedStream.enabled) { + updatedInput.enabled = true; + } else if ( + datasourceInput.enabled && + !newStreams.find(stream => stream.enabled) + ) { + updatedInput.enabled = false; + } + + updateDatasourceInput(updatedInput); + }} + /> + + + + ) : null; + })} + + ) : null} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_stream_config.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_stream_config.tsx new file mode 100644 index 00000000000000..e4b138932cb534 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_stream_config.tsx @@ -0,0 +1,132 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState, Fragment } from 'react'; +import ReactMarkdown from 'react-markdown'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiFlexGrid, + EuiFlexGroup, + EuiFlexItem, + EuiSwitch, + EuiText, + EuiSpacer, + EuiButtonEmpty, +} from '@elastic/eui'; +import { DatasourceInputStream, RegistryStream, RegistryVarsEntry } from '../../../../types'; +import { DatasourceInputVarField } from './datasource_input_var_field'; + +export const DatasourceInputStreamConfig: React.FunctionComponent<{ + packageInputStream: RegistryStream; + datasourceInputStream: DatasourceInputStream; + updateDatasourceInputStream: (updatedStream: Partial) => void; +}> = ({ packageInputStream, datasourceInputStream, updateDatasourceInputStream }) => { + // Showing advanced options toggle state + const [isShowingAdvanced, setIsShowingAdvanced] = useState(false); + + const requiredVars: RegistryVarsEntry[] = []; + const advancedVars: RegistryVarsEntry[] = []; + + if (packageInputStream.vars && packageInputStream.vars.length) { + packageInputStream.vars.forEach(varDef => { + if (varDef.required && !varDef.default) { + requiredVars.push(varDef); + } else { + advancedVars.push(varDef); + } + }); + } + + return ( + + + { + const enabled = e.target.checked; + updateDatasourceInputStream({ + enabled, + }); + }} + /> + {packageInputStream.description ? ( + + + + + + + ) : null} + + + + {requiredVars.map(varDef => { + const varName = varDef.name; + const value = datasourceInputStream.config![varName]; + return ( + + { + updateDatasourceInputStream({ + config: { + ...datasourceInputStream.config, + [varName]: newValue, + }, + }); + }} + /> + + ); + })} + {advancedVars.length ? ( + + + {/* Wrapper div to prevent button from going full width */} +
+ setIsShowingAdvanced(!isShowingAdvanced)} + flush="left" + > + + +
+ + {isShowingAdvanced + ? advancedVars.map(varDef => { + const varName = varDef.name; + const value = datasourceInputStream.config![varName]; + return ( + + { + updateDatasourceInputStream({ + config: { + ...datasourceInputStream.config, + [varName]: newValue, + }, + }); + }} + /> + + ); + }) + : null} + + ) : null} + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_var_field.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_var_field.tsx new file mode 100644 index 00000000000000..4186b6a488f54d --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/datasource_input_var_field.tsx @@ -0,0 +1,48 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import ReactMarkdown from 'react-markdown'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { EuiFormRow, EuiFieldText, EuiComboBox, EuiText } from '@elastic/eui'; +import { RegistryVarsEntry } from '../../../../types'; + +export const DatasourceInputVarField: React.FunctionComponent<{ + varDef: RegistryVarsEntry; + value: any; + onChange: (newValue: any) => void; +}> = ({ varDef, value, onChange }) => { + return ( + + + + ) : null + } + helpText={} + > + {varDef.multi ? ( + ({ label: val }))} + onCreateOption={(newVal: any) => { + onChange([...value, newVal]); + }} + onChange={(newVals: any[]) => { + onChange(newVals.map(val => val.label)); + }} + /> + ) : ( + onChange(e.target.value)} /> + )} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/index.ts new file mode 100644 index 00000000000000..e5f18e1449d1b2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/index.ts @@ -0,0 +1,7 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export { CreateDatasourcePageLayout } from './layout'; +export { DatasourceInputPanel } from './datasource_input_panel'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/layout.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/layout.tsx new file mode 100644 index 00000000000000..c063155c571d25 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/layout.tsx @@ -0,0 +1,123 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiFlexGroup, + EuiFlexItem, + EuiText, + EuiDescriptionList, + EuiDescriptionListTitle, + EuiDescriptionListDescription, + EuiButtonEmpty, + EuiSpacer, +} from '@elastic/eui'; +import { WithHeaderLayout } from '../../../../layouts'; +import { AgentConfig, PackageInfo } from '../../../../types'; +import { PackageIcon } from '../../../epm/components'; +import { CreateDatasourceFrom, CreateDatasourceStep } from '../types'; +import { CreateDatasourceStepsNavigation } from './navigation'; + +export const CreateDatasourcePageLayout: React.FunctionComponent<{ + from: CreateDatasourceFrom; + basePath: string; + cancelUrl: string; + maxStep: CreateDatasourceStep | ''; + currentStep: CreateDatasourceStep; + agentConfig?: AgentConfig; + packageInfo?: PackageInfo; + restrictWidth?: number; +}> = ({ + from, + basePath, + cancelUrl, + maxStep, + currentStep, + agentConfig, + packageInfo, + restrictWidth, + children, +}) => { + return ( + + + + + + + + +

+ +

+ + + + + + {agentConfig || from === 'config' ? ( + + + + + + + {agentConfig?.name || '-'} + + + + ) : null} + {packageInfo || from === 'package' ? ( + + + + + + + + + + + + {packageInfo?.title || packageInfo?.name || '-'} + + + + + + ) : null} + + + + } + rightColumn={ + + } + > + {children} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/navigation.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/navigation.tsx new file mode 100644 index 00000000000000..099a7a83caa106 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/components/navigation.tsx @@ -0,0 +1,85 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import styled from 'styled-components'; +import { useHistory } from 'react-router-dom'; +import { i18n } from '@kbn/i18n'; +import { EuiStepsHorizontal } from '@elastic/eui'; +import { CreateDatasourceFrom, CreateDatasourceStep } from '../types'; +import { WeightedCreateDatasourceSteps, CREATE_DATASOURCE_STEP_PATHS } from '../constants'; + +const StepsHorizontal = styled(EuiStepsHorizontal)` + background: none; +`; + +export const CreateDatasourceStepsNavigation: React.FunctionComponent<{ + from: CreateDatasourceFrom; + basePath: string; + maxStep: CreateDatasourceStep | ''; + currentStep: CreateDatasourceStep; +}> = ({ from, basePath, maxStep, currentStep }) => { + const history = useHistory(); + + const steps = [ + from === 'config' + ? { + title: i18n.translate('xpack.ingestManager.createDatasource.stepSelectPackageLabel', { + defaultMessage: 'Select package', + }), + isSelected: currentStep === 'selectPackage', + isComplete: + WeightedCreateDatasourceSteps.indexOf('selectPackage') <= + WeightedCreateDatasourceSteps.indexOf(maxStep), + onClick: () => { + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.selectPackage}`); + }, + } + : { + title: i18n.translate('xpack.ingestManager.createDatasource.stepSelectConfigLabel', { + defaultMessage: 'Select configuration', + }), + isSelected: currentStep === 'selectConfig', + isComplete: + WeightedCreateDatasourceSteps.indexOf('selectConfig') <= + WeightedCreateDatasourceSteps.indexOf(maxStep), + onClick: () => { + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.selectConfig}`); + }, + }, + { + title: i18n.translate('xpack.ingestManager.createDatasource.stepConfigureDatasourceLabel', { + defaultMessage: 'Configure data source', + }), + isSelected: currentStep === 'configure', + isComplete: + WeightedCreateDatasourceSteps.indexOf('configure') <= + WeightedCreateDatasourceSteps.indexOf(maxStep), + disabled: + WeightedCreateDatasourceSteps.indexOf(maxStep) < + WeightedCreateDatasourceSteps.indexOf('configure') - 1, + onClick: () => { + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.configure}`); + }, + }, + { + title: i18n.translate('xpack.ingestManager.createDatasource.stepReviewLabel', { + defaultMessage: 'Review', + }), + isSelected: currentStep === 'review', + isComplete: + WeightedCreateDatasourceSteps.indexOf('review') <= + WeightedCreateDatasourceSteps.indexOf(maxStep), + disabled: + WeightedCreateDatasourceSteps.indexOf(maxStep) < + WeightedCreateDatasourceSteps.indexOf('review') - 1, + onClick: () => { + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.review}`); + }, + }, + ]; + + return ; +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/constants.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/constants.ts new file mode 100644 index 00000000000000..eea18179560a10 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/constants.ts @@ -0,0 +1,18 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export const WeightedCreateDatasourceSteps = [ + 'selectConfig', + 'selectPackage', + 'configure', + 'review', +]; + +export const CREATE_DATASOURCE_STEP_PATHS = { + selectConfig: '/select-config', + selectPackage: '/select-package', + configure: '/configure', + review: '/review', +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/index.tsx new file mode 100644 index 00000000000000..23d0f3317a6673 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/index.tsx @@ -0,0 +1,267 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { + useRouteMatch, + HashRouter as Router, + Switch, + Route, + Redirect, + useHistory, +} from 'react-router-dom'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { EuiButtonEmpty } from '@elastic/eui'; +import { AGENT_CONFIG_DETAILS_PATH } from '../../../constants'; +import { AgentConfig, PackageInfo, NewDatasource } from '../../../types'; +import { useLink, sendCreateDatasource } from '../../../hooks'; +import { useLinks as useEPMLinks } from '../../epm/hooks'; +import { CreateDatasourcePageLayout } from './components'; +import { CreateDatasourceFrom, CreateDatasourceStep } from './types'; +import { CREATE_DATASOURCE_STEP_PATHS } from './constants'; +import { StepSelectPackage } from './step_select_package'; +import { StepSelectConfig } from './step_select_config'; +import { StepConfigureDatasource } from './step_configure_datasource'; +import { StepReviewDatasource } from './step_review'; + +export const CreateDatasourcePage: React.FunctionComponent = () => { + const { + params: { configId, pkgkey }, + path: matchPath, + url: basePath, + } = useRouteMatch(); + const history = useHistory(); + const from: CreateDatasourceFrom = configId ? 'config' : 'package'; + const [maxStep, setMaxStep] = useState(''); + const [isSaving, setIsSaving] = useState(false); + + // Agent config and package info states + const [agentConfig, setAgentConfig] = useState(); + const [packageInfo, setPackageInfo] = useState(); + + // New datasource state + const [datasource, setDatasource] = useState({ + name: '', + description: '', + config_id: '', + enabled: true, + output_id: '', // TODO: Blank for now as we only support default output + inputs: [], + }); + + // Update package info method + const updatePackageInfo = (updatedPackageInfo: PackageInfo | undefined) => { + if (updatedPackageInfo) { + setPackageInfo(updatedPackageInfo); + } else { + setPackageInfo(undefined); + setMaxStep(''); + } + + // eslint-disable-next-line no-console + console.debug('Package info updated', updatedPackageInfo); + }; + + // Update agent config method + const updateAgentConfig = (updatedAgentConfig: AgentConfig | undefined) => { + if (updatedAgentConfig) { + setAgentConfig(updatedAgentConfig); + } else { + setAgentConfig(undefined); + setMaxStep(''); + } + + // eslint-disable-next-line no-console + console.debug('Agent config updated', updatedAgentConfig); + }; + + // Update datasource method + const updateDatasource = (updatedFields: Partial) => { + const newDatasource = { + ...datasource, + ...updatedFields, + }; + setDatasource(newDatasource); + + // eslint-disable-next-line no-console + console.debug('Datasource updated', newDatasource); + }; + + // Cancel url + const CONFIG_URL = useLink( + `${AGENT_CONFIG_DETAILS_PATH}${agentConfig ? agentConfig.id : configId}` + ); + const PACKAGE_URL = useEPMLinks().toDetailView({ + name: (pkgkey || '-').split('-')[0], + version: (pkgkey || '-').split('-')[1], + }); + const cancelUrl = from === 'config' ? CONFIG_URL : PACKAGE_URL; + + // Redirect to first step + const redirectToFirstStep = + from === 'config' ? ( + + ) : ( + + ); + + // Url to first and second steps + const SELECT_PACKAGE_URL = useLink(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.selectPackage}`); + const SELECT_CONFIG_URL = useLink(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.selectConfig}`); + const CONFIGURE_DATASOURCE_URL = useLink(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.configure}`); + const firstStepUrl = from === 'config' ? SELECT_PACKAGE_URL : SELECT_CONFIG_URL; + const secondStepUrl = CONFIGURE_DATASOURCE_URL; + + // Redirect to second step + const redirectToSecondStep = ( + + ); + + // Save datasource + const saveDatasource = async () => { + setIsSaving(true); + const result = await sendCreateDatasource(datasource); + setIsSaving(false); + return result; + }; + + const layoutProps = { + from, + basePath, + cancelUrl, + maxStep, + agentConfig, + packageInfo, + restrictWidth: 770, + }; + + return ( + + + {/* Redirect to first step from `/` */} + {from === 'config' ? ( + + ) : ( + + )} + + {/* First step, either render select package or select config depending on entry */} + {from === 'config' ? ( + + + { + setMaxStep('selectPackage'); + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.configure}`); + }} + /> + + + ) : ( + + + { + setMaxStep('selectConfig'); + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.configure}`); + }} + /> + + + )} + + {/* Second step to configure data source, redirect to first step if agent config */} + {/* or package info isn't defined (i.e. after full page reload) */} + + + {!agentConfig || !packageInfo ? ( + redirectToFirstStep + ) : ( + + {from === 'config' ? ( + + ) : ( + + )} + + } + cancelUrl={cancelUrl} + onNext={() => { + setMaxStep('configure'); + history.push(`${basePath}${CREATE_DATASOURCE_STEP_PATHS.review}`); + }} + /> + )} + + + + {/* Third step to review, redirect to second step if data source name is missing */} + {/* (i.e. after full page reload) */} + + + {!agentConfig || !datasource.name ? ( + redirectToSecondStep + ) : ( + + + + } + onSubmit={async () => { + const { error } = await saveDatasource(); + if (!error) { + history.push( + `${AGENT_CONFIG_DETAILS_PATH}${agentConfig ? agentConfig.id : configId}` + ); + } else { + // TODO: Handle save datasource error + } + }} + /> + )} + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_configure_datasource.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_configure_datasource.tsx new file mode 100644 index 00000000000000..484ea3f1d94a0f --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_configure_datasource.tsx @@ -0,0 +1,289 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useEffect, useState, Fragment } from 'react'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiSteps, + EuiPanel, + EuiFlexGrid, + EuiFlexGroup, + EuiFlexItem, + EuiFormRow, + EuiFieldText, + EuiButtonEmpty, + EuiSpacer, + EuiEmptyPrompt, + EuiText, + EuiButton, + EuiComboBox, +} from '@elastic/eui'; +import { + AgentConfig, + PackageInfo, + Datasource, + NewDatasource, + DatasourceInput, +} from '../../../types'; +import { packageToConfigDatasourceInputs } from '../../../services'; +import { DatasourceInputPanel } from './components'; + +export const StepConfigureDatasource: React.FunctionComponent<{ + agentConfig: AgentConfig; + packageInfo: PackageInfo; + datasource: NewDatasource; + updateDatasource: (fields: Partial) => void; + backLink: JSX.Element; + cancelUrl: string; + onNext: () => void; +}> = ({ agentConfig, packageInfo, datasource, updateDatasource, backLink, cancelUrl, onNext }) => { + // Form show/hide states + const [isShowingAdvancedDefine, setIsShowingAdvancedDefine] = useState(false); + + // Update datasource's package and config info + useEffect(() => { + const dsPackage = datasource.package; + const currentPkgKey = dsPackage ? `${dsPackage.name}-${dsPackage.version}` : ''; + const pkgKey = `${packageInfo.name}-${packageInfo.version}`; + + // If package has changed, create shell datasource with input&stream values based on package info + if (currentPkgKey !== pkgKey) { + // Existing datasources on the agent config using the package name, retrieve highest number appended to datasource name + const dsPackageNamePattern = new RegExp(`${packageInfo.name}-(\\d+)`); + const dsWithMatchingNames = (agentConfig.datasources as Datasource[]) + .filter(ds => Boolean(ds.name.match(dsPackageNamePattern))) + .map(ds => parseInt(ds.name.match(dsPackageNamePattern)![1], 10)) + .sort(); + + updateDatasource({ + name: `${packageInfo.name}-${ + dsWithMatchingNames.length ? dsWithMatchingNames[dsWithMatchingNames.length - 1] + 1 : 1 + }`, + package: { + name: packageInfo.name, + title: packageInfo.title, + version: packageInfo.version, + }, + inputs: packageToConfigDatasourceInputs(packageInfo), + }); + } + + // If agent config has changed, update datasource's config ID and namespace + if (datasource.config_id !== agentConfig.id) { + updateDatasource({ + config_id: agentConfig.id, + namespace: agentConfig.namespace, + }); + } + }, [datasource.package, datasource.config_id, agentConfig, packageInfo, updateDatasource]); + + // Step A, define datasource + const DefineDatasource = ( + + + + + } + > + + updateDatasource({ + name: e.target.value, + }) + } + /> + + + + + } + labelAppend={ + + + + } + > + + updateDatasource({ + description: e.target.value, + }) + } + /> + + + + + setIsShowingAdvancedDefine(!isShowingAdvancedDefine)} + > + + + {/* Todo: Populate list of existing namespaces */} + {isShowingAdvancedDefine ? ( + + + + + + } + > + { + updateDatasource({ + namespace: newNamespace, + }); + }} + onChange={(newNamespaces: Array<{ label: string }>) => { + updateDatasource({ + namespace: newNamespaces.length ? newNamespaces[0].label : '', + }); + }} + /> + + + + + ) : null} + + ); + + // Step B, configure inputs (and their streams) + // Assume packages only export one datasource for now + const ConfigureInputs = + packageInfo.datasources && packageInfo.datasources[0] ? ( + + {packageInfo.datasources[0].inputs.map(packageInput => { + const datasourceInput = datasource.inputs.find(input => input.type === packageInput.type); + return datasourceInput ? ( + + ) => { + const indexOfUpdatedInput = datasource.inputs.findIndex( + input => input.type === packageInput.type + ); + const newInputs = [...datasource.inputs]; + newInputs[indexOfUpdatedInput] = { + ...newInputs[indexOfUpdatedInput], + ...updatedInput, + }; + updateDatasource({ + inputs: newInputs, + }); + }} + /> + + ) : null; + })} + + ) : ( + + +

+ +

+ + } + /> + + ); + + return ( + + + + + + {backLink} + + + + + + + + + + + + + + + + onNext()}> + + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_review.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_review.tsx new file mode 100644 index 00000000000000..355bf2febdf5f4 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_review.tsx @@ -0,0 +1,189 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { Fragment, useState, useEffect } from 'react'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiFlexGroup, + EuiFlexItem, + EuiButtonEmpty, + EuiButton, + EuiTitle, + EuiCallOut, + EuiText, + EuiCheckbox, + EuiTabbedContent, + EuiCodeBlock, + EuiSpacer, +} from '@elastic/eui'; +import { dump } from 'js-yaml'; +import { NewDatasource, AgentConfig } from '../../../types'; +import { useConfig, sendGetAgentStatus } from '../../../hooks'; +import { storedDatasourceToAgentDatasource } from '../../../services'; + +export const StepReviewDatasource: React.FunctionComponent<{ + agentConfig: AgentConfig; + datasource: NewDatasource; + backLink: JSX.Element; + cancelUrl: string; + onSubmit: () => void; + isSubmitLoading: boolean; +}> = ({ agentConfig, datasource, backLink, cancelUrl, onSubmit, isSubmitLoading }) => { + // Agent count info states + const [agentCount, setAgentCount] = useState(0); + const [agentCountChecked, setAgentCountChecked] = useState(false); + + // Config information + const { + fleet: { enabled: isFleetEnabled }, + } = useConfig(); + + // Retrieve agent count + useEffect(() => { + const getAgentCount = async () => { + const { data } = await sendGetAgentStatus({ configId: agentConfig.id }); + if (data?.results.total) { + setAgentCount(data.results.total); + } + }; + + if (isFleetEnabled) { + getAgentCount(); + } + }, [agentConfig.id, isFleetEnabled]); + + const showAgentDisclaimer = isFleetEnabled && agentCount; + const fullAgentDatasource = storedDatasourceToAgentDatasource(datasource); + + return ( + + + + + +

+ +

+ + + {backLink} + + + + {/* Agents affected warning */} + {showAgentDisclaimer ? ( + + + } + > + +

+ {agentConfig.name}, + }} + /> +

+ + + + ) : null} + + {/* Preview and YAML view */} + {/* TODO: Implement preview tab */} + + + + + {dump(fullAgentDatasource)} + + + ), + }, + ]} + /> + + + {/* Confirm agents affected */} + {showAgentDisclaimer ? ( + + + + +

+ +

+ + + + + } + checked={agentCountChecked} + onChange={e => setAgentCountChecked(e.target.checked)} + /> + + + + ) : null} + + + + + + + + + + onSubmit()} + isLoading={isSubmitLoading} + disabled={isSubmitLoading || Boolean(showAgentDisclaimer && !agentCountChecked)} + > + + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_select_config.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_select_config.tsx new file mode 100644 index 00000000000000..2ddfc170069a1e --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_select_config.tsx @@ -0,0 +1,259 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useEffect, useState, Fragment } from 'react'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiButtonEmpty, + EuiButton, + EuiFlexGroup, + EuiFlexItem, + EuiTitle, + EuiSelectable, + EuiSpacer, + EuiTextColor, +} from '@elastic/eui'; +import { Error } from '../../../components'; +import { AGENT_CONFIG_PATH } from '../../../constants'; +import { useCapabilities, useLink } from '../../../hooks'; +import { AgentConfig, PackageInfo, GetAgentConfigsResponseItem } from '../../../types'; +import { useGetPackageInfoByKey, useGetAgentConfigs, sendGetOneAgentConfig } from '../../../hooks'; + +export const StepSelectConfig: React.FunctionComponent<{ + pkgkey: string; + updatePackageInfo: (packageInfo: PackageInfo | undefined) => void; + agentConfig: AgentConfig | undefined; + updateAgentConfig: (config: AgentConfig | undefined) => void; + cancelUrl: string; + onNext: () => void; +}> = ({ pkgkey, updatePackageInfo, agentConfig, updateAgentConfig, cancelUrl, onNext }) => { + const hasWriteCapabilites = useCapabilities().write; + // Selected config state + const [selectedConfigId, setSelectedConfigId] = useState( + agentConfig ? agentConfig.id : undefined + ); + const [selectedConfigLoading, setSelectedConfigLoading] = useState(false); + const [selectedConfigError, setSelectedConfigError] = useState(); + + // Todo: replace with create agent config flyout + const CREATE_NEW_CONFIG_URI = useLink(AGENT_CONFIG_PATH); + + // Fetch package info + const { data: packageInfoData, error: packageInfoError } = useGetPackageInfoByKey(pkgkey); + + // Fetch agent configs info + const { + data: agentConfigsData, + error: agentConfigsError, + isLoading: isAgentConfigsLoading, + } = useGetAgentConfigs(); + const agentConfigs = agentConfigsData?.items || []; + const agentConfigsById = agentConfigs.reduce( + (acc: { [key: string]: GetAgentConfigsResponseItem }, config) => { + acc[config.id] = config; + return acc; + }, + {} + ); + + // Update parent package state + useEffect(() => { + if (packageInfoData && packageInfoData.response) { + updatePackageInfo(packageInfoData.response); + } + }, [packageInfoData, updatePackageInfo]); + + // Update parent selected agent config state + useEffect(() => { + const fetchAgentConfigInfo = async () => { + if (selectedConfigId) { + setSelectedConfigLoading(true); + const { data, error } = await sendGetOneAgentConfig(selectedConfigId); + setSelectedConfigLoading(false); + if (error) { + setSelectedConfigError(error); + updateAgentConfig(undefined); + } else if (data && data.item) { + setSelectedConfigError(undefined); + updateAgentConfig(data.item); + } + } else { + setSelectedConfigError(undefined); + updateAgentConfig(undefined); + } + }; + if (!agentConfig || selectedConfigId !== agentConfig.id) { + fetchAgentConfigInfo(); + } + }, [selectedConfigId, agentConfig, updateAgentConfig]); + + // Display package error if there is one + if (packageInfoError) { + return ( + + } + error={packageInfoError} + /> + ); + } + + // Display agent configs list error if there is one + if (agentConfigsError) { + return ( + + } + error={agentConfigsError} + /> + ); + } + + return ( + + + + + +

+ +

+ + + + + + + + + + + { + return { + label: name, + key: id, + checked: selectedConfigId === id ? 'on' : undefined, + }; + })} + renderOption={option => ( + + {option.label} + + + {agentConfigsById[option.key!].description} + + + + + + + + + )} + listProps={{ + bordered: true, + }} + searchProps={{ + placeholder: i18n.translate( + 'xpack.ingestManager.createDatasource.StepSelectConfig.filterAgentConfigsInputPlaceholder', + { + defaultMessage: 'Search for agent configurations', + } + ), + }} + height={240} + onChange={options => { + const selectedOption = options.find(option => option.checked === 'on'); + if (selectedOption) { + setSelectedConfigId(selectedOption.key); + } else { + setSelectedConfigId(undefined); + } + }} + > + {(list, search) => ( + + {search} + + {list} + + )} + + + {/* Display selected agent config error if there is one */} + {selectedConfigError ? ( + + + } + error={selectedConfigError} + /> + + ) : null} + + + + + + + + + onNext()} + > + + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_select_package.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_select_package.tsx new file mode 100644 index 00000000000000..f90e7f0ab0460f --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/step_select_package.tsx @@ -0,0 +1,210 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useEffect, useState, Fragment } from 'react'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiButtonEmpty, + EuiButton, + EuiFlexGroup, + EuiFlexItem, + EuiTitle, + EuiSelectable, + EuiSpacer, +} from '@elastic/eui'; +import { Error } from '../../../components'; +import { AgentConfig, PackageInfo } from '../../../types'; +import { useGetOneAgentConfig, useGetPackages, sendGetPackageInfoByKey } from '../../../hooks'; +import { PackageIcon } from '../../epm/components'; + +export const StepSelectPackage: React.FunctionComponent<{ + agentConfigId: string; + updateAgentConfig: (config: AgentConfig | undefined) => void; + packageInfo?: PackageInfo; + updatePackageInfo: (packageInfo: PackageInfo | undefined) => void; + cancelUrl: string; + onNext: () => void; +}> = ({ agentConfigId, updateAgentConfig, packageInfo, updatePackageInfo, cancelUrl, onNext }) => { + // Selected package state + const [selectedPkgKey, setSelectedPkgKey] = useState( + packageInfo ? `${packageInfo.name}-${packageInfo.version}` : undefined + ); + const [selectedPkgLoading, setSelectedPkgLoading] = useState(false); + const [selectedPkgError, setSelectedPkgError] = useState(); + + // Fetch agent config info + const { data: agentConfigData, error: agentConfigError } = useGetOneAgentConfig(agentConfigId); + + // Fetch packages info + const { + data: packagesData, + error: packagesError, + isLoading: isPackagesLoading, + } = useGetPackages(); + const packages = packagesData?.response || []; + + // Update parent agent config state + useEffect(() => { + if (agentConfigData && agentConfigData.item) { + updateAgentConfig(agentConfigData.item); + } + }, [agentConfigData, updateAgentConfig]); + + // Update parent selected package state + useEffect(() => { + const fetchPackageInfo = async () => { + if (selectedPkgKey) { + setSelectedPkgLoading(true); + const { data, error } = await sendGetPackageInfoByKey(selectedPkgKey); + setSelectedPkgLoading(false); + if (error) { + setSelectedPkgError(error); + updatePackageInfo(undefined); + } else if (data && data.response) { + setSelectedPkgError(undefined); + updatePackageInfo(data.response); + } + } else { + setSelectedPkgError(undefined); + updatePackageInfo(undefined); + } + }; + if (!packageInfo || selectedPkgKey !== `${packageInfo.name}-${packageInfo.version}`) { + fetchPackageInfo(); + } + }, [selectedPkgKey, packageInfo, updatePackageInfo]); + + // Display agent config error if there is one + if (agentConfigError) { + return ( + + } + error={agentConfigError} + /> + ); + } + + // Display packages list error if there is one + if (packagesError) { + return ( + + } + error={packagesError} + /> + ); + } + + return ( + + + +

+ +

+ + + + { + const pkgkey = `${name}-${version}`; + return { + label: title || name, + key: pkgkey, + prepend: , + checked: selectedPkgKey === pkgkey ? 'on' : undefined, + }; + })} + listProps={{ + bordered: true, + }} + searchProps={{ + placeholder: i18n.translate( + 'xpack.ingestManager.createDatasource.stepSelectPackage.filterPackagesInputPlaceholder', + { + defaultMessage: 'Search for packages', + } + ), + }} + height={240} + onChange={options => { + const selectedOption = options.find(option => option.checked === 'on'); + if (selectedOption) { + setSelectedPkgKey(selectedOption.key); + } else { + setSelectedPkgKey(undefined); + } + }} + > + {(list, search) => ( + + {search} + + {list} + + )} + + + {/* Display selected package error if there is one */} + {selectedPkgError ? ( + + + } + error={selectedPkgError} + /> + + ) : null} + + + + + + + + + onNext()} + > + + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/types.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/types.ts new file mode 100644 index 00000000000000..bd05be2d8a558d --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/create_datasource_page/types.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export type CreateDatasourceFrom = 'package' | 'config'; +export type CreateDatasourceStep = 'selectConfig' | 'selectPackage' | 'configure' | 'review'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/config_form.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/config_form.tsx new file mode 100644 index 00000000000000..c4f8d944ceb142 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/config_form.tsx @@ -0,0 +1,94 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useState } from 'react'; +import { EuiFieldText, EuiForm, EuiFormRow } from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { AgentConfig } from '../../../../types'; + +interface ValidationResults { + [key: string]: JSX.Element[]; +} + +export const configFormValidation = (config: Partial): ValidationResults => { + const errors: ValidationResults = {}; + + if (!config.name?.trim()) { + errors.name = [ + , + ]; + } + + return errors; +}; + +interface Props { + config: Partial; + updateConfig: (u: Partial) => void; + validation: ValidationResults; +} + +export const ConfigForm: React.FunctionComponent = ({ + config, + updateConfig, + validation, +}) => { + const [touchedFields, setTouchedFields] = useState<{ [key: string]: boolean }>({}); + const fields: Array<{ name: 'name' | 'description' | 'namespace'; label: JSX.Element }> = [ + { + name: 'name', + label: ( + + ), + }, + { + name: 'description', + label: ( + + ), + }, + { + name: 'namespace', + label: ( + + ), + }, + ]; + + return ( + + {fields.map(({ name, label }) => { + return ( + + updateConfig({ [name]: e.target.value })} + isInvalid={Boolean(touchedFields[name] && validation[name])} + onBlur={() => setTouchedFields({ ...touchedFields, [name]: true })} + /> + + ); + })} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/datasources_table.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/datasources_table.tsx new file mode 100644 index 00000000000000..3c982747e1d226 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/datasources_table.tsx @@ -0,0 +1,131 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { EuiInMemoryTable, EuiInMemoryTableProps, EuiBadge } from '@elastic/eui'; +import { Datasource } from '../../../../types'; + +type DatasourceWithConfig = Datasource & { configs?: string[] }; + +interface InMemoryDatasource { + id: string; + name: string; + streams: number; + packageName?: string; + packageTitle?: string; + packageVersion?: string; + configs: number; +} + +interface Props { + datasources?: DatasourceWithConfig[]; + withConfigsCount?: boolean; + loading?: EuiInMemoryTableProps['loading']; + message?: EuiInMemoryTableProps['message']; + search?: EuiInMemoryTableProps['search']; + selection?: EuiInMemoryTableProps['selection']; + isSelectable?: EuiInMemoryTableProps['isSelectable']; +} + +export const DatasourcesTable: React.FunctionComponent = ( + { datasources: originalDatasources, withConfigsCount, ...rest } = { + datasources: [], + withConfigsCount: false, + } +) => { + // Flatten some values so that they can be searched via in-memory table search + const datasources = + originalDatasources?.map(({ id, name, inputs, package: datasourcePackage, configs }) => ({ + id, + name, + streams: inputs.reduce( + (streamsCount, input) => + streamsCount + + (input.enabled ? input.streams.filter(stream => stream.enabled).length : 0), + 0 + ), + packageName: datasourcePackage?.name, + packageTitle: datasourcePackage?.title, + packageVersion: datasourcePackage?.version, + configs: configs?.length || 0, + })) || []; + + const columns: EuiInMemoryTableProps['columns'] = [ + { + field: 'name', + name: i18n.translate('xpack.ingestManager.configDetails.datasourcesTable.nameColumnTitle', { + defaultMessage: 'Name', + }), + }, + { + field: 'packageTitle', + name: i18n.translate( + 'xpack.ingestManager.configDetails.datasourcesTable.packageNameColumnTitle', + { + defaultMessage: 'Package', + } + ), + }, + { + field: 'packageVersion', + name: i18n.translate( + 'xpack.ingestManager.configDetails.datasourcesTable.packageVersionColumnTitle', + { + defaultMessage: 'Version', + } + ), + }, + { + field: 'streams', + name: i18n.translate( + 'xpack.ingestManager.configDetails.datasourcesTable.streamsCountColumnTitle', + { + defaultMessage: 'Streams', + } + ), + }, + ]; + + if (withConfigsCount) { + columns.splice(columns.length - 1, 0, { + field: 'configs', + name: i18n.translate( + 'xpack.ingestManager.configDetails.datasourcesTable.configsColumnTitle', + { + defaultMessage: 'Configs', + } + ), + render: (configs: number) => { + return configs === 0 ? ( + + + + ) : ( + configs + ); + }, + }); + } + + return ( + + itemId="id" + items={datasources || ([] as InMemoryDatasource[])} + columns={columns} + sorting={{ + sort: { + field: 'name', + direction: 'asc', + }, + }} + {...rest} + /> + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/donut_chart.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/donut_chart.tsx new file mode 100644 index 00000000000000..408ccc6e951f68 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/donut_chart.tsx @@ -0,0 +1,65 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useEffect, useRef } from 'react'; +import d3 from 'd3'; +import { EuiFlexItem } from '@elastic/eui'; + +interface DonutChartProps { + data: { + [key: string]: number; + }; + height: number; + width: number; +} + +export const DonutChart = ({ height, width, data }: DonutChartProps) => { + const chartElement = useRef(null); + + useEffect(() => { + if (chartElement.current !== null) { + // we must remove any existing paths before painting + d3.selectAll('g').remove(); + const svgElement = d3 + .select(chartElement.current) + .append('g') + .attr('transform', `translate(${width / 2}, ${height / 2})`); + const color = d3.scale + .ordinal() + // @ts-ignore + .domain(data) + .range(['#017D73', '#98A2B3', '#BD271E']); + const pieGenerator = d3.layout + .pie() + .value(({ value }: any) => value) + // these start/end angles will reverse the direction of the pie, + // which matches our design + .startAngle(2 * Math.PI) + .endAngle(0); + + svgElement + .selectAll('g') + // @ts-ignore + .data(pieGenerator(d3.entries(data))) + .enter() + .append('path') + .attr( + 'd', + // @ts-ignore attr does not expect a param of type Arc but it behaves as desired + d3.svg + .arc() + .innerRadius(width * 0.28) + .outerRadius(Math.min(width, height) / 2 - 10) + ) + .attr('fill', (d: any) => color(d.data.key)); + } + }, [data, height, width]); + return ( + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/edit_config.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/edit_config.tsx new file mode 100644 index 00000000000000..65eb86d7d871f2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/edit_config.tsx @@ -0,0 +1,135 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { + EuiFlyout, + EuiFlyoutHeader, + EuiTitle, + EuiFlyoutBody, + EuiFlyoutFooter, + EuiFlexGroup, + EuiFlexItem, + EuiButtonEmpty, + EuiButton, +} from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { useCore, sendRequest } from '../../../../hooks'; +import { agentConfigRouteService } from '../../../../services'; +import { AgentConfig } from '../../../../types'; +import { ConfigForm, configFormValidation } from './config_form'; + +interface Props { + agentConfig: AgentConfig; + onClose: () => void; +} + +export const EditConfigFlyout: React.FunctionComponent = ({ + agentConfig: originalAgentConfig, + onClose, +}) => { + const { notifications } = useCore(); + const [config, setConfig] = useState>({ + name: originalAgentConfig.name, + description: originalAgentConfig.description, + }); + const [isLoading, setIsLoading] = useState(false); + const updateConfig = (updatedFields: Partial) => { + setConfig({ + ...config, + ...updatedFields, + }); + }; + const validation = configFormValidation(config); + + const header = ( + + +

+ +

+ + + ); + + const body = ( + + + + ); + + const footer = ( + + + + + + + + + 0} + onClick={async () => { + setIsLoading(true); + try { + const { error } = await sendRequest({ + path: agentConfigRouteService.getUpdatePath(originalAgentConfig.id), + method: 'put', + body: JSON.stringify(config), + }); + if (!error) { + notifications.toasts.addSuccess( + i18n.translate('xpack.ingestManager.editConfig.successNotificationTitle', { + defaultMessage: "Agent config '{name}' updated", + values: { name: config.name }, + }) + ); + } else { + notifications.toasts.addDanger( + error + ? error.message + : i18n.translate('xpack.ingestManager.editConfig.errorNotificationTitle', { + defaultMessage: 'Unable to update agent config', + }) + ); + } + } catch (e) { + notifications.toasts.addDanger( + i18n.translate('xpack.ingestManager.editConfig.errorNotificationTitle', { + defaultMessage: 'Unable to update agent config', + }) + ); + } + setIsLoading(false); + onClose(); + }} + > + + + + + + ); + + return ( + + {header} + {body} + {footer} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/index.ts new file mode 100644 index 00000000000000..51834268ffa5b1 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/components/index.ts @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export { DatasourcesTable } from './datasources_table'; +export { DonutChart } from './donut_chart'; +export { EditConfigFlyout } from './edit_config'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/constants.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/constants.ts new file mode 100644 index 00000000000000..787791f985c7da --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/constants.ts @@ -0,0 +1,10 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { AGENT_CONFIG_DETAILS_PATH } from '../../../constants'; + +export const DETAILS_ROUTER_PATH = `${AGENT_CONFIG_DETAILS_PATH}:configId`; +export const DETAILS_ROUTER_SUB_PATH = `${DETAILS_ROUTER_PATH}/:tabId`; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/index.ts new file mode 100644 index 00000000000000..19be93676a7346 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/index.ts @@ -0,0 +1,7 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export { useGetAgentStatus, AgentStatusRefreshContext } from './use_agent_status'; +export { ConfigRefreshContext } from './use_config'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_agent_status.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_agent_status.tsx new file mode 100644 index 00000000000000..214deb81f535c8 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_agent_status.tsx @@ -0,0 +1,36 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { useRequest } from '../../../../hooks'; +import { GetAgentStatusResponse } from '../../../../types'; +import { agentRouteService } from '../../../../services'; +import { UseRequestConfig } from '../../../../hooks/use_request/use_request'; + +type RequestOptions = Pick, 'pollIntervalMs'>; + +export function useGetAgentStatus(configId?: string, options?: RequestOptions) { + const agentStatusRequest = useRequest({ + path: agentRouteService.getStatusPath(), + query: { + configId, + }, + method: 'get', + ...options, + }); + + return { + isLoading: agentStatusRequest.isLoading, + data: agentStatusRequest.data, + error: agentStatusRequest.error, + refreshAgentStatus: () => agentStatusRequest.sendRequest, + }; +} + +export const AgentStatusRefreshContext = React.createContext({ refresh: () => {} }); + +export function useAgentStatusRefresh() { + return React.useContext(AgentStatusRefreshContext).refresh; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_config.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_config.tsx new file mode 100644 index 00000000000000..b61986e7abb4fd --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_config.tsx @@ -0,0 +1,12 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; + +export const ConfigRefreshContext = React.createContext({ refresh: () => {} }); + +export function useConfigRefresh() { + return React.useContext(ConfigRefreshContext).refresh; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_details_uri.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_details_uri.ts new file mode 100644 index 00000000000000..df43d8e908e41f --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/hooks/use_details_uri.ts @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { useMemo } from 'react'; +import { generatePath } from 'react-router-dom'; +import { useLink } from '../../../../hooks'; +import { AGENT_CONFIG_PATH } from '../../../../constants'; +import { DETAILS_ROUTER_PATH, DETAILS_ROUTER_SUB_PATH } from '../constants'; + +export const useDetailsUri = (configId: string) => { + const BASE_URI = useLink(''); + return useMemo(() => { + const AGENT_CONFIG_DETAILS = `${BASE_URI}${generatePath(DETAILS_ROUTER_PATH, { configId })}`; + + return { + ADD_DATASOURCE: `${AGENT_CONFIG_DETAILS}/add-datasource`, + AGENT_CONFIG_LIST: `${BASE_URI}${AGENT_CONFIG_PATH}`, + AGENT_CONFIG_DETAILS, + AGENT_CONFIG_DETAILS_YAML: `${BASE_URI}${generatePath(DETAILS_ROUTER_SUB_PATH, { + configId, + tabId: 'yaml', + })}`, + AGENT_CONFIG_DETAILS_SETTINGS: `${BASE_URI}${generatePath(DETAILS_ROUTER_SUB_PATH, { + configId, + tabId: 'settings', + })}`, + }; + }, [BASE_URI, configId]); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/index.tsx new file mode 100644 index 00000000000000..6f72977cb333f1 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/details_page/index.tsx @@ -0,0 +1,365 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { Fragment, memo, useCallback, useMemo, useState } from 'react'; +import { Redirect, useRouteMatch, Switch, Route } from 'react-router-dom'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage, FormattedDate } from '@kbn/i18n/react'; +import { + EuiFlexGroup, + EuiFlexItem, + EuiCallOut, + EuiText, + EuiSpacer, + EuiTitle, + EuiButton, + EuiButtonEmpty, + EuiEmptyPrompt, + EuiI18nNumber, + EuiDescriptionList, + EuiDescriptionListTitle, + EuiDescriptionListDescription, +} from '@elastic/eui'; +import { Props as EuiTabProps } from '@elastic/eui/src/components/tabs/tab'; +import styled from 'styled-components'; +import { useCapabilities, useGetOneAgentConfig } from '../../../hooks'; +import { Datasource } from '../../../types'; +import { Loading } from '../../../components'; +import { WithHeaderLayout } from '../../../layouts'; +import { ConfigRefreshContext, useGetAgentStatus, AgentStatusRefreshContext } from './hooks'; +import { DatasourcesTable, EditConfigFlyout } from './components'; +import { LinkedAgentCount } from '../components'; +import { useDetailsUri } from './hooks/use_details_uri'; +import { DETAILS_ROUTER_PATH, DETAILS_ROUTER_SUB_PATH } from './constants'; + +const Divider = styled.div` + width: 0; + height: 100%; + border-left: ${props => props.theme.eui.euiBorderThin}; +`; + +export const AgentConfigDetailsPage = memo(() => { + return ( + + + + + + + + + ); +}); + +export const AgentConfigDetailsLayout: React.FunctionComponent = () => { + const { + params: { configId, tabId = '' }, + } = useRouteMatch<{ configId: string; tabId?: string }>(); + const hasWriteCapabilites = useCapabilities().write; + const agentConfigRequest = useGetOneAgentConfig(configId); + const agentConfig = agentConfigRequest.data ? agentConfigRequest.data.item : null; + const { isLoading, error, sendRequest: refreshAgentConfig } = agentConfigRequest; + const [redirectToAgentConfigList] = useState(false); + const agentStatusRequest = useGetAgentStatus(configId); + const { refreshAgentStatus } = agentStatusRequest; + const agentStatus = agentStatusRequest.data?.results; + const URI = useDetailsUri(configId); + + // Flyout states + const [isEditConfigFlyoutOpen, setIsEditConfigFlyoutOpen] = useState(false); + + const refreshData = useCallback(() => { + refreshAgentConfig(); + refreshAgentStatus(); + }, [refreshAgentConfig, refreshAgentStatus]); + + const headerLeftContent = useMemo( + () => ( + + + + + +
+ + + +
+ +

+ {(agentConfig && agentConfig.name) || ( + + )} +

+ + + + {agentConfig && agentConfig.description ? ( + + + + {agentConfig.description} + + + ) : null} + + + + + ), + [URI.AGENT_CONFIG_LIST, agentConfig, configId] + ); + + const headerRightContent = useMemo( + () => ( + + {[ + { + label: i18n.translate('xpack.ingestManager.configDetails.summary.revision', { + defaultMessage: 'Revision', + }), + content: '999', // FIXME: implement version - see: https://github.com/elastic/kibana/issues/56750 + }, + { isDivider: true }, + { + label: i18n.translate('xpack.ingestManager.configDetails.summary.datasources', { + defaultMessage: 'Data sources', + }), + content: ( + + ), + }, + { isDivider: true }, + { + label: i18n.translate('xpack.ingestManager.configDetails.summary.usedBy', { + defaultMessage: 'Used by', + }), + content: ( + + ), + }, + { isDivider: true }, + { + label: i18n.translate('xpack.ingestManager.configDetails.summary.lastUpdated', { + defaultMessage: 'Last updated on', + }), + content: + (agentConfig && ( + + )) || + '', + }, + ].map((item, index) => ( + + {item.isDivider ?? false ? ( + + ) : ( + + {item.label} + {item.content} + + )} + + ))} + + ), + [agentConfig, agentStatus] + ); + + const headerTabs = useMemo(() => { + return [ + { + id: 'datasources', + name: i18n.translate('xpack.ingestManager.configDetails.subTabs.datasouces', { + defaultMessage: 'Data sources', + }), + href: URI.AGENT_CONFIG_DETAILS, + isSelected: tabId === '', + }, + { + id: 'yaml', + name: i18n.translate('xpack.ingestManager.configDetails.subTabs.yamlFile', { + defaultMessage: 'YAML File', + }), + href: URI.AGENT_CONFIG_DETAILS_YAML, + isSelected: tabId === 'yaml', + }, + { + id: 'settings', + name: i18n.translate('xpack.ingestManager.configDetails.subTabs.settings', { + defaultMessage: 'Settings', + }), + href: URI.AGENT_CONFIG_DETAILS_SETTINGS, + isSelected: tabId === 'settings', + }, + ]; + }, [ + URI.AGENT_CONFIG_DETAILS, + URI.AGENT_CONFIG_DETAILS_SETTINGS, + URI.AGENT_CONFIG_DETAILS_YAML, + tabId, + ]); + + if (redirectToAgentConfigList) { + return ; + } + + if (isLoading) { + return ; + } + + if (error) { + return ( + + +

+ {error.message} +

+ + + ); + } + + if (!agentConfig) { + return ( + + + + ); + } + + return ( + + + + {isEditConfigFlyoutOpen ? ( + { + setIsEditConfigFlyoutOpen(false); + refreshData(); + }} + agentConfig={agentConfig} + /> + ) : null} + + + { + // TODO: YAML implementation tracked via https://github.com/elastic/kibana/issues/57958 + return
YAML placeholder
; + }} + /> + { + // TODO: Settings implementation tracked via: https://github.com/elastic/kibana/issues/57959 + return
Settings placeholder
; + }} + /> + { + return ( + + + + } + actions={ + + + + } + /> + ) : null + } + search={{ + toolsRight: [ + + + , + ], + box: { + incremental: true, + schema: true, + }, + }} + isSelectable={false} + /> + ); + }} + /> + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/index.tsx index c80c4496198beb..71ada155373bfb 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/index.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/index.tsx @@ -6,10 +6,18 @@ import React from 'react'; import { HashRouter as Router, Switch, Route } from 'react-router-dom'; import { AgentConfigListPage } from './list_page'; +import { AgentConfigDetailsPage } from './details_page'; +import { CreateDatasourcePage } from './create_datasource_page'; export const AgentConfigApp: React.FunctionComponent = () => ( + + + + + + diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/components/create_config.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/components/create_config.tsx index c6fea7b22bcd18..2373d6ad2ad176 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/components/create_config.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/components/create_config.tsx @@ -16,9 +16,10 @@ import { EuiFlexItem, EuiButtonEmpty, EuiButton, + EuiText, } from '@elastic/eui'; import { NewAgentConfig } from '../../../../types'; -import { useCore, sendCreateAgentConfig } from '../../../../hooks'; +import { useCapabilities, useCore, sendCreateAgentConfig } from '../../../../hooks'; import { AgentConfigForm, agentConfigFormValidation } from '../../components'; interface Props { @@ -27,11 +28,12 @@ interface Props { export const CreateAgentConfigFlyout: React.FunctionComponent = ({ onClose }) => { const { notifications } = useCore(); - + const hasWriteCapabilites = useCapabilities().write; const [agentConfig, setAgentConfig] = useState({ name: '', description: '', namespace: '', + is_default: undefined, }); const [isLoading, setIsLoading] = useState(false); const validation = agentConfigFormValidation(agentConfig); @@ -53,10 +55,16 @@ export const CreateAgentConfigFlyout: React.FunctionComponent = ({ onClos

+ + + ); @@ -85,7 +93,7 @@ export const CreateAgentConfigFlyout: React.FunctionComponent = ({ onClos 0} + isDisabled={!hasWriteCapabilites || isLoading || Object.keys(validation).length > 0} onClick={async () => { setIsLoading(true); try { @@ -125,7 +133,7 @@ export const CreateAgentConfigFlyout: React.FunctionComponent = ({ onClos > @@ -134,7 +142,7 @@ export const CreateAgentConfigFlyout: React.FunctionComponent = ({ onClos ); return ( - + {header} {body} {footer} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/index.tsx index ef5a38d4869015..35915fab6f1434 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/index.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_config/list_page/index.tsx @@ -3,7 +3,7 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import React, { useState } from 'react'; +import React, { CSSProperties, useCallback, useEffect, useMemo, useState } from 'react'; import { EuiSpacer, EuiText, @@ -11,21 +11,45 @@ import { EuiFlexItem, EuiButton, EuiEmptyPrompt, - // @ts-ignore - EuiSearchBar, EuiBasicTable, EuiLink, - EuiBadge, + EuiTableActionsColumnType, + EuiTableFieldDataColumnType, + EuiTextColor, + EuiPopover, + EuiContextMenuPanel, + EuiContextMenuItem, + EuiButtonIcon, } from '@elastic/eui'; import { i18n } from '@kbn/i18n'; -import { FormattedMessage } from '@kbn/i18n/react'; +import { FormattedMessage, FormattedDate } from '@kbn/i18n/react'; +import styled from 'styled-components'; +import { useHistory } from 'react-router-dom'; import { AgentConfig } from '../../../types'; -import { DEFAULT_AGENT_CONFIG_ID, AGENT_CONFIG_DETAILS_PATH } from '../../../constants'; +import { + AGENT_CONFIG_DETAILS_PATH, + AGENT_CONFIG_SAVED_OBJECT_TYPE, + AGENT_CONFIG_PATH, +} from '../../../constants'; import { WithHeaderLayout } from '../../../layouts'; -// import { SearchBar } from '../../../components'; -import { useGetAgentConfigs, usePagination, useLink } from '../../../hooks'; +import { + useCapabilities, + useGetAgentConfigs, + usePagination, + useLink, + useConfig, + useUrlParams, +} from '../../../hooks'; import { AgentConfigDeleteProvider } from '../components'; import { CreateAgentConfigFlyout } from './components'; +import { SearchBar } from '../../../components/search_bar'; +import { LinkedAgentCount } from '../components'; + +const NO_WRAP_TRUNCATE_STYLE: CSSProperties = Object.freeze({ + overflow: 'hidden', + textOverflow: 'ellipsis', + whiteSpace: 'nowrap', +}); const AgentConfigListPageLayout: React.FunctionComponent = ({ children }) => ( ( ); +const DangerEuiContextMenuItem = styled(EuiContextMenuItem)` + color: ${props => props.theme.eui.textColors.danger}; +`; + +const RowActions = React.memo<{ config: AgentConfig; onDelete: () => void }>( + ({ config, onDelete }) => { + const hasWriteCapabilites = useCapabilities().write; + const DETAILS_URI = useLink(`${AGENT_CONFIG_DETAILS_PATH}${config.id}`); + const ADD_DATASOURCE_URI = `${DETAILS_URI}/add-datasource`; + + const [isOpen, setIsOpen] = useState(false); + const handleCloseMenu = useCallback(() => setIsOpen(false), [setIsOpen]); + const handleToggleMenu = useCallback(() => setIsOpen(!isOpen), [isOpen]); + + return ( + + } + isOpen={isOpen} + closePopover={handleCloseMenu} + > + + + , + + + + , + + + + , + + + {deleteAgentConfigsPrompt => { + return ( + deleteAgentConfigsPrompt([config.id], onDelete)} + > + + + ); + }} + , + ]} + /> + + ); + } +); + export const AgentConfigListPage: React.FunctionComponent<{}> = () => { - // Create agent config flyout state - const [isCreateAgentConfigFlyoutOpen, setIsCreateAgentConfigFlyoutOpen] = useState( - false - ); + // Config information + const hasWriteCapabilites = useCapabilities().write; + const { + fleet: { enabled: isFleetEnabled }, + } = useConfig(); + + // Base URL paths + const DETAILS_URI = useLink(AGENT_CONFIG_DETAILS_PATH); // Table and search states const [search, setSearch] = useState(''); - const { pagination, setPagination } = usePagination(); + const { pagination, pageSizeOptions, setPagination } = usePagination(); const [selectedAgentConfigs, setSelectedAgentConfigs] = useState([]); + const history = useHistory(); + const { urlParams, toUrlParams } = useUrlParams(); + const isCreateAgentConfigFlyoutOpen = 'create' in urlParams; + const setIsCreateAgentConfigFlyoutOpen = useCallback( + (isOpen: boolean) => { + if (isOpen !== isCreateAgentConfigFlyoutOpen) { + if (isOpen) { + history.push(`${AGENT_CONFIG_PATH}?${toUrlParams({ ...urlParams, create: null })}`); + } else { + const { create, ...params } = urlParams; + history.push(`${AGENT_CONFIG_PATH}?${toUrlParams(params)}`); + } + } + }, + [history, isCreateAgentConfigFlyoutOpen, toUrlParams, urlParams] + ); // Fetch agent configs - const { isLoading, data: agentConfigData, sendRequest } = useGetAgentConfigs(); + const { isLoading, data: agentConfigData, sendRequest } = useGetAgentConfigs({ + page: pagination.currentPage, + perPage: pagination.pageSize, + kuery: search, + }); - // Base path for config details - const DETAILS_URI = useLink(AGENT_CONFIG_DETAILS_PATH); + // If `kuery` url param changes trigger a search + useEffect(() => { + const kuery = Array.isArray(urlParams.kuery) + ? urlParams.kuery[urlParams.kuery.length - 1] + : urlParams.kuery ?? ''; + if (kuery !== search) { + setSearch(kuery); + } + }, [search, urlParams]); // Some configs retrieved, set up table props - const columns = [ - { - field: 'name', - name: i18n.translate('xpack.ingestManager.agentConfigList.nameColumnTitle', { - defaultMessage: 'Name', - }), - render: (name: string, agentConfig: AgentConfig) => name || agentConfig.id, - }, - { - field: 'namespace', - name: i18n.translate('xpack.ingestManager.agentConfigList.namespaceColumnTitle', { - defaultMessage: 'Namespace', - }), - render: (namespace: string) => (namespace ? {namespace} : null), - }, - { - field: 'description', - name: i18n.translate('xpack.ingestManager.agentConfigList.descriptionColumnTitle', { - defaultMessage: 'Description', - }), - }, - { - field: 'datasources', - name: i18n.translate('xpack.ingestManager.agentConfigList.datasourcesCountColumnTitle', { - defaultMessage: 'Datasources', - }), - render: (datasources: AgentConfig['datasources']) => (datasources ? datasources.length : 0), - }, - { - name: i18n.translate('xpack.ingestManager.agentConfigList.actionsColumnTitle', { - defaultMessage: 'Actions', - }), - actions: [ - { - render: ({ id }: AgentConfig) => { - return ( - + const columns = useMemo(() => { + const cols: Array< + EuiTableFieldDataColumnType | EuiTableActionsColumnType + > = [ + { + field: 'name', + name: i18n.translate('xpack.ingestManager.agentConfigList.nameColumnTitle', { + defaultMessage: 'Name', + }), + width: '20%', + // FIXME: use version once available - see: https://github.com/elastic/kibana/issues/56750 + render: (name: string, agentConfig: AgentConfig) => ( + + + + {name || agentConfig.id} + + + + - - ); + + + + ), + }, + { + field: 'description', + name: i18n.translate('xpack.ingestManager.agentConfigList.descriptionColumnTitle', { + defaultMessage: 'Description', + }), + width: '35%', + truncateText: true, + render: (description: AgentConfig['description']) => ( + + {description} + + ), + }, + { + field: 'updated_on', + name: i18n.translate('xpack.ingestManager.agentConfigList.updatedOnColumnTitle', { + defaultMessage: 'Last updated on', + }), + render: (date: AgentConfig['updated_on']) => ( + + ), + }, + { + field: 'agents', + name: i18n.translate('xpack.ingestManager.agentConfigList.agentsColumnTitle', { + defaultMessage: 'Agents', + }), + dataType: 'number', + render: (agents: number, config: AgentConfig) => ( + + ), + }, + { + field: 'datasources', + name: i18n.translate('xpack.ingestManager.agentConfigList.datasourcesCountColumnTitle', { + defaultMessage: 'Data sources', + }), + dataType: 'number', + render: (datasources: AgentConfig['datasources']) => (datasources ? datasources.length : 0), + }, + { + name: i18n.translate('xpack.ingestManager.agentConfigList.actionsColumnTitle', { + defaultMessage: 'Actions', + }), + actions: [ + { + render: (config: AgentConfig) => ( + sendRequest()} /> + ), }, - }, - ], - width: '100px', - }, - ]; + ], + }, + ]; - const emptyPrompt = ( - - - - } - actions={ - setIsCreateAgentConfigFlyoutOpen(true)} - > - - - } - /> + // If Fleet is not enabled, then remove the `agents` column + if (!isFleetEnabled) { + return cols.filter(col => ('field' in col ? col.field !== 'agents' : true)); + } + + return cols; + }, [DETAILS_URI, isFleetEnabled, sendRequest]); + + const createAgentConfigButton = useMemo( + () => ( + setIsCreateAgentConfigFlyoutOpen(true)} + > + + + ), + [hasWriteCapabilites, setIsCreateAgentConfigFlyoutOpen] + ); + + const emptyPrompt = useMemo( + () => ( + + + + } + actions={hasWriteCapabilites ?? createAgentConfigButton} + /> + ), + [hasWriteCapabilites, createAgentConfigButton] ); return ( @@ -191,50 +382,40 @@ export const AgentConfigListPage: React.FunctionComponent<{}> = () => { ) : null} - {/* { - setPagination({ - ...pagination, - currentPage: 1, - }); - setSearch(newSearch); - }} - fieldPrefix={AGENT_CONFIG_SAVED_OBJECT_TYPE} - /> */} + { + setPagination({ + ...pagination, + currentPage: 1, + }); + setSearch(newSearch); + }} + fieldPrefix={AGENT_CONFIG_SAVED_OBJECT_TYPE} + /> - sendRequest()}> + sendRequest()}> - - setIsCreateAgentConfigFlyoutOpen(true)} - > - - - + {createAgentConfigButton} - ) : !search.trim() && agentConfigData?.total === 0 ? ( + ) : !search.trim() && (agentConfigData?.total ?? 0) === 0 ? ( emptyPrompt ) : ( = () => { columns={columns} isSelectable={true} selection={{ - selectable: (agentConfig: AgentConfig) => agentConfig.id !== DEFAULT_AGENT_CONFIG_ID, + selectable: (agentConfig: AgentConfig) => !agentConfig.is_default, onSelectionChange: (newSelectedAgentConfigs: AgentConfig[]) => { setSelectedAgentConfigs(newSelectedAgentConfigs); }, @@ -267,6 +448,7 @@ export const AgentConfigListPage: React.FunctionComponent<{}> = () => { pageIndex: pagination.currentPage - 1, pageSize: pagination.pageSize, totalItemCount: agentConfigData ? agentConfigData.total : 0, + pageSizeOptions, }} onChange={({ page }: { page: { index: number; size: number } }) => { const newPagination = { @@ -275,7 +457,6 @@ export const AgentConfigListPage: React.FunctionComponent<{}> = () => { pageSize: page.size, }; setPagination(newPagination); - sendRequest(); // todo: fix this to send pagination options }} /> diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/assets_facet_group.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/assets_facet_group.tsx new file mode 100644 index 00000000000000..219896dd27ef7b --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/assets_facet_group.tsx @@ -0,0 +1,102 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + EuiFacetButton, + EuiFacetGroup, + EuiFlexGroup, + EuiFlexItem, + EuiIcon, + EuiText, + EuiTextColor, + EuiTitle, +} from '@elastic/eui'; +import React, { Fragment } from 'react'; +import styled from 'styled-components'; +import { + AssetsGroupedByServiceByType, + AssetTypeToParts, + KibanaAssetType, + entries, +} from '../../../types'; +import { + AssetIcons, + AssetTitleMap, + DisplayedAssets, + ServiceIcons, + ServiceTitleMap, +} from '../constants'; + +export function AssetsFacetGroup({ assets }: { assets: AssetsGroupedByServiceByType }) { + const FirstHeaderRow = styled(EuiFlexGroup)` + padding: 0 0 ${props => props.theme.eui.paddingSizes.m} 0; + `; + + const HeaderRow = styled(EuiFlexGroup)` + padding: ${props => props.theme.eui.paddingSizes.m} 0; + `; + + const FacetGroup = styled(EuiFacetGroup)` + flex-grow: 0; + `; + + return ( + + {entries(assets).map(([service, typeToParts], index) => { + const Header = index === 0 ? FirstHeaderRow : HeaderRow; + // filter out assets we are not going to display + const filteredTypes: AssetTypeToParts = entries(typeToParts).reduce( + (acc: any, [asset, value]) => { + if (DisplayedAssets[service].includes(asset)) acc[asset] = value; + return acc; + }, + {} + ); + return ( + +
+ + + + + + + +

{ServiceTitleMap[service]} Assets

+ + + +
+ + + {entries(filteredTypes).map(([_type, parts]) => { + const type = _type as KibanaAssetType; + // only kibana assets have icons + const iconType = type in AssetIcons && AssetIcons[type]; + const iconNode = iconType ? : ''; + const FacetButton = styled(EuiFacetButton)` + padding: '${props => props.theme.eui.paddingSizes.xs} 0'; + height: 'unset'; + `; + return ( + {}} + > + {AssetTitleMap[type]} + + ); + })} + + + ); + })} + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/icon_panel.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/icon_panel.tsx new file mode 100644 index 00000000000000..7ce386ed56f5f2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/icon_panel.tsx @@ -0,0 +1,31 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiIcon, EuiPanel, IconType } from '@elastic/eui'; +import React from 'react'; +import styled from 'styled-components'; + +export function IconPanel({ iconType }: { iconType: IconType }) { + const Panel = styled(EuiPanel)` + /* 🤢🤷 https://www.styled-components.com/docs/faqs#how-can-i-override-styles-with-higher-specificity */ + &&& { + position: absolute; + text-align: center; + vertical-align: middle; + padding: ${props => props.theme.eui.spacerSizes.xl}; + svg { + height: ${props => props.theme.eui.euiKeyPadMenuSize}; + width: ${props => props.theme.eui.euiKeyPadMenuSize}; + } + } + `; + + return ( + + + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/index.ts new file mode 100644 index 00000000000000..2cb940e2ff40c5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/index.ts @@ -0,0 +1,7 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export { PackageIcon } from './package_icon'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/nav_button_back.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/nav_button_back.tsx new file mode 100644 index 00000000000000..0c01bb72b339af --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/nav_button_back.tsx @@ -0,0 +1,19 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiButtonEmpty } from '@elastic/eui'; +import React from 'react'; +import styled from 'styled-components'; + +export function NavButtonBack({ href, text }: { href: string; text: string }) { + const ButtonEmpty = styled(EuiButtonEmpty)` + margin-right: ${props => props.theme.eui.spacerSizes.xl}; + `; + return ( + + {text} + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_card.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_card.tsx new file mode 100644 index 00000000000000..d1d7cfc180cad9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_card.tsx @@ -0,0 +1,47 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import styled from 'styled-components'; +import { EuiCard } from '@elastic/eui'; +import { PackageInfo, PackageListItem } from '../../../types'; +import { useLinks } from '../hooks'; +import { PackageIcon } from './package_icon'; + +export interface BadgeProps { + showInstalledBadge?: boolean; +} + +type PackageCardProps = (PackageListItem | PackageInfo) & BadgeProps; + +// adding the `href` causes EuiCard to use a `a` instead of a `button` +// `a` tags use `euiLinkColor` which results in blueish Badge text +const Card = styled(EuiCard)` + color: inherit; +`; + +export function PackageCard({ + description, + name, + title, + version, + showInstalledBadge, + status, + icons, +}: PackageCardProps) { + const { toDetailView } = useLinks(); + const url = toDetailView({ name, version }); + + return ( + } + href={url} + /> + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_icon.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_icon.tsx new file mode 100644 index 00000000000000..dd2f46adc31885 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_icon.tsx @@ -0,0 +1,26 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { ICON_TYPES, EuiIcon, EuiIconProps } from '@elastic/eui'; +import { PackageInfo, PackageListItem } from '../../../types'; +import { useLinks } from '../hooks'; + +type Package = PackageInfo | PackageListItem; + +export const PackageIcon: React.FunctionComponent<{ + packageName: string; + icons?: Package['icons']; +} & Omit> = ({ packageName, icons, ...euiIconProps }) => { + const { toImage } = useLinks(); + // try to find a logo in EUI + const euiLogoIcon = ICON_TYPES.find(key => key.toLowerCase() === `logo${packageName}`); + const svgIcons = icons?.filter(icon => icon.type === 'image/svg+xml'); + const localIcon = svgIcons && Array.isArray(svgIcons) && svgIcons[0]; + const pathToLocal = localIcon && toImage(localIcon.src); + const euiIconType = pathToLocal || euiLogoIcon || 'package'; + + return ; +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_list_grid.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_list_grid.tsx new file mode 100644 index 00000000000000..34e1763c44255e --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/package_list_grid.tsx @@ -0,0 +1,63 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiFlexGrid, EuiFlexGroup, EuiFlexItem, EuiSpacer, EuiText } from '@elastic/eui'; +import React, { Fragment, ReactNode } from 'react'; +import { PackageList } from '../../../types'; +import { BadgeProps, PackageCard } from './package_card'; + +type ListProps = { + controls?: ReactNode; + title: string; + list: PackageList; +} & BadgeProps; + +export function PackageListGrid({ controls, title, list, showInstalledBadge }: ListProps) { + const controlsContent = ; + const gridContent = ; + + return ( + + {controlsContent} + {gridContent} + + ); +} + +interface ControlsColumnProps { + controls: ReactNode; + title: string; +} + +function ControlsColumn({ controls, title }: ControlsColumnProps) { + return ( + + +

{title}

+ + + + {controls} + + + + ); +} + +type GridColumnProps = { + list: PackageList; +} & BadgeProps; + +function GridColumn({ list }: GridColumnProps) { + return ( + + {list.map(item => ( + + + + ))} + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/requirements.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/requirements.tsx new file mode 100644 index 00000000000000..f60d2d83ed45e1 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/requirements.tsx @@ -0,0 +1,57 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiFlexGroup, EuiFlexItem, EuiIcon, EuiText, EuiTextColor, EuiTitle } from '@elastic/eui'; +import React, { Fragment } from 'react'; +import styled from 'styled-components'; +import { RequirementsByServiceName, entries } from '../../../types'; +import { ServiceTitleMap } from '../constants'; +import { Version } from './version'; + +export interface RequirementsProps { + requirements: RequirementsByServiceName; +} + +const FlexGroup = styled(EuiFlexGroup)` + padding: 0 0 ${props => props.theme.eui.paddingSizes.m} 0; + margin: 0; +`; +const StyledVersion = styled(Version)` + font-size: ${props => props.theme.eui.euiFontSizeXS}; +`; + +export function Requirements(props: RequirementsProps) { + const { requirements } = props; + + return ( + + + + + + + + +

Elastic Stack Compatibility

+ + + + + {entries(requirements).map(([service, requirement]) => ( + + + + {ServiceTitleMap[service]}: + + + + + + + ))} + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/version.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/version.tsx new file mode 100644 index 00000000000000..537f6201dea067 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/components/version.tsx @@ -0,0 +1,22 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React from 'react'; +import styled from 'styled-components'; +import { RequirementVersion } from '../../../types'; + +const CodeText = styled.span` + font-family: ${props => props.theme.eui.euiCodeFontFamily}; +`; +export function Version({ + className, + version, +}: { + className?: string; + version: RequirementVersion; +}) { + return {version}; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/constants.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/constants.tsx new file mode 100644 index 00000000000000..3a6dfe4a87dafe --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/constants.tsx @@ -0,0 +1,44 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { IconType } from '@elastic/eui'; +import { AssetType, ElasticsearchAssetType, KibanaAssetType, ServiceName } from '../../types'; + +// only allow Kibana assets for the kibana key, ES asssets for elasticsearch, etc +type ServiceNameToAssetTypes = Record, KibanaAssetType[]> & + Record, ElasticsearchAssetType[]>; + +export const DisplayedAssets: ServiceNameToAssetTypes = { + kibana: Object.values(KibanaAssetType), + elasticsearch: Object.values(ElasticsearchAssetType), +}; + +export const AssetTitleMap: Record = { + dashboard: 'Dashboard', + 'ilm-policy': 'ILM Policy', + 'ingest-pipeline': 'Ingest Pipeline', + 'index-pattern': 'Index Pattern', + 'index-template': 'Index Template', + search: 'Saved Search', + visualization: 'Visualization', + input: 'Agent input', +}; + +export const ServiceTitleMap: Record = { + elasticsearch: 'Elasticsearch', + kibana: 'Kibana', +}; + +export const AssetIcons: Record = { + dashboard: 'dashboardApp', + 'index-pattern': 'indexPatternApp', + search: 'searchProfilerApp', + visualization: 'visualizeApp', +}; + +export const ServiceIcons: Record = { + elasticsearch: 'logoElasticsearch', + kibana: 'logoKibana', +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/index.tsx new file mode 100644 index 00000000000000..589ce5f5dbd251 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/index.tsx @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +// export { useBreadcrumbs } from './use_breadcrumbs'; +export { useLinks } from './use_links'; +export { + PackageInstallProvider, + useDeletePackage, + useGetPackageInstallStatus, + useInstallPackage, + useSetPackageInstallStatus, +} from './use_package_install'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_breadcrumbs.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_breadcrumbs.tsx new file mode 100644 index 00000000000000..6222d346432c3e --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_breadcrumbs.tsx @@ -0,0 +1,13 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { ChromeBreadcrumb } from '../../../../../../../../../src/core/public'; +import { useCore } from '../../../hooks'; + +export function useBreadcrumbs(newBreadcrumbs: ChromeBreadcrumb[]) { + const { chrome } = useCore(); + return chrome.setBreadcrumbs(newBreadcrumbs); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_links.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_links.tsx new file mode 100644 index 00000000000000..d4ed3624a6e685 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_links.tsx @@ -0,0 +1,61 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { generatePath } from 'react-router-dom'; +import { useCore } from '../../../hooks/use_core'; +import { PLUGIN_ID } from '../../../constants'; +import { epmRouteService } from '../../../services'; +import { DetailViewPanelName } from '../../../types'; +import { BASE_PATH, EPM_PATH, EPM_DETAIL_VIEW_PATH } from '../../../constants'; + +// TODO: get this from server/packages/handlers.ts (move elsewhere?) +// seems like part of the name@version change +interface DetailParams { + name: string; + version: string; + panel?: DetailViewPanelName; + withAppRoot?: boolean; +} + +const removeRelativePath = (relativePath: string): string => + new URL(relativePath, 'http://example.com').pathname; + +export function useLinks() { + const { http } = useCore(); + function appRoot(path: string) { + // include '#' because we're using HashRouter + return http.basePath.prepend(BASE_PATH + '#' + path); + } + + return { + toAssets: (path: string) => + http.basePath.prepend( + `/plugins/${PLUGIN_ID}/applications/ingest_manager/sections/epm/assets/${path}` + ), + toImage: (path: string) => http.basePath.prepend(epmRouteService.getFilePath(path)), + toRelativeImage: ({ + path, + packageName, + version, + }: { + path: string; + packageName: string; + version: string; + }) => { + const imagePath = removeRelativePath(path); + const pkgkey = `${packageName}-${version}`; + const filePath = `${epmRouteService.getInfoPath(pkgkey)}/${imagePath}`; + return http.basePath.prepend(filePath); + }, + toListView: () => appRoot(EPM_PATH), + toDetailView: ({ name, version, panel, withAppRoot = true }: DetailParams) => { + // panel is optional, but `generatePath` won't accept `path: undefined` + // so use this to pass `{ pkgkey }` or `{ pkgkey, panel }` + const params = Object.assign({ pkgkey: `${name}-${version}` }, panel ? { panel } : {}); + const path = generatePath(EPM_DETAIL_VIEW_PATH, params); + return withAppRoot ? appRoot(path) : path; + }, + }; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_package_install.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_package_install.tsx new file mode 100644 index 00000000000000..537a2616f17862 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/hooks/use_package_install.tsx @@ -0,0 +1,142 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import createContainer from 'constate'; +import React, { useCallback, useState } from 'react'; +import { NotificationsStart } from 'src/core/public'; +import { useLinks } from '.'; +import { toMountPoint } from '../../../../../../../../../src/plugins/kibana_react/public'; +import { PackageInfo } from '../../../types'; +import { sendInstallPackage, sendRemovePackage } from '../../../hooks'; +import { InstallStatus } from '../../../types'; + +interface PackagesInstall { + [key: string]: PackageInstallItem; +} + +interface PackageInstallItem { + status: InstallStatus; +} + +type InstallPackageProps = Pick; + +function usePackageInstall({ notifications }: { notifications: NotificationsStart }) { + const [packages, setPackage] = useState({}); + const { toDetailView } = useLinks(); + + const setPackageInstallStatus = useCallback( + ({ name, status }: { name: PackageInfo['name']; status: InstallStatus }) => { + setPackage((prev: PackagesInstall) => ({ + ...prev, + [name]: { status }, + })); + }, + [] + ); + + const installPackage = useCallback( + async ({ name, version, title }: InstallPackageProps) => { + setPackageInstallStatus({ name, status: InstallStatus.installing }); + const pkgkey = `${name}-${version}`; + + const res = await sendInstallPackage(pkgkey); + if (res.error) { + setPackageInstallStatus({ name, status: InstallStatus.notInstalled }); + notifications.toasts.addWarning({ + title: `Failed to install ${title} package`, + text: + 'Something went wrong while trying to install this package. Please try again later.', + iconType: 'alert', + }); + } else { + setPackageInstallStatus({ name, status: InstallStatus.installed }); + const SuccessMsg =

Successfully installed {name}

; + + notifications.toasts.addSuccess({ + title: `Installed ${title} package`, + text: toMountPoint(SuccessMsg), + }); + + // TODO: this should probably live somewhere else and use , + // this hook could return the request state and a component could + // use that state. the component should be able to unsubscribe to prevent memory leaks + const packageUrl = toDetailView({ name, version }); + const dataSourcesUrl = toDetailView({ + name, + version, + panel: 'data-sources', + withAppRoot: false, + }); + if (window.location.href.includes(packageUrl)) window.location.hash = dataSourcesUrl; + } + }, + [notifications.toasts, setPackageInstallStatus, toDetailView] + ); + + const getPackageInstallStatus = useCallback( + (pkg: string): InstallStatus => { + return packages[pkg].status; + }, + [packages] + ); + + const deletePackage = useCallback( + async ({ name, version, title }: Pick) => { + setPackageInstallStatus({ name, status: InstallStatus.uninstalling }); + const pkgkey = `${name}-${version}`; + + const res = await sendRemovePackage(pkgkey); + if (res.error) { + setPackageInstallStatus({ name, status: InstallStatus.installed }); + notifications.toasts.addWarning({ + title: `Failed to delete ${title} package`, + text: 'Something went wrong while trying to delete this package. Please try again later.', + iconType: 'alert', + }); + } else { + setPackageInstallStatus({ name, status: InstallStatus.notInstalled }); + + const SuccessMsg =

Successfully deleted {title}

; + + notifications.toasts.addSuccess({ + title: `Deleted ${title} package`, + text: toMountPoint(SuccessMsg), + }); + + const packageUrl = toDetailView({ name, version }); + const dataSourcesUrl = toDetailView({ + name, + version, + panel: 'data-sources', + }); + if (window.location.href.includes(packageUrl)) window.location.href = dataSourcesUrl; + } + }, + [notifications.toasts, setPackageInstallStatus, toDetailView] + ); + + return { + packages, + installPackage, + setPackageInstallStatus, + getPackageInstallStatus, + deletePackage, + }; +} + +export const [ + PackageInstallProvider, + useInstallPackage, + useSetPackageInstallStatus, + useGetPackageInstallStatus, + useDeletePackage, +] = createContainer( + usePackageInstall, + value => value.installPackage, + value => value.setPackageInstallStatus, + value => value.getPackageInstallStatus, + value => value.deletePackage +); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/index.tsx index 777c2353226c41..b8dd08eb46a54d 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/index.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/index.tsx @@ -5,74 +5,28 @@ */ import React from 'react'; -import styled from 'styled-components'; -import { EuiFlexGroup, EuiFlexItem, EuiText, EuiImage } from '@elastic/eui'; -import { FormattedMessage } from '@kbn/i18n/react'; -import { PLUGIN_ID } from '../../constants'; -import { WithHeaderLayout } from '../../layouts'; -import { useConfig, useCore } from '../../hooks'; - -const ImageWrapper = styled.div` - margin-bottom: -62px; -`; +import { HashRouter as Router, Switch, Route } from 'react-router-dom'; +import { useConfig } from '../../hooks'; +import { CreateDatasourcePage } from '../agent_config/create_datasource_page'; +import { Home } from './screens/home'; +import { Detail } from './screens/detail'; export const EPMApp: React.FunctionComponent = () => { const { epm } = useConfig(); - const { http } = useCore(); - - if (!epm.enabled) { - return null; - } - return ( - - - -

- -

- - - - -

- -

- - - - } - rightColumn={ - - - - } - tabs={[ - { - id: 'all_packages', - name: 'All packages', - isSelected: true, - }, - { - id: 'installed_packages', - name: 'Installed packages', - }, - ]} - > - hello world - fleet app - - ); + return epm.enabled ? ( + + + + + + + + + + + + + + ) : null; }; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/confirm_package_delete.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/confirm_package_delete.tsx new file mode 100644 index 00000000000000..2b3be04ac476b4 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/confirm_package_delete.tsx @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiCallOut, EuiConfirmModal, EuiOverlayMask } from '@elastic/eui'; +import React from 'react'; + +interface ConfirmPackageDeleteProps { + onCancel: () => void; + onConfirm: () => void; + packageName: string; + numOfAssets: number; +} +export const ConfirmPackageDelete = (props: ConfirmPackageDeleteProps) => { + const { onCancel, onConfirm, packageName, numOfAssets } = props; + return ( + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/confirm_package_install.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/confirm_package_install.tsx new file mode 100644 index 00000000000000..137d9cf226b4d2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/confirm_package_install.tsx @@ -0,0 +1,36 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiCallOut, EuiConfirmModal, EuiOverlayMask, EuiSpacer } from '@elastic/eui'; +import React from 'react'; + +interface ConfirmPackageInstallProps { + onCancel: () => void; + onConfirm: () => void; + packageName: string; + numOfAssets: number; +} +export const ConfirmPackageInstall = (props: ConfirmPackageInstallProps) => { + const { onCancel, onConfirm, packageName, numOfAssets } = props; + return ( + + + + +

+ and will only be accessible to users who have permission to view this Space. Elasticsearch + assets are installed globally and will be accessible to all Kibana users. +

+ + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/content.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/content.tsx new file mode 100644 index 00000000000000..384cbbeed378e7 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/content.tsx @@ -0,0 +1,83 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiFlexGroup, EuiFlexItem, EuiHorizontalRule, EuiSpacer } from '@elastic/eui'; +import React from 'react'; +import styled from 'styled-components'; +import { DEFAULT_PANEL, DetailParams } from '.'; +import { PackageInfo } from '../../../../types'; +import { AssetsFacetGroup } from '../../components/assets_facet_group'; +import { Requirements } from '../../components/requirements'; +import { CenterColumn, LeftColumn, RightColumn } from './layout'; +import { OverviewPanel } from './overview_panel'; +import { SideNavLinks } from './side_nav_links'; +import { DataSourcesPanel } from './data_sources_panel'; + +type ContentProps = PackageInfo & Pick & { hasIconPanel: boolean }; +export function Content(props: ContentProps) { + const { hasIconPanel, name, panel, version } = props; + const SideNavColumn = hasIconPanel + ? styled(LeftColumn)` + /* 🤢🤷 https://www.styled-components.com/docs/faqs#how-can-i-override-styles-with-higher-specificity */ + &&& { + margin-top: 77px; + } + ` + : LeftColumn; + + // fixes IE11 problem with nested flex items + const ContentFlexGroup = styled(EuiFlexGroup)` + flex: 0 0 auto !important; + `; + return ( + + + + + + + + + + + + ); +} + +type ContentPanelProps = PackageInfo & Pick; +export function ContentPanel(props: ContentPanelProps) { + const { panel, name, version } = props; + switch (panel) { + case 'data-sources': + return ; + case 'overview': + default: + return ; + } +} + +type RightColumnContentProps = PackageInfo & Pick; +function RightColumnContent(props: RightColumnContentProps) { + const { assets, requirement, panel } = props; + switch (panel) { + case 'overview': + return ( + + + + + + + + + + + + ); + default: + return ; + } +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/content_collapse.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/content_collapse.tsx new file mode 100644 index 00000000000000..9d5614debb42b8 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/content_collapse.tsx @@ -0,0 +1,96 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiButton, EuiButtonEmpty, EuiHorizontalRule, EuiSpacer } from '@elastic/eui'; +import React, { Fragment, useCallback, useLayoutEffect, useRef, useState } from 'react'; +import styled from 'styled-components'; + +const BottomFade = styled.div` + width: 100%; + background: ${props => + `linear-gradient(${props.theme.eui.euiColorEmptyShade}00 0%, ${props.theme.eui.euiColorEmptyShade} 100%)`}; + margin-top: -${props => parseInt(props.theme.eui.spacerSizes.xl, 10) * 2}px; + height: ${props => parseInt(props.theme.eui.spacerSizes.xl, 10) * 2}px; + position: absolute; +`; +const ContentCollapseContainer = styled.div` + position: relative; +`; +const CollapseButtonContainer = styled.div` + display: inline-block; + background-color: ${props => props.theme.eui.euiColorEmptyShade}; + position: absolute; + left: 50%; + transform: translateX(-50%); + top: ${props => parseInt(props.theme.eui.euiButtonHeight, 10) / 2}px; +`; +const CollapseButtonTop = styled(EuiButtonEmpty)` + float: right; +`; + +const CollapseButton = ({ + open, + toggleCollapse, +}: { + open: boolean; + toggleCollapse: () => void; +}) => { + return ( +
+ + + + + {open ? 'Collapse' : 'Read more'} + + +
+ ); +}; + +export const ContentCollapse = ({ children }: { children: React.ReactNode }) => { + const [open, setOpen] = useState(false); + const [height, setHeight] = useState('auto'); + const [collapsible, setCollapsible] = useState(true); + const contentEl = useRef(null); + const collapsedHeight = 360; + + // if content is too small, don't collapse + useLayoutEffect( + () => + contentEl.current && contentEl.current.clientHeight < collapsedHeight + ? setCollapsible(false) + : setHeight(collapsedHeight), + [] + ); + + const clickOpen = useCallback(() => { + setOpen(!open); + }, [open]); + + return ( + + {collapsible ? ( + +
+ {open && ( + + Collapse + + )} + {children} +
+ {!open && } + + + ) : ( +
{children}
+ )} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/data_sources_panel.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/data_sources_panel.tsx new file mode 100644 index 00000000000000..fa3245aec02c5b --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/data_sources_panel.tsx @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { Fragment } from 'react'; +import { EuiTitle } from '@elastic/eui'; +import { Redirect } from 'react-router-dom'; +import { useLinks, useGetPackageInstallStatus } from '../../hooks'; +import { InstallStatus } from '../../../../types'; + +interface DataSourcesPanelProps { + name: string; + version: string; +} +export const DataSourcesPanel = ({ name, version }: DataSourcesPanelProps) => { + const { toDetailView } = useLinks(); + const getPackageInstallStatus = useGetPackageInstallStatus(); + const packageInstallStatus = getPackageInstallStatus(name); + // if they arrive at this page and the package is not installed, send them to overview + // this happens if they arrive with a direct url or they uninstall while on this tab + if (packageInstallStatus !== InstallStatus.installed) + return ( + + ); + return ( + + + Data Sources + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/header.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/header.tsx new file mode 100644 index 00000000000000..5a51515d494865 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/header.tsx @@ -0,0 +1,81 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { Fragment } from 'react'; +import styled from 'styled-components'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { EuiFlexGroup, EuiFlexItem, EuiPage, EuiTitle, IconType, EuiButton } from '@elastic/eui'; +import { PackageInfo } from '../../../../types'; +import { EPM_PATH } from '../../../../constants'; +import { useCapabilities, useLink } from '../../../../hooks'; +import { IconPanel } from '../../components/icon_panel'; +import { NavButtonBack } from '../../components/nav_button_back'; +import { Version } from '../../components/version'; +import { useLinks } from '../../hooks'; +import { CenterColumn, LeftColumn, RightColumn } from './layout'; + +const FullWidthNavRow = styled(EuiPage)` + /* no left padding so link is against column left edge */ + padding-left: 0; +`; + +const Text = styled.span` + margin-right: ${props => props.theme.eui.euiSizeM}; +`; + +const StyledVersion = styled(Version)` + font-size: ${props => props.theme.eui.euiFontSizeS}; + color: ${props => props.theme.eui.euiColorDarkShade}; +`; + +type HeaderProps = PackageInfo & { iconType?: IconType }; + +export function Header(props: HeaderProps) { + const { iconType, name, title, version } = props; + const hasWriteCapabilites = useCapabilities().write; + const { toListView } = useLinks(); + // useBreadcrumbs([{ text: PLUGIN.TITLE, href: toListView() }, { text: title }]); + + const ADD_DATASOURCE_URI = useLink(`${EPM_PATH}/${name}-${version}/add-datasource`); + + return ( + + + + + + {iconType ? ( + + + + ) : null} + + +

+ {title} + +

+ + + + + + + + + + + + + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/index.tsx new file mode 100644 index 00000000000000..4bc90c6a0f8fd8 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/index.tsx @@ -0,0 +1,82 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiPage, EuiPageBody, EuiPageProps, ICON_TYPES } from '@elastic/eui'; +import React, { Fragment, useEffect, useState } from 'react'; +import { useParams } from 'react-router-dom'; +import styled from 'styled-components'; +import { DetailViewPanelName, InstallStatus } from '../../../../types'; +import { PackageInfo } from '../../../../types'; +import { useSetPackageInstallStatus } from '../../hooks'; +import { Content } from './content'; +import { Header } from './header'; +import { sendGetPackageInfoByKey } from '../../../../hooks'; + +export const DEFAULT_PANEL: DetailViewPanelName = 'overview'; + +export interface DetailParams { + pkgkey: string; + panel?: DetailViewPanelName; +} + +export function Detail() { + // TODO: fix forced cast if possible + const { pkgkey, panel = DEFAULT_PANEL } = useParams() as DetailParams; + + const [info, setInfo] = useState(null); + const setPackageInstallStatus = useSetPackageInstallStatus(); + useEffect(() => { + sendGetPackageInfoByKey(pkgkey).then(response => { + const packageInfo = response.data?.response; + const title = packageInfo?.title; + const name = packageInfo?.name; + const status: InstallStatus = packageInfo?.status as any; + + // track install status state + if (name) { + setPackageInstallStatus({ name, status }); + } + if (packageInfo) { + setInfo({ ...packageInfo, title: title || '' }); + } + }); + }, [pkgkey, setPackageInstallStatus]); + + if (!info) return null; + + return ; +} + +const FullWidthHeader = styled(EuiPage)` + border-bottom: ${props => props.theme.eui.euiBorderThin}; + padding-bottom: ${props => props.theme.eui.paddingSizes.xl}; +`; + +const FullWidthContent = styled(EuiPage)` + background-color: ${props => props.theme.eui.euiColorEmptyShade}; + padding-top: ${props => parseInt(props.theme.eui.paddingSizes.xl, 10) * 1.25}px; + flex-grow: 1; +`; + +type LayoutProps = PackageInfo & Pick & Pick; +export function DetailLayout(props: LayoutProps) { + const { name, restrictWidth } = props; + const iconType = ICON_TYPES.find(key => key.toLowerCase() === `logo${name}`); + + return ( + + + +
+ + + + + + + + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/installation_button.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/installation_button.tsx new file mode 100644 index 00000000000000..8a8afed5570ed9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/installation_button.tsx @@ -0,0 +1,97 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiButton } from '@elastic/eui'; +import React, { Fragment, useCallback, useMemo, useState } from 'react'; +import { PackageInfo, InstallStatus } from '../../../../types'; +import { useCapabilities } from '../../../../hooks'; +import { useDeletePackage, useGetPackageInstallStatus, useInstallPackage } from '../../hooks'; +import { ConfirmPackageDelete } from './confirm_package_delete'; +import { ConfirmPackageInstall } from './confirm_package_install'; + +interface InstallationButtonProps { + package: PackageInfo; +} + +export function InstallationButton(props: InstallationButtonProps) { + const { assets, name, title, version } = props.package; + const hasWriteCapabilites = useCapabilities().write; + const installPackage = useInstallPackage(); + const deletePackage = useDeletePackage(); + const getPackageInstallStatus = useGetPackageInstallStatus(); + const installationStatus = getPackageInstallStatus(name); + + const isInstalling = installationStatus === InstallStatus.installing; + const isRemoving = installationStatus === InstallStatus.uninstalling; + const isInstalled = installationStatus === InstallStatus.installed; + const [isModalVisible, setModalVisible] = useState(false); + const toggleModal = useCallback(() => { + setModalVisible(!isModalVisible); + }, [isModalVisible]); + + const handleClickInstall = useCallback(() => { + installPackage({ name, version, title }); + toggleModal(); + }, [installPackage, name, title, toggleModal, version]); + + const handleClickDelete = useCallback(() => { + deletePackage({ name, version, title }); + toggleModal(); + }, [deletePackage, name, title, toggleModal, version]); + + const numOfAssets = useMemo( + () => + Object.entries(assets).reduce( + (acc, [serviceName, serviceNameValue]) => + acc + + Object.entries(serviceNameValue).reduce( + (acc2, [assetName, assetNameValue]) => acc2 + assetNameValue.length, + 0 + ), + 0 + ), + [assets] + ); + + const installButton = ( + + {isInstalling ? 'Installing' : 'Install package'} + + ); + + const installedButton = ( + + {isInstalling ? 'Deleting' : 'Delete package'} + + ); + + const deletionModal = ( + + ); + + const installationModal = ( + + ); + + return hasWriteCapabilites ? ( + + {isInstalled ? installedButton : installButton} + {isModalVisible && (isInstalled ? deletionModal : installationModal)} + + ) : null; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/layout.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/layout.tsx new file mode 100644 index 00000000000000..a802e35add7dbe --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/layout.tsx @@ -0,0 +1,37 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiFlexItem } from '@elastic/eui'; +import React, { FunctionComponent, ReactNode } from 'react'; + +interface ColumnProps { + children?: ReactNode; + className?: string; +} + +export const LeftColumn: FunctionComponent = ({ children, ...rest }) => { + return ( + + {children} + + ); +}; + +export const CenterColumn: FunctionComponent = ({ children, ...rest }) => { + return ( + + {children} + + ); +}; + +export const RightColumn: FunctionComponent = ({ children, ...rest }) => { + return ( + + {children} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/markdown_renderers.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/markdown_renderers.tsx new file mode 100644 index 00000000000000..2e321e8bfc36ff --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/markdown_renderers.tsx @@ -0,0 +1,70 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + EuiCodeBlock, + EuiLink, + EuiTableHeaderCell, + EuiTableRow, + EuiTableRowCell, + EuiText, +} from '@elastic/eui'; +import React from 'react'; + +/** prevents links to the new pages from accessing `window.opener` */ +const REL_NOOPENER = 'noopener'; + +/** prevents search engine manipulation by noting the linked document is not trusted or endorsed by us */ +const REL_NOFOLLOW = 'nofollow'; + +/** prevents the browser from sending the current address as referrer via the Referer HTTP header */ +const REL_NOREFERRER = 'noreferrer'; + +export const markdownRenderers = { + root: ({ children }: { children: React.ReactNode[] }) => ( + {children} + ), + table: ({ children }: { children: React.ReactNode[] }) => ( + + {children} +
+ ), + tableRow: ({ children }: { children: React.ReactNode[] }) => ( + {children} + ), + tableCell: ({ isHeader, children }: { isHeader: boolean; children: React.ReactNode[] }) => { + return isHeader ? ( + {children} + ) : ( + {children} + ); + }, + // the headings used in markdown don't match our page so mapping them to the appropriate one + heading: ({ level, children }: { level: number; children: React.ReactNode[] }) => { + switch (level) { + case 1: + return

{children}

; + case 2: + return

{children}

; + case 3: + return
{children}
; + default: + return
{children}
; + } + }, + link: ({ children, href }: { children: React.ReactNode[]; href?: string }) => ( + + {children} + + ), + code: ({ language, value }: { language: string; value: string }) => { + return ( + + {value} + + ); + }, +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/overview_panel.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/overview_panel.tsx new file mode 100644 index 00000000000000..ca6aceabe7f36e --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/overview_panel.tsx @@ -0,0 +1,21 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiSpacer } from '@elastic/eui'; +import React, { Fragment } from 'react'; +import { PackageInfo } from '../../../../types'; +import { Readme } from './readme'; +import { Screenshots } from './screenshots'; + +export function OverviewPanel(props: PackageInfo) { + const { screenshots, readme, name, version } = props; + return ( + + {readme && } + + {screenshots && } + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/readme.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/readme.tsx new file mode 100644 index 00000000000000..72e2d779c39be0 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/readme.tsx @@ -0,0 +1,67 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiLoadingContent, EuiText } from '@elastic/eui'; +import React, { Fragment, useEffect, useState } from 'react'; +import ReactMarkdown from 'react-markdown'; +import { useLinks } from '../../hooks'; +import { ContentCollapse } from './content_collapse'; +import { markdownRenderers } from './markdown_renderers'; +import { sendGetFileByPath } from '../../../../hooks'; + +export function Readme({ + readmePath, + packageName, + version, +}: { + readmePath: string; + packageName: string; + version: string; +}) { + const [markdown, setMarkdown] = useState(undefined); + const { toRelativeImage } = useLinks(); + const handleImageUri = React.useCallback( + (uri: string) => { + const isRelative = + uri.indexOf('http://') === 0 || uri.indexOf('https://') === 0 ? false : true; + const fullUri = isRelative ? toRelativeImage({ packageName, version, path: uri }) : uri; + return fullUri; + }, + [toRelativeImage, packageName, version] + ); + + useEffect(() => { + sendGetFileByPath(readmePath).then(res => { + setMarkdown(res.data || ''); + }); + }, [readmePath]); + + return ( + + {markdown !== undefined ? ( + + + + ) : ( + + {/* simulates a long page of text loading */} +

+ +

+

+ +

+

+ +

+ + )} + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/screenshots.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/screenshots.tsx new file mode 100644 index 00000000000000..10cf9c97723c0c --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/screenshots.tsx @@ -0,0 +1,77 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiFlexGroup, EuiFlexItem, EuiImage, EuiSpacer, EuiText, EuiTitle } from '@elastic/eui'; +import React, { Fragment } from 'react'; +import styled from 'styled-components'; +import { ScreenshotItem } from '../../../../types'; +import { useLinks } from '../../hooks'; + +interface ScreenshotProps { + images: ScreenshotItem[]; +} + +export function Screenshots(props: ScreenshotProps) { + const { toImage } = useLinks(); + const { images } = props; + + // for now, just get first image + const image = images[0]; + const hasCaption = image.title ? true : false; + + const getHorizontalPadding = (styledProps: any): number => + parseInt(styledProps.theme.eui.paddingSizes.xl, 10) * 2; + const getVerticalPadding = (styledProps: any): number => + parseInt(styledProps.theme.eui.paddingSizes.xl, 10) * 1.75; + const getPadding = (styledProps: any) => + hasCaption + ? `${styledProps.theme.eui.paddingSizes.xl} ${getHorizontalPadding( + styledProps + )}px ${getVerticalPadding(styledProps)}px` + : `${getHorizontalPadding(styledProps)}px ${getVerticalPadding(styledProps)}px`; + + const ScreenshotsContainer = styled(EuiFlexGroup)` + background: linear-gradient(360deg, rgba(0, 0, 0, 0.2) 0%, rgba(0, 0, 0, 0) 100%), + ${styledProps => styledProps.theme.eui.euiColorPrimary}; + padding: ${styledProps => getPadding(styledProps)}; + flex: 0 0 auto; + border-radius: ${styledProps => styledProps.theme.eui.euiBorderRadius}; + `; + + // fixes ie11 problems with nested flex items + const NestedEuiFlexItem = styled(EuiFlexItem)` + flex: 0 0 auto !important; + `; + return ( + + +

Screenshots

+ + + + {hasCaption && ( + + + {image.title} + + + + )} + + {/* By default EuiImage sets width to 100% and Figure to 22.5rem for size=l images, + set image to same width. Will need to update if size changes. + */} + + + + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/side_nav_links.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/side_nav_links.tsx new file mode 100644 index 00000000000000..39a6fca2e43180 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/detail/side_nav_links.tsx @@ -0,0 +1,50 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiButtonEmpty, EuiButtonEmptyProps } from '@elastic/eui'; +import React, { Fragment } from 'react'; +import styled from 'styled-components'; +import { PackageInfo, entries, DetailViewPanelName, InstallStatus } from '../../../../types'; +import { useLinks, useGetPackageInstallStatus } from '../../hooks'; + +export type NavLinkProps = Pick & { + active: DetailViewPanelName; +}; + +const PanelDisplayNames: Record = { + overview: 'Overview', + 'data-sources': 'Data Sources', +}; + +export function SideNavLinks({ name, version, active }: NavLinkProps) { + const { toDetailView } = useLinks(); + const getPackageInstallStatus = useGetPackageInstallStatus(); + const packageInstallStatus = getPackageInstallStatus(name); + + return ( + + {entries(PanelDisplayNames).map(([panel, display]) => { + const Link = styled(EuiButtonEmpty).attrs({ + href: toDetailView({ name, version, panel }), + })` + font-weight: ${p => + active === panel + ? p.theme.eui.euiFontWeightSemiBold + : p.theme.eui.euiFontWeightRegular}; + `; + // don't display Data Sources tab if the package is not installed + if (packageInstallStatus !== InstallStatus.installed && panel === 'data-sources') + return null; + + return ( +
+ {display} +
+ ); + })} + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/category_facets.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/category_facets.tsx new file mode 100644 index 00000000000000..e138f9f531a392 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/category_facets.tsx @@ -0,0 +1,36 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { EuiFacetButton, EuiFacetGroup } from '@elastic/eui'; +import React from 'react'; +import { CategorySummaryItem, CategorySummaryList } from '../../../../types'; + +export function CategoryFacets({ + categories, + selectedCategory, + onCategoryChange, +}: { + categories: CategorySummaryList; + selectedCategory: string; + onCategoryChange: (category: CategorySummaryItem) => unknown; +}) { + const controls = ( + + {categories.map(category => ( + onCategoryChange(category)} + > + {category.title} + + ))} + + ); + + return controls; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/header.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/header.tsx new file mode 100644 index 00000000000000..2cb5aca39c8073 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/header.tsx @@ -0,0 +1,54 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiFlexGroup, EuiFlexItem, EuiImage, EuiText } from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; +import React from 'react'; +import styled from 'styled-components'; +import { useLinks } from '../../hooks'; + +export function HeroCopy() { + return ( + + + +

+ +

+ + + + +

+ +

+ + + + ); +} + +export function HeroImage() { + const { toAssets } = useLinks(); + const ImageWrapper = styled.div` + margin-bottom: -38px; // revert to -62px when tabs are restored + `; + + return ( + + + + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/hooks.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/hooks.tsx new file mode 100644 index 00000000000000..c3e29f723dcba5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/hooks.tsx @@ -0,0 +1,47 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { useEffect, useRef, useState } from 'react'; +import { PackageList } from '../../../../types'; +import { fieldsToSearch, LocalSearch, searchIdField } from './search_packages'; + +export function useAllPackages(selectedCategory: string, categoryPackages: PackageList = []) { + const [allPackages, setAllPackages] = useState([]); + + useEffect(() => { + if (!selectedCategory) setAllPackages(categoryPackages); + }, [selectedCategory, categoryPackages]); + + return [allPackages, setAllPackages] as [typeof allPackages, typeof setAllPackages]; +} + +export function useLocalSearch(allPackages: PackageList) { + const localSearchRef = useRef(null); + + useEffect(() => { + if (!allPackages.length) return; + + const localSearch = new LocalSearch(searchIdField); + fieldsToSearch.forEach(field => localSearch.addIndex(field)); + localSearch.addDocuments(allPackages); + localSearchRef.current = localSearch; + }, [allPackages]); + + return localSearchRef; +} + +export function useInstalledPackages(allPackages: PackageList) { + const [installedPackages, setInstalledPackages] = useState([]); + + useEffect(() => { + setInstalledPackages(allPackages.filter(({ status }) => status === 'installed')); + }, [allPackages]); + + return [installedPackages, setInstalledPackages] as [ + typeof installedPackages, + typeof setInstalledPackages + ]; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/index.tsx new file mode 100644 index 00000000000000..640e4a30a40ca5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/index.tsx @@ -0,0 +1,139 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + EuiHorizontalRule, + // @ts-ignore + EuiSearchBar, + EuiSpacer, +} from '@elastic/eui'; +import React, { Fragment, useState } from 'react'; +import { useGetCategories, useGetPackages } from '../../../../hooks'; +import { WithHeaderLayout } from '../../../../layouts'; +import { CategorySummaryItem, PackageList } from '../../../../types'; +import { PackageListGrid } from '../../components/package_list_grid'; +// import { useBreadcrumbs, useLinks } from '../../hooks'; +import { CategoryFacets } from './category_facets'; +import { HeroCopy, HeroImage } from './header'; +import { useAllPackages, useInstalledPackages, useLocalSearch } from './hooks'; +import { SearchPackages } from './search_packages'; + +export function Home() { + // useBreadcrumbs([{ text: PLUGIN.TITLE, href: toListView() }]); + + const state = useHomeState(); + const searchBar = ( + { + state.setSearchTerm(userInput); + }} + /> + ); + const body = state.searchTerm ? ( + + ) : ( + + {state.installedPackages.length ? ( + + + + + ) : null} + + + ); + + return ( + } + rightColumn={} + // tabs={[ + // { + // id: 'all_packages', + // name: 'All packages', + // isSelected: true, + // }, + // { + // id: 'installed_packages', + // name: 'Installed packages', + // }, + // ]} + > + {searchBar} + + {body} + + ); +} + +type HomeState = ReturnType; + +export function useHomeState() { + const [searchTerm, setSearchTerm] = useState(''); + const [selectedCategory, setSelectedCategory] = useState(''); + const { data: categoriesRes } = useGetCategories(); + const categories = categoriesRes?.response; + const { data: categoryPackagesRes } = useGetPackages({ category: selectedCategory }); + const categoryPackages = categoryPackagesRes?.response; + const [allPackages, setAllPackages] = useAllPackages(selectedCategory, categoryPackages); + const localSearchRef = useLocalSearch(allPackages); + const [installedPackages, setInstalledPackages] = useInstalledPackages(allPackages); + + return { + searchTerm, + setSearchTerm, + selectedCategory, + setSelectedCategory, + categories, + allPackages, + setAllPackages, + installedPackages, + localSearchRef, + setInstalledPackages, + categoryPackages, + }; +} + +function InstalledPackages({ list }: { list: PackageList }) { + const title = 'Your Packages'; + + return ; +} + +function AvailablePackages({ + allPackages, + categories, + categoryPackages, + selectedCategory, + setSelectedCategory, +}: HomeState) { + const title = 'Available Packages'; + const noFilter = { + id: '', + title: 'All', + count: allPackages.length, + }; + + const controls = categories ? ( + setSelectedCategory(id)} + /> + ) : null; + + return ; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/search_packages.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/search_packages.tsx new file mode 100644 index 00000000000000..adffdefd30a4f9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/search_packages.tsx @@ -0,0 +1,33 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Search as LocalSearch } from 'js-search'; +import React from 'react'; +import { PackageList, PackageListItem } from '../../../../types'; +import { SearchResults } from './search_results'; + +export { LocalSearch }; +export type SearchField = keyof PackageListItem; +export const searchIdField: SearchField = 'name'; +export const fieldsToSearch: SearchField[] = ['description', 'name', 'title']; + +interface SearchPackagesProps { + searchTerm: string; + localSearchRef: React.MutableRefObject; + allPackages: PackageList; +} + +export function SearchPackages({ searchTerm, localSearchRef, allPackages }: SearchPackagesProps) { + // this means the search index hasn't been built yet. + // i.e. the intial fetch of all packages hasn't finished + if (!localSearchRef.current) return
Still fetching matches. Try again in a moment.
; + + const matches = localSearchRef.current.search(searchTerm) as PackageList; + const matchingIds = matches.map(match => match[searchIdField]); + const filtered = allPackages.filter(item => matchingIds.includes(item[searchIdField])); + + return ; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/search_results.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/search_results.tsx new file mode 100644 index 00000000000000..fbdcaac01931b1 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/epm/screens/home/search_results.tsx @@ -0,0 +1,33 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiText, EuiTitle } from '@elastic/eui'; +import React from 'react'; +import { PackageList } from '../../../../types'; +import { PackageListGrid } from '../../components/package_list_grid'; + +interface SearchResultsProps { + term: string; + results: PackageList; +} + +export function SearchResults({ term, results }: SearchResultsProps) { + const title = 'Search results'; + return ( + + + {results.length} results for "{term}" + + + } + /> + ); +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/agent_events_table.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/agent_events_table.tsx new file mode 100644 index 00000000000000..c0f55a5a275fd2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/agent_events_table.tsx @@ -0,0 +1,154 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useState } from 'react'; +import { + EuiBasicTable, + // @ts-ignore + EuiSuggest, + EuiFlexGroup, + EuiButton, + EuiSpacer, + EuiFlexItem, + EuiTitle, +} from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage, FormattedTime } from '@kbn/i18n/react'; +import { Agent, AgentEvent } from '../../../../types'; +import { usePagination, useGetOneAgentEvents } from '../../../../hooks'; +import { SearchBar } from '../../../../components/search_bar'; + +function useSearch() { + const [state, setState] = useState<{ search: string }>({ + search: '', + }); + + const setSearch = (s: string) => + setState({ + search: s, + }); + + return { + ...state, + setSearch, + }; +} + +export const AgentEventsTable: React.FunctionComponent<{ agent: Agent }> = ({ agent }) => { + const { pageSizeOptions, pagination, setPagination } = usePagination(); + const { search, setSearch } = useSearch(); + + const { isLoading, data, sendRequest } = useGetOneAgentEvents(agent.id, { + page: pagination.currentPage, + perPage: pagination.pageSize, + kuery: search && search.trim() !== '' ? search.trim() : undefined, + }); + + const refresh = () => sendRequest(); + + const total = data ? data.total : 0; + const list = data ? data.list : []; + const paginationOptions = { + pageIndex: pagination.currentPage - 1, + pageSize: pagination.pageSize, + totalItemCount: total, + pageSizeOptions, + }; + + const columns = [ + { + field: 'timestamp', + name: i18n.translate('xpack.ingestManager.agentEventsList.timestampColumnTitle', { + defaultMessage: 'Timestamp', + }), + render: (timestamp: string) => ( + + ), + sortable: true, + }, + { + field: 'type', + name: i18n.translate('xpack.ingestManager.agentEventsList.typeColumnTitle', { + defaultMessage: 'Type', + }), + width: '90px', + }, + { + field: 'subtype', + name: i18n.translate('xpack.ingestManager.agentEventsList.subtypeColumnTitle', { + defaultMessage: 'Subtype', + }), + width: '90px', + }, + { + field: 'message', + name: i18n.translate('xpack.ingestManager.agentEventsList.messageColumnTitle', { + defaultMessage: 'Message', + }), + }, + { + field: 'payload', + name: i18n.translate('xpack.ingestManager.agentEventsList.paylodColumnTitle', { + defaultMessage: 'Payload', + }), + truncateText: true, + render: (payload: any) => ( + + {payload && JSON.stringify(payload, null, 2)} + + ), + }, + ]; + + const onClickRefresh = () => { + refresh(); + }; + + const onChange = ({ page }: { page: { index: number; size: number } }) => { + const newPagination = { + ...pagination, + currentPage: page.index + 1, + pageSize: page.size, + }; + + setPagination(newPagination); + }; + + return ( + <> + +

+ +

+ + + + + + + + + + + + + + + onChange={onChange} + items={list} + columns={columns} + pagination={paginationOptions} + loading={isLoading} + /> + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/details_section.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/details_section.tsx new file mode 100644 index 00000000000000..0844368dc214bd --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/details_section.tsx @@ -0,0 +1,157 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useState, Fragment } from 'react'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiTitle, + EuiSpacer, + EuiFlexGroup, + EuiFlexItem, + EuiDescriptionList, + EuiButton, + EuiDescriptionListTitle, + EuiDescriptionListDescription, + EuiButtonEmpty, + EuiIconTip, + EuiTextColor, +} from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { useAgentRefresh } from '../hooks'; +import { AgentMetadataFlyout } from './metadata_flyout'; +import { Agent } from '../../../../types'; +import { AgentHealth } from '../../components/agent_health'; +import { useCapabilities, useGetOneAgentConfig } from '../../../../hooks'; +import { Loading } from '../../../../components'; +import { ConnectedLink } from '../../components'; +import { AgentUnenrollProvider } from '../../components/agent_unenroll_provider'; + +const Item: React.FunctionComponent<{ label: string }> = ({ label, children }) => { + return ( + + + {label} + {children} + + + ); +}; + +function useFlyout() { + const [isVisible, setVisible] = useState(false); + return { + isVisible, + show: () => setVisible(true), + hide: () => setVisible(false), + }; +} + +interface Props { + agent: Agent; +} +export const AgentDetailSection: React.FunctionComponent = ({ agent }) => { + const hasWriteCapabilites = useCapabilities().write; + const metadataFlyout = useFlyout(); + const refreshAgent = useAgentRefresh(); + + // Fetch AgentConfig information + const { isLoading: isAgentConfigLoading, data: agentConfigData } = useGetOneAgentConfig( + agent.config_id as string + ); + + const items = [ + { + title: i18n.translate('xpack.ingestManager.agentDetails.statusLabel', { + defaultMessage: 'Status', + }), + description: , + }, + { + title: i18n.translate('xpack.ingestManager.agentDetails.idLabel', { + defaultMessage: 'ID', + }), + description: agent.id, + }, + { + title: i18n.translate('xpack.ingestManager.agentDetails.typeLabel', { + defaultMessage: 'Type', + }), + description: agent.type, + }, + { + title: i18n.translate('xpack.ingestManager.agentDetails.agentConfigLabel', { + defaultMessage: 'AgentConfig', + }), + description: isAgentConfigLoading ? ( + + ) : agentConfigData && agentConfigData.item ? ( + + {agentConfigData.item.name} + + ) : ( + + + } + />{' '} + {agent.config_id} + + ), + }, + ]; + + return ( + <> + + + +

+ +

+ + + + + {unenrollAgentsPrompt => ( + { + unenrollAgentsPrompt([agent.id], 1, refreshAgent); + }} + > + + + )} + + + + + + {items.map((item, idx) => ( + + {item.description} + + ))} + + metadataFlyout.show()}>View metadata + + + {metadataFlyout.isVisible && } + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/index.ts new file mode 100644 index 00000000000000..9dffa54aeaf7fc --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/index.ts @@ -0,0 +1,7 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export { AgentEventsTable } from './agent_events_table'; +export { AgentDetailSection } from './details_section'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/metadata_flyout.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/metadata_flyout.tsx new file mode 100644 index 00000000000000..ee43385e601c28 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/metadata_flyout.tsx @@ -0,0 +1,76 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React from 'react'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiTitle, + EuiSpacer, + EuiDescriptionList, + EuiFlyout, + EuiFlyoutHeader, + EuiFlyoutBody, + EuiHorizontalRule, +} from '@elastic/eui'; +import { MetadataForm } from './metadata_form'; +import { Agent } from '../../../../types'; + +interface Props { + agent: Agent; + flyout: { hide: () => void }; +} +export const AgentMetadataFlyout: React.FunctionComponent = ({ agent, flyout }) => { + const mapMetadata = (obj: { [key: string]: string } | undefined) => { + return Object.keys(obj || {}).map(key => ({ + title: key, + description: obj ? obj[key] : '', + })); + }; + + const localItems = mapMetadata(agent.local_metadata); + const userProvidedItems = mapMetadata(agent.user_provided_metadata); + + return ( + flyout.hide()} size="s" aria-labelledby="flyoutTitle"> + + +

+ +

+ + + + +

+ +

+ + + + + +

+ +

+ + + + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/metadata_form.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/metadata_form.tsx new file mode 100644 index 00000000000000..ce28bbdc590b06 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/components/metadata_form.tsx @@ -0,0 +1,160 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useState } from 'react'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { + EuiButtonEmpty, + EuiPopover, + EuiFormRow, + EuiButton, + EuiFlexItem, + EuiFieldText, + EuiFlexGroup, + EuiForm, +} from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { AxiosError } from 'axios'; +import { useAgentRefresh } from '../hooks'; +import { useInput, sendRequest } from '../../../../hooks'; +import { Agent } from '../../../../types'; +import { agentRouteService } from '../../../../services'; + +function useAddMetadataForm(agent: Agent, done: () => void) { + const refreshAgent = useAgentRefresh(); + const keyInput = useInput(); + const valueInput = useInput(); + const [state, setState] = useState<{ + isLoading: boolean; + error: null | string; + }>({ + isLoading: false, + error: null, + }); + + function clearInputs() { + keyInput.clear(); + valueInput.clear(); + } + + function setError(error: AxiosError) { + setState({ + isLoading: false, + error: error.response && error.response.data ? error.response.data.message : error.message, + }); + } + + async function success() { + await refreshAgent(); + setState({ + isLoading: false, + error: null, + }); + clearInputs(); + done(); + } + + return { + state, + onSubmit: async (e: React.FormEvent | React.MouseEvent) => { + e.preventDefault(); + setState({ + ...state, + isLoading: true, + }); + + try { + const { error } = await sendRequest({ + path: agentRouteService.getUpdatePath(agent.id), + method: 'put', + body: JSON.stringify({ + user_provided_metadata: { + ...agent.user_provided_metadata, + [keyInput.value]: valueInput.value, + }, + }), + }); + + if (error) { + throw error; + } + await success(); + } catch (error) { + setError(error); + } + }, + inputs: { + keyInput, + valueInput, + }, + }; +} + +export const MetadataForm: React.FunctionComponent<{ agent: Agent }> = ({ agent }) => { + const [isOpen, setOpen] = useState(false); + + const form = useAddMetadataForm(agent, () => { + setOpen(false); + }); + const { keyInput, valueInput } = form.inputs; + + const button = ( + setOpen(true)} color={'text'}> + + + ); + return ( + <> + setOpen(false)} + initialFocus="[id=fleet-details-metadata-form]" + > +
+ + + + + + + + + + + + + + + + + + + + + +
+ + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/hooks/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/hooks/index.ts new file mode 100644 index 00000000000000..c4e8ea13a89e0f --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/hooks/index.ts @@ -0,0 +1,6 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export { useAgentRefresh, AgentRefreshContext } from './use_agent'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/hooks/use_agent.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/hooks/use_agent.tsx new file mode 100644 index 00000000000000..78b04260f384f3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/hooks/use_agent.tsx @@ -0,0 +1,12 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; + +export const AgentRefreshContext = React.createContext({ refresh: () => {} }); + +export function useAgentRefresh() { + return React.useContext(AgentRefreshContext).refresh; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/index.tsx new file mode 100644 index 00000000000000..f8ba829135f3cf --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_details_page/index.tsx @@ -0,0 +1,67 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { useRouteMatch } from 'react-router-dom'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { EuiCallOut, EuiText } from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { AgentEventsTable, AgentDetailSection } from './components'; +import { AgentRefreshContext } from './hooks'; +import { Loading } from '../../../components'; +import { useGetOneAgent } from '../../../hooks'; +import { WithHeaderLayout } from '../../../layouts'; + +export const AgentDetailsPage: React.FunctionComponent = () => { + const { + params: { agentId }, + } = useRouteMatch(); + const agentRequest = useGetOneAgent(agentId, { + pollIntervalMs: 5000, + }); + + if (agentRequest.isLoading && agentRequest.isInitialRequest) { + return ; + } + + if (agentRequest.error) { + return ( + + +

+ {agentRequest.error.message} +

+ + + ); + } + + if (!agentRequest.data) { + return ( + + + + ); + } + + const agent = agentRequest.data.item; + + return ( + agentRequest.sendRequest() }}> + }> + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/index.tsx new file mode 100644 index 00000000000000..9c14a2e9dfed13 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/index.tsx @@ -0,0 +1,77 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { + EuiFlyout, + EuiFlyoutBody, + EuiFlyoutHeader, + EuiSpacer, + EuiTitle, + EuiFlexGroup, + EuiFlexItem, + EuiButtonEmpty, + EuiButton, + EuiFlyoutFooter, +} from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { AgentConfig } from '../../../../../types'; +import { APIKeySelection } from './key_selection'; +import { EnrollmentInstructions } from './instructions'; + +interface Props { + onClose: () => void; + agentConfigs: AgentConfig[]; +} + +export const AgentEnrollmentFlyout: React.FunctionComponent = ({ + onClose, + agentConfigs = [], +}) => { + const [selectedAPIKeyId, setSelectedAPIKeyId] = useState(null); + + return ( + + + +

+ +

+ + + + setSelectedAPIKeyId(keyId)} + /> + + + + + + + + + + + + + + + + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/instructions.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/instructions.tsx new file mode 100644 index 00000000000000..97434d21788529 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/instructions.tsx @@ -0,0 +1,126 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { i18n } from '@kbn/i18n'; +import { EuiSpacer, EuiText, EuiButtonGroup, EuiSteps } from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { useEnrollmentApiKey } from '../enrollment_api_keys'; +import { ShellEnrollmentInstructions, ManualInstructions } from '../enrollment_instructions'; +import { useCore, useGetAgents } from '../../../../../hooks'; +import { Loading } from '../../../components'; + +interface Props { + selectedAPIKeyId: string | null; +} +function useNewEnrolledAgents() { + // New enrolled agents + const [timestamp] = useState(new Date().toISOString()); + const agentsRequest = useGetAgents( + { + perPage: 100, + page: 1, + showInactive: false, + }, + { + pollIntervalMs: 3000, + } + ); + return React.useMemo(() => { + if (!agentsRequest.data) { + return []; + } + + return agentsRequest.data.list.filter(agent => agent.enrolled_at >= timestamp); + }, [agentsRequest.data, timestamp]); +} + +export const EnrollmentInstructions: React.FunctionComponent = ({ selectedAPIKeyId }) => { + const core = useCore(); + const [installType, setInstallType] = useState<'quickInstall' | 'manual'>('quickInstall'); + + const apiKey = useEnrollmentApiKey(selectedAPIKeyId); + + const newAgents = useNewEnrolledAgents(); + if (!apiKey.data) { + return null; + } + + return ( + <> + { + setInstallType(installType === 'manual' ? 'quickInstall' : 'manual'); + }} + buttonSize="m" + isFullWidth + /> + + {installType === 'manual' ? ( + + ) : ( + + ), + }, + { + title: i18n.translate('xpack.ingestManager.agentEnrollment.stepTestAgents', { + defaultMessage: 'Test Agents', + }), + children: ( + + {!newAgents.length ? ( + <> + + + + ) : ( + <> + + + )} + + ), + }, + ]} + /> + )} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/key_selection.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/key_selection.tsx new file mode 100644 index 00000000000000..89801bc6bee1ed --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/agent_enrollment_flyout/key_selection.tsx @@ -0,0 +1,205 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { i18n } from '@kbn/i18n'; +import { + EuiFlexGroup, + EuiFlexItem, + EuiFormRow, + EuiSelect, + EuiSpacer, + EuiText, + EuiLink, + EuiFieldText, +} from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { useEnrollmentApiKeys } from '../enrollment_api_keys'; +import { AgentConfig } from '../../../../../types'; +import { useInput, useCore, sendRequest } from '../../../../../hooks'; +import { enrollmentAPIKeyRouteService } from '../../../../../services'; + +interface Props { + onKeyChange: (keyId: string | null) => void; + agentConfigs: AgentConfig[]; +} + +function useCreateApiKeyForm(configId: string | null, onSuccess: (keyId: string) => void) { + const { notifications } = useCore(); + const [isLoading, setIsLoading] = useState(false); + const apiKeyNameInput = useInput(''); + + const onSubmit = async (event: React.FormEvent) => { + event.preventDefault(); + setIsLoading(true); + try { + const res = await sendRequest({ + method: 'post', + path: enrollmentAPIKeyRouteService.getCreatePath(), + body: JSON.stringify({ + name: apiKeyNameInput.value, + config_id: configId, + }), + }); + apiKeyNameInput.clear(); + setIsLoading(false); + onSuccess(res.data.item.id); + } catch (err) { + notifications.toasts.addError(err as Error, { + title: 'Error', + }); + setIsLoading(false); + } + }; + + return { + isLoading, + onSubmit, + apiKeyNameInput, + }; +} + +export const APIKeySelection: React.FunctionComponent = ({ onKeyChange, agentConfigs }) => { + const enrollmentAPIKeysRequest = useEnrollmentApiKeys({ + currentPage: 1, + pageSize: 1000, + }); + + const [selectedState, setSelectedState] = useState<{ + agentConfigId: string | null; + enrollmentAPIKeyId: string | null; + }>({ + agentConfigId: agentConfigs.length ? agentConfigs[0].id : null, + enrollmentAPIKeyId: null, + }); + const filteredEnrollmentAPIKeys = React.useMemo(() => { + if (!selectedState.agentConfigId || !enrollmentAPIKeysRequest.data) { + return []; + } + + return enrollmentAPIKeysRequest.data.list.filter( + key => key.config_id === selectedState.agentConfigId + ); + }, [enrollmentAPIKeysRequest.data, selectedState.agentConfigId]); + + // Select first API key when config change + React.useEffect(() => { + if (!selectedState.enrollmentAPIKeyId && filteredEnrollmentAPIKeys.length > 0) { + const enrollmentAPIKeyId = filteredEnrollmentAPIKeys[0].id; + setSelectedState({ + agentConfigId: selectedState.agentConfigId, + enrollmentAPIKeyId, + }); + onKeyChange(enrollmentAPIKeyId); + } + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [filteredEnrollmentAPIKeys, selectedState.enrollmentAPIKeyId, selectedState.agentConfigId]); + + const [showAPIKeyForm, setShowAPIKeyForm] = useState(false); + const apiKeyForm = useCreateApiKeyForm(selectedState.agentConfigId, async (keyId: string) => { + const res = await enrollmentAPIKeysRequest.refresh(); + setSelectedState({ + ...selectedState, + enrollmentAPIKeyId: res.data?.list.find(key => key.id === keyId)?.id ?? null, + }); + setShowAPIKeyForm(false); + }); + + return ( + <> + + + + + + + + } + > + ({ + value: agentConfig.id, + text: agentConfig.name, + }))} + value={selectedState.agentConfigId || undefined} + onChange={e => + setSelectedState({ + agentConfigId: e.target.value, + enrollmentAPIKeyId: null, + }) + } + /> + + + + + } + labelAppend={ + + setShowAPIKeyForm(!showAPIKeyForm)} color="primary"> + {showAPIKeyForm ? ( + + ) : ( + + )} + + + } + > + {showAPIKeyForm ? ( +
+ + + ) : ( + ({ + value: key.id, + text: key.name, + }))} + value={selectedState.enrollmentAPIKeyId || undefined} + onChange={e => { + setSelectedState({ + ...selectedState, + enrollmentAPIKeyId: e.target.value, + }); + onKeyChange(selectedState.enrollmentAPIKeyId); + }} + /> + )} +
+ + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/donut_chart.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/donut_chart.tsx new file mode 100644 index 00000000000000..8cd363576aa85c --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/donut_chart.tsx @@ -0,0 +1,65 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { useEffect, useRef } from 'react'; +import d3 from 'd3'; +import { EuiFlexItem } from '@elastic/eui'; + +interface DonutChartProps { + data: { + [key: string]: number; + }; + height: number; + width: number; +} + +export const DonutChart = ({ height, width, data }: DonutChartProps) => { + const chartElement = useRef(null); + + useEffect(() => { + if (chartElement.current !== null) { + // we must remove any existing paths before painting + d3.selectAll('g').remove(); + const svgElement = d3 + .select(chartElement.current) + .append('g') + .attr('transform', `translate(${width / 2}, ${height / 2})`); + const color = d3.scale + .ordinal() + // @ts-ignore + .domain(data) + .range(['#017D73', '#98A2B3', '#BD271E']); + const pieGenerator = d3.layout + .pie() + .value(({ value }: any) => value) + // these start/end angles will reverse the direction of the pie, + // which matches our design + .startAngle(2 * Math.PI) + .endAngle(0); + + svgElement + .selectAll('g') + // @ts-ignore + .data(pieGenerator(d3.entries(data))) + .enter() + .append('path') + .attr( + 'd', + // @ts-ignore attr does not expect a param of type Arc but it behaves as desired + d3.svg + .arc() + .innerRadius(width * 0.36) + .outerRadius(Math.min(width, height) / 2) + ) + .attr('fill', (d: any) => color(d.data.key)); + } + }, [data, height, width]); + return ( + + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/confirm_delete_modal.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/confirm_delete_modal.tsx new file mode 100644 index 00000000000000..8ce20a85e14b82 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/confirm_delete_modal.tsx @@ -0,0 +1,46 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React from 'react'; +import { EuiOverlayMask, EuiConfirmModal } from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; + +export const ConfirmDeleteModal: React.FunctionComponent<{ + onConfirm: () => void; + onCancel: () => void; + apiKeyId: string; +}> = ({ onConfirm, onCancel, apiKeyId }) => { + return ( + + + } + onCancel={onCancel} + onConfirm={onConfirm} + cancelButtonText={ + + } + confirmButtonText={ + + } + buttonColor="danger" + /> + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/create_api_key_form.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/create_api_key_form.tsx new file mode 100644 index 00000000000000..009080a4da1860 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/create_api_key_form.tsx @@ -0,0 +1,94 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React from 'react'; +import { + EuiFlexGroup, + EuiFlexItem, + EuiFormRow, + EuiFieldText, + EuiButton, + EuiSelect, +} from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { useInput, sendRequest } from '../../../../../hooks'; +import { useConfigs } from './hooks'; +import { enrollmentAPIKeyRouteService } from '../../../../../services'; + +export const CreateApiKeyForm: React.FunctionComponent<{ onChange: () => void }> = ({ + onChange, +}) => { + const { data: configs } = useConfigs(); + const { inputs, onSubmit, submitted } = useCreateApiKey(() => onChange()); + + return ( + + + + + + + + + ({ + value: config.id, + text: config.name, + }))} + /> + + + + + onSubmit()}> + + + + + + ); +}; + +function useCreateApiKey(onSuccess: () => void) { + const [submitted, setSubmitted] = React.useState(false); + const inputs = { + nameInput: useInput(), + configIdInput: useInput('default'), + }; + + const onSubmit = async () => { + setSubmitted(true); + await sendRequest({ + method: 'post', + path: enrollmentAPIKeyRouteService.getCreatePath(), + body: JSON.stringify({ + name: inputs.nameInput.value, + config_id: inputs.configIdInput.value, + }), + }); + setSubmitted(false); + onSuccess(); + }; + + return { + inputs, + onSubmit, + submitted, + }; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/hooks.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/hooks.tsx new file mode 100644 index 00000000000000..957e1201fd43b5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/hooks.tsx @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + Pagination, + useGetAgentConfigs, + useGetEnrollmentAPIKeys, + useGetOneEnrollmentAPIKey, +} from '../../../../../hooks'; + +export function useEnrollmentApiKeys(pagination: Pagination) { + const request = useGetEnrollmentAPIKeys(); + + return { + data: request.data, + isLoading: request.isLoading, + refresh: () => request.sendRequest(), + }; +} + +export function useConfigs() { + const request = useGetAgentConfigs(); + + return { + data: request.data ? request.data.items : [], + isLoading: request.isLoading, + }; +} + +export function useEnrollmentApiKey(apiKeyId: string | null) { + const request = useGetOneEnrollmentAPIKey(apiKeyId as string); + + return { + data: request.data, + isLoading: request.isLoading, + }; +} diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/index.tsx new file mode 100644 index 00000000000000..19957e7827680b --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_api_keys/index.tsx @@ -0,0 +1,152 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { EuiBasicTable, EuiButtonEmpty, EuiSpacer, EuiPopover, EuiLink } from '@elastic/eui'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { i18n } from '@kbn/i18n'; +import { usePagination, sendRequest } from '../../../../../hooks'; +import { useEnrollmentApiKeys, useEnrollmentApiKey } from './hooks'; +import { ConfirmDeleteModal } from './confirm_delete_modal'; +import { CreateApiKeyForm } from './create_api_key_form'; +import { EnrollmentAPIKey } from '../../../../../types'; +import { useCapabilities } from '../../../../../hooks'; +import { enrollmentAPIKeyRouteService } from '../../../../../services'; +export { useEnrollmentApiKeys, useEnrollmentApiKey } from './hooks'; + +export const EnrollmentApiKeysTable: React.FunctionComponent<{ + onChange: () => void; +}> = ({ onChange }) => { + const [confirmDeleteApiKeyId, setConfirmDeleteApiKeyId] = useState(null); + const { pagination } = usePagination(); + const { data, isLoading, refresh } = useEnrollmentApiKeys(pagination); + + const columns: any[] = [ + { + field: 'name', + name: i18n.translate('xpack.ingestManager.apiKeysList.nameColumnTitle', { + defaultMessage: 'Name', + }), + width: '300px', + }, + { + field: 'config_id', + name: i18n.translate('xpack.ingestManager.apiKeysList.configColumnTitle', { + defaultMessage: 'Config', + }), + width: '100px', + }, + { + field: null, + name: i18n.translate('xpack.ingestManager.apiKeysList.apiKeyColumnTitle', { + defaultMessage: 'API Key', + }), + render: (key: EnrollmentAPIKey) => , + }, + { + field: null, + width: '50px', + render: (key: EnrollmentAPIKey) => { + return ( + setConfirmDeleteApiKeyId(key.id)} iconType={'trash'} /> + ); + }, + }, + ]; + + return ( + <> + {confirmDeleteApiKeyId && ( + setConfirmDeleteApiKeyId(null)} + onConfirm={async () => { + await sendRequest({ + method: 'delete', + path: enrollmentAPIKeyRouteService.getDeletePath(confirmDeleteApiKeyId), + }); + setConfirmDeleteApiKeyId(null); + refresh(); + }} + /> + )} + + } + items={data ? data.list : []} + itemId="id" + columns={columns} + /> + + { + refresh(); + onChange(); + }} + /> + + ); +}; + +export const CreateApiKeyButton: React.FunctionComponent<{ onChange: () => void }> = ({ + onChange, +}) => { + const hasWriteCapabilites = useCapabilities().write; + const [isOpen, setIsOpen] = React.useState(false); + + return ( + setIsOpen(true)} color="primary"> + + + } + isOpen={isOpen} + closePopover={() => setIsOpen(false)} + > + { + setIsOpen(false); + onChange(); + }} + /> + + ); + return <>; +}; + +const ApiKeyField: React.FunctionComponent<{ apiKeyId: string }> = ({ apiKeyId }) => { + const [visible, setVisible] = useState(false); + const { data } = useEnrollmentApiKey(apiKeyId); + + return ( + <> + {visible && data ? data.item.api_key : '••••••••••••••••••••••••••••'} + setVisible(!visible)}> + {visible ? ( + + ) : ( + + )} + {' '} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/index.tsx new file mode 100644 index 00000000000000..34233a00e630ab --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/index.tsx @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export { ShellEnrollmentInstructions } from './shell'; +export { ManualInstructions } from './manual'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/manual/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/manual/index.tsx new file mode 100644 index 00000000000000..b1da4583b74cc2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/manual/index.tsx @@ -0,0 +1,36 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiText, EuiSpacer } from '@elastic/eui'; +import React from 'react'; + +export const ManualInstructions: React.FunctionComponent = () => { + return ( + <> + + Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam vestibulum ullamcorper + turpis vitae interdum. Maecenas orci magna, auctor volutpat pellentesque eu, consectetur id + est. Nunc orci lacus, condimentum vel congue ac, fringilla eget tortor. Aliquam blandit, + nisi et congue euismod, leo lectus blandit risus, eu blandit erat metus sit amet leo. Nam + dictum lobortis condimentum. + + + + Vivamus sem sapien, dictum eu tellus vel, rutrum aliquam purus. Cras quis cursus nibh. + Aliquam fermentum ipsum nec turpis luctus lobortis. Nulla facilisi. Etiam nec fringilla + urna, sed vehicula ipsum. Quisque vel pellentesque lorem, at egestas enim. Nunc semper elit + lectus, in sollicitudin erat fermentum in. Pellentesque tempus massa eget purus pharetra + blandit. + + + + Mauris congue enim nulla, nec semper est posuere non. Donec et eros eu nisi gravida + malesuada eget in velit. Morbi placerat semper euismod. Suspendisse potenti. Morbi quis + porta erat, quis cursus nulla. Aenean mauris lorem, mollis in mattis et, lobortis a lectus. + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/shell/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/shell/index.tsx new file mode 100644 index 00000000000000..04e84902bc9d40 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/enrollment_instructions/shell/index.tsx @@ -0,0 +1,90 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { + EuiButtonEmpty, + EuiContextMenuItem, + EuiContextMenuPanel, + EuiCopy, + EuiFieldText, + EuiPopover, +} from '@elastic/eui'; +import { EnrollmentAPIKey } from '../../../../../../types'; + +// No need for i18n as these are platform names +const PLATFORMS = { + macos: 'macOS', + windows: 'Windows', + linux: 'Linux', +}; + +interface Props { + kibanaUrl: string; + kibanaCASha256?: string; + apiKey: EnrollmentAPIKey; +} + +export const ShellEnrollmentInstructions: React.FunctionComponent = ({ + kibanaUrl, + kibanaCASha256, + apiKey, +}) => { + // Platform state + const [currentPlatform, setCurrentPlatform] = useState('macos'); + const [isPlatformOptionsOpen, setIsPlatformOptionsOpen] = useState(false); + + // Build quick installation command + const quickInstallInstructions = `${ + kibanaCASha256 ? `CA_SHA256=${kibanaCASha256} ` : '' + }API_KEY=${ + apiKey.api_key + } sh -c "$(curl ${kibanaUrl}/api/ingest_manager/fleet/install/${currentPlatform})"`; + + return ( + <> + setIsPlatformOptionsOpen(true)} + > + {PLATFORMS[currentPlatform]} + + } + isOpen={isPlatformOptionsOpen} + closePopover={() => setIsPlatformOptionsOpen(false)} + > + ( + { + setCurrentPlatform(platform as typeof currentPlatform); + setIsPlatformOptionsOpen(false); + }} + > + {name} + + ))} + /> + + } + append={ + + {copy => } + + } + /> + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/index.ts new file mode 100644 index 00000000000000..c82c82db6f7138 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/components/index.ts @@ -0,0 +1,6 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +export { AgentEnrollmentFlyout } from './agent_enrollment_flyout'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/index.scss b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/index.scss new file mode 100644 index 00000000000000..10e809c5f5566b --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/index.scss @@ -0,0 +1,6 @@ +.fleet__agentList__table .euiTableFooterCell { + .euiTableCellContent, + .euiTableCellContent__text { + overflow: visible; + } +} \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/index.tsx new file mode 100644 index 00000000000000..acf09dedc25f75 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/agent_list_page/index.tsx @@ -0,0 +1,729 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState, useCallback } from 'react'; +import styled, { CSSProperties } from 'styled-components'; +import { + EuiBasicTable, + EuiButton, + EuiEmptyPrompt, + EuiFilterButton, + EuiFilterGroup, + EuiFilterSelectItem, + EuiFlexGroup, + EuiFlexItem, + EuiLink, + EuiPopover, + EuiSpacer, + EuiSwitch, + EuiText, + EuiTitle, + EuiStat, + EuiI18nNumber, + EuiHealth, + EuiButtonIcon, + EuiContextMenuPanel, + EuiContextMenuItem, +} from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage, FormattedRelative } from '@kbn/i18n/react'; +import { AgentEnrollmentFlyout } from './components'; +import { WithHeaderLayout } from '../../../layouts'; +import { Agent } from '../../../types'; +import { + usePagination, + useCapabilities, + useGetAgentConfigs, + useGetAgents, + useUrlParams, + useLink, +} from '../../../hooks'; +import { ConnectedLink } from '../components'; +import { SearchBar } from '../../../components/search_bar'; +import { AgentHealth } from '../components/agent_health'; +import { AgentUnenrollProvider } from '../components/agent_unenroll_provider'; +import { DonutChart } from './components/donut_chart'; +import { useGetAgentStatus } from '../../agent_config/details_page/hooks'; +import { AgentStatusKueryHelper } from '../../../services'; +import { FLEET_AGENT_DETAIL_PATH, AGENT_CONFIG_DETAILS_PATH } from '../../../constants'; + +const Divider = styled.div` + width: 0; + height: 100%; + border-left: ${props => props.theme.eui.euiBorderThin}; + height: 45px; +`; +const NO_WRAP_TRUNCATE_STYLE: CSSProperties = Object.freeze({ + overflow: 'hidden', + textOverflow: 'ellipsis', + whiteSpace: 'nowrap', +}); +const REFRESH_INTERVAL_MS = 5000; + +const statusFilters = [ + { + status: 'online', + label: i18n.translate('xpack.ingestManager.agentList.statusOnlineFilterText', { + defaultMessage: 'Online', + }), + }, + { + status: 'offline', + label: i18n.translate('xpack.ingestManager.agentList.statusOfflineFilterText', { + defaultMessage: 'Offline', + }), + }, + , + { + status: 'error', + label: i18n.translate('xpack.ingestManager.agentList.statusErrorFilterText', { + defaultMessage: 'Error', + }), + }, +] as Array<{ label: string; status: string }>; + +const RowActions = React.memo<{ agent: Agent; refresh: () => void }>(({ agent, refresh }) => { + const hasWriteCapabilites = useCapabilities().write; + const DETAILS_URI = useLink(FLEET_AGENT_DETAIL_PATH); + const [isOpen, setIsOpen] = useState(false); + const handleCloseMenu = useCallback(() => setIsOpen(false), [setIsOpen]); + const handleToggleMenu = useCallback(() => setIsOpen(!isOpen), [isOpen]); + + return ( + + } + isOpen={isOpen} + closePopover={handleCloseMenu} + > + + + , + + + {unenrollAgentsPrompt => ( + { + unenrollAgentsPrompt([agent.id], 1, () => { + refresh(); + }); + }} + > + + + )} + , + ]} + /> + + ); +}); + +export const AgentListPage: React.FunctionComponent<{}> = () => { + const defaultKuery: string = (useUrlParams().urlParams.kuery as string) || ''; + const hasWriteCapabilites = useCapabilities().write; + // Agent data states + const [showInactive, setShowInactive] = useState(false); + + // Table and search states + const [search, setSearch] = useState(defaultKuery); + const { pagination, pageSizeOptions, setPagination } = usePagination(); + const [selectedAgents, setSelectedAgents] = useState([]); + const [areAllAgentsSelected, setAreAllAgentsSelected] = useState(false); + + // Configs state (for filtering) + const [isConfigsFilterOpen, setIsConfigsFilterOpen] = useState(false); + const [selectedConfigs, setSelectedConfigs] = useState([]); + // Status for filtering + const [isStatusFilterOpen, setIsStatutsFilterOpen] = useState(false); + const [selectedStatus, setSelectedStatus] = useState([]); + + // Add a config id to current search + const addConfigFilter = (configId: string) => { + setSelectedConfigs([...selectedConfigs, configId]); + }; + + // Remove a config id from current search + const removeConfigFilter = (configId: string) => { + setSelectedConfigs(selectedConfigs.filter(config => config !== configId)); + }; + + // Agent enrollment flyout state + const [isEnrollmentFlyoutOpen, setIsEnrollmentFlyoutOpen] = useState(false); + + let kuery = search.trim(); + if (selectedConfigs.length) { + if (kuery) { + kuery = `(${kuery}) and`; + } + kuery = `${kuery} agents.config_id : (${selectedConfigs + .map(config => `"${config}"`) + .join(' or ')})`; + } + + if (selectedStatus.length) { + if (kuery) { + kuery = `(${kuery}) and`; + } + + kuery = selectedStatus + .map(status => { + switch (status) { + case 'online': + return AgentStatusKueryHelper.buildKueryForOnlineAgents(); + case 'offline': + return AgentStatusKueryHelper.buildKueryForOfflineAgents(); + case 'error': + return AgentStatusKueryHelper.buildKueryForErrorAgents(); + } + + return ''; + }) + .join(' or '); + } + + const agentStatusRequest = useGetAgentStatus(undefined, { + pollIntervalMs: REFRESH_INTERVAL_MS, + }); + const agentStatus = agentStatusRequest.data?.results; + + const agentsRequest = useGetAgents( + { + page: pagination.currentPage, + perPage: pagination.pageSize, + kuery: kuery && kuery !== '' ? kuery : undefined, + showInactive, + }, + { + pollIntervalMs: REFRESH_INTERVAL_MS, + } + ); + + const agents = agentsRequest.data ? agentsRequest.data.list : []; + const totalAgents = agentsRequest.data ? agentsRequest.data.total : 0; + const { isLoading } = agentsRequest; + + const agentConfigsRequest = useGetAgentConfigs({ + page: 1, + perPage: 1000, + }); + + const agentConfigs = agentConfigsRequest.data ? agentConfigsRequest.data.items : []; + const { isLoading: isAgentConfigsLoading } = agentConfigsRequest; + + const CONFIG_DETAILS_URI = useLink(AGENT_CONFIG_DETAILS_PATH); + + const columns = [ + { + field: 'local_metadata.host', + name: i18n.translate('xpack.ingestManager.agentList.hostColumnTitle', { + defaultMessage: 'Host', + }), + render: (host: string, agent: Agent) => ( + + {host} + + ), + footer: () => { + if (selectedAgents.length === agents.length && totalAgents > selectedAgents.length) { + return areAllAgentsSelected ? ( + setAreAllAgentsSelected(false)}> + + + ), + }} + /> + ) : ( + setAreAllAgentsSelected(true)}> + + + ), + }} + /> + ); + } + return null; + }, + }, + { + field: 'active', + name: i18n.translate('xpack.ingestManager.agentList.statusColumnTitle', { + defaultMessage: 'Status', + }), + render: (active: boolean, agent: any) => , + }, + { + field: 'config_id', + name: i18n.translate('xpack.ingestManager.agentList.configColumnTitle', { + defaultMessage: 'Configuration', + }), + render: (configId: string) => { + const configName = agentConfigs.find(p => p.id === configId)?.name; + return ( + + + + {configName || configId} + + + + + + + + + ); + }, + }, + { + field: 'local_metadata.agent_version', + name: i18n.translate('xpack.ingestManager.agentList.versionTitle', { + defaultMessage: 'Version', + }), + }, + { + field: 'last_checkin', + name: i18n.translate('xpack.ingestManager.agentList.lastCheckinTitle', { + defaultMessage: 'Last activity', + }), + render: (lastCheckin: string, agent: any) => + lastCheckin ? : null, + }, + { + name: i18n.translate('xpack.ingestManager.agentList.actionsColumnTitle', { + defaultMessage: 'Actions', + }), + actions: [ + { + render: (agent: Agent) => { + return agentsRequest.sendRequest()} />; + }, + }, + ], + width: '100px', + }, + ]; + + const emptyPrompt = ( + + + + } + actions={ + hasWriteCapabilites ? ( + setIsEnrollmentFlyoutOpen(true)}> + + + ) : null + } + /> + ); + const headerRightColumn = ( + + + } + description={i18n.translate('xpack.ingestManager.agentListStatus.totalLabel', { + defaultMessage: 'Agents', + })} + /> + + + + + + + {' '} + + + } + description={i18n.translate('xpack.ingestManager.agentListStatus.onlineLabel', { + defaultMessage: 'Online', + })} + /> + + + } + description={i18n.translate('xpack.ingestManager.agentListStatus.offlineLabel', { + defaultMessage: 'Offline', + })} + /> + + + } + description={i18n.translate('xpack.ingestManager.agentListStatus.errorLabel', { + defaultMessage: 'Error', + })} + /> + + {hasWriteCapabilites && ( + <> + + + + + setIsEnrollmentFlyoutOpen(true)}> + + + + + )} + + ); + const headerLeftColumn = ( + + + +

+ +

+ + + + +

+ +

+ + + + ); + + return ( + + {isEnrollmentFlyoutOpen ? ( + setIsEnrollmentFlyoutOpen(false)} + /> + ) : null} + +

+ +

+ + + + + + + + + + + + setShowInactive(!showInactive)} + /> + + + + + + {selectedAgents.length ? ( + + + {unenrollAgentsPrompt => ( + { + unenrollAgentsPrompt( + areAllAgentsSelected ? search : selectedAgents.map(agent => agent.id), + areAllAgentsSelected ? totalAgents : selectedAgents.length, + () => { + // Reload agents if on first page and no search query, otherwise + // reset to first page and reset search, which will trigger a reload + if (pagination.currentPage === 1 && !search) { + agentsRequest.sendRequest(); + } else { + setPagination({ + ...pagination, + currentPage: 1, + }); + setSearch(''); + } + + setAreAllAgentsSelected(false); + setSelectedAgents([]); + } + ); + }} + > + + + )} + + + ) : null} + + + + { + setPagination({ + ...pagination, + currentPage: 1, + }); + setSearch(newSearch); + }} + fieldPrefix="agents" + /> + + + + setIsStatutsFilterOpen(!isStatusFilterOpen)} + isSelected={isStatusFilterOpen} + hasActiveFilters={selectedStatus.length > 0} + numActiveFilters={selectedStatus.length} + disabled={isAgentConfigsLoading} + > + + + } + isOpen={isStatusFilterOpen} + closePopover={() => setIsStatutsFilterOpen(false)} + panelPaddingSize="none" + > +
+ {statusFilters.map(({ label, status }, idx) => ( + { + if (selectedStatus.includes(status)) { + setSelectedStatus([...selectedStatus.filter(s => s !== status)]); + } else { + setSelectedStatus([...selectedStatus, status]); + } + }} + > + {label} + + ))} +
+ + setIsConfigsFilterOpen(!isConfigsFilterOpen)} + isSelected={isConfigsFilterOpen} + hasActiveFilters={selectedConfigs.length > 0} + numActiveFilters={selectedConfigs.length} + numFilters={agentConfigs.length} + disabled={isAgentConfigsLoading} + > + + + } + isOpen={isConfigsFilterOpen} + closePopover={() => setIsConfigsFilterOpen(false)} + panelPaddingSize="none" + > +
+ {agentConfigs.map((config, index) => ( + { + if (selectedConfigs.includes(config.id)) { + removeConfigFilter(config.id); + } else { + addConfigFilter(config.id); + } + }} + > + {config.name} + + ))} +
+ + + + + + + + + + className="fleet__agentList__table" + loading={isLoading && agentsRequest.isInitialRequest} + hasActions={true} + noItemsMessage={ + isLoading ? ( + + ) : !search.trim() && selectedConfigs.length === 0 && totalAgents === 0 ? ( + emptyPrompt + ) : ( + setSearch('')}> + + + ), + }} + /> + ) + } + items={totalAgents ? agents : []} + itemId="id" + columns={columns} + isSelectable={true} + selection={{ + selectable: (agent: Agent) => agent.active, + onSelectionChange: (newSelectedAgents: Agent[]) => { + setSelectedAgents(newSelectedAgents); + setAreAllAgentsSelected(false); + }, + }} + pagination={{ + pageIndex: pagination.currentPage - 1, + pageSize: pagination.pageSize, + totalItemCount: totalAgents, + pageSizeOptions, + }} + onChange={({ page }: { page: { index: number; size: number } }) => { + const newPagination = { + ...pagination, + currentPage: page.index + 1, + pageSize: page.size, + }; + setPagination(newPagination); + }} + /> + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/agent_health.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/agent_health.tsx new file mode 100644 index 00000000000000..181ebe35042227 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/agent_health.tsx @@ -0,0 +1,105 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { FormattedMessage, FormattedRelative } from '@kbn/i18n/react'; +import { EuiHealth, EuiToolTip } from '@elastic/eui'; +import { Agent } from '../../../types'; + +interface Props { + agent: Agent; +} + +const Status = { + Online: ( + + + + ), + Offline: ( + + + + ), + Inactive: ( + + + + ), + Warning: ( + + + + ), + Error: ( + + + + ), +}; + +function getStatusComponent(agent: Agent): React.ReactElement { + switch (agent.status) { + case 'error': + return Status.Error; + case 'inactive': + return Status.Inactive; + case 'offline': + return Status.Offline; + case 'warning': + return Status.Warning; + default: + return Status.Online; + } +} + +export const AgentHealth: React.FunctionComponent = ({ agent }) => { + const { last_checkin: lastCheckIn } = agent; + const msLastCheckIn = new Date(lastCheckIn || 0).getTime(); + + return ( + + , + }} + /> + {agent.current_error_events.map((event, idx) => ( +

{event.message}

+ ))} + + ) : ( + + ) + } + > + {getStatusComponent(agent)} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/agent_unenroll_provider.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/agent_unenroll_provider.tsx new file mode 100644 index 00000000000000..25499495a78979 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/agent_unenroll_provider.tsx @@ -0,0 +1,184 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React, { Fragment, useRef, useState } from 'react'; +import { EuiConfirmModal, EuiOverlayMask } from '@elastic/eui'; +import { i18n } from '@kbn/i18n'; +import { FormattedMessage } from '@kbn/i18n/react'; +import { useCore, sendRequest } from '../../../hooks'; +import { PostAgentUnenrollResponse } from '../../../types'; +import { agentRouteService } from '../../../services'; + +interface Props { + children: (unenrollAgents: UnenrollAgents) => React.ReactElement; +} + +export type UnenrollAgents = ( + agents: string[] | string, + agentsCount: number, + onSuccess?: OnSuccessCallback +) => void; + +type OnSuccessCallback = (agentsUnenrolled: string[]) => void; + +export const AgentUnenrollProvider: React.FunctionComponent = ({ children }) => { + const core = useCore(); + const [agents, setAgents] = useState([]); + const [agentsCount, setAgentsCount] = useState(0); + const [isModalOpen, setIsModalOpen] = useState(false); + const [isLoading, setIsLoading] = useState(false); + const onSuccessCallback = useRef(null); + + const unenrollAgentsPrompt: UnenrollAgents = ( + agentsToUnenroll, + agentsToUnenrollCount, + onSuccess = () => undefined + ) => { + if ( + agentsToUnenroll === undefined || + (Array.isArray(agentsToUnenroll) && agentsToUnenroll.length === 0) + ) { + throw new Error('No agents specified for unenrollment'); + } + setIsModalOpen(true); + setAgents(agentsToUnenroll); + setAgentsCount(agentsToUnenrollCount); + onSuccessCallback.current = onSuccess; + }; + + const closeModal = () => { + setAgents([]); + setAgentsCount(0); + setIsLoading(false); + setIsModalOpen(false); + }; + + const unenrollAgents = async () => { + setIsLoading(true); + + try { + const unenrollByKuery = typeof agents === 'string'; + const { data, error } = await sendRequest({ + path: agentRouteService.getUnenrollPath(), + method: 'post', + body: JSON.stringify({ + kuery: unenrollByKuery ? agents : undefined, + ids: !unenrollByKuery ? agents : undefined, + }), + }); + + if (error) { + throw new Error(error.message); + } + + const results = data ? data.results : []; + + const successfulResults = results.filter(result => result.success); + const failedResults = results.filter(result => !result.success); + + if (successfulResults.length) { + const hasMultipleSuccesses = successfulResults.length > 1; + const successMessage = hasMultipleSuccesses + ? i18n.translate('xpack.ingestManager.unenrollAgents.successMultipleNotificationTitle', { + defaultMessage: 'Unenrolled {count} agents', + values: { count: successfulResults.length }, + }) + : i18n.translate('xpack.ingestManager.unenrollAgents.successSingleNotificationTitle', { + defaultMessage: "Unenrolled agent '{id}'", + values: { id: successfulResults[0].id }, + }); + core.notifications.toasts.addSuccess(successMessage); + } + + if (failedResults.length) { + const hasMultipleFailures = failedResults.length > 1; + const failureMessage = hasMultipleFailures + ? i18n.translate('xpack.ingestManager.unenrollAgents.failureMultipleNotificationTitle', { + defaultMessage: 'Error unenrolling {count} agents', + values: { count: failedResults.length }, + }) + : i18n.translate('xpack.ingestManager.unenrollAgents.failureSingleNotificationTitle', { + defaultMessage: "Error unenrolling agent '{id}'", + values: { id: failedResults[0].id }, + }); + core.notifications.toasts.addDanger(failureMessage); + } + + if (onSuccessCallback.current) { + onSuccessCallback.current(successfulResults.map(result => result.id)); + } + } catch (e) { + core.notifications.toasts.addDanger( + i18n.translate('xpack.ingestManager.unenrollAgents.fatalErrorNotificationTitle', { + defaultMessage: 'Error unenrolling agents', + }) + ); + } + + closeModal(); + }; + + const renderModal = () => { + if (!isModalOpen) { + return null; + } + + const unenrollByKuery = typeof agents === 'string'; + const isSingle = agentsCount === 1; + + return ( + + + ) : ( + + ) + } + onCancel={closeModal} + onConfirm={unenrollAgents} + cancelButtonText={ + + } + confirmButtonText={ + isLoading ? ( + + ) : ( + + ) + } + buttonColor="danger" + confirmButtonDisabled={isLoading} + /> + + ); + }; + + return ( + + {children(unenrollAgentsPrompt)} + {renderModal()} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/index.tsx new file mode 100644 index 00000000000000..19378fe2fb9520 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/index.tsx @@ -0,0 +1,9 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export * from './loading'; +export * from './navigation/child_routes'; +export * from './navigation/connected_link'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/loading.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/loading.tsx new file mode 100644 index 00000000000000..5dcd2ff4d24779 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/loading.tsx @@ -0,0 +1,16 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import React from 'react'; +import { EuiFlexGroup, EuiFlexItem, EuiLoadingSpinner } from '@elastic/eui'; + +export const Loading: React.FunctionComponent<{}> = () => ( + + + + + +); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/navigation/child_routes.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/navigation/child_routes.tsx new file mode 100644 index 00000000000000..8af0e0a5cbc257 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/navigation/child_routes.tsx @@ -0,0 +1,38 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { Route, Switch } from 'react-router-dom'; + +interface RouteConfig { + path: string; + component: React.ComponentType; + routes?: RouteConfig[]; +} + +export const ChildRoutes: React.FunctionComponent<{ + routes?: RouteConfig[]; + useSwitch?: boolean; + [other: string]: any; +}> = ({ routes, useSwitch = true, ...rest }) => { + if (!routes) { + return null; + } + const Parent = useSwitch ? Switch : React.Fragment; + return ( + + {routes.map(route => ( + { + const Component = route.component; + return ; + }} + /> + ))} + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/navigation/connected_link.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/navigation/connected_link.tsx new file mode 100644 index 00000000000000..489ee85ffe28ac --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/components/navigation/connected_link.tsx @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { EuiLink } from '@elastic/eui'; +import { Link, withRouter } from 'react-router-dom'; + +export function ConnectedLinkComponent({ + location, + path, + query, + disabled, + children, + ...props +}: { + location: any; + path: string; + disabled: boolean; + query: any; + [key: string]: any; +}) { + if (disabled) { + return ; + } + + // Shorthand for pathname + const pathname = path || _.get(props.to, 'pathname') || location.pathname; + + return ( + + ); +} + +export const ConnectedLink = withRouter(ConnectedLinkComponent); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/components/no_data_layout.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/components/no_data_layout.tsx new file mode 100644 index 00000000000000..02134513edd644 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/components/no_data_layout.tsx @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { EuiEmptyPrompt, EuiFlexGroup, EuiFlexItem, EuiPageContent } from '@elastic/eui'; +import React from 'react'; +import { withRouter } from 'react-router-dom'; + +interface LayoutProps { + title: string | React.ReactNode; + actionSection?: React.ReactNode; + modalClosePath?: string; +} + +export const NoDataLayout: React.FunctionComponent = withRouter< + any, + React.FunctionComponent +>(({ actionSection, title, modalClosePath, children }) => { + return ( + + + + {title}} + body={children} + actions={actionSection} + /> + + + + ); +}) as any; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/enforce_security.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/enforce_security.tsx new file mode 100644 index 00000000000000..e131da159d6cba --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/enforce_security.tsx @@ -0,0 +1,26 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { FormattedMessage, injectI18n } from '@kbn/i18n/react'; + +import { NoDataLayout } from './components/no_data_layout'; + +export const EnforceSecurityPage = injectI18n(({ intl }) => ( + +

+ +

+ +)); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/invalid_license.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/invalid_license.tsx new file mode 100644 index 00000000000000..883e41fea95b85 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/invalid_license.tsx @@ -0,0 +1,27 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { FormattedMessage, injectI18n } from '@kbn/i18n/react'; + +import { NoDataLayout } from './components/no_data_layout'; + +export const InvalidLicensePage = injectI18n(({ intl }) => ( + +

+ +

+ +)); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/no_access.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/no_access.tsx new file mode 100644 index 00000000000000..5a3afd62168243 --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/error_pages/no_access.tsx @@ -0,0 +1,27 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React from 'react'; +import { FormattedMessage, injectI18n } from '@kbn/i18n/react'; + +import { NoDataLayout } from './components/no_data_layout'; + +export const NoAccessPage = injectI18n(({ intl }) => ( + +

+ +

+ +)); diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/index.tsx index c4e8c576a1d7db..edc2b5b7eb87f9 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/index.tsx +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/index.tsx @@ -4,53 +4,53 @@ * you may not use this file except in compliance with the Elastic License. */ import React from 'react'; -import { EuiFlexGroup, EuiFlexItem, EuiText } from '@elastic/eui'; -import { FormattedMessage } from '@kbn/i18n/react'; -import { WithHeaderLayout } from '../../layouts'; -import { useConfig } from '../../hooks'; +import { HashRouter as Router, Route, Switch, Redirect } from 'react-router-dom'; +import { Loading } from '../../components'; +import { useConfig, useCore, useRequest } from '../../hooks'; +import { AgentListPage } from './agent_list_page'; +import { SetupPage } from './setup_page'; +import { AgentDetailsPage } from './agent_details_page'; +import { NoAccessPage } from './error_pages/no_access'; +import { fleetSetupRouteService } from '../../services'; export const FleetApp: React.FunctionComponent = () => { + const core = useCore(); const { fleet } = useConfig(); - if (!fleet.enabled) { - return null; + + const setupRequest = useRequest({ + method: 'get', + path: fleetSetupRouteService.getFleetSetupPath(), + }); + + if (!fleet.enabled) return null; + if (setupRequest.isLoading) { + return ; + } + + if (setupRequest.data.isInitialized === false) { + return ( + { + await setupRequest.sendRequest(); + }} + /> + ); + } + if (!core.application.capabilities.ingestManager.read) { + return ; } return ( - - - -

- -

- - - - -

- -

- - - - } - tabs={[ - { - id: 'agents', - name: 'Agents', - isSelected: true, - }, - { - id: 'enrollment_keys', - name: 'Enrollment keys', - }, - ]} - > - hello world - fleet app - + + + } /> + + + + + + + + ); }; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/setup_page/index.tsx b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/setup_page/index.tsx new file mode 100644 index 00000000000000..31e5e99ad284bd --- /dev/null +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/fleet/setup_page/index.tsx @@ -0,0 +1,80 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import React, { useState } from 'react'; +import { + EuiPageBody, + EuiPageContent, + EuiForm, + EuiFormRow, + EuiFieldText, + EuiFieldPassword, + EuiText, + EuiButton, + EuiCallOut, + EuiTitle, + EuiSpacer, +} from '@elastic/eui'; +import { sendRequest, useInput, useCore } from '../../../hooks'; +import { fleetSetupRouteService } from '../../../services'; + +export const SetupPage: React.FunctionComponent<{ + refresh: () => Promise; +}> = ({ refresh }) => { + const [isFormLoading, setIsFormLoading] = useState(false); + const core = useCore(); + const usernameInput = useInput(); + const passwordInput = useInput(); + + const onSubmit = async (e: React.FormEvent) => { + e.preventDefault(); + setIsFormLoading(true); + try { + await sendRequest({ + method: 'post', + path: fleetSetupRouteService.postFleetSetupPath(), + body: JSON.stringify({ + admin_username: usernameInput.value, + admin_password: passwordInput.value, + }), + }); + await refresh(); + } catch (error) { + core.notifications.toasts.addDanger(error.message); + setIsFormLoading(false); + } + }; + + return ( + + + +

Setup

+ + + + + To setup fleet and ingest you need to a enable a user that can create API Keys and write + to logs-* and metrics-* + + + + +
+ + + + + + + + Submit + +
+ + + + ); +}; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/services/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/services/index.ts index 6502b0fff71234..0aa08602e4d4de 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/services/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/services/index.ts @@ -4,4 +4,15 @@ * you may not use this file except in compliance with the Elastic License. */ -export { agentConfigRouteService } from '../../../../common'; +export { + agentConfigRouteService, + datasourceRouteService, + fleetSetupRouteService, + agentRouteService, + enrollmentAPIKeyRouteService, + epmRouteService, + setupRouteService, + packageToConfigDatasourceInputs, + storedDatasourceToAgentDatasource, + AgentStatusKueryHelper, +} from '../../../../common'; diff --git a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/types/index.ts b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/types/index.ts index 8597d6fd59323e..a59fb06145a3a5 100644 --- a/x-pack/plugins/ingest_manager/public/applications/ingest_manager/types/index.ts +++ b/x-pack/plugins/ingest_manager/public/applications/ingest_manager/types/index.ts @@ -4,16 +4,69 @@ * you may not use this file except in compliance with the Elastic License. */ export { + // utility function + entries, // Object types + Agent, AgentConfig, NewAgentConfig, - // API schemas + AgentEvent, + EnrollmentAPIKey, + Datasource, + NewDatasource, + DatasourceInput, + DatasourceInputStream, + // API schemas - Agent Config GetAgentConfigsResponse, + GetAgentConfigsResponseItem, GetOneAgentConfigResponse, - CreateAgentConfigRequestSchema, + CreateAgentConfigRequest, CreateAgentConfigResponse, - UpdateAgentConfigRequestSchema, + UpdateAgentConfigRequest, UpdateAgentConfigResponse, - DeleteAgentConfigsRequestSchema, + DeleteAgentConfigsRequest, DeleteAgentConfigsResponse, + // API schemas - Datasource + CreateDatasourceRequest, + CreateDatasourceResponse, + // API schemas - Agents + GetAgentsResponse, + GetAgentsRequest, + GetOneAgentResponse, + PostAgentUnenrollResponse, + GetOneAgentEventsRequest, + GetOneAgentEventsResponse, + GetAgentStatusRequest, + GetAgentStatusResponse, + // API schemas - Enrollment API Keys + GetEnrollmentAPIKeysResponse, + GetOneEnrollmentAPIKeyResponse, + // EPM types + AssetReference, + AssetsGroupedByServiceByType, + AssetType, + AssetTypeToParts, + CategoryId, + CategorySummaryItem, + CategorySummaryList, + ElasticsearchAssetType, + KibanaAssetType, + PackageInfo, + RegistryVarsEntry, + RegistryInput, + RegistryStream, + PackageList, + PackageListItem, + PackagesGroupedByStatus, + RequirementsByServiceName, + RequirementVersion, + ScreenshotItem, + ServiceName, + GetCategoriesResponse, + GetPackagesResponse, + GetInfoResponse, + InstallPackageResponse, + DeletePackageResponse, + DetailViewPanelName, + InstallStatus, } from '../../../../common'; diff --git a/x-pack/plugins/ingest_manager/public/plugin.ts b/x-pack/plugins/ingest_manager/public/plugin.ts index ae244e7ebec3d1..a1dc2c057e9e5d 100644 --- a/x-pack/plugins/ingest_manager/public/plugin.ts +++ b/x-pack/plugins/ingest_manager/public/plugin.ts @@ -6,15 +6,16 @@ import { AppMountParameters, CoreSetup, - CoreStart, Plugin, PluginInitializerContext, + CoreStart, } from 'kibana/public'; import { i18n } from '@kbn/i18n'; import { DEFAULT_APP_CATEGORIES } from '../../../../src/core/utils'; -import { DataPublicPluginSetup } from '../../../../src/plugins/data/public'; +import { DataPublicPluginSetup, DataPublicPluginStart } from '../../../../src/plugins/data/public'; import { LicensingPluginSetup } from '../../licensing/public'; import { PLUGIN_ID } from '../common/constants'; + import { IngestManagerConfigType } from '../common/types'; export { IngestManagerConfigType } from '../common/types'; @@ -27,6 +28,10 @@ export interface IngestManagerSetupDeps { data: DataPublicPluginSetup; } +export interface IngestManagerStartDeps { + data: DataPublicPluginStart; +} + export class IngestManagerPlugin implements Plugin { private config: IngestManagerConfigType; @@ -36,7 +41,6 @@ export class IngestManagerPlugin implements Plugin { public setup(core: CoreSetup, deps: IngestManagerSetupDeps) { const config = this.config; - // Register main Ingest Manager app core.application.register({ id: PLUGIN_ID, @@ -44,9 +48,12 @@ export class IngestManagerPlugin implements Plugin { title: i18n.translate('xpack.ingestManager.appTitle', { defaultMessage: 'Ingest Manager' }), euiIconType: 'savedObjectsApp', async mount(params: AppMountParameters) { - const [coreStart] = await core.getStartServices(); + const [coreStart, startDeps] = (await core.getStartServices()) as [ + CoreStart, + IngestManagerStartDeps + ]; const { renderApp } = await import('./applications/ingest_manager'); - return renderApp(coreStart, params, deps, config); + return renderApp(coreStart, params, deps, startDeps, config); }, }); } diff --git a/x-pack/plugins/ingest_manager/scripts/dev_agent/index.js b/x-pack/plugins/ingest_manager/scripts/dev_agent/index.js new file mode 100644 index 00000000000000..75d9a27e9f2415 --- /dev/null +++ b/x-pack/plugins/ingest_manager/scripts/dev_agent/index.js @@ -0,0 +1,8 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +require('../../../../../src/setup_node_env'); +require('./script'); diff --git a/x-pack/plugins/ingest_manager/scripts/dev_agent/script.ts b/x-pack/plugins/ingest_manager/scripts/dev_agent/script.ts new file mode 100644 index 00000000000000..c7b8edd0c332ef --- /dev/null +++ b/x-pack/plugins/ingest_manager/scripts/dev_agent/script.ts @@ -0,0 +1,134 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { createFlagError, run, ToolingLog } from '@kbn/dev-utils'; +import fetch from 'node-fetch'; +import os from 'os'; +import { + Agent as _Agent, + PostAgentCheckinRequest, + PostAgentCheckinResponse, + PostAgentEnrollRequest, + PostAgentEnrollResponse, +} from '../../common/types'; + +const CHECKIN_INTERVAL = 3000; // 3 seconds + +type Agent = Pick<_Agent, 'id' | 'access_api_key'>; + +let closing = false; + +process.once('SIGINT', () => { + closing = true; +}); + +run( + async ({ flags, log }) => { + if (!flags.kibanaUrl || typeof flags.kibanaUrl !== 'string') { + throw createFlagError('please provide a single --path flag'); + } + + if (!flags.enrollmentApiKey || typeof flags.enrollmentApiKey !== 'string') { + throw createFlagError('please provide a single --enrollmentApiKey flag'); + } + const kibanaUrl = flags.kibanaUrl || 'http://localhost:5601'; + const agent = await enroll(kibanaUrl, flags.enrollmentApiKey, log); + + log.info('Enrolled with sucess', agent); + + while (!closing) { + await checkin(kibanaUrl, agent, log); + await new Promise((resolve, reject) => setTimeout(() => resolve(), CHECKIN_INTERVAL)); + } + }, + { + description: ` + Run a fleet development agent. + `, + flags: { + string: ['kibanaUrl', 'enrollmentApiKey'], + help: ` + --kibanaUrl kibanaURL to run the fleet agent + --enrollmentApiKey enrollment api key + `, + }, + } +); + +async function checkin(kibanaURL: string, agent: Agent, log: ToolingLog) { + const body: PostAgentCheckinRequest['body'] = { + events: [ + { + type: 'STATE', + subtype: 'RUNNING', + message: 'state changed from STOPPED to RUNNING', + timestamp: new Date().toISOString(), + payload: { + random: 'data', + state: 'RUNNING', + previous_state: 'STOPPED', + }, + agent_id: agent.id, + }, + ], + }; + const res = await fetch(`${kibanaURL}/api/ingest_manager/fleet/agents/${agent.id}/checkin`, { + method: 'POST', + body: JSON.stringify(body), + headers: { + 'kbn-xsrf': 'xxx', + Authorization: `ApiKey ${agent.access_api_key}`, + 'Content-Type': 'application/json', + }, + }); + + if (res.status === 403) { + closing = true; + log.info('Unenrolling agent'); + return; + } + + const obj: PostAgentCheckinResponse = await res.json(); + log.info('checkin', obj); +} + +async function enroll(kibanaURL: string, apiKey: string, log: ToolingLog): Promise { + const body: PostAgentEnrollRequest['body'] = { + type: 'PERMANENT', + metadata: { + local: { + host: 'localhost', + ip: '127.0.0.1', + system: `${os.type()} ${os.release()}`, + memory: os.totalmem(), + }, + user_provided: { + dev_agent_version: '0.0.1', + region: 'us-east', + }, + }, + }; + const res = await fetch(`${kibanaURL}/api/ingest_manager/fleet/agents/enroll`, { + method: 'POST', + body: JSON.stringify(body), + headers: { + 'kbn-xsrf': 'xxx', + Authorization: `ApiKey ${apiKey}`, + 'Content-Type': 'application/json', + }, + }); + const obj: PostAgentEnrollResponse = await res.json(); + + if (!obj.success) { + log.error(JSON.stringify(obj, null, 2)); + throw new Error('unable to enroll'); + } + + return { + id: obj.item.id, + access_api_key: obj.item.access_api_key, + }; +} diff --git a/x-pack/plugins/ingest_manager/scripts/readme.md b/x-pack/plugins/ingest_manager/scripts/readme.md new file mode 100644 index 00000000000000..efec40b0aba1e4 --- /dev/null +++ b/x-pack/plugins/ingest_manager/scripts/readme.md @@ -0,0 +1,8 @@ +### Dev agents + +You can run a development fleet agent that is going to enroll and checkin every 3 seconds. +For this you can run the following command in the fleet pluging directory. + +``` +node scripts/dev_agent --enrollmentApiKey= --kibanaUrl=http://localhost:5603/qed +``` diff --git a/x-pack/plugins/ingest_manager/server/constants/index.ts b/x-pack/plugins/ingest_manager/server/constants/index.ts index 6b54afa1d81cb1..f6ee475614c5e3 100644 --- a/x-pack/plugins/ingest_manager/server/constants/index.ts +++ b/x-pack/plugins/ingest_manager/server/constants/index.ts @@ -4,19 +4,31 @@ * you may not use this file except in compliance with the Elastic License. */ export { + AGENT_TYPE_PERMANENT, + AGENT_TYPE_EPHEMERAL, + AGENT_TYPE_TEMPORARY, + AGENT_POLLING_THRESHOLD_MS, + AGENT_POLLING_INTERVAL, // Routes PLUGIN_ID, EPM_API_ROUTES, DATASOURCE_API_ROUTES, + AGENT_API_ROUTES, AGENT_CONFIG_API_ROUTES, FLEET_SETUP_API_ROUTES, + ENROLLMENT_API_KEY_ROUTES, + INSTALL_SCRIPT_API_ROUTES, + SETUP_API_ROUTE, // Saved object types + AGENT_EVENT_SAVED_OBJECT_TYPE, + AGENT_SAVED_OBJECT_TYPE, AGENT_CONFIG_SAVED_OBJECT_TYPE, DATASOURCE_SAVED_OBJECT_TYPE, OUTPUT_SAVED_OBJECT_TYPE, + PACKAGES_SAVED_OBJECT_TYPE, + INDEX_PATTERN_SAVED_OBJECT_TYPE, + ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, // Defaults - DEFAULT_AGENT_CONFIG_ID, DEFAULT_AGENT_CONFIG, - DEFAULT_OUTPUT_ID, DEFAULT_OUTPUT, } from '../../common'; diff --git a/x-pack/plugins/ingest_manager/server/index.ts b/x-pack/plugins/ingest_manager/server/index.ts index 5228f1e0e34694..b732cb8005efb7 100644 --- a/x-pack/plugins/ingest_manager/server/index.ts +++ b/x-pack/plugins/ingest_manager/server/index.ts @@ -3,8 +3,7 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ - -import { schema } from '@kbn/config-schema'; +import { schema, TypeOf } from '@kbn/config-schema'; import { PluginInitializerContext } from 'kibana/server'; import { IngestManagerPlugin } from './plugin'; @@ -21,11 +20,20 @@ export const config = { }), fleet: schema.object({ enabled: schema.boolean({ defaultValue: false }), - defaultOutputHost: schema.string({ defaultValue: 'http://localhost:9200' }), + kibana: schema.object({ + host: schema.maybe(schema.string()), + ca_sha256: schema.maybe(schema.string()), + }), + elasticsearch: schema.object({ + host: schema.string({ defaultValue: 'http://localhost:9200' }), + ca_sha256: schema.maybe(schema.string()), + }), }), }), }; +export type IngestManagerConfigType = TypeOf; + export const plugin = (initializerContext: PluginInitializerContext) => { return new IngestManagerPlugin(initializerContext); }; @@ -37,7 +45,5 @@ export { OUTPUT_SAVED_OBJECT_TYPE, AGENT_CONFIG_SAVED_OBJECT_TYPE, DATASOURCE_SAVED_OBJECT_TYPE, + PACKAGES_SAVED_OBJECT_TYPE, } from './constants'; - -// TODO: Temporary exports for Fleet dependencies, remove once Fleet moved into this plugin -export { agentConfigService, outputService } from './services'; diff --git a/x-pack/plugins/ingest_manager/server/integration_tests/router.test.ts b/x-pack/plugins/ingest_manager/server/integration_tests/router.test.ts index c2546454e21316..bfd84282226430 100644 --- a/x-pack/plugins/ingest_manager/server/integration_tests/router.test.ts +++ b/x-pack/plugins/ingest_manager/server/integration_tests/router.test.ts @@ -51,11 +51,11 @@ describe('ingestManager', () => { }); it('does not have EPM api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/epm').expect(404); + await kbnTestServer.request.get(root, '/api/ingest_manager/epm/packages').expect(404); }); it('does not have Fleet api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/fleet').expect(404); + await kbnTestServer.request.get(root, '/api/ingest_manager/fleet/setup').expect(404); }); }); @@ -84,7 +84,7 @@ describe('ingestManager', () => { }); it('does not have EPM api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/epm/list').expect(404); + await kbnTestServer.request.get(root, '/api/ingest_manager/epm/packages').expect(404); }); it('does not have Fleet api', async () => { @@ -122,8 +122,8 @@ describe('ingestManager', () => { await kbnTestServer.request.get(root, '/api/ingest_manager/datasources').expect(200); }); - it('does not have EPM api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/epm/list').expect(404); + it('does have EPM api', async () => { + await kbnTestServer.request.get(root, '/api/ingest_manager/epm/packages').expect(500); }); it('does not have Fleet api', async () => { @@ -137,7 +137,6 @@ describe('ingestManager', () => { beforeAll(async () => { const ingestManagerConfig = { enabled: true, - epm: { enabled: true }, fleet: { enabled: true }, }; root = createXPackRoot({ @@ -158,11 +157,11 @@ describe('ingestManager', () => { }); it('does not have EPM api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/epm/list').expect(404); + await kbnTestServer.request.get(root, '/api/ingest_manager/epm/packages').expect(404); }); - it('does not have Fleet api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/fleet/setup').expect(404); + it('does have Fleet api', async () => { + await kbnTestServer.request.get(root, '/api/ingest_manager/fleet/setup').expect(200); }); }); @@ -192,12 +191,12 @@ describe('ingestManager', () => { await kbnTestServer.request.get(root, '/api/ingest_manager/datasources').expect(200); }); - it('does not have EPM api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/epm/list').expect(404); + it('does have EPM api', async () => { + await kbnTestServer.request.get(root, '/api/ingest_manager/epm/packages').expect(500); }); - it('does not have Fleet api', async () => { - await kbnTestServer.request.get(root, '/api/ingest_manager/fleet/setup').expect(404); + it('does have Fleet api', async () => { + await kbnTestServer.request.get(root, '/api/ingest_manager/fleet/setup').expect(200); }); }); }); diff --git a/x-pack/plugins/ingest_manager/server/plugin.ts b/x-pack/plugins/ingest_manager/server/plugin.ts index 4f30a171ab0c02..c162ea5fadabe8 100644 --- a/x-pack/plugins/ingest_manager/server/plugin.ts +++ b/x-pack/plugins/ingest_manager/server/plugin.ts @@ -4,15 +4,41 @@ * you may not use this file except in compliance with the Elastic License. */ import { Observable } from 'rxjs'; -import { CoreSetup, CoreStart, Plugin, PluginInitializerContext } from 'kibana/server'; +import { first } from 'rxjs/operators'; +import { + CoreSetup, + CoreStart, + Plugin, + PluginInitializerContext, + SavedObjectsServiceStart, +} from 'kibana/server'; import { LicensingPluginSetup } from '../../licensing/server'; import { EncryptedSavedObjectsPluginStart } from '../../encrypted_saved_objects/server'; import { SecurityPluginSetup } from '../../security/server'; import { PluginSetupContract as FeaturesPluginSetup } from '../../features/server'; -import { PLUGIN_ID } from './constants'; -import { appContextService } from './services'; -import { registerDatasourceRoutes, registerAgentConfigRoutes } from './routes'; +import { + PLUGIN_ID, + OUTPUT_SAVED_OBJECT_TYPE, + AGENT_CONFIG_SAVED_OBJECT_TYPE, + DATASOURCE_SAVED_OBJECT_TYPE, + PACKAGES_SAVED_OBJECT_TYPE, + AGENT_SAVED_OBJECT_TYPE, + AGENT_EVENT_SAVED_OBJECT_TYPE, + ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, +} from './constants'; + +import { + registerEPMRoutes, + registerDatasourceRoutes, + registerAgentConfigRoutes, + registerSetupRoutes, + registerAgentRoutes, + registerEnrollmentApiKeyRoutes, + registerInstallScriptRoutes, +} from './routes'; + import { IngestManagerConfigType } from '../common'; +import { appContextService } from './services'; export interface IngestManagerSetupDeps { licensing: LicensingPluginSetup; @@ -24,8 +50,19 @@ export interface IngestManagerAppContext { encryptedSavedObjects: EncryptedSavedObjectsPluginStart; security?: SecurityPluginSetup; config$?: Observable; + savedObjects: SavedObjectsServiceStart; } +const allSavedObjectTypes = [ + OUTPUT_SAVED_OBJECT_TYPE, + AGENT_CONFIG_SAVED_OBJECT_TYPE, + DATASOURCE_SAVED_OBJECT_TYPE, + PACKAGES_SAVED_OBJECT_TYPE, + AGENT_SAVED_OBJECT_TYPE, + AGENT_EVENT_SAVED_OBJECT_TYPE, + ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, +]; + export class IngestManagerPlugin implements Plugin { private config$: Observable; private security: SecurityPluginSetup | undefined; @@ -50,37 +87,47 @@ export class IngestManagerPlugin implements Plugin { app: [PLUGIN_ID, 'kibana'], privileges: { all: { - api: [PLUGIN_ID], + api: [`${PLUGIN_ID}-read`, `${PLUGIN_ID}-all`], savedObject: { - all: [], + all: allSavedObjectTypes, read: [], }, - ui: ['show'], + ui: ['show', 'read', 'write'], }, read: { - api: [PLUGIN_ID], + api: [`${PLUGIN_ID}-read`], savedObject: { all: [], - read: [], + read: allSavedObjectTypes, }, - ui: ['show'], + ui: ['show', 'read'], }, }, }); } - // Create router const router = core.http.createRouter(); + const config = await this.config$.pipe(first()).toPromise(); // Register routes registerAgentConfigRoutes(router); registerDatasourceRoutes(router); - // Optional route registration depending on Kibana config - // restore when EPM & Fleet features are added - // const config = await this.config$.pipe(first()).toPromise(); - // if (config.epm.enabled) registerEPMRoutes(router); - // if (config.fleet.enabled) registerFleetSetupRoutes(router); + // Conditional routes + if (config.epm.enabled) { + registerEPMRoutes(router); + } + + if (config.fleet.enabled) { + registerSetupRoutes(router); + registerAgentRoutes(router); + registerEnrollmentApiKeyRoutes(router); + registerInstallScriptRoutes({ + router, + serverInfo: core.http.getServerInfo(), + basePath: core.http.basePath, + }); + } } public async start( @@ -93,6 +140,7 @@ export class IngestManagerPlugin implements Plugin { encryptedSavedObjects: plugins.encryptedSavedObjects, security: this.security, config$: this.config$, + savedObjects: core.savedObjects, }); } diff --git a/x-pack/plugins/ingest_manager/server/routes/agent/handlers.ts b/x-pack/plugins/ingest_manager/server/routes/agent/handlers.ts new file mode 100644 index 00000000000000..cb4e4d557d74fd --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/routes/agent/handlers.ts @@ -0,0 +1,411 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { RequestHandler, KibanaRequest } from 'kibana/server'; +import { TypeOf } from '@kbn/config-schema'; +import { + GetAgentsResponse, + GetOneAgentResponse, + GetOneAgentEventsResponse, + PostAgentCheckinResponse, + PostAgentEnrollResponse, + PostAgentUnenrollResponse, + GetAgentStatusResponse, +} from '../../../common/types'; +import { + GetAgentsRequestSchema, + GetOneAgentRequestSchema, + UpdateAgentRequestSchema, + DeleteAgentRequestSchema, + GetOneAgentEventsRequestSchema, + PostAgentCheckinRequestSchema, + PostAgentEnrollRequestSchema, + PostAgentAcksRequestSchema, + PostAgentUnenrollRequestSchema, + GetAgentStatusRequestSchema, +} from '../../types'; +import * as AgentService from '../../services/agents'; +import * as APIKeyService from '../../services/api_keys'; +import { appContextService } from '../../services/app_context'; + +function getInternalUserSOClient(request: KibanaRequest) { + // soClient as kibana internal users, be carefull on how you use it, security is not enabled + return appContextService.getSavedObjects().getScopedClient(request, { + excludedWrappers: ['security'], + }); +} + +export const getAgentHandler: RequestHandler> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const agent = await AgentService.getAgent(soClient, request.params.agentId); + + const body: GetOneAgentResponse = { + item: { + ...agent, + status: AgentService.getAgentStatus(agent), + }, + success: true, + }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom && e.output.statusCode === 404) { + return response.notFound({ + body: { message: `Agent ${request.params.agentId} not found` }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getAgentEventsHandler: RequestHandler< + TypeOf, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const { page, perPage, kuery } = request.query; + const { items, total } = await AgentService.getAgentEvents(soClient, request.params.agentId, { + page, + perPage, + kuery, + }); + + const body: GetOneAgentEventsResponse = { + list: items, + total, + success: true, + page, + perPage, + }; + + return response.ok({ + body, + }); + } catch (e) { + if (e.isBoom && e.output.statusCode === 404) { + return response.notFound({ + body: { message: `Agent ${request.params.agentId} not found` }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const deleteAgentHandler: RequestHandler> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + await AgentService.deleteAgent(soClient, request.params.agentId); + + const body = { + success: true, + action: 'deleted', + }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom) { + return response.customError({ + statusCode: e.output.statusCode, + body: { message: `Agent ${request.params.agentId} not found` }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const updateAgentHandler: RequestHandler< + TypeOf, + undefined, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + await AgentService.updateAgent(soClient, request.params.agentId, { + userProvidedMetatada: request.body.user_provided_metadata, + }); + const agent = await AgentService.getAgent(soClient, request.params.agentId); + + const body = { + item: { + ...agent, + status: AgentService.getAgentStatus(agent), + }, + success: true, + }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom && e.output.statusCode === 404) { + return response.notFound({ + body: { message: `Agent ${request.params.agentId} not found` }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const postAgentCheckinHandler: RequestHandler< + TypeOf, + undefined, + TypeOf +> = async (context, request, response) => { + try { + const soClient = getInternalUserSOClient(request); + const res = APIKeyService.parseApiKey(request.headers); + const agent = await AgentService.getAgentByAccessAPIKeyId(soClient, res.apiKeyId); + const { actions } = await AgentService.agentCheckin( + soClient, + agent, + request.body.events || [], + request.body.local_metadata + ); + const body: PostAgentCheckinResponse = { + action: 'checkin', + success: true, + actions: actions.map(a => ({ + type: a.type, + data: a.data ? JSON.parse(a.data) : a.data, + id: a.id, + created_at: a.created_at, + })), + }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom && e.output.statusCode === 404) { + return response.notFound({ + body: { message: `Agent ${request.params.agentId} not found` }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const postAgentAcksHandler: RequestHandler< + TypeOf, + undefined, + TypeOf +> = async (context, request, response) => { + try { + const soClient = getInternalUserSOClient(request); + const res = APIKeyService.parseApiKey(request.headers); + const agent = await AgentService.getAgentByAccessAPIKeyId(soClient, res.apiKeyId as string); + + await AgentService.acknowledgeAgentActions(soClient, agent, request.body.action_ids); + + const body = { + action: 'acks', + success: true, + }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom) { + return response.customError({ + statusCode: e.output.statusCode, + body: { message: e.message }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const postAgentEnrollHandler: RequestHandler< + undefined, + undefined, + TypeOf +> = async (context, request, response) => { + try { + const soClient = getInternalUserSOClient(request); + const { apiKeyId } = APIKeyService.parseApiKey(request.headers); + const enrollmentAPIKey = await APIKeyService.getEnrollmentAPIKeyById(soClient, apiKeyId); + + if (!enrollmentAPIKey || !enrollmentAPIKey.active) { + return response.unauthorized({ + body: { message: 'Invalid Enrollment API Key' }, + }); + } + + const agent = await AgentService.enroll( + soClient, + request.body.type, + enrollmentAPIKey.config_id as string, + { + userProvided: request.body.metadata.user_provided, + local: request.body.metadata.local, + }, + request.body.shared_id + ); + const body: PostAgentEnrollResponse = { + action: 'created', + success: true, + item: { + ...agent, + status: AgentService.getAgentStatus(agent), + }, + }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom) { + return response.customError({ + statusCode: e.output.statusCode, + body: { message: e.message }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getAgentsHandler: RequestHandler< + undefined, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const { agents, total, page, perPage } = await AgentService.listAgents(soClient, { + page: request.query.page, + perPage: request.query.perPage, + showInactive: request.query.showInactive, + kuery: request.query.kuery, + }); + + const body: GetAgentsResponse = { + list: agents.map(agent => ({ + ...agent, + status: AgentService.getAgentStatus(agent), + })), + success: true, + total, + page, + perPage, + }; + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const postAgentsUnenrollHandler: RequestHandler< + undefined, + undefined, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const kuery = (request.body as { kuery: string }).kuery; + let toUnenrollIds: string[] = (request.body as { ids: string[] }).ids || []; + + if (kuery) { + let hasMore = true; + let page = 1; + while (hasMore) { + const { agents } = await AgentService.listAgents(soClient, { + page: page++, + perPage: 100, + kuery, + showInactive: true, + }); + if (agents.length === 0) { + hasMore = false; + } + const agentIds = agents.filter(a => a.active).map(a => a.id); + toUnenrollIds = toUnenrollIds.concat(agentIds); + } + } + const results = (await AgentService.unenrollAgents(soClient, toUnenrollIds)).map( + ({ + success, + id, + error, + }): { + success: boolean; + id: string; + action: 'unenrolled'; + error?: { + message: string; + }; + } => { + return { + success, + id, + action: 'unenrolled', + error: error && { + message: error.message, + }, + }; + } + ); + + const body: PostAgentUnenrollResponse = { + results, + success: results.every(result => result.success), + }; + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getAgentStatusForConfigHandler: RequestHandler< + undefined, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + // TODO change path + const results = await AgentService.getAgentStatusForConfig(soClient, request.query.configId); + + const body: GetAgentStatusResponse = { results, success: true }; + + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/agent/index.ts b/x-pack/plugins/ingest_manager/server/routes/agent/index.ts new file mode 100644 index 00000000000000..8a65fa9c50e8b3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/routes/agent/index.ts @@ -0,0 +1,135 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { IRouter } from 'kibana/server'; +import { PLUGIN_ID, AGENT_API_ROUTES } from '../../constants'; +import { + GetAgentsRequestSchema, + GetOneAgentRequestSchema, + GetOneAgentEventsRequestSchema, + UpdateAgentRequestSchema, + DeleteAgentRequestSchema, + PostAgentCheckinRequestSchema, + PostAgentEnrollRequestSchema, + PostAgentAcksRequestSchema, + PostAgentUnenrollRequestSchema, + GetAgentStatusRequestSchema, +} from '../../types'; +import { + getAgentsHandler, + getAgentHandler, + updateAgentHandler, + deleteAgentHandler, + getAgentEventsHandler, + postAgentCheckinHandler, + postAgentEnrollHandler, + postAgentAcksHandler, + postAgentsUnenrollHandler, + getAgentStatusForConfigHandler, +} from './handlers'; + +export const registerRoutes = (router: IRouter) => { + // Get one + router.get( + { + path: AGENT_API_ROUTES.INFO_PATTERN, + validate: GetOneAgentRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getAgentHandler + ); + // Update + router.put( + { + path: AGENT_API_ROUTES.UPDATE_PATTERN, + validate: UpdateAgentRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, + }, + updateAgentHandler + ); + // Delete + router.delete( + { + path: AGENT_API_ROUTES.DELETE_PATTERN, + validate: DeleteAgentRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, + }, + deleteAgentHandler + ); + // List + router.get( + { + path: AGENT_API_ROUTES.LIST_PATTERN, + validate: GetAgentsRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getAgentsHandler + ); + + // Agent checkin + router.post( + { + path: AGENT_API_ROUTES.CHECKIN_PATTERN, + validate: PostAgentCheckinRequestSchema, + options: { tags: [] }, + }, + postAgentCheckinHandler + ); + + // Agent enrollment + router.post( + { + path: AGENT_API_ROUTES.ENROLL_PATTERN, + validate: PostAgentEnrollRequestSchema, + options: { tags: [] }, + }, + postAgentEnrollHandler + ); + + // Agent acks + router.post( + { + path: AGENT_API_ROUTES.ACKS_PATTERN, + validate: PostAgentAcksRequestSchema, + options: { tags: [] }, + }, + postAgentAcksHandler + ); + + router.post( + { + path: AGENT_API_ROUTES.UNENROLL_PATTERN, + validate: PostAgentUnenrollRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, + }, + postAgentsUnenrollHandler + ); + + // Get agent events + router.get( + { + path: AGENT_API_ROUTES.EVENTS_PATTERN, + validate: GetOneAgentEventsRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getAgentEventsHandler + ); + + // Get agent status for config + router.get( + { + path: AGENT_API_ROUTES.STATUS_PATTERN, + validate: GetAgentStatusRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getAgentStatusForConfigHandler + ); +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/agent_config/handlers.ts b/x-pack/plugins/ingest_manager/server/routes/agent_config/handlers.ts index 67da6a4cf2f1d2..8c3ca82f327b09 100644 --- a/x-pack/plugins/ingest_manager/server/routes/agent_config/handlers.ts +++ b/x-pack/plugins/ingest_manager/server/routes/agent_config/handlers.ts @@ -5,19 +5,25 @@ */ import { TypeOf } from '@kbn/config-schema'; import { RequestHandler } from 'kibana/server'; +import bluebird from 'bluebird'; import { appContextService, agentConfigService } from '../../services'; +import { listAgents } from '../../services/agents'; import { GetAgentConfigsRequestSchema, - GetAgentConfigsResponse, GetOneAgentConfigRequestSchema, - GetOneAgentConfigResponse, CreateAgentConfigRequestSchema, - CreateAgentConfigResponse, UpdateAgentConfigRequestSchema, - UpdateAgentConfigResponse, DeleteAgentConfigsRequestSchema, - DeleteAgentConfigsResponse, + GetFullAgentConfigRequestSchema, } from '../../types'; +import { + GetAgentConfigsResponse, + GetOneAgentConfigResponse, + CreateAgentConfigResponse, + UpdateAgentConfigResponse, + DeleteAgentConfigsResponse, + GetFullAgentConfigResponse, +} from '../../../common'; export const getAgentConfigsHandler: RequestHandler< undefined, @@ -33,6 +39,19 @@ export const getAgentConfigsHandler: RequestHandler< perPage, success: true, }; + + await bluebird.map( + items, + agentConfig => + listAgents(soClient, { + showInactive: true, + perPage: 0, + page: 1, + kuery: `agents.config_id:${agentConfig.id}`, + }).then(({ total: agentTotal }) => (agentConfig.agents = agentTotal)), + { concurrency: 10 } + ); + return response.ok({ body }); } catch (e) { return response.customError({ @@ -142,3 +161,35 @@ export const deleteAgentConfigsHandler: RequestHandler< }); } }; + +export const getFullAgentConfig: RequestHandler> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + + try { + const fullAgentConfig = await agentConfigService.getFullConfig( + soClient, + request.params.agentConfigId + ); + if (fullAgentConfig) { + const body: GetFullAgentConfigResponse = { + item: fullAgentConfig, + success: true, + }; + return response.ok({ + body, + }); + } else { + return response.customError({ + statusCode: 404, + body: { message: 'Agent config not found' }, + }); + } + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/agent_config/index.ts b/x-pack/plugins/ingest_manager/server/routes/agent_config/index.ts index 67ad915b71e45a..c3b3c00a9574cd 100644 --- a/x-pack/plugins/ingest_manager/server/routes/agent_config/index.ts +++ b/x-pack/plugins/ingest_manager/server/routes/agent_config/index.ts @@ -11,6 +11,7 @@ import { CreateAgentConfigRequestSchema, UpdateAgentConfigRequestSchema, DeleteAgentConfigsRequestSchema, + GetFullAgentConfigRequestSchema, } from '../../types'; import { getAgentConfigsHandler, @@ -18,6 +19,7 @@ import { createAgentConfigHandler, updateAgentConfigHandler, deleteAgentConfigsHandler, + getFullAgentConfig, } from './handlers'; export const registerRoutes = (router: IRouter) => { @@ -26,7 +28,7 @@ export const registerRoutes = (router: IRouter) => { { path: AGENT_CONFIG_API_ROUTES.LIST_PATTERN, validate: GetAgentConfigsRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, getAgentConfigsHandler ); @@ -36,7 +38,7 @@ export const registerRoutes = (router: IRouter) => { { path: AGENT_CONFIG_API_ROUTES.INFO_PATTERN, validate: GetOneAgentConfigRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, getOneAgentConfigHandler ); @@ -46,7 +48,7 @@ export const registerRoutes = (router: IRouter) => { { path: AGENT_CONFIG_API_ROUTES.CREATE_PATTERN, validate: CreateAgentConfigRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, createAgentConfigHandler ); @@ -56,7 +58,7 @@ export const registerRoutes = (router: IRouter) => { { path: AGENT_CONFIG_API_ROUTES.UPDATE_PATTERN, validate: UpdateAgentConfigRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, updateAgentConfigHandler ); @@ -66,8 +68,18 @@ export const registerRoutes = (router: IRouter) => { { path: AGENT_CONFIG_API_ROUTES.DELETE_PATTERN, validate: DeleteAgentConfigsRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, deleteAgentConfigsHandler ); + + // Get one full agent config + router.get( + { + path: AGENT_CONFIG_API_ROUTES.FULL_INFO_PATTERN, + validate: GetFullAgentConfigRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getFullAgentConfig + ); }; diff --git a/x-pack/plugins/ingest_manager/server/routes/datasource/handlers.ts b/x-pack/plugins/ingest_manager/server/routes/datasource/handlers.ts index 78cad2e21c5faf..349e88d8fb59df 100644 --- a/x-pack/plugins/ingest_manager/server/routes/datasource/handlers.ts +++ b/x-pack/plugins/ingest_manager/server/routes/datasource/handlers.ts @@ -5,15 +5,16 @@ */ import { TypeOf } from '@kbn/config-schema'; import { RequestHandler } from 'kibana/server'; -import { datasourceService } from '../../services'; +import { appContextService, datasourceService } from '../../services'; +import { ensureInstalledPackage } from '../../services/epm/packages'; import { GetDatasourcesRequestSchema, GetOneDatasourceRequestSchema, CreateDatasourceRequestSchema, UpdateDatasourceRequestSchema, DeleteDatasourcesRequestSchema, - DeleteDatasourcesResponse, } from '../../types'; +import { CreateDatasourceResponse, DeleteDatasourcesResponse } from '../../../common'; export const getDatasourcesHandler: RequestHandler< undefined, @@ -72,10 +73,23 @@ export const createDatasourceHandler: RequestHandler< TypeOf > = async (context, request, response) => { const soClient = context.core.savedObjects.client; + const callCluster = context.core.elasticsearch.adminClient.callAsCurrentUser; + const user = (await appContextService.getSecurity()?.authc.getCurrentUser(request)) || undefined; try { - const datasource = await datasourceService.create(soClient, request.body); + // Make sure the datasource package is installed + if (request.body.package?.name) { + await ensureInstalledPackage({ + savedObjectsClient: soClient, + pkgName: request.body.package.name, + callCluster, + }); + } + + // Create datasource + const datasource = await datasourceService.create(soClient, request.body, { user }); + const body: CreateDatasourceResponse = { item: datasource, success: true }; return response.ok({ - body: { item: datasource, success: true }, + body, }); } catch (e) { return response.customError({ @@ -91,11 +105,13 @@ export const updateDatasourceHandler: RequestHandler< TypeOf > = async (context, request, response) => { const soClient = context.core.savedObjects.client; + const user = (await appContextService.getSecurity()?.authc.getCurrentUser(request)) || undefined; try { const datasource = await datasourceService.update( soClient, request.params.datasourceId, - request.body + request.body, + { user } ); return response.ok({ body: { item: datasource, success: true }, @@ -108,16 +124,18 @@ export const updateDatasourceHandler: RequestHandler< } }; -export const deleteDatasourcesHandler: RequestHandler< +export const deleteDatasourceHandler: RequestHandler< unknown, unknown, TypeOf > = async (context, request, response) => { const soClient = context.core.savedObjects.client; + const user = (await appContextService.getSecurity()?.authc.getCurrentUser(request)) || undefined; try { const body: DeleteDatasourcesResponse = await datasourceService.delete( soClient, - request.body.datasourceIds + request.body.datasourceIds, + { user } ); return response.ok({ body, diff --git a/x-pack/plugins/ingest_manager/server/routes/datasource/index.ts b/x-pack/plugins/ingest_manager/server/routes/datasource/index.ts index d9e3ba9de8838f..e5891cc7377e96 100644 --- a/x-pack/plugins/ingest_manager/server/routes/datasource/index.ts +++ b/x-pack/plugins/ingest_manager/server/routes/datasource/index.ts @@ -17,7 +17,7 @@ import { getOneDatasourceHandler, createDatasourceHandler, updateDatasourceHandler, - deleteDatasourcesHandler, + deleteDatasourceHandler, } from './handlers'; export const registerRoutes = (router: IRouter) => { @@ -26,7 +26,7 @@ export const registerRoutes = (router: IRouter) => { { path: DATASOURCE_API_ROUTES.LIST_PATTERN, validate: GetDatasourcesRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, getDatasourcesHandler ); @@ -36,7 +36,7 @@ export const registerRoutes = (router: IRouter) => { { path: DATASOURCE_API_ROUTES.INFO_PATTERN, validate: GetOneDatasourceRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, getOneDatasourceHandler ); @@ -46,7 +46,7 @@ export const registerRoutes = (router: IRouter) => { { path: DATASOURCE_API_ROUTES.CREATE_PATTERN, validate: CreateDatasourceRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, createDatasourceHandler ); @@ -56,7 +56,7 @@ export const registerRoutes = (router: IRouter) => { { path: DATASOURCE_API_ROUTES.UPDATE_PATTERN, validate: UpdateDatasourceRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, updateDatasourceHandler ); @@ -68,6 +68,6 @@ export const registerRoutes = (router: IRouter) => { validate: DeleteDatasourcesRequestSchema, options: { tags: [`access:${PLUGIN_ID}`] }, }, - deleteDatasourcesHandler + deleteDatasourceHandler ); }; diff --git a/x-pack/plugins/ingest_manager/server/routes/enrollment_api_key/handler.ts b/x-pack/plugins/ingest_manager/server/routes/enrollment_api_key/handler.ts new file mode 100644 index 00000000000000..478078a934186f --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/routes/enrollment_api_key/handler.ts @@ -0,0 +1,112 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { RequestHandler } from 'kibana/server'; +import { TypeOf } from '@kbn/config-schema'; +import { + GetEnrollmentAPIKeysRequestSchema, + PostEnrollmentAPIKeyRequestSchema, + DeleteEnrollmentAPIKeyRequestSchema, + GetOneEnrollmentAPIKeyRequestSchema, +} from '../../types'; +import { + GetEnrollmentAPIKeysResponse, + GetOneEnrollmentAPIKeyResponse, + DeleteEnrollmentAPIKeyResponse, + PostEnrollmentAPIKeyResponse, +} from '../../../common'; +import * as APIKeyService from '../../services/api_keys'; + +export const getEnrollmentApiKeysHandler: RequestHandler< + undefined, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const { items, total, page, perPage } = await APIKeyService.listEnrollmentApiKeys(soClient, { + page: request.query.page, + perPage: request.query.perPage, + kuery: request.query.kuery, + }); + const body: GetEnrollmentAPIKeysResponse = { list: items, success: true, total, page, perPage }; + + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; +export const postEnrollmentApiKeyHandler: RequestHandler< + undefined, + undefined, + TypeOf +> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const apiKey = await APIKeyService.generateEnrollmentAPIKey(soClient, { + name: request.body.name, + expiration: request.body.expiration, + configId: request.body.config_id, + }); + + const body: PostEnrollmentAPIKeyResponse = { item: apiKey, success: true, action: 'created' }; + + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const deleteEnrollmentApiKeyHandler: RequestHandler> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + await APIKeyService.deleteEnrollmentApiKey(soClient, request.params.keyId); + + const body: DeleteEnrollmentAPIKeyResponse = { action: 'deleted', success: true }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom && e.output.statusCode === 404) { + return response.notFound({ + body: { message: `EnrollmentAPIKey ${request.params.keyId} not found` }, + }); + } + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getOneEnrollmentApiKeyHandler: RequestHandler> = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + try { + const apiKey = await APIKeyService.getEnrollmentAPIKey(soClient, request.params.keyId); + const body: GetOneEnrollmentAPIKeyResponse = { item: apiKey, success: true }; + + return response.ok({ body }); + } catch (e) { + if (e.isBoom && e.output.statusCode === 404) { + return response.notFound({ + body: { message: `EnrollmentAPIKey ${request.params.keyId} not found` }, + }); + } + + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/enrollment_api_key/index.ts b/x-pack/plugins/ingest_manager/server/routes/enrollment_api_key/index.ts new file mode 100644 index 00000000000000..6df5299d30bd44 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/routes/enrollment_api_key/index.ts @@ -0,0 +1,57 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { IRouter } from 'kibana/server'; +import { PLUGIN_ID, ENROLLMENT_API_KEY_ROUTES } from '../../constants'; +import { + GetEnrollmentAPIKeysRequestSchema, + GetOneEnrollmentAPIKeyRequestSchema, + DeleteEnrollmentAPIKeyRequestSchema, + PostEnrollmentAPIKeyRequestSchema, +} from '../../types'; +import { + getEnrollmentApiKeysHandler, + getOneEnrollmentApiKeyHandler, + deleteEnrollmentApiKeyHandler, + postEnrollmentApiKeyHandler, +} from './handler'; + +export const registerRoutes = (router: IRouter) => { + router.get( + { + path: ENROLLMENT_API_KEY_ROUTES.INFO_PATTERN, + validate: GetOneEnrollmentAPIKeyRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getOneEnrollmentApiKeyHandler + ); + + router.delete( + { + path: ENROLLMENT_API_KEY_ROUTES.DELETE_PATTERN, + validate: DeleteEnrollmentAPIKeyRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, + }, + deleteEnrollmentApiKeyHandler + ); + + router.get( + { + path: ENROLLMENT_API_KEY_ROUTES.LIST_PATTERN, + validate: GetEnrollmentAPIKeysRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + getEnrollmentApiKeysHandler + ); + + router.post( + { + path: ENROLLMENT_API_KEY_ROUTES.CREATE_PATTERN, + validate: PostEnrollmentAPIKeyRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, + }, + postEnrollmentApiKeyHandler + ); +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/epm/handlers.ts b/x-pack/plugins/ingest_manager/server/routes/epm/handlers.ts new file mode 100644 index 00000000000000..6b1dde92ec0e1d --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/routes/epm/handlers.ts @@ -0,0 +1,163 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { TypeOf } from '@kbn/config-schema'; +import { RequestHandler, CustomHttpResponseOptions } from 'kibana/server'; +import { + GetPackagesRequestSchema, + GetFileRequestSchema, + GetInfoRequestSchema, + InstallPackageRequestSchema, + DeletePackageRequestSchema, +} from '../../types'; +import { + GetInfoResponse, + InstallPackageResponse, + DeletePackageResponse, + GetCategoriesResponse, + GetPackagesResponse, +} from '../../../common'; +import { + getCategories, + getPackages, + getFile, + getPackageInfo, + installPackage, + removeInstallation, +} from '../../services/epm/packages'; + +export const getCategoriesHandler: RequestHandler = async (context, request, response) => { + try { + const res = await getCategories(); + const body: GetCategoriesResponse = { + response: res, + success: true, + }; + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getListHandler: RequestHandler< + undefined, + TypeOf +> = async (context, request, response) => { + try { + const savedObjectsClient = context.core.savedObjects.client; + const res = await getPackages({ + savedObjectsClient, + category: request.query.category, + }); + const body: GetPackagesResponse = { + response: res, + success: true, + }; + return response.ok({ + body, + }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getFileHandler: RequestHandler> = async ( + context, + request, + response +) => { + try { + const { pkgkey, filePath } = request.params; + const registryResponse = await getFile(`/package/${pkgkey}/${filePath}`); + const contentType = registryResponse.headers.get('Content-Type'); + const customResponseObj: CustomHttpResponseOptions = { + body: registryResponse.body, + statusCode: registryResponse.status, + }; + if (contentType !== null) { + customResponseObj.headers = { 'Content-Type': contentType }; + } + return response.custom(customResponseObj); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const getInfoHandler: RequestHandler> = async ( + context, + request, + response +) => { + try { + const { pkgkey } = request.params; + const savedObjectsClient = context.core.savedObjects.client; + const res = await getPackageInfo({ savedObjectsClient, pkgkey }); + const body: GetInfoResponse = { + response: res, + success: true, + }; + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const installPackageHandler: RequestHandler> = async (context, request, response) => { + try { + const { pkgkey } = request.params; + const savedObjectsClient = context.core.savedObjects.client; + const callCluster = context.core.elasticsearch.adminClient.callAsCurrentUser; + const res = await installPackage({ + savedObjectsClient, + pkgkey, + callCluster, + }); + const body: InstallPackageResponse = { + response: res, + success: true, + }; + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const deletePackageHandler: RequestHandler> = async (context, request, response) => { + try { + const { pkgkey } = request.params; + const savedObjectsClient = context.core.savedObjects.client; + const callCluster = context.core.elasticsearch.adminClient.callAsCurrentUser; + const res = await removeInstallation({ savedObjectsClient, pkgkey, callCluster }); + const body: DeletePackageResponse = { + response: res, + success: true, + }; + return response.ok({ body }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/epm/index.ts b/x-pack/plugins/ingest_manager/server/routes/epm/index.ts index 7bdcafe6338431..cb9ec5cc532c49 100644 --- a/x-pack/plugins/ingest_manager/server/routes/epm/index.ts +++ b/x-pack/plugins/ingest_manager/server/routes/epm/index.ts @@ -5,71 +5,74 @@ */ import { IRouter } from 'kibana/server'; import { PLUGIN_ID, EPM_API_ROUTES } from '../../constants'; +import { + getCategoriesHandler, + getListHandler, + getFileHandler, + getInfoHandler, + installPackageHandler, + deletePackageHandler, +} from './handlers'; +import { + GetPackagesRequestSchema, + GetFileRequestSchema, + GetInfoRequestSchema, + InstallPackageRequestSchema, + DeletePackageRequestSchema, +} from '../../types'; export const registerRoutes = (router: IRouter) => { router.get( { path: EPM_API_ROUTES.CATEGORIES_PATTERN, validate: false, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, - async (context, req, res) => { - return res.ok({ body: { hello: 'world' } }); - } + getCategoriesHandler ); router.get( { path: EPM_API_ROUTES.LIST_PATTERN, - validate: false, - options: { tags: [`access:${PLUGIN_ID}`] }, + validate: GetPackagesRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, - async (context, req, res) => { - return res.ok({ body: { hello: 'world' } }); - } + getListHandler ); router.get( { - path: `${EPM_API_ROUTES.INFO_PATTERN}/{filePath*}`, - validate: false, - options: { tags: [`access:${PLUGIN_ID}`] }, + path: EPM_API_ROUTES.FILEPATH_PATTERN, + validate: GetFileRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, - async (context, req, res) => { - return res.ok({ body: { hello: 'world' } }); - } + getFileHandler ); router.get( { path: EPM_API_ROUTES.INFO_PATTERN, - validate: false, - options: { tags: [`access:${PLUGIN_ID}`] }, + validate: GetInfoRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, - async (context, req, res) => { - return res.ok({ body: { hello: 'world' } }); - } + getInfoHandler ); - router.get( + router.post( { path: EPM_API_ROUTES.INSTALL_PATTERN, - validate: false, - options: { tags: [`access:${PLUGIN_ID}`] }, + validate: InstallPackageRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, - async (context, req, res) => { - return res.ok({ body: { hello: 'world' } }); - } + installPackageHandler ); - router.get( + router.delete( { path: EPM_API_ROUTES.DELETE_PATTERN, - validate: false, - options: { tags: [`access:${PLUGIN_ID}`] }, + validate: DeletePackageRequestSchema, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, - async (context, req, res) => { - return res.ok({ body: { hello: 'world' } }); - } + deletePackageHandler ); }; diff --git a/x-pack/plugins/ingest_manager/server/routes/index.ts b/x-pack/plugins/ingest_manager/server/routes/index.ts index b458ef31dee454..33d75f3ab82cda 100644 --- a/x-pack/plugins/ingest_manager/server/routes/index.ts +++ b/x-pack/plugins/ingest_manager/server/routes/index.ts @@ -6,4 +6,7 @@ export { registerRoutes as registerAgentConfigRoutes } from './agent_config'; export { registerRoutes as registerDatasourceRoutes } from './datasource'; export { registerRoutes as registerEPMRoutes } from './epm'; -export { registerRoutes as registerFleetSetupRoutes } from './fleet_setup'; +export { registerRoutes as registerSetupRoutes } from './setup'; +export { registerRoutes as registerAgentRoutes } from './agent'; +export { registerRoutes as registerEnrollmentApiKeyRoutes } from './enrollment_api_key'; +export { registerRoutes as registerInstallScriptRoutes } from './install_script'; diff --git a/x-pack/plugins/ingest_manager/server/routes/install_script/index.ts b/x-pack/plugins/ingest_manager/server/routes/install_script/index.ts new file mode 100644 index 00000000000000..5470df31adbddb --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/routes/install_script/index.ts @@ -0,0 +1,44 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import url from 'url'; +import { IRouter, BasePath, HttpServerInfo, KibanaRequest } from 'kibana/server'; +import { INSTALL_SCRIPT_API_ROUTES } from '../../constants'; +import { getScript } from '../../services/install_script'; +import { InstallScriptRequestSchema } from '../../types'; + +export const registerRoutes = ({ + router, + basePath, + serverInfo, +}: { + router: IRouter; + basePath: Pick; + serverInfo: HttpServerInfo; +}) => { + const kibanaUrl = url.format({ + protocol: serverInfo.protocol, + hostname: serverInfo.host, + port: serverInfo.port, + pathname: basePath.serverBasePath, + }); + + router.get( + { + path: INSTALL_SCRIPT_API_ROUTES, + validate: InstallScriptRequestSchema, + options: { tags: [], authRequired: false }, + }, + async function getInstallScriptHandler( + context, + request: KibanaRequest<{ osType: 'macos' }>, + response + ) { + const script = getScript(request.params.osType, kibanaUrl); + + return response.ok({ body: script }); + } + ); +}; diff --git a/x-pack/plugins/ingest_manager/server/routes/fleet_setup/handlers.ts b/x-pack/plugins/ingest_manager/server/routes/setup/handlers.ts similarity index 58% rename from x-pack/plugins/ingest_manager/server/routes/fleet_setup/handlers.ts rename to x-pack/plugins/ingest_manager/server/routes/setup/handlers.ts index 72fe34eb23c5ff..30e725bb5ad4a9 100644 --- a/x-pack/plugins/ingest_manager/server/routes/fleet_setup/handlers.ts +++ b/x-pack/plugins/ingest_manager/server/routes/setup/handlers.ts @@ -5,17 +5,18 @@ */ import { TypeOf } from '@kbn/config-schema'; import { RequestHandler } from 'kibana/server'; -import { DEFAULT_OUTPUT_ID } from '../../constants'; import { outputService, agentConfigService } from '../../services'; import { CreateFleetSetupRequestSchema, CreateFleetSetupResponse } from '../../types'; +import { setup } from '../../services/setup'; +import { generateEnrollmentAPIKey } from '../../services/api_keys'; export const getFleetSetupHandler: RequestHandler = async (context, request, response) => { const soClient = context.core.savedObjects.client; const successBody: CreateFleetSetupResponse = { isInitialized: true }; const failureBody: CreateFleetSetupResponse = { isInitialized: false }; try { - const output = await outputService.get(soClient, DEFAULT_OUTPUT_ID); - if (output) { + const adminUser = await outputService.getAdminUser(soClient); + if (adminUser) { return response.ok({ body: successBody, }); @@ -38,11 +39,31 @@ export const createFleetSetupHandler: RequestHandler< > = async (context, request, response) => { const soClient = context.core.savedObjects.client; try { - await outputService.createDefaultOutput(soClient, { - username: request.body.admin_username, - password: request.body.admin_password, + await outputService.updateOutput(soClient, await outputService.getDefaultOutputId(soClient), { + admin_username: request.body.admin_username, + admin_password: request.body.admin_password, }); - await agentConfigService.ensureDefaultAgentConfig(soClient); + await generateEnrollmentAPIKey(soClient, { + name: 'Default', + configId: await agentConfigService.getDefaultAgentConfigId(soClient), + }); + + return response.ok({ + body: { isInitialized: true }, + }); + } catch (e) { + return response.customError({ + statusCode: 500, + body: { message: e.message }, + }); + } +}; + +export const ingestManagerSetupHandler: RequestHandler = async (context, request, response) => { + const soClient = context.core.savedObjects.client; + const callCluster = context.core.elasticsearch.adminClient.callAsCurrentUser; + try { + await setup(soClient, callCluster); return response.ok({ body: { isInitialized: true }, }); diff --git a/x-pack/plugins/ingest_manager/server/routes/fleet_setup/index.ts b/x-pack/plugins/ingest_manager/server/routes/setup/index.ts similarity index 51% rename from x-pack/plugins/ingest_manager/server/routes/fleet_setup/index.ts rename to x-pack/plugins/ingest_manager/server/routes/setup/index.ts index c23164db6b6ebc..7e09d8dbef1f65 100644 --- a/x-pack/plugins/ingest_manager/server/routes/fleet_setup/index.ts +++ b/x-pack/plugins/ingest_manager/server/routes/setup/index.ts @@ -4,27 +4,42 @@ * you may not use this file except in compliance with the Elastic License. */ import { IRouter } from 'kibana/server'; -import { PLUGIN_ID, FLEET_SETUP_API_ROUTES } from '../../constants'; +import { PLUGIN_ID, FLEET_SETUP_API_ROUTES, SETUP_API_ROUTE } from '../../constants'; import { GetFleetSetupRequestSchema, CreateFleetSetupRequestSchema } from '../../types'; -import { getFleetSetupHandler, createFleetSetupHandler } from './handlers'; +import { + getFleetSetupHandler, + createFleetSetupHandler, + ingestManagerSetupHandler, +} from './handlers'; export const registerRoutes = (router: IRouter) => { - // Get + // Ingest manager setup + router.post( + { + path: SETUP_API_ROUTE, + validate: false, + // if this route is set to `-all`, a read-only user get a 404 for this route + // and will see `Unable to initialize Ingest Manager` in the UI + options: { tags: [`access:${PLUGIN_ID}-read`] }, + }, + ingestManagerSetupHandler + ); + // Get Fleet setup router.get( { path: FLEET_SETUP_API_ROUTES.INFO_PATTERN, validate: GetFleetSetupRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-read`] }, }, getFleetSetupHandler ); - // Create + // Create Fleet setup router.post( { path: FLEET_SETUP_API_ROUTES.CREATE_PATTERN, validate: CreateFleetSetupRequestSchema, - options: { tags: [`access:${PLUGIN_ID}`] }, + options: { tags: [`access:${PLUGIN_ID}-all`] }, }, createFleetSetupHandler ); diff --git a/x-pack/plugins/ingest_manager/server/saved_objects.ts b/x-pack/plugins/ingest_manager/server/saved_objects.ts index 976556f388acf8..860b95b58c7f75 100644 --- a/x-pack/plugins/ingest_manager/server/saved_objects.ts +++ b/x-pack/plugins/ingest_manager/server/saved_objects.ts @@ -7,6 +7,10 @@ import { OUTPUT_SAVED_OBJECT_TYPE, AGENT_CONFIG_SAVED_OBJECT_TYPE, DATASOURCE_SAVED_OBJECT_TYPE, + PACKAGES_SAVED_OBJECT_TYPE, + AGENT_SAVED_OBJECT_TYPE, + AGENT_EVENT_SAVED_OBJECT_TYPE, + ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, } from './constants'; /* @@ -15,28 +19,87 @@ import { * Please update typings in `/common/types` if mappings are updated. */ export const savedObjectMappings = { + [AGENT_SAVED_OBJECT_TYPE]: { + properties: { + shared_id: { type: 'keyword' }, + type: { type: 'keyword' }, + active: { type: 'boolean' }, + enrolled_at: { type: 'date' }, + access_api_key_id: { type: 'keyword' }, + version: { type: 'keyword' }, + user_provided_metadata: { type: 'text' }, + local_metadata: { type: 'text' }, + config_id: { type: 'keyword' }, + last_updated: { type: 'date' }, + last_checkin: { type: 'date' }, + config_updated_at: { type: 'date' }, + // FIXME_INGEST https://github.com/elastic/kibana/issues/56554 + default_api_key: { type: 'keyword' }, + updated_at: { type: 'date' }, + current_error_events: { type: 'text' }, + // FIXME_INGEST https://github.com/elastic/kibana/issues/56554 + actions: { + type: 'nested', + properties: { + id: { type: 'keyword' }, + type: { type: 'keyword' }, + data: { type: 'text' }, + sent_at: { type: 'date' }, + created_at: { type: 'date' }, + }, + }, + }, + }, + [AGENT_EVENT_SAVED_OBJECT_TYPE]: { + properties: { + type: { type: 'keyword' }, + subtype: { type: 'keyword' }, + agent_id: { type: 'keyword' }, + action_id: { type: 'keyword' }, + config_id: { type: 'keyword' }, + stream_id: { type: 'keyword' }, + timestamp: { type: 'date' }, + message: { type: 'text' }, + payload: { type: 'text' }, + data: { type: 'text' }, + }, + }, [AGENT_CONFIG_SAVED_OBJECT_TYPE]: { properties: { id: { type: 'keyword' }, name: { type: 'text' }, + is_default: { type: 'boolean' }, namespace: { type: 'keyword' }, description: { type: 'text' }, status: { type: 'keyword' }, datasources: { type: 'keyword' }, updated_on: { type: 'keyword' }, updated_by: { type: 'keyword' }, + revision: { type: 'integer' }, + }, + }, + [ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE]: { + properties: { + name: { type: 'keyword' }, + type: { type: 'keyword' }, + // FIXME_INGEST https://github.com/elastic/kibana/issues/56554 + api_key: { type: 'binary' }, + api_key_id: { type: 'keyword' }, + config_id: { type: 'keyword' }, + created_at: { type: 'date' }, + updated_at: { type: 'date' }, + expire_at: { type: 'date' }, + active: { type: 'boolean' }, }, }, [OUTPUT_SAVED_OBJECT_TYPE]: { properties: { - id: { type: 'keyword' }, name: { type: 'keyword' }, type: { type: 'keyword' }, - username: { type: 'keyword' }, - password: { type: 'keyword' }, - index_name: { type: 'keyword' }, - ingest_pipeline: { type: 'keyword' }, + is_default: { type: 'boolean' }, hosts: { type: 'keyword' }, + ca_sha256: { type: 'keyword' }, + // FIXME_INGEST https://github.com/elastic/kibana/issues/56554 api_key: { type: 'keyword' }, admin_username: { type: 'binary' }, admin_password: { type: 'binary' }, @@ -45,32 +108,49 @@ export const savedObjectMappings = { }, [DATASOURCE_SAVED_OBJECT_TYPE]: { properties: { - id: { type: 'keyword' }, name: { type: 'keyword' }, + description: { type: 'text' }, namespace: { type: 'keyword' }, - read_alias: { type: 'keyword' }, - agent_config_id: { type: 'keyword' }, + config_id: { type: 'keyword' }, + enabled: { type: 'boolean' }, package: { properties: { - assets: { + name: { type: 'keyword' }, + title: { type: 'keyword' }, + version: { type: 'keyword' }, + }, + }, + output_id: { type: 'keyword' }, + inputs: { + type: 'nested', + properties: { + type: { type: 'keyword' }, + enabled: { type: 'boolean' }, + processors: { type: 'keyword' }, + streams: { + type: 'nested', properties: { id: { type: 'keyword' }, - type: { type: 'keyword' }, + enabled: { type: 'boolean' }, + dataset: { type: 'keyword' }, + processors: { type: 'keyword' }, + config: { type: 'flattened' }, }, }, - description: { type: 'keyword' }, - name: { type: 'keyword' }, - title: { type: 'keyword' }, - version: { type: 'keyword' }, }, }, - streams: { + revision: { type: 'integer' }, + }, + }, + [PACKAGES_SAVED_OBJECT_TYPE]: { + properties: { + name: { type: 'keyword' }, + version: { type: 'keyword' }, + installed: { + type: 'nested', properties: { - config: { type: 'flattened' }, id: { type: 'keyword' }, - input: { type: 'flattened' }, - output_id: { type: 'keyword' }, - processors: { type: 'keyword' }, + type: { type: 'keyword' }, }, }, }, diff --git a/x-pack/plugins/ingest_manager/server/services/agent_config.ts b/x-pack/plugins/ingest_manager/server/services/agent_config.ts index 0690e115ca862f..c0eb614102b29a 100644 --- a/x-pack/plugins/ingest_manager/server/services/agent_config.ts +++ b/x-pack/plugins/ingest_manager/server/services/agent_config.ts @@ -5,37 +5,29 @@ */ import { SavedObjectsClientContract } from 'kibana/server'; import { AuthenticatedUser } from '../../../security/server'; +import { DEFAULT_AGENT_CONFIG, AGENT_CONFIG_SAVED_OBJECT_TYPE } from '../constants'; import { - DEFAULT_AGENT_CONFIG_ID, - DEFAULT_AGENT_CONFIG, - AGENT_CONFIG_SAVED_OBJECT_TYPE, -} from '../constants'; -import { + Datasource, NewAgentConfig, AgentConfig, + FullAgentConfig, AgentConfigStatus, - AgentConfigUpdateHandler, ListWithKuery, - DeleteAgentConfigsResponse, } from '../types'; +import { DeleteAgentConfigsResponse, storedDatasourceToAgentDatasource } from '../../common'; import { datasourceService } from './datasource'; +import { outputService } from './output'; +import { agentConfigUpdateEventHandler } from './agent_config_update'; const SAVED_OBJECT_TYPE = AGENT_CONFIG_SAVED_OBJECT_TYPE; class AgentConfigService { - private eventsHandler: AgentConfigUpdateHandler[] = []; - - public registerAgentConfigUpdateHandler(handler: AgentConfigUpdateHandler) { - this.eventsHandler.push(handler); - } - - public triggerAgentConfigUpdatedEvent: AgentConfigUpdateHandler = async ( - action, - agentConfigId + private triggerAgentConfigUpdatedEvent = async ( + soClient: SavedObjectsClientContract, + action: string, + agentConfigId: string ) => { - for (const handler of this.eventsHandler) { - await handler(action, agentConfigId); - } + return agentConfigUpdateEventHandler(soClient, action, agentConfigId); }; private async _update( @@ -44,37 +36,51 @@ class AgentConfigService { agentConfig: Partial, user?: AuthenticatedUser ): Promise { + const oldAgentConfig = await this.get(soClient, id, false); + + if (!oldAgentConfig) { + throw new Error('Agent config not found'); + } + + if ( + oldAgentConfig.status === AgentConfigStatus.Inactive && + agentConfig.status !== AgentConfigStatus.Active + ) { + throw new Error( + `Agent config ${id} cannot be updated because it is ${oldAgentConfig.status}` + ); + } + await soClient.update(SAVED_OBJECT_TYPE, id, { ...agentConfig, + revision: oldAgentConfig.revision + 1, updated_on: new Date().toString(), updated_by: user ? user.username : 'system', }); - await this.triggerAgentConfigUpdatedEvent('updated', id); + await this.triggerAgentConfigUpdatedEvent(soClient, 'updated', id); return (await this.get(soClient, id)) as AgentConfig; } public async ensureDefaultAgentConfig(soClient: SavedObjectsClientContract) { - let defaultAgentConfig; - - try { - defaultAgentConfig = await this.get(soClient, DEFAULT_AGENT_CONFIG_ID); - } catch (err) { - if (!err.isBoom || err.output.statusCode !== 404) { - throw err; - } - } + const configs = await soClient.find({ + type: AGENT_CONFIG_SAVED_OBJECT_TYPE, + filter: 'agent_configs.attributes.is_default:true', + }); - if (!defaultAgentConfig) { + if (configs.total === 0) { const newDefaultAgentConfig: NewAgentConfig = { ...DEFAULT_AGENT_CONFIG, }; - await this.create(soClient, newDefaultAgentConfig, { - id: DEFAULT_AGENT_CONFIG_ID, - }); + return this.create(soClient, newDefaultAgentConfig); } + + return { + id: configs.saved_objects[0].id, + ...configs.saved_objects[0].attributes, + }; } public async create( @@ -86,13 +92,16 @@ class AgentConfigService { SAVED_OBJECT_TYPE, { ...agentConfig, + revision: 1, updated_on: new Date().toISOString(), updated_by: options?.user?.username || 'system', } as AgentConfig, options ); - await this.triggerAgentConfigUpdatedEvent('created', newSo.id); + if (!agentConfig.is_default) { + await this.triggerAgentConfigUpdatedEvent(soClient, 'created', newSo.id); + } return { id: newSo.id, @@ -100,7 +109,11 @@ class AgentConfigService { }; } - public async get(soClient: SavedObjectsClientContract, id: string): Promise { + public async get( + soClient: SavedObjectsClientContract, + id: string, + withDatasources: boolean = true + ): Promise { const agentConfigSO = await soClient.get(SAVED_OBJECT_TYPE, id); if (!agentConfigSO) { return null; @@ -110,15 +123,20 @@ class AgentConfigService { throw new Error(agentConfigSO.error.message); } - return { + const agentConfig: AgentConfig = { id: agentConfigSO.id, ...agentConfigSO.attributes, - datasources: + }; + + if (withDatasources) { + agentConfig.datasources = (await datasourceService.getByIDs( soClient, (agentConfigSO.attributes.datasources as string[]) || [] - )) || [], - }; + )) || []; + } + + return agentConfig; } public async list( @@ -159,31 +177,24 @@ class AgentConfigService { agentConfig: Partial, options?: { user?: AuthenticatedUser } ): Promise { - const oldAgentConfig = await this.get(soClient, id); - - if (!oldAgentConfig) { - throw new Error('Agent config not found'); - } - - if ( - oldAgentConfig.status === AgentConfigStatus.Inactive && - agentConfig.status !== AgentConfigStatus.Active - ) { - throw new Error( - `Agent config ${id} cannot be updated because it is ${oldAgentConfig.status}` - ); - } - return this._update(soClient, id, agentConfig, options?.user); } + public async bumpRevision( + soClient: SavedObjectsClientContract, + id: string, + options?: { user?: AuthenticatedUser } + ): Promise { + return this._update(soClient, id, {}, options?.user); + } + public async assignDatasources( soClient: SavedObjectsClientContract, id: string, datasourceIds: string[], options?: { user?: AuthenticatedUser } ): Promise { - const oldAgentConfig = await this.get(soClient, id); + const oldAgentConfig = await this.get(soClient, id, false); if (!oldAgentConfig) { throw new Error('Agent config not found'); @@ -206,7 +217,7 @@ class AgentConfigService { datasourceIds: string[], options?: { user?: AuthenticatedUser } ): Promise { - const oldAgentConfig = await this.get(soClient, id); + const oldAgentConfig = await this.get(soClient, id, false); if (!oldAgentConfig) { throw new Error('Agent config not found'); @@ -225,20 +236,34 @@ class AgentConfigService { ); } + public async getDefaultAgentConfigId(soClient: SavedObjectsClientContract) { + const configs = await soClient.find({ + type: AGENT_CONFIG_SAVED_OBJECT_TYPE, + filter: 'agent_configs.attributes.is_default:true', + }); + + if (configs.saved_objects.length === 0) { + throw new Error('No default agent config'); + } + + return configs.saved_objects[0].id; + } + public async delete( soClient: SavedObjectsClientContract, ids: string[] ): Promise { const result: DeleteAgentConfigsResponse = []; + const defaultConfigId = await this.getDefaultAgentConfigId(soClient); - if (ids.includes(DEFAULT_AGENT_CONFIG_ID)) { + if (ids.includes(defaultConfigId)) { throw new Error('The default agent configuration cannot be deleted'); } for (const id of ids) { try { await soClient.delete(SAVED_OBJECT_TYPE, id); - await this.triggerAgentConfigUpdatedEvent('deleted', id); + await this.triggerAgentConfigUpdatedEvent(soClient, 'deleted', id); result.push({ id, success: true, @@ -253,6 +278,50 @@ class AgentConfigService { return result; } + + public async getFullConfig( + soClient: SavedObjectsClientContract, + id: string + ): Promise { + let config; + + try { + config = await this.get(soClient, id); + } catch (err) { + if (!err.isBoom || err.output.statusCode !== 404) { + throw err; + } + } + + if (!config) { + return null; + } + + const agentConfig: FullAgentConfig = { + id: config.id, + outputs: { + // TEMPORARY as we only support a default output + ...[ + await outputService.get(soClient, await outputService.getDefaultOutputId(soClient)), + ].reduce((outputs, { config: outputConfig, name, type, hosts, ca_sha256, api_key }) => { + outputs[name] = { + type, + hosts, + ca_sha256, + api_key, + ...outputConfig, + }; + return outputs; + }, {} as FullAgentConfig['outputs']), + }, + datasources: (config.datasources as Datasource[]).map(ds => + storedDatasourceToAgentDatasource(ds) + ), + revision: config.revision, + }; + + return agentConfig; + } } export const agentConfigService = new AgentConfigService(); diff --git a/x-pack/plugins/ingest_manager/server/services/agent_config_update.ts b/x-pack/plugins/ingest_manager/server/services/agent_config_update.ts new file mode 100644 index 00000000000000..38894ff321a0b6 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agent_config_update.ts @@ -0,0 +1,30 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { generateEnrollmentAPIKey, deleteEnrollmentApiKeyForConfigId } from './api_keys'; +import { updateAgentsForConfigId, unenrollForConfigId } from './agents'; + +export async function agentConfigUpdateEventHandler( + soClient: SavedObjectsClientContract, + action: string, + configId: string +) { + if (action === 'created') { + await generateEnrollmentAPIKey(soClient, { + configId, + }); + } + + if (action === 'updated') { + await updateAgentsForConfigId(soClient, configId); + } + + if (action === 'deleted') { + await unenrollForConfigId(soClient, configId); + await deleteEnrollmentApiKeyForConfigId(soClient, configId); + } +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/acks.ts b/x-pack/plugins/ingest_manager/server/services/agents/acks.ts new file mode 100644 index 00000000000000..1732ff9cf5b5c8 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/acks.ts @@ -0,0 +1,28 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { Agent, AgentSOAttributes } from '../../types'; +import { AGENT_SAVED_OBJECT_TYPE } from '../../constants'; + +export async function acknowledgeAgentActions( + soClient: SavedObjectsClientContract, + agent: Agent, + actionIds: string[] +) { + const now = new Date().toISOString(); + + const updatedActions = agent.actions.map(action => { + if (action.sent_at) { + return action; + } + return { ...action, sent_at: actionIds.indexOf(action.id) >= 0 ? now : undefined }; + }); + + await soClient.update(AGENT_SAVED_OBJECT_TYPE, agent.id, { + actions: updatedActions, + }); +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/checkin.ts b/x-pack/plugins/ingest_manager/server/services/agents/checkin.ts new file mode 100644 index 00000000000000..76dfc0867fb4ed --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/checkin.ts @@ -0,0 +1,162 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract, SavedObjectsBulkCreateObject } from 'kibana/server'; +import uuid from 'uuid'; +import { + Agent, + AgentEvent, + AgentAction, + AgentSOAttributes, + AgentEventSOAttributes, +} from '../../types'; + +import { agentConfigService } from '../agent_config'; +import * as APIKeysService from '../api_keys'; +import { AGENT_SAVED_OBJECT_TYPE, AGENT_EVENT_SAVED_OBJECT_TYPE } from '../../constants'; + +export async function agentCheckin( + soClient: SavedObjectsClientContract, + agent: Agent, + events: AgentEvent[], + localMetadata?: any +) { + const updateData: { + last_checkin: string; + default_api_key?: string; + actions?: AgentAction[]; + local_metadata?: string; + current_error_events?: string; + } = { + last_checkin: new Date().toISOString(), + }; + + const actions = filterActionsForCheckin(agent); + + // Generate new agent config if config is updated + if (isNewAgentConfig(agent) && agent.config_id) { + const config = await agentConfigService.getFullConfig(soClient, agent.config_id); + if (config) { + // Assign output API keys + // We currently only support default ouput + if (!agent.default_api_key) { + updateData.default_api_key = await APIKeysService.generateOutputApiKey( + soClient, + 'default', + agent.id + ); + } + // Mutate the config to set the api token for this agent + config.outputs.default.api_key = agent.default_api_key || updateData.default_api_key; + + const configChangeAction: AgentAction = { + id: uuid.v4(), + type: 'CONFIG_CHANGE', + created_at: new Date().toISOString(), + data: JSON.stringify({ + config, + }), + sent_at: undefined, + }; + actions.push(configChangeAction); + // persist new action + updateData.actions = actions; + } + } + if (localMetadata) { + updateData.local_metadata = JSON.stringify(localMetadata); + } + + const { updatedErrorEvents } = await processEventsForCheckin(soClient, agent, events); + + // Persist changes + if (updatedErrorEvents) { + updateData.current_error_events = JSON.stringify(updatedErrorEvents); + } + + await soClient.update(AGENT_SAVED_OBJECT_TYPE, agent.id, updateData); + + return { actions }; +} + +async function processEventsForCheckin( + soClient: SavedObjectsClientContract, + agent: Agent, + events: AgentEvent[] +) { + const acknowledgedActionIds: string[] = []; + const updatedErrorEvents = [...agent.current_error_events]; + for (const event of events) { + // @ts-ignore + event.config_id = agent.config_id; + + if (isActionEvent(event)) { + acknowledgedActionIds.push(event.action_id as string); + } + + if (isErrorOrState(event)) { + // Remove any global or specific to a stream event + const existingEventIndex = updatedErrorEvents.findIndex(e => e.stream_id === event.stream_id); + if (existingEventIndex >= 0) { + updatedErrorEvents.splice(existingEventIndex, 1); + } + if (event.type === 'ERROR') { + updatedErrorEvents.push(event); + } + } + } + + if (events.length > 0) { + await createEventsForAgent(soClient, agent.id, events); + } + + return { + acknowledgedActionIds, + updatedErrorEvents, + }; +} + +async function createEventsForAgent( + soClient: SavedObjectsClientContract, + agentId: string, + events: AgentEvent[] +) { + const objects: Array> = events.map( + eventData => { + return { + attributes: { + ...eventData, + payload: eventData.payload ? JSON.stringify(eventData.payload) : undefined, + }, + type: AGENT_EVENT_SAVED_OBJECT_TYPE, + }; + } + ); + + return soClient.bulkCreate(objects); +} + +function isErrorOrState(event: AgentEvent) { + return event.type === 'STATE' || event.type === 'ERROR'; +} + +function isActionEvent(event: AgentEvent) { + return ( + event.type === 'ACTION' && (event.subtype === 'ACKNOWLEDGED' || event.subtype === 'UNKNOWN') + ); +} + +function isNewAgentConfig(agent: Agent) { + const isFirstCheckin = !agent.last_checkin; + const isConfigUpdatedSinceLastCheckin = + agent.last_checkin && agent.config_updated_at && agent.last_checkin <= agent.config_updated_at; + + return isFirstCheckin || isConfigUpdatedSinceLastCheckin; +} + +function filterActionsForCheckin(agent: Agent): AgentAction[] { + return agent.actions.filter((a: AgentAction) => !a.sent_at); +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/crud.ts b/x-pack/plugins/ingest_manager/server/services/agents/crud.ts new file mode 100644 index 00000000000000..bcd825fee87258 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/crud.ts @@ -0,0 +1,156 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import Boom from 'boom'; +import { SavedObjectsClientContract } from 'kibana/server'; +import { + AGENT_SAVED_OBJECT_TYPE, + AGENT_EVENT_SAVED_OBJECT_TYPE, + AGENT_TYPE_EPHEMERAL, + AGENT_POLLING_THRESHOLD_MS, +} from '../../constants'; +import { AgentSOAttributes, Agent, AgentEventSOAttributes } from '../../types'; +import { savedObjectToAgent } from './saved_objects'; + +export async function listAgents( + soClient: SavedObjectsClientContract, + options: { + page: number; + perPage: number; + kuery?: string; + showInactive: boolean; + } +) { + const { page, perPage, kuery, showInactive = false } = options; + + const filters = []; + + if (kuery && kuery !== '') { + // To ensure users dont need to know about SO data structure... + filters.push(kuery.replace(/agents\./g, 'agents.attributes.')); + } + + if (showInactive === false) { + const agentActiveCondition = `agents.attributes.active:true AND not agents.attributes.type:${AGENT_TYPE_EPHEMERAL}`; + const recentlySeenEphemeralAgent = `agents.attributes.active:true AND agents.attributes.type:${AGENT_TYPE_EPHEMERAL} AND agents.attributes.last_checkin > ${Date.now() - + 3 * AGENT_POLLING_THRESHOLD_MS}`; + filters.push(`(${agentActiveCondition}) OR (${recentlySeenEphemeralAgent})`); + } + + const { saved_objects, total } = await soClient.find({ + type: AGENT_SAVED_OBJECT_TYPE, + page, + perPage, + filter: _joinFilters(filters), + ..._getSortFields(), + }); + + const agents: Agent[] = saved_objects.map(savedObjectToAgent); + + return { + agents, + total, + page, + perPage, + }; +} + +export async function getAgent(soClient: SavedObjectsClientContract, agentId: string) { + const agent = savedObjectToAgent( + await soClient.get(AGENT_SAVED_OBJECT_TYPE, agentId) + ); + return agent; +} + +export async function getAgentByAccessAPIKeyId( + soClient: SavedObjectsClientContract, + accessAPIKeyId: string +) { + const response = await soClient.find({ + type: AGENT_SAVED_OBJECT_TYPE, + searchFields: ['access_api_key_id'], + search: accessAPIKeyId, + }); + + const [agent] = response.saved_objects.map(savedObjectToAgent); + + if (!agent) { + throw Boom.notFound('Agent not found'); + } + if (!agent.active) { + throw Boom.forbidden('Agent inactive'); + } + + return agent; +} + +export async function updateAgent( + soClient: SavedObjectsClientContract, + agentId: string, + data: { + userProvidedMetatada: any; + } +) { + await soClient.update(AGENT_SAVED_OBJECT_TYPE, agentId, { + user_provided_metadata: JSON.stringify(data.userProvidedMetatada), + }); +} + +export async function deleteAgent(soClient: SavedObjectsClientContract, agentId: string) { + const agent = await getAgent(soClient, agentId); + if (agent.type === 'EPHEMERAL') { + // Delete events + let more = true; + while (more === true) { + const { saved_objects: events } = await soClient.find({ + type: AGENT_EVENT_SAVED_OBJECT_TYPE, + fields: ['id'], + search: agentId, + searchFields: ['agent_id'], + perPage: 1000, + }); + if (events.length === 0) { + more = false; + } + for (const event of events) { + await soClient.delete(AGENT_EVENT_SAVED_OBJECT_TYPE, event.id); + } + } + await soClient.delete(AGENT_SAVED_OBJECT_TYPE, agentId); + return; + } + + await soClient.update(AGENT_SAVED_OBJECT_TYPE, agentId, { + active: false, + }); +} + +function _getSortFields(sortOption?: string) { + switch (sortOption) { + case 'ASC': + return { + sortField: 'enrolled_at', + sortOrder: 'ASC', + }; + + case 'DESC': + default: + return { + sortField: 'enrolled_at', + sortOrder: 'DESC', + }; + } +} + +function _joinFilters(filters: string[], operator = 'AND') { + return filters.reduce((acc: string | undefined, filter) => { + if (acc) { + return `${acc} ${operator} (${filter})`; + } + + return `(${filter})`; + }, undefined); +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/enroll.ts b/x-pack/plugins/ingest_manager/server/services/agents/enroll.ts new file mode 100644 index 00000000000000..b48d311da4440a --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/enroll.ts @@ -0,0 +1,84 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import Boom from 'boom'; +import { SavedObjectsClientContract } from 'kibana/server'; +import { AgentType, Agent, AgentSOAttributes } from '../../types'; +import { savedObjectToAgent } from './saved_objects'; +import { AGENT_SAVED_OBJECT_TYPE } from '../../constants'; +import * as APIKeyService from '../api_keys'; + +export async function enroll( + soClient: SavedObjectsClientContract, + type: AgentType, + configId: string, + metadata?: { local: any; userProvided: any }, + sharedId?: string +): Promise { + const existingAgent = sharedId ? await getAgentBySharedId(soClient, sharedId) : null; + + if (existingAgent && existingAgent.active === true) { + throw Boom.badRequest('Impossible to enroll an already active agent'); + } + + const enrolledAt = new Date().toISOString(); + + const agentData: AgentSOAttributes = { + shared_id: sharedId, + active: true, + config_id: configId, + type, + enrolled_at: enrolledAt, + user_provided_metadata: JSON.stringify(metadata?.userProvided ?? {}), + local_metadata: JSON.stringify(metadata?.local ?? {}), + current_error_events: undefined, + actions: [], + access_api_key_id: undefined, + config_updated_at: undefined, + last_checkin: undefined, + default_api_key: undefined, + }; + + let agent; + if (existingAgent) { + await soClient.update(AGENT_SAVED_OBJECT_TYPE, existingAgent.id, agentData); + agent = { + ...existingAgent, + ...agentData, + user_provided_metadata: metadata?.userProvided ?? {}, + local_metadata: metadata?.local ?? {}, + current_error_events: [], + } as Agent; + } else { + agent = savedObjectToAgent( + await soClient.create(AGENT_SAVED_OBJECT_TYPE, agentData) + ); + } + + const accessAPIKey = await APIKeyService.generateAccessApiKey(soClient, agent.id, configId); + + await soClient.update(AGENT_SAVED_OBJECT_TYPE, agent.id, { + access_api_key_id: accessAPIKey.id, + }); + + return { ...agent, access_api_key: accessAPIKey.key }; +} + +async function getAgentBySharedId(soClient: SavedObjectsClientContract, sharedId: string) { + const response = await soClient.find({ + type: AGENT_SAVED_OBJECT_TYPE, + searchFields: ['shared_id'], + search: sharedId, + }); + + const agents = response.saved_objects.map(savedObjectToAgent); + + if (agents.length > 0) { + return agents[0]; + } + + return null; +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/events.ts b/x-pack/plugins/ingest_manager/server/services/agents/events.ts new file mode 100644 index 00000000000000..908d289fbc4bba --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/events.ts @@ -0,0 +1,45 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { AGENT_EVENT_SAVED_OBJECT_TYPE } from '../../constants'; +import { AgentEventSOAttributes, AgentEvent } from '../../types'; + +export async function getAgentEvents( + soClient: SavedObjectsClientContract, + agentId: string, + options: { + kuery?: string; + page: number; + perPage: number; + } +) { + const { page, perPage, kuery } = options; + + const { total, saved_objects } = await soClient.find({ + type: AGENT_EVENT_SAVED_OBJECT_TYPE, + filter: + kuery && kuery !== '' + ? kuery.replace(/agent_events\./g, 'agent_events.attributes.') + : undefined, + perPage, + page, + sortField: 'timestamp', + sortOrder: 'DESC', + defaultSearchOperator: 'AND', + search: agentId, + searchFields: ['agent_id'], + }); + + const items: AgentEvent[] = saved_objects.map(so => { + return { + ...so.attributes, + payload: so.attributes.payload ? JSON.parse(so.attributes.payload) : undefined, + }; + }); + + return { items, total }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/index.ts b/x-pack/plugins/ingest_manager/server/services/agents/index.ts new file mode 100644 index 00000000000000..477f081d1900b1 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/index.ts @@ -0,0 +1,14 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export * from './acks'; +export * from './events'; +export * from './checkin'; +export * from './enroll'; +export * from './unenroll'; +export * from './status'; +export * from './crud'; +export * from './update'; diff --git a/x-pack/plugins/ingest_manager/server/services/agents/saved_objects.ts b/x-pack/plugins/ingest_manager/server/services/agents/saved_objects.ts new file mode 100644 index 00000000000000..adb096a444903f --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/saved_objects.ts @@ -0,0 +1,26 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObject } from 'kibana/server'; +import { Agent, AgentSOAttributes } from '../../types'; + +export function savedObjectToAgent(so: SavedObject): Agent { + if (so.error) { + throw new Error(so.error.message); + } + + return { + id: so.id, + ...so.attributes, + current_error_events: so.attributes.current_error_events + ? JSON.parse(so.attributes.current_error_events) + : [], + local_metadata: JSON.parse(so.attributes.local_metadata), + user_provided_metadata: JSON.parse(so.attributes.user_provided_metadata), + access_api_key: undefined, + status: undefined, + }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/status.ts b/x-pack/plugins/ingest_manager/server/services/agents/status.ts new file mode 100644 index 00000000000000..f6477bf1c7334e --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/status.ts @@ -0,0 +1,95 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { listAgents } from './crud'; +import { AGENT_EVENT_SAVED_OBJECT_TYPE } from '../../constants'; +import { AgentStatus, Agent } from '../../types'; + +import { + AGENT_POLLING_THRESHOLD_MS, + AGENT_TYPE_PERMANENT, + AGENT_TYPE_TEMPORARY, + AGENT_TYPE_EPHEMERAL, +} from '../../constants'; +import { AgentStatusKueryHelper } from '../../../common/services'; + +export function getAgentStatus(agent: Agent, now: number = Date.now()): AgentStatus { + const { type, last_checkin: lastCheckIn } = agent; + const msLastCheckIn = new Date(lastCheckIn || 0).getTime(); + const msSinceLastCheckIn = new Date().getTime() - msLastCheckIn; + const intervalsSinceLastCheckIn = Math.floor(msSinceLastCheckIn / AGENT_POLLING_THRESHOLD_MS); + if (!agent.active) { + return 'inactive'; + } + if (agent.current_error_events.length > 0) { + return 'error'; + } + switch (type) { + case AGENT_TYPE_PERMANENT: + if (intervalsSinceLastCheckIn >= 4) { + return 'error'; + } + if (intervalsSinceLastCheckIn >= 2) { + return 'warning'; + } + case AGENT_TYPE_TEMPORARY: + if (intervalsSinceLastCheckIn >= 3) { + return 'offline'; + } + case AGENT_TYPE_EPHEMERAL: + if (intervalsSinceLastCheckIn >= 3) { + return 'inactive'; + } + } + return 'online'; +} + +export async function getAgentStatusForConfig( + soClient: SavedObjectsClientContract, + configId?: string +) { + const [all, error, offline] = await Promise.all( + [ + undefined, + AgentStatusKueryHelper.buildKueryForErrorAgents(), + AgentStatusKueryHelper.buildKueryForOfflineAgents(), + ].map(kuery => + listAgents(soClient, { + showInactive: true, + perPage: 0, + page: 1, + kuery: configId + ? kuery + ? `(${kuery}) and (agents.config_id:"${configId}")` + : `agents.config_id:"${configId}"` + : kuery, + }) + ) + ); + + return { + events: await getEventsCount(soClient, configId), + total: all.total, + online: all.total - error.total - offline.total, + error: error.total, + offline: offline.total, + }; +} + +async function getEventsCount(soClient: SavedObjectsClientContract, configId?: string) { + const { total } = await soClient.find({ + type: AGENT_EVENT_SAVED_OBJECT_TYPE, + filter: configId ? `agent_events.attributes.config_id:"${configId}"` : undefined, + perPage: 0, + page: 1, + sortField: 'timestamp', + sortOrder: 'DESC', + defaultSearchOperator: 'AND', + }); + + return total; +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/unenroll.ts b/x-pack/plugins/ingest_manager/server/services/agents/unenroll.ts new file mode 100644 index 00000000000000..e45620c3cf5888 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/unenroll.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { AgentSOAttributes } from '../../types'; +import { AGENT_SAVED_OBJECT_TYPE } from '../../constants'; + +export async function unenrollAgents( + soClient: SavedObjectsClientContract, + toUnenrollIds: string[] +) { + const response = []; + for (const id of toUnenrollIds) { + try { + await soClient.update(AGENT_SAVED_OBJECT_TYPE, id, { + active: false, + }); + response.push({ + id, + success: true, + }); + } catch (error) { + response.push({ + id, + error, + success: false, + }); + } + } + + return response; +} diff --git a/x-pack/plugins/ingest_manager/server/services/agents/update.ts b/x-pack/plugins/ingest_manager/server/services/agents/update.ts new file mode 100644 index 00000000000000..8452c05d53a1f4 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/agents/update.ts @@ -0,0 +1,59 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { listAgents } from './crud'; +import { AGENT_SAVED_OBJECT_TYPE } from '../../constants'; +import { unenrollAgents } from './unenroll'; + +export async function updateAgentsForConfigId( + soClient: SavedObjectsClientContract, + configId: string +) { + let hasMore = true; + let page = 1; + const now = new Date().toISOString(); + while (hasMore) { + const { agents } = await listAgents(soClient, { + kuery: `agents.config_id:"${configId}"`, + page: page++, + perPage: 1000, + showInactive: true, + }); + if (agents.length === 0) { + hasMore = false; + break; + } + const agentUpdate = agents.map(agent => ({ + id: agent.id, + type: AGENT_SAVED_OBJECT_TYPE, + attributes: { config_updated_at: now }, + })); + + await soClient.bulkUpdate(agentUpdate); + } +} + +export async function unenrollForConfigId(soClient: SavedObjectsClientContract, configId: string) { + let hasMore = true; + let page = 1; + while (hasMore) { + const { agents } = await listAgents(soClient, { + kuery: `agents.config_id:"${configId}"`, + page: page++, + perPage: 1000, + showInactive: true, + }); + + if (agents.length === 0) { + hasMore = false; + } + await unenrollAgents( + soClient, + agents.map(a => a.id) + ); + } +} diff --git a/x-pack/plugins/ingest_manager/server/services/api_keys/enrollment_api_key.ts b/x-pack/plugins/ingest_manager/server/services/api_keys/enrollment_api_key.ts new file mode 100644 index 00000000000000..9a1a91f9ed8a9f --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/api_keys/enrollment_api_key.ts @@ -0,0 +1,128 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import uuid from 'uuid'; +import { SavedObjectsClientContract, SavedObject } from 'kibana/server'; +import { EnrollmentAPIKey, EnrollmentAPIKeySOAttributes } from '../../types'; +import { ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE } from '../../constants'; +import { createAPIKey, invalidateAPIKey } from './security'; +import { agentConfigService } from '../agent_config'; + +export async function listEnrollmentApiKeys( + soClient: SavedObjectsClientContract, + options: { + page?: number; + perPage?: number; + kuery?: string; + showInactive?: boolean; + } +): Promise<{ items: EnrollmentAPIKey[]; total: any; page: any; perPage: any }> { + const { page = 1, perPage = 20, kuery } = options; + + const { saved_objects, total } = await soClient.find({ + type: ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, + page, + perPage, + filter: + kuery && kuery !== '' + ? kuery.replace(/enrollment_api_keys\./g, 'enrollment_api_keys.attributes.') + : undefined, + }); + + const items = saved_objects.map(savedObjectToEnrollmentApiKey); + + return { + items, + total, + page, + perPage, + }; +} + +export async function getEnrollmentAPIKey(soClient: SavedObjectsClientContract, id: string) { + return savedObjectToEnrollmentApiKey( + await soClient.get(ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, id) + ); +} + +export async function deleteEnrollmentApiKey(soClient: SavedObjectsClientContract, id: string) { + const enrollmentApiKey = await getEnrollmentAPIKey(soClient, id); + + await invalidateAPIKey(soClient, enrollmentApiKey.api_key_id); + + await soClient.delete(ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, id); +} + +export async function deleteEnrollmentApiKeyForConfigId( + soClient: SavedObjectsClientContract, + configId: string +) { + let hasMore = true; + let page = 1; + while (hasMore) { + const { items } = await listEnrollmentApiKeys(soClient, { + page: page++, + perPage: 100, + kuery: `enrollment_api_keys.config_id:${configId}`, + }); + + if (items.length === 0) { + hasMore = false; + } + + for (const apiKey of items) { + await deleteEnrollmentApiKey(soClient, apiKey.id); + } + } +} + +export async function generateEnrollmentAPIKey( + soClient: SavedObjectsClientContract, + data: { + name?: string; + expiration?: string; + configId?: string; + } +) { + const id = uuid.v4(); + const { name: providedKeyName } = data; + const configId = data.configId ?? (await agentConfigService.getDefaultAgentConfigId(soClient)); + + const name = providedKeyName ? `${providedKeyName} (${id})` : id; + + const key = await createAPIKey(soClient, name, {}); + + if (!key) { + throw new Error('Unable to create an enrollment api key'); + } + + const apiKey = Buffer.from(`${key.id}:${key.api_key}`).toString('base64'); + + return savedObjectToEnrollmentApiKey( + await soClient.create(ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, { + active: true, + api_key_id: key.id, + api_key: apiKey, + name, + config_id: configId, + }) + ); +} + +function savedObjectToEnrollmentApiKey({ + error, + attributes, + id, +}: SavedObject): EnrollmentAPIKey { + if (error) { + throw new Error(error.message); + } + + return { + id, + ...attributes, + }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/api_keys/index.ts b/x-pack/plugins/ingest_manager/server/services/api_keys/index.ts new file mode 100644 index 00000000000000..6482c2a045a17c --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/api_keys/index.ts @@ -0,0 +1,112 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract, SavedObject, KibanaRequest } from 'kibana/server'; +import { ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE } from '../../constants'; +import { EnrollmentAPIKeySOAttributes, EnrollmentAPIKey } from '../../types'; +import { createAPIKey } from './security'; + +export * from './enrollment_api_key'; + +export async function generateOutputApiKey( + soClient: SavedObjectsClientContract, + outputId: string, + agentId: string +): Promise { + const name = `${agentId}:${outputId}`; + const key = await createAPIKey(soClient, name, { + 'fleet-output': { + cluster: ['monitor'], + index: [ + { + names: ['logs-*', 'metrics-*'], + privileges: ['write'], + }, + ], + }, + }); + + if (!key) { + throw new Error('Unable to create an output api key'); + } + + return `${key.id}:${key.api_key}`; +} + +export async function generateAccessApiKey( + soClient: SavedObjectsClientContract, + agentId: string, + configId: string +) { + const key = await createAPIKey(soClient, agentId, { + 'fleet-agent': {}, + }); + + if (!key) { + throw new Error('Unable to create an access api key'); + } + + return { id: key.id, key: Buffer.from(`${key.id}:${key.api_key}`).toString('base64') }; +} + +export async function getEnrollmentAPIKeyById( + soClient: SavedObjectsClientContract, + apiKeyId: string +) { + const [enrollmentAPIKey] = ( + await soClient.find({ + type: ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE, + searchFields: ['api_key_id'], + search: apiKeyId, + }) + ).saved_objects.map(_savedObjectToEnrollmentApiKey); + + return enrollmentAPIKey; +} + +export function parseApiKey(headers: KibanaRequest['headers']) { + const authorizationHeader = headers.authorization; + + if (!authorizationHeader) { + throw new Error('Authorization header must be set'); + } + + if (Array.isArray(authorizationHeader)) { + throw new Error('Authorization header must be `string` not `string[]`'); + } + + if (!authorizationHeader.startsWith('ApiKey ')) { + throw new Error('Authorization header is malformed'); + } + + const apiKey = authorizationHeader.split(' ')[1]; + if (!apiKey) { + throw new Error('Authorization header is malformed'); + } + const apiKeyId = Buffer.from(apiKey, 'base64') + .toString('utf8') + .split(':')[0]; + + return { + apiKey, + apiKeyId, + }; +} + +function _savedObjectToEnrollmentApiKey({ + error, + attributes, + id, +}: SavedObject): EnrollmentAPIKey { + if (error) { + throw new Error(error.message); + } + + return { + id, + ...attributes, + }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/api_keys/security.ts b/x-pack/plugins/ingest_manager/server/services/api_keys/security.ts new file mode 100644 index 00000000000000..ffc269bca94eb2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/api_keys/security.ts @@ -0,0 +1,70 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { KibanaRequest, FakeRequest, SavedObjectsClientContract } from 'kibana/server'; +import { CallESAsCurrentUser } from '../../types'; +import { appContextService } from '../app_context'; +import { outputService } from '../output'; + +export async function createAPIKey( + soClient: SavedObjectsClientContract, + name: string, + roleDescriptors: any +) { + const adminUser = await outputService.getAdminUser(soClient); + if (!adminUser) { + throw new Error('No admin user configured'); + } + const request: FakeRequest = { + headers: { + authorization: `Basic ${Buffer.from(`${adminUser.username}:${adminUser.password}`).toString( + 'base64' + )}`, + }, + }; + const security = appContextService.getSecurity(); + if (!security) { + throw new Error('Missing security plugin'); + } + + return security.authc.createAPIKey(request as KibanaRequest, { + name, + role_descriptors: roleDescriptors, + }); +} +export async function authenticate(callCluster: CallESAsCurrentUser) { + try { + await callCluster('transport.request', { + path: '/_security/_authenticate', + method: 'GET', + }); + } catch (e) { + throw new Error('ApiKey is not valid: impossible to authenticate user'); + } +} + +export async function invalidateAPIKey(soClient: SavedObjectsClientContract, id: string) { + const adminUser = await outputService.getAdminUser(soClient); + if (!adminUser) { + throw new Error('No admin user configured'); + } + const request: FakeRequest = { + headers: { + authorization: `Basic ${Buffer.from(`${adminUser.username}:${adminUser.password}`).toString( + 'base64' + )}`, + }, + }; + + const security = appContextService.getSecurity(); + if (!security) { + throw new Error('Missing security plugin'); + } + + return security.authc.invalidateAPIKey(request as KibanaRequest, { + id, + }); +} diff --git a/x-pack/plugins/ingest_manager/server/services/app_context.ts b/x-pack/plugins/ingest_manager/server/services/app_context.ts index 69a014fca37fbd..c06b282389fc76 100644 --- a/x-pack/plugins/ingest_manager/server/services/app_context.ts +++ b/x-pack/plugins/ingest_manager/server/services/app_context.ts @@ -5,6 +5,7 @@ */ import { BehaviorSubject, Observable } from 'rxjs'; import { first } from 'rxjs/operators'; +import { SavedObjectsServiceStart } from 'kibana/server'; import { EncryptedSavedObjectsPluginStart } from '../../../encrypted_saved_objects/server'; import { SecurityPluginSetup } from '../../../security/server'; import { IngestManagerConfigType } from '../../common'; @@ -15,10 +16,12 @@ class AppContextService { private security: SecurityPluginSetup | undefined; private config$?: Observable; private configSubject$?: BehaviorSubject; + private savedObjects: SavedObjectsServiceStart | undefined; public async start(appContext: IngestManagerAppContext) { this.encryptedSavedObjects = appContext.encryptedSavedObjects; this.security = appContext.security; + this.savedObjects = appContext.savedObjects; if (appContext.config$) { this.config$ = appContext.config$; @@ -45,6 +48,13 @@ class AppContextService { public getConfig$() { return this.config$; } + + public getSavedObjects() { + if (!this.savedObjects) { + throw new Error('Saved objects start service not set.'); + } + return this.savedObjects; + } } export const appContextService = new AppContextService(); diff --git a/x-pack/plugins/ingest_manager/server/services/config.ts b/x-pack/plugins/ingest_manager/server/services/config.ts new file mode 100644 index 00000000000000..9043b1cc634a0d --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/config.ts @@ -0,0 +1,37 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { Observable, Subscription } from 'rxjs'; +import { IngestManagerConfigType } from '../'; + +/** + * Kibana config observable service, *NOT* agent config + */ +class ConfigService { + private observable: Observable | null = null; + private subscription: Subscription | null = null; + private config: IngestManagerConfigType | null = null; + + private updateInformation(config: IngestManagerConfigType) { + this.config = config; + } + + public start(config$: Observable) { + this.observable = config$; + this.subscription = this.observable.subscribe(this.updateInformation.bind(this)); + } + + public stop() { + if (this.subscription) { + this.subscription.unsubscribe(); + } + } + + public getConfig() { + return this.config; + } +} + +export const configService = new ConfigService(); diff --git a/x-pack/plugins/ingest_manager/server/services/datasource.ts b/x-pack/plugins/ingest_manager/server/services/datasource.ts index b305ccaab777b6..615b29087ba1e9 100644 --- a/x-pack/plugins/ingest_manager/server/services/datasource.ts +++ b/x-pack/plugins/ingest_manager/server/services/datasource.ts @@ -4,8 +4,11 @@ * you may not use this file except in compliance with the Elastic License. */ import { SavedObjectsClientContract } from 'kibana/server'; +import { AuthenticatedUser } from '../../../security/server'; +import { DeleteDatasourcesResponse } from '../../common'; import { DATASOURCE_SAVED_OBJECT_TYPE } from '../constants'; -import { NewDatasource, Datasource, DeleteDatasourcesResponse, ListWithKuery } from '../types'; +import { NewDatasource, Datasource, ListWithKuery } from '../types'; +import { agentConfigService } from './agent_config'; const SAVED_OBJECT_TYPE = DATASOURCE_SAVED_OBJECT_TYPE; @@ -13,14 +16,22 @@ class DatasourceService { public async create( soClient: SavedObjectsClientContract, datasource: NewDatasource, - options?: { id?: string } + options?: { id?: string; user?: AuthenticatedUser } ): Promise { - const newSo = await soClient.create( + const newSo = await soClient.create>( SAVED_OBJECT_TYPE, - datasource as Datasource, + { + ...datasource, + revision: 1, + }, options ); + // Assign it to the given agent config + await agentConfigService.assignDatasources(soClient, datasource.config_id, [newSo.id], { + user: options?.user, + }); + return { id: newSo.id, ...newSo.attributes, @@ -98,20 +109,47 @@ class DatasourceService { public async update( soClient: SavedObjectsClientContract, id: string, - datasource: NewDatasource + datasource: NewDatasource, + options?: { user?: AuthenticatedUser } ): Promise { - await soClient.update(SAVED_OBJECT_TYPE, id, datasource); + const oldDatasource = await this.get(soClient, id); + + if (!oldDatasource) { + throw new Error('Datasource not found'); + } + + await soClient.update(SAVED_OBJECT_TYPE, id, { + ...datasource, + revision: oldDatasource.revision + 1, + }); + + // Bump revision of associated agent config + await agentConfigService.bumpRevision(soClient, datasource.config_id, { user: options?.user }); + return (await this.get(soClient, id)) as Datasource; } public async delete( soClient: SavedObjectsClientContract, - ids: string[] + ids: string[], + options?: { user?: AuthenticatedUser } ): Promise { const result: DeleteDatasourcesResponse = []; for (const id of ids) { try { + const oldDatasource = await this.get(soClient, id); + if (!oldDatasource) { + throw new Error('Datasource not found'); + } + await agentConfigService.unassignDatasources( + soClient, + oldDatasource.config_id, + [oldDatasource.id], + { + user: options?.user, + } + ); await soClient.delete(SAVED_OBJECT_TYPE, id); result.push({ id, diff --git a/x-pack/plugins/ingest_manager/server/services/epm/agent/agent.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/agent/agent.test.ts new file mode 100644 index 00000000000000..4f75ba03324183 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/agent/agent.test.ts @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import fs from 'fs'; +import * as yaml from 'js-yaml'; +import path from 'path'; +import { createInput } from './agent'; + +test('test converting input and manifest into template', () => { + const manifest = yaml.safeLoad( + fs.readFileSync(path.join(__dirname, 'tests/manifest.yml'), 'utf8') + ); + + const inputTemplate = fs.readFileSync(path.join(__dirname, 'tests/input.yml'), 'utf8'); + const output = createInput(manifest.vars, inputTemplate); + + // Golden file path + const generatedFile = path.join(__dirname, './tests/input.generated.yaml'); + + // Regenerate the file if `-generate` flag is used + if (process.argv.includes('-generate')) { + fs.writeFileSync(generatedFile, output); + } + + const outputData = fs.readFileSync(generatedFile, 'utf-8'); + + // Check that content file and generated file are equal + expect(outputData).toBe(output); +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/agent/agent.ts b/x-pack/plugins/ingest_manager/server/services/epm/agent/agent.ts new file mode 100644 index 00000000000000..c7dd3dab38bc1c --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/agent/agent.ts @@ -0,0 +1,23 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import Handlebars from 'handlebars'; +import { RegistryVarsEntry } from '../../../types'; + +/** + * This takes a dataset object as input and merges it with the input template. + * It returns the resolved template as a string. + */ +export function createInput(vars: RegistryVarsEntry[], inputTemplate: string): string { + const view: Record = {}; + + for (const v of vars) { + view[v.name] = v.default; + } + + const template = Handlebars.compile(inputTemplate); + return template(view); +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/input.generated.yaml b/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/input.generated.yaml new file mode 100644 index 00000000000000..451ed554ce2593 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/input.generated.yaml @@ -0,0 +1,5 @@ +type: log +paths: + - "/var/log/nginx/access.log*" + +tags: nginx diff --git a/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/input.yml b/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/input.yml new file mode 100644 index 00000000000000..65a23fc2fa9adb --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/input.yml @@ -0,0 +1,7 @@ +type: log +paths: +{{#each paths}} + - "{{this}}" +{{/each}} + +tags: {{tags}} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/manifest.yml b/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/manifest.yml new file mode 100644 index 00000000000000..46a38179fe1321 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/agent/tests/manifest.yml @@ -0,0 +1,20 @@ +title: Nginx Acess Logs +release: beta +type: logs +ingest_pipeline: default + +vars: + - name: paths + # Should we define this as array? How will the UI best make sense of it? + type: textarea + default: + - /var/log/nginx/access.log* + # I suggest to use ECS fields for this config options here: https://github.com/elastic/ecs/blob/master/schemas/os.yml + # This would need to be based on a predefined definition on what can be filtered on + os.darwin: + - /usr/local/var/log/nginx/access.log* + os.windows: + - c:/programdata/nginx/logs/*access.log* + - name: tags + default: [nginx] + type: text diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ilm/install.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ilm/install.ts new file mode 100644 index 00000000000000..c56322239f27be --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ilm/install.ts @@ -0,0 +1,48 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { CallESAsCurrentUser, ElasticsearchAssetType } from '../../../../types'; +import * as Registry from '../../registry'; + +export async function installILMPolicy(pkgkey: string, callCluster: CallESAsCurrentUser) { + const ilmPaths = await Registry.getArchiveInfo(pkgkey, (entry: Registry.ArchiveEntry) => + isILMPolicy(entry) + ); + if (!ilmPaths.length) return; + await Promise.all( + ilmPaths.map(async path => { + const body = Registry.getAsset(path).toString('utf-8'); + const { file } = Registry.pathParts(path); + const name = file.substr(0, file.lastIndexOf('.')); + try { + if (await policyExists(name, callCluster)) return; + await callCluster('transport.request', { + method: 'PUT', + path: '/_ilm/policy/' + name, + body, + }); + } catch (err) { + throw new Error(err.message); + } + }) + ); +} +const isILMPolicy = ({ path }: Registry.ArchiveEntry) => { + const pathParts = Registry.pathParts(path); + return pathParts.type === ElasticsearchAssetType.ilmPolicy; +}; +export async function policyExists( + name: string, + callCluster: CallESAsCurrentUser +): Promise { + const response = await callCluster('transport.request', { + method: 'GET', + path: '/_ilm/policy/?filter_path=' + name, + }); + + // If the response contains a key, it means the policy exists + return Object.keys(response).length > 0; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/index.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/index.test.ts new file mode 100644 index 00000000000000..851a3bc2dd7206 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/index.test.ts @@ -0,0 +1,22 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Dataset } from '../../../types'; +import { getDatasetAssetBaseName } from './index'; + +test('getBaseName', () => { + const dataset: Dataset = { + id: 'nginx.access', + title: 'Nginx Acess Logs', + release: 'beta', + type: 'logs', + ingest_pipeline: 'default', + package: 'nginx', + path: 'access', + }; + const name = getDatasetAssetBaseName(dataset); + expect(name).toStrictEqual('logs-nginx.access'); +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/index.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/index.ts new file mode 100644 index 00000000000000..e00b9db71db10b --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/index.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Dataset } from '../../../types'; + +/** + * Creates the base name for Elasticsearch assets in the form of + * {type}-{id} + */ +export function getDatasetAssetBaseName(dataset: Dataset): string { + return `${dataset.type}-${dataset.id}`; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/ingest_pipelines.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/ingest_pipelines.test.ts new file mode 100644 index 00000000000000..c3b135993105e5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/ingest_pipelines.test.ts @@ -0,0 +1,134 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { readFileSync } from 'fs'; +import path from 'path'; +import { rewriteIngestPipeline, getPipelineNameForInstallation } from './install'; +import { Dataset } from '../../../../types'; + +test('a json-format pipeline with pipeline references is correctly rewritten', () => { + const inputStandard = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/real_input_standard.json'), + 'utf-8' + ); + const inputBeats = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/real_input_beats.json'), + 'utf-8' + ); + const output = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/real_output.json') + ).toString('utf-8'); + + const substitutions = [ + { + source: 'pipeline-json', + target: 'new-pipeline-json', + templateFunction: 'IngestPipeline', + }, + { + source: 'pipeline-plaintext', + target: 'new-pipeline-plaintext', + templateFunction: 'IngestPipeline', + }, + ]; + expect(rewriteIngestPipeline(inputStandard, substitutions)).toBe(output); + expect(rewriteIngestPipeline(inputBeats, substitutions)).toBe(output); +}); + +test('a yml-format pipeline with pipeline references is correctly rewritten', () => { + const inputStandard = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/real_input_standard.yml') + ).toString('utf-8'); + const inputBeats = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/real_input_beats.yml') + ).toString('utf-8'); + const output = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/real_output.yml') + ).toString('utf-8'); + + const substitutions = [ + { + source: 'pipeline-json', + target: 'new-pipeline-json', + templateFunction: 'IngestPipeline', + }, + { + source: 'pipeline-plaintext', + target: 'new-pipeline-plaintext', + templateFunction: 'IngestPipeline', + }, + ]; + expect(rewriteIngestPipeline(inputStandard, substitutions)).toBe(output); + expect(rewriteIngestPipeline(inputBeats, substitutions)).toBe(output); +}); + +test('a json-format pipeline with no pipeline references stays unchanged', () => { + const input = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/no_replacement.json') + ).toString('utf-8'); + + const substitutions = [ + { + source: 'pipeline-json', + target: 'new-pipeline-json', + templateFunction: 'IngestPipeline', + }, + { + source: 'pipeline-plaintext', + target: 'new-pipeline-plaintext', + templateFunction: 'IngestPipeline', + }, + ]; + expect(rewriteIngestPipeline(input, substitutions)).toBe(input); +}); + +test('a yml-format pipeline with no pipeline references stays unchanged', () => { + const input = readFileSync( + path.join(__dirname, '/tests/ingest_pipelines/no_replacement.yml') + ).toString('utf-8'); + + const substitutions = [ + { + source: 'pipeline-json', + target: 'new-pipeline-json', + templateFunction: 'IngestPipeline', + }, + { + source: 'pipeline-plaintext', + target: 'new-pipeline-plaintext', + templateFunction: 'IngestPipeline', + }, + ]; + expect(rewriteIngestPipeline(input, substitutions)).toBe(input); +}); + +test('getPipelineNameForInstallation gets correct name', () => { + const dataset: Dataset = { + id: 'coredns.log', + title: 'CoreDNS logs', + release: 'ga', + type: 'logs', + ingest_pipeline: 'pipeline-entry', + package: 'coredns', + path: 'log', + }; + const packageVersion = '1.0.1'; + const pipelineRefName = 'pipeline-json'; + const pipelineEntryNameForInstallation = getPipelineNameForInstallation({ + pipelineName: dataset.ingest_pipeline, + dataset, + packageVersion, + }); + const pipelineRefNameForInstallation = getPipelineNameForInstallation({ + pipelineName: pipelineRefName, + dataset, + packageVersion, + }); + expect(pipelineEntryNameForInstallation).toBe(`${dataset.type}-${dataset.id}-${packageVersion}`); + expect(pipelineRefNameForInstallation).toBe( + `${dataset.type}-${dataset.id}-${packageVersion}-${pipelineRefName}` + ); +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/install.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/install.ts new file mode 100644 index 00000000000000..4b65e5554567e2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/install.ts @@ -0,0 +1,195 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + AssetReference, + Dataset, + ElasticsearchAssetType, + IngestAssetType, + RegistryPackage, +} from '../../../../types'; +import * as Registry from '../../registry'; +import { CallESAsCurrentUser } from '../../../../types'; + +interface RewriteSubstitution { + source: string; + target: string; + templateFunction: string; +} + +export const installPipelines = async ( + registryPackage: RegistryPackage, + callCluster: CallESAsCurrentUser +) => { + const datasets = registryPackage.datasets; + if (datasets) { + const pipelines = datasets.reduce>>((acc, dataset) => { + if (dataset.ingest_pipeline) { + acc.push( + installPipelinesForDataset({ + pkgkey: Registry.pkgToPkgKey(registryPackage), + dataset, + callCluster, + packageName: registryPackage.name, + packageVersion: registryPackage.version, + }) + ); + } + return acc; + }, []); + return Promise.all(pipelines).then(results => results.flat()); + } + return []; +}; + +export function rewriteIngestPipeline( + pipeline: string, + substitutions: RewriteSubstitution[] +): string { + substitutions.forEach(sub => { + const { source, target, templateFunction } = sub; + // This fakes the use of the golang text/template expression {{SomeTemplateFunction 'some-param'}} + // cf. https://github.com/elastic/beats/blob/master/filebeat/fileset/fileset.go#L294 + + // "Standard style" uses '{{' and '}}' as delimiters + const matchStandardStyle = `{{\\s?${templateFunction}\\s+['"]${source}['"]\\s?}}`; + // "Beats style" uses '{<' and '>}' as delimiters because this is current practice in the beats project + const matchBeatsStyle = `{<\\s?${templateFunction}\\s+['"]${source}['"]\\s?>}`; + + const regexStandardStyle = new RegExp(matchStandardStyle); + const regexBeatsStyle = new RegExp(matchBeatsStyle); + pipeline = pipeline.replace(regexStandardStyle, target).replace(regexBeatsStyle, target); + }); + return pipeline; +} + +export async function installPipelinesForDataset({ + callCluster, + pkgkey, + dataset, + packageName, + packageVersion, +}: { + callCluster: CallESAsCurrentUser; + pkgkey: string; + dataset: Dataset; + packageName: string; + packageVersion: string; +}): Promise { + const pipelinePaths = await Registry.getArchiveInfo(pkgkey, (entry: Registry.ArchiveEntry) => + isDatasetPipeline(entry, dataset.path) + ); + let pipelines: any[] = []; + const substitutions: RewriteSubstitution[] = []; + + pipelinePaths.forEach(path => { + const { name, extension } = getNameAndExtension(path); + const nameForInstallation = getPipelineNameForInstallation({ + pipelineName: name, + dataset, + packageVersion, + }); + const content = Registry.getAsset(path).toString('utf-8'); + pipelines.push({ + name, + nameForInstallation, + content, + extension, + }); + substitutions.push({ + source: name, + target: nameForInstallation, + templateFunction: 'IngestPipeline', + }); + }); + + pipelines = pipelines.map(pipeline => { + return { + ...pipeline, + contentForInstallation: rewriteIngestPipeline(pipeline.content, substitutions), + }; + }); + + const installationPromises = pipelines.map(async pipeline => { + return installPipeline({ callCluster, pipeline }); + }); + + return Promise.all(installationPromises); +} + +async function installPipeline({ + callCluster, + pipeline, +}: { + callCluster: CallESAsCurrentUser; + pipeline: any; +}): Promise { + const callClusterParams: { + method: string; + path: string; + ignore: number[]; + body: any; + headers?: any; + } = { + method: 'PUT', + path: `/_ingest/pipeline/${pipeline.nameForInstallation}`, + ignore: [404], + body: pipeline.contentForInstallation, + }; + if (pipeline.extension === 'yml') { + callClusterParams.headers = { ['Content-Type']: 'application/yaml' }; + } + + // This uses the catch-all endpoint 'transport.request' because we have to explicitly + // set the Content-Type header above for sending yml data. Setting the headers is not + // exposed in the convenience endpoint 'ingest.putPipeline' of elasticsearch-js-legacy + // which we could otherwise use. + // See src/core/server/elasticsearch/api_types.ts for available endpoints. + await callCluster('transport.request', callClusterParams); + return { id: pipeline.nameForInstallation, type: IngestAssetType.IngestPipeline }; +} + +const isDirectory = ({ path }: Registry.ArchiveEntry) => path.endsWith('/'); +const isDatasetPipeline = ({ path }: Registry.ArchiveEntry, datasetName: string) => { + // TODO: better way to get particular assets + const pathParts = Registry.pathParts(path); + return ( + !isDirectory({ path }) && + pathParts.type === ElasticsearchAssetType.ingestPipeline && + pathParts.dataset !== undefined && + datasetName === pathParts.dataset + ); +}; + +// XXX: assumes path/to/file.ext -- 0..n '/' and exactly one '.' +const getNameAndExtension = ( + path: string +): { + name: string; + extension: string; +} => { + const splitPath = path.split('/'); + const filename = splitPath[splitPath.length - 1]; + return { + name: filename.split('.')[0], + extension: filename.split('.')[1], + }; +}; + +export const getPipelineNameForInstallation = ({ + pipelineName, + dataset, + packageVersion, +}: { + pipelineName: string; + dataset: Dataset; + packageVersion: string; +}): string => { + const isPipelineEntry = pipelineName === dataset.ingest_pipeline; + const suffix = isPipelineEntry ? '' : `-${pipelineName}`; + // if this is the pipeline entry, don't add a suffix + return `${dataset.type}-${dataset.id}-${packageVersion}${suffix}`; +}; diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipeline_template.json b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipeline_template.json new file mode 100644 index 00000000000000..87fbb0c36ee84e --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipeline_template.json @@ -0,0 +1,101 @@ +{ + "description": "Pipeline for normalizing Kubernetes coredns logs", + "processors": [ + { + "pipeline": { + "if": "ctx.message.charAt(0) == (char)(\"{\")", + "name": "{{IngestPipeline 'pipeline-json' }}" + } + }, + { + "pipeline": { + "if": "ctx.message.charAt(0) != (char)(\"{\")", + "name": "{{IngestPipeline 'pipeline-plaintext' }}" + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.created = ctx['@timestamp']; ctx['@timestamp'] = ctx['timestamp']; ctx.remove('timestamp');", + "ignore_failure" : true + } + }, + { + "script": { + "lang": "painless", + "source": "ctx['source'] = new HashMap(); if (ctx.temp.source.charAt(0) == (char)(\"[\")) { def p = ctx.temp.source.indexOf (']'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(1, p); ctx.source.port = ctx.temp.source.substring(p+2, l);} else { def p = ctx.temp.source.indexOf (':'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(0, p); ctx.source.port = ctx.temp.source.substring(p+1, l);} ctx.remove('temp');", + "if": "ctx.temp?.source != null" + } + }, + { + "set": { + "field": "source.ip", + "value": "{{source.address}}", + "if": "ctx.source?.address != null" + } + }, + { + "convert" : { + "field" : "source.port", + "type": "integer" + } + }, + { + "convert" : { + "field" : "coredns.duration", + "type": "double" + } + }, + { + "convert" : { + "field" : "coredns.query.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.response.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.dnssec_ok", + "type": "boolean" + } + }, + { + "uppercase": { + "field": "coredns.response.flags" + } + }, + { + "split": { + "field": "coredns.response.flags", + "separator": "," + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.duration = Math.round(ctx.coredns.duration * params.scale)", + "params": { + "scale": 1000000000 + }, + "if": "ctx.coredns?.duration != null" + } + }, + { + "remove": { + "field": "coredns.duration", + "ignore_missing": true + } + } + ], + "on_failure" : [{ + "set" : { + "field" : "error.message", + "value" : "{{ _ingest.on_failure_message }}" + } + }] +} \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/no_replacement.json b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/no_replacement.json new file mode 100644 index 00000000000000..14d4c0f9907423 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/no_replacement.json @@ -0,0 +1,49 @@ +{ + "description": "Pipeline for dissecting message field in JSON logs", + "processors": [ + { + "json" : { + "field" : "message", + "target_field" : "json" + } + }, + { + "dissect": { + "field": "json.message", + "pattern": "%{timestamp} [%{log.level}] %{temp.source} - %{coredns.id} \"%{coredns.query.type} %{coredns.query.class} %{coredns.query.name} %{network.transport} %{coredns.query.size} %{coredns.dnssec_ok} %{bufsize}\" %{coredns.response.code} %{coredns.response.flags} %{coredns.response.size} %{coredns.duration}s" + } + }, + { + "remove": { + "field": ["message"], + "ignore_failure" : true + } + }, + { + "rename": { + "field": "json.message", + "target_field": "message", + "ignore_failure" : true + } + }, + { + "rename": { + "field": "json.kubernetes", + "target_field": "kubernetes", + "ignore_failure" : true + } + }, + { + "remove": { + "field": ["json", "bufsize"], + "ignore_failure" : true + } + } + ], + "on_failure" : [{ + "set" : { + "field" : "error.message", + "value" : "{{ _ingest.on_failure_message }}" + } + }] +} \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/no_replacement.yml b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/no_replacement.yml new file mode 100644 index 00000000000000..df3094fbfad5b9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/no_replacement.yml @@ -0,0 +1,51 @@ +description: Pipeline for Cisco IOS logs. + +processors: + # IP Geolocation Lookup + - geoip: + field: source.ip + target_field: source.geo + ignore_missing: true + - geoip: + field: destination.ip + target_field: destination.geo + ignore_missing: true + + # IP Autonomous System (AS) Lookup + - geoip: + database_file: GeoLite2-ASN.mmdb + field: source.ip + target_field: source.as + properties: + - asn + - organization_name + ignore_missing: true + - geoip: + database_file: GeoLite2-ASN.mmdb + field: destination.ip + target_field: destination.as + properties: + - asn + - organization_name + ignore_missing: true + - rename: + field: source.as.asn + target_field: source.as.number + ignore_missing: true + - rename: + field: source.as.organization_name + target_field: source.as.organization.name + ignore_missing: true + - rename: + field: destination.as.asn + target_field: destination.as.number + ignore_missing: true + - rename: + field: destination.as.organization_name + target_field: destination.as.organization.name + ignore_missing: true + +on_failure: + - set: + field: error.message + value: '{{ _ingest.on_failure_message }}' \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_beats.json b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_beats.json new file mode 100644 index 00000000000000..a1188ea08c7624 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_beats.json @@ -0,0 +1,101 @@ +{ + "description": "Pipeline for normalizing Kubernetes coredns logs", + "processors": [ + { + "pipeline": { + "if": "ctx.message.charAt(0) == (char)(\"{\")", + "name": "{}" + } + }, + { + "pipeline": { + "if": "ctx.message.charAt(0) != (char)(\"{\")", + "name": "{}" + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.created = ctx['@timestamp']; ctx['@timestamp'] = ctx['timestamp']; ctx.remove('timestamp');", + "ignore_failure" : true + } + }, + { + "script": { + "lang": "painless", + "source": "ctx['source'] = new HashMap(); if (ctx.temp.source.charAt(0) == (char)(\"[\")) { def p = ctx.temp.source.indexOf (']'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(1, p); ctx.source.port = ctx.temp.source.substring(p+2, l);} else { def p = ctx.temp.source.indexOf (':'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(0, p); ctx.source.port = ctx.temp.source.substring(p+1, l);} ctx.remove('temp');", + "if": "ctx.temp?.source != null" + } + }, + { + "set": { + "field": "source.ip", + "value": "{{source.address}}", + "if": "ctx.source?.address != null" + } + }, + { + "convert" : { + "field" : "source.port", + "type": "integer" + } + }, + { + "convert" : { + "field" : "coredns.duration", + "type": "double" + } + }, + { + "convert" : { + "field" : "coredns.query.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.response.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.dnssec_ok", + "type": "boolean" + } + }, + { + "uppercase": { + "field": "coredns.response.flags" + } + }, + { + "split": { + "field": "coredns.response.flags", + "separator": "," + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.duration = Math.round(ctx.coredns.duration * params.scale)", + "params": { + "scale": 1000000000 + }, + "if": "ctx.coredns?.duration != null" + } + }, + { + "remove": { + "field": "coredns.duration", + "ignore_missing": true + } + } + ], + "on_failure" : [{ + "set" : { + "field" : "error.message", + "value" : "{{ _ingest.on_failure_message }}" + } + }] +} \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_beats.yml b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_beats.yml new file mode 100644 index 00000000000000..4aabb8c0cf1ef9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_beats.yml @@ -0,0 +1,113 @@ +--- +description: Pipeline for normalizing Kubernetes CoreDNS logs. +processors: + - pipeline: + if: ctx.message.charAt(0) == (char)("{") + name: '{}' + - pipeline: + if: ctx.message.charAt(0) != (char)("{") + name: '{}' + - script: + lang: painless + source: > + ctx.event.created = ctx['@timestamp']; + ctx['@timestamp'] = ctx['timestamp']; + ctx.remove('timestamp'); + ignore_failure: true + - script: + lang: painless + if: ctx.temp?.source != null + source: > + ctx['source'] = new HashMap(); + if (ctx.temp.source.charAt(0) == (char)("[")) { + def p = ctx.temp.source.indexOf (']'); + def l = ctx.temp.source.length(); + ctx.source.address = ctx.temp.source.substring(1, p); + ctx.source.port = ctx.temp.source.substring(p+2, l); + } else { + def p = ctx.temp.source.indexOf(':'); + def l = ctx.temp.source.length(); + ctx.source.address = ctx.temp.source.substring(0, p); + ctx.source.port = ctx.temp.source.substring(p+1, l); + } + ctx.remove('temp'); + - set: + field: source.ip + value: "{{source.address}}" + if: ctx.source?.address != null + - convert: + field: source.port + type: integer + - convert: + field: coredns.duration + type: double + - convert: + field: coredns.query.size + type: long + - convert: + field: coredns.response.size + type: long + - convert: + field: coredns.dnssec_ok + type: boolean + - uppercase: + field: dns.header_flags + - split: + field: dns.header_flags + separator: "," + - append: + if: ctx.coredns?.dnssec_ok + field: dns.header_flags + value: DO + - script: + lang: painless + source: ctx.event.duration = Math.round(ctx.coredns.duration * params.scale); + params: + scale: 1000000000 + if: ctx.coredns?.duration != null + - remove: + field: + - coredns.duration + ignore_missing: true + # The following copies values from dns namespace (ECS) to the coredns + # namespace to avoid introducing breaking change. This should be removed + # for 8.0.0. Additionally coredns.dnssec_ok can be removed. + - set: + if: ctx.dns?.id != null + field: coredns.id + value: '{{dns.id}}' + - set: + if: ctx.dns?.question?.class != null + field: coredns.query.class + value: '{{dns.question.class}}' + - set: + if: ctx.dns?.question?.name != null + field: coredns.query.name + value: '{{dns.question.name}}' + - set: + if: ctx.dns?.question?.type != null + field: coredns.query.type + value: '{{dns.question.type}}' + - set: + if: ctx.dns?.response_code != null + field: coredns.response.code + value: '{{dns.response_code}}' + - script: + if: ctx.dns?.header_flags != null + lang: painless + source: > + ctx.coredns.response.flags = ctx.dns.header_flags; + # Right trim the trailing dot from domain names. + - script: + if: ctx.dns?.question?.name != null + lang: painless + source: > + def q = ctx.dns.question.name; + def end = q.length() - 1; + if (q.charAt(end) == (char) '.') { + ctx.dns.question.name = q.substring(0, end); + } +on_failure: + - set: + field: error.message + value: "{{ _ingest.on_failure_message }}" \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_standard.json b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_standard.json new file mode 100644 index 00000000000000..87fbb0c36ee84e --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_standard.json @@ -0,0 +1,101 @@ +{ + "description": "Pipeline for normalizing Kubernetes coredns logs", + "processors": [ + { + "pipeline": { + "if": "ctx.message.charAt(0) == (char)(\"{\")", + "name": "{{IngestPipeline 'pipeline-json' }}" + } + }, + { + "pipeline": { + "if": "ctx.message.charAt(0) != (char)(\"{\")", + "name": "{{IngestPipeline 'pipeline-plaintext' }}" + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.created = ctx['@timestamp']; ctx['@timestamp'] = ctx['timestamp']; ctx.remove('timestamp');", + "ignore_failure" : true + } + }, + { + "script": { + "lang": "painless", + "source": "ctx['source'] = new HashMap(); if (ctx.temp.source.charAt(0) == (char)(\"[\")) { def p = ctx.temp.source.indexOf (']'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(1, p); ctx.source.port = ctx.temp.source.substring(p+2, l);} else { def p = ctx.temp.source.indexOf (':'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(0, p); ctx.source.port = ctx.temp.source.substring(p+1, l);} ctx.remove('temp');", + "if": "ctx.temp?.source != null" + } + }, + { + "set": { + "field": "source.ip", + "value": "{{source.address}}", + "if": "ctx.source?.address != null" + } + }, + { + "convert" : { + "field" : "source.port", + "type": "integer" + } + }, + { + "convert" : { + "field" : "coredns.duration", + "type": "double" + } + }, + { + "convert" : { + "field" : "coredns.query.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.response.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.dnssec_ok", + "type": "boolean" + } + }, + { + "uppercase": { + "field": "coredns.response.flags" + } + }, + { + "split": { + "field": "coredns.response.flags", + "separator": "," + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.duration = Math.round(ctx.coredns.duration * params.scale)", + "params": { + "scale": 1000000000 + }, + "if": "ctx.coredns?.duration != null" + } + }, + { + "remove": { + "field": "coredns.duration", + "ignore_missing": true + } + } + ], + "on_failure" : [{ + "set" : { + "field" : "error.message", + "value" : "{{ _ingest.on_failure_message }}" + } + }] +} \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_standard.yml b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_standard.yml new file mode 100644 index 00000000000000..f5e3491fedbcd4 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_input_standard.yml @@ -0,0 +1,113 @@ +--- +description: Pipeline for normalizing Kubernetes CoreDNS logs. +processors: + - pipeline: + if: ctx.message.charAt(0) == (char)("{") + name: '{{IngestPipeline "pipeline-json" }}' + - pipeline: + if: ctx.message.charAt(0) != (char)("{") + name: '{{IngestPipeline "pipeline-plaintext" }}' + - script: + lang: painless + source: > + ctx.event.created = ctx['@timestamp']; + ctx['@timestamp'] = ctx['timestamp']; + ctx.remove('timestamp'); + ignore_failure: true + - script: + lang: painless + if: ctx.temp?.source != null + source: > + ctx['source'] = new HashMap(); + if (ctx.temp.source.charAt(0) == (char)("[")) { + def p = ctx.temp.source.indexOf (']'); + def l = ctx.temp.source.length(); + ctx.source.address = ctx.temp.source.substring(1, p); + ctx.source.port = ctx.temp.source.substring(p+2, l); + } else { + def p = ctx.temp.source.indexOf(':'); + def l = ctx.temp.source.length(); + ctx.source.address = ctx.temp.source.substring(0, p); + ctx.source.port = ctx.temp.source.substring(p+1, l); + } + ctx.remove('temp'); + - set: + field: source.ip + value: "{{source.address}}" + if: ctx.source?.address != null + - convert: + field: source.port + type: integer + - convert: + field: coredns.duration + type: double + - convert: + field: coredns.query.size + type: long + - convert: + field: coredns.response.size + type: long + - convert: + field: coredns.dnssec_ok + type: boolean + - uppercase: + field: dns.header_flags + - split: + field: dns.header_flags + separator: "," + - append: + if: ctx.coredns?.dnssec_ok + field: dns.header_flags + value: DO + - script: + lang: painless + source: ctx.event.duration = Math.round(ctx.coredns.duration * params.scale); + params: + scale: 1000000000 + if: ctx.coredns?.duration != null + - remove: + field: + - coredns.duration + ignore_missing: true + # The following copies values from dns namespace (ECS) to the coredns + # namespace to avoid introducing breaking change. This should be removed + # for 8.0.0. Additionally coredns.dnssec_ok can be removed. + - set: + if: ctx.dns?.id != null + field: coredns.id + value: '{{dns.id}}' + - set: + if: ctx.dns?.question?.class != null + field: coredns.query.class + value: '{{dns.question.class}}' + - set: + if: ctx.dns?.question?.name != null + field: coredns.query.name + value: '{{dns.question.name}}' + - set: + if: ctx.dns?.question?.type != null + field: coredns.query.type + value: '{{dns.question.type}}' + - set: + if: ctx.dns?.response_code != null + field: coredns.response.code + value: '{{dns.response_code}}' + - script: + if: ctx.dns?.header_flags != null + lang: painless + source: > + ctx.coredns.response.flags = ctx.dns.header_flags; + # Right trim the trailing dot from domain names. + - script: + if: ctx.dns?.question?.name != null + lang: painless + source: > + def q = ctx.dns.question.name; + def end = q.length() - 1; + if (q.charAt(end) == (char) '.') { + ctx.dns.question.name = q.substring(0, end); + } +on_failure: + - set: + field: error.message + value: "{{ _ingest.on_failure_message }}" \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_output.json b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_output.json new file mode 100644 index 00000000000000..91b54fdf664a90 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_output.json @@ -0,0 +1,101 @@ +{ + "description": "Pipeline for normalizing Kubernetes coredns logs", + "processors": [ + { + "pipeline": { + "if": "ctx.message.charAt(0) == (char)(\"{\")", + "name": "new-pipeline-json" + } + }, + { + "pipeline": { + "if": "ctx.message.charAt(0) != (char)(\"{\")", + "name": "new-pipeline-plaintext" + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.created = ctx['@timestamp']; ctx['@timestamp'] = ctx['timestamp']; ctx.remove('timestamp');", + "ignore_failure" : true + } + }, + { + "script": { + "lang": "painless", + "source": "ctx['source'] = new HashMap(); if (ctx.temp.source.charAt(0) == (char)(\"[\")) { def p = ctx.temp.source.indexOf (']'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(1, p); ctx.source.port = ctx.temp.source.substring(p+2, l);} else { def p = ctx.temp.source.indexOf (':'); def l = ctx.temp.source.length(); ctx.source.address = ctx.temp.source.substring(0, p); ctx.source.port = ctx.temp.source.substring(p+1, l);} ctx.remove('temp');", + "if": "ctx.temp?.source != null" + } + }, + { + "set": { + "field": "source.ip", + "value": "{{source.address}}", + "if": "ctx.source?.address != null" + } + }, + { + "convert" : { + "field" : "source.port", + "type": "integer" + } + }, + { + "convert" : { + "field" : "coredns.duration", + "type": "double" + } + }, + { + "convert" : { + "field" : "coredns.query.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.response.size", + "type": "long" + } + }, + { + "convert" : { + "field" : "coredns.dnssec_ok", + "type": "boolean" + } + }, + { + "uppercase": { + "field": "coredns.response.flags" + } + }, + { + "split": { + "field": "coredns.response.flags", + "separator": "," + } + }, + { + "script": { + "lang": "painless", + "source": "ctx.event.duration = Math.round(ctx.coredns.duration * params.scale)", + "params": { + "scale": 1000000000 + }, + "if": "ctx.coredns?.duration != null" + } + }, + { + "remove": { + "field": "coredns.duration", + "ignore_missing": true + } + } + ], + "on_failure" : [{ + "set" : { + "field" : "error.message", + "value" : "{{ _ingest.on_failure_message }}" + } + }] +} \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_output.yml b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_output.yml new file mode 100644 index 00000000000000..0e5b588f03b0de --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/ingest_pipeline/tests/ingest_pipelines/real_output.yml @@ -0,0 +1,113 @@ +--- +description: Pipeline for normalizing Kubernetes CoreDNS logs. +processors: + - pipeline: + if: ctx.message.charAt(0) == (char)("{") + name: 'new-pipeline-json' + - pipeline: + if: ctx.message.charAt(0) != (char)("{") + name: 'new-pipeline-plaintext' + - script: + lang: painless + source: > + ctx.event.created = ctx['@timestamp']; + ctx['@timestamp'] = ctx['timestamp']; + ctx.remove('timestamp'); + ignore_failure: true + - script: + lang: painless + if: ctx.temp?.source != null + source: > + ctx['source'] = new HashMap(); + if (ctx.temp.source.charAt(0) == (char)("[")) { + def p = ctx.temp.source.indexOf (']'); + def l = ctx.temp.source.length(); + ctx.source.address = ctx.temp.source.substring(1, p); + ctx.source.port = ctx.temp.source.substring(p+2, l); + } else { + def p = ctx.temp.source.indexOf(':'); + def l = ctx.temp.source.length(); + ctx.source.address = ctx.temp.source.substring(0, p); + ctx.source.port = ctx.temp.source.substring(p+1, l); + } + ctx.remove('temp'); + - set: + field: source.ip + value: "{{source.address}}" + if: ctx.source?.address != null + - convert: + field: source.port + type: integer + - convert: + field: coredns.duration + type: double + - convert: + field: coredns.query.size + type: long + - convert: + field: coredns.response.size + type: long + - convert: + field: coredns.dnssec_ok + type: boolean + - uppercase: + field: dns.header_flags + - split: + field: dns.header_flags + separator: "," + - append: + if: ctx.coredns?.dnssec_ok + field: dns.header_flags + value: DO + - script: + lang: painless + source: ctx.event.duration = Math.round(ctx.coredns.duration * params.scale); + params: + scale: 1000000000 + if: ctx.coredns?.duration != null + - remove: + field: + - coredns.duration + ignore_missing: true + # The following copies values from dns namespace (ECS) to the coredns + # namespace to avoid introducing breaking change. This should be removed + # for 8.0.0. Additionally coredns.dnssec_ok can be removed. + - set: + if: ctx.dns?.id != null + field: coredns.id + value: '{{dns.id}}' + - set: + if: ctx.dns?.question?.class != null + field: coredns.query.class + value: '{{dns.question.class}}' + - set: + if: ctx.dns?.question?.name != null + field: coredns.query.name + value: '{{dns.question.name}}' + - set: + if: ctx.dns?.question?.type != null + field: coredns.query.type + value: '{{dns.question.type}}' + - set: + if: ctx.dns?.response_code != null + field: coredns.response.code + value: '{{dns.response_code}}' + - script: + if: ctx.dns?.header_flags != null + lang: painless + source: > + ctx.coredns.response.flags = ctx.dns.header_flags; + # Right trim the trailing dot from domain names. + - script: + if: ctx.dns?.question?.name != null + lang: painless + source: > + def q = ctx.dns.question.name; + def end = q.length() - 1; + if (q.charAt(end) == (char) '.') { + ctx.dns.question.name = q.substring(0, end); + } +on_failure: + - set: + field: error.message + value: "{{ _ingest.on_failure_message }}" \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/__snapshots__/template.test.ts.snap b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/__snapshots__/template.test.ts.snap new file mode 100644 index 00000000000000..ad4d636164d71e --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/__snapshots__/template.test.ts.snap @@ -0,0 +1,79 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`tests loading fields.yml: base.yml 1`] = ` +{ + "order": 1, + "index_patterns": [ + "foo-*" + ], + "settings": { + "index": { + "lifecycle": { + "name": "logs-default" + }, + "codec": "best_compression", + "mapping": { + "total_fields": { + "limit": "10000" + } + }, + "refresh_interval": "5s", + "number_of_shards": "1", + "query": { + "default_field": [ + "message" + ] + }, + "number_of_routing_shards": "30" + } + }, + "mappings": { + "_meta": { + "package": "foo" + }, + "dynamic_templates": [ + { + "strings_as_keyword": { + "mapping": { + "ignore_above": 1024, + "type": "keyword" + }, + "match_mapping_type": "string" + } + } + ], + "date_detection": false, + "properties": { + "user": { + "properties": { + "auid": { + "type": "keyword" + }, + "euid": { + "type": "keyword" + } + } + }, + "long": { + "properties": { + "nested": { + "properties": { + "foo": { + "type": "keyword" + } + } + } + } + }, + "nested": { + "properties": { + "bar": { + "type": "keyword" + } + } + } + } + }, + "aliases": {} +} +`; diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/install.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/install.ts new file mode 100644 index 00000000000000..005bb78e458e30 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/install.ts @@ -0,0 +1,120 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { + AssetReference, + Dataset, + RegistryPackage, + IngestAssetType, + ElasticsearchAssetType, +} from '../../../../types'; +import { CallESAsCurrentUser } from '../../../../types'; +import { Field, loadFieldsFromYaml } from '../../fields/field'; +import { getPipelineNameForInstallation } from '../ingest_pipeline/install'; +import { generateMappings, generateTemplateName, getTemplate } from './template'; +import * as Registry from '../../registry'; + +export const installTemplates = async ( + registryPackage: RegistryPackage, + callCluster: CallESAsCurrentUser, + pkgkey: string +) => { + // install any pre-built index template assets, + // atm, this is only the base package's global template + installPreBuiltTemplates(pkgkey, callCluster); + + // build templates per dataset from yml files + const datasets = registryPackage.datasets; + if (datasets) { + const templates = datasets.reduce>>((acc, dataset) => { + acc.push( + installTemplateForDataset({ + pkg: registryPackage, + callCluster, + dataset, + }) + ); + return acc; + }, []); + return Promise.all(templates).then(results => results.flat()); + } + return []; +}; + +// this is temporary until we update the registry to use index templates v2 structure +const installPreBuiltTemplates = async (pkgkey: string, callCluster: CallESAsCurrentUser) => { + const templatePaths = await Registry.getArchiveInfo(pkgkey, (entry: Registry.ArchiveEntry) => + isTemplate(entry) + ); + templatePaths.forEach(async path => { + const { file } = Registry.pathParts(path); + const templateName = file.substr(0, file.lastIndexOf('.')); + const content = JSON.parse(Registry.getAsset(path).toString('utf8')); + await callCluster('indices.putTemplate', { + name: templateName, + body: content, + }); + }); +}; +const isTemplate = ({ path }: Registry.ArchiveEntry) => { + const pathParts = Registry.pathParts(path); + return pathParts.type === ElasticsearchAssetType.indexTemplate; +}; +/** + * installTemplatesForDataset installs one template for each dataset + * + * The template is currently loaded with the pkgey-package-dataset + */ + +export async function installTemplateForDataset({ + pkg, + callCluster, + dataset, +}: { + pkg: RegistryPackage; + callCluster: CallESAsCurrentUser; + dataset: Dataset; +}): Promise { + const fields = await loadFieldsFromYaml(pkg, dataset.path); + return installTemplate({ + callCluster, + fields, + dataset, + packageVersion: pkg.version, + }); +} + +export async function installTemplate({ + callCluster, + fields, + dataset, + packageVersion, +}: { + callCluster: CallESAsCurrentUser; + fields: Field[]; + dataset: Dataset; + packageVersion: string; +}): Promise { + const mappings = generateMappings(fields); + const templateName = generateTemplateName(dataset); + let pipelineName; + if (dataset.ingest_pipeline) { + pipelineName = getPipelineNameForInstallation({ + pipelineName: dataset.ingest_pipeline, + dataset, + packageVersion, + }); + } + const template = getTemplate(dataset.type, templateName, mappings, pipelineName); + // TODO: Check return values for errors + await callCluster('indices.putTemplate', { + name: templateName, + body: template, + }); + + // The id of a template is its name + return { id: templateName, type: IngestAssetType.IndexTemplate }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.test.ts new file mode 100644 index 00000000000000..aa5be59b6a5cde --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.test.ts @@ -0,0 +1,42 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { readFileSync } from 'fs'; +import { safeLoad } from 'js-yaml'; +import path from 'path'; +import { Field, processFields } from '../../fields/field'; +import { generateMappings, getTemplate } from './template'; + +// Add our own serialiser to just do JSON.stringify +expect.addSnapshotSerializer({ + print(val) { + return JSON.stringify(val, null, 2); + }, + + test(val) { + return val; + }, +}); + +test('get template', () => { + const templateName = 'logs-nginx-access-abcd'; + + const template = getTemplate('logs', templateName, { properties: {} }); + expect(template.index_patterns).toStrictEqual([`${templateName}-*`]); +}); + +test('tests loading fields.yml', () => { + // Load fields.yml file + const ymlPath = path.join(__dirname, '../../fields/tests/base.yml'); + const fieldsYML = readFileSync(ymlPath, 'utf-8'); + const fields: Field[] = safeLoad(fieldsYML); + + processFields(fields); + const mappings = generateMappings(fields); + const template = getTemplate('logs', 'foo', mappings); + + expect(template).toMatchSnapshot(path.basename(ymlPath)); +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts new file mode 100644 index 00000000000000..f075771e9808a3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts @@ -0,0 +1,128 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Field } from '../../fields/field'; +import { Dataset, IndexTemplate } from '../../../../types'; +import { getDatasetAssetBaseName } from '../index'; + +interface Properties { + [key: string]: any; +} +interface Mappings { + properties: any; +} +/** + * getTemplate retrieves the default template but overwrites the index pattern with the given value. + * + * @param indexPattern String with the index pattern + */ +export function getTemplate( + type: string, + templateName: string, + mappings: Mappings, + pipelineName?: string | undefined +): IndexTemplate { + const template = getBaseTemplate(type, templateName, mappings); + if (pipelineName) { + template.settings.index.default_pipeline = pipelineName; + } + return template; +} + +/** + * Generate mapping takes the given fields array and creates the Elasticsearch + * mapping properties out of it. + * + * @param fields + */ +export function generateMappings(fields: Field[]): Mappings { + const props: Properties = {}; + fields.forEach(field => { + // Are there more fields inside this field? Build them recursively + if (field.fields && field.fields.length > 0) { + props[field.name] = generateMappings(field.fields); + return; + } + + // If not type is defined, take keyword + const type = field.type || 'keyword'; + // Only add keyword fields for now + // TODO: add support for other field types + if (type === 'keyword') { + props[field.name] = { type }; + } + }); + return { properties: props }; +} + +/** + * Generates the template name out of the given information + */ +export function generateTemplateName(dataset: Dataset): string { + return getDatasetAssetBaseName(dataset); +} + +function getBaseTemplate(type: string, templateName: string, mappings: Mappings): IndexTemplate { + return { + // We need to decide which order we use for the templates + order: 1, + // To be completed with the correct index patterns + index_patterns: [`${templateName}-*`], + settings: { + index: { + // ILM Policy must be added here, for now point to the default global ILM policy name + lifecycle: { + name: `${type}-default`, + }, + // What should be our default for the compression? + codec: 'best_compression', + // W + mapping: { + total_fields: { + limit: '10000', + }, + }, + // This is the default from Beats? So far seems to be a good value + refresh_interval: '5s', + // Default in the stack now, still good to have it in + number_of_shards: '1', + // All the default fields which should be queried have to be added here. + // So far we add all keyword and text fields here. + query: { + default_field: ['message'], + }, + // We are setting 30 because it can be devided by several numbers. Useful when shrinking. + number_of_routing_shards: '30', + }, + }, + mappings: { + // To be filled with interesting information about this specific index + _meta: { + package: 'foo', + }, + // All the dynamic field mappings + dynamic_templates: [ + // This makes sure all mappings are keywords by default + { + strings_as_keyword: { + mapping: { + ignore_above: 1024, + type: 'keyword', + }, + match_mapping_type: 'string', + }, + }, + ], + // As we define fields ahead, we don't need any automatic field detection + // This makes sure all the fields are mapped to keyword by default to prevent mapping conflicts + date_detection: false, + // All the properties we know from the fields.yml file + properties: mappings.properties, + }, + // To be filled with the aliases that we need + aliases: {}, + }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/fields/__snapshots__/field.test.ts.snap b/x-pack/plugins/ingest_manager/server/services/epm/fields/__snapshots__/field.test.ts.snap new file mode 100644 index 00000000000000..76991bde77008b --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/fields/__snapshots__/field.test.ts.snap @@ -0,0 +1,101 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`tests loading fields.yml: base.yml 1`] = ` +[ + { + "name": "user", + "type": "group", + "fields": [ + { + "name": "auid" + }, + { + "name": "euid" + } + ] + }, + { + "name": "long", + "type": "group", + "fields": [ + { + "name": "nested", + "type": "group", + "fields": [ + { + "name": "foo" + } + ] + } + ] + }, + { + "name": "nested", + "type": "group", + "fields": [ + { + "name": "bar" + } + ] + } +] +`; + +exports[`tests loading fields.yml: coredns.logs.yml 1`] = ` +[ + { + "name": "coredns", + "type": "group", + "description": "coredns fields after normalization\\n", + "fields": [ + { + "name": "id", + "type": "keyword", + "description": "id of the DNS transaction\\n" + }, + { + "name": "query.size", + "type": "integer", + "format": "bytes", + "description": "size of the DNS query\\n" + }, + { + "name": "query.class", + "type": "keyword", + "description": "DNS query class\\n" + }, + { + "name": "query.name", + "type": "keyword", + "description": "DNS query name\\n" + }, + { + "name": "query.type", + "type": "keyword", + "description": "DNS query type\\n" + }, + { + "name": "response.code", + "type": "keyword", + "description": "DNS response code\\n" + }, + { + "name": "response.flags", + "type": "keyword", + "description": "DNS response flags\\n" + }, + { + "name": "response.size", + "type": "integer", + "format": "bytes", + "description": "size of the DNS response\\n" + }, + { + "name": "dnssec_ok", + "type": "boolean", + "description": "dnssec flag\\n" + } + ] + } +] +`; diff --git a/x-pack/plugins/ingest_manager/server/services/epm/fields/field.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/fields/field.test.ts new file mode 100644 index 00000000000000..3cdf011d9d0e32 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/fields/field.test.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { readFileSync } from 'fs'; +import glob from 'glob'; +import { safeLoad } from 'js-yaml'; +import path from 'path'; +import { Field, processFields } from './field'; + +// Add our own serialiser to just do JSON.stringify +expect.addSnapshotSerializer({ + print(val) { + return JSON.stringify(val, null, 2); + }, + + test(val) { + return val; + }, +}); + +test('tests loading fields.yml', () => { + // Find all .yml files to run tests on + const files = glob.sync(path.join(__dirname, '/tests/*.yml')); + for (const file of files) { + const fieldsYML = readFileSync(file, 'utf-8'); + const fields: Field[] = safeLoad(fieldsYML); + processFields(fields); + + // Check that content file and generated file are equal + expect(fields).toMatchSnapshot(path.basename(file)); + } +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/fields/field.ts b/x-pack/plugins/ingest_manager/server/services/epm/fields/field.ts new file mode 100644 index 00000000000000..eb515f5652f36e --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/fields/field.ts @@ -0,0 +1,113 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { safeLoad } from 'js-yaml'; +import { RegistryPackage } from '../../../types'; +import { getAssetsData } from '../packages/assets'; + +// This should become a copy of https://github.com/elastic/beats/blob/d9a4c9c240a9820fab15002592e5bb6db318543b/libbeat/mapping/field.go#L39 +export interface Field { + name: string; + type?: string; + description?: string; + format?: string; + fields?: Fields; + enabled?: boolean; + path?: string; + index?: boolean; + required?: boolean; + multi_fields?: Fields; + doc_values?: boolean; + + // Kibana specific + analyzed?: boolean; + count?: number; + searchable?: boolean; + aggregatable?: boolean; + script?: string; + readFromDocValues?: boolean; + + // Kibana field format params + pattern?: string; + input_format?: string; + output_format?: string; + output_precision?: number; + label_template?: string; + url_template?: string; + open_link_in_current_tab?: boolean; +} + +export type Fields = Field[]; + +/** + * ProcessFields takes the given fields read from yaml and expands it. + * There are dotted fields in the field.yml like `foo.bar`. These should + * be stored as an object inside an object and is the main purpose of this + * preprocessing. + * + * Note: This function modifies the passed field param. + */ +export function processFields(fields: Fields) { + fields.forEach((field, key) => { + const fieldName = field.name; + + // If the field name contains a dot, it means we need to create sub objects + if (fieldName.includes('.')) { + // Split up the name by dots to extract first and other parts + const nameParts = fieldName.split('.'); + + // Getting first part of the name for the new field + const newNameTop = nameParts[0]; + delete nameParts[0]; + + // Put back together the parts again for the new field name + const newName = nameParts.length === 1 ? nameParts[0] : nameParts.slice(1).join('.'); + + field.name = newName; + + // Create the new field with the old field inside + const newField: Field = { + name: newNameTop, + type: 'group', + fields: [field], + }; + // Replace the old field in the array + fields[key] = newField; + if (newField.fields) { + processFields(newField.fields); + } + } + }); +} + +const isFields = (path: string) => { + return path.includes('/fields/'); +}; + +/** + * loadFieldsFromYaml + * + * Gets all field files, optionally filtered by dataset, extracts .yml files, merges them together + */ + +export const loadFieldsFromYaml = async ( + pkg: RegistryPackage, + datasetName?: string +): Promise => { + // Fetch all field definition files + const fieldDefinitionFiles = await getAssetsData(pkg, isFields, datasetName); + return fieldDefinitionFiles.reduce((acc, file) => { + // Make sure it is defined as it is optional. Should never happen. + if (file.buffer) { + const tmpFields = safeLoad(file.buffer.toString()); + // safeLoad() returns undefined for empty files, we don't want that + if (tmpFields) { + acc = acc.concat(tmpFields); + } + } + return acc; + }, []); +}; diff --git a/x-pack/plugins/ingest_manager/server/services/epm/fields/tests/base.yml b/x-pack/plugins/ingest_manager/server/services/epm/fields/tests/base.yml new file mode 100644 index 00000000000000..86b61245aa3b80 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/fields/tests/base.yml @@ -0,0 +1,7 @@ +- name: user + type: group + fields: + - name: auid + - name: euid +- name: long.nested.foo +- name: nested.bar diff --git a/x-pack/plugins/ingest_manager/server/services/epm/fields/tests/coredns.logs.yml b/x-pack/plugins/ingest_manager/server/services/epm/fields/tests/coredns.logs.yml new file mode 100644 index 00000000000000..439849742a8515 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/fields/tests/coredns.logs.yml @@ -0,0 +1,51 @@ +- name: coredns + type: group + description: > + coredns fields after normalization + fields: + - name: id + type: keyword + description: > + id of the DNS transaction + + - name: query.size + type: integer + format: bytes + description: > + size of the DNS query + + - name: query.class + type: keyword + description: > + DNS query class + + - name: query.name + type: keyword + description: > + DNS query name + + - name: query.type + type: keyword + description: > + DNS query type + + - name: response.code + type: keyword + description: > + DNS response code + + - name: response.flags + type: keyword + description: > + DNS response flags + + - name: response.size + type: integer + format: bytes + description: > + size of the DNS response + + - name: dnssec_ok + type: boolean + description: > + dnssec flag \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/__snapshots__/install.test.ts.snap b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/__snapshots__/install.test.ts.snap new file mode 100644 index 00000000000000..79f778f9bba8f7 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/__snapshots__/install.test.ts.snap @@ -0,0 +1,1326 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`creating index patterns from yaml fields createFieldFormatMap creates correct map based on inputs all variations and all the params get passed through: createFieldFormatMap 1`] = ` +{ + "fieldPattern": { + "params": { + "pattern": "patternVal" + } + }, + "fieldFormat": { + "id": "formatVal" + }, + "fieldFormatWithParam": { + "id": "formatVal", + "params": { + "outputPrecision": 2 + } + }, + "fieldFormatAndPattern": { + "id": "formatVal", + "params": { + "pattern": "patternVal" + } + }, + "fieldFormatAndAllParams": { + "id": "formatVal", + "params": { + "pattern": "pattenVal", + "inputFormat": "inputFormatVal", + "outputFormat": "outputFormalVal", + "outputPrecision": 3, + "labelTemplate": "labelTemplateVal", + "urlTemplate": "urlTemplateVal" + } + } +} +`; + +exports[`creating index patterns from yaml fields createIndexPattern function creates Kibana index pattern: createIndexPattern 1`] = ` +{ + "title": "logs-*", + "timeFieldName": "@timestamp", + "fields": "[{\\"name\\":\\"coredns.id\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"coredns.allParams\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"number\\"},{\\"name\\":\\"coredns.query.length\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"number\\"},{\\"name\\":\\"coredns.query.size\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"number\\"},{\\"name\\":\\"coredns.query.class\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"coredns.query.name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"coredns.query.type\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"coredns.response.code\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"coredns.response.flags\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"coredns.response.size\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"number\\"},{\\"name\\":\\"coredns.dnssec_ok\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"boolean\\"},{\\"name\\":\\"@timestamp\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"date\\"},{\\"name\\":\\"labels\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"message\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":false,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"tags\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"agent.ephemeral_id\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"agent.id\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"agent.name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"agent.type\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"agent.version\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"as.number\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"number\\"},{\\"name\\":\\"as.organization.name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"nginx.access.remote_ip_list\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.body_sent.bytes\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.user_name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.method\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.url\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.http_version\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.response_code\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.referrer\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.agent\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.user_agent.device\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.user_agent.name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.user_agent.os\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.user_agent.os_name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.user_agent.original\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.geoip.continent_name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":false,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"nginx.access.geoip.country_iso_code\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.geoip.location\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.geoip.region_name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.geoip.city_name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"nginx.access.geoip.region_iso_code\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true},{\\"name\\":\\"source.geo.continent_name\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":false,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"country\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"country.keyword\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":true,\\"doc_values\\":true,\\"type\\":\\"string\\"},{\\"name\\":\\"country.text\\",\\"count\\":0,\\"scripted\\":false,\\"indexed\\":true,\\"analyzed\\":false,\\"searchable\\":true,\\"aggregatable\\":false,\\"doc_values\\":true,\\"type\\":\\"string\\"}]", + "fieldFormatMap": "{\\"coredns.allParams\\":{\\"id\\":\\"bytes\\",\\"params\\":{\\"pattern\\":\\"patternValQueryWeight\\",\\"inputFormat\\":\\"inputFormatVal,\\",\\"outputFormat\\":\\"outputFormalVal,\\",\\"outputPrecision\\":\\"3,\\",\\"labelTemplate\\":\\"labelTemplateVal,\\",\\"urlTemplate\\":\\"urlTemplateVal,\\"}},\\"coredns.query.length\\":{\\"params\\":{\\"pattern\\":\\"patternValQueryLength\\"}},\\"coredns.query.size\\":{\\"id\\":\\"bytes\\",\\"params\\":{\\"pattern\\":\\"patternValQuerySize\\"}},\\"coredns.response.size\\":{\\"id\\":\\"bytes\\"}}" +} +`; + +exports[`creating index patterns from yaml fields createIndexPatternFields function creates Kibana index pattern fields and fieldFormatMap: createIndexPatternFields 1`] = ` +{ + "indexPatternFields": [ + { + "name": "coredns.id", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "coredns.allParams", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "number" + }, + { + "name": "coredns.query.length", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "number" + }, + { + "name": "coredns.query.size", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "number" + }, + { + "name": "coredns.query.class", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "coredns.query.name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "coredns.query.type", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "coredns.response.code", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "coredns.response.flags", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "coredns.response.size", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "number" + }, + { + "name": "coredns.dnssec_ok", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "boolean" + }, + { + "name": "@timestamp", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "date" + }, + { + "name": "labels", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "message", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": false, + "doc_values": true, + "type": "string" + }, + { + "name": "tags", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "agent.ephemeral_id", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "agent.id", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "agent.name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "agent.type", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "agent.version", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "as.number", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "number" + }, + { + "name": "as.organization.name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "nginx.access.remote_ip_list", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.body_sent.bytes", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.user_name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.method", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.url", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.http_version", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.response_code", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.referrer", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.agent", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.user_agent.device", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.user_agent.name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.user_agent.os", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.user_agent.os_name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.user_agent.original", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.geoip.continent_name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": false, + "doc_values": true, + "type": "string" + }, + { + "name": "nginx.access.geoip.country_iso_code", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.geoip.location", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.geoip.region_name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.geoip.city_name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "nginx.access.geoip.region_iso_code", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true + }, + { + "name": "source.geo.continent_name", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": false, + "doc_values": true, + "type": "string" + }, + { + "name": "country", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "country.keyword", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": true, + "doc_values": true, + "type": "string" + }, + { + "name": "country.text", + "count": 0, + "scripted": false, + "indexed": true, + "analyzed": false, + "searchable": true, + "aggregatable": false, + "doc_values": true, + "type": "string" + } + ], + "fieldFormatMap": { + "coredns.allParams": { + "id": "bytes", + "params": { + "pattern": "patternValQueryWeight", + "inputFormat": "inputFormatVal,", + "outputFormat": "outputFormalVal,", + "outputPrecision": "3,", + "labelTemplate": "labelTemplateVal,", + "urlTemplate": "urlTemplateVal," + } + }, + "coredns.query.length": { + "params": { + "pattern": "patternValQueryLength" + } + }, + "coredns.query.size": { + "id": "bytes", + "params": { + "pattern": "patternValQuerySize" + } + }, + "coredns.response.size": { + "id": "bytes" + } + } +} +`; + +exports[`creating index patterns from yaml fields dedupFields function remove duplicated fields when parsing multiple files: dedupeFields 1`] = ` +[ + { + "name": "coredns", + "type": "group", + "description": "coredns fields after normalization\\n", + "fields": [ + { + "name": "id", + "type": "keyword", + "description": "id of the DNS transaction\\n" + }, + { + "name": "allParams", + "type": "integer", + "format": "bytes", + "pattern": "patternValQueryWeight", + "input_format": "inputFormatVal,", + "output_format": "outputFormalVal,", + "output_precision": "3,", + "label_template": "labelTemplateVal,", + "url_template": "urlTemplateVal,", + "openLinkInCurrentTab": "true,", + "description": "weight of the DNS query\\n" + }, + { + "name": "query.length", + "type": "integer", + "pattern": "patternValQueryLength", + "description": "length of the DNS query\\n" + }, + { + "name": "query.size", + "type": "integer", + "format": "bytes", + "pattern": "patternValQuerySize", + "description": "size of the DNS query\\n" + }, + { + "name": "query.class", + "type": "keyword", + "description": "DNS query class\\n" + }, + { + "name": "query.name", + "type": "keyword", + "description": "DNS query name\\n" + }, + { + "name": "query.type", + "type": "keyword", + "description": "DNS query type\\n" + }, + { + "name": "response.code", + "type": "keyword", + "description": "DNS response code\\n" + }, + { + "name": "response.flags", + "type": "keyword", + "description": "DNS response flags\\n" + }, + { + "name": "response.size", + "type": "integer", + "format": "bytes", + "description": "size of the DNS response\\n" + }, + { + "name": "dnssec_ok", + "type": "boolean", + "description": "dnssec flag\\n" + } + ] + }, + { + "name": "@timestamp", + "level": "core", + "required": true, + "type": "date", + "description": "Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events.", + "example": "2016-05-23T08:05:34.853Z" + }, + { + "name": "labels", + "level": "core", + "type": "object", + "object_type": "keyword", + "description": "Custom key/value pairs. Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword. Example: \`docker\` and \`k8s\` labels.", + "example": { + "application": "foo-bar", + "env": "production" + } + }, + { + "name": "message", + "level": "core", + "type": "text", + "description": "For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.", + "example": "Hello World" + }, + { + "name": "tags", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "List of keywords used to tag each event.", + "example": "[\\"production\\", \\"env2\\"]" + }, + { + "name": "agent", + "title": "Agent", + "group": 2, + "description": "The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include Beats. Agents may also run on observers. ECS agent.* fields shall be populated with details of the agent running on the host or observer where the event happened or the measurement was taken.", + "footnote": "Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the agent running in the app/service. The agent information does not change if data is sent through queuing systems like Kafka, Redis, or processing systems such as Logstash or APM Server.", + "type": "group", + "fields": [ + { + "name": "ephemeral_id", + "level": "extended", + "type": "keyword", + "ignore_above": 1024, + "description": "Ephemeral identifier of this agent (if one exists). This id normally changes across restarts, but \`agent.id\` does not.", + "example": "8a4f500f" + }, + { + "name": "id", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.", + "example": "8a4f500d" + }, + { + "name": "name", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Custom name of the agent. This is a name that can be given to an agent. This can be helpful if for example two Filebeat instances are running on the same host but a human readable separation is needed on which Filebeat instance data is coming from. If no name is given, the name is often left empty.", + "example": "foo" + }, + { + "name": "type", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Type of the agent. The agent type stays always the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.", + "example": "filebeat" + }, + { + "name": "version", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Version of the agent.", + "example": "6.0.0-rc2" + } + ] + }, + { + "name": "as", + "title": "Autonomous System", + "group": 2, + "description": "An autonomous system (AS) is a collection of connected Internet Protocol (IP) routing prefixes under the control of one or more network operators on behalf of a single administrative entity or domain that presents a common, clearly defined routing policy to the internet.", + "type": "group", + "fields": [ + { + "name": "number", + "level": "extended", + "type": "long", + "description": "Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.", + "example": 15169 + }, + { + "name": "organization.name", + "level": "extended", + "type": "keyword", + "ignore_above": 1024, + "description": "Organization name.", + "example": "Google LLC" + } + ] + }, + { + "name": "nginx.access", + "type": "group", + "description": "Contains fields for the Nginx access logs.\\n", + "fields": [ + { + "name": "group_disabled", + "type": "group", + "enabled": false, + "fields": [ + { + "name": "message", + "type": "text" + } + ] + }, + { + "name": "remote_ip_list", + "type": "array", + "description": "An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like \`X-Forwarded-For\`. Real source IP is restored to \`source.ip\`.\\n" + }, + { + "name": "body_sent.bytes", + "type": "alias", + "path": "http.response.body.bytes", + "migration": true + }, + { + "name": "user_name", + "type": "alias", + "path": "user.name", + "migration": true + }, + { + "name": "method", + "type": "alias", + "path": "http.request.method", + "migration": true + }, + { + "name": "url", + "type": "alias", + "path": "url.original", + "migration": true + }, + { + "name": "http_version", + "type": "alias", + "path": "http.version", + "migration": true + }, + { + "name": "response_code", + "type": "alias", + "path": "http.response.status_code", + "migration": true + }, + { + "name": "referrer", + "type": "alias", + "path": "http.request.referrer", + "migration": true + }, + { + "name": "agent", + "type": "alias", + "path": "user_agent.original", + "migration": true + }, + { + "name": "user_agent", + "type": "group", + "fields": [ + { + "name": "device", + "type": "alias", + "path": "user_agent.device.name", + "migration": true + }, + { + "name": "name", + "type": "alias", + "path": "user_agent.name", + "migration": true + }, + { + "name": "os", + "type": "alias", + "path": "user_agent.os.full_name", + "migration": true + }, + { + "name": "os_name", + "type": "alias", + "path": "user_agent.os.name", + "migration": true + }, + { + "name": "original", + "type": "alias", + "path": "user_agent.original", + "migration": true + } + ] + }, + { + "name": "geoip", + "type": "group", + "fields": [ + { + "name": "continent_name", + "type": "alias", + "path": "source.geo.continent_name", + "migration": true + }, + { + "name": "country_iso_code", + "type": "alias", + "path": "source.geo.country_iso_code", + "migration": true + }, + { + "name": "location", + "type": "alias", + "path": "source.geo.location", + "migration": true + }, + { + "name": "region_name", + "type": "alias", + "path": "source.geo.region_name", + "migration": true + }, + { + "name": "city_name", + "type": "alias", + "path": "source.geo.city_name", + "migration": true + }, + { + "name": "region_iso_code", + "type": "alias", + "path": "source.geo.region_iso_code", + "migration": true + } + ] + } + ] + }, + { + "name": "source", + "type": "group", + "fields": [ + { + "name": "geo", + "type": "group", + "fields": [ + { + "name": "continent_name", + "type": "text" + } + ] + } + ] + }, + { + "name": "country", + "type": "", + "multi_fields": [ + { + "name": "keyword", + "type": "keyword" + }, + { + "name": "text", + "type": "text" + } + ] + } +] +`; + +exports[`creating index patterns from yaml fields flattenFields function flattens recursively and handles copying alias fields: flattenFields 1`] = ` +[ + { + "name": "coredns.id", + "type": "keyword", + "description": "id of the DNS transaction\\n" + }, + { + "name": "coredns.allParams", + "type": "integer", + "format": "bytes", + "pattern": "patternValQueryWeight", + "input_format": "inputFormatVal,", + "output_format": "outputFormalVal,", + "output_precision": "3,", + "label_template": "labelTemplateVal,", + "url_template": "urlTemplateVal,", + "openLinkInCurrentTab": "true,", + "description": "weight of the DNS query\\n" + }, + { + "name": "coredns.query.length", + "type": "integer", + "pattern": "patternValQueryLength", + "description": "length of the DNS query\\n" + }, + { + "name": "coredns.query.size", + "type": "integer", + "format": "bytes", + "pattern": "patternValQuerySize", + "description": "size of the DNS query\\n" + }, + { + "name": "coredns.query.class", + "type": "keyword", + "description": "DNS query class\\n" + }, + { + "name": "coredns.query.name", + "type": "keyword", + "description": "DNS query name\\n" + }, + { + "name": "coredns.query.type", + "type": "keyword", + "description": "DNS query type\\n" + }, + { + "name": "coredns.response.code", + "type": "keyword", + "description": "DNS response code\\n" + }, + { + "name": "coredns.response.flags", + "type": "keyword", + "description": "DNS response flags\\n" + }, + { + "name": "coredns.response.size", + "type": "integer", + "format": "bytes", + "description": "size of the DNS response\\n" + }, + { + "name": "coredns.dnssec_ok", + "type": "boolean", + "description": "dnssec flag\\n" + }, + { + "name": "@timestamp", + "level": "core", + "required": true, + "type": "date", + "description": "Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events.", + "example": "2016-05-23T08:05:34.853Z" + }, + { + "name": "labels", + "level": "core", + "type": "object", + "object_type": "keyword", + "description": "Custom key/value pairs. Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword. Example: \`docker\` and \`k8s\` labels.", + "example": { + "application": "foo-bar", + "env": "production" + } + }, + { + "name": "message", + "level": "core", + "type": "text", + "description": "For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.", + "example": "Hello World" + }, + { + "name": "tags", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "List of keywords used to tag each event.", + "example": "[\\"production\\", \\"env2\\"]" + }, + { + "name": "agent.ephemeral_id", + "level": "extended", + "type": "keyword", + "ignore_above": 1024, + "description": "Ephemeral identifier of this agent (if one exists). This id normally changes across restarts, but \`agent.id\` does not.", + "example": "8a4f500f" + }, + { + "name": "agent.id", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.", + "example": "8a4f500d" + }, + { + "name": "agent.name", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Custom name of the agent. This is a name that can be given to an agent. This can be helpful if for example two Filebeat instances are running on the same host but a human readable separation is needed on which Filebeat instance data is coming from. If no name is given, the name is often left empty.", + "example": "foo" + }, + { + "name": "agent.type", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Type of the agent. The agent type stays always the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.", + "example": "filebeat" + }, + { + "name": "agent.version", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Version of the agent.", + "example": "6.0.0-rc2" + }, + { + "name": "as.number", + "level": "extended", + "type": "long", + "description": "Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.", + "example": 15169 + }, + { + "name": "as.organization.name", + "level": "extended", + "type": "keyword", + "ignore_above": 1024, + "description": "Organization name.", + "example": "Google LLC" + }, + { + "name": "@timestamp", + "level": "core", + "required": true, + "type": "date", + "description": "Date/time when the event originated. This is the date/time extracted from the event, typically representing when the event was generated by the source. If the event source has no original timestamp, this value is typically populated by the first time the event was received by the pipeline. Required field for all events.", + "example": "2016-05-23T08:05:34.853Z" + }, + { + "name": "labels", + "level": "core", + "type": "object", + "object_type": "keyword", + "description": "Custom key/value pairs. Can be used to add meta information to events. Should not contain nested objects. All values are stored as keyword. Example: \`docker\` and \`k8s\` labels.", + "example": { + "application": "foo-bar", + "env": "production" + } + }, + { + "name": "message", + "level": "core", + "type": "text", + "description": "For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.", + "example": "Hello World" + }, + { + "name": "tags", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "List of keywords used to tag each event.", + "example": "[\\"production\\", \\"env2\\"]" + }, + { + "name": "agent.ephemeral_id", + "level": "extended", + "type": "keyword", + "ignore_above": 1024, + "description": "Ephemeral identifier of this agent (if one exists). This id normally changes across restarts, but \`agent.id\` does not.", + "example": "8a4f500f" + }, + { + "name": "agent.id", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id.", + "example": "8a4f500d" + }, + { + "name": "agent.name", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Custom name of the agent. This is a name that can be given to an agent. This can be helpful if for example two Filebeat instances are running on the same host but a human readable separation is needed on which Filebeat instance data is coming from. If no name is given, the name is often left empty.", + "example": "foo" + }, + { + "name": "agent.type", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Type of the agent. The agent type stays always the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.", + "example": "filebeat" + }, + { + "name": "agent.version", + "level": "core", + "type": "keyword", + "ignore_above": 1024, + "description": "Version of the agent.", + "example": "6.0.0-rc2" + }, + { + "name": "as.number", + "level": "extended", + "type": "long", + "description": "Unique number allocated to the autonomous system. The autonomous system number (ASN) uniquely identifies each network on the Internet.", + "example": 15169 + }, + { + "name": "as.organization.name", + "level": "extended", + "type": "keyword", + "ignore_above": 1024, + "description": "Organization name.", + "example": "Google LLC" + }, + { + "name": "nginx.access.remote_ip_list", + "type": "array", + "description": "An array of remote IP addresses. It is a list because it is common to include, besides the client IP address, IP addresses from headers like \`X-Forwarded-For\`. Real source IP is restored to \`source.ip\`.\\n" + }, + { + "name": "nginx.access.body_sent.bytes", + "type": "alias", + "path": "http.response.body.bytes", + "migration": true + }, + { + "name": "nginx.access.user_name", + "type": "alias", + "path": "user.name", + "migration": true + }, + { + "name": "nginx.access.method", + "type": "alias", + "path": "http.request.method", + "migration": true + }, + { + "name": "nginx.access.url", + "type": "alias", + "path": "url.original", + "migration": true + }, + { + "name": "nginx.access.http_version", + "type": "alias", + "path": "http.version", + "migration": true + }, + { + "name": "nginx.access.response_code", + "type": "alias", + "path": "http.response.status_code", + "migration": true + }, + { + "name": "nginx.access.referrer", + "type": "alias", + "path": "http.request.referrer", + "migration": true + }, + { + "name": "nginx.access.agent", + "type": "alias", + "path": "user_agent.original", + "migration": true + }, + { + "name": "nginx.access.user_agent.device", + "type": "alias", + "path": "user_agent.device.name", + "migration": true + }, + { + "name": "nginx.access.user_agent.name", + "type": "alias", + "path": "user_agent.name", + "migration": true + }, + { + "name": "nginx.access.user_agent.os", + "type": "alias", + "path": "user_agent.os.full_name", + "migration": true + }, + { + "name": "nginx.access.user_agent.os_name", + "type": "alias", + "path": "user_agent.os.name", + "migration": true + }, + { + "name": "nginx.access.user_agent.original", + "type": "alias", + "path": "user_agent.original", + "migration": true + }, + { + "name": "nginx.access.geoip.continent_name", + "type": "text", + "path": "source.geo.continent_name" + }, + { + "name": "nginx.access.geoip.country_iso_code", + "type": "alias", + "path": "source.geo.country_iso_code", + "migration": true + }, + { + "name": "nginx.access.geoip.location", + "type": "alias", + "path": "source.geo.location", + "migration": true + }, + { + "name": "nginx.access.geoip.region_name", + "type": "alias", + "path": "source.geo.region_name", + "migration": true + }, + { + "name": "nginx.access.geoip.city_name", + "type": "alias", + "path": "source.geo.city_name", + "migration": true + }, + { + "name": "nginx.access.geoip.region_iso_code", + "type": "alias", + "path": "source.geo.region_iso_code", + "migration": true + }, + { + "name": "source.geo.continent_name", + "type": "text" + }, + { + "name": "country", + "type": "", + "multi_fields": [ + { + "name": "keyword", + "type": "keyword" + }, + { + "name": "text", + "type": "text" + } + ] + }, + { + "name": "country.keyword", + "type": "keyword" + }, + { + "name": "country.text", + "type": "text" + } +] +`; diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/install.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/install.test.ts new file mode 100644 index 00000000000000..5e883772957d25 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/install.test.ts @@ -0,0 +1,311 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import path from 'path'; +import { readFileSync } from 'fs'; +import glob from 'glob'; +import { safeLoad } from 'js-yaml'; +import { + flattenFields, + dedupeFields, + transformField, + findFieldByPath, + IndexPatternField, + createFieldFormatMap, + createIndexPatternFields, + createIndexPattern, +} from './install'; +import { Fields, Field } from '../../fields/field'; + +// Add our own serialiser to just do JSON.stringify +expect.addSnapshotSerializer({ + print(val) { + return JSON.stringify(val, null, 2); + }, + + test(val) { + return val; + }, +}); +const files = glob.sync(path.join(__dirname, '/tests/*.yml')); +let fields: Fields = []; +for (const file of files) { + const fieldsYML = readFileSync(file, 'utf-8'); + fields = fields.concat(safeLoad(fieldsYML)); +} + +describe('creating index patterns from yaml fields', () => { + interface Test { + fields: Field[]; + expect: string | number | boolean | undefined; + } + + const name = 'testField'; + + test('createIndexPatternFields function creates Kibana index pattern fields and fieldFormatMap', () => { + const indexPatternFields = createIndexPatternFields(fields); + expect(indexPatternFields).toMatchSnapshot('createIndexPatternFields'); + }); + + test('createIndexPattern function creates Kibana index pattern', () => { + const indexPattern = createIndexPattern('logs', fields); + expect(indexPattern).toMatchSnapshot('createIndexPattern'); + }); + + test('flattenFields function flattens recursively and handles copying alias fields', () => { + const flattened = flattenFields(fields); + expect(flattened).toMatchSnapshot('flattenFields'); + }); + + test('dedupFields function remove duplicated fields when parsing multiple files', () => { + const deduped = dedupeFields(fields); + expect(deduped).toMatchSnapshot('dedupeFields'); + }); + + describe('getFieldByPath searches recursively for field in fields given dot separated path', () => { + const searchFields: Fields = [ + { + name: '1', + fields: [ + { + name: '1-1', + }, + { + name: '1-2', + }, + ], + }, + { + name: '2', + fields: [ + { + name: '2-1', + }, + { + name: '2-2', + fields: [ + { + name: '2-2-1', + }, + { + name: '2-2-2', + }, + ], + }, + ], + }, + ]; + test('returns undefined when the field does not exist', () => { + expect(findFieldByPath(searchFields, '0')).toBe(undefined); + }); + test('returns undefined if the field is not a leaf node', () => { + expect(findFieldByPath(searchFields, '1')?.name).toBe(undefined); + }); + test('returns undefined searching for a nested field that does not exist', () => { + expect(findFieldByPath(searchFields, '1.1-3')?.name).toBe(undefined); + }); + test('returns nested field that is a leaf node', () => { + expect(findFieldByPath(searchFields, '2.2-2.2-2-1')?.name).toBe('2-2-1'); + }); + }); + + test('transformField maps field types to kibana index pattern data types', () => { + const tests: Test[] = [ + { fields: [{ name: 'testField' }], expect: 'string' }, + { fields: [{ name: 'testField', type: 'half_float' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'scaled_float' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'float' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'integer' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'long' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'short' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'byte' }], expect: 'number' }, + { fields: [{ name: 'testField', type: 'keyword' }], expect: 'string' }, + { fields: [{ name: 'testField', type: 'invalidType' }], expect: undefined }, + { fields: [{ name: 'testField', type: 'text' }], expect: 'string' }, + { fields: [{ name: 'testField', type: 'date' }], expect: 'date' }, + { fields: [{ name: 'testField', type: 'geo_point' }], expect: 'geo_point' }, + ]; + + tests.forEach(test => { + const res = test.fields.map(transformField); + expect(res[0].type).toBe(test.expect); + }); + }); + + test('transformField changes values based on other values', () => { + interface TestWithAttr extends Test { + attr: keyof IndexPatternField; + } + + const tests: TestWithAttr[] = [ + // count + { fields: [{ name }], expect: 0, attr: 'count' }, + { fields: [{ name, count: 4 }], expect: 4, attr: 'count' }, + + // searchable + { fields: [{ name }], expect: true, attr: 'searchable' }, + { fields: [{ name, searchable: true }], expect: true, attr: 'searchable' }, + { fields: [{ name, searchable: false }], expect: false, attr: 'searchable' }, + { fields: [{ name, type: 'binary' }], expect: false, attr: 'searchable' }, + { fields: [{ name, searchable: true, type: 'binary' }], expect: false, attr: 'searchable' }, + { + fields: [{ name, searchable: true, type: 'object', enabled: false }], + expect: false, + attr: 'searchable', + }, + + // aggregatable + { fields: [{ name }], expect: true, attr: 'aggregatable' }, + { fields: [{ name, aggregatable: true }], expect: true, attr: 'aggregatable' }, + { fields: [{ name, aggregatable: false }], expect: false, attr: 'aggregatable' }, + { fields: [{ name, type: 'binary' }], expect: false, attr: 'aggregatable' }, + { + fields: [{ name, aggregatable: true, type: 'binary' }], + expect: false, + attr: 'aggregatable', + }, + { fields: [{ name, type: 'keyword' }], expect: true, attr: 'aggregatable' }, + { fields: [{ name, type: 'text', aggregatable: true }], expect: false, attr: 'aggregatable' }, + { fields: [{ name, type: 'text' }], expect: false, attr: 'aggregatable' }, + { + fields: [{ name, aggregatable: true, type: 'object', enabled: false }], + expect: false, + attr: 'aggregatable', + }, + + // analyzed + { fields: [{ name }], expect: false, attr: 'analyzed' }, + { fields: [{ name, analyzed: true }], expect: true, attr: 'analyzed' }, + { fields: [{ name, analyzed: false }], expect: false, attr: 'analyzed' }, + { fields: [{ name, type: 'binary' }], expect: false, attr: 'analyzed' }, + { fields: [{ name, analyzed: true, type: 'binary' }], expect: false, attr: 'analyzed' }, + { + fields: [{ name, analyzed: true, type: 'object', enabled: false }], + expect: false, + attr: 'analyzed', + }, + + // doc_values always set to true except for meta fields + { fields: [{ name }], expect: true, attr: 'doc_values' }, + { fields: [{ name, doc_values: true }], expect: true, attr: 'doc_values' }, + { fields: [{ name, doc_values: false }], expect: false, attr: 'doc_values' }, + { fields: [{ name, script: 'doc[]' }], expect: false, attr: 'doc_values' }, + { fields: [{ name, doc_values: true, script: 'doc[]' }], expect: false, attr: 'doc_values' }, + { fields: [{ name, type: 'binary' }], expect: false, attr: 'doc_values' }, + { fields: [{ name, doc_values: true, type: 'binary' }], expect: true, attr: 'doc_values' }, + { + fields: [{ name, doc_values: true, type: 'object', enabled: false }], + expect: false, + attr: 'doc_values', + }, + + // enabled - only applies to objects (and only if set) + { fields: [{ name, type: 'binary', enabled: false }], expect: undefined, attr: 'enabled' }, + { fields: [{ name, type: 'binary', enabled: true }], expect: undefined, attr: 'enabled' }, + { fields: [{ name, type: 'object', enabled: true }], expect: true, attr: 'enabled' }, + { fields: [{ name, type: 'object', enabled: false }], expect: false, attr: 'enabled' }, + { + fields: [{ name, type: 'object', enabled: false }], + expect: false, + attr: 'doc_values', + }, + + // indexed + { fields: [{ name, type: 'binary' }], expect: false, attr: 'indexed' }, + { + fields: [{ name, index: true, type: 'binary' }], + expect: false, + attr: 'indexed', + }, + { + fields: [{ name, index: true, type: 'object', enabled: false }], + expect: false, + attr: 'indexed', + }, + + // script, scripted + { fields: [{ name }], expect: false, attr: 'scripted' }, + { fields: [{ name }], expect: undefined, attr: 'script' }, + { fields: [{ name, script: 'doc[]' }], expect: true, attr: 'scripted' }, + { fields: [{ name, script: 'doc[]' }], expect: 'doc[]', attr: 'script' }, + + // lang + { fields: [{ name }], expect: undefined, attr: 'lang' }, + { fields: [{ name, script: 'doc[]' }], expect: 'painless', attr: 'lang' }, + ]; + tests.forEach(test => { + const res = test.fields.map(transformField); + expect(res[0][test.attr]).toBe(test.expect); + }); + }); + + describe('createFieldFormatMap creates correct map based on inputs', () => { + test('field with no format or pattern have empty fieldFormatMap', () => { + const fieldsToFormat = [{ name: 'fieldName', input_format: 'inputFormatVal' }]; + const fieldFormatMap = createFieldFormatMap(fieldsToFormat); + expect(fieldFormatMap).toEqual({}); + }); + test('field with pattern and no format creates fieldFormatMap with no id', () => { + const fieldsToFormat = [ + { name: 'fieldName', pattern: 'patternVal', input_format: 'inputFormatVal' }, + ]; + const fieldFormatMap = createFieldFormatMap(fieldsToFormat); + const expectedFieldFormatMap = { + fieldName: { + params: { + pattern: 'patternVal', + inputFormat: 'inputFormatVal', + }, + }, + }; + expect(fieldFormatMap).toEqual(expectedFieldFormatMap); + }); + + test('field with format and params creates fieldFormatMap with id', () => { + const fieldsToFormat = [ + { + name: 'fieldName', + format: 'formatVal', + pattern: 'patternVal', + input_format: 'inputFormatVal', + }, + ]; + const fieldFormatMap = createFieldFormatMap(fieldsToFormat); + const expectedFieldFormatMap = { + fieldName: { + id: 'formatVal', + params: { + pattern: 'patternVal', + inputFormat: 'inputFormatVal', + }, + }, + }; + expect(fieldFormatMap).toEqual(expectedFieldFormatMap); + }); + + test('all variations and all the params get passed through', () => { + const fieldsToFormat = [ + { name: 'fieldPattern', pattern: 'patternVal' }, + { name: 'fieldFormat', format: 'formatVal' }, + { name: 'fieldFormatWithParam', format: 'formatVal', output_precision: 2 }, + { name: 'fieldFormatAndPattern', format: 'formatVal', pattern: 'patternVal' }, + { + name: 'fieldFormatAndAllParams', + format: 'formatVal', + pattern: 'pattenVal', + input_format: 'inputFormatVal', + output_format: 'outputFormalVal', + output_precision: 3, + label_template: 'labelTemplateVal', + url_template: 'urlTemplateVal', + openLinkInCurrentTab: true, + }, + ]; + const fieldFormatMap = createFieldFormatMap(fieldsToFormat); + expect(fieldFormatMap).toMatchSnapshot('createFieldFormatMap'); + }); + }); +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/install.ts b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/install.ts new file mode 100644 index 00000000000000..264000f9892ba0 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/install.ts @@ -0,0 +1,338 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { INDEX_PATTERN_SAVED_OBJECT_TYPE } from '../../../../constants'; +import * as Registry from '../../registry'; +import { loadFieldsFromYaml, Fields, Field } from '../../fields/field'; +import { getPackageKeysByStatus } from '../../packages/get'; +import { InstallationStatus, RegistryPackage } from '../../../../types'; + +interface FieldFormatMap { + [key: string]: FieldFormatMapItem; +} +interface FieldFormatMapItem { + id?: string; + params?: FieldFormatParams; +} +interface FieldFormatParams { + pattern?: string; + inputFormat?: string; + outputFormat?: string; + outputPrecision?: number; + labelTemplate?: string; + urlTemplate?: string; + openLinkInCurrentTab?: boolean; +} +/* this should match https://github.com/elastic/beats/blob/d9a4c9c240a9820fab15002592e5bb6db318543b/libbeat/kibana/fields_transformer.go */ +interface TypeMap { + [key: string]: string; +} +const typeMap: TypeMap = { + binary: 'binary', + half_float: 'number', + scaled_float: 'number', + float: 'number', + integer: 'number', + long: 'number', + short: 'number', + byte: 'number', + text: 'string', + keyword: 'string', + '': 'string', + geo_point: 'geo_point', + date: 'date', + ip: 'ip', + boolean: 'boolean', +}; + +export interface IndexPatternField { + name: string; + type?: string; + count: number; + scripted: boolean; + indexed: boolean; + analyzed: boolean; + searchable: boolean; + aggregatable: boolean; + doc_values: boolean; + enabled?: boolean; + script?: string; + lang?: string; +} +export enum IndexPatternType { + logs = 'logs', + metrics = 'metrics', + events = 'events', +} + +export async function installIndexPatterns( + savedObjectsClient: SavedObjectsClientContract, + pkgkey?: string +) { + // get all user installed packages + const installedPackages = await getPackageKeysByStatus( + savedObjectsClient, + InstallationStatus.installed + ); + // add this package + if (pkgkey) installedPackages.push(pkgkey); + + // get each package's registry info + const installedPackagesFetchInfoPromise = installedPackages.map(pkg => Registry.fetchInfo(pkg)); + const installedPackagesInfo = await Promise.all(installedPackagesFetchInfoPromise); + + // for each index pattern type, create an index pattern + const indexPatternTypes = [ + IndexPatternType.logs, + IndexPatternType.metrics, + IndexPatternType.events, + ]; + indexPatternTypes.forEach(async indexPatternType => { + // if this is an update because a package is being unisntalled (no pkgkey argument passed) and no other packages are installed, remove the index pattern + if (!pkgkey && installedPackages.length === 0) { + try { + await savedObjectsClient.delete( + INDEX_PATTERN_SAVED_OBJECT_TYPE, + `epm-ip-${indexPatternType}` + ); + } catch (err) { + // index pattern was probably deleted by the user already + } + return; + } + + // get all dataset fields from all installed packages + const fields = await getAllDatasetFieldsByType(installedPackagesInfo, indexPatternType); + + const kibanaIndexPattern = createIndexPattern(indexPatternType, fields); + // create or overwrite the index pattern + await savedObjectsClient.create(INDEX_PATTERN_SAVED_OBJECT_TYPE, kibanaIndexPattern, { + id: `epm-ip-${indexPatternType}`, + overwrite: true, + }); + }); +} + +// loops through all given packages and returns an array +// of all fields from all datasets matching datasetType +export const getAllDatasetFieldsByType = async ( + packages: RegistryPackage[], + datasetType: IndexPatternType +): Promise => { + const datasetsPromises = packages.reduce>>((acc, pkg) => { + if (pkg.datasets) { + // filter out datasets by datasetType + const matchingDatasets = pkg.datasets.filter(dataset => dataset.type === datasetType); + matchingDatasets.forEach(dataset => acc.push(loadFieldsFromYaml(pkg, dataset.path))); + } + return acc; + }, []); + + // get all the datasets for each installed package into one array + const allDatasetFields: Fields[] = await Promise.all(datasetsPromises); + return allDatasetFields.flat(); +}; + +// creates or updates index pattern +export const createIndexPattern = (indexPatternType: string, fields: Fields) => { + const { indexPatternFields, fieldFormatMap } = createIndexPatternFields(fields); + + return { + title: `${indexPatternType}-*`, + timeFieldName: '@timestamp', + fields: JSON.stringify(indexPatternFields), + fieldFormatMap: JSON.stringify(fieldFormatMap), + }; +}; + +// takes fields from yaml files and transforms into Kibana Index Pattern fields +// and also returns the fieldFormatMap +export const createIndexPatternFields = ( + fields: Fields +): { indexPatternFields: IndexPatternField[]; fieldFormatMap: FieldFormatMap } => { + const dedupedFields = dedupeFields(fields); + const flattenedFields = flattenFields(dedupedFields); + const fieldFormatMap = createFieldFormatMap(flattenedFields); + const transformedFields = flattenedFields.map(transformField); + return { indexPatternFields: transformedFields, fieldFormatMap }; +}; + +export const dedupeFields = (fields: Fields) => { + const uniqueObj = fields.reduce<{ [name: string]: Field }>((acc, field) => { + if (!acc[field.name]) { + acc[field.name] = field; + } + return acc; + }, {}); + + return Object.values(uniqueObj); +}; + +/** + * search through fields with field's path property + * returns undefined if field not found or field is not a leaf node + * @param allFields fields to search + * @param path dot separated path from field.path + */ +export const findFieldByPath = (allFields: Fields, path: string): Field | undefined => { + const pathParts = path.split('.'); + return getField(allFields, pathParts); +}; + +const getField = (fields: Fields, pathNames: string[]): Field | undefined => { + if (!pathNames.length) return undefined; + // get the first rest of path names + const [name, ...restPathNames] = pathNames; + for (const field of fields) { + if (field.name === name) { + // check field's fields, passing in the remaining path names + if (field.fields && field.fields.length > 0) { + return getField(field.fields, restPathNames); + } + // no nested fields to search, but still more names - not found + if (restPathNames.length) { + return undefined; + } + return field; + } + } + return undefined; +}; + +export const transformField = (field: Field, i: number, fields: Fields): IndexPatternField => { + const newField: IndexPatternField = { + name: field.name, + count: field.count ?? 0, + scripted: false, + indexed: field.index ?? true, + analyzed: field.analyzed ?? false, + searchable: field.searchable ?? true, + aggregatable: field.aggregatable ?? true, + doc_values: field.doc_values ?? true, + }; + + // if type exists, check if it exists in the map + if (field.type) { + // if no type match type is not set (undefined) + if (typeMap[field.type]) { + newField.type = typeMap[field.type]; + } + // if type isn't set, default to string + } else { + newField.type = 'string'; + } + + if (newField.type === 'binary') { + newField.aggregatable = false; + newField.analyzed = false; + newField.doc_values = field.doc_values ?? false; + newField.indexed = false; + newField.searchable = false; + } + + if (field.type === 'object' && field.hasOwnProperty('enabled')) { + const enabled = field.enabled ?? true; + newField.enabled = enabled; + if (!enabled) { + newField.aggregatable = false; + newField.analyzed = false; + newField.doc_values = false; + newField.indexed = false; + newField.searchable = false; + } + } + + if (field.type === 'text') { + newField.aggregatable = false; + } + + if (field.hasOwnProperty('script')) { + newField.scripted = true; + newField.script = field.script; + newField.lang = 'painless'; + newField.doc_values = false; + } + + return newField; +}; + +/** + * flattenFields + * + * flattens fields and renames them with a path of the parent names + */ + +export const flattenFields = (allFields: Fields): Fields => { + const flatten = (fields: Fields): Fields => + fields.reduce((acc, field) => { + // recurse through nested fields + if (field.type === 'group' && field.fields?.length) { + // skip if field.enabled is not explicitly set to false + if (!field.hasOwnProperty('enabled') || field.enabled === true) { + acc = renameAndFlatten(field, field.fields, [...acc]); + } + } else { + // handle alias type fields + if (field.type === 'alias' && field.path) { + const foundField = findFieldByPath(allFields, field.path); + // if aliased leaf field is found copy its props over except path and name + if (foundField) { + const { path, name } = field; + field = { ...foundField, path, name }; + } + } + // add field before going through multi_fields because we still want to add the parent field + acc.push(field); + + // for each field in multi_field add new field + if (field.multi_fields?.length) { + acc = renameAndFlatten(field, field.multi_fields, [...acc]); + } + } + return acc; + }, []); + + // helper function to call flatten() and rename the fields + const renameAndFlatten = (field: Field, fields: Fields, acc: Fields): Fields => { + const flattenedFields = flatten(fields); + flattenedFields.forEach(nestedField => { + acc.push({ + ...nestedField, + name: `${field.name}.${nestedField.name}`, + }); + }); + return acc; + }; + + return flatten(allFields); +}; + +export const createFieldFormatMap = (fields: Fields): FieldFormatMap => + fields.reduce((acc, field) => { + if (field.format || field.pattern) { + const fieldFormatMapItem: FieldFormatMapItem = {}; + if (field.format) { + fieldFormatMapItem.id = field.format; + } + const params = getFieldFormatParams(field); + if (Object.keys(params).length) fieldFormatMapItem.params = params; + acc[field.name] = fieldFormatMapItem; + } + return acc; + }, {}); + +const getFieldFormatParams = (field: Field): FieldFormatParams => { + const params: FieldFormatParams = {}; + if (field.pattern) params.pattern = field.pattern; + if (field.input_format) params.inputFormat = field.input_format; + if (field.output_format) params.outputFormat = field.output_format; + if (field.output_precision) params.outputPrecision = field.output_precision; + if (field.label_template) params.labelTemplate = field.label_template; + if (field.url_template) params.urlTemplate = field.url_template; + if (field.open_link_in_current_tab) params.openLinkInCurrentTab = field.open_link_in_current_tab; + return params; +}; diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/coredns.logs.yml b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/coredns.logs.yml new file mode 100644 index 00000000000000..d66a4cf62bc41f --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/coredns.logs.yml @@ -0,0 +1,71 @@ +- name: coredns + type: group + description: > + coredns fields after normalization + fields: + - name: id + type: keyword + description: > + id of the DNS transaction + + - name: allParams + type: integer + format: bytes + pattern: patternValQueryWeight + input_format: inputFormatVal, + output_format: outputFormalVal, + output_precision: 3, + label_template: labelTemplateVal, + url_template: urlTemplateVal, + openLinkInCurrentTab: true, + description: > + weight of the DNS query + + - name: query.length + type: integer + pattern: patternValQueryLength + description: > + length of the DNS query + + - name: query.size + type: integer + format: bytes + pattern: patternValQuerySize + description: > + size of the DNS query + + - name: query.class + type: keyword + description: > + DNS query class + + - name: query.name + type: keyword + description: > + DNS query name + + - name: query.type + type: keyword + description: > + DNS query type + + - name: response.code + type: keyword + description: > + DNS response code + + - name: response.flags + type: keyword + description: > + DNS response flags + + - name: response.size + type: integer + format: bytes + description: > + size of the DNS response + + - name: dnssec_ok + type: boolean + description: > + dnssec flag diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.access.ecs.yml b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.access.ecs.yml new file mode 100644 index 00000000000000..51090a0fe7cf01 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.access.ecs.yml @@ -0,0 +1,112 @@ +- name: '@timestamp' + level: core + required: true + type: date + description: 'Date/time when the event originated. + This is the date/time extracted from the event, typically representing when + the event was generated by the source. + If the event source has no original timestamp, this value is typically populated + by the first time the event was received by the pipeline. + Required field for all events.' + example: '2016-05-23T08:05:34.853Z' +- name: labels + level: core + type: object + object_type: keyword + description: 'Custom key/value pairs. + Can be used to add meta information to events. Should not contain nested objects. + All values are stored as keyword. + Example: `docker` and `k8s` labels.' + example: + application: foo-bar + env: production +- name: message + level: core + type: text + description: 'For log events the message field contains the log message, optimized + for viewing in a log viewer. + For structured logs without an original message field, other fields can be concatenated + to form a human-readable summary of the event. + If multiple messages exist, they can be combined into one message.' + example: Hello World +- name: tags + level: core + type: keyword + ignore_above: 1024 + description: List of keywords used to tag each event. + example: '["production", "env2"]' +- name: agent + title: Agent + group: 2 + description: 'The agent fields contain the data about the software entity, if + any, that collects, detects, or observes events on a host, or takes measurements + on a host. + Examples include Beats. Agents may also run on observers. ECS agent.* fields + shall be populated with details of the agent running on the host or observer + where the event happened or the measurement was taken.' + footnote: 'Examples: In the case of Beats for logs, the agent.name is filebeat. + For APM, it is the agent running in the app/service. The agent information does + not change if data is sent through queuing systems like Kafka, Redis, or processing + systems such as Logstash or APM Server.' + type: group + fields: + - name: ephemeral_id + level: extended + type: keyword + ignore_above: 1024 + description: 'Ephemeral identifier of this agent (if one exists). + This id normally changes across restarts, but `agent.id` does not.' + example: 8a4f500f + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique identifier of this agent (if one exists). + Example: For Beats this would be beat.id.' + example: 8a4f500d + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Custom name of the agent. + This is a name that can be given to an agent. This can be helpful if for example + two Filebeat instances are running on the same host but a human readable separation + is needed on which Filebeat instance data is coming from. + If no name is given, the name is often left empty.' + example: foo + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of the agent. + The agent type stays always the same and should be given by the agent used. + In case of Filebeat the agent would always be Filebeat also if two Filebeat + instances are run on the same machine.' + example: filebeat + - name: version + level: core + type: keyword + ignore_above: 1024 + description: Version of the agent. + example: 6.0.0-rc2 +- name: as + title: Autonomous System + group: 2 + description: An autonomous system (AS) is a collection of connected Internet Protocol + (IP) routing prefixes under the control of one or more network operators on + behalf of a single administrative entity or domain that presents a common, clearly + defined routing policy to the internet. + type: group + fields: + - name: number + level: extended + type: long + description: Unique number allocated to the autonomous system. The autonomous + system number (ASN) uniquely identifies each network on the Internet. + example: 15169 + - name: organization.name + level: extended + type: keyword + ignore_above: 1024 + description: Organization name. + example: Google LLC \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.error.ecs.yml b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.error.ecs.yml new file mode 100644 index 00000000000000..51090a0fe7cf01 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.error.ecs.yml @@ -0,0 +1,112 @@ +- name: '@timestamp' + level: core + required: true + type: date + description: 'Date/time when the event originated. + This is the date/time extracted from the event, typically representing when + the event was generated by the source. + If the event source has no original timestamp, this value is typically populated + by the first time the event was received by the pipeline. + Required field for all events.' + example: '2016-05-23T08:05:34.853Z' +- name: labels + level: core + type: object + object_type: keyword + description: 'Custom key/value pairs. + Can be used to add meta information to events. Should not contain nested objects. + All values are stored as keyword. + Example: `docker` and `k8s` labels.' + example: + application: foo-bar + env: production +- name: message + level: core + type: text + description: 'For log events the message field contains the log message, optimized + for viewing in a log viewer. + For structured logs without an original message field, other fields can be concatenated + to form a human-readable summary of the event. + If multiple messages exist, they can be combined into one message.' + example: Hello World +- name: tags + level: core + type: keyword + ignore_above: 1024 + description: List of keywords used to tag each event. + example: '["production", "env2"]' +- name: agent + title: Agent + group: 2 + description: 'The agent fields contain the data about the software entity, if + any, that collects, detects, or observes events on a host, or takes measurements + on a host. + Examples include Beats. Agents may also run on observers. ECS agent.* fields + shall be populated with details of the agent running on the host or observer + where the event happened or the measurement was taken.' + footnote: 'Examples: In the case of Beats for logs, the agent.name is filebeat. + For APM, it is the agent running in the app/service. The agent information does + not change if data is sent through queuing systems like Kafka, Redis, or processing + systems such as Logstash or APM Server.' + type: group + fields: + - name: ephemeral_id + level: extended + type: keyword + ignore_above: 1024 + description: 'Ephemeral identifier of this agent (if one exists). + This id normally changes across restarts, but `agent.id` does not.' + example: 8a4f500f + - name: id + level: core + type: keyword + ignore_above: 1024 + description: 'Unique identifier of this agent (if one exists). + Example: For Beats this would be beat.id.' + example: 8a4f500d + - name: name + level: core + type: keyword + ignore_above: 1024 + description: 'Custom name of the agent. + This is a name that can be given to an agent. This can be helpful if for example + two Filebeat instances are running on the same host but a human readable separation + is needed on which Filebeat instance data is coming from. + If no name is given, the name is often left empty.' + example: foo + - name: type + level: core + type: keyword + ignore_above: 1024 + description: 'Type of the agent. + The agent type stays always the same and should be given by the agent used. + In case of Filebeat the agent would always be Filebeat also if two Filebeat + instances are run on the same machine.' + example: filebeat + - name: version + level: core + type: keyword + ignore_above: 1024 + description: Version of the agent. + example: 6.0.0-rc2 +- name: as + title: Autonomous System + group: 2 + description: An autonomous system (AS) is a collection of connected Internet Protocol + (IP) routing prefixes under the control of one or more network operators on + behalf of a single administrative entity or domain that presents a common, clearly + defined routing policy to the internet. + type: group + fields: + - name: number + level: extended + type: long + description: Unique number allocated to the autonomous system. The autonomous + system number (ASN) uniquely identifies each network on the Internet. + example: 15169 + - name: organization.name + level: extended + type: keyword + ignore_above: 1024 + description: Organization name. + example: Google LLC \ No newline at end of file diff --git a/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.fields.yml b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.fields.yml new file mode 100644 index 00000000000000..220225a2c246b9 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/kibana/index_pattern/tests/nginx.fields.yml @@ -0,0 +1,118 @@ +- name: nginx.access + type: group + description: > + Contains fields for the Nginx access logs. + fields: + - name: group_disabled + type: group + enabled: false + fields: + - name: message + type: text + - name: remote_ip_list + type: array + description: > + An array of remote IP addresses. It is a list because it is common to include, besides the client + IP address, IP addresses from headers like `X-Forwarded-For`. + Real source IP is restored to `source.ip`. + + - name: body_sent.bytes + type: alias + path: http.response.body.bytes + migration: true + - name: user_name + type: alias + path: user.name + migration: true + - name: method + type: alias + path: http.request.method + migration: true + - name: url + type: alias + path: url.original + migration: true + - name: http_version + type: alias + path: http.version + migration: true + - name: response_code + type: alias + path: http.response.status_code + migration: true + - name: referrer + type: alias + path: http.request.referrer + migration: true + - name: agent + type: alias + path: user_agent.original + migration: true + + - name: user_agent + type: group + fields: + - name: device + type: alias + path: user_agent.device.name + migration: true + - name: name + type: alias + path: user_agent.name + migration: true + - name: os + type: alias + path: user_agent.os.full_name + migration: true + - name: os_name + type: alias + path: user_agent.os.name + migration: true + - name: original + type: alias + path: user_agent.original + migration: true + + - name: geoip + type: group + fields: + - name: continent_name + type: alias + path: source.geo.continent_name + migration: true + - name: country_iso_code + type: alias + path: source.geo.country_iso_code + migration: true + - name: location + type: alias + path: source.geo.location + migration: true + - name: region_name + type: alias + path: source.geo.region_name + migration: true + - name: city_name + type: alias + path: source.geo.city_name + migration: true + - name: region_iso_code + type: alias + path: source.geo.region_iso_code + migration: true + +- name: source + type: group + fields: + - name: geo + type: group + fields: + - name: continent_name + type: text +- name: country + type: "" + multi_fields: + - name: keyword + type: keyword + - name: text + type: text diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/assets.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/assets.test.ts new file mode 100644 index 00000000000000..5153f9205dde73 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/assets.test.ts @@ -0,0 +1,66 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { RegistryPackage } from '../../../types'; +import { getAssets } from './assets'; + +const tests = [ + { + package: { + assets: [ + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-plaintext.json', + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-json.json', + ], + name: 'coredns', + version: '1.0.1', + }, + dataset: 'log', + filter: (path: string) => { + return true; + }, + expected: [ + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-plaintext.json', + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-json.json', + ], + }, + { + package: { + assets: [ + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-plaintext.json', + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-json.json', + ], + name: 'coredns', + version: '1.0.1', + }, + // Non existant dataset + dataset: 'foo', + filter: (path: string) => { + return true; + }, + expected: [], + }, + { + package: { + assets: [ + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-plaintext.json', + '/package/coredns-1.0.1/dataset/log/elasticsearch/ingest-pipeline/pipeline-json.json', + ], + }, + // Filter which does not exist + filter: (path: string) => { + return path.includes('foo'); + }, + expected: [], + }, +]; + +test('testGetAssets', () => { + for (const value of tests) { + // as needed to pretent it is a RegistryPackage + const assets = getAssets(value.package as RegistryPackage, value.filter, value.dataset); + expect(assets).toStrictEqual(value.expected); + } +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/assets.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/assets.ts new file mode 100644 index 00000000000000..ecc882d9c2e70c --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/assets.ts @@ -0,0 +1,72 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { RegistryPackage } from '../../../types'; +import * as Registry from '../registry'; +import { cacheHas } from '../registry/cache'; + +// paths from RegistryPackage are routes to the assets on EPR +// e.g. `/package/nginx-1.2.0/dataset/access/fields/fields.yml` +// paths for ArchiveEntry are routes to the assets in the archive +// e.g. `nginx-1.2.0/dataset/access/fields/fields.yml` +// RegistryPackage paths have a `/package/` prefix compared to ArchiveEntry paths +const EPR_PATH_PREFIX = '/package'; +function registryPathToArchivePath(registryPath: RegistryPackage['path']): string { + const archivePath = registryPath.replace(`${EPR_PATH_PREFIX}/`, ''); + return archivePath; +} + +export function getAssets( + packageInfo: RegistryPackage, + filter = (path: string): boolean => true, + datasetName?: string +): string[] { + const assets: string[] = []; + if (!packageInfo?.assets) return assets; + + // Skip directories + for (const path of packageInfo.assets) { + if (path.endsWith('/')) { + continue; + } + + // if dataset, filter for them + if (datasetName) { + // TODO: Filter for dataset path + const comparePath = `${EPR_PATH_PREFIX}/${packageInfo.name}-${packageInfo.version}/dataset/${datasetName}`; + if (!path.includes(comparePath)) { + continue; + } + } + if (!filter(path)) { + continue; + } + + assets.push(path); + } + return assets; +} + +export async function getAssetsData( + packageInfo: RegistryPackage, + filter = (path: string): boolean => true, + datasetName?: string +): Promise { + // TODO: Needs to be called to fill the cache but should not be required + const pkgkey = packageInfo.name + '-' + packageInfo.version; + if (!cacheHas(pkgkey)) await Registry.getArchiveInfo(pkgkey); + + // Gather all asset data + const assets = getAssets(packageInfo, filter, datasetName); + const entries: Registry.ArchiveEntry[] = assets.map(registryPath => { + const archivePath = registryPathToArchivePath(registryPath); + const buffer = Registry.getAsset(archivePath); + + return { path: registryPath, buffer }; + }); + + return entries; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/get.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/get.ts new file mode 100644 index 00000000000000..58416b7f66d2d2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/get.ts @@ -0,0 +1,128 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'src/core/server/'; +import { PACKAGES_SAVED_OBJECT_TYPE } from '../../../constants'; +import { Installation, InstallationStatus, PackageInfo } from '../../../types'; +import * as Registry from '../registry'; +import { createInstallableFrom } from './index'; + +export { fetchFile as getFile, SearchParams } from '../registry'; + +function nameAsTitle(name: string) { + return name.charAt(0).toUpperCase() + name.substr(1).toLowerCase(); +} + +export async function getCategories() { + return Registry.fetchCategories(); +} + +export async function getPackages( + options: { + savedObjectsClient: SavedObjectsClientContract; + } & Registry.SearchParams +) { + const { savedObjectsClient } = options; + const registryItems = await Registry.fetchList({ category: options.category }).then(items => { + return items.map(item => + Object.assign({}, item, { title: item.title || nameAsTitle(item.name) }) + ); + }); + const searchObjects = registryItems.map(({ name, version }) => ({ + type: PACKAGES_SAVED_OBJECT_TYPE, + id: `${name}-${version}`, + })); + const results = await savedObjectsClient.bulkGet(searchObjects); + const savedObjects = results.saved_objects.filter(o => !o.error); // ignore errors for now + const packageList = registryItems + .map(item => + createInstallableFrom( + item, + savedObjects.find(({ id }) => id === `${item.name}-${item.version}`) + ) + ) + .sort(sortByName); + return packageList; +} + +export async function getPackageKeysByStatus( + savedObjectsClient: SavedObjectsClientContract, + status: InstallationStatus +) { + const allPackages = await getPackages({ savedObjectsClient }); + return allPackages.reduce((acc, pkg) => { + if (pkg.status === status) { + acc.push(`${pkg.name}-${pkg.version}`); + } + return acc; + }, []); +} + +export async function getPackageInfo(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; +}): Promise { + const { savedObjectsClient, pkgkey } = options; + const [item, savedObject] = await Promise.all([ + Registry.fetchInfo(pkgkey), + getInstallationObject({ savedObjectsClient, pkgkey }), + Registry.getArchiveInfo(pkgkey), + ] as const); + // adding `as const` due to regression in TS 3.7.2 + // see https://github.com/microsoft/TypeScript/issues/34925#issuecomment-550021453 + // and https://github.com/microsoft/TypeScript/pull/33707#issuecomment-550718523 + + // add properties that aren't (or aren't yet) on Registry response + const updated = { + ...item, + title: item.title || nameAsTitle(item.name), + assets: Registry.groupPathsByService(item?.assets || []), + }; + return createInstallableFrom(updated, savedObject); +} + +export async function getInstallationObject(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; +}) { + const { savedObjectsClient, pkgkey } = options; + return savedObjectsClient + .get(PACKAGES_SAVED_OBJECT_TYPE, pkgkey) + .catch(e => undefined); +} + +export async function getInstallation(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; +}) { + const savedObject = await getInstallationObject(options); + return savedObject?.attributes; +} + +export async function findInstalledPackageByName(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgName: string; +}): Promise { + const { savedObjectsClient, pkgName } = options; + + const res = await savedObjectsClient.find({ + type: PACKAGES_SAVED_OBJECT_TYPE, + search: pkgName, + searchFields: ['name'], + }); + if (res.saved_objects.length) return res.saved_objects[0].attributes; + return undefined; +} + +function sortByName(a: { name: string }, b: { name: string }) { + if (a.name > b.name) { + return 1; + } else if (a.name < b.name) { + return -1; + } else { + return 0; + } +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/get_objects.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/get_objects.ts new file mode 100644 index 00000000000000..e0424aa8a36f53 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/get_objects.ts @@ -0,0 +1,72 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObject, SavedObjectsBulkCreateObject } from 'src/core/server/'; +import { AssetType } from '../../../types'; +import * as Registry from '../registry'; + +type ArchiveAsset = Pick; +type SavedObjectToBe = Required & { type: AssetType }; + +export async function getObjects( + pkgkey: string, + filter = (entry: Registry.ArchiveEntry): boolean => true +): Promise { + // Create a Map b/c some values, especially index-patterns, are referenced multiple times + const objects: Map = new Map(); + + // Get paths which match the given filter + const paths = await Registry.getArchiveInfo(pkgkey, filter); + + // Get all objects which matched filter. Add them to the Map + const rootObjects = await Promise.all(paths.map(getObject)); + rootObjects.forEach(obj => objects.set(obj.id, obj)); + + // Each of those objects might have `references` property like [{id, type, name}] + for (const object of rootObjects) { + // For each of those objects, if they have references + for (const reference of object.references) { + // Get the referenced objects. Call same function with a new filter + const referencedObjects = await getObjects(pkgkey, (entry: Registry.ArchiveEntry) => { + // Skip anything we've already stored + if (objects.has(reference.id)) return false; + + // Is the archive entry the reference we want? + const { type, file } = Registry.pathParts(entry.path); + const isType = type === reference.type; + const isJson = file === `${reference.id}.json`; + + return isType && isJson; + }); + + // Add referenced objects to the Map + referencedObjects.forEach(ro => objects.set(ro.id, ro)); + } + } + + // return the array of unique objects + return Array.from(objects.values()); +} + +export async function getObject(key: string) { + const buffer = Registry.getAsset(key); + + // cache values are buffers. convert to string / JSON + const json = buffer.toString('utf8'); + // convert that to an object + const asset: ArchiveAsset = JSON.parse(json); + + const { type, file } = Registry.pathParts(key); + const savedObject: SavedObjectToBe = { + type, + id: file.replace('.json', ''), + attributes: asset.attributes, + references: asset.references || [], + migrationVersion: asset.migrationVersion || {}, + }; + + return savedObject; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/index.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/index.ts new file mode 100644 index 00000000000000..2f84ea5b6f8db7 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/index.ts @@ -0,0 +1,53 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObject } from '../../../../../../../src/core/server'; +import { + AssetType, + Installable, + Installation, + InstallationStatus, + KibanaAssetType, +} from '../../../../common/types/models/epm'; + +export { + getCategories, + getFile, + getInstallationObject, + getInstallation, + getPackageInfo, + getPackages, + SearchParams, + findInstalledPackageByName, +} from './get'; + +export { installKibanaAssets, installPackage, ensureInstalledPackage } from './install'; +export { removeInstallation } from './remove'; + +export class PackageNotInstalledError extends Error { + constructor(pkgkey: string) { + super(`${pkgkey} is not installed`); + } +} + +// only Kibana Assets use Saved Objects at this point +export const savedObjectTypes: AssetType[] = Object.values(KibanaAssetType); + +export function createInstallableFrom( + from: T, + savedObject?: SavedObject +): Installable { + return savedObject + ? { + ...from, + status: InstallationStatus.installed, + savedObject, + } + : { + ...from, + status: InstallationStatus.notInstalled, + }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/install.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/install.ts new file mode 100644 index 00000000000000..acf77998fdb3c3 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/install.ts @@ -0,0 +1,199 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObject, SavedObjectsClientContract } from 'src/core/server/'; +import { PACKAGES_SAVED_OBJECT_TYPE } from '../../../constants'; +import { + AssetReference, + Installation, + KibanaAssetType, + CallESAsCurrentUser, + DefaultPackages, +} from '../../../types'; +import { installIndexPatterns } from '../kibana/index_pattern/install'; +import * as Registry from '../registry'; +import { getObject } from './get_objects'; +import { getInstallation, findInstalledPackageByName } from './index'; +import { installTemplates } from '../elasticsearch/template/install'; +import { installPipelines } from '../elasticsearch/ingest_pipeline/install'; +import { installILMPolicy } from '../elasticsearch/ilm/install'; + +export async function installLatestPackage(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgName: string; + callCluster: CallESAsCurrentUser; +}): Promise { + const { savedObjectsClient, pkgName, callCluster } = options; + try { + const latestPackage = await Registry.fetchFindLatestPackage(pkgName); + const pkgkey = Registry.pkgToPkgKey({ + name: latestPackage.name, + version: latestPackage.version, + }); + return installPackage({ savedObjectsClient, pkgkey, callCluster }); + } catch (err) { + throw err; + } +} + +export async function ensureInstalledDefaultPackages( + savedObjectsClient: SavedObjectsClientContract, + callCluster: CallESAsCurrentUser +): Promise { + const installations = []; + for (const pkgName in DefaultPackages) { + if (!DefaultPackages.hasOwnProperty(pkgName)) continue; + const installation = await ensureInstalledPackage({ + savedObjectsClient, + pkgName, + callCluster, + }); + if (installation) installations.push(installation); + } + + return installations; +} + +export async function ensureInstalledPackage(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgName: string; + callCluster: CallESAsCurrentUser; +}): Promise { + const { savedObjectsClient, pkgName, callCluster } = options; + const installedPackage = await findInstalledPackageByName({ savedObjectsClient, pkgName }); + if (installedPackage) { + return installedPackage; + } + // if the requested packaged was not found to be installed, try installing + try { + await installLatestPackage({ + savedObjectsClient, + pkgName, + callCluster, + }); + return await findInstalledPackageByName({ savedObjectsClient, pkgName }); + } catch (err) { + throw new Error(err.message); + } +} + +export async function installPackage(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; + callCluster: CallESAsCurrentUser; +}): Promise { + const { savedObjectsClient, pkgkey, callCluster } = options; + const registryPackageInfo = await Registry.fetchInfo(pkgkey); + const { name: pkgName, version: pkgVersion } = registryPackageInfo; + + const installKibanaAssetsPromise = installKibanaAssets({ + savedObjectsClient, + pkgkey, + }); + const installPipelinePromises = installPipelines(registryPackageInfo, callCluster); + const installTemplatePromises = installTemplates(registryPackageInfo, callCluster, pkgkey); + + // index patterns and ilm policies are not currently associated with a particular package + // so we do not save them in the package saved object state. at some point ILM policies can be installed/modified + // per dataset and we should then save them + await installIndexPatterns(savedObjectsClient, pkgkey); + // currenly only the base package has an ILM policy + await installILMPolicy(pkgkey, callCluster); + + const res = await Promise.all([ + installKibanaAssetsPromise, + installPipelinePromises, + installTemplatePromises, + ]); + + const toSave = res.flat(); + // Save those references in the package manager's state saved object + await saveInstallationReferences({ + savedObjectsClient, + pkgkey, + pkgName, + pkgVersion, + toSave, + }); + return toSave; +} + +// TODO: make it an exhaustive list +// e.g. switch statement with cases for each enum key returning `never` for default case +export async function installKibanaAssets(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; +}) { + const { savedObjectsClient, pkgkey } = options; + + // Only install Kibana assets during package installation. + const kibanaAssetTypes = Object.values(KibanaAssetType); + const installationPromises = kibanaAssetTypes.map(async assetType => + installKibanaSavedObjects({ savedObjectsClient, pkgkey, assetType }) + ); + + // installKibanaSavedObjects returns AssetReference[], so .map creates AssetReference[][] + // call .flat to flatten into one dimensional array + return Promise.all(installationPromises).then(results => results.flat()); +} + +export async function saveInstallationReferences(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; + pkgName: string; + pkgVersion: string; + toSave: AssetReference[]; +}) { + const { savedObjectsClient, pkgkey, pkgName, pkgVersion, toSave } = options; + const installation = await getInstallation({ savedObjectsClient, pkgkey }); + const savedRefs = installation?.installed || []; + const mergeRefsReducer = (current: AssetReference[], pending: AssetReference) => { + const hasRef = current.find(c => c.id === pending.id && c.type === pending.type); + if (!hasRef) current.push(pending); + return current; + }; + + const toInstall = toSave.reduce(mergeRefsReducer, savedRefs); + await savedObjectsClient.create( + PACKAGES_SAVED_OBJECT_TYPE, + { installed: toInstall, name: pkgName, version: pkgVersion }, + { id: pkgkey, overwrite: true } + ); + + return toInstall; +} + +async function installKibanaSavedObjects({ + savedObjectsClient, + pkgkey, + assetType, +}: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; + assetType: KibanaAssetType; +}) { + const isSameType = ({ path }: Registry.ArchiveEntry) => + assetType === Registry.pathParts(path).type; + const paths = await Registry.getArchiveInfo(pkgkey, isSameType); + const toBeSavedObjects = await Promise.all(paths.map(getObject)); + + if (toBeSavedObjects.length === 0) { + return []; + } else { + const createResults = await savedObjectsClient.bulkCreate(toBeSavedObjects, { + overwrite: true, + }); + const createdObjects = createResults.saved_objects; + const installed = createdObjects.map(toAssetReference); + return installed; + } +} + +function toAssetReference({ id, type }: SavedObject) { + const reference: AssetReference = { id, type: type as KibanaAssetType }; + + return reference; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/packages/remove.ts b/x-pack/plugins/ingest_manager/server/services/epm/packages/remove.ts new file mode 100644 index 00000000000000..e57729a7ab2ba2 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/packages/remove.ts @@ -0,0 +1,59 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'src/core/server/'; +import { PACKAGES_SAVED_OBJECT_TYPE } from '../../../constants'; +import { AssetReference, AssetType, ElasticsearchAssetType } from '../../../types'; +import { CallESAsCurrentUser } from '../../../types'; +import { getInstallation, savedObjectTypes } from './index'; +import { installIndexPatterns } from '../kibana/index_pattern/install'; + +export async function removeInstallation(options: { + savedObjectsClient: SavedObjectsClientContract; + pkgkey: string; + callCluster: CallESAsCurrentUser; +}): Promise { + const { savedObjectsClient, pkgkey, callCluster } = options; + const installation = await getInstallation({ savedObjectsClient, pkgkey }); + const installedObjects = installation?.installed || []; + + // Delete the manager saved object with references to the asset objects + // could also update with [] or some other state + await savedObjectsClient.delete(PACKAGES_SAVED_OBJECT_TYPE, pkgkey); + + // recreate or delete index patterns when a package is uninstalled + await installIndexPatterns(savedObjectsClient); + + // Delete the installed assets + const deletePromises = installedObjects.map(async ({ id, type }) => { + const assetType = type as AssetType; + if (savedObjectTypes.includes(assetType)) { + savedObjectsClient.delete(assetType, id); + } else if (assetType === ElasticsearchAssetType.ingestPipeline) { + deletePipeline(callCluster, id); + } else if (assetType === ElasticsearchAssetType.indexTemplate) { + deleteTemplate(callCluster, id); + } + }); + await Promise.all([...deletePromises]); + + // successful delete's in SO client return {}. return something more useful + return installedObjects; +} + +async function deletePipeline(callCluster: CallESAsCurrentUser, id: string): Promise { + // '*' shouldn't ever appear here, but it still would delete all ingest pipelines + if (id && id !== '*') { + await callCluster('ingest.deletePipeline', { id }); + } +} + +async function deleteTemplate(callCluster: CallESAsCurrentUser, name: string): Promise { + // '*' shouldn't ever appear here, but it still would delete all templates + if (name && name !== '*') { + await callCluster('indices.deleteTemplate', { name }); + } +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts b/x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts new file mode 100644 index 00000000000000..17d52bc745a55f --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/registry/cache.ts @@ -0,0 +1,10 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +const cache: Map = new Map(); +export const cacheGet = (key: string) => cache.get(key); +export const cacheSet = (key: string, value: Buffer) => cache.set(key, value); +export const cacheHas = (key: string) => cache.has(key); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/registry/extract.ts b/x-pack/plugins/ingest_manager/server/services/epm/registry/extract.ts new file mode 100644 index 00000000000000..feed2236f06ebe --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/registry/extract.ts @@ -0,0 +1,32 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import tar from 'tar'; +import { bufferToStream, streamToBuffer } from './streams'; + +export interface ArchiveEntry { + path: string; + buffer?: Buffer; +} + +export async function untarBuffer( + buffer: Buffer, + filter = (entry: ArchiveEntry): boolean => true, + onEntry = (entry: ArchiveEntry): void => {} +): Promise { + const deflatedStream = bufferToStream(buffer); + // use tar.list vs .extract to avoid writing to disk + const inflateStream = tar.list().on('entry', (entry: tar.FileStat) => { + const path = entry.header.path || ''; + if (!filter({ path })) return; + streamToBuffer(entry).then(entryBuffer => onEntry({ buffer: entryBuffer, path })); + }); + + return new Promise((resolve, reject) => { + inflateStream.on('end', resolve).on('error', reject); + deflatedStream.pipe(inflateStream); + }); +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/registry/index.test.ts b/x-pack/plugins/ingest_manager/server/services/epm/registry/index.test.ts new file mode 100644 index 00000000000000..eae84275a49b91 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/registry/index.test.ts @@ -0,0 +1,50 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { AssetParts } from '../../../types'; +import { pathParts } from './index'; + +const testPaths = [ + { + path: 'foo-1.1.0/service/type/file.yml', + assetParts: { + dataset: undefined, + file: 'file.yml', + path: 'foo-1.1.0/service/type/file.yml', + pkgkey: 'foo-1.1.0', + service: 'service', + type: 'type', + }, + }, + { + path: 'iptables-1.0.4/kibana/visualization/683402b0-1f29-11e9-8ec4-cf5d91a864b3-ecs.json', + assetParts: { + dataset: undefined, + file: '683402b0-1f29-11e9-8ec4-cf5d91a864b3-ecs.json', + path: 'iptables-1.0.4/kibana/visualization/683402b0-1f29-11e9-8ec4-cf5d91a864b3-ecs.json', + pkgkey: 'iptables-1.0.4', + service: 'kibana', + type: 'visualization', + }, + }, + { + path: 'coredns-1.0.1/dataset/stats/fields/coredns.stats.yml', + assetParts: { + dataset: 'stats', + file: 'coredns.stats.yml', + path: 'coredns-1.0.1/dataset/stats/fields/coredns.stats.yml', + pkgkey: 'coredns-1.0.1', + service: '', + type: 'fields', + }, + }, +]; + +test('testPathParts', () => { + for (const value of testPaths) { + expect(pathParts(value.path)).toStrictEqual(value.assetParts as AssetParts); + } +}); diff --git a/x-pack/plugins/ingest_manager/server/services/epm/registry/index.ts b/x-pack/plugins/ingest_manager/server/services/epm/registry/index.ts new file mode 100644 index 00000000000000..ba4b3135aac1de --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/registry/index.ts @@ -0,0 +1,180 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { Response } from 'node-fetch'; +import { URL } from 'url'; +import { + AssetParts, + AssetsGroupedByServiceByType, + CategoryId, + CategorySummaryList, + KibanaAssetType, + RegistryPackage, + RegistrySearchResults, + RegistrySearchResult, +} from '../../../types'; +import { appContextService } from '../../'; +import { cacheGet, cacheSet } from './cache'; +import { ArchiveEntry, untarBuffer } from './extract'; +import { fetchUrl, getResponse, getResponseStream } from './requests'; +import { streamToBuffer } from './streams'; + +export { ArchiveEntry } from './extract'; + +export interface SearchParams { + category?: CategoryId; +} + +export const pkgToPkgKey = ({ name, version }: { name: string; version: string }) => + `${name}-${version}`; + +export async function fetchList(params?: SearchParams): Promise { + const registryUrl = appContextService.getConfig()?.epm.registryUrl; + const url = new URL(`${registryUrl}/search`); + if (params && params.category) { + url.searchParams.set('category', params.category); + } + + return fetchUrl(url.toString()).then(JSON.parse); +} + +export async function fetchFindLatestPackage( + packageName: string, + internal: boolean = true +): Promise { + const registryUrl = appContextService.getConfig()?.epm.registryUrl; + const url = new URL(`${registryUrl}/search?package=${packageName}&internal=${internal}`); + const res = await fetchUrl(url.toString()); + const searchResults = JSON.parse(res); + if (searchResults.length) { + return searchResults[0]; + } else { + throw new Error('package not found'); + } +} + +export async function fetchInfo(pkgkey: string): Promise { + const registryUrl = appContextService.getConfig()?.epm.registryUrl; + return fetchUrl(`${registryUrl}/package/${pkgkey}`).then(JSON.parse); +} + +export async function fetchFile(filePath: string): Promise { + const registryUrl = appContextService.getConfig()?.epm.registryUrl; + return getResponse(`${registryUrl}${filePath}`); +} + +export async function fetchCategories(): Promise { + const registryUrl = appContextService.getConfig()?.epm.registryUrl; + return fetchUrl(`${registryUrl}/categories`).then(JSON.parse); +} + +export async function getArchiveInfo( + pkgkey: string, + filter = (entry: ArchiveEntry): boolean => true +): Promise { + const paths: string[] = []; + const onEntry = (entry: ArchiveEntry) => { + const { path, buffer } = entry; + const { file } = pathParts(path); + if (!file) return; + if (buffer) { + cacheSet(path, buffer); + paths.push(path); + } + }; + + await extract(pkgkey, filter, onEntry); + + return paths; +} + +export function pathParts(path: string): AssetParts { + let dataset; + + let [pkgkey, service, type, file] = path.split('/'); + + // if it's a dataset + if (service === 'dataset') { + // save the dataset name + dataset = type; + // drop the `dataset/dataset-name` portion & re-parse + [pkgkey, service, type, file] = path.replace(`dataset/${dataset}/`, '').split('/'); + } + + // This is to cover for the fields.yml files inside the "fields" directory + if (file === undefined) { + file = type; + type = 'fields'; + service = ''; + } + + return { + pkgkey, + service, + type, + file, + dataset, + path, + } as AssetParts; +} + +async function extract( + pkgkey: string, + filter = (entry: ArchiveEntry): boolean => true, + onEntry: (entry: ArchiveEntry) => void +) { + const archiveBuffer = await getOrFetchArchiveBuffer(pkgkey); + + return untarBuffer(archiveBuffer, filter, onEntry); +} + +async function getOrFetchArchiveBuffer(pkgkey: string): Promise { + // assume .tar.gz for now. add support for .zip if/when we need it + const key = `${pkgkey}.tar.gz`; + let buffer = cacheGet(key); + if (!buffer) { + buffer = await fetchArchiveBuffer(pkgkey); + cacheSet(key, buffer); + } + + if (buffer) { + return buffer; + } else { + throw new Error(`no archive buffer for ${key}`); + } +} + +async function fetchArchiveBuffer(key: string): Promise { + const { download: archivePath } = await fetchInfo(key); + const registryUrl = appContextService.getConfig()?.epm.registryUrl; + return getResponseStream(`${registryUrl}${archivePath}`).then(streamToBuffer); +} + +export function getAsset(key: string) { + const buffer = cacheGet(key); + if (buffer === undefined) throw new Error(`Cannot find asset ${key}`); + + return buffer; +} + +export function groupPathsByService(paths: string[]): AssetsGroupedByServiceByType { + // ASK: best way, if any, to avoid `any`? + const assets = paths.reduce((map: any, path) => { + const parts = pathParts(path.replace(/^\/package\//, '')); + if (parts.type in KibanaAssetType) { + if (!map[parts.service]) map[parts.service] = {}; + if (!map[parts.service][parts.type]) map[parts.service][parts.type] = []; + map[parts.service][parts.type].push(parts); + } + + return map; + }, {}); + + return { + kibana: assets.kibana, + // elasticsearch: assets.elasticsearch, + }; +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts b/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts new file mode 100644 index 00000000000000..654aa8aae13559 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts @@ -0,0 +1,31 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import Boom from 'boom'; +import fetch, { Response } from 'node-fetch'; +import { streamToString } from './streams'; + +export async function getResponse(url: string): Promise { + try { + const response = await fetch(url); + if (response.ok) { + return response; + } else { + throw new Boom(response.statusText, { statusCode: response.status }); + } + } catch (e) { + throw Boom.boomify(e); + } +} + +export async function getResponseStream(url: string): Promise { + const res = await getResponse(url); + return res.body; +} + +export async function fetchUrl(url: string): Promise { + return getResponseStream(url).then(streamToString); +} diff --git a/x-pack/plugins/ingest_manager/server/services/epm/registry/streams.ts b/x-pack/plugins/ingest_manager/server/services/epm/registry/streams.ts new file mode 100644 index 00000000000000..e174c5f2e4d720 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/epm/registry/streams.ts @@ -0,0 +1,30 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { PassThrough } from 'stream'; + +export function bufferToStream(buffer: Buffer): PassThrough { + const stream = new PassThrough(); + stream.end(buffer); + return stream; +} + +export function streamToString(stream: NodeJS.ReadableStream): Promise { + return new Promise((resolve, reject) => { + const body: string[] = []; + stream.on('data', (chunk: string) => body.push(chunk)); + stream.on('end', () => resolve(body.join(''))); + stream.on('error', reject); + }); +} + +export function streamToBuffer(stream: NodeJS.ReadableStream): Promise { + return new Promise((resolve, reject) => { + const chunks: Buffer[] = []; + stream.on('data', chunk => chunks.push(Buffer.from(chunk))); + stream.on('end', () => resolve(Buffer.concat(chunks))); + stream.on('error', reject); + }); +} diff --git a/x-pack/plugins/ingest_manager/server/services/install_script/index.ts b/x-pack/plugins/ingest_manager/server/services/install_script/index.ts new file mode 100644 index 00000000000000..7e7f8d2a3734b0 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/install_script/index.ts @@ -0,0 +1,18 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { macosInstallTemplate } from './install_templates/macos'; + +export function getScript(osType: 'macos', kibanaUrl: string): string { + const variables = { kibanaUrl }; + + switch (osType) { + case 'macos': + return macosInstallTemplate(variables); + default: + throw new Error(`${osType} is not supported.`); + } +} diff --git a/x-pack/plugins/ingest_manager/server/services/install_script/install_templates/macos.ts b/x-pack/plugins/ingest_manager/server/services/install_script/install_templates/macos.ts new file mode 100644 index 00000000000000..e59dc6174b40ff --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/install_script/install_templates/macos.ts @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { resolve } from 'path'; +import { InstallTemplateFunction } from './types'; + +const PROJECT_ROOT = resolve(__dirname, '../../../../'); +export const macosInstallTemplate: InstallTemplateFunction = variables => `#!/bin/sh + +eval "node ${PROJECT_ROOT}/scripts/dev_agent --enrollmentApiKey=$API_KEY --kibanaUrl=${variables.kibanaUrl}" + +`; diff --git a/x-pack/plugins/ingest_manager/server/services/install_script/install_templates/types.ts b/x-pack/plugins/ingest_manager/server/services/install_script/install_templates/types.ts new file mode 100644 index 00000000000000..a478beaa96cfca --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/install_script/install_templates/types.ts @@ -0,0 +1,7 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export type InstallTemplateFunction = (variables: { kibanaUrl: string }) => string; diff --git a/x-pack/plugins/ingest_manager/server/services/license.ts b/x-pack/plugins/ingest_manager/server/services/license.ts new file mode 100644 index 00000000000000..bd96dbc7e3affb --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/license.ts @@ -0,0 +1,38 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { Observable, Subscription } from 'rxjs'; +import { ILicense } from '../../../licensing/server'; + +class LicenseService { + private observable: Observable | null = null; + private subscription: Subscription | null = null; + private licenseInformation: ILicense | null = null; + + private updateInformation(licenseInformation: ILicense) { + this.licenseInformation = licenseInformation; + } + + public start(license$: Observable) { + this.observable = license$; + this.subscription = this.observable.subscribe(this.updateInformation.bind(this)); + } + + public stop() { + if (this.subscription) { + this.subscription.unsubscribe(); + } + } + + public getLicenseInformation() { + return this.licenseInformation; + } + + public getLicenseInformation$() { + return this.observable; + } +} + +export const licenseService = new LicenseService(); diff --git a/x-pack/plugins/ingest_manager/server/services/output.ts b/x-pack/plugins/ingest_manager/server/services/output.ts index 8f60ed295205d9..066f8e8a316a58 100644 --- a/x-pack/plugins/ingest_manager/server/services/output.ts +++ b/x-pack/plugins/ingest_manager/server/services/output.ts @@ -3,47 +3,66 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { SavedObjectsClientContract, KibanaRequest } from 'kibana/server'; +import { SavedObjectsClientContract } from 'kibana/server'; import { NewOutput, Output } from '../types'; -import { DEFAULT_OUTPUT, DEFAULT_OUTPUT_ID, OUTPUT_SAVED_OBJECT_TYPE } from '../constants'; +import { DEFAULT_OUTPUT, OUTPUT_SAVED_OBJECT_TYPE } from '../constants'; import { appContextService } from './app_context'; const SAVED_OBJECT_TYPE = OUTPUT_SAVED_OBJECT_TYPE; class OutputService { - public async createDefaultOutput( - soClient: SavedObjectsClientContract, - adminUser: { username: string; password: string } - ) { - let defaultOutput; - - try { - defaultOutput = await this.get(soClient, DEFAULT_OUTPUT_ID); - } catch (err) { - if (!err.isBoom || err.output.statusCode !== 404) { - throw err; - } - } + public async ensureDefaultOutput(soClient: SavedObjectsClientContract) { + const outputs = await soClient.find({ + type: OUTPUT_SAVED_OBJECT_TYPE, + filter: 'outputs.attributes.is_default:true', + }); - if (!defaultOutput) { + if (!outputs.saved_objects.length) { const newDefaultOutput = { ...DEFAULT_OUTPUT, - hosts: [appContextService.getConfig()!.fleet.defaultOutputHost], - api_key: await this.createDefaultOutputApiKey(adminUser.username, adminUser.password), - admin_username: adminUser.username, - admin_password: adminUser.password, + hosts: [appContextService.getConfig()!.fleet.elasticsearch.host], + ca_sha256: appContextService.getConfig()!.fleet.elasticsearch.ca_sha256, } as NewOutput; - await this.create(soClient, newDefaultOutput, { - id: DEFAULT_OUTPUT_ID, - }); + return await this.create(soClient, newDefaultOutput); } + + return { + id: outputs.saved_objects[0].id, + ...outputs.saved_objects[0].attributes, + }; } - public async getAdminUser() { + public async updateOutput( + soClient: SavedObjectsClientContract, + id: string, + data: Partial + ) { + await soClient.update(SAVED_OBJECT_TYPE, id, data); + } + + public async getDefaultOutputId(soClient: SavedObjectsClientContract) { + const outputs = await soClient.find({ + type: OUTPUT_SAVED_OBJECT_TYPE, + filter: 'outputs.attributes.is_default:true', + }); + + if (!outputs.saved_objects.length) { + throw new Error('No default output'); + } + + return outputs.saved_objects[0].id; + } + + public async getAdminUser(soClient: SavedObjectsClientContract) { + const defaultOutputId = await this.getDefaultOutputId(soClient); const so = await appContextService .getEncryptedSavedObjects() - ?.getDecryptedAsInternalUser(OUTPUT_SAVED_OBJECT_TYPE, DEFAULT_OUTPUT_ID); + ?.getDecryptedAsInternalUser(OUTPUT_SAVED_OBJECT_TYPE, defaultOutputId); + + if (!so || !so.attributes.admin_username || !so.attributes.admin_password) { + return null; + } return { username: so!.attributes.admin_username, @@ -51,35 +70,6 @@ class OutputService { }; } - // TODO: TEMPORARY this is going to be per agent - private async createDefaultOutputApiKey(username: string, password: string): Promise { - const key = await appContextService.getSecurity()?.authc.createAPIKey( - { - headers: { - authorization: `Basic ${Buffer.from(`${username}:${password}`).toString('base64')}`, - }, - } as KibanaRequest, - { - name: 'fleet-default-output', - role_descriptors: { - 'fleet-output': { - cluster: ['monitor'], - index: [ - { - names: ['logs-*', 'metrics-*'], - privileges: ['write'], - }, - ], - }, - }, - } - ); - if (!key) { - throw new Error('An error occured while creating default API Key'); - } - return `${key.id}:${key.api_key}`; - } - public async create( soClient: SavedObjectsClientContract, output: NewOutput, @@ -93,11 +83,8 @@ class OutputService { }; } - public async get(soClient: SavedObjectsClientContract, id: string): Promise { + public async get(soClient: SavedObjectsClientContract, id: string): Promise { const outputSO = await soClient.get(SAVED_OBJECT_TYPE, id); - if (!outputSO) { - return null; - } if (outputSO.error) { throw new Error(outputSO.error.message); diff --git a/x-pack/plugins/ingest_manager/server/services/setup.ts b/x-pack/plugins/ingest_manager/server/services/setup.ts new file mode 100644 index 00000000000000..f770e9d17ffb15 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/services/setup.ts @@ -0,0 +1,87 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { SavedObjectsClientContract } from 'kibana/server'; +import { CallESAsCurrentUser } from '../types'; +import { agentConfigService } from './agent_config'; +import { outputService } from './output'; +import { ensureInstalledDefaultPackages } from './epm/packages/install'; +import { + packageToConfigDatasourceInputs, + Datasource, + AgentConfig, + Installation, + Output, + DEFAULT_AGENT_CONFIGS_PACKAGES, +} from '../../common'; +import { getPackageInfo } from './epm/packages'; +import { datasourceService } from './datasource'; + +export async function setup( + soClient: SavedObjectsClientContract, + callCluster: CallESAsCurrentUser +) { + const [installedPackages, defaultOutput, config] = await Promise.all([ + // packages installed by default + ensureInstalledDefaultPackages(soClient, callCluster), + outputService.ensureDefaultOutput(soClient), + agentConfigService.ensureDefaultAgentConfig(soClient), + ]); + + // ensure default packages are added to the default conifg + const configWithDatasource = await agentConfigService.get(soClient, config.id, true); + if (!configWithDatasource) { + throw new Error('Config not found'); + } + if ( + configWithDatasource.datasources.length && + typeof configWithDatasource.datasources[0] === 'string' + ) { + throw new Error('Config not found'); + } + for (const installedPackage of installedPackages) { + const packageShouldBeInstalled = DEFAULT_AGENT_CONFIGS_PACKAGES.some( + packageName => installedPackage.name === packageName + ); + if (!packageShouldBeInstalled) { + continue; + } + + const isInstalled = configWithDatasource.datasources.some((d: Datasource | string) => { + return typeof d !== 'string' && d.package?.name === installedPackage.name; + }); + + if (!isInstalled) { + await addPackageToConfig(soClient, installedPackage, configWithDatasource, defaultOutput); + } + } +} + +async function addPackageToConfig( + soClient: SavedObjectsClientContract, + packageToInstall: Installation, + config: AgentConfig, + defaultOutput: Output +) { + const packageInfo = await getPackageInfo({ + savedObjectsClient: soClient, + pkgkey: `${packageToInstall.name}-${packageToInstall.version}`, + }); + const datasource = await datasourceService.create(soClient, { + name: `${packageInfo.name}-1`, + enabled: true, + package: { + name: packageInfo.name, + title: packageInfo.title, + version: packageInfo.version, + }, + inputs: packageToConfigDatasourceInputs(packageInfo), + config_id: config.id, + output_id: defaultOutput.id, + }); + // Assign it to the given agent config + await agentConfigService.assignDatasources(soClient, datasource.config_id, [datasource.id]); +} diff --git a/x-pack/plugins/ingest_manager/server/types/index.tsx b/x-pack/plugins/ingest_manager/server/types/index.tsx index f44d03923d4244..c9a4bf79f35165 100644 --- a/x-pack/plugins/ingest_manager/server/types/index.tsx +++ b/x-pack/plugins/ingest_manager/server/types/index.tsx @@ -3,10 +3,57 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -export * from './models'; -export * from './rest_spec'; +import { ScopedClusterClient } from 'src/core/server/'; + +export { + // Object types + Agent, + AgentSOAttributes, + AgentStatus, + AgentType, + AgentEvent, + AgentEventSOAttributes, + AgentAction, + Datasource, + NewDatasource, + FullAgentConfigDatasource, + FullAgentConfig, + AgentConfig, + NewAgentConfig, + AgentConfigStatus, + Output, + NewOutput, + OutputType, + EnrollmentAPIKey, + EnrollmentAPIKeySOAttributes, + Installation, + InstallationStatus, + PackageInfo, + RegistryVarsEntry, + Dataset, + AssetReference, + ElasticsearchAssetType, + IngestAssetType, + RegistryPackage, + AssetType, + Installable, + KibanaAssetType, + AssetParts, + AssetsGroupedByServiceByType, + CategoryId, + CategorySummaryList, + IndexTemplate, + RegistrySearchResults, + RegistrySearchResult, + DefaultPackages, +} from '../../common'; + +export type CallESAsCurrentUser = ScopedClusterClient['callAsCurrentUser']; export type AgentConfigUpdateHandler = ( action: 'created' | 'updated' | 'deleted', agentConfigId: string ) => Promise; + +export * from './models'; +export * from './rest_spec'; diff --git a/x-pack/plugins/ingest_manager/server/types/models/agent.ts b/x-pack/plugins/ingest_manager/server/types/models/agent.ts new file mode 100644 index 00000000000000..276dddf9e3d1c5 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/types/models/agent.ts @@ -0,0 +1,49 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { schema } from '@kbn/config-schema'; + +import { AGENT_TYPE_EPHEMERAL, AGENT_TYPE_PERMANENT, AGENT_TYPE_TEMPORARY } from '../../../common'; + +export const AgentTypeSchema = schema.oneOf([ + schema.literal(AGENT_TYPE_EPHEMERAL), + schema.literal(AGENT_TYPE_PERMANENT), + schema.literal(AGENT_TYPE_TEMPORARY), +]); + +const AgentEventBase = { + type: schema.oneOf([ + schema.literal('STATE'), + schema.literal('ERROR'), + schema.literal('ACTION_RESULT'), + schema.literal('ACTION'), + ]), + subtype: schema.oneOf([ + // State + schema.literal('RUNNING'), + schema.literal('STARTING'), + schema.literal('IN_PROGRESS'), + schema.literal('CONFIG'), + schema.literal('FAILED'), + schema.literal('STOPPING'), + schema.literal('STOPPED'), + // Action results + schema.literal('DATA_DUMP'), + // Actions + schema.literal('ACKNOWLEDGED'), + schema.literal('UNKNOWN'), + ]), + timestamp: schema.string(), + message: schema.string(), + payload: schema.maybe(schema.any()), + agent_id: schema.string(), + action_id: schema.maybe(schema.string()), + config_id: schema.maybe(schema.string()), + stream_id: schema.maybe(schema.string()), +}; + +export const AgentEventSchema = schema.object({ + ...AgentEventBase, +}); diff --git a/x-pack/plugins/ingest_manager/server/types/models/agent_config.ts b/x-pack/plugins/ingest_manager/server/types/models/agent_config.ts index d3acb0c167837b..040b2eb16289ab 100644 --- a/x-pack/plugins/ingest_manager/server/types/models/agent_config.ts +++ b/x-pack/plugins/ingest_manager/server/types/models/agent_config.ts @@ -3,17 +3,13 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { schema, TypeOf } from '@kbn/config-schema'; +import { schema } from '@kbn/config-schema'; import { DatasourceSchema } from './datasource'; - -export enum AgentConfigStatus { - Active = 'active', - Inactive = 'inactive', -} +import { AgentConfigStatus } from '../../../common'; const AgentConfigBaseSchema = { name: schema.string(), - namespace: schema.string(), + namespace: schema.maybe(schema.string()), description: schema.maybe(schema.string()), }; @@ -32,7 +28,3 @@ export const AgentConfigSchema = schema.object({ updated_on: schema.string(), updated_by: schema.string(), }); - -export type NewAgentConfig = TypeOf; - -export type AgentConfig = TypeOf; diff --git a/x-pack/plugins/ingest_manager/server/types/models/datasource.ts b/x-pack/plugins/ingest_manager/server/types/models/datasource.ts index 4179d4c3a511a2..94d0a1cc1aabf7 100644 --- a/x-pack/plugins/ingest_manager/server/types/models/datasource.ts +++ b/x-pack/plugins/ingest_manager/server/types/models/datasource.ts @@ -3,40 +3,37 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { schema, TypeOf } from '@kbn/config-schema'; +import { schema } from '@kbn/config-schema'; +export { Datasource, NewDatasource } from '../../../common'; const DatasourceBaseSchema = { name: schema.string(), + description: schema.maybe(schema.string()), namespace: schema.maybe(schema.string()), - read_alias: schema.maybe(schema.string()), - agent_config_id: schema.string(), + config_id: schema.string(), + enabled: schema.boolean(), package: schema.maybe( schema.object({ - assets: schema.arrayOf( - schema.object({ - id: schema.string(), - type: schema.string(), - }) - ), - description: schema.string(), name: schema.string(), title: schema.string(), version: schema.string(), }) ), - streams: schema.arrayOf( + output_id: schema.string(), + inputs: schema.arrayOf( schema.object({ - config: schema.recordOf(schema.string(), schema.any()), - input: schema.object({ - type: schema.string(), - config: schema.recordOf(schema.string(), schema.any()), - fields: schema.maybe(schema.arrayOf(schema.recordOf(schema.string(), schema.any()))), - ilm_policy: schema.maybe(schema.string()), - index_template: schema.maybe(schema.string()), - ingest_pipelines: schema.maybe(schema.arrayOf(schema.string())), - }), - output_id: schema.string(), + type: schema.string(), + enabled: schema.boolean(), processors: schema.maybe(schema.arrayOf(schema.string())), + streams: schema.arrayOf( + schema.object({ + id: schema.string(), + enabled: schema.boolean(), + dataset: schema.string(), + processors: schema.maybe(schema.arrayOf(schema.string())), + config: schema.recordOf(schema.string(), schema.any()), + }) + ), }) ), }; @@ -49,7 +46,3 @@ export const DatasourceSchema = schema.object({ ...DatasourceBaseSchema, id: schema.string(), }); - -export type NewDatasource = TypeOf; - -export type Datasource = TypeOf; diff --git a/x-pack/plugins/ingest_manager/server/types/models/enrollment_api_key.ts b/x-pack/plugins/ingest_manager/server/types/models/enrollment_api_key.ts new file mode 100644 index 00000000000000..e563b39e53f507 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/types/models/enrollment_api_key.ts @@ -0,0 +1,25 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { schema } from '@kbn/config-schema'; + +export const EnrollmentAPIKeySchema = schema.object({ + id: schema.string(), + api_key_id: schema.string(), + api_key: schema.string(), + name: schema.maybe(schema.string()), + active: schema.boolean(), + config_id: schema.maybe(schema.string()), +}); + +export const EnrollmentAPIKeySOAttributesSchema = schema.object({ + api_key_id: schema.string(), + api_key: schema.string(), + name: schema.maybe(schema.string()), + active: schema.boolean(), + config_id: schema.maybe(schema.string()), + // ASK: Is this allowUnknown? How do we type this with config-schema? + // [k: string]: schema.any(), // allow to use it as saved object attributes type +}); diff --git a/x-pack/plugins/ingest_manager/server/types/models/index.ts b/x-pack/plugins/ingest_manager/server/types/models/index.ts index 959dfe1d937b96..7da36c8a18ad22 100644 --- a/x-pack/plugins/ingest_manager/server/types/models/index.ts +++ b/x-pack/plugins/ingest_manager/server/types/models/index.ts @@ -4,5 +4,7 @@ * you may not use this file except in compliance with the Elastic License. */ export * from './agent_config'; +export * from './agent'; export * from './datasource'; export * from './output'; +export * from './enrollment_api_key'; diff --git a/x-pack/plugins/ingest_manager/server/types/models/output.ts b/x-pack/plugins/ingest_manager/server/types/models/output.ts index 610fa6796cc2d9..8c8f4c76af7feb 100644 --- a/x-pack/plugins/ingest_manager/server/types/models/output.ts +++ b/x-pack/plugins/ingest_manager/server/types/models/output.ts @@ -3,7 +3,8 @@ * or more contributor license agreements. Licensed under the Elastic License; * you may not use this file except in compliance with the Elastic License. */ -import { schema, TypeOf } from '@kbn/config-schema'; +import { schema } from '@kbn/config-schema'; +export { Output, NewOutput } from '../../../common'; export enum OutputType { Elasticsearch = 'elasticsearch', @@ -12,10 +13,6 @@ export enum OutputType { const OutputBaseSchema = { name: schema.string(), type: schema.oneOf([schema.literal(OutputType.Elasticsearch)]), - username: schema.maybe(schema.string()), - password: schema.maybe(schema.string()), - index_name: schema.maybe(schema.string()), - ingest_pipeline: schema.maybe(schema.string()), hosts: schema.maybe(schema.arrayOf(schema.string())), api_key: schema.maybe(schema.string()), admin_username: schema.maybe(schema.string()), @@ -31,7 +28,3 @@ export const OutputSchema = schema.object({ ...OutputBaseSchema, id: schema.string(), }); - -export type NewOutput = TypeOf; - -export type Output = TypeOf; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/agent.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/agent.ts new file mode 100644 index 00000000000000..92422274d5cf46 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/agent.ts @@ -0,0 +1,96 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { schema } from '@kbn/config-schema'; +import { AgentEventSchema, AgentTypeSchema } from '../models'; + +export const GetAgentsRequestSchema = { + query: schema.object({ + page: schema.number({ defaultValue: 1 }), + perPage: schema.number({ defaultValue: 20 }), + kuery: schema.maybe(schema.string()), + showInactive: schema.boolean({ defaultValue: false }), + }), +}; + +export const GetOneAgentRequestSchema = { + params: schema.object({ + agentId: schema.string(), + }), +}; + +export const PostAgentCheckinRequestSchema = { + params: schema.object({ + agentId: schema.string(), + }), + body: schema.object({ + local_metadata: schema.maybe(schema.recordOf(schema.string(), schema.any())), + events: schema.maybe(schema.arrayOf(AgentEventSchema)), + }), +}; + +export const PostAgentEnrollRequestSchema = { + body: schema.object({ + type: AgentTypeSchema, + shared_id: schema.maybe(schema.string()), + metadata: schema.object({ + local: schema.recordOf(schema.string(), schema.any()), + user_provided: schema.recordOf(schema.string(), schema.any()), + }), + }), +}; + +export const PostAgentAcksRequestSchema = { + body: schema.object({ + action_ids: schema.arrayOf(schema.string()), + }), + params: schema.object({ + agentId: schema.string(), + }), +}; + +export const PostAgentUnenrollRequestSchema = { + body: schema.oneOf([ + schema.object({ + kuery: schema.string(), + }), + schema.object({ + ids: schema.arrayOf(schema.string()), + }), + ]), +}; + +export const GetOneAgentEventsRequestSchema = { + params: schema.object({ + agentId: schema.string(), + }), + query: schema.object({ + page: schema.number({ defaultValue: 1 }), + perPage: schema.number({ defaultValue: 20 }), + kuery: schema.maybe(schema.string()), + }), +}; + +export const DeleteAgentRequestSchema = { + params: schema.object({ + agentId: schema.string(), + }), +}; + +export const UpdateAgentRequestSchema = { + params: schema.object({ + agentId: schema.string(), + }), + body: schema.object({ + user_provided_metadata: schema.recordOf(schema.string(), schema.any()), + }), +}; + +export const GetAgentStatusRequestSchema = { + query: schema.object({ + configId: schema.maybe(schema.string()), + }), +}; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/agent_config.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/agent_config.ts index cb4680a4eed04c..7c40cc1b70009d 100644 --- a/x-pack/plugins/ingest_manager/server/types/rest_spec/agent_config.ts +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/agent_config.ts @@ -4,58 +4,36 @@ * you may not use this file except in compliance with the Elastic License. */ import { schema } from '@kbn/config-schema'; -import { AgentConfig, NewAgentConfigSchema } from '../models'; -import { ListWithKuerySchema } from './common'; +import { NewAgentConfigSchema } from '../models'; +import { ListWithKuerySchema } from './index'; export const GetAgentConfigsRequestSchema = { query: ListWithKuerySchema, }; -export interface GetAgentConfigsResponse { - items: AgentConfig[]; - total: number; - page: number; - perPage: number; - success: boolean; -} - export const GetOneAgentConfigRequestSchema = { params: schema.object({ agentConfigId: schema.string(), }), }; -export interface GetOneAgentConfigResponse { - item: AgentConfig; - success: boolean; -} - export const CreateAgentConfigRequestSchema = { body: NewAgentConfigSchema, }; -export interface CreateAgentConfigResponse { - item: AgentConfig; - success: boolean; -} - export const UpdateAgentConfigRequestSchema = { ...GetOneAgentConfigRequestSchema, body: NewAgentConfigSchema, }; -export interface UpdateAgentConfigResponse { - item: AgentConfig; - success: boolean; -} - export const DeleteAgentConfigsRequestSchema = { body: schema.object({ agentConfigIds: schema.arrayOf(schema.string()), }), }; -export type DeleteAgentConfigsResponse = Array<{ - id: string; - success: boolean; -}>; +export const GetFullAgentConfigRequestSchema = { + params: schema.object({ + agentConfigId: schema.string(), + }), +}; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/datasource.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/datasource.ts index 5c165517e9c9d2..fce2c94b282bd4 100644 --- a/x-pack/plugins/ingest_manager/server/types/rest_spec/datasource.ts +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/datasource.ts @@ -5,7 +5,7 @@ */ import { schema } from '@kbn/config-schema'; import { NewDatasourceSchema } from '../models'; -import { ListWithKuerySchema } from './common'; +import { ListWithKuerySchema } from './index'; export const GetDatasourcesRequestSchema = { query: ListWithKuerySchema, @@ -31,8 +31,3 @@ export const DeleteDatasourcesRequestSchema = { datasourceIds: schema.arrayOf(schema.string()), }), }; - -export type DeleteDatasourcesResponse = Array<{ - id: string; - success: boolean; -}>; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/enrollment_api_key.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/enrollment_api_key.ts new file mode 100644 index 00000000000000..ff342bd1657702 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/enrollment_api_key.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { schema } from '@kbn/config-schema'; + +export const GetEnrollmentAPIKeysRequestSchema = { + query: schema.object({ + page: schema.number({ defaultValue: 1 }), + perPage: schema.number({ defaultValue: 20 }), + kuery: schema.maybe(schema.string()), + }), +}; + +export const GetOneEnrollmentAPIKeyRequestSchema = { + params: schema.object({ + keyId: schema.string(), + }), +}; + +export const DeleteEnrollmentAPIKeyRequestSchema = { + params: schema.object({ + keyId: schema.string(), + }), +}; + +export const PostEnrollmentAPIKeyRequestSchema = { + body: schema.object({ + name: schema.maybe(schema.string()), + config_id: schema.string(), + expiration: schema.maybe(schema.string()), + }), +}; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/epm.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/epm.ts new file mode 100644 index 00000000000000..2ca83276b02289 --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/epm.ts @@ -0,0 +1,37 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ +import { schema } from '@kbn/config-schema'; + +export const GetPackagesRequestSchema = { + query: schema.object({ + category: schema.maybe(schema.string()), + }), +}; + +export const GetFileRequestSchema = { + params: schema.object({ + pkgkey: schema.string(), + filePath: schema.string(), + }), +}; + +export const GetInfoRequestSchema = { + params: schema.object({ + pkgkey: schema.string(), + }), +}; + +export const InstallPackageRequestSchema = { + params: schema.object({ + pkgkey: schema.string(), + }), +}; + +export const DeletePackageRequestSchema = { + params: schema.object({ + pkgkey: schema.string(), + }), +}; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/index.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/index.ts index 7d0d7e67f2db01..c143cd3b35f91a 100644 --- a/x-pack/plugins/ingest_manager/server/types/rest_spec/index.ts +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/index.ts @@ -4,6 +4,10 @@ * you may not use this file except in compliance with the Elastic License. */ export * from './common'; -export * from './datasource'; export * from './agent_config'; +export * from './agent'; +export * from './datasource'; +export * from './epm'; +export * from './enrollment_api_key'; export * from './fleet_setup'; +export * from './install_script'; diff --git a/x-pack/plugins/ingest_manager/server/types/rest_spec/install_script.ts b/x-pack/plugins/ingest_manager/server/types/rest_spec/install_script.ts new file mode 100644 index 00000000000000..cf676129cce7ad --- /dev/null +++ b/x-pack/plugins/ingest_manager/server/types/rest_spec/install_script.ts @@ -0,0 +1,13 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { schema } from '@kbn/config-schema'; + +export const InstallScriptRequestSchema = { + params: schema.object({ + osType: schema.oneOf([schema.literal('macos')]), + }), +}; diff --git a/x-pack/plugins/ingest_manager/yarn.lock b/x-pack/plugins/ingest_manager/yarn.lock new file mode 120000 index 00000000000000..6e09764ec763b0 --- /dev/null +++ b/x-pack/plugins/ingest_manager/yarn.lock @@ -0,0 +1 @@ +../../../yarn.lock \ No newline at end of file diff --git a/x-pack/test/api_integration/apis/features/features/features.ts b/x-pack/test/api_integration/apis/features/features/features.ts index 34b2c5e3241874..7a0196adbfffd6 100644 --- a/x-pack/test/api_integration/apis/features/features/features.ts +++ b/x-pack/test/api_integration/apis/features/features/features.ts @@ -115,6 +115,7 @@ export default function({ getService }: FtrProviderContext) { 'uptime', 'siem', 'endpoint', + 'ingestManager', ].sort() ); }); diff --git a/x-pack/test/api_integration/apis/fleet/agents/acks.ts b/x-pack/test/api_integration/apis/fleet/agents/acks.ts new file mode 100644 index 00000000000000..1ab54554d62f0e --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/agents/acks.ts @@ -0,0 +1,79 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import uuid from 'uuid'; + +import { FtrProviderContext } from '../../../ftr_provider_context'; +import { getSupertestWithoutAuth } from './services'; + +export default function(providerContext: FtrProviderContext) { + const { getService } = providerContext; + const esArchiver = getService('esArchiver'); + const esClient = getService('es'); + + const supertest = getSupertestWithoutAuth(providerContext); + let apiKey: { id: string; api_key: string }; + + describe('fleet_agents_acks', () => { + before(async () => { + await esArchiver.loadIfNeeded('fleet/agents'); + + const { body: apiKeyBody } = await esClient.security.createApiKey({ + body: { + name: `test access api key: ${uuid.v4()}`, + }, + }); + apiKey = apiKeyBody; + const { + body: { _source: agentDoc }, + } = await esClient.get({ + index: '.kibana', + id: 'agents:agent1', + }); + agentDoc.agents.access_api_key_id = apiKey.id; + await esClient.update({ + index: '.kibana', + id: 'agents:agent1', + refresh: 'true', + body: { + doc: agentDoc, + }, + }); + }); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should return a 401 if this a not a valid acks access', async () => { + await supertest + .post(`/api/ingest_manager/fleet/agents/agent1/acks`) + .set('kbn-xsrf', 'xx') + .set('Authorization', 'ApiKey NOT_A_VALID_TOKEN') + .send({ + action_ids: [], + }) + .expect(401); + }); + + it('should return a 200 if this a valid acks access', async () => { + const { body: apiResponse } = await supertest + .post(`/api/ingest_manager/fleet/agents/agent1/acks`) + .set('kbn-xsrf', 'xx') + .set( + 'Authorization', + `ApiKey ${Buffer.from(`${apiKey.id}:${apiKey.api_key}`).toString('base64')}` + ) + .send({ + action_ids: ['action1'], + }) + .expect(200); + + expect(apiResponse.action).to.be('acks'); + expect(apiResponse.success).to.be(true); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/agents/checkin.ts b/x-pack/test/api_integration/apis/fleet/agents/checkin.ts new file mode 100644 index 00000000000000..ca51676126e73d --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/agents/checkin.ts @@ -0,0 +1,105 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import uuid from 'uuid'; + +import { FtrProviderContext } from '../../../ftr_provider_context'; +import { getSupertestWithoutAuth } from './services'; + +export default function(providerContext: FtrProviderContext) { + const { getService } = providerContext; + const esArchiver = getService('esArchiver'); + const esClient = getService('es'); + + const supertest = getSupertestWithoutAuth(providerContext); + let apiKey: { id: string; api_key: string }; + + describe('fleet_agents_checkin', () => { + before(async () => { + await esArchiver.loadIfNeeded('fleet/agents'); + + const { body: apiKeyBody } = await esClient.security.createApiKey({ + body: { + name: `test access api key: ${uuid.v4()}`, + }, + }); + apiKey = apiKeyBody; + const { + body: { _source: agentDoc }, + } = await esClient.get({ + index: '.kibana', + id: 'agents:agent1', + }); + agentDoc.agents.access_api_key_id = apiKey.id; + await esClient.update({ + index: '.kibana', + id: 'agents:agent1', + refresh: 'true', + body: { + doc: agentDoc, + }, + }); + }); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should return a 401 if this a not a valid checkin access', async () => { + await supertest + .post(`/api/ingest_manager/fleet/agents/agent1/checkin`) + .set('kbn-xsrf', 'xx') + .set('Authorization', 'ApiKey NOT_A_VALID_TOKEN') + .send({ + events: [], + }) + .expect(401); + }); + + it('should return a 400 if for a malformed request payload', async () => { + await supertest + .post(`/api/ingest_manager/fleet/agents/agent1/checkin`) + .set('kbn-xsrf', 'xx') + .set( + 'Authorization', + `ApiKey ${Buffer.from(`${apiKey.id}:${apiKey.api_key}`).toString('base64')}` + ) + .send({ + events: ['i-am-not-valid-event'], + metadata: {}, + }) + .expect(400); + }); + + it('should return a 200 if this a valid checkin access', async () => { + const { body: apiResponse } = await supertest + .post(`/api/ingest_manager/fleet/agents/agent1/checkin`) + .set('kbn-xsrf', 'xx') + .set( + 'Authorization', + `ApiKey ${Buffer.from(`${apiKey.id}:${apiKey.api_key}`).toString('base64')}` + ) + .send({ + events: [ + { + type: 'STATE', + timestamp: '2019-01-04T14:32:03.36764-05:00', + subtype: 'STARTING', + message: 'State change: STARTING', + agent_id: 'agent1', + }, + ], + local_metadata: { + cpu: 12, + }, + }) + .expect(200); + + expect(apiResponse.action).to.be('checkin'); + expect(apiResponse.success).to.be(true); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/agents/enroll.ts b/x-pack/test/api_integration/apis/fleet/agents/enroll.ts new file mode 100644 index 00000000000000..666d97452ad3d5 --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/agents/enroll.ts @@ -0,0 +1,108 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import uuid from 'uuid'; +import { FtrProviderContext } from '../../../ftr_provider_context'; +import { getSupertestWithoutAuth, setupIngest } from './services'; + +export default function(providerContext: FtrProviderContext) { + const { getService } = providerContext; + + const esArchiver = getService('esArchiver'); + const esClient = getService('es'); + + const supertest = getSupertestWithoutAuth(providerContext); + let apiKey: { id: string; api_key: string }; + + describe('fleet_agents_enroll', () => { + before(async () => { + await esArchiver.loadIfNeeded('fleet/agents'); + + const { body: apiKeyBody } = await esClient.security.createApiKey({ + body: { + name: `test access api key: ${uuid.v4()}`, + }, + }); + apiKey = apiKeyBody; + const { + body: { _source: enrollmentApiKeyDoc }, + } = await esClient.get({ + index: '.kibana', + id: 'enrollment_api_keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0', + }); + // @ts-ignore + enrollmentApiKeyDoc.enrollment_api_keys.api_key_id = apiKey.id; + await esClient.update({ + index: '.kibana', + id: 'enrollment_api_keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0', + refresh: 'true', + body: { + doc: enrollmentApiKeyDoc, + }, + }); + }); + setupIngest(providerContext); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should not allow to enroll an agent with a invalid enrollment', async () => { + await supertest + .post(`/api/ingest_manager/fleet/agents/enroll`) + .set('kbn-xsrf', 'xxx') + .set('Authorization', 'ApiKey NOTAVALIDKEY') + .send({ + type: 'PERMANENT', + metadata: { + local: {}, + user_provided: {}, + }, + }) + .expect(401); + }); + + it('should not allow to enroll an agent with a shared id if it already exists ', async () => { + const { body: apiResponse } = await supertest + .post(`/api/ingest_manager/fleet/agents/enroll`) + .set('kbn-xsrf', 'xxx') + .set( + 'authorization', + `ApiKey ${Buffer.from(`${apiKey.id}:${apiKey.api_key}`).toString('base64')}` + ) + .send({ + shared_id: 'agent2_filebeat', + type: 'PERMANENT', + metadata: { + local: {}, + user_provided: {}, + }, + }) + .expect(400); + expect(apiResponse.message).to.match(/Impossible to enroll an already active agent/); + }); + + it('should allow to enroll an agent with a valid enrollment token', async () => { + const { body: apiResponse } = await supertest + .post(`/api/ingest_manager/fleet/agents/enroll`) + .set('kbn-xsrf', 'xxx') + .set( + 'Authorization', + `ApiKey ${Buffer.from(`${apiKey.id}:${apiKey.api_key}`).toString('base64')}` + ) + .send({ + type: 'PERMANENT', + metadata: { + local: {}, + user_provided: {}, + }, + }) + .expect(200); + expect(apiResponse.success).to.eql(true); + expect(apiResponse.item).to.have.keys('id', 'active', 'access_api_key', 'type', 'config_id'); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/agents/events.ts b/x-pack/test/api_integration/apis/fleet/agents/events.ts new file mode 100644 index 00000000000000..ac5eb9d1779d89 --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/agents/events.ts @@ -0,0 +1,36 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; + +import { FtrProviderContext } from '../../../ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + const esArchiver = getService('esArchiver'); + const supertest = getService('supertest'); + + describe('fleet_agents_events', () => { + before(async () => { + await esArchiver.loadIfNeeded('fleet/agents'); + }); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should return a 200 and the events for a given agent', async () => { + const { body: apiResponse } = await supertest + .get(`/api/ingest_manager/fleet/agents/agent1/events`) + .expect(200); + expect(apiResponse).to.have.keys(['list', 'total', 'page']); + expect(apiResponse.total).to.be(2); + expect(apiResponse.page).to.be(1); + + const event = apiResponse.list[0]; + expect(event).to.have.keys('type', 'subtype', 'message', 'payload'); + expect(event.payload).to.have.keys('previous_state'); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/agents/services.ts b/x-pack/test/api_integration/apis/fleet/agents/services.ts new file mode 100644 index 00000000000000..5c111b8ea9a847 --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/agents/services.ts @@ -0,0 +1,35 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import supertestAsPromised from 'supertest-as-promised'; +import url from 'url'; + +import { FtrProviderContext } from '../../../ftr_provider_context'; + +export function getSupertestWithoutAuth({ getService }: FtrProviderContext) { + const config = getService('config'); + const kibanaUrl = config.get('servers.kibana'); + kibanaUrl.auth = null; + kibanaUrl.password = null; + + return supertestAsPromised(url.format(kibanaUrl)); +} + +export function setupIngest({ getService }: FtrProviderContext) { + before(async () => { + await getService('supertest') + .post(`/api/ingest_manager/setup`) + .set('kbn-xsrf', 'xxx') + .send(); + await getService('supertest') + .post(`/api/ingest_manager/fleet/setup`) + .set('kbn-xsrf', 'xxx') + .send({ + admin_username: 'elastic', + admin_password: 'changeme', + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/delete_agent.ts b/x-pack/test/api_integration/apis/fleet/delete_agent.ts new file mode 100644 index 00000000000000..bddfcfbf6715ec --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/delete_agent.ts @@ -0,0 +1,96 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import { FtrProviderContext } from '../../ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + const esArchiver = getService('esArchiver'); + const supertest = getService('supertestWithoutAuth'); + const security = getService('security'); + const users: { [rollName: string]: { username: string; password: string; permissions?: any } } = { + fleet_user: { + permissions: { + feature: { + ingestManager: ['read'], + }, + spaces: ['*'], + }, + username: 'fleet_user', + password: 'changeme', + }, + fleet_admin: { + permissions: { + feature: { + ingestManager: ['all'], + }, + spaces: ['*'], + }, + username: 'fleet_admin', + password: 'changeme', + }, + }; + describe('fleet_delete_agent', () => { + before(async () => { + for (const roleName in users) { + if (users.hasOwnProperty(roleName)) { + const user = users[roleName]; + + if (user.permissions) { + await security.role.create(roleName, { + kibana: [user.permissions], + }); + } + + // Import a repository first + await security.user.create(user.username, { + password: user.password, + roles: [roleName], + full_name: user.username, + }); + } + } + + await esArchiver.loadIfNeeded('fleet/agents'); + }); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should return a 404 if user lacks fleet-write permissions', async () => { + const { body: apiResponse } = await supertest + .delete(`/api/ingest_manager/fleet/agents/agent1`) + .auth(users.fleet_user.username, users.fleet_user.password) + .set('kbn-xsrf', 'xx') + .expect(404); + + expect(apiResponse).not.to.eql({ + success: true, + action: 'deleted', + }); + }); + + it('should return a 404 if there is no agent to delete', async () => { + await supertest + .delete(`/api/ingest_manager/fleet/agents/i-do-not-exist`) + .auth(users.fleet_admin.username, users.fleet_admin.password) + .set('kbn-xsrf', 'xx') + .expect(404); + }); + + it('should return a 200 after deleting an agent', async () => { + const { body: apiResponse } = await supertest + .delete(`/api/ingest_manager/fleet/agents/agent1`) + .auth(users.fleet_admin.username, users.fleet_admin.password) + .set('kbn-xsrf', 'xx') + .expect(200); + expect(apiResponse).to.eql({ + success: true, + action: 'deleted', + }); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/enrollment_api_keys/crud.ts b/x-pack/test/api_integration/apis/fleet/enrollment_api_keys/crud.ts new file mode 100644 index 00000000000000..800e0147528e50 --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/enrollment_api_keys/crud.ts @@ -0,0 +1,83 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; + +import { FtrProviderContext } from '../../../ftr_provider_context'; +import { setupIngest } from '../agents/services'; + +const ENROLLMENT_KEY_ID = 'ed22ca17-e178-4cfe-8b02-54ea29fbd6d0'; + +export default function({ getService }: FtrProviderContext) { + const esArchiver = getService('esArchiver'); + const supertest = getService('supertest'); + + describe('fleet_enrollment_api_keys_crud', () => { + before(async () => { + await esArchiver.loadIfNeeded('fleet/agents'); + }); + setupIngest({ getService } as FtrProviderContext); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + describe('GET /fleet/enrollment-api-keys', async () => { + it('should list existing api keys', async () => { + const { body: apiResponse } = await supertest + .get(`/api/ingest_manager/fleet/enrollment-api-keys`) + .expect(200); + + expect(apiResponse.total).to.be(2); + expect(apiResponse.list[0]).to.have.keys('id', 'api_key_id', 'name'); + }); + }); + + describe('GET /fleet/enrollment-api-keys/{id}', async () => { + it('should allow to retrieve existing api keys', async () => { + const { body: apiResponse } = await supertest + .get(`/api/ingest_manager/fleet/enrollment-api-keys/${ENROLLMENT_KEY_ID}`) + .expect(200); + + expect(apiResponse.item).to.have.keys('id', 'api_key_id', 'name'); + }); + }); + + describe('GET /fleet/enrollment-api-keys/{id}', async () => { + it('should allow to retrieve existing api keys', async () => { + const { body: apiResponse } = await supertest + .delete(`/api/ingest_manager/fleet/enrollment-api-keys/${ENROLLMENT_KEY_ID}`) + .set('kbn-xsrf', 'xxx') + .expect(200); + + expect(apiResponse.success).to.eql(true); + }); + }); + + describe('POST /fleet/enrollment-api-keys', () => { + it('should not accept bad parameters', async () => { + await supertest + .post(`/api/ingest_manager/fleet/enrollment-api-keys`) + .set('kbn-xsrf', 'xxx') + .send({ + raoul: 'raoul', + }) + .expect(400); + }); + + it('should allow to create an enrollment api key with a policy', async () => { + const { body: apiResponse } = await supertest + .post(`/api/ingest_manager/fleet/enrollment-api-keys`) + .set('kbn-xsrf', 'xxx') + .send({ + config_id: 'policy1', + }) + .expect(200); + + expect(apiResponse.success).to.eql(true); + expect(apiResponse.item).to.have.keys('id', 'api_key', 'api_key_id', 'name', 'config_id'); + }); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/index.js b/x-pack/test/api_integration/apis/fleet/index.js new file mode 100644 index 00000000000000..69d30291f030bf --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/index.js @@ -0,0 +1,19 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export default function loadTests({ loadTestFile }) { + describe('Fleet Endpoints', () => { + loadTestFile(require.resolve('./delete_agent')); + loadTestFile(require.resolve('./list_agent')); + loadTestFile(require.resolve('./unenroll_agent')); + loadTestFile(require.resolve('./agents/enroll')); + loadTestFile(require.resolve('./agents/checkin')); + loadTestFile(require.resolve('./agents/events')); + loadTestFile(require.resolve('./agents/acks')); + loadTestFile(require.resolve('./enrollment_api_keys/crud')); + loadTestFile(require.resolve('./install')); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/install.ts b/x-pack/test/api_integration/apis/fleet/install.ts new file mode 100644 index 00000000000000..f5d8a062271510 --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/install.ts @@ -0,0 +1,25 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import { FtrProviderContext } from '../../ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + const supertest = getService('supertest'); + + describe('fleet_install', () => { + it('should return a 400 if we try download an install script for a not supported OS', async () => { + await supertest.get(`/api/ingest_manager/fleet/install/gameboy`).expect(400); + }); + + it('should return an install script for a supported OS', async () => { + const { text: apiResponse } = await supertest + .get(`/api/ingest_manager/fleet/install/macos`) + .expect(200); + expect(apiResponse).match(/^#!\/bin\/sh/); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/list_agent.ts b/x-pack/test/api_integration/apis/fleet/list_agent.ts new file mode 100644 index 00000000000000..e9798c71e030ea --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/list_agent.ts @@ -0,0 +1,101 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; + +import { FtrProviderContext } from '../../ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + const esArchiver = getService('esArchiver'); + const supertest = getService('supertestWithoutAuth'); + const security = getService('security'); + const users: { [rollName: string]: { username: string; password: string; permissions?: any } } = { + kibana_basic_user: { + permissions: { + feature: { + dashboards: ['read'], + }, + spaces: ['*'], + }, + username: 'kibana_basic_user', + password: 'changeme', + }, + fleet_user: { + permissions: { + feature: { + ingestManager: ['read'], + }, + spaces: ['*'], + }, + username: 'fleet_user', + password: 'changeme', + }, + fleet_admin: { + permissions: { + feature: { + ingestManager: ['all'], + }, + spaces: ['*'], + }, + username: 'fleet_admin', + password: 'changeme', + }, + }; + + describe('fleet_list_agent', () => { + before(async () => { + for (const roleName in users) { + if (users.hasOwnProperty(roleName)) { + const user = users[roleName]; + + if (user.permissions) { + await security.role.create(roleName, { + kibana: [user.permissions], + }); + } + + // Import a repository first + await security.user.create(user.username, { + password: user.password, + roles: [roleName], + full_name: user.username, + }); + } + } + + await esArchiver.loadIfNeeded('fleet/agents'); + }); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should return the list of agents when requesting as a user with fleet write permissions', async () => { + const { body: apiResponse } = await supertest + .get(`/api/ingest_manager/fleet/agents`) + .auth(users.fleet_admin.username, users.fleet_admin.password) + .expect(200); + + expect(apiResponse).to.have.keys('success', 'page', 'total', 'list'); + expect(apiResponse.success).to.eql(true); + expect(apiResponse.total).to.eql(4); + }); + it('should return the list of agents when requesting as a user with fleet read permissions', async () => { + const { body: apiResponse } = await supertest + .get(`/api/ingest_manager/fleet/agents`) + .auth(users.fleet_user.username, users.fleet_user.password) + .expect(200); + expect(apiResponse).to.have.keys('success', 'page', 'total', 'list'); + expect(apiResponse.success).to.eql(true); + expect(apiResponse.total).to.eql(4); + }); + it('should not return the list of agents when requesting as a user without fleet permissions', async () => { + await supertest + .get(`/api/ingest_manager/fleet/agents`) + .auth(users.kibana_basic_user.username, users.kibana_basic_user.password) + .expect(404); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/fleet/unenroll_agent.ts b/x-pack/test/api_integration/apis/fleet/unenroll_agent.ts new file mode 100644 index 00000000000000..4b6b28e3d6350f --- /dev/null +++ b/x-pack/test/api_integration/apis/fleet/unenroll_agent.ts @@ -0,0 +1,77 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; + +import { FtrProviderContext } from '../../ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + const esArchiver = getService('esArchiver'); + const supertest = getService('supertest'); + + describe('fleet_unenroll_agent', () => { + before(async () => { + await esArchiver.loadIfNeeded('fleet/agents'); + }); + after(async () => { + await esArchiver.unload('fleet/agents'); + }); + + it('should not allow both ids and kuery in the payload', async () => { + await supertest + .post(`/api/ingest_manager/fleet/agents/unenroll`) + .set('kbn-xsrf', 'xxx') + .send({ + ids: ['agent:1'], + kuery: ['agents.id:1'], + }) + .expect(400); + }); + + it('should not allow no ids or kuery in the payload', async () => { + await supertest + .post(`/api/ingest_manager/fleet/agents/unenroll`) + .set('kbn-xsrf', 'xxx') + .send({}) + .expect(400); + }); + + it('allow to unenroll using a list of ids', async () => { + const { body } = await supertest + .post(`/api/ingest_manager/fleet/agents/unenroll`) + .set('kbn-xsrf', 'xxx') + .send({ + ids: ['agent1'], + }) + .expect(200); + + expect(body).to.have.keys('results', 'success'); + expect(body.success).to.be(true); + expect(body.results).to.have.length(1); + expect(body.results[0].success).to.be(true); + }); + + it('allow to unenroll using a kibana query', async () => { + const { body } = await supertest + .post(`/api/ingest_manager/fleet/agents/unenroll`) + .set('kbn-xsrf', 'xxx') + .send({ + kuery: 'agents.shared_id:agent2_filebeat OR agents.shared_id:agent3_metricbeat', + }) + .expect(200); + + expect(body).to.have.keys('results', 'success'); + expect(body.success).to.be(true); + expect(body.results).to.have.length(2); + expect(body.results[0].success).to.be(true); + + const agentsUnenrolledIds = body.results.map((r: { id: string }) => r.id); + + expect(agentsUnenrolledIds).to.contain('agent2'); + expect(agentsUnenrolledIds).to.contain('agent3'); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/index.js b/x-pack/test/api_integration/apis/index.js index 5af941dde525fe..0a87dcb4b5bb0d 100644 --- a/x-pack/test/api_integration/apis/index.js +++ b/x-pack/test/api_integration/apis/index.js @@ -27,6 +27,8 @@ export default function({ loadTestFile }) { loadTestFile(require.resolve('./siem')); loadTestFile(require.resolve('./short_urls')); loadTestFile(require.resolve('./lens')); + loadTestFile(require.resolve('./fleet')); + loadTestFile(require.resolve('./ingest')); loadTestFile(require.resolve('./endpoint')); loadTestFile(require.resolve('./ml')); }); diff --git a/x-pack/test/api_integration/apis/ingest/index.js b/x-pack/test/api_integration/apis/ingest/index.js new file mode 100644 index 00000000000000..5dac999a861673 --- /dev/null +++ b/x-pack/test/api_integration/apis/ingest/index.js @@ -0,0 +1,11 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export default function loadTests({ loadTestFile }) { + describe('Ingest Endpoints', () => { + loadTestFile(require.resolve('./policies')); + }); +} diff --git a/x-pack/test/api_integration/apis/ingest/policies.ts b/x-pack/test/api_integration/apis/ingest/policies.ts new file mode 100644 index 00000000000000..bfc77952ca325f --- /dev/null +++ b/x-pack/test/api_integration/apis/ingest/policies.ts @@ -0,0 +1,63 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; + +import { FtrProviderContext } from '../../ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + const esArchiver = getService('esArchiver'); + const supertest = getService('supertest'); + function useFixtures() { + before(async () => { + await esArchiver.loadIfNeeded('ingest/policies'); + }); + after(async () => { + await esArchiver.unload('ingest/policies'); + }); + } + + describe.skip('ingest_policies', () => { + describe('POST /api/ingest/policies', () => { + useFixtures(); + + it('should return a 400 if the request is not valid', async () => { + await supertest + .post(`/api/ingest/policies`) + .set('kbn-xsrf', 'xxx') + .send({}) + .expect(400); + }); + + it('should allow to create a new policy', async () => { + const { body: apiResponse } = await supertest + .post(`/api/ingest/policies`) + .set('kbn-xsrf', 'xxx') + .send({ + name: 'Policy from test 1', + description: 'I am a policy', + }) + .expect(200); + + expect(apiResponse.success).to.eql(true); + expect(apiResponse).to.have.keys('success', 'item', 'action'); + expect(apiResponse.item).to.have.keys('id', 'name', 'status', 'description'); + }); + }); + describe('GET /api/ingest/policies', () => { + useFixtures(); + it('should return the list of policies grouped by shared id', async () => { + const { body: apiResponse } = await supertest.get(`/api/ingest/policies`).expect(200); + expect(apiResponse).to.have.keys('success', 'page', 'total', 'list'); + expect(apiResponse.success).to.eql(true); + const policiesIds = (apiResponse.list as Array<{ id: string }>).map(i => i.id); + expect(policiesIds.length).to.eql(3); + expect(policiesIds).to.contain('1'); + expect(policiesIds).to.contain('3'); + }); + }); + }); +} diff --git a/x-pack/test/api_integration/apis/security/privileges.ts b/x-pack/test/api_integration/apis/security/privileges.ts index 4068b88cd30bcf..0b29fc1cac7de7 100644 --- a/x-pack/test/api_integration/apis/security/privileges.ts +++ b/x-pack/test/api_integration/apis/security/privileges.ts @@ -35,6 +35,7 @@ export default function({ getService }: FtrProviderContext) { apm: ['all', 'read'], siem: ['all', 'read'], endpoint: ['all', 'read'], + ingestManager: ['all', 'read'], }, global: ['all', 'read'], space: ['all', 'read'], diff --git a/x-pack/test/api_integration/config.js b/x-pack/test/api_integration/config.js index 70a7763bca6c55..182a9105a7df80 100644 --- a/x-pack/test/api_integration/config.js +++ b/x-pack/test/api_integration/config.js @@ -26,6 +26,8 @@ export async function getApiIntegrationConfig({ readConfigFile }) { '--xpack.security.session.idleTimeout=3600000', // 1 hour '--optimize.enabled=false', '--xpack.endpoint.enabled=true', + '--xpack.ingestManager.enabled=true', + '--xpack.ingestManager.fleet.enabled=true', '--xpack.endpoint.alertResultListDefaultDateRange.from=2018-01-10T00:00:00.000Z', ], }, diff --git a/x-pack/test/epm_api_integration/apis/file.ts b/x-pack/test/epm_api_integration/apis/file.ts new file mode 100644 index 00000000000000..2989263af40a7e --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/file.ts @@ -0,0 +1,151 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import ServerMock from 'mock-http-server'; +import { FtrProviderContext } from '../../api_integration/ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + describe('package file', () => { + const server = new ServerMock({ host: 'localhost', port: 6666 }); + beforeEach(() => { + server.start(() => {}); + }); + afterEach(() => { + server.stop(() => {}); + }); + it('fetches a .png screenshot image', async () => { + server.on({ + method: 'GET', + path: '/package/auditd-2.0.4/img/screenshots/auditbeat-file-integrity-dashboard.png', + reply: { + headers: { 'content-type': 'image/png' }, + }, + }); + + const supertest = getService('supertest'); + await supertest + .get( + '/api/ingest_manager/epm/packages/auditd-2.0.4/img/screenshots/auditbeat-file-integrity-dashboard.png' + ) + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'image/png') + .expect(200); + }); + + it('fetches an .svg icon image', async () => { + server.on({ + method: 'GET', + path: '/package/auditd-2.0.4/img/icon.svg', + reply: { + headers: { 'content-type': 'image/svg' }, + }, + }); + + const supertest = getService('supertest'); + await supertest + .get('/api/ingest_manager/epm/packages/auditd-2.0.4/img/icon.svg') + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'image/svg'); + }); + + it('fetches an auditbeat .conf rule file', async () => { + server.on({ + method: 'GET', + path: '/package/auditd-2.0.4/auditbeat/rules/sample-rules-linux-32bit.conf', + }); + + const supertest = getService('supertest'); + await supertest + .get( + '/api/ingest_manager/epm/packages/auditd-2.0.4/auditbeat/rules/sample-rules-linux-32bit.conf' + ) + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'application/json; charset=utf-8') + .expect(200); + }); + + it('fetches an auditbeat .yml config file', async () => { + server.on({ + method: 'GET', + path: '/package/auditd-2.0.4/auditbeat/config/config.yml', + reply: { + headers: { 'content-type': 'text/yaml; charset=UTF-8' }, + }, + }); + + const supertest = getService('supertest'); + await supertest + .get('/api/ingest_manager/epm/packages/auditd-2.0.4/auditbeat/config/config.yml') + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'text/yaml; charset=UTF-8') + .expect(200); + }); + + it('fetches a .json kibana visualization file', async () => { + server.on({ + method: 'GET', + path: + '/package/auditd-2.0.4/kibana/visualization/b21e0c70-c252-11e7-8692-232bd1143e8a-ecs.json', + }); + + const supertest = getService('supertest'); + await supertest + .get( + '/api/ingest_manager/epm/packages/auditd-2.0.4/kibana/visualization/b21e0c70-c252-11e7-8692-232bd1143e8a-ecs.json' + ) + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'application/json; charset=utf-8') + .expect(200); + }); + + it('fetches a .json kibana dashboard file', async () => { + server.on({ + method: 'GET', + path: + '/package/auditd-2.0.4/kibana/dashboard/7de391b0-c1ca-11e7-8995-936807a28b16-ecs.json', + }); + + const supertest = getService('supertest'); + await supertest + .get( + '/api/ingest_manager/epm/packages/auditd-2.0.4/kibana/dashboard/7de391b0-c1ca-11e7-8995-936807a28b16-ecs.json' + ) + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'application/json; charset=utf-8') + .expect(200); + }); + + it('fetches an .json index pattern file', async () => { + server.on({ + method: 'GET', + path: '/package/auditd-2.0.4/kibana/index-pattern/auditbeat-*.json', + }); + + const supertest = getService('supertest'); + await supertest + .get('/api/ingest_manager/epm/packages/auditd-2.0.4/kibana/index-pattern/auditbeat-*.json') + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'application/json; charset=utf-8') + .expect(200); + }); + + it('fetches a .json search file', async () => { + server.on({ + method: 'GET', + path: '/package/auditd-2.0.4/kibana/search/0f10c430-c1c3-11e7-8995-936807a28b16-ecs.json', + }); + + const supertest = getService('supertest'); + await supertest + .get( + '/api/ingest_manager/epm/packages/auditd-2.0.4/kibana/search/0f10c430-c1c3-11e7-8995-936807a28b16-ecs.json' + ) + .set('kbn-xsrf', 'xxx') + .expect('Content-Type', 'application/json; charset=utf-8') + .expect(200); + }); + }); +} diff --git a/x-pack/test/epm_api_integration/apis/fixtures/packages/epr/yamlpipeline_1.0.0.tar.gz b/x-pack/test/epm_api_integration/apis/fixtures/packages/epr/yamlpipeline_1.0.0.tar.gz new file mode 100644 index 00000000000000..ca8695f111d023 Binary files /dev/null and b/x-pack/test/epm_api_integration/apis/fixtures/packages/epr/yamlpipeline_1.0.0.tar.gz differ diff --git a/x-pack/test/epm_api_integration/apis/fixtures/packages/package/yamlpipeline_1.0.0 b/x-pack/test/epm_api_integration/apis/fixtures/packages/package/yamlpipeline_1.0.0 new file mode 100644 index 00000000000000..bf167c583aab3c --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/fixtures/packages/package/yamlpipeline_1.0.0 @@ -0,0 +1,32 @@ +{ + "name": "yamlpipeline", + "title": "Yaml Pipeline package", + "version": "1.0.0", + "description": "This package contains a yaml pipeline.\n", + "type": "integration", + "categories": [ + "logs" + ], + "requirement": { + "kibana": {} + }, + "assets": [ + "/package/yamlpipeline-1.0.0/manifest.yml", + "/package/yamlpipeline-1.0.0/dataset/log/manifest.yml", + "/package/yamlpipeline-1.0.0/dataset/log/elasticsearch/ingest-pipeline/pipeline-entry.yml", + "/package/yamlpipeline-1.0.0/dataset/log/elasticsearch/ingest-pipeline/pipeline-json.yml", + "/package/yamlpipeline-1.0.0/dataset/log/elasticsearch/ingest-pipeline/pipeline-plaintext.yml" + ], + "format_version": "1.0.0", + "datasets": [ + { + "title": "Log Yaml pipeline", + "name": "log", + "release": "", + "type": "logs", + "ingest_pipeline": "" + } + ], + "download": "/epr/yamlpipeline/yamlpipeline-1.0.0.tar.gz", + "path": "/package/yamlpipeline-1.0.0" +} \ No newline at end of file diff --git a/x-pack/test/epm_api_integration/apis/ilm.ts b/x-pack/test/epm_api_integration/apis/ilm.ts new file mode 100644 index 00000000000000..fb33071361cbda --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/ilm.ts @@ -0,0 +1,40 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import { FtrProviderContext } from '../../api_integration/ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + describe('ilm', () => { + it('setup policy', async () => { + const policyName = 'foo'; + const es = getService('es'); + const policy = { + policy: { + phases: { + hot: { + actions: { + rollover: { + max_size: '50gb', + max_age: '30d', + }, + }, + }, + }, + }, + }; + + const data = await es.transport.request({ + method: 'PUT', + path: '/_ilm/policy/' + policyName, + body: policy, + }); + + expect(data.body.acknowledged).to.eql(true); + expect(data.statusCode).to.eql(200); + }); + }); +} diff --git a/x-pack/test/epm_api_integration/apis/index.js b/x-pack/test/epm_api_integration/apis/index.js new file mode 100644 index 00000000000000..cfdfd5baf1e590 --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/index.js @@ -0,0 +1,15 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +export default function({ loadTestFile }) { + describe('EPM Endpoints', function() { + this.tags('ciGroup7'); + loadTestFile(require.resolve('./list')); + loadTestFile(require.resolve('./file')); + loadTestFile(require.resolve('./template')); + loadTestFile(require.resolve('./ilm')); + }); +} diff --git a/x-pack/test/epm_api_integration/apis/list.ts b/x-pack/test/epm_api_integration/apis/list.ts new file mode 100644 index 00000000000000..d0d921af6016b3 --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/list.ts @@ -0,0 +1,124 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import ServerMock from 'mock-http-server'; +import { FtrProviderContext } from '../../api_integration/ftr_provider_context'; + +export default function({ getService }: FtrProviderContext) { + describe('list', () => { + const server = new ServerMock({ host: 'localhost', port: 6666 }); + beforeEach(() => { + server.start(() => {}); + }); + afterEach(() => { + server.stop(() => {}); + }); + it('lists all packages from the registry', async () => { + const searchResponse = [ + { + description: 'First integration package', + download: '/package/first-1.0.1.tar.gz', + name: 'first', + title: 'First', + type: 'integration', + version: '1.0.1', + }, + { + description: 'Second integration package', + download: '/package/second-2.0.4.tar.gz', + icons: [ + { + src: '/package/second-2.0.4/img/icon.svg', + type: 'image/svg+xml', + }, + ], + name: 'second', + title: 'Second', + type: 'integration', + version: '2.0.4', + }, + ]; + server.on({ + method: 'GET', + path: '/search', + reply: { + status: 200, + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(searchResponse), + }, + }); + + const supertest = getService('supertest'); + const fetchPackageList = async () => { + const response = await supertest + .get('/api/ingest_manager/epm/packages') + .set('kbn-xsrf', 'xxx') + .expect(200); + return response.body; + }; + + const listResponse = await fetchPackageList(); + expect(listResponse.response.length).to.be(2); + expect(listResponse.response[0]).to.eql({ ...searchResponse[0], status: 'not_installed' }); + expect(listResponse.response[1]).to.eql({ ...searchResponse[1], status: 'not_installed' }); + }); + + it('sorts the packages even if the registry sends them unsorted', async () => { + const searchResponse = [ + { + description: 'BBB integration package', + download: '/package/bbb-1.0.1.tar.gz', + name: 'bbb', + title: 'BBB', + type: 'integration', + version: '1.0.1', + }, + { + description: 'CCC integration package', + download: '/package/ccc-2.0.4.tar.gz', + name: 'ccc', + title: 'CCC', + type: 'integration', + version: '2.0.4', + }, + { + description: 'AAA integration package', + download: '/package/aaa-0.0.1.tar.gz', + name: 'aaa', + title: 'AAA', + type: 'integration', + version: '0.0.1', + }, + ]; + server.on({ + method: 'GET', + path: '/search', + reply: { + status: 200, + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(searchResponse), + }, + }); + + const supertest = getService('supertest'); + const fetchPackageList = async () => { + const response = await supertest + .get('/api/ingest_manager/epm/packages') + .set('kbn-xsrf', 'xxx') + .expect(200); + return response.body; + }; + + const listResponse = await fetchPackageList(); + + expect(listResponse.response.length).to.be(3); + expect(listResponse.response[0].name).to.eql('aaa'); + expect(listResponse.response[1].name).to.eql('bbb'); + expect(listResponse.response[2].name).to.eql('ccc'); + }); + }); +} diff --git a/x-pack/test/epm_api_integration/apis/mock_http_server.d.ts b/x-pack/test/epm_api_integration/apis/mock_http_server.d.ts new file mode 100644 index 00000000000000..b037445893c95e --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/mock_http_server.d.ts @@ -0,0 +1,9 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +// No types for mock-http-server available, but we don't need them. + +declare module 'mock-http-server'; diff --git a/x-pack/test/epm_api_integration/apis/template.ts b/x-pack/test/epm_api_integration/apis/template.ts new file mode 100644 index 00000000000000..923dd1d3676b51 --- /dev/null +++ b/x-pack/test/epm_api_integration/apis/template.ts @@ -0,0 +1,44 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import expect from '@kbn/expect'; +import { FtrProviderContext } from '../../api_integration/ftr_provider_context'; +import { getTemplate } from '../../../plugins/ingest_manager/server/services/epm/elasticsearch/template/template'; + +export default function({ getService }: FtrProviderContext) { + const indexPattern = 'foo'; + const templateName = 'bar'; + const es = getService('es'); + const mappings = { + properties: { + foo: { + type: 'keyword', + }, + }, + }; + // This test was inspired by https://github.com/elastic/kibana/blob/master/x-pack/test/api_integration/apis/monitoring/common/mappings_exist.js + describe('template', async () => { + it('can be loaded', async () => { + const template = getTemplate('logs', indexPattern, mappings); + + // This test is not an API integration test with Kibana + // We want to test here if the template is valid and for this we need a running ES instance. + // If the ES instance takes the template, we assume it is a valid template. + const { body: response1 } = await es.indices.putTemplate({ + name: templateName, + body: template, + }); + // Checks if template loading worked as expected + expect(response1).to.eql({ acknowledged: true }); + + const { body: response2 } = await es.indices.getTemplate({ name: templateName }); + // Checks if the content of the template that was loaded is as expected + // We already know based on the above test that the template was valid + // but we check here also if we wrote the index pattern inside the template as expected + expect(response2[templateName].index_patterns).to.eql([`${indexPattern}-*`]); + }); + }); +} diff --git a/x-pack/test/epm_api_integration/config.ts b/x-pack/test/epm_api_integration/config.ts new file mode 100644 index 00000000000000..e95a389ef20edf --- /dev/null +++ b/x-pack/test/epm_api_integration/config.ts @@ -0,0 +1,36 @@ +/* + * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one + * or more contributor license agreements. Licensed under the Elastic License; + * you may not use this file except in compliance with the Elastic License. + */ + +import { FtrConfigProviderContext } from '@kbn/test/types/ftr'; + +export default async function({ readConfigFile }: FtrConfigProviderContext) { + const xPackAPITestsConfig = await readConfigFile(require.resolve('../api_integration/config.js')); + + return { + testFiles: [require.resolve('./apis')], + servers: xPackAPITestsConfig.get('servers'), + services: { + supertest: xPackAPITestsConfig.get('services.supertest'), + es: xPackAPITestsConfig.get('services.es'), + }, + junit: { + reportName: 'X-Pack EPM API Integration Tests', + }, + + esTestCluster: { + ...xPackAPITestsConfig.get('esTestCluster'), + }, + + kbnTestServer: { + ...xPackAPITestsConfig.get('kbnTestServer'), + serverArgs: [ + ...xPackAPITestsConfig.get('kbnTestServer.serverArgs'), + '--xpack.ingestManager.epm.enabled=true', + '--xpack.ingestManager.epm.registryUrl=http://localhost:6666', + ], + }, + }; +} diff --git a/x-pack/test/functional/es_archives/fleet/agents/data.json b/x-pack/test/functional/es_archives/fleet/agents/data.json new file mode 100644 index 00000000000000..36928018d15a05 --- /dev/null +++ b/x-pack/test/functional/es_archives/fleet/agents/data.json @@ -0,0 +1,147 @@ +{ + "type": "doc", + "value": { + "id": "agents:agent1", + "index": ".kibana", + "source": { + "type": "agents", + "agents": { + "access_api_key_id": "api-key-2", + "active": true, + "shared_id": "agent1_filebeat", + "config_id": "1", + "type": "PERMANENT", + "local_metadata": "{}", + "user_provided_metadata": "{}", + "actions": [{ + "id": "37ed51ff-e80f-4f2a-a62d-f4fa975e7d85", + "created_at": "2019-09-04T15:04:07+0000", + "type": "RESUME" + }, + { + "id": "b400439c-bbbf-43d5-83cb-cf8b7e32506f", + "type": "PAUSE", + "created_at": "2019-09-04T15:01:07+0000", + "sent_at": "2019-09-04T15:03:07+0000" + }] + } + } + } +} + +{ + "type": "doc", + "value": { + "id": "agents:agent2", + "index": ".kibana", + "source": { + "type": "agents", + "agents": { + "access_api_key_id": "api-key-2", + "active": true, + "shared_id": "agent2_filebeat", + "type": "PERMANENT", + "local_metadata": "{}", + "user_provided_metadata": "{}", + "actions": [] + } + } + } +} + +{ + "type": "doc", + "value": { + "id": "agents:agent3", + "index": ".kibana", + "source": { + "type": "agents", + "agents": { + "access_api_key_id": "api-key-3", + "active": true, + "shared_id": "agent3_metricbeat", + "type": "PERMANENT", + "local_metadata": "{}", + "user_provided_metadata": "{}", + "actions": [] + } + } + } +} + +{ + "type": "doc", + "value": { + "id": "agents:agent4", + "index": ".kibana", + "source": { + "type": "agents", + "agents": { + "access_api_key_id": "api-key-4", + "active": true, + "shared_id": "agent4_metricbeat", + "type": "PERMANENT", + "local_metadata": "{}", + "user_provided_metadata": "{}", + "actions": [] + } + } + } +} + +{ + "type": "doc", + "value": { + "id": "enrollment_api_keys:ed22ca17-e178-4cfe-8b02-54ea29fbd6d0", + "index": ".kibana", + "source": { + "enrollment_api_keys" : { + "created_at" : "2019-10-10T16:31:12.518Z", + "name": "FleetEnrollmentKey:1", + "api_key_id" : "key", + "config_id" : "policy:1", + "active" : true + }, + "type" : "enrollment_api_keys", + "references": [] + } + } +} + +{ + "type": "doc", + "value": { + "id": "events:event1", + "index": ".kibana", + "source": { + "type": "agent_events", + "agent_events": { + "agent_id": "agent1", + "type": "STATE", + "subtype": "STARTED", + "message": "State changed from STOPPED to STARTED", + "payload": "{\"previous_state\": \"STOPPED\"}", + "timestamp": "2019-09-20T17:30:22.950Z" + } + } + } +} + +{ + "type": "doc", + "value": { + "id": "events:event2", + "index": ".kibana", + "source": { + "type": "agent_events", + "agent_events": { + "agent_id": "agent1", + "type": "STATE", + "subtype": "STOPPED", + "message": "State changed from RUNNING to STOPPED", + "payload": "{\"previous_state\": \"RUNNING\"}", + "timestamp": "2019-09-20T17:30:25.950Z" + } + } + } +} diff --git a/x-pack/test/functional/es_archives/fleet/agents/mappings.json b/x-pack/test/functional/es_archives/fleet/agents/mappings.json new file mode 100644 index 00000000000000..0f632b7333ee76 --- /dev/null +++ b/x-pack/test/functional/es_archives/fleet/agents/mappings.json @@ -0,0 +1,1549 @@ +{ + "type": "index", + "value": { + "aliases": { + ".kibana": {} + }, + "index": ".kibana_1", + "mappings": { + "dynamic": "strict", + "_meta": { + "migrationMappingPropertyHashes": { + "ml-telemetry": "257fd1d4b4fdbb9cb4b8a3b27da201e9", + "server": "ec97f1c5da1a19609a60874e5af1100c", + "visualization": "52d7a13ad68a150c4525b292d23e12cc", + "references": "7997cf5a56cc02bdc9c93361bde732b0", + "graph-workspace": "cd7ba1330e6682e9cc00b78850874be1", + "siem-ui-timeline-note": "8874706eedc49059d4cf0f5094559084", + "policies": "1a096b98c98c2efebfdba77cefcfe54a", + "type": "2f4316de49999235636386fe51dc06c1", + "lens": "21c3ea0763beb1ecb0162529706b88c5", + "space": "c5ca8acafa0beaa4d08d014a97b6bc6b", + "infrastructure-ui-source": "ddc0ecb18383f6b26101a2fadb2dab0c", + "sample-data-telemetry": "7d3cfeb915303c9641c59681967ffeb4", + "search": "181661168bbadd1eff5902361e2a0d5c", + "updated_at": "00da57df13e94e9d98437d13ace4bfe0", + "canvas-workpad": "b0a1706d356228dbdcb4a17e6b9eb231", + "map": "23d7aa4a720d4938ccde3983f87bd58d", + "dashboard": "d00f614b29a80360e1190193fd333bab", + "apm-services-telemetry": "07ee1939fa4302c62ddc052ec03fed90", + "metrics-explorer-view": "53c5365793677328df0ccb6138bf3cdd", + "epm": "abf5b64aa599932bd181efc86dce14a7", + "siem-ui-timeline": "6485ab095be8d15246667b98a1a34295", + "agent_events": "8060c5567d33f6697164e1fd5c81b8ed", + "file-upload-telemetry": "0ed4d3e1983d1217a30982630897092e", + "query": "11aaeb7f5f7fa5bb43f25e18ce26e7d9", + "kql-telemetry": "d12a98a6f19a2d273696597547e064ee", + "ui-metric": "0d409297dc5ebe1e3a1da691c6ee32e3", + "url": "c7f66a0df8b1b52f17c28c4adb111105", + "apm-indices": "c69b68f3fe2bf27b4788d4191c1d6011", + "agents": "1c8e942384219bd899f381fd40e407d7", + "migrationVersion": "4a1746014a75ade3a714e1db5763276f", + "inventory-view": "84b320fd67209906333ffce261128462", + "enrollment_api_keys": "90e66b79e8e948e9c15434fdb3ae576e", + "upgrade-assistant-reindex-operation": "a53a20fe086b72c9a86da3cc12dad8a6", + "index-pattern": "66eccb05066c5a89924f48a9e9736499", + "canvas-element": "7390014e1091044523666d97247392fc", + "datasources": "2fed9e9883b9622cd59a73ee5550ef4f", + "maps-telemetry": "a4229f8b16a6820c6d724b7e0c1f729d", + "namespace": "2f4316de49999235636386fe51dc06c1", + "telemetry": "358ffaa88ba34a97d55af0933a117de4", + "siem-ui-timeline-pinned-event": "20638091112f0e14f0e443d512301c29", + "timelion-sheet": "9a2a2748877c7a7b582fef201ab1d4cf", + "config": "87aca8fdb053154f11383fce3dbf3edf", + "upgrade-assistant-telemetry": "56702cec857e0a9dacfb696655b4ff7b", + "lens-ui-telemetry": "509bfa5978586998e05f9e303c07a327" + } + }, + "properties": { + "agent_events": { + "properties": { + "agent_id": { + "type": "keyword" + }, + "data": { + "type": "text" + }, + "message": { + "type": "text" + }, + "payload": { + "type": "text" + }, + "subtype": { + "type": "keyword" + }, + "timestamp": { + "type": "date" + }, + "type": { + "type": "keyword" + } + } + }, + "agents": { + "properties": { + "access_api_key_id": { + "type": "keyword" + }, + "actions": { + "type": "nested", + "properties": { + "created_at": { + "type": "date" + }, + "data": { + "type": "text" + }, + "id": { + "type": "keyword" + }, + "sent_at": { + "type": "date" + }, + "type": { + "type": "keyword" + } + } + }, + "active": { + "type": "boolean" + }, + "enrolled_at": { + "type": "date" + }, + "last_checkin": { + "type": "date" + }, + "last_updated": { + "type": "date" + }, + "local_metadata": { + "type": "text" + }, + "config_id": { + "type": "keyword" + }, + "shared_id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + }, + "updated_at": { + "type": "date" + }, + "user_provided_metadata": { + "type": "text" + }, + "current_error_events": { + "type": "text" + }, + "version": { + "type": "keyword" + } + } + }, + "apm-indices": { + "properties": { + "apm_oss": { + "properties": { + "apmAgentConfigurationIndex": { + "type": "keyword" + }, + "errorIndices": { + "type": "keyword" + }, + "metricsIndices": { + "type": "keyword" + }, + "onboardingIndices": { + "type": "keyword" + }, + "sourcemapIndices": { + "type": "keyword" + }, + "spanIndices": { + "type": "keyword" + }, + "transactionIndices": { + "type": "keyword" + } + } + } + } + }, + "apm-services-telemetry": { + "properties": { + "has_any_services": { + "type": "boolean" + }, + "services_per_agent": { + "properties": { + "dotnet": { + "type": "long", + "null_value": 0 + }, + "go": { + "type": "long", + "null_value": 0 + }, + "java": { + "type": "long", + "null_value": 0 + }, + "js-base": { + "type": "long", + "null_value": 0 + }, + "nodejs": { + "type": "long", + "null_value": 0 + }, + "python": { + "type": "long", + "null_value": 0 + }, + "ruby": { + "type": "long", + "null_value": 0 + }, + "rum-js": { + "type": "long", + "null_value": 0 + } + } + } + } + }, + "canvas-element": { + "dynamic": "false", + "properties": { + "@created": { + "type": "date" + }, + "@timestamp": { + "type": "date" + }, + "content": { + "type": "text" + }, + "help": { + "type": "text" + }, + "image": { + "type": "text" + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword" + } + } + } + } + }, + "canvas-workpad": { + "dynamic": "false", + "properties": { + "@created": { + "type": "date" + }, + "@timestamp": { + "type": "date" + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword" + } + } + } + } + }, + "config": { + "dynamic": "true", + "properties": { + "buildNum": { + "type": "keyword" + } + } + }, + "dashboard": { + "properties": { + "description": { + "type": "text" + }, + "hits": { + "type": "integer" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "optionsJSON": { + "type": "text" + }, + "panelsJSON": { + "type": "text" + }, + "refreshInterval": { + "properties": { + "display": { + "type": "keyword" + }, + "pause": { + "type": "boolean" + }, + "section": { + "type": "integer" + }, + "value": { + "type": "integer" + } + } + }, + "timeFrom": { + "type": "keyword" + }, + "timeRestore": { + "type": "boolean" + }, + "timeTo": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "datasources": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + }, + "package": { + "properties": { + "assets": { + "properties": { + "id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "description": { + "type": "keyword" + }, + "name": { + "type": "keyword" + }, + "title": { + "type": "keyword" + }, + "version": { + "type": "keyword" + } + } + }, + "read_alias": { + "type": "keyword" + }, + "streams": { + "properties": { + "config": { + "type": "flattened" + }, + "id": { + "type": "keyword" + }, + "input": { + "properties": { + "config": { + "type": "flattened" + }, + "fields": { + "type": "flattened" + }, + "id": { + "type": "keyword" + }, + "ilm_policy": { + "type": "keyword" + }, + "index_template": { + "type": "keyword" + }, + "ingest_pipelines": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "output_id": { + "type": "keyword" + }, + "processors": { + "type": "keyword" + } + } + } + } + }, + "enrollment_api_keys": { + "properties": { + "active": { + "type": "boolean" + }, + "api_key": { + "type": "binary" + }, + "api_key_id": { + "type": "keyword" + }, + "created_at": { + "type": "date" + }, + "enrollment_rules": { + "type": "nested", + "properties": { + "created_at": { + "type": "date" + }, + "id": { + "type": "keyword" + }, + "ip_ranges": { + "type": "keyword" + }, + "types": { + "type": "keyword" + }, + "updated_at": { + "type": "date" + }, + "window_duration": { + "type": "nested", + "properties": { + "from": { + "type": "date" + }, + "to": { + "type": "date" + } + } + } + } + }, + "expire_at": { + "type": "date" + }, + "name": { + "type": "keyword" + }, + "config_id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + }, + "updated_at": { + "type": "date" + } + } + }, + "epm": { + "properties": { + "installed": { + "type": "nested", + "properties": { + "id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + } + } + }, + "file-upload-telemetry": { + "properties": { + "filesUploadedTotalCount": { + "type": "long" + } + } + }, + "graph-workspace": { + "properties": { + "description": { + "type": "text" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "numLinks": { + "type": "integer" + }, + "numVertices": { + "type": "integer" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + }, + "wsState": { + "type": "text" + } + } + }, + "index-pattern": { + "properties": { + "fieldFormatMap": { + "type": "text" + }, + "fields": { + "type": "text" + }, + "intervalName": { + "type": "keyword" + }, + "notExpandable": { + "type": "boolean" + }, + "sourceFilters": { + "type": "text" + }, + "timeFieldName": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "type": { + "type": "keyword" + }, + "typeMeta": { + "type": "keyword" + } + } + }, + "infrastructure-ui-source": { + "properties": { + "description": { + "type": "text" + }, + "fields": { + "properties": { + "container": { + "type": "keyword" + }, + "host": { + "type": "keyword" + }, + "pod": { + "type": "keyword" + }, + "tiebreaker": { + "type": "keyword" + }, + "timestamp": { + "type": "keyword" + } + } + }, + "logAlias": { + "type": "keyword" + }, + "logColumns": { + "type": "nested", + "properties": { + "fieldColumn": { + "properties": { + "field": { + "type": "keyword" + }, + "id": { + "type": "keyword" + } + } + }, + "messageColumn": { + "properties": { + "id": { + "type": "keyword" + } + } + }, + "timestampColumn": { + "properties": { + "id": { + "type": "keyword" + } + } + } + } + }, + "metricAlias": { + "type": "keyword" + }, + "name": { + "type": "text" + } + } + }, + "inventory-view": { + "properties": { + "autoBounds": { + "type": "boolean" + }, + "autoReload": { + "type": "boolean" + }, + "boundsOverride": { + "properties": { + "max": { + "type": "integer" + }, + "min": { + "type": "integer" + } + } + }, + "customOptions": { + "type": "nested", + "properties": { + "field": { + "type": "keyword" + }, + "text": { + "type": "keyword" + } + } + }, + "filterQuery": { + "properties": { + "expression": { + "type": "keyword" + }, + "kind": { + "type": "keyword" + } + } + }, + "groupBy": { + "type": "nested", + "properties": { + "field": { + "type": "keyword" + }, + "label": { + "type": "keyword" + } + } + }, + "metric": { + "properties": { + "type": { + "type": "keyword" + } + } + }, + "name": { + "type": "keyword" + }, + "nodeType": { + "type": "keyword" + }, + "time": { + "type": "integer" + }, + "view": { + "type": "keyword" + } + } + }, + "kql-telemetry": { + "properties": { + "optInCount": { + "type": "long" + }, + "optOutCount": { + "type": "long" + } + } + }, + "lens": { + "properties": { + "expression": { + "type": "keyword", + "index": false + }, + "state": { + "type": "flattened" + }, + "title": { + "type": "text" + }, + "visualizationType": { + "type": "keyword" + } + } + }, + "lens-ui-telemetry": { + "properties": { + "count": { + "type": "integer" + }, + "date": { + "type": "date" + }, + "name": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "map": { + "properties": { + "bounds": { + "type": "geo_shape" + }, + "description": { + "type": "text" + }, + "layerListJSON": { + "type": "text" + }, + "mapStateJSON": { + "type": "text" + }, + "title": { + "type": "text" + }, + "uiStateJSON": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "maps-telemetry": { + "properties": { + "attributesPerMap": { + "properties": { + "dataSourcesCount": { + "properties": { + "avg": { + "type": "long" + }, + "max": { + "type": "long" + }, + "min": { + "type": "long" + } + } + }, + "emsVectorLayersCount": { + "type": "object", + "dynamic": "true" + }, + "layerTypesCount": { + "type": "object", + "dynamic": "true" + }, + "layersCount": { + "properties": { + "avg": { + "type": "long" + }, + "max": { + "type": "long" + }, + "min": { + "type": "long" + } + } + } + } + }, + "mapsTotalCount": { + "type": "long" + }, + "timeCaptured": { + "type": "date" + } + } + }, + "metrics-explorer-view": { + "properties": { + "chartOptions": { + "properties": { + "stack": { + "type": "boolean" + }, + "type": { + "type": "keyword" + }, + "yAxisMode": { + "type": "keyword" + } + } + }, + "currentTimerange": { + "properties": { + "from": { + "type": "keyword" + }, + "interval": { + "type": "keyword" + }, + "to": { + "type": "keyword" + } + } + }, + "name": { + "type": "keyword" + }, + "options": { + "properties": { + "aggregation": { + "type": "keyword" + }, + "filterQuery": { + "type": "keyword" + }, + "groupBy": { + "type": "keyword" + }, + "limit": { + "type": "integer" + }, + "metrics": { + "type": "nested", + "properties": { + "aggregation": { + "type": "keyword" + }, + "color": { + "type": "keyword" + }, + "field": { + "type": "keyword" + }, + "label": { + "type": "keyword" + } + } + } + } + } + } + }, + "migrationVersion": { + "dynamic": "true", + "properties": { + "space": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } + } + }, + "ml-telemetry": { + "properties": { + "file_data_visualizer": { + "properties": { + "index_creation_count": { + "type": "long" + } + } + } + } + }, + "namespace": { + "type": "keyword" + }, + "policies": { + "properties": { + "datasources": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "id": { + "type": "keyword" + }, + "label": { + "type": "keyword" + }, + "name": { + "type": "text" + }, + "status": { + "type": "keyword" + }, + "updated_by": { + "type": "keyword" + }, + "updated_on": { + "type": "keyword" + } + } + }, + "query": { + "properties": { + "description": { + "type": "text" + }, + "filters": { + "type": "object", + "enabled": false + }, + "query": { + "properties": { + "language": { + "type": "keyword" + }, + "query": { + "type": "keyword", + "index": false + } + } + }, + "timefilter": { + "type": "object", + "enabled": false + }, + "title": { + "type": "text" + } + } + }, + "references": { + "type": "nested", + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "sample-data-telemetry": { + "properties": { + "installCount": { + "type": "long" + }, + "unInstallCount": { + "type": "long" + } + } + }, + "search": { + "properties": { + "columns": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "hits": { + "type": "integer" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "sort": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "server": { + "properties": { + "uuid": { + "type": "keyword" + } + } + }, + "siem-ui-timeline": { + "properties": { + "columns": { + "properties": { + "aggregatable": { + "type": "boolean" + }, + "category": { + "type": "keyword" + }, + "columnHeaderType": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "example": { + "type": "text" + }, + "id": { + "type": "keyword" + }, + "indexes": { + "type": "keyword" + }, + "name": { + "type": "text" + }, + "placeholder": { + "type": "text" + }, + "searchable": { + "type": "boolean" + }, + "type": { + "type": "keyword" + } + } + }, + "created": { + "type": "date" + }, + "createdBy": { + "type": "text" + }, + "dataProviders": { + "properties": { + "and": { + "properties": { + "enabled": { + "type": "boolean" + }, + "excluded": { + "type": "boolean" + }, + "id": { + "type": "keyword" + }, + "kqlQuery": { + "type": "text" + }, + "name": { + "type": "text" + }, + "queryMatch": { + "properties": { + "displayField": { + "type": "text" + }, + "displayValue": { + "type": "text" + }, + "field": { + "type": "text" + }, + "operator": { + "type": "text" + }, + "value": { + "type": "text" + } + } + } + } + }, + "enabled": { + "type": "boolean" + }, + "excluded": { + "type": "boolean" + }, + "id": { + "type": "keyword" + }, + "kqlQuery": { + "type": "text" + }, + "name": { + "type": "text" + }, + "queryMatch": { + "properties": { + "displayField": { + "type": "text" + }, + "displayValue": { + "type": "text" + }, + "field": { + "type": "text" + }, + "operator": { + "type": "text" + }, + "value": { + "type": "text" + } + } + } + } + }, + "dateRange": { + "properties": { + "end": { + "type": "date" + }, + "start": { + "type": "date" + } + } + }, + "description": { + "type": "text" + }, + "favorite": { + "properties": { + "favoriteDate": { + "type": "date" + }, + "fullName": { + "type": "text" + }, + "keySearch": { + "type": "text" + }, + "userName": { + "type": "text" + } + } + }, + "filters": { + "properties": { + "exists": { + "type": "text" + }, + "match_all": { + "type": "text" + }, + "meta": { + "properties": { + "alias": { + "type": "text" + }, + "controlledBy": { + "type": "text" + }, + "disabled": { + "type": "boolean" + }, + "field": { + "type": "text" + }, + "formattedValue": { + "type": "text" + }, + "index": { + "type": "keyword" + }, + "key": { + "type": "keyword" + }, + "negate": { + "type": "boolean" + }, + "params": { + "type": "text" + }, + "type": { + "type": "keyword" + }, + "value": { + "type": "text" + } + } + }, + "missing": { + "type": "text" + }, + "query": { + "type": "text" + }, + "range": { + "type": "text" + }, + "script": { + "type": "text" + } + } + }, + "kqlMode": { + "type": "keyword" + }, + "kqlQuery": { + "properties": { + "filterQuery": { + "properties": { + "kuery": { + "properties": { + "expression": { + "type": "text" + }, + "kind": { + "type": "keyword" + } + } + }, + "serializedQuery": { + "type": "text" + } + } + } + } + }, + "savedQueryId": { + "type": "keyword" + }, + "sort": { + "properties": { + "columnId": { + "type": "keyword" + }, + "sortDirection": { + "type": "keyword" + } + } + }, + "title": { + "type": "text" + }, + "updated": { + "type": "date" + }, + "updatedBy": { + "type": "text" + } + } + }, + "siem-ui-timeline-note": { + "properties": { + "created": { + "type": "date" + }, + "createdBy": { + "type": "text" + }, + "eventId": { + "type": "keyword" + }, + "note": { + "type": "text" + }, + "timelineId": { + "type": "keyword" + }, + "updated": { + "type": "date" + }, + "updatedBy": { + "type": "text" + } + } + }, + "siem-ui-timeline-pinned-event": { + "properties": { + "created": { + "type": "date" + }, + "createdBy": { + "type": "text" + }, + "eventId": { + "type": "keyword" + }, + "timelineId": { + "type": "keyword" + }, + "updated": { + "type": "date" + }, + "updatedBy": { + "type": "text" + } + } + }, + "space": { + "properties": { + "_reserved": { + "type": "boolean" + }, + "color": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "disabledFeatures": { + "type": "keyword" + }, + "imageUrl": { + "type": "text", + "index": false + }, + "initials": { + "type": "keyword" + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 2048 + } + } + } + } + }, + "telemetry": { + "properties": { + "enabled": { + "type": "boolean" + }, + "lastReported": { + "type": "date" + }, + "lastVersionChecked": { + "type": "keyword", + "ignore_above": 256 + }, + "sendUsageFrom": { + "type": "keyword", + "ignore_above": 256 + }, + "userHasSeenNotice": { + "type": "boolean" + } + } + }, + "timelion-sheet": { + "properties": { + "description": { + "type": "text" + }, + "hits": { + "type": "integer" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "timelion_chart_height": { + "type": "integer" + }, + "timelion_columns": { + "type": "integer" + }, + "timelion_interval": { + "type": "keyword" + }, + "timelion_other_interval": { + "type": "keyword" + }, + "timelion_rows": { + "type": "integer" + }, + "timelion_sheet": { + "type": "text" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "type": { + "type": "keyword" + }, + "ui-metric": { + "properties": { + "count": { + "type": "integer" + } + } + }, + "updated_at": { + "type": "date" + }, + "upgrade-assistant-reindex-operation": { + "dynamic": "true", + "properties": { + "indexName": { + "type": "keyword" + }, + "status": { + "type": "integer" + } + } + }, + "upgrade-assistant-telemetry": { + "properties": { + "features": { + "properties": { + "deprecation_logging": { + "properties": { + "enabled": { + "type": "boolean", + "null_value": true + } + } + } + } + }, + "ui_open": { + "properties": { + "cluster": { + "type": "long", + "null_value": 0 + }, + "indices": { + "type": "long", + "null_value": 0 + }, + "overview": { + "type": "long", + "null_value": 0 + } + } + }, + "ui_reindex": { + "properties": { + "close": { + "type": "long", + "null_value": 0 + }, + "open": { + "type": "long", + "null_value": 0 + }, + "start": { + "type": "long", + "null_value": 0 + }, + "stop": { + "type": "long", + "null_value": 0 + } + } + } + } + }, + "url": { + "properties": { + "accessCount": { + "type": "long" + }, + "accessDate": { + "type": "date" + }, + "createDate": { + "type": "date" + }, + "url": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 2048 + } + } + } + } + }, + "visualization": { + "properties": { + "description": { + "type": "text" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "savedSearchRefName": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "uiStateJSON": { + "type": "text" + }, + "version": { + "type": "integer" + }, + "visState": { + "type": "text" + } + } + } + } + }, + "settings": { + "index": { + "auto_expand_replicas": "0-1", + "number_of_replicas": "0", + "number_of_shards": "1" + } + } + } +} diff --git a/x-pack/test/functional/es_archives/ingest/policies/data.json b/x-pack/test/functional/es_archives/ingest/policies/data.json new file mode 100644 index 00000000000000..78cf18d501a3e7 --- /dev/null +++ b/x-pack/test/functional/es_archives/ingest/policies/data.json @@ -0,0 +1,59 @@ +{ + "type": "doc", + "value": { + "index": ".kibana", + "id": "policies:1", + "source": { + "policies": { + "name": "Policy 1", + "description": "Amazing policy", + "status": "active", + "updated_on": "2019-09-20T17:35:09+0000", + "updated_by": "nchaulet" + }, + "type": "policies", + "references": [], + "updated_at": "2019-09-20T17:30:22.950Z" + } + } +} + +{ + "type": "doc", + "value": { + "index": ".kibana", + "id": "policies:2", + "source": { + "policies": { + "name": "Policy", + "description": "Amazing policy", + "status": "active", + "updated_on": "2019-09-20T17:35:09+0000", + "updated_by": "nchaulet" + }, + "type": "policies", + "references": [], + "updated_at": "2019-09-20T17:30:22.950Z" + } + } +} + +{ + "type": "doc", + "value": { + "index": ".kibana", + "id": "policies:3", + "source": { + "policies": { + "name": "Policy 3", + "description": "Amazing policy", + "status": "active", + "updated_on": "2019-09-20T17:35:09+0000", + "updated_by": "nchaulet" + }, + "type": "policies", + "references": [], + "updated_at": "2019-09-20T17:30:22.950Z" + } + } +} diff --git a/x-pack/test/functional/es_archives/ingest/policies/mappings.json b/x-pack/test/functional/es_archives/ingest/policies/mappings.json new file mode 100644 index 00000000000000..878d6aa58c225e --- /dev/null +++ b/x-pack/test/functional/es_archives/ingest/policies/mappings.json @@ -0,0 +1,1545 @@ +{ + "type": "index", + "value": { + "aliases": { + ".kibana": {} + }, + "index": ".kibana_1", + "mappings": { + "dynamic": "strict", + "_meta": { + "migrationMappingPropertyHashes": { + "ml-telemetry": "257fd1d4b4fdbb9cb4b8a3b27da201e9", + "server": "ec97f1c5da1a19609a60874e5af1100c", + "visualization": "52d7a13ad68a150c4525b292d23e12cc", + "references": "7997cf5a56cc02bdc9c93361bde732b0", + "graph-workspace": "cd7ba1330e6682e9cc00b78850874be1", + "siem-ui-timeline-note": "8874706eedc49059d4cf0f5094559084", + "policies": "1a096b98c98c2efebfdba77cefcfe54a", + "type": "2f4316de49999235636386fe51dc06c1", + "lens": "21c3ea0763beb1ecb0162529706b88c5", + "space": "c5ca8acafa0beaa4d08d014a97b6bc6b", + "infrastructure-ui-source": "ddc0ecb18383f6b26101a2fadb2dab0c", + "sample-data-telemetry": "7d3cfeb915303c9641c59681967ffeb4", + "search": "181661168bbadd1eff5902361e2a0d5c", + "updated_at": "00da57df13e94e9d98437d13ace4bfe0", + "canvas-workpad": "b0a1706d356228dbdcb4a17e6b9eb231", + "map": "23d7aa4a720d4938ccde3983f87bd58d", + "dashboard": "d00f614b29a80360e1190193fd333bab", + "apm-services-telemetry": "07ee1939fa4302c62ddc052ec03fed90", + "metrics-explorer-view": "53c5365793677328df0ccb6138bf3cdd", + "epm": "abf5b64aa599932bd181efc86dce14a7", + "siem-ui-timeline": "6485ab095be8d15246667b98a1a34295", + "agent_events": "8060c5567d33f6697164e1fd5c81b8ed", + "file-upload-telemetry": "0ed4d3e1983d1217a30982630897092e", + "query": "11aaeb7f5f7fa5bb43f25e18ce26e7d9", + "kql-telemetry": "d12a98a6f19a2d273696597547e064ee", + "ui-metric": "0d409297dc5ebe1e3a1da691c6ee32e3", + "url": "c7f66a0df8b1b52f17c28c4adb111105", + "apm-indices": "c69b68f3fe2bf27b4788d4191c1d6011", + "agents": "1c8e942384219bd899f381fd40e407d7", + "migrationVersion": "4a1746014a75ade3a714e1db5763276f", + "inventory-view": "84b320fd67209906333ffce261128462", + "enrollment_api_keys": "90e66b79e8e948e9c15434fdb3ae576e", + "upgrade-assistant-reindex-operation": "a53a20fe086b72c9a86da3cc12dad8a6", + "index-pattern": "66eccb05066c5a89924f48a9e9736499", + "canvas-element": "7390014e1091044523666d97247392fc", + "datasources": "2fed9e9883b9622cd59a73ee5550ef4f", + "maps-telemetry": "a4229f8b16a6820c6d724b7e0c1f729d", + "namespace": "2f4316de49999235636386fe51dc06c1", + "telemetry": "358ffaa88ba34a97d55af0933a117de4", + "siem-ui-timeline-pinned-event": "20638091112f0e14f0e443d512301c29", + "timelion-sheet": "9a2a2748877c7a7b582fef201ab1d4cf", + "config": "87aca8fdb053154f11383fce3dbf3edf", + "upgrade-assistant-telemetry": "56702cec857e0a9dacfb696655b4ff7b", + "lens-ui-telemetry": "509bfa5978586998e05f9e303c07a327" + } + }, + "properties": { + "agent_events": { + "properties": { + "agent_id": { + "type": "keyword" + }, + "data": { + "type": "text" + }, + "message": { + "type": "text" + }, + "payload": { + "type": "text" + }, + "subtype": { + "type": "keyword" + }, + "timestamp": { + "type": "date" + }, + "type": { + "type": "keyword" + } + } + }, + "agents": { + "properties": { + "access_api_key_id": { + "type": "keyword" + }, + "actions": { + "type": "nested", + "properties": { + "created_at": { + "type": "date" + }, + "data": { + "type": "text" + }, + "id": { + "type": "keyword" + }, + "sent_at": { + "type": "date" + }, + "type": { + "type": "keyword" + } + } + }, + "active": { + "type": "boolean" + }, + "enrolled_at": { + "type": "date" + }, + "last_checkin": { + "type": "date" + }, + "last_updated": { + "type": "date" + }, + "local_metadata": { + "type": "text" + }, + "config_id": { + "type": "keyword" + }, + "shared_id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + }, + "updated_at": { + "type": "date" + }, + "user_provided_metadata": { + "type": "text" + }, + "version": { + "type": "keyword" + } + } + }, + "apm-indices": { + "properties": { + "apm_oss": { + "properties": { + "apmAgentConfigurationIndex": { + "type": "keyword" + }, + "errorIndices": { + "type": "keyword" + }, + "metricsIndices": { + "type": "keyword" + }, + "onboardingIndices": { + "type": "keyword" + }, + "sourcemapIndices": { + "type": "keyword" + }, + "spanIndices": { + "type": "keyword" + }, + "transactionIndices": { + "type": "keyword" + } + } + } + } + }, + "apm-services-telemetry": { + "properties": { + "has_any_services": { + "type": "boolean" + }, + "services_per_agent": { + "properties": { + "dotnet": { + "type": "long", + "null_value": 0 + }, + "go": { + "type": "long", + "null_value": 0 + }, + "java": { + "type": "long", + "null_value": 0 + }, + "js-base": { + "type": "long", + "null_value": 0 + }, + "nodejs": { + "type": "long", + "null_value": 0 + }, + "python": { + "type": "long", + "null_value": 0 + }, + "ruby": { + "type": "long", + "null_value": 0 + }, + "rum-js": { + "type": "long", + "null_value": 0 + } + } + } + } + }, + "canvas-element": { + "dynamic": "false", + "properties": { + "@created": { + "type": "date" + }, + "@timestamp": { + "type": "date" + }, + "content": { + "type": "text" + }, + "help": { + "type": "text" + }, + "image": { + "type": "text" + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword" + } + } + } + } + }, + "canvas-workpad": { + "dynamic": "false", + "properties": { + "@created": { + "type": "date" + }, + "@timestamp": { + "type": "date" + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword" + } + } + } + } + }, + "config": { + "dynamic": "true", + "properties": { + "buildNum": { + "type": "keyword" + } + } + }, + "dashboard": { + "properties": { + "description": { + "type": "text" + }, + "hits": { + "type": "integer" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "optionsJSON": { + "type": "text" + }, + "panelsJSON": { + "type": "text" + }, + "refreshInterval": { + "properties": { + "display": { + "type": "keyword" + }, + "pause": { + "type": "boolean" + }, + "section": { + "type": "integer" + }, + "value": { + "type": "integer" + } + } + }, + "timeFrom": { + "type": "keyword" + }, + "timeRestore": { + "type": "boolean" + }, + "timeTo": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "datasources": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + }, + "package": { + "properties": { + "assets": { + "properties": { + "id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "description": { + "type": "keyword" + }, + "name": { + "type": "keyword" + }, + "title": { + "type": "keyword" + }, + "version": { + "type": "keyword" + } + } + }, + "read_alias": { + "type": "keyword" + }, + "streams": { + "properties": { + "config": { + "type": "flattened" + }, + "id": { + "type": "keyword" + }, + "input": { + "properties": { + "config": { + "type": "flattened" + }, + "fields": { + "type": "flattened" + }, + "id": { + "type": "keyword" + }, + "ilm_policy": { + "type": "keyword" + }, + "index_template": { + "type": "keyword" + }, + "ingest_pipelines": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "output_id": { + "type": "keyword" + }, + "processors": { + "type": "keyword" + } + } + } + } + }, + "enrollment_api_keys": { + "properties": { + "active": { + "type": "boolean" + }, + "api_key": { + "type": "binary" + }, + "api_key_id": { + "type": "keyword" + }, + "created_at": { + "type": "date" + }, + "enrollment_rules": { + "type": "nested", + "properties": { + "created_at": { + "type": "date" + }, + "id": { + "type": "keyword" + }, + "ip_ranges": { + "type": "keyword" + }, + "types": { + "type": "keyword" + }, + "updated_at": { + "type": "date" + }, + "window_duration": { + "type": "nested", + "properties": { + "from": { + "type": "date" + }, + "to": { + "type": "date" + } + } + } + } + }, + "expire_at": { + "type": "date" + }, + "name": { + "type": "keyword" + }, + "config_id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + }, + "updated_at": { + "type": "date" + } + } + }, + "epm": { + "properties": { + "installed": { + "type": "nested", + "properties": { + "id": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + } + } + }, + "file-upload-telemetry": { + "properties": { + "filesUploadedTotalCount": { + "type": "long" + } + } + }, + "graph-workspace": { + "properties": { + "description": { + "type": "text" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "numLinks": { + "type": "integer" + }, + "numVertices": { + "type": "integer" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + }, + "wsState": { + "type": "text" + } + } + }, + "index-pattern": { + "properties": { + "fieldFormatMap": { + "type": "text" + }, + "fields": { + "type": "text" + }, + "intervalName": { + "type": "keyword" + }, + "notExpandable": { + "type": "boolean" + }, + "sourceFilters": { + "type": "text" + }, + "timeFieldName": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "type": { + "type": "keyword" + }, + "typeMeta": { + "type": "keyword" + } + } + }, + "infrastructure-ui-source": { + "properties": { + "description": { + "type": "text" + }, + "fields": { + "properties": { + "container": { + "type": "keyword" + }, + "host": { + "type": "keyword" + }, + "pod": { + "type": "keyword" + }, + "tiebreaker": { + "type": "keyword" + }, + "timestamp": { + "type": "keyword" + } + } + }, + "logAlias": { + "type": "keyword" + }, + "logColumns": { + "type": "nested", + "properties": { + "fieldColumn": { + "properties": { + "field": { + "type": "keyword" + }, + "id": { + "type": "keyword" + } + } + }, + "messageColumn": { + "properties": { + "id": { + "type": "keyword" + } + } + }, + "timestampColumn": { + "properties": { + "id": { + "type": "keyword" + } + } + } + } + }, + "metricAlias": { + "type": "keyword" + }, + "name": { + "type": "text" + } + } + }, + "inventory-view": { + "properties": { + "autoBounds": { + "type": "boolean" + }, + "autoReload": { + "type": "boolean" + }, + "boundsOverride": { + "properties": { + "max": { + "type": "integer" + }, + "min": { + "type": "integer" + } + } + }, + "customOptions": { + "type": "nested", + "properties": { + "field": { + "type": "keyword" + }, + "text": { + "type": "keyword" + } + } + }, + "filterQuery": { + "properties": { + "expression": { + "type": "keyword" + }, + "kind": { + "type": "keyword" + } + } + }, + "groupBy": { + "type": "nested", + "properties": { + "field": { + "type": "keyword" + }, + "label": { + "type": "keyword" + } + } + }, + "metric": { + "properties": { + "type": { + "type": "keyword" + } + } + }, + "name": { + "type": "keyword" + }, + "nodeType": { + "type": "keyword" + }, + "time": { + "type": "integer" + }, + "view": { + "type": "keyword" + } + } + }, + "kql-telemetry": { + "properties": { + "optInCount": { + "type": "long" + }, + "optOutCount": { + "type": "long" + } + } + }, + "lens": { + "properties": { + "expression": { + "type": "keyword", + "index": false + }, + "state": { + "type": "flattened" + }, + "title": { + "type": "text" + }, + "visualizationType": { + "type": "keyword" + } + } + }, + "lens-ui-telemetry": { + "properties": { + "count": { + "type": "integer" + }, + "date": { + "type": "date" + }, + "name": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "map": { + "properties": { + "bounds": { + "type": "geo_shape" + }, + "description": { + "type": "text" + }, + "layerListJSON": { + "type": "text" + }, + "mapStateJSON": { + "type": "text" + }, + "title": { + "type": "text" + }, + "uiStateJSON": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "maps-telemetry": { + "properties": { + "attributesPerMap": { + "properties": { + "dataSourcesCount": { + "properties": { + "avg": { + "type": "long" + }, + "max": { + "type": "long" + }, + "min": { + "type": "long" + } + } + }, + "emsVectorLayersCount": { + "type": "object", + "dynamic": "true" + }, + "layerTypesCount": { + "type": "object", + "dynamic": "true" + }, + "layersCount": { + "properties": { + "avg": { + "type": "long" + }, + "max": { + "type": "long" + }, + "min": { + "type": "long" + } + } + } + } + }, + "mapsTotalCount": { + "type": "long" + }, + "timeCaptured": { + "type": "date" + } + } + }, + "metrics-explorer-view": { + "properties": { + "chartOptions": { + "properties": { + "stack": { + "type": "boolean" + }, + "type": { + "type": "keyword" + }, + "yAxisMode": { + "type": "keyword" + } + } + }, + "currentTimerange": { + "properties": { + "from": { + "type": "keyword" + }, + "interval": { + "type": "keyword" + }, + "to": { + "type": "keyword" + } + } + }, + "name": { + "type": "keyword" + }, + "options": { + "properties": { + "aggregation": { + "type": "keyword" + }, + "filterQuery": { + "type": "keyword" + }, + "groupBy": { + "type": "keyword" + }, + "limit": { + "type": "integer" + }, + "metrics": { + "type": "nested", + "properties": { + "aggregation": { + "type": "keyword" + }, + "color": { + "type": "keyword" + }, + "field": { + "type": "keyword" + }, + "label": { + "type": "keyword" + } + } + } + } + } + } + }, + "migrationVersion": { + "dynamic": "true", + "properties": { + "space": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } + } + }, + "ml-telemetry": { + "properties": { + "file_data_visualizer": { + "properties": { + "index_creation_count": { + "type": "long" + } + } + } + } + }, + "namespace": { + "type": "keyword" + }, + "policies": { + "properties": { + "datasources": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "id": { + "type": "keyword" + }, + "label": { + "type": "keyword" + }, + "name": { + "type": "text" + }, + "status": { + "type": "keyword" + }, + "updated_by": { + "type": "keyword" + }, + "updated_on": { + "type": "keyword" + } + } + }, + "query": { + "properties": { + "description": { + "type": "text" + }, + "filters": { + "type": "object", + "enabled": false + }, + "query": { + "properties": { + "language": { + "type": "keyword" + }, + "query": { + "type": "keyword", + "index": false + } + } + }, + "timefilter": { + "type": "object", + "enabled": false + }, + "title": { + "type": "text" + } + } + }, + "references": { + "type": "nested", + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + }, + "type": { + "type": "keyword" + } + } + }, + "sample-data-telemetry": { + "properties": { + "installCount": { + "type": "long" + }, + "unInstallCount": { + "type": "long" + } + } + }, + "search": { + "properties": { + "columns": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "hits": { + "type": "integer" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "sort": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "server": { + "properties": { + "uuid": { + "type": "keyword" + } + } + }, + "siem-ui-timeline": { + "properties": { + "columns": { + "properties": { + "aggregatable": { + "type": "boolean" + }, + "category": { + "type": "keyword" + }, + "columnHeaderType": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "example": { + "type": "text" + }, + "id": { + "type": "keyword" + }, + "indexes": { + "type": "keyword" + }, + "name": { + "type": "text" + }, + "placeholder": { + "type": "text" + }, + "searchable": { + "type": "boolean" + }, + "type": { + "type": "keyword" + } + } + }, + "created": { + "type": "date" + }, + "createdBy": { + "type": "text" + }, + "dataProviders": { + "properties": { + "and": { + "properties": { + "enabled": { + "type": "boolean" + }, + "excluded": { + "type": "boolean" + }, + "id": { + "type": "keyword" + }, + "kqlQuery": { + "type": "text" + }, + "name": { + "type": "text" + }, + "queryMatch": { + "properties": { + "displayField": { + "type": "text" + }, + "displayValue": { + "type": "text" + }, + "field": { + "type": "text" + }, + "operator": { + "type": "text" + }, + "value": { + "type": "text" + } + } + } + } + }, + "enabled": { + "type": "boolean" + }, + "excluded": { + "type": "boolean" + }, + "id": { + "type": "keyword" + }, + "kqlQuery": { + "type": "text" + }, + "name": { + "type": "text" + }, + "queryMatch": { + "properties": { + "displayField": { + "type": "text" + }, + "displayValue": { + "type": "text" + }, + "field": { + "type": "text" + }, + "operator": { + "type": "text" + }, + "value": { + "type": "text" + } + } + } + } + }, + "dateRange": { + "properties": { + "end": { + "type": "date" + }, + "start": { + "type": "date" + } + } + }, + "description": { + "type": "text" + }, + "favorite": { + "properties": { + "favoriteDate": { + "type": "date" + }, + "fullName": { + "type": "text" + }, + "keySearch": { + "type": "text" + }, + "userName": { + "type": "text" + } + } + }, + "filters": { + "properties": { + "exists": { + "type": "text" + }, + "match_all": { + "type": "text" + }, + "meta": { + "properties": { + "alias": { + "type": "text" + }, + "controlledBy": { + "type": "text" + }, + "disabled": { + "type": "boolean" + }, + "field": { + "type": "text" + }, + "formattedValue": { + "type": "text" + }, + "index": { + "type": "keyword" + }, + "key": { + "type": "keyword" + }, + "negate": { + "type": "boolean" + }, + "params": { + "type": "text" + }, + "type": { + "type": "keyword" + }, + "value": { + "type": "text" + } + } + }, + "missing": { + "type": "text" + }, + "query": { + "type": "text" + }, + "range": { + "type": "text" + }, + "script": { + "type": "text" + } + } + }, + "kqlMode": { + "type": "keyword" + }, + "kqlQuery": { + "properties": { + "filterQuery": { + "properties": { + "kuery": { + "properties": { + "expression": { + "type": "text" + }, + "kind": { + "type": "keyword" + } + } + }, + "serializedQuery": { + "type": "text" + } + } + } + } + }, + "savedQueryId": { + "type": "keyword" + }, + "sort": { + "properties": { + "columnId": { + "type": "keyword" + }, + "sortDirection": { + "type": "keyword" + } + } + }, + "title": { + "type": "text" + }, + "updated": { + "type": "date" + }, + "updatedBy": { + "type": "text" + } + } + }, + "siem-ui-timeline-note": { + "properties": { + "created": { + "type": "date" + }, + "createdBy": { + "type": "text" + }, + "eventId": { + "type": "keyword" + }, + "note": { + "type": "text" + }, + "timelineId": { + "type": "keyword" + }, + "updated": { + "type": "date" + }, + "updatedBy": { + "type": "text" + } + } + }, + "siem-ui-timeline-pinned-event": { + "properties": { + "created": { + "type": "date" + }, + "createdBy": { + "type": "text" + }, + "eventId": { + "type": "keyword" + }, + "timelineId": { + "type": "keyword" + }, + "updated": { + "type": "date" + }, + "updatedBy": { + "type": "text" + } + } + }, + "space": { + "properties": { + "_reserved": { + "type": "boolean" + }, + "color": { + "type": "keyword" + }, + "description": { + "type": "text" + }, + "disabledFeatures": { + "type": "keyword" + }, + "imageUrl": { + "type": "text", + "index": false + }, + "initials": { + "type": "keyword" + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 2048 + } + } + } + } + }, + "telemetry": { + "properties": { + "enabled": { + "type": "boolean" + }, + "lastReported": { + "type": "date" + }, + "lastVersionChecked": { + "type": "keyword", + "ignore_above": 256 + }, + "sendUsageFrom": { + "type": "keyword", + "ignore_above": 256 + }, + "userHasSeenNotice": { + "type": "boolean" + } + } + }, + "timelion-sheet": { + "properties": { + "description": { + "type": "text" + }, + "hits": { + "type": "integer" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "timelion_chart_height": { + "type": "integer" + }, + "timelion_columns": { + "type": "integer" + }, + "timelion_interval": { + "type": "keyword" + }, + "timelion_other_interval": { + "type": "keyword" + }, + "timelion_rows": { + "type": "integer" + }, + "timelion_sheet": { + "type": "text" + }, + "title": { + "type": "text" + }, + "version": { + "type": "integer" + } + } + }, + "type": { + "type": "keyword" + }, + "ui-metric": { + "properties": { + "count": { + "type": "integer" + } + } + }, + "updated_at": { + "type": "date" + }, + "upgrade-assistant-reindex-operation": { + "dynamic": "true", + "properties": { + "indexName": { + "type": "keyword" + }, + "status": { + "type": "integer" + } + } + }, + "upgrade-assistant-telemetry": { + "properties": { + "features": { + "properties": { + "deprecation_logging": { + "properties": { + "enabled": { + "type": "boolean", + "null_value": true + } + } + } + } + }, + "ui_open": { + "properties": { + "cluster": { + "type": "long", + "null_value": 0 + }, + "indices": { + "type": "long", + "null_value": 0 + }, + "overview": { + "type": "long", + "null_value": 0 + } + } + }, + "ui_reindex": { + "properties": { + "close": { + "type": "long", + "null_value": 0 + }, + "open": { + "type": "long", + "null_value": 0 + }, + "start": { + "type": "long", + "null_value": 0 + }, + "stop": { + "type": "long", + "null_value": 0 + } + } + } + } + }, + "url": { + "properties": { + "accessCount": { + "type": "long" + }, + "accessDate": { + "type": "date" + }, + "createDate": { + "type": "date" + }, + "url": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 2048 + } + } + } + } + }, + "visualization": { + "properties": { + "description": { + "type": "text" + }, + "kibanaSavedObjectMeta": { + "properties": { + "searchSourceJSON": { + "type": "text" + } + } + }, + "savedSearchRefName": { + "type": "keyword" + }, + "title": { + "type": "text" + }, + "uiStateJSON": { + "type": "text" + }, + "version": { + "type": "integer" + }, + "visState": { + "type": "text" + } + } + } + } + }, + "settings": { + "index": { + "number_of_replicas": "0", + "number_of_shards": "1" + } + } + } +} diff --git a/x-pack/test_utils/jest/contract_tests/__memorize_snapshots__/example.contract.test.ts.snap b/x-pack/test_utils/jest/contract_tests/__memorize_snapshots__/example.contract.test.ts.snap index 56d50a67e8ff57..f800abbfa2c0c9 100644 --- a/x-pack/test_utils/jest/contract_tests/__memorize_snapshots__/example.contract.test.ts.snap +++ b/x-pack/test_utils/jest/contract_tests/__memorize_snapshots__/example.contract.test.ts.snap @@ -4,3 +4,108 @@ exports['Example contract tests should run online or offline - example_test_snap "serverExists": true } } + +exports['Example contract tests should run online or offline - example_test_snapshot'] = { + "results": { + "serverExists": true + } +} + +exports['Example contract tests should have loaded sample data use esArchive - sample_data'] = { + "results": { + "took": 2, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 4, + "relation": "eq" + }, + "max_score": 0.90445626, + "hits": [ + { + "_index": ".management-beats", + "_type": "_doc", + "_id": "beat:qux", + "_score": 0.90445626, + "_source": { + "beat": { + "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjcmVhdGVkIjoiMjAxOC0wNi0zMFQwMzo0MjoxNS4yMzBaIiwiaWF0IjoxNTMwMzMwMTM1fQ.SSsX2Byyo1B1bGxV8C3G4QldhE5iH87EY_1r21-bwbI", + "active": true, + "host_ip": "1.2.3.4", + "host_name": "foo.bar.com", + "id": "qux", + "name": "qux_filebeat", + "type": "filebeat" + }, + "type": "beat" + } + }, + { + "_index": ".management-beats", + "_type": "_doc", + "_id": "beat:baz", + "_score": 0.90445626, + "_source": { + "beat": { + "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjcmVhdGVkIjoiMjAxOC0wNi0zMFQwMzo0MjoxNS4yMzBaIiwiaWF0IjoxNTMwMzMwMTM1fQ.SSsX2Byyo1B1bGxV8C3G4QldhE5iH87EY_1r21-bwbI", + "active": true, + "host_ip": "22.33.11.44", + "host_name": "baz.bar.com", + "id": "baz", + "name": "baz_metricbeat", + "type": "metricbeat" + }, + "type": "beat" + } + }, + { + "_index": ".management-beats", + "_type": "_doc", + "_id": "beat:foo", + "_score": 0.90445626, + "_source": { + "beat": { + "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjcmVhdGVkIjoiMjAxOC0wNi0zMFQwMzo0MjoxNS4yMzBaIiwiaWF0IjoxNTMwMzMwMTM1fQ.SSsX2Byyo1B1bGxV8C3G4QldhE5iH87EY_1r21-bwbI", + "active": true, + "host_ip": "1.2.3.4", + "host_name": "foo.bar.com", + "id": "foo", + "name": "foo_metricbeat", + "tags": [ + "production", + "qa" + ], + "type": "metricbeat", + "verified_on": "2018-05-15T16:25:38.924Z" + }, + "type": "beat" + } + }, + { + "_index": ".management-beats", + "_type": "_doc", + "_id": "beat:bar", + "_score": 0.90445626, + "_source": { + "beat": { + "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjcmVhdGVkIjoiMjAxOC0wNi0zMFQwMzo0MjoxNS4yMzBaIiwiaWF0IjoxNTMwMzMwMTM1fQ.SSsX2Byyo1B1bGxV8C3G4QldhE5iH87EY_1r21-bwbI", + "active": true, + "host_ip": "11.22.33.44", + "host_name": "foo.com", + "id": "bar", + "name": "bar_filebeat", + "type": "filebeat" + }, + "type": "beat" + } + } + ] + } + } +} diff --git a/x-pack/test_utils/jest/contract_tests/servers.ts b/x-pack/test_utils/jest/contract_tests/servers.ts index 44bad3ea702cc1..66dd0da39bacf2 100644 --- a/x-pack/test_utils/jest/contract_tests/servers.ts +++ b/x-pack/test_utils/jest/contract_tests/servers.ts @@ -64,7 +64,13 @@ export async function _createSharedServer() { const servers = await kbnTestServer.createTestServers({ // adjustTimeout function is required by createTestServers fn adjustTimeout: (t: number) => {}, - settings: TestKbnServerConfig, + settings: { + ...TestKbnServerConfig, + es: { + ...TestKbnServerConfig.es, + esArgs: ['xpack.security.authc.api_key.enabled=true'], + }, + }, }); ESServer = await servers.startES(); const { hosts, username, password } = ESServer; @@ -101,19 +107,21 @@ export function getSharedESServer(): ESServerConfig { export async function createKibanaServer(xpackOption = {}) { if (jest && jest.setTimeout) { // Allow kibana to start - jest.setTimeout(120000); + jest.setTimeout(240000); } const root = kbnTestServer.createRootWithCorePlugins({ elasticsearch: { ...getSharedESServer() }, plugins: { paths: [PLUGIN_PATHS] }, xpack: xpackOption, + oss: false, }); + await root.setup(); await root.start(); const { server } = (root as any).server.legacy.kbnServer; return { - shutdown: () => root.shutdown(), + shutdown: async () => await root.shutdown(), kbnServer: server, root, }; diff --git a/yarn.lock b/yarn.lock index 8348ec12387d86..412c2f9fb2a6ae 100644 --- a/yarn.lock +++ b/yarn.lock @@ -3457,10 +3457,10 @@ resolved "https://registry.yarnpkg.com/@mapbox/whoots-js/-/whoots-js-3.1.0.tgz#497c67a1cef50d1a2459ba60f315e448d2ad87fe" integrity sha512-Es6WcD0nO5l+2BOQS4uLfNPYQaNDfbot3X1XUoloz+x0mPDS3eeORZJl06HXjwBG1fOGwCRnzK88LMdxKRrd6Q== -"@mattapperson/slapshot@1.4.0": - version "1.4.0" - resolved "https://registry.yarnpkg.com/@mattapperson/slapshot/-/slapshot-1.4.0.tgz#d1ba04ad23ca139601fcf7a74ef43ec3dd184d5e" - integrity sha512-P73fBpfevcMKAtDFsq2Z2VVW+tUZ2yNeNje3du5kb+ZH9RJUYFmpo+lCQpTJGyq9lzyhKlF3JIhfoGhz5RzgWg== +"@mattapperson/slapshot@1.4.3": + version "1.4.3" + resolved "https://registry.yarnpkg.com/@mattapperson/slapshot/-/slapshot-1.4.3.tgz#f5b81b297a3708f43f7d9242b46b37c60c1dd9ed" + integrity sha512-5BgwWHAzpethrotEFErzYtWhWyZSq6y+Yek3wzgOquCIqH+/2QoCQT3ru2ina+oIqdtSp3+4BDUMMkCKIa2uhg== dependencies: caller-callsite "^4.0.0" get-caller-file "^2.0.5" @@ -5333,6 +5333,11 @@ dependencies: "@types/sizzle" "*" +"@types/js-search@^1.4.0": + version "1.4.0" + resolved "https://registry.yarnpkg.com/@types/js-search/-/js-search-1.4.0.tgz#f2d4afa176a4fc7b17fb46a1593847887fa1fb7b" + integrity sha1-8tSvoXak/HsX+0ahWThHiH+h+3s= + "@types/js-yaml@^3.11.1", "@types/js-yaml@^3.12.1": version "3.12.1" resolved "https://registry.yarnpkg.com/@types/js-yaml/-/js-yaml-3.12.1.tgz#5c6f4a1eabca84792fbd916f0cb40847f123c656" @@ -5480,6 +5485,13 @@ resolved "https://registry.yarnpkg.com/@types/minimatch/-/minimatch-2.0.29.tgz#5002e14f75e2d71e564281df0431c8c1b4a2a36a" integrity sha1-UALhT3Xi1x5WQoHfBDHIwbSio2o= +"@types/minipass@*": + version "2.2.0" + resolved "https://registry.yarnpkg.com/@types/minipass/-/minipass-2.2.0.tgz#51ad404e8eb1fa961f75ec61205796807b6f9651" + integrity sha512-wuzZksN4w4kyfoOv/dlpov4NOunwutLA/q7uc00xU02ZyUY+aoM5PWIXEKBMnm0NHd4a+N71BMjq+x7+2Af1fg== + dependencies: + "@types/node" "*" + "@types/mocha@^5.2.7": version "5.2.7" resolved "https://registry.yarnpkg.com/@types/mocha/-/mocha-5.2.7.tgz#315d570ccb56c53452ff8638738df60726d5b6ea" @@ -5949,6 +5961,14 @@ dependencies: "@types/node" "*" +"@types/tar@^4.0.3": + version "4.0.3" + resolved "https://registry.yarnpkg.com/@types/tar/-/tar-4.0.3.tgz#e2cce0b8ff4f285293243f5971bd7199176ac489" + integrity sha512-Z7AVMMlkI8NTWF0qGhC4QIX0zkV/+y0J8x7b/RsHrN0310+YNjoJd8UrApCiGBCWtKjxS9QhNqLi2UJNToh5hA== + dependencies: + "@types/minipass" "*" + "@types/node" "*" + "@types/tempy@^0.2.0": version "0.2.0" resolved "https://registry.yarnpkg.com/@types/tempy/-/tempy-0.2.0.tgz#8b7a93f6912aef25cc0b8d8a80ff974151478685" @@ -8649,7 +8669,7 @@ body-parser@1.18.3: raw-body "2.3.3" type-is "~1.6.16" -body-parser@1.19.0, body-parser@^1.18.3: +body-parser@1.19.0, body-parser@^1.18.1, body-parser@^1.18.3: version "1.19.0" resolved "https://registry.yarnpkg.com/body-parser/-/body-parser-1.19.0.tgz#96b2709e57c9c4e09a6fd66a8fd979844f69f08a" integrity sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw== @@ -10521,6 +10541,16 @@ connect-history-api-fallback@^1.6.0: resolved "https://registry.yarnpkg.com/connect-history-api-fallback/-/connect-history-api-fallback-1.6.0.tgz#8b32089359308d111115d81cad3fceab888f97bc" integrity sha512-e54B99q/OUoH64zYYRf3HBP5z24G38h5D3qXu23JGRoigpX5Ss4r9ZnDk3g0Z8uQC2x2lPaJ+UlWBc1ZWBWdLg== +connect@^3.4.0: + version "3.7.0" + resolved "https://registry.yarnpkg.com/connect/-/connect-3.7.0.tgz#5d49348910caa5e07a01800b030d0c35f20484f8" + integrity sha512-ZqRXc+tZukToSNmh5C2iWMSoV3X1YUcPbqEM4DkEG5tNQXrQUZCNVGGv3IuicnkMtPfGf3Xtp8WCXs295iQ1pQ== + dependencies: + debug "2.6.9" + finalhandler "1.1.2" + parseurl "~1.3.3" + utils-merge "1.0.1" + connect@^3.6.0: version "3.6.6" resolved "https://registry.yarnpkg.com/connect/-/connect-3.6.6.tgz#09eff6c55af7236e137135a72574858b6786f524" @@ -14368,6 +14398,13 @@ fbjs@^0.8.4, fbjs@^0.8.9: setimmediate "^1.0.5" ua-parser-js "^0.7.9" +fd-slicer@1.1.0, fd-slicer@~1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/fd-slicer/-/fd-slicer-1.1.0.tgz#25c7c89cb1f9077f8891bbe61d8f390eae256f1e" + integrity sha1-JcfInLH5B3+IkbvmHY85Dq4lbx4= + dependencies: + pend "~1.2.0" + fd-slicer@~1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/fd-slicer/-/fd-slicer-1.0.1.tgz#8b5bcbd9ec327c5041bf9ab023fd6750f1177e65" @@ -14375,13 +14412,6 @@ fd-slicer@~1.0.1: dependencies: pend "~1.2.0" -fd-slicer@~1.1.0: - version "1.1.0" - resolved "https://registry.yarnpkg.com/fd-slicer/-/fd-slicer-1.1.0.tgz#25c7c89cb1f9077f8891bbe61d8f390eae256f1e" - integrity sha1-JcfInLH5B3+IkbvmHY85Dq4lbx4= - dependencies: - pend "~1.2.0" - fecha@^2.3.3: version "2.3.3" resolved "https://registry.yarnpkg.com/fecha/-/fecha-2.3.3.tgz#948e74157df1a32fd1b12c3a3c3cdcb6ec9d96cd" @@ -14598,7 +14628,7 @@ finalhandler@1.1.1: statuses "~1.4.0" unpipe "~1.0.0" -finalhandler@~1.1.2: +finalhandler@1.1.2, finalhandler@~1.1.2: version "1.1.2" resolved "https://registry.yarnpkg.com/finalhandler/-/finalhandler-1.1.2.tgz#b7e7d000ffd11938d0fdb053506f6ebabe9f587d" integrity sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA== @@ -17014,6 +17044,17 @@ http-errors@1.7.2, http-errors@~1.7.2: statuses ">= 1.5.0 < 2" toidentifier "1.0.0" +http-errors@~1.7.0: + version "1.7.3" + resolved "https://registry.yarnpkg.com/http-errors/-/http-errors-1.7.3.tgz#6c619e4f9c60308c38519498c14fbb10aacebb06" + integrity sha512-ZTTX0MWrsQ2ZAhA1cejAwDLycFsd7I7nVtnkT3Ol0aqodaKW+0CTZDQ1uBv5whptCnc8e8HeRRJxRs0kmm/Qfw== + dependencies: + depd "~1.1.2" + inherits "2.0.4" + setprototypeof "1.1.1" + statuses ">= 1.5.0 < 2" + toidentifier "1.0.0" + http-headers@^3.0.2: version "3.0.2" resolved "https://registry.yarnpkg.com/http-headers/-/http-headers-3.0.2.tgz#5147771292f0b39d6778d930a3a59a76fc7ef44d" @@ -17401,7 +17442,7 @@ inflight@^1.0.4: once "^1.3.0" wrappy "1" -inherits@2, inherits@~2.0.3, inherits@~2.0.4: +inherits@2, inherits@2.0.4, inherits@~2.0.3, inherits@~2.0.4: version "2.0.4" resolved "https://registry.yarnpkg.com/inherits/-/inherits-2.0.4.tgz#0fa2c64f932917c3433a0ded55363aae37416b7c" integrity sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ== @@ -19122,6 +19163,11 @@ js-levenshtein@^1.1.3: resolved "https://registry.yarnpkg.com/js-levenshtein/-/js-levenshtein-1.1.3.tgz#3ef627df48ec8cf24bacf05c0f184ff30ef413c5" integrity sha512-/812MXr9RBtMObviZ8gQBhHO8MOrGj8HlEE+4ccMTElNA/6I3u39u+bhny55Lk921yn44nSZFy9naNLElL5wgQ== +js-search@^1.4.3: + version "1.4.3" + resolved "https://registry.yarnpkg.com/js-search/-/js-search-1.4.3.tgz#23a86d7e064ca53a473930edc48615b6b1c1954a" + integrity sha512-Sny5pf00kX1sM1KzvUC9nGYWXOvBfy30rmvZWeRktpg+esQKedIXrXNee/I2CAnsouCyaTjitZpRflDACx4toA== + js-sha3@0.8.0: version "0.8.0" resolved "https://registry.yarnpkg.com/js-sha3/-/js-sha3-0.8.0.tgz#b9b7a5da73afad7dedd0f8c463954cbde6818840" @@ -21581,6 +21627,16 @@ mochawesome@^4.1.0: strip-ansi "^5.0.0" uuid "^3.3.2" +mock-http-server@1.3.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/mock-http-server/-/mock-http-server-1.3.0.tgz#d2c2ffe65f77d3a4da8302c91d3bf687e5b51519" + integrity sha512-WC1fQ4kfOiiRZZ6IEOispJcfvz66m7VVbVFmnWsv1pOwL3psqYyLQGjFXg//zjPeZ//y/rxa8e2eh1Bb58cN7g== + dependencies: + body-parser "^1.18.1" + connect "^3.4.0" + multiparty "^4.1.2" + underscore "^1.8.3" + module-definition@^3.0.0, module-definition@^3.1.0: version "3.2.0" resolved "https://registry.yarnpkg.com/module-definition/-/module-definition-3.2.0.tgz#a1741d5ddf60d76c60d5b1f41ba8744ba08d3ef4" @@ -21711,6 +21767,16 @@ multimatch@^4.0.0: arrify "^2.0.1" minimatch "^3.0.4" +multiparty@^4.1.2: + version "4.2.1" + resolved "https://registry.yarnpkg.com/multiparty/-/multiparty-4.2.1.tgz#d9b6c46d8b8deab1ee70c734b0af771dd46e0b13" + integrity sha512-AvESCnNoQlZiOfP9R4mxN8M9csy2L16EIbWIkt3l4FuGti9kXBS8QVzlfyg4HEnarJhrzZilgNFlZtqmoiAIIA== + dependencies: + fd-slicer "1.1.0" + http-errors "~1.7.0" + safe-buffer "5.1.2" + uid-safe "2.1.5" + multistream@^2.1.1: version "2.1.1" resolved "https://registry.yarnpkg.com/multistream/-/multistream-2.1.1.tgz#629d3a29bd76623489980d04519a2c365948148c" @@ -24690,6 +24756,11 @@ randexp@0.4.6: discontinuous-range "1.0.0" ret "~0.1.10" +random-bytes@~1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/random-bytes/-/random-bytes-1.0.0.tgz#4f68a1dc0ae58bd3fb95848c30324db75d64360b" + integrity sha1-T2ih3Arli9P7lYSMMDJNt11kNgs= + random-poly-fill@^1.0.1: version "1.0.1" resolved "https://registry.yarnpkg.com/random-poly-fill/-/random-poly-fill-1.0.1.tgz#13634dc0255a31ecf85d4a182d92c40f9bbcf5ed" @@ -25227,6 +25298,20 @@ react-markdown@^4.0.6: unist-util-visit "^1.3.0" xtend "^4.0.1" +react-markdown@^4.2.2: + version "4.3.1" + resolved "https://registry.yarnpkg.com/react-markdown/-/react-markdown-4.3.1.tgz#39f0633b94a027445b86c9811142d05381300f2f" + integrity sha512-HQlWFTbDxTtNY6bjgp3C3uv1h2xcjCSi1zAEzfBW9OwJJvENSYiLXWNXN5hHLsoqai7RnZiiHzcnWdXk2Splzw== + dependencies: + html-to-react "^1.3.4" + mdast-add-list-metadata "1.0.1" + prop-types "^15.7.2" + react-is "^16.8.6" + remark-parse "^5.0.0" + unified "^6.1.5" + unist-util-visit "^1.3.0" + xtend "^4.0.1" + react-moment-proptypes@^1.7.0: version "1.7.0" resolved "https://registry.yarnpkg.com/react-moment-proptypes/-/react-moment-proptypes-1.7.0.tgz#89881479840a76c13574a86e3bb214c4ba564e7a" @@ -30649,6 +30734,13 @@ uid-number@0.0.5: resolved "https://registry.yarnpkg.com/uid-number/-/uid-number-0.0.5.tgz#5a3db23ef5dbd55b81fce0ec9a2ac6fccdebb81e" integrity sha1-Wj2yPvXb1VuB/ODsmirG/M3ruB4= +uid-safe@2.1.5: + version "2.1.5" + resolved "https://registry.yarnpkg.com/uid-safe/-/uid-safe-2.1.5.tgz#2b3d5c7240e8fc2e58f8aa269e5ee49c0857bd3a" + integrity sha512-KPHm4VL5dDXKz01UuEd88Df+KzynaohSL9fBh096KWAxSKZQDI2uBrVqtvRM4rwrIrRRKsdLNML/lnaaVSRioA== + dependencies: + random-bytes "~1.0.0" + ultron@~1.1.0: version "1.1.1" resolved "https://registry.yarnpkg.com/ultron/-/ultron-1.1.1.tgz#9fe1536a10a664a65266a1e3ccf85fd36302bc9c" @@ -30675,6 +30767,11 @@ underscore.string@~3.3.4: sprintf-js "^1.0.3" util-deprecate "^1.0.2" +underscore@^1.8.3: + version "1.9.1" + resolved "https://registry.yarnpkg.com/underscore/-/underscore-1.9.1.tgz#06dce34a0e68a7babc29b365b8e74b8925203961" + integrity sha512-5/4etnCkd9c8gwgowi5/om/mYO5ajCaOgdzj/oW+0eQV9WxKBDZw5+ycmKmeaTXjInS/W0BzpGLo2xR2aBwZdg== + underscore@~1.6.0: version "1.6.0" resolved "https://registry.yarnpkg.com/underscore/-/underscore-1.6.0.tgz#8b38b10cacdef63337b8b24e4ff86d45aea529a8"