[7.x] [Ingest] Add Fleet & EPM features (#59376)

docs/epm/index.asciidoc

@@ -0,0 +1,146 @@
+[role="xpack"]
+[[epm]]
+== Elastic Package Manager
+
+These are the docs for the Elastic Package Manager (EPM).
+
+
+=== Configuration
+
+The Elastic Package Manager by default access `epr.elastic.co` to retrieve the package. The url can be configured with:
+
+```
+xpack.epm.registryUrl: 'http://localhost:8080'
+```
+
+=== API
+
+The Package Manager offers an API. Here an example on how they can be used.
+
+List installed packages:
+
+```
+curl localhost:5601/api/ingest_manager/epm/packages
+```
+
+Install a package:
+
+```
+curl -X POST localhost:5601/api/ingest_manager/epm/packages/iptables-1.0.4
+```
+
+Delete a package:
+
+```
+curl -X DELETE localhost:5601/api/ingest_manager/epm/packages/iptables-1.0.4
+```
+
+=== Definitions
+
+This section is to define terms used across ingest management.
+
+==== Elastic Agent
+A single, unified agent that users can deploy to hosts or containers. It controls which data is collected from the host or containers and where the data is sent. It will run Beats, Endpoint or other monitoring programs as needed. It can operate standalone or pull a configuration policy from Fleet.
+
+==== Namespace
+A user-specified string that will be used to part of the index name in Elasticsearch. It helps users identify logs coming from a specific environment (like prod or test), an application, or other identifiers.
+
+==== Package
+
+A package contains all the assets for the Elastic Stack. A more detailed definition of a package can be found under https://github.com/elastic/package-registry .
+
+
+== Indexing Strategy
+
+Ingest Management enforces an indexing strategy to allow the system to automically detect indices and run queries on it. In short the indexing strategy looks as following:
+
+```
+{type}-{dataset}-{namespace}
+```
+
+The `{type}` can be `logs` or `metrics`. The `{namespace}` is the part where the user can use free form. The only two requirement are that it has only characters allowed in an Elasticsearch index name and does NOT contain a `-`. The `dataset` is defined by the data that is indexed. The same requirements as for the namespace apply. It is expected that the fields for type, namespace and dataset are part of each event and are constant keywords.
+
+Note: More `{type}`s might be added in the future like `apm` and `endpoint`.
+
+This indexing strategy has a few advantages:
+
+* Each index contains only the fields which are relevant for the dataset. This leads to more dense indices and better field completion.
+* ILM policies can be applied per namespace per dataset.
+* Rollups can be specified per namespace per dataset.
+* Having the namespace user configurable makes setting security permissions possible.
+* Having a global metrics and logs template, allows to create new indices on demand which still follow the convention. This is common in the case of k8s as an example.
+* Constant keywords allow to narrow down the indices we need to access for querying very efficiently. This is especially relevant in environments which a large number of indices or with indices on slower nodes.
+
+=== Ingest Pipeline
+
+The ingest pipelines for a specific dataset will have the following naming scheme:
+
+```
+{type}-{dataset}-{package.version}
+```
+
+As an example, the ingest pipeline for the Nginx access logs is called `logs-nginx.access-3.4.1`. The same ingest pipeline is used for all namespaces. It is possible that a dataset has multiple ingest pipelines in which case a suffix is added to the name.
+
+The version is included in each pipeline to allow upgrades. The pipeline itself is listed in the index template and is automatically applied at ingest time.
+
+=== Templates & ILM Policies
+
+To make the above strategy possible, alias templates are required. For each type there is a basic alias template with a default ILM policy. These default templates apply to all indices which follow the indexing strategy and do not have a more specific dataset alias template. 
+
+The `metrics` and `logs` alias template contain all the basic fields from ECS.
+
+Each type template contains an ILM policy. Modifying this default ILM policy will affect all data covered by the default templates.
+
+The templates for a dataset are called as following:
+
+```
+{type}-{dataset}
+```
+
+The pattern used inside the index template is `{type}-{dataset}-*` to match all namespaces.
+
+=== Defaults
+
+If the Elastic Agent is used to ingest data and only the type is specified, `default` for the namespace is used and `generic` for the dataset.
+
+=== Data filtering
+
+Filtering for data in queries for example in visualizations or dashboards should always be done on the constant keyword fields. Visualizations needing data for the nginx.access dataset should query on `type:logs AND dataset:nginx.access`. As these are constant keywords the prefiltering is very efficient.
+
+=== Security permissions
+
+Security permissions can be set on different levels. To set special permissions for the access on the prod namespace an index pattern as below can be used:
+
+```
+/(logs|metrics)-[^-]+-prod-$/
+```
+
+To set specific permissions on the logs index, the following can be used:
+
+```
+/^(logs|metrics)-.*/
+```
+
+Todo: The above queries need to be tested.
+
+
+
+== Package Manager
+
+=== Package Upgrades
+
+When upgrading a package between a bugfix or a minor version, no breaking changes should happen. Upgrading a package has the following effect:
+
+* Removal of existing dashboards
+* Installation of new dashboards
+* Write new ingest pipelines with the version
+* Write new Elasticsearch alias templates
+* Trigger a rollover for all the affected indices
+
+The new ingest pipeline is expected to still work with the data coming from older configurations. In most cases this means some of the fields can be missing. For this to work, each event must contain the version of config / package it is coming from to make such a decision.
+
+In case of a breaking change in the data structure, the new ingest pipeline is also expected to deal with this change. In case there are breaking changes which cannot be dealt with in an ingest pipeline, a new package has to be created.
+
+Each package lists its minimal required agent version. In case there are agents enrolled with an older version, the user is notified to upgrade these agents as otherwise the new configs cannot be rolled out.
+
+

package.json

@@ -135,6 +135,7 @@
     "@kbn/test-subj-selector": "0.2.1",
     "@kbn/ui-framework": "1.0.0",
     "@kbn/ui-shared-deps": "1.0.0",
+    "@types/tar": "^4.0.3",
     "JSONStream": "1.3.5",
     "abortcontroller-polyfill": "^1.4.0",
     "angular": "^1.7.9",
@@ -449,6 +450,7 @@
     "listr": "^0.14.1",
     "load-grunt-config": "^3.0.1",
     "mocha": "^6.2.2",
+    "mock-http-server": "1.3.0",
     "multistream": "^2.1.1",
     "murmurhash3js": "3.0.1",
     "mutation-observer": "^1.0.3",

src/test_utils/kbn_server.ts

@@ -249,7 +249,7 @@ export function createTestServers({
 
   return {
     startES: async () => {
-      await es.start();
+      await es.start(get(settings, 'es.esArgs', []));
 
       if (['gold', 'trial'].includes(license)) {
         await setupUsers({

x-pack/legacy/plugins/ingest_manager/index.ts

@@ -9,6 +9,7 @@ import {
   OUTPUT_SAVED_OBJECT_TYPE,
   AGENT_CONFIG_SAVED_OBJECT_TYPE,
   DATASOURCE_SAVED_OBJECT_TYPE,
+  PACKAGES_SAVED_OBJECT_TYPE,
 } from '../../../plugins/ingest_manager/server';
 
 // TODO https://github.com/elastic/kibana/issues/46373
@@ -34,6 +35,10 @@ export function ingestManager(kibana: any) {
           isNamespaceAgnostic: true,
           // indexPattern: INDEX_NAMES.INGEST,
         },
+        [PACKAGES_SAVED_OBJECT_TYPE]: {
+          isNamespaceAgnostic: true,
+          // indexPattern: INDEX_NAMES.INGEST,
+        },
       },
       mappings: savedObjectMappings,
     },

x-pack/package.json

@@ -33,7 +33,7 @@
     "@kbn/plugin-helpers": "9.0.2",
     "@kbn/test": "1.0.0",
     "@kbn/utility-types": "1.0.0",
-    "@mattapperson/slapshot": "1.4.0",
+    "@mattapperson/slapshot": "1.4.3",
     "@storybook/addon-actions": "^5.2.6",
     "@storybook/addon-console": "^1.2.1",
     "@storybook/addon-knobs": "^5.2.6",
@@ -69,6 +69,7 @@
     "@types/history": "^4.7.3",
     "@types/jest": "24.0.19",
     "@types/joi": "^13.4.2",
+    "@types/js-search": "^1.4.0",
     "@types/js-yaml": "^3.11.1",
     "@types/jsdom": "^12.2.4",
     "@types/json-stable-stringify": "^1.0.32",
@@ -258,6 +259,7 @@
     "isbinaryfile": "4.0.2",
     "joi": "^13.5.2",
     "jquery": "^3.4.1",
+    "js-search": "^1.4.3",
     "js-yaml": "3.13.1",
     "json-stable-stringify": "^1.0.1",
     "jsonwebtoken": "^8.5.1",

x-pack/plugins/ingest_manager/README.md

@@ -1,20 +1,81 @@
 # Ingest Manager
+## Plugin
+  - No features enabled by default. See the TypeScript type for the [the available plugin configuration options](https://github.com/elastic/kibana/blob/feature-ingest/x-pack/plugins/ingest_manager/common/types/index.ts#L9-L19)
+  - Setting `xpack.ingestManager.enabled=true` is required to enable the plugin. It adds the `DATASOURCE_API_ROUTES` and `AGENT_CONFIG_API_ROUTES` values in [`common/constants/routes.ts`](./common/constants/routes.ts)
+  - Adding `--xpack.ingestManager.epm.enabled=true` will add the EPM API & UI
+  - Adding `--xpack.ingestManager.fleet.enabled=true` will add the Fleet API & UI
+    - [code for adding the routes](https://github.com/elastic/kibana/blob/1f27d349533b1c2865c10c45b2cf705d7416fb36/x-pack/plugins/ingest_manager/server/plugin.ts#L115-L133)
+    - [Integration tests](server/integration_tests/router.test.ts)
+  - Both EPM and Fleet require `ingestManager` be enabled. They are not standalone features.
 
-## Getting started
-See the Kibana docs for [how to set up your dev environment](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#setting-up-your-development-environment), [run Elasticsearch](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-elasticsearch), and [start Kibana](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-kibana).
+## Development
 
-One common workflow is:
+### Getting started
+See the Kibana docs for [how to set up your dev environment](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#setting-up-your-development-environment), [run Elasticsearch](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-elasticsearch), and [start Kibana](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#running-kibana)
 
- 1. `yarn es snapshot`
- 1. In another shell: `yarn start --xpack.ingestManager.enabled=true` (or set in `config.yml`)
 
-## HTTP API
-  1. Nothing by default. If `xpack.ingestManager.enabled=true`, it adds the `DATASOURCE_API_ROUTES` and `AGENT_CONFIG_API_ROUTES` values in [`common/constants/routes.ts`](./common/constants/routes.ts)
-  1. [Integration tests](../../test/api_integration/apis/ingest_manager/endpoints.ts)
-  1. In later versions the EPM and Fleet routes will be added when their flags are enabled. See the [currently disabled logic to add those routes](https://github.com/jfsiii/kibana/blob/feature-ingest-manager/x-pack/plugins/ingest_manager/server/plugin.ts#L86-L90).
+One common development workflow is:
+ - Start Elasticsearch in one shell
+    ```
+    yarn es snapshot -E xpack.security.authc.api_key.enabled=true
+    ```
+ - Start Kibana in another shell
+    ```
+    yarn start --xpack.ingestManager.enabled=true --xpack.ingestManager.epm.enabled=true --xpack.ingestManager.fleet.enabled=true
+    ```
 
-## Plugin architecture
-Follows the `common`, `server`, `public` structure from the [Architecture Style Guide
-](https://github.com/elastic/kibana/blob/master/style_guides/architecture_style_guide.md#file-and-folder-structure).
+This plugin follows the `common`, `server`, `public` structure from the [Architecture Style Guide
+](https://github.com/elastic/kibana/blob/master/style_guides/architecture_style_guide.md#file-and-folder-structure). We also follow the pattern of developing feature branches under your personal fork of Kibana.
 
-We use New Platform approach (structure, APIs, etc) where possible. There's a `kibana.json` manifest, and the server uses the `server/{index,plugin}.ts` approach from [`MIGRATION.md`](https://github.com/elastic/kibana/blob/master/src/core/MIGRATION.md#architecture). 
+### API Tests
+#### Ingest & Fleet
+  1. In one terminal, change to the `x-pack` directory and start the test server with
+      ```
+      node scripts/functional_tests_server.js --config test/api_integration/config.ts
+      ```
+
+  1. in a second terminal, run the tests from the Kibana root directory with
+      ```
+      node scripts/functional_test_runner.js --config x-pack/test/api_integration/config.ts
+      ```
+#### EPM
+  1. In one terminal, change to the `x-pack` directory and start the test server with
+      ```
+      node scripts/functional_tests_server.js --config test/epm_api_integration/config.ts
+      ```
+
+  1. in a second terminal, run the tests from the Kibana root directory with
+      ```
+      node scripts/functional_test_runner.js --config x-pack/test/epm_api_integration/config.ts
+      ```
+
+ ### Staying up-to-date with `master`
+ While we're developing in the `feature-ingest` feature branch, here's is more information on keeping up to date with upstream kibana.
+
+<details>
+  <summary>merge upstream <code>master</code> into <code>feature-ingest</code></summary>
+
+```bash
+## checkout feature branch to your fork
+git checkout -B feature-ingest origin/feature-ingest
+
+## make sure your feature branch is current with upstream feature branch
+git pull upstream feature-ingest
+
+## pull in changes from upstream master
+git pull upstream master
+
+## push changes to your remote
+git push origin
+
+# /!\ Open a DRAFT PR /!\
+# Normal PRs will re-notify authors of commits already merged
+# Draft PR will trigger CI run. Once CI is green ...
+# /!\ DO NOT USE THE GITHUB UI TO MERGE THE PR /!\
+
+## push your changes to upstream feature branch from the terminal; not GitHub UI
+git push upstream
+```
+</details>
+
+See https://github.com/elastic/kibana/pull/37950 for an example.

x-pack/plugins/ingest_manager/common/constants/agent.ts

@@ -0,0 +1,16 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export const AGENT_SAVED_OBJECT_TYPE = 'agents';
+
+export const AGENT_EVENT_SAVED_OBJECT_TYPE = 'agent_events';
+
+export const AGENT_TYPE_PERMANENT = 'PERMANENT';
+export const AGENT_TYPE_EPHEMERAL = 'EPHEMERAL';
+export const AGENT_TYPE_TEMPORARY = 'TEMPORARY';
+
+export const AGENT_POLLING_THRESHOLD_MS = 30000;
+export const AGENT_POLLING_INTERVAL = 1000;

x-pack/plugins/ingest_manager/common/constants/agent_config.ts

@@ -3,16 +3,17 @@
  * or more contributor license agreements. Licensed under the Elastic License;
  * you may not use this file except in compliance with the Elastic License.
  */
-import { AgentConfigStatus } from '../types';
+import { AgentConfigStatus, DefaultPackages } from '../types';
 
 export const AGENT_CONFIG_SAVED_OBJECT_TYPE = 'agent_configs';
 
-export const DEFAULT_AGENT_CONFIG_ID = 'default';
-
 export const DEFAULT_AGENT_CONFIG = {
   name: 'Default config',
   namespace: 'default',
   description: 'Default agent configuration created by Kibana',
   status: AgentConfigStatus.Active,
   datasources: [],
+  is_default: true,
 };
+
+export const DEFAULT_AGENT_CONFIGS_PACKAGES = [DefaultPackages.system];

x-pack/plugins/ingest_manager/common/constants/enrollment_api_key.ts

@@ -0,0 +1,7 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export const ENROLLMENT_API_KEYS_SAVED_OBJECT_TYPE = 'enrollment_api_keys';

x-pack/plugins/ingest_manager/common/constants/epm.ts

@@ -0,0 +1,8 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export const PACKAGES_SAVED_OBJECT_TYPE = 'epm-package';
+export const INDEX_PATTERN_SAVED_OBJECT_TYPE = 'index-pattern';

x-pack/plugins/ingest_manager/common/constants/index.ts

@@ -6,6 +6,9 @@
 export * from './plugin';
 export * from './routes';
 
+export * from './agent';
 export * from './agent_config';
 export * from './datasource';
+export * from './epm';
 export * from './output';
+export * from './enrollment_api_key';

x-pack/plugins/ingest_manager/common/constants/output.ts

@@ -7,12 +7,10 @@ import { OutputType } from '../types';
 
 export const OUTPUT_SAVED_OBJECT_TYPE = 'outputs';
 
-export const DEFAULT_OUTPUT_ID = 'default';
-
 export const DEFAULT_OUTPUT = {
-  name: DEFAULT_OUTPUT_ID,
+  name: 'default',
+  is_default: true,
   type: OutputType.Elasticsearch,
   hosts: [''],
-  ingest_pipeline: DEFAULT_OUTPUT_ID,
   api_key: '',
 };

x-pack/plugins/ingest_manager/common/constants/plugin.ts

@@ -3,5 +3,4 @@
  * or more contributor license agreements. Licensed under the Elastic License;
  * you may not use this file except in compliance with the Elastic License.
  */
-
 export const PLUGIN_ID = 'ingestManager';