Merge branch 'master' into apm-update-header-layout

dev_docs/key_concepts/saved_objects.mdx

@@ -0,0 +1,74 @@
+---
+id: kibDevDocsSavedObjectsIntro
+slug: /kibana-dev-docs/saved-objects-intro
+title: Saved Objects
+summary: Saved Objects are a key concept to understand when building a Kibana plugin.
+date: 2021-02-02
+tags: ['kibana','dev', 'contributor', 'api docs']
+---
+
+"Saved Objects" are developer defined, persisted entities, stored in the Kibana system index (which is also sometimes referred to as the `.kibana` index).
+The Saved Objects service allows Kibana plugins to use Elasticsearch like a primary database. Think of it as an Object Document Mapper for Elasticsearch.
+ Some examples of Saved Object types are dashboards, lens, canvas workpads, index patterns, cases, ml jobs, and advanced settings. Some Saved Object types are 
+ exposed to the user in the [Saved Object management UI](https://www.elastic.co/guide/en/kibana/current/managing-saved-objects.html), but not all.
+
+Developers create and manage their Saved Objects using the SavedObjectClient, while other data in Elasticsearch should be accessed via the data plugin's search
+services.
+
+![image](../assets/saved_object_vs_data_indices.png)
+
+
+<DocLink id="kibDevTutorialSavedObject" text="Tutorial: Register a new Saved Object type"/>
+
+## References
+
+In order to support import and export, and space-sharing capabilities, Saved Objects need to explicitly list any references they contain to other Saved Objects.
+The parent should have a reference to it's children, not the other way around. That way when a "parent" is exported (or shared to a space),
+ all the "children" will be automatically included. However, when a "child" is exported, it will not include all "parents".
+
+<DocLink id="kibDevTutorialSavedObject" section="references" text="Learn how to define Saved Object references"/>
+
+## Migrations and Backward compatibility
+
+As your plugin evolves, you may need to change your Saved Object type in a breaking way (for example, changing the type of an attribtue, or removing
+an attribute). If that happens, you should write a migration to upgrade the Saved Objects that existed prior to the change.
+
+<DocLink id="kibDevTutorialSavedObject" section="migrations" text="How to write a migration"/>.
+
+## Security
+
+Saved Objects can be secured using Kibana's Privileges model, unlike data that comes from data indices, which is secured using Elasticsearch's Privileges model.
+
+### Space awareness
+
+Saved Objects are "space aware". They exist in the space they were created in, and any spaces they have been shared with.
+
+### Feature controls and RBAC
+
+Feature controls provide another level of isolation and shareability for Saved Objects. Admins can give users and roles read, write or none permissions for each Saved Object type.
+
+### Object level security (OLS) 
+
+OLS is an oft-requested feature that is not implemented yet. When it is, it will provide users with even more sharing and privacy flexibility. Individual
+objects can be private to the user, shared with a selection of others, or made public. Much like how sharing Google Docs works.
+
+## Scalability
+
+By default all saved object types go into a single index. If you expect your saved object type to have a lot of unique fields, or if you expect there
+to be many of them, you can have your objects go in a separate index by using the `indexPattern` field. Reporting and task manager are two 
+examples of features that use this capability.
+
+## Searchability
+
+Because saved objects are stored in system indices, they cannot be searched like other data can. If you see the phrase “[X] as data” it is 
+referring to this searching limitation. Users will not be able to create custom dashboards using saved object data, like they would for data stored
+in Elasticsearch data indices.
+
+## Saved Objects by value
+
+Sometimes Saved Objects end up persisted inside another Saved Object. We call these Saved Objects “by value”, as opposed to "by
+ reference". If an end user creates a visualization and adds it to a dashboard without saving it to the visualization 
+ library, the data ends up nested inside the dashboard Saved Object. This helps keep the visualization library smaller. It also avoids
+ issues with edits propagating - since an entity can only exist in a single place.
+  Note that from the end user stand point, we don’t use these terms “by reference” and “by value”.
+

dev_docs/tutorials/saved_objects.mdx

@@ -0,0 +1,250 @@
+---
+id: kibDevTutorialSavedObject
+slug: /kibana-dev-docs/tutorial/saved-objects
+title: Register a new saved object type
+summary: Learn how to register a new saved object type.
+date: 2021-02-05
+tags: ['kibana','onboarding', 'dev', 'architecture', 'tutorials']
+---
+
+Saved Object type definitions should be defined in their own `my_plugin/server/saved_objects` directory.
+
+The folder should contain a file per type, named after the snake_case name of the type, and an index.ts file exporting all the types.
+
+**src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts**
+
+```ts
+import { SavedObjectsType } from 'src/core/server';
+
+export const dashboardVisualization: SavedObjectsType = {
+  name: 'dashboard_visualization', [1]
+  hidden: false,
+  namespaceType: 'single',
+  mappings: {
+    dynamic: false,
+    properties: {
+      description: {
+        type: 'text',
+      },
+      hits: {
+        type: 'integer',
+      },
+    },
+  },
+  migrations: {
+    '1.0.0': migratedashboardVisualizationToV1,
+    '2.0.0': migratedashboardVisualizationToV2,
+  },
+};
+```
+
+[1] Since the name of a Saved Object type forms part of the url path for the public Saved Objects HTTP API, 
+these should follow our API URL path convention and always be written as snake case.
+
+**src/plugins/my_plugin/server/saved_objects/index.ts**
+
+```ts
+export { dashboardVisualization } from './dashboard_visualization';
+export { dashboard } from './dashboard';
+```
+
+**src/plugins/my_plugin/server/plugin.ts**
+
+```ts
+import { dashboard, dashboardVisualization } from './saved_objects';
+
+export class MyPlugin implements Plugin {
+  setup({ savedObjects }) {
+    savedObjects.registerType(dashboard);
+    savedObjects.registerType(dashboardVisualization);
+  }
+}
+```
+
+## Mappings
+
+Each Saved Object type can define its own Elasticsearch field mappings. Because multiple Saved Object
+types can share the same index, mappings defined by a type will be nested under a top-level field that matches the type name.
+
+For example, the mappings defined by the dashboard_visualization Saved Object type:
+
+**src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts**
+
+```ts
+import { SavedObjectsType } from 'src/core/server';
+
+export const dashboardVisualization: SavedObjectsType = {
+  name: 'dashboard_visualization',
+  ...
+  mappings: {
+    properties: {
+      dynamic: false,
+      description: {
+        type: 'text',
+      },
+      hits: {
+        type: 'integer',
+      },
+    },
+  },
+  migrations: { ... },
+};
+```
+
+Will result in the following mappings being applied to the .kibana index:
+
+```ts
+{
+  "mappings": {
+    "dynamic": "strict",
+    "properties": {
+      ...
+      "dashboard_vizualization": {
+        "dynamic": false,
+        "properties": {
+          "description": {
+            "type": "text",
+          },
+          "hits": {
+            "type": "integer",
+          },
+        },
+      }
+    }
+  }
+}
+```
+Do not use field mappings like you would use data types for the columns of a SQL database. Instead, field mappings are analogous to a 
+SQL index. Only specify field mappings for the fields you wish to search on or query. By specifying `dynamic: false`
+ in any level of your mappings, Elasticsearch will accept and store any other fields even if they are not specified in your mappings.
+
+Since Elasticsearch has a default limit of 1000 fields per index, plugins should carefully consider the 
+fields they add to the mappings. Similarly, Saved Object types should never use `dynamic: true` as this can cause an arbitrary
+ amount of fields to be added to the .kibana index.
+
+ ## References
+
+Declare <DocLink id="kibDevDocsSavedObjectsIntro" section="References" text="Saved Object references"/> by adding an id, type and name to the
+ `references` array.
+
+```ts
+router.get(
+  { path: '/some-path', validate: false },
+  async (context, req, res) => {
+    const object = await context.core.savedObjects.client.create(
+      'dashboard',
+      {
+        title: 'my dashboard',
+        panels: [
+          { visualization: 'vis1' }, [1]
+        ],
+        indexPattern: 'indexPattern1'
+      },
+      { references: [
+          { id: '...', type: 'visualization', name: 'vis1' },
+          { id: '...', type: 'index_pattern', name: 'indexPattern1' },
+        ]
+      }
+    )
+    ...
+  }
+);
+```
+[1] Note how `dashboard.panels[0].visualization` stores the name property of the reference (not the id directly) to be able to uniquely 
+identify this reference. This guarantees that the id the reference points to always remains up to date. If a
+ visualization id was directly stored in `dashboard.panels[0].visualization` there is a risk that this id gets updated without 
+ updating the reference in the references array.
+
+## Writing migrations
+
+Saved Objects support schema changes between Kibana versions, which we call migrations. Migrations are
+ applied when a Kibana installation is upgraded from one version to the next, when exports are imported via
+  the Saved Objects Management UI, or when a new object is created via the HTTP API.
+
+Each Saved Object type may define migrations for its schema. Migrations are specified by the Kibana version number, receive an input document,
+ and must return the fully migrated document to be persisted to Elasticsearch.
+
+Let’s say we want to define two migrations: - In version 1.1.0, we want to drop the subtitle field and append it to the title - In version
+ 1.4.0, we want to add a new id field to every panel with a newly generated UUID.
+
+First, the current mappings should always reflect the latest or "target" schema. Next, we should define a migration function for each step in the schema evolution:
+
+**src/plugins/my_plugin/server/saved_objects/dashboard_visualization.ts**
+
+```ts
+import { SavedObjectsType, SavedObjectMigrationFn } from 'src/core/server';
+import uuid from 'uuid';
+
+interface DashboardVisualizationPre110 {
+  title: string;
+  subtitle: string;
+  panels: Array<{}>;
+}
+interface DashboardVisualization110 {
+  title: string;
+  panels: Array<{}>;
+}
+
+interface DashboardVisualization140 {
+  title: string;
+  panels: Array<{ id: string }>;
+}
+
+const migrateDashboardVisualization110: SavedObjectMigrationFn<
+  DashboardVisualizationPre110, [1]
+  DashboardVisualization110
+> = (doc) => {
+  const { subtitle, ...attributesWithoutSubtitle } = doc.attributes;
+  return {
+    ...doc, [2]
+    attributes: {
+      ...attributesWithoutSubtitle,
+      title: `${doc.attributes.title} - ${doc.attributes.subtitle}`,
+    },
+  };
+};
+
+const migrateDashboardVisualization140: SavedObjectMigrationFn<
+  DashboardVisualization110,
+  DashboardVisualization140
+> = (doc) => {
+  const outPanels = doc.attributes.panels?.map((panel) => {
+    return { ...panel, id: uuid.v4() };
+  });
+  return {
+    ...doc,
+    attributes: {
+      ...doc.attributes,
+      panels: outPanels,
+    },
+  };
+};
+
+export const dashboardVisualization: SavedObjectsType = {
+  name: 'dashboard_visualization', [1]
+  /** ... */
+  migrations: {
+    // Takes a pre 1.1.0 doc, and converts it to 1.1.0
+    '1.1.0': migrateDashboardVisualization110,
+
+    // Takes a 1.1.0 doc, and converts it to 1.4.0
+    '1.4.0': migrateDashboardVisualization140, [3]
+  },
+};
+```
+[1] It is useful to define an interface for each version of the schema. This allows TypeScript to ensure that you are properly handling the input and output 
+ types correctly as the schema evolves.
+
+[2] Returning a shallow copy is necessary to avoid type errors when using different types for the input and output shape.
+
+[3] Migrations do not have to be defined for every version. The version number of a migration must always be the earliest Kibana version
+ in which this migration was released. So if you are creating a migration which will
+ be part of the v7.10.0 release, but will also be backported and released as v7.9.3, the migration version should be: 7.9.3.
+
+ Migrations should be written defensively, an exception in a migration function will prevent a Kibana upgrade from succeeding and will cause downtime for our users. 
+ Having said that, if a 
+ document is encountered that is not in the expected shape, migrations are encouraged to throw an exception to abort the upgrade. In most scenarios, it is better to
+  fail an upgrade than to silently ignore a corrupt document which can cause unexpected behaviour at some future point in time.
+
+It is critical that you have extensive tests to ensure that migrations behave as expected with all possible input documents. Given how simple it is to test all the branch 
+conditions in a migration function and the high impact of a bug in this code, there’s really no reason not to aim for 100% test code coverage.

docs/development/core/public/kibana-plugin-core-public.doclinksstart.links.md

@@ -91,7 +91,9 @@ readonly links: {
             readonly scriptAggs: string;
             readonly painless: string;
             readonly painlessApi: string;
+            readonly painlessLangSpec: string;
             readonly painlessSyntax: string;
+            readonly painlessWalkthrough: string;
             readonly luceneExpressions: string;
         };
         readonly indexPatterns: {
@@ -131,6 +133,7 @@ readonly links: {
             openIndex: string;
             putComponentTemplate: string;
             painlessExecute: string;
+            painlessExecuteAPIContexts: string;
             putComponentTemplateMetadata: string;
             putWatch: string;
             updateTransform: string;