Move elasticsearch client creation/access API from core setup to start contract

[pgayvallet]

In #55012 we decided to remove any API that could allow to access/query savedObjects from core setup contract, and move them to the start contract instead.

#55396 plans to do the same thing regarding UiSettings.

Steps:

• Expose Elasticsearch from start and deprecate setup API's Expose Elasticsearch from start and deprecate from setup #59886
• add legacy prefix for API exposed from setup Deprecate es API exposed from setup contract #67596
• Move context.elasticsearch.adminClient/dataClient -> context.elasticsearch.legacy.client Mark elasticsearch client exposed via request context as deprecated #67319
• Refactor Core API (SavedObjectsService, RequestContext, etc.) to use Elasticsearch start API's Populate RequestHandlerContext with start API only  #67579
• Refactor plugins to use Elasticsearch start API's Refactor Plugins to access elasticsearch from CoreStart #59915
• Refactor plugins to create custom Elasticsearch clients from start API's
  Use ES API from start contract #66157

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[rudolf]

I've audited all the x-pack plugins which use the Elasticsearch client from CoreSetup:

• Security constructs an Elasticsearch client with a custom plugin and uses it to register http authentication and authorization and http routes. Once we get a valid licence it also uses the Elalsticsearch client to register privileges with Elasticsearch.
  

  kibana/x-pack/plugins/security/server/plugin.ts

  Lines 115 to 117 in 18c3e8c

  	this.clusterClient = core.elasticsearch.createClient('security', {
  	plugins: [elasticsearchClientPlugin],
  	});

  kibana/x-pack/plugins/security/server/authorization/register_privileges_with_cluster.ts

  Line 60 in ad356f5

  const existingPrivileges = await clusterClient.callAsInternalUser('shield.getPrivilege', {

• Reporting validates the Kibana vs ES max content length settings. According to the comments this should be done "early" presumably to provide an warning before any work is started.

  kibana/x-pack/legacy/plugins/reporting/server/lib/validate/validate_max_content_length.ts

  Lines 23 to 39 in c2a6e75

  	const elasticClusterSettingsResponse = await callAsInternalUser('cluster.getSettings', {
  	includeDefaults: true,
  	});
  	const { persistent, transient, defaults: defaultSettings } = elasticClusterSettingsResponse;
  	const elasticClusterSettings = defaults({}, persistent, transient, defaultSettings);
  	const elasticSearchMaxContent = get(elasticClusterSettings, 'http.max_content_length', '100mb');
  	const elasticSearchMaxContentBytes = numeral().unformat(elasticSearchMaxContent.toUpperCase());
  	const kibanaMaxContentBytes: number = config.get(KIBANA_MAX_SIZE_BYTES_PATH);
  	if (kibanaMaxContentBytes > elasticSearchMaxContentBytes) {
  	// TODO this should simply throw an error and let the handler conver it to a warning mesasge. See validateServerHost.
  	logger.warning(
  	`${KIBANA_MAX_SIZE_BYTES_PATH} (${kibanaMaxContentBytes}) is higher than ElasticSearch's ${ES_MAX_SIZE_BYTES_PATH} (${elasticSearchMaxContentBytes}). ` +
  	`Please set ${ES_MAX_SIZE_BYTES_PATH} in ElasticSearch to match, or lower your ${KIBANA_MAX_SIZE_BYTES_PATH} in Kibana to avoid this warning.`
  	);
  	}

• Telemetry

  kibana/src/plugins/telemetry/server/plugin.ts

  Line 94 in d72de0e

  registerCollection(telemetryCollectionManager, elasticsearch.dataClient);

• Monitoring uses dataClient to create callWithRequest for the legacy shim

  kibana/x-pack/plugins/monitoring/server/plugin.ts

  Line 134 in dacc95f

  esDataClient: core.elasticsearch.dataClient,

  
  

  kibana/x-pack/plugins/monitoring/server/plugin.ts

  Line 376 in dacc95f

  name === 'monitoring' ? cluster : this.legacyShimDependencies.esDataClient;

• Monitoring to get cluster uuid. can be moved as it's an async operation

  kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js

  Line 61 in 398b660

  elasticsearch.adminClient.callAsInternalUser('info').then(data => {

• Rollup creates an Elasticsearch client with a custom plugin and then uses this client from it's registered routes. Rollup would have to use getStartServices to get access to ElasticsearchService

  kibana/x-pack/legacy/plugins/rollup/server/lib/call_with_request_factory/call_with_request_factory.ts

  Line 13 in 86e186c

  return elasticsearchService.createClient('rollup', config);

  kibana/x-pack/legacy/plugins/rollup/server/routes/api/jobs.ts

  Lines 15 to 16 in 86e186c

  	const getJobsHandler: RequestHandler<any, any, any> = async (ctx, request, response) => {
  	const callWithRequest = callWithRequestFactory(deps.elasticsearchService, request);

• Actions accesses the Elasticsearch adminClient from setup to use it in start

  kibana/x-pack/plugins/actions/server/plugin.ts

  Line 156 in 5d3797e

  this.adminClient = core.elasticsearch.adminClient;

• Alerting accesses the Elasticsearch adminClient from setup to use it in start

  kibana/x-pack/plugins/alerting/server/plugin.ts

  Line 95 in c07ff71

  this.adminClient = core.elasticsearch.adminClient;

• APM uses Elasticsearch from setup to create it's own indices

  kibana/x-pack/plugins/apm/server/plugin.ts

  Lines 64 to 75 in 2817d6e

  	// create agent configuration index without blocking setup lifecycle
  	createApmAgentConfigurationIndex({
  	esClient: core.elasticsearch.dataClient,
  	config: currentConfig,
  	logger
  	});
  	// create custom action index without blocking setup lifecycle
  	createApmCustomLinkIndex({
  	esClient: core.elasticsearch.dataClient,
  	config: currentConfig,
  	logger
  	});

• EventLog constructs an object during setup which then initialises it's index during start

  kibana/x-pack/plugins/event_log/server/plugin.ts

  Line 86 in 5d3797e

  this.esContext.initialize();

• Lens registers a task to collect telemetry during setup

  kibana/x-pack/plugins/lens/server/usage/task.ts

  Lines 180 to 187 in 27dd6dd

  	export function telemetryTaskRunner(
  	logger: Logger,
  	core: CoreSetup,
  	config: Observable<{ kibana: { index: string } }>
  	) {
  	return ({ taskInstance }: RunContext) => {
  	const { state } = taskInstance;
  	const callCluster = core.elasticsearch.adminClient.callAsInternalUser;

• Licensing constructs a licence poller during setup

  kibana/x-pack/plugins/licensing/server/plugin.ts

  Lines 97 to 100 in 56041f0

  	const { refresh, license$ } = this.createLicensePoller(
  	dataClient,
  	pollingFrequency.asMilliseconds()
  	);

• ML constructs an Elasticsearch client with custom plugin and then exposes it through the request context

  kibana/x-pack/plugins/ml/server/plugin.ts

  Lines 88 to 96 in d55524c

  	const mlClient = coreSetup.elasticsearch.createClient(PLUGIN_ID, {
  	plugins: [elasticsearchJsPlugin],
  	});
  	coreSetup.http.registerRouteHandlerContext(PLUGIN_ID, (context, request) => {
  	return {
  	mlClient: mlClient.asScoped(request),
  	};
  	});

• OssTelemetry registers a task definition for collecting telemetry during setup and then schedules the tasks to run during start

  kibana/x-pack/plugins/oss_telemetry/server/lib/tasks/index.ts

  Lines 33 to 43 in 91c9652

  	taskManager.registerTaskDefinitions({
  	[VIS_TELEMETRY_TASK]: {
  	title: 'X-Pack telemetry calculator for Visualizations',
  	type: VIS_TELEMETRY_TASK,
  	createTaskRunner({ taskInstance }: { taskInstance: TaskInstance }) {
  	return {
  	run: visualizationsTaskRunner(taskInstance, config, elasticsearch),
  	};
  	},
  	},
  	});

• RemoteClusters seems to have an unused dependency on Elasticsearch during setup

  kibana/x-pack/plugins/remote_clusters/server/plugin.ts

  Lines 40 to 41 in bb86bf2

  	elasticsearch,
  	elasticsearchService,

• SnapshotRestore creates a elasticsearch client with custom plugin during setup and exposes it through the route context

  kibana/x-pack/plugins/snapshot_restore/server/plugin.ts

  Lines 75 to 81 in a05c321

  	const esClientConfig = { plugins: [elasticsearchJsPlugin] };
  	const snapshotRestoreESClient = elasticsearch.createClient('snapshotRestore', esClientConfig);
  	http.registerRouteHandlerContext('snapshotRestore', (ctx, request) => {
  	return {
  	client: snapshotRestoreESClient.asScoped(request),
  	};
  	});

• Spaces client uses Elasticsearch to create a spaces SavedObjectsClient wrapper. It's also used to create a default space saved object document which happens after the licensing plugin emits a valid licence.

  kibana/x-pack/plugins/spaces/server/spaces_service/spaces_service.ts

  Lines 75 to 86 in 8def60e

  	const internalRepository = this.getLegacyAPI().savedObjects.getSavedObjectsRepository(
  	elasticsearch.adminClient.callAsInternalUser,
  	['space']
  	);
  	const callCluster = elasticsearch.adminClient.asScoped(request).callAsCurrentUser;
  	const callWithRequestRepository = this.getLegacyAPI().savedObjects.getSavedObjectsRepository(
  	callCluster,
  	['space']
  	);

  kibana/x-pack/plugins/spaces/server/plugin.ts

  Lines 175 to 180 in 91330d2

  	createDefaultSpace: async () => {
  	return await createDefaultSpace({
  	esClient: core.elasticsearch.adminClient,
  	savedObjects: this.getLegacyAPI().savedObjects,
  	});
  	},

• TaskManager constructs a task manager using the Elasticsearch client, but only starts it in the start lifecycle (currently after all legacy plugins are loaded).

  kibana/x-pack/plugins/task_manager/server/plugin.ts

  Lines 48 to 54 in a02232d

  	createTaskManager(core, {
  	logger,
  	config,
  	elasticsearch,
  	savedObjectsRepository,
  	savedObjectsSerializer: savedObjects.createSerializer(),
  	})

  kibana/x-pack/legacy/plugins/task_manager/server/index.ts

  Lines 61 to 63 in ea7c78d

  	this.kbnServer.afterPluginsInit(() => {
  	taskManagerPlugin.start();
  	});

• Transform creates an elasticsearch client with custom plugin and exposes it through the request handler context

  kibana/x-pack/plugins/transform/server/plugin.ts

  Lines 75 to 82 in bb55e8a

  	const transformClient = elasticsearch.createClient('transform', {
  	plugins: [elasticsearchJsPlugin],
  	});
  	http.registerRouteHandlerContext('transform', (context, request) => {
  	return {
  	dataClient: transformClient.asScoped(request),
  	};
  	});

• UpgradeAssistant pull Elasticsearch client during setup and uses it during start

  kibana/x-pack/plugins/upgrade_assistant/server/plugin.ts

  Line 65 in 899270a

  this.elasticSearchService = elasticsearch;

• Watcher creates an Elasticsearch client with custom plugin and exposes it through the request handler context

  kibana/x-pack/plugins/watcher/server/plugin.ts

  Lines 58 to 64 in b07aa09

  	const config = { plugins: [elasticsearchJsPlugin] };
  	const watcherESClient = elasticsearchService.createClient('watcher', config);
  	http.registerRouteHandlerContext('watcher', (ctx, request) => {
  	return {
  	client: watcherESClient.asScoped(request),
  	};
  	});

• Expressions implements a bulk fetch route

  kibana/src/plugins/expressions/server/legacy.ts

  Line 115 in 454d0a2

  const scopedClient = core.elasticsearch.dataClient.asScoped(request);

• Observability uses dataClient in getScopedAnnotationsClient to get scoped client
  

  kibana/x-pack/plugins/observability/server/lib/annotations/bootstrap_annotations.ts

  Lines 34 to 40 in 399eed7

  	getScopedAnnotationsClient: (request: KibanaRequest) => {
  	return createAnnotationsClient({
  	index,
  	apiCaller: core.elasticsearch.dataClient.asScoped(request).callAsCurrentUser,
  	logger,
  	});
  	},

• APM reporting

  kibana/x-pack/plugins/apm/server/lib/apm_telemetry/index.ts

  Line 60 in 94127d8

  const esClient = core.elasticsearch.dataClient;

• File uploader

  kibana/x-pack/plugins/file_upload/server/plugin.js

  Line 18 in c4458ca

  setElasticsearchClientServices(core.elasticsearch);

OSS usages:

[rudolf]

Several plugins construct Elasticsearch clients with custom plugins and then inject these into their routes by registering a route handler context. To refactor these to get their Elasticsearch client from start gets awkward:

--- a/x-pack/plugins/snapshot_restore/server/plugin.ts
+++ b/x-pack/plugins/snapshot_restore/server/plugin.ts
@@ -72,11 +72,18 @@ export class SnapshotRestoreServerPlugin implements Plugin<void, void, any, any>
       }
     );
 
-    const esClientConfig = { plugins: [elasticsearchJsPlugin] };
-    const snapshotRestoreESClient = elasticsearch.createClient('snapshotRestore', esClientConfig);
+    const getClient = async () => {
+      const esClientConfig = { plugins: [elasticsearchJsPlugin] };
+      const coreStart = (await getStartServices())[0];
+      return coreStart.elasticsearch.createClient('snapshotRestore', esClientConfig);
+    };
+
     http.registerRouteHandlerContext('snapshotRestore', (ctx, request) => {
       return {
-        client: snapshotRestoreESClient.asScoped(request),
+        client: {
+          callAsCurrentUser: async (...args) =>
+            (await getClient()).asScoped(request).callAsCurrentUser(...args),
+          callAsInternalUser: async (...args) =>
+            (await getClient()).asScoped(request).callAsInternalUser(...args),
+        },
       };
     });

Although it would require downstream changes, the other alternative is to expose the client as an async function:

--- a/x-pack/plugins/snapshot_restore/server/plugin.ts
+++ b/x-pack/plugins/snapshot_restore/server/plugin.ts
@@ -72,11 +72,15 @@ export class SnapshotRestoreServerPlugin implements Plugin<void, void, any, any>
       }
     );
 
-    const esClientConfig = { plugins: [elasticsearchJsPlugin] };
-    const snapshotRestoreESClient = elasticsearch.createClient('snapshotRestore', esClientConfig);
+    const getClient = async () => {
+      const esClientConfig = { plugins: [elasticsearchJsPlugin] };
+      const coreStart = (await getStartServices())[0];
+      return coreStart.elasticsearch.createClient('snapshotRestore', esClientConfig);
+    };
+
     http.registerRouteHandlerContext('snapshotRestore', (ctx, request) => {
       return {
-        client: snapshotRestoreESClient.asScoped(request),
+        getClient,
       };
     });

[joshdover]

Several plugins construct Elasticsearch clients with custom plugins and then inject these into their routes by registering a route handler context.

I know it breaks the convention of "register in setup, do things in start", but I think it would make sense if we exposed the registerRequestHandlerContext API during start. The objects that are exposed in context are things that should only be available during start anyways. Even core can't register it's request context with the ES client until start.

WDYT?

[mshustov]

Even core can't register it's request context with the ES client until start.

It would work as long as we start listening on a port after plugins register their context providers. The problems will start to appear when we deprecate async lifecycles, as we can no longer guarantee that the context contains all the registered API. Everything that we can do to avoid this problem is to educate how to register the async API...

const depsPromise = doAsync();
core.registerRequestHandlerContext(() => {
   async getFoo() {
     return (await depsPromise).getFoo()
   }
})
      

[rudolf]

Yeah we could say CoreStart.registerRequestHandlerContext can only be used synchronously from start() so that we know we can start handling requests after all plugins' start return. But most start API's remain available during the entire lifecycle so I think this will become very confusing.

[pgayvallet]

I know it breaks the convention of "register in setup, do things in start", but I think it would make sense if we exposed the registerRequestHandlerContext API during start. The objects that are exposed in context are things that should only be available during start anyways. Even core can't register it's request context with the ES client until start.

registerRouteHandlerContext could also have a reference to core's start contract one way or another, to be able to keep calling it from setup while exposing the start APIs to it, as it's mean to be used/resolved during startup anyway.

http.registerRouteHandlerContext('snapshotRestore',
      async (context, req, res)=> {...}

could become something like

http.registerRouteHandlerContext('snapshotRestore', coreStart => {
         const snapshotRestoreESClient = coreStart.elasticsearch.createClient('snapshotRestore', esClientConfig);
         return async (context, req, res) => ({
            client: snapshotRestoreESClient.asScoped(request),
         })

This is more complicated to implement probably, but would keep the 'register during setup' convention.

We could even replace the coreStart parameter with [coreStart, pluginStartDeps] if we want to.

[joshdover]

This is more complicated to implement probably, but would keep the 'register during setup' convention.

One thing I'd like to note is that the CoreStart interface is plugin-specific. So you'll need to keep a reference to this value for each plugin and use it when building the context. It is currently stored in the PluginWrapper in src/core/plugins/{public,server}/plugin.ts, so you'll just need a way to pipe these values into the ContextService.

[pgayvallet]

Yea, the implementation would requires some low-level refactoring with the contextService.

However as @rudolf said on slack yesterday

@delvedor said the new client supports the entire ES API, so I don't think any plugins should have to create custom clients.
So if this is going away soon, we wouldn't want it to affect Core's API in a lasting way

So probably this registerRouteHandlerContext changes should now longer be an option we want?

[mshustov]

Reporting

Reporting validates the Kibana vs ES max content length settings. According to the comments this should be done "early" presumably to provide a warning before any work is started.

This validation should be moved away from the setup, as we are deprecating async lifecycles. Added a comment to their migration issue #53898 (comment)

Licensing

#67291 migrates to elasticsearch client to the provided from the start contract.

Security

Creates a custom elasticsearch plugin and provides access to it via public API. Switching to elasticsearch client exposed from start requires public API refactoring.
Created issue #66405

Observability & APM

#67263 migrates to the context-provided elasticsearch client and elasticsearch start.

Monitoring

Provides creates a custom es client and provides it to alerting plugin. Added a comment to the migration issue #33671 (comment)

File upload

#67277 switches to es client provided via request handler context.

Telemetry

#67279 an attempt to migrate to es client provided from start contract.

[mshustov]

Not all the API provided from the setup contract has been migrated. The list of leftovers #55397 (comment)
It's up to consumers to prioritize their API migration.