[Fleet] Support for APM Agent Central Config with zero configuration

[axw]

Agent central config depends on APM Server querying Kibana, and having privileges as laid out in https://www.elastic.co/guide/en/apm/server/current/privileges-agent-central-config.html.

When running under Fleet, APM Server will by default not have the necessary privileges to query central config. We will have to either inject secondary credentials for querying Kibana, or somehow extend the privileges of the primary API Key provided by Fleet to APM Server.

Users should not need to care about configuring API Keys for this; they should rather be concerned about enabling or disabling the central config feature, which implicitly updates privileges.

Alternatively: we can reimplement APM Agent Central Config to use Fleet. We would need to:

• Update the APM app in Kibana to store agent config in (and retrieve from) Fleet policies
  • This config would be pushed down to APM Server, which would store it in memory.
  • Updating APM Agent config must not cause APM Server to restart.
• Add a means for APM Server to record status for each APM Agent config block (so we can see if the config has been applied)

[simitt]

Created a Kibana issue to track work for this for Fleet elastic/kibana#89311

[axw]

Reopening and changing the description so we don't lose sight of enabling this feature without manually injecting API Keys.

[axw]

We will have to either inject secondary credentials for querying Kibana, or somehow extend the privileges of the primary API Key provided by Fleet to APM Server.

Fleet provides extension points for the package policy create and update APIs.

Information courtesy of Paul Tavares:

API hooks available for Package policies (aka: integration policies): one for Create and another for Update. To use them, a dependency must be set on your kibana Plugin to Fleet, which will then expose FleetStartContract during the start phase of the plugin's lifecycle. This interface includes a method named registerExternalCallback() which allows you to register a callback function for the following:

fleetStart.registerExternalCallback('packagePolicyCraete', async () => {});
and 
fleetStart.registerExternalCallback('packagePolicyUpdate', async () => {});
The Types for the callbacks supported are defined here: https://github.com/elastic/kibana/blob/a99ccc27d7a9d2ad7be162c21f545e8eebf59c69/x-pack/plugins/fleet/server/plugin.ts#L132-L148 

So what we should be able to do is have the APM app hook into these API calls, and inject one or more additional API Keys. This could be done without elastic/kibana#89311.

Alternatively/additionally we could use the same mechanism to inject APM agent config into APM package policies. When a package policy is created or updated we would inject the APM agent config through the hook, and in subsequent changes to APM agent config we would update the package policy to keep them in sync. With this kind of approach we would need an alternative means of marking agent config as "applied".

I think the latter approach is probably what we should aim for, but in the near term we should keep the number of changes down and inject additional privileges. This will be necessary anyway for tail-based sampling.

[simitt]

So what we should be able to do is have the APM app hook into these API calls, and inject one or more additional API Keys. This could be done without elastic/kibana#89311.

In scenarios where the Kibana API Key needs to be revoked, I believe it is fair to let the users revoke it manually. But we would need a way to convince the hook to actually create a new one.

[simitt]

No direct work on the APM Server side, created elastic/kibana#93420 for the suggested solution.

[axw]

We have elastic/kibana#93420 for the interim solution, and I've just opened #5018 and elastic/kibana#95501 for the final solution. Closing this one, let's iterate on those issues as needed.

[axw]

See elastic/kibana#93420 (comment) for proof of concepts