[Fleet] APM Server managed by Elastic Agent with Fleet - 7.12

[jalvz]

This is a continuation of #4004.
This issue will be regularly updated.

First public release

• Variables in data stream names spec/64, [Fleet] Add service name to datasets #4492, kibana/88307, #4674
• Dynamic config reloading #4376
• Default and/or configurable ILM policies #4620
• Short term solution for Central config management and Sourcemaps apm-server#4670
• Documentation #4494
• Screenshots #4495
• Add new default index patterns in Kibana app kibana/87501

Internal

• Complete apm-integration-testing support apm-integration-testing/1004
• Manual test plan #4493
• Package Integration tests

Bundling

• Bundle apm-server with Elastic Agent beats/23545

Other

• Investigate short term alternative for sourcemap enrich processor

Bugs

• Forbid more than 1 APM integration per policy #4539
• Kibana asset counter doesn't include ES assets kibana/85968
• YAML parsing error kibana/91401

[axw]

Central config and API Keys for APM agents

@ruflin a while back we talked about the need for providing APM Server with additional API Keys. In particular, it will need privileges for:

• the Kibana APM app (https://www.elastic.co/guide/en/apm/server/current/privileges-agent-central-config.html)
• managing API Keys for APM Agent auth (https://www.elastic.co/guide/en/apm/server/current/privileges-api-key.html) (Silvia pointed out to me that we only need privileges for creating API Keys, and that's not necessary for the integration.)

We discussed an enhancement whereby Fleet would create and inject API Keys into APM Server's config, and manage the lifetime of those keys (deleting on agent/policy removal). Is there an existing issue we can link to?

[ruflin]

@axw Endpoint does something similar where they have an API key in their config. @ph @kevinlog Which API do they take here currently and how is it generated and managed?

[axw]

@ruflin after further pondering, I think ideally an integration should be able to list additional privileges that are either required for installation or are optional along with an explanation of what they're used for. Users can then opt out (foregoing some functionality) of those at integration installation time. The privileges would be all on the one API Key provided to the integration. Such API Keys should probably be more narrowly scoped to the integration being run by an Agent, not shared by all integrations run by the Agent.

WDYT?

[ruflin]

In this scenario, the API Key with additional privileges would be an additional API besides the one we already ship down for output I assume? Would this additional API Key be the same for a single instance of an integration or different per Elastic Agent like the output keys?

[axw]

In this scenario, the API Key with additional privileges would be an additional API besides the one we already ship down for output I assume?

It could also be the same, but as mentioned above we must then ensure it's not shared with other integration processes.

Would this additional API Key be the same for a single instance of an integration or different per Elastic Agent like the output keys?

I don't think it matters for APM Server.
I'll create a more specific issue to continue this discussion, and perhaps we can follow that up with a feature request on Fleet.

[axw]

@ruflin let's continue discussing API Keys with additional privileges in #4573

[jalvz]

Update 13/01

Highlights of the year so far, in no particular order:

• We got 2 PRs adding capabilities to apm-integration-testing: One already merged to start a package-registry container with any package distribution and optionally a local apm package, and another still WIP to install the apm integration automatically.
  We might need to follow up with something to pass a local apm-server binary to the elastic agent, but apm-integration-testing is looking quite complete now.

• After discussing a proposal, there is a PR open to the spec to change the index-pattern generated for APM templates. This will need Kibana support, and then is just a matter of appending the service name to the dataset in apm-server code.

• A new version 0.1.0-dev.3 was released to the snapshot distribution with a bug fix to amend the secret token type, and another will be released soon with another bug fix to amend wrong templates for labels fields. Besides these problems, and a missing bit in Kibana, and a duplicated bit in beats, manual testing of the APM integration went generally well.

• Finally, a couple of other unimportant, small PRs have been merged in apm-server, and I opened an issue in Kibana to add the index patterns to match the indexing strategy.

The biggest unknowns now are: the sourcemap enrich processor needs ingest node support (with unknown ETA), central config is still much in the air, and we need to figure out how to bundle agents in apm-server.

[jalvz]

Update 10/02

Long due update, here we go:

• Apm-server is now bundled with Elastic Agent in all installations.

• Support for for a short term solution for central config and sourcemaps is about to be merged (waiting Beats approval). This requires users to generate and enter API keys in the policy editor, which is not ideal. Thinking long term we should get better Fleet support for specific privileges.
  Regarding sourcemaps, we will rely on Ingest Node. Recently the Elasticsearch team merged a fingerprint processor that we needed for this.

• kibana/89870 has been merged adding support for service names in datasets. Server side PR is up, waiting on an up-to-date Kibana in the snapshot registry for testing.

• We found a bug in Kibana, somewhat hard to reproduce.

• The APM UI team is working on updating the default index names.

• A couple of issues will miss the 7.12 release:

  • Change logic in Kibana to check for APM setup kibana/86462
  • Fleet API key renewal kibana/85777

• Finally, we (well, mainly Felix) researched the option of bundling apm agents with apm server, and it turns out it is going to be very complicated. Not the bundling itself, but telling the user where the APM agent is. Because of this, we will go down the route of downloading the java agent attacher from Maven central, and running it for the user (targeting 7.13), and look at k8s later.