[APM] Java agent metrics custom API endpoint

[jasonrhodes]

For "Metrics Phase 2", we will be focusing on displaying custom metrics from the Java agent. To do this, we need our APM Kibana API to expose an endpoint (or multiple?) for 5 new Java agent-specific graphs.

AC:

• API endpoint exists to return data needed for the following 3 Java agent-specific graphs:
  • Memory (remove the existing aggregate memory graph and replace with)
    • Heap - several lines- max, committed and used (area graph)
    • Non-heap - similar to heap
  • Thread
    • Thread count

Note:
These 2 graphs will be added to the endpoint's results once the data is ready to be queried. See elastic/apm-server#2148

• GC
  - Time - multiple lines, a line per pool*
  - Count - same

*@Eyal-Koren is it true that we don't ever expect the GC time graph to ever have more than 4-5 pools, which is why we are okay with showing a line per pool there?

[elasticmachine]

Pinging @elastic/apm-ui

[eyalkoren]

@jasonrhodes never say "ever" 😄
In the JVM implementations I know, there are only two GCs- one for minor collections and one for major collections. So, I think it is safe enough to assume that a JVM will have no more than two.
However, each implementation can have different names for them, so if the filter we apply is wide enough to include data from multiple JVMs of different implementations/versions, we can have more lines on that graph.
I believe that for a MVP, it is ok to assume that the number of GCs will be 2-4. If we see this is not enough in the future, we can add Java agent code that consolidates collectors in known JVMs to only "minor" and "major".

[jasonrhodes]

@Eyal-Koren I still don't have the queries for these 5 graphs so I'm going to take a stab at putting them together tomorrow. Two questions for you:

1. Can you list the exact name of the metrics needed for all 5 graphs here? The keys used in the ES documents, specifically.
2. Does the current version of the Java agent already send all of these metrics so that I can test them all using the integration testing setup?

[eyalkoren]

@jasonrhodes All required metrics are available in Java agent version 1.6.0.

Metric data:

• Memory
  • Heap - jvm.memory.heap.committed/max/used
  • Non-heap - jvm.memory.non_heap.committed/max/used
• Thread count - jvm.thread.count
• GC - those are stored in separate documents for each GC, but on the same index and with same processor event type, so probably doesn't matter. The metric data is under jvm.gc.count/time and the name of the GC is at context.tags.name

Let me know if there's anything else you need.

[eyalkoren]

@roncohen I know we said that we can rely on the new timeseries.instance field for the GC ever-increasing counters, but can we use ECS's agent.ephemeral_id for that? It should be really easy to implement - just compute a random ID in each agent when it starts and send it in the metadata. It doesn't even need to have a specific format. Should save some ingestion CPU for hashing as well I suppose...

[roncohen]

@eyalkoren yes, this could work fine as a start 👍

[jasonrhodes]

All required metrics are available in Java agent version 1.6.0.

@eyalkoren Looks like apm-integration-testing is only pulling version 1.5.1-SNAPSHOT as of right now ... any chance someone can make a 1.6 snapshot that our cloud setup can use?

[eyalkoren]

@jasonrhodes if you refer to the opbeans docker image, I triggered a build that will take the latest agent. Try it out in a couple of hours and see if you get 1.6.1-SNAPSHOT. Otherwise, start with 1.5.1 - it has everything except from the thread count, which is likely the simplest one you have.

@roncohen what do you want to do with that? Create an issue in the apm repo for all agents, or just do it in the Java agent for now? Also, if I send it in the metadata as server.agent.ephemeral_id, is that enough to make its way to the stored metric documents, or is there work to be done on the APM server?

[jasonrhodes]

@Eyal-Koren ok, I just upgraded and still seem to have 1.5 ... can you just let me know when the snapshot is built and ready or give me a way to check on that myself?

For you and for @roncohen : as of now I only have the metrics required to build the queries for 2/5 graphs. When can I expect to have data that will represent everything needed to make all 5 graphs? Hopefully thread count will land tomorrow once the java agent snapshot is built, but what about whatever you and @Eyal-Koren are talking about re: garbage collection? I don't understand enough of it to follow along.

[roncohen]

@eyalkoren Generally, I was trying to avoid adding agent.ephemeral_id now because generic graphing solutions like the upcoming metrics explorer will not be able to rely on it solely. The idea was to use instead rely on elastic/beats#10293. However, since that is still some way out, and we control the graphing for now, let's go with agent.ephemeral_id initially. Since we're focusing on the java agent for now I think we should just implement it in the Java agent. Will you work with the @elastic/apm-server team to have it added to metadata in the appropriate manner?

And let's make sure we get @jasonrhodes the metrics he needs as soon as possible

[eyalkoren]

Related server issue: elastic/apm-server#2148

@jasonrhodes I don't know exactly what you tested. I guessed it was opbeans and I created a docker image that uses Java agent version 1.6.1-SNAPSHOT. I just tried ./scripts/compose.py start <stack-version> --with-opbeans-java and it starts the stack and the opbeans app. I don't think you need to make requests to the app, but if so it should be listening on port 3002. Maybe you need to clear all java opbeans related images from your docker cache for it to pull the latest one. I hope this won't delay you any further.

[jasonrhodes]

@eyalkoren thanks, I'll try again.

@roncohen do you think you can explain to me what the "ephemeral_id" should be used for with regard to the garbage collection queries? I'm not sure how those queries need to be constructed.

[jasonrhodes]

I'm going to modify this ticket to just deal with the Java memory and thread count metrics, and then move on to building out those 3 graphs. We'll circle back to the garbage collection graphs when that data is ready to be queried. Thanks all!