proposal for correlating synthetics traces

[vigneshshanmugam]

• Address Correlate Synthetics monitor execution with APM  #823
• Adds the docs around Integrating/Crosslinking the Synthetics waterfall traces with the APM UI and vice versa.
• Design is still being worked on https://github.com/elastic/apm-dev/issues/1052 and See previous discussion on this topic - https://github.com/elastic/apm-dev/issues/1024

[felixbarny]

Another challenge is that native OTel agents don't capture all HTTP headers by default, IINM.

[vigneshshanmugam]

I believe we cant do much here except the users would see a rootless trace unless Synthetics itself is also instrumented using APM?

I can add this to the limitation list if its a huge concern?

[felixbarny]

I don't think it's a huge concern but we should still add it to the limitations.
The concern is not necessarily about the trace being rootless. I think it's more about that we aren't able to tell synthetics requests apart from normal requests. I think that's a limitation we can live with but it doesn't allow us to build features like "show only requests from synthetics".

[gregkalapos]

The requirement level of the user_agent.original attribute (which is the Value of the HTTP User-Agent header sent by the client.) is recommended. I also think this is trivial to capture, so I expect most OTel implementations will have this out of the box.

Furthermore, what we could also do is this: for some languages, it's likely we'll end up with custom OTel distributions - with that we have control on how the OTel agent is configured. To be on a safe-side, we could add something like this to this spec:

Custom Elastic OTel agent distributions MUST capture the user_agent.original attribute in order to enable APM-Synthetic correlation.

In most cases, this will have no effect, since I expect vanilla OTel agents already doing so. Nevertheless with this we can mitigate the issue of not being able to recognize synthetic calls in OTel Agents.

[felixbarny]

A couple of smaller nits and suggestions. Overall LGTM.

[felixbarny]

I don't think it's a huge concern but we should still add it to the limitations.
The concern is not necessarily about the trace being rootless. I think it's more about that we aren't able to tell synthetics requests apart from normal requests. I think that's a limitation we can live with but it doesn't allow us to build features like "show only requests from synthetics".

[gregkalapos]

Looks very nice 👍

As already discussed on slack, I agree with the traceparent approach instead of using baggage in order to mitigate the challenges caused by potentially unsampled traces.

Left some comments.