[Heartbeat] Elastic Agent diagnostics ZIP does not contain Synthetics data (Heartbeat error)

[paulb-elastic]

Running Elastic Agent (elastic-agent-complete) 8.8.0, the diagnostics ZIP export does not contain Synthetics related data under the components sub-folders, but instead contains an error.txt file with the contents of context deadline exceeded.

This seems to be ok with 8.7.0, but failing in 8.8.0 (not the same upgraded environment, but separate environments and seen in a few different 8.8.0 instances). Have also tried with 8.8.1-SNAPSHOT and it still has the problem.

Steps to reproduce

• Set up a Private Location with Elastic Agent (elastic-agent-complete image) 8.8.0
• Configure a browser monitor to run on the Private Location
• Go to Fleet > Agents
• Select the Host running the Agent with the Private Location configured
• Go to Diagnostics tab
• Click Request diagnostics .zip button
• Once generated, download and view the ZIP file

Under the components/synthetics-browser-default folder, each sub-folder contains an error.txt file with the contents of context deadline exceeded, instead of other files (like heap.txt, mutex.txt, allocs.txt etc.)

There are also the following Heartbeat errors when attempting to generate the diags (which include references to agentDiagnosticHook):

18:54:59.050
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] panic: runtime error: invalid memory address or nil pointer dereference
18:54:59.050
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x55ed5cac6aad]
18:54:59.051
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] goroutine 1199 [running]:
18:54:59.082
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/beats/v7/libbeat/publisher/processing.(*builder).Processors(0xc0009811e0?)
18:54:59.082
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/beats/v7/libbeat/publisher/processing/default.go:264 +0x2d
18:54:59.082
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).agentDiagnosticHook(0x203001?)
18:54:59.082
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:855 +0x3a
18:54:59.082
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).tryPerformDiagnostics.func1()
18:54:59.082
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:763 +0x183
18:54:59.083
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] created by github.com/elastic/elastic-agent-client/v7/pkg/client.(*clientV2).tryPerformDiagnostics
18:54:59.083
elastic_agent.heartbeat
[elastic_agent.heartbeat][error] github.com/elastic/elastic-agent-client/[email protected]/pkg/client/client_v2.go:760 +0x5c5

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[paulb-elastic]

Updated the description to mention it's still a problem with 8.8.1-SNAPSHOT.

[paulb-elastic]

Post FF testing looks good, e.g. content in the ../components/synthetics-browser-default/synthetics-browser-default folder