Compare Kibana server performance with/without in-built APM

[dmlemeshko]

Test environment:

Bare metal machine EX62
OS Linux, Node v16.13.2

Test scenarios:

DiscoverJourney

• 1 min warm up with 20 conc. users
• 3 min ramping up from 20 to 700 conc. users

CanvasJourney

• 1 min warm up with 20 conc. users
• 3 min ramping up from 20 to 100 conc. users

LensJourney

• 1 min warm up with 20 conc. users
• 3 min ramping up from 20 to 700 conc. users

APM configuration:

ELASTIC_APM_ACTIVE: true,
ELASTIC_APM_CONTEXT_PROPAGATION_ONLY: 'false',
ELASTIC_APM_ENVIRONMENT: 'ci',
ELASTIC_APM_TRANSACTION_SAMPLE_RATE: '1.0',
ELASTIC_APM_MAX_QUEUE_SIZE=20480
ELASTIC_APM_CAPTURE_SPAN_STACK_TRACES=false
ELASTIC_APM_METRICS_INTERVAL=60s

Results are available on Kibana-stats))

Example of 2 end-points called during DiscoverJourney

Full results:

The slowest end-points:

[lizozom]

Wondering if we could repeat this with the current recommended configs?
elastic/kibana#117492

[dmlemeshko]

Wondering if we could repeat this with the current recommended configs?

Here are the results:

I should admit I did not set elastic.apm.propagateTracestate: true since I did not find the way of doing it via env variable. The APM configuration I used for testing can be reviewed here

Results for individual end-points are way better as well:

[lizozom]

@dmlemeshko are these results implying that APM improves overall performance by upto 10% :-) ?
The large variance and negative values in diffs bother me a bit.