Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Stack monitoring] log more detailed memory information #171060

Closed
rudolf opened this issue Nov 10, 2023 · 2 comments
Closed

[Stack monitoring] log more detailed memory information #171060

rudolf opened this issue Nov 10, 2023 · 2 comments
Assignees
@pgayvallet
Labels
Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@rudolf
Copy link
Contributor

rudolf commented Nov 10, 2023

Our current monitoring collection only contains a subset of what's available from https://nodejs.org/api/process.html#processmemoryusage

  • rss: 4935680 - useful for understanding how much memory nodejs is consuming in total
  • heapTotal - already tracked
  • heapUsed - already tracked
  • external - memory consumed by libuv / c++ extensions
  • arrayBuffers - memory consumed by buffers (subset of external memory)

In addition to information about the nodejs process itself, it could be useful to understand how much resources are consumed by the cgroup (all processes running in this docker container). We already collect Cgroups v2 monitoring data https://github.com/elastic/kibana/pull/147082/files but I'm not sure if it's being ingested by metricbeat or if we're collecting the memory.

Somewhat related issue about better eventloop monitoring #134452
@rudolf rudolf added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. labels Nov 10, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@rudolf rudolf assigned pgayvallet and unassigned Bamieh Nov 28, 2023
@pgayvallet pgayvallet mentioned this issue Nov 29, 2023
Merged
pgayvallet added a commit that referenced this issue Nov 30, 2023
@pgayvallet
Core metrics collection - collect more memory related metrics (#172146)
b323fc9 
## Summary

Part of #171060

Add additional memory metrics for collection:

**process**:
- `process.memory.array_buffers_in_bytes` the
`process.memoryUsage().arrayBuffers` value
- `process.memory.external_in_bytes` the
`process.memoryUsage().external` value

**cgroup**:  (v2 only)
- `os.cgroupMemory.current_in_bytes` value from
`/sys/fs/cgroup/{group}/memory.current`
- `os.cgroupMemory.swap_current_in_bytes` value from
`/sys/fs/cgroup/{group}/memory.swap.current`
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 30, 2023
@pgayvallet
Core metrics collection - collect more memory related metrics (elasti…
8feae01 
…c#172146)

## Summary

Part of elastic#171060

Add additional memory metrics for collection:

**process**:
- `process.memory.array_buffers_in_bytes` the
`process.memoryUsage().arrayBuffers` value
- `process.memory.external_in_bytes` the
`process.memoryUsage().external` value

**cgroup**:  (v2 only)
- `os.cgroupMemory.current_in_bytes` value from
`/sys/fs/cgroup/{group}/memory.current`
- `os.cgroupMemory.swap_current_in_bytes` value from
`/sys/fs/cgroup/{group}/memory.swap.current`

(cherry picked from commit b323fc9)
kibanamachine referenced this issue Nov 30, 2023
@kibanamachine @pgayvallet
[8.11] Core metrics collection - collect more memory related metrics (#…
4131708 
…172146) (#172271)

# Backport

This will backport the following commits from `main` to `8.11`:
- [Core metrics collection - collect more memory related metrics
(#172146)](#172146)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Pierre
Gayvallet","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-11-30T10:47:46Z","message":"Core
metrics collection - collect more memory related metrics (#172146)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana/issues/171060\r\n\r\nAdd additional
memory metrics for collection:\r\n\r\n**process**:\r\n-
`process.memory.array_buffers_in_bytes`
the\r\n`process.memoryUsage().arrayBuffers` value\r\n-
`process.memory.external_in_bytes`
the\r\n`process.memoryUsage().external` value\r\n\r\n**cgroup**: (v2
only)\r\n- `os.cgroupMemory.current_in_bytes` value
from\r\n`/sys/fs/cgroup/{group}/memory.current`\r\n-
`os.cgroupMemory.swap_current_in_bytes` value
from\r\n`/sys/fs/cgroup/{group}/memory.swap.current`","sha":"b323fc90a81dd404686c1858a3c525ae1cc922ae","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","Feature:Stack
Monitoring","backport:prev-minor","v8.12.0"],"number":172146,"url":"https://github.com/elastic/kibana/pull/172146","mergeCommit":{"message":"Core
metrics collection - collect more memory related metrics (#172146)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana/issues/171060\r\n\r\nAdd additional
memory metrics for collection:\r\n\r\n**process**:\r\n-
`process.memory.array_buffers_in_bytes`
the\r\n`process.memoryUsage().arrayBuffers` value\r\n-
`process.memory.external_in_bytes`
the\r\n`process.memoryUsage().external` value\r\n\r\n**cgroup**: (v2
only)\r\n- `os.cgroupMemory.current_in_bytes` value
from\r\n`/sys/fs/cgroup/{group}/memory.current`\r\n-
`os.cgroupMemory.swap_current_in_bytes` value
from\r\n`/sys/fs/cgroup/{group}/memory.swap.current`","sha":"b323fc90a81dd404686c1858a3c525ae1cc922ae"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/172146","number":172146,"mergeCommit":{"message":"Core
metrics collection - collect more memory related metrics (#172146)\n\n##
Summary\r\n\r\nPart of
https://github.com/elastic/kibana/issues/171060\r\n\r\nAdd additional
memory metrics for collection:\r\n\r\n**process**:\r\n-
`process.memory.array_buffers_in_bytes`
the\r\n`process.memoryUsage().arrayBuffers` value\r\n-
`process.memory.external_in_bytes`
the\r\n`process.memoryUsage().external` value\r\n\r\n**cgroup**: (v2
only)\r\n- `os.cgroupMemory.current_in_bytes` value
from\r\n`/sys/fs/cgroup/{group}/memory.current`\r\n-
`os.cgroupMemory.swap_current_in_bytes` value
from\r\n`/sys/fs/cgroup/{group}/memory.swap.current`","sha":"b323fc90a81dd404686c1858a3c525ae1cc922ae"}}]}]
BACKPORT-->

Co-authored-by: Pierre Gayvallet <[email protected]>
@rudolf rudolf closed this as completed Nov 30, 2023
@rudolf rudolf reopened this Nov 30, 2023
@pgayvallet
Copy link
Contributor

pgayvallet commented Nov 30, 2023 •
edited
Loading

Kibana PRs:

Metricbeats PRs:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pgayvallet pgayvallet

Labels
Supportability Improve our (devs, SREs, support eng, users) ability to troubleshoot/self-service product better. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@rudolf @pgayvallet @Bamieh @elasticmachine