The memory usage of jetstream pod is large [v2.10.20]

[AllenZMC]

Observed behavior

The memory usage of jetstream pod is large and remains at 2Gi. At this time, the jetstream stream does not have any memory data. Why is this?

In addition, when the pod is restarted, the memory usage will be very small, about tens of MB.

What is the reason for this?

Expected behavior

When the jetstream stream does not have any memory data, it will theoretically reduce memory usage.

Server and client version

nats-server: v2.10.20

Host environment

No response

Steps to reproduce

No response

[wallyqs]

if the system is running in k8s, need to make sure that you are setting GOMEMLIMIT so that that GC does not work based on the host memory instead.

[neilalexander]

Can you please provide memory profiles so we can see what's going on?

Either nats server request profile allocs using the system account, or https://docs.nats.io/running-a-nats-service/nats_admin/profiling.

[AllenZMC]

Can you please provide memory profiles so we can see what's going on?

Either nats server request profile allocs using the system account, or https://docs.nats.io/running-a-nats-service/nats_admin/profiling.

@neilalexander
How to configure prof_port = 65432 in jetstream？
The jetstream pod fails to run with an error message: "This parameter is not recognized"

[neilalexander]

It's a configuration option on the NATS Server config file, if you are doing it under Kubernetes you may need to do something like this:

config:
  merge:
     prof_port: 65432

[neilalexander]

Also note that you don't need prof_port enabled to use the NATS CLI, you just need access to the system account.

[AllenZMC]

mem.prof.zip
@neilalexander Here is the memory profile.

[derekcollison]

Thanks @AllenZMC! @neilalexander will take a look.

[neilalexander]

Thanks for the profile, was this taken when the memory usage was inflated? It is only showing 30MB heap allocations. Are you using GOMEMLIMIT?

[AllenZMC]

Thanks for the profile, was this taken when the memory usage was inflated? It is only showing 30MB heap allocations. Are you using GOMEMLIMIT?

It is indeed obtained when the memory is very large. I also feel confused.

And what is GOMEMLIMIT? NATS runs in a container. Is this configuration in the Go program or somewhere else?

[derekcollison]

How are you measuring memory?

[AllenZMC]

How are you measuring memory?

container_memory_working_set_bytes{pod='a', namespace='b', container='c'}

[derekcollison]

Is that RSS or VSS?

[AllenZMC]

Is that RSS or VSS?

container_memory_working_set_bytes is a metric of the k8s container platform, and it is the amount of physical memory actually used by the container, excluding memory that can be reclaimed, not RSS or VSS.