Testing GC Heap Counts with Containers

[richlander]

Testing GC Heap Counts with Containers

@Maoni0 asked me do to a test pass to validate the container limits doc we're updating. I thought it would be useful to share the results. The short version is the product did what I expected for the scenarios I tested.

Note: the cgroup values are only read on process startup. Changing these values afterwards will have no effect. Enabling a dynamic scenario could be considered in future.

FYI: @omajid @jkotas

Let's test!

The first set of tests are using my Apple M1 laptop. It's 8 cores but Docker Desktop limits itself to 4 cores by default (and I didn't change that).

I'll show all the steps the first time and then skip for the remaining examples.

dotnet-dump tool: https://docs.microsoft.com/dotnet/core/diagnostics/dotnet-dump

FYI: One of my reviewers mentioned that ps wasn't needed since dotnet-dump ps does the same thing but only lists .NET process, which actually makes the process easier.

1 Core

Note: that I'm using the linux-arm64 version of dotnet-dump. Link: https://aka.ms/dotnet-dump/linux-arm64

% docker run --rm --cpus 1 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
1b27f1306ab066735d4d49fe85bf2265673404883df788ccb013065833c898ac
% docker exec -it aspnetapp bash
/app# cd
~# apt update && apt install -y curl procps
~# curl -Lo dotnet-dump https://aka.ms/dotnet-dump/linux-arm64
~# chmod +x dotnet-dump 
~# ps x
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:01 dotnet aspnetapp.dll
   22 pts/0    Ss     0:00 bash
  496 pts/0    R+     0:00 ps x
~# ./dotnet-dump collect -p 1 -o dump.dmp
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number          
Number of GC Heaps: 1

2 cores

% docker run --rm --cpus 2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 2

2 cores + DOTNET_PROCESSOR_COUNT=3

% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3

2 cores + DOTNET_PROCESSOR_COUNT=10

% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=10 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 4

Note: Docker desktop only gave me 4 cores on my laptop, as mentioned at the top. The DOTNET_PROCESSOR_COUNT=10 value is asking for more cores than the machine has.

3 cores (CPU affinity)

% docker run --rm --cpuset-cpus 0,2,3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3

Low memory; cores unconstrained

% docker run --rm -m 40mb -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 2

Ubconstrained

% docker run --rm -m -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 4

Unconstrained + DOTNET_GcHeapCount=2

% docker run --rm -e DOTNET_GCHeapCount=2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 2

Big machine unconstrained

Note: I've switched to a big 64 core x64 machine in Azure, so am now using the linux-x64 version of dotnet-dump. Link: https://aka.ms/dotnet-dump/linux-x64.

~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number 
Number of GC Heaps: 64

Big machine -- memory constrained

~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number 
Number of GC Heaps: 12

[ghost]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Testing GC Heap Counts with Containers

@Maoni0 asked me do to a test pass to validate a doc we're working on. I thought it would be useful to share the results. The short version is the product did what I expected for the scenarios I tested.

FYI: @omajid @jkotas

Context: dotnet/designs#267

Let's test!

The first set of tests are using my Apple M1 laptop. It's 8 cores but Docker Desktop limits itself to 4 cores by default (and I didn't change that).

I'll show all the steps the first time and then skip for the remaining examples.

dotnet-dump tool: https://docs.microsoft.com/dotnet/core/diagnostics/dotnet-dump

1 Core

Note: that I'm using the linux-arm64 version of dotnet-dump. Linke: https://aka.ms/dotnet-dump/linux-arm64

% docker run --rm --cpus 1 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
1b27f1306ab066735d4d49fe85bf2265673404883df788ccb013065833c898ac
% docker exec -it aspnetapp bash
/app# cd
~# apt update && apt install -y curl procps
~# curl -Lo dotnet-dump https://aka.ms/dotnet-dump/linux-arm64
~# chmod +x dotnet-dump 
~# ps x
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:01 dotnet aspnetapp.dll
   22 pts/0    Ss     0:00 bash
  496 pts/0    R+     0:00 ps x
~# ./dotnet-dump collect -p 1 -o dump.dmp
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number          
Number of GC Heaps: 1

2 cores

% docker run --rm --cpus 2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash

/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 2

2 cores + DOTNET_PROCESSOR_COUNT=3

% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3

2 cores + DOTNET_PROCESSOR_COUNT=10

% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=10 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 4

3 cores (CPU affinity)

% docker run --rm --cpuset-cpus 0,2,3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3

Low memory; cores unconstrained

% docker run --rm -m 40mb -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 2

Ubconstrained

% docker run --rm -m -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 4

Ubconstrained + DOTNET_GcHeapCount=2

% docker run --rm -e DOTNET_GCHeapCount=2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 2

Big machine unconstrained

Note: I've switched to a big 64 core x64 machine in Azure, so am now using the linux-x64 version of dotnet-dump. Link: https://aka.ms/dotnet-dump/linux-x64.

~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number 
Number of GC Heaps: 64

Big machine -- memory constrained

~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number 
Number of GC Heaps: 12

Author:	richlander
Assignees:	-
Labels:	area-Diagnostics-coreclr
Milestone:	-

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Testing GC Heap Counts with Containers

@Maoni0 asked me do to a test pass to validate the container limits doc we're updating. I thought it would be useful to share the results. The short version is the product did what I expected for the scenarios I tested.

FYI: @omajid @jkotas

Let's test!

The first set of tests are using my Apple M1 laptop. It's 8 cores but Docker Desktop limits itself to 4 cores by default (and I didn't change that).

I'll show all the steps the first time and then skip for the remaining examples.

dotnet-dump tool: https://docs.microsoft.com/dotnet/core/diagnostics/dotnet-dump

1 Core

Note: that I'm using the linux-arm64 version of dotnet-dump. Linke: https://aka.ms/dotnet-dump/linux-arm64

% docker run --rm --cpus 1 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
1b27f1306ab066735d4d49fe85bf2265673404883df788ccb013065833c898ac
% docker exec -it aspnetapp bash
/app# cd
~# apt update && apt install -y curl procps
~# curl -Lo dotnet-dump https://aka.ms/dotnet-dump/linux-arm64
~# chmod +x dotnet-dump 
~# ps x
  PID TTY      STAT   TIME COMMAND
    1 ?        Ssl    0:01 dotnet aspnetapp.dll
   22 pts/0    Ss     0:00 bash
  496 pts/0    R+     0:00 ps x
~# ./dotnet-dump collect -p 1 -o dump.dmp
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number          
Number of GC Heaps: 1

2 cores

% docker run --rm --cpus 2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash

/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 2

2 cores + DOTNET_PROCESSOR_COUNT=3

% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3

2 cores + DOTNET_PROCESSOR_COUNT=10

% docker run --rm --cpus 2 -e DOTNET_PROCESSOR_COUNT=10 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 4

3 cores (CPU affinity)

% docker run --rm --cpuset-cpus 0,2,3 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
/# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number
Number of GC Heaps: 3

Low memory; cores unconstrained

% docker run --rm -m 40mb -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 2

Ubconstrained

% docker run --rm -m -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 4

Ubconstrained + DOTNET_GcHeapCount=2

% docker run --rm -e DOTNET_GCHeapCount=2 -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
% docker exec -it aspnetapp bash
# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number  
Number of GC Heaps: 2

Big machine unconstrained

Note: I've switched to a big 64 core x64 machine in Azure, so am now using the linux-x64 version of dotnet-dump. Link: https://aka.ms/dotnet-dump/linux-x64.

~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number 
Number of GC Heaps: 64

Big machine -- memory constrained

~$ docker run --rm -d --name aspnetapp -p 8000:80 mcr.microsoft.com/dotnet/samples:aspnetapp
~$ docker exec -it aspnetapp bash
~# ./dotnet-dump analyze dump.dmp -c "eeheap -gc" -c "exit" | grep Number 
Number of GC Heaps: 12

Author:	richlander
Assignees:	-
Labels:	area-Diagnostics-coreclr, area-GC-coreclr, untriaged
Milestone:	-

[omajid]

Cc @tmds @aslicerh

[cshung]

I am wondering if it makes sense to mention in the doc that these cgroup or job object settings are only read during process startup. Once the GC is initialized, it won't read these settings anymore and therefore it won't pick up any changes.

There is a proposal and a prototype for refreshing these limits on user request, but this is not in the scope of .NET 7. The prototype cannot adjust the number of heaps dynamically, that would be much more involved.

[richlander]

I added a point about that. Good call.

[Maoni0]

@richlander can this be closed?

[richlander]

Yup.