From bc4a845dce600506e99f45c7b0d8ef908f3f14ff Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 17:06:14 -0700 Subject: [PATCH 01/13] Add --cpus context --- accepted/2019/support-for-memory-limits.md | 49 +++++++++++++++------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index f43f2a614..858236596 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -2,14 +2,16 @@ **Owner** [Rich Lander](https://github.com/richlander) -.NET Core has support for [control groups](https://en.wikipedia.org/wiki/Cgroups) (cgroups), which is the basis of [Docker limits](https://docs.docker.com/config/containers/resource_constraints/). We found that the algorithm we use to honor cgroups works well for larger memory size limits (for example, >500MB), but that it is not possible to configure a .NET Core application to run indefinitely at lower memory levels. This document proposes an approach to support low memory size limits, <100MB. +.NET Core has support for [control groups](https://en.wikipedia.org/wiki/Cgroups) (cgroups), which is the basis of [Docker limits](https://docs.docker.com/config/containers/resource_constraints/). We found that the algorithm we use to honor cgroups works well for larger memory size limits (for example, >500MB), but that it is not possible to configure a .NET Core application to run indefinitely at lower memory levels. This document proposes an approach to support low memory size limits, for example <100MB. -Note: Windows has a concept similar to cgroups called [job objects](https://docs.microsoft.com/windows/desktop/ProcThread/job-objects). .NET Core should honor job objects in the same way as cgroups, as appropriate. This document will focus on cgroups throughout. +Note: Windows has a concept similar to cgroups called [job objects](https://docs.microsoft.com/windows/desktop/ProcThread/job-objects). .NET 6+ correctly honors job objects in the same way as cgroups. This document will focus on cgroups throughout. -It is critical to provide effective and well defined experiences when .NET Core applications are run within memory-limited cgroups. An application should run indefinitely given a sensible configuration for that application. We considered relying on orchestators to manage failing applications (that can no longer satisfy the configuration of a cgroup), but believe this to be antithetical as a primary solution for building reliable systems. We also expect that there are scenarios where orchestrators will be unavailable or primitive or hardware will be constrainted, and therefore not tolerant of frequently failing applications. As a result, we need a better tuned algorithm for cgroup support to the end of running reliable software within constrained environments. +It is critical to provide effective and well defined experiences when .NET Core applications are run within memory-limited cgroups. An application should run indefinitely given a sensible configuration for that application. We considered relying on orchestators to manage failing applications (that can no longer satisfy the configuration of a cgroup), but believe this to be antithetical as a primary solution for building reliable systems. We also expect that there are scenarios where orchestrators will be unavailable or primitive or hardware will be constrained, and therefore not tolerant of frequently failing applications. As a result, we need a better tuned algorithm for cgroup support to the end of running reliable software within constrained environments. See [implementing hard limit for GC heap dotnet/coreclr #22180](https://github.com/dotnet/coreclr/pull/22180). +See [Validate container improvements with .NET 6](https://github.com/dotnet/runtime/issues/53149). + ## GC Heap Hard Limit The following configuration knobs will be exposed to enable developers to configure their applications: @@ -29,29 +31,44 @@ The GC will more aggressive perform GCs as the GC heap grows closer to the `GCHe The GC will throw an `OutOfMemoryException` for allocations that would cause the committed heap size to exceed the `GCHeapHardLimit` memory size, even after a full compacting GC. -## GC Heap Heap Minimum Size +## GC Heap Minimum Size Using Server GC, there are multiple GC heaps created, up to one per core. This model doesn't scale well when a small memory limit is set on a machine with many cores. -The minimum _reserved_ segment size per heap: 16mb +The minimum _reserved_ segment size per heap: `16 MiB` + +Example -- CPU unconstrained: -Example: +```bash +docker run --rm -m 256mb mcr.microsoft.com/dotnet/samples +``` * 48 core machine -* cgroup has a 200MB memory limit +* cgroup has a 256MiB memory limit * cgroup has no CPU/core limit -* 160MB `GCHeapHardLimit` -* Server GC will create 10 GC heaps -* All 48 cores can be used by the application +* 192MB `GCHeapHardLimit` +* Server GC will create 12 GC heaps, with 16 MiB reserved memory +* All 48 cores can be used by the application, per [container policy](https://docs.docker.com/config/containers/resource_constraints/#cpu) + +`heaps = (256 * .75) / 16` +`heaps = 12` -Example: +Example -- CPU constrained: + +```bash +docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples +``` * 48 core machine -* cgroup has a 200MB memory limit -* cgroup has 4 CPU/core limit -* 160MB `GCHeapHardLimit` -* Server GC will create 4 GC heaps -* Only 4 cores can be used by the application +* cgroup has a 256MiB memory limit +* cgroup has 2 CPU/core limit +* 192MB `GCHeapHardLimit` +* Server GC will create 2 GC heaps, with 16 MiB reserved memory +* Only 2 cores can be used by the application + +There are other scenarios, like using `--cpuset-cpus` but they all follow from these two examples. + +If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. ## Previous behavior From 172b30d4baa9aa450c7432a425148c2b0e692ef1 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 17:15:59 -0700 Subject: [PATCH 02/13] Add comment about .NET Framework --- accepted/2019/support-for-memory-limits.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 858236596..56eb4b13f 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -70,6 +70,8 @@ There are other scenarios, like using `--cpuset-cpus` but they all follow from t If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. +Note: .NET Framework has the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. + ## Previous behavior Previously, the **maximum GC heap size** matched the cgroup limit. From 1558f19ef63b92736811139bd7ad5eed959ef972 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 17:22:14 -0700 Subject: [PATCH 03/13] Clarify CPU affinity --- accepted/2019/support-for-memory-limits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 56eb4b13f..5936e20f6 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -66,7 +66,7 @@ docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples * Server GC will create 2 GC heaps, with 16 MiB reserved memory * Only 2 cores can be used by the application -There are other scenarios, like using `--cpuset-cpus` but they all follow from these two examples. +There are other scenarios, like using `--cpuset-cpus` (CPU affinity) but they all follow from these two examples. If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. From e7527c966af84c48036d0ff5f27c44b8ed472782 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 20:27:28 -0700 Subject: [PATCH 04/13] Update accepted/2019/support-for-memory-limits.md Co-authored-by: Jan Kotas --- accepted/2019/support-for-memory-limits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 5936e20f6..30c29aec0 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -35,7 +35,7 @@ The GC will throw an `OutOfMemoryException` for allocations that would cause the Using Server GC, there are multiple GC heaps created, up to one per core. This model doesn't scale well when a small memory limit is set on a machine with many cores. -The minimum _reserved_ segment size per heap: `16 MiB` +The minimum _reserved_ memory size per heap: `16 MiB` Example -- CPU unconstrained: From 8aced4a638bbfd40251671a2133c7dc5c3d61097 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 21:27:02 -0700 Subject: [PATCH 05/13] Update accepted/2019/support-for-memory-limits.md --- accepted/2019/support-for-memory-limits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 30c29aec0..7b9d8df9c 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -70,7 +70,7 @@ There are other scenarios, like using `--cpuset-cpus` (CPU affinity) but they al If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. -Note: .NET Framework has the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. +Note: [.NET Framework 4.8 and 4.8.1](https://github.com/microsoft/dotnet-framework-docker/discussions/935) have the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. ## Previous behavior From d6b79ac52892ca17b04795a842014d769fb469ed Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 22:14:42 -0700 Subject: [PATCH 06/13] Add more context --- accepted/2019/support-for-memory-limits.md | 76 +++++++++++++++++----- 1 file changed, 58 insertions(+), 18 deletions(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 7b9d8df9c..0a02c1924 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -1,27 +1,45 @@ -# .NET Core GC Support for Docker Limits +# .NET GC Support for Docker Limits **Owner** [Rich Lander](https://github.com/richlander) -.NET Core has support for [control groups](https://en.wikipedia.org/wiki/Cgroups) (cgroups), which is the basis of [Docker limits](https://docs.docker.com/config/containers/resource_constraints/). We found that the algorithm we use to honor cgroups works well for larger memory size limits (for example, >500MB), but that it is not possible to configure a .NET Core application to run indefinitely at lower memory levels. This document proposes an approach to support low memory size limits, for example <100MB. +.NET has support for [control groups (cgroups)](https://en.wikipedia.org/wiki/Cgroups) , which is the basis of [Docker resource limits](https://docs.docker.com/config/containers/resource_constraints/). .NET has supported cgroups since .NET Core 2.1. -Note: Windows has a concept similar to cgroups called [job objects](https://docs.microsoft.com/windows/desktop/ProcThread/job-objects). .NET 6+ correctly honors job objects in the same way as cgroups. This document will focus on cgroups throughout. +Windows has a concept similar to cgroups called [job objects](https://docs.microsoft.com/windows/desktop/ProcThread/job-objects). .NET 6+ correctly honors job objects in the same way as cgroups. This document will focus on cgroups throughout. -It is critical to provide effective and well defined experiences when .NET Core applications are run within memory-limited cgroups. An application should run indefinitely given a sensible configuration for that application. We considered relying on orchestators to manage failing applications (that can no longer satisfy the configuration of a cgroup), but believe this to be antithetical as a primary solution for building reliable systems. We also expect that there are scenarios where orchestrators will be unavailable or primitive or hardware will be constrained, and therefore not tolerant of frequently failing applications. As a result, we need a better tuned algorithm for cgroup support to the end of running reliable software within constrained environments. +It is critical to provide effective and well-defined capabilities for .NET applications within memory-limited cgroups. An application should run indefinitely given a sensible configuration for that application. It is important that .NET developers have good controls to optimize their container hosted applications. Our goal is that certain classes of .NET applications can be run with <100 MiB memory constraints. -See [implementing hard limit for GC heap dotnet/coreclr #22180](https://github.com/dotnet/coreclr/pull/22180). +Related: -See [Validate container improvements with .NET 6](https://github.com/dotnet/runtime/issues/53149). +- [implementing hard limit for GC heap dotnet/coreclr #22180](https://github.com/dotnet/coreclr/pull/22180). +- [Validate container improvements with .NET 6](https://github.com/dotnet/runtime/issues/53149). + +## Cgroup constraints + +cgroups control two main resources: memory and cores. Both are relevant to the .NET GC. + +Memory constraints defines the maximum memory available to the cgroup. This memory is used by the guest operating system, the .NET runtime, the GC heap, and potentially other users. If a cgroup has `100 MiB` available, the app will have less than that. The cgroup will be terminated (AKA `OOMKilled`) when the memory limit is reached. + +Core constraints determine how many GC heaps should be created, at maximum. The maximum heap value matches `Environment.ProcessorCount`. There are three primary ways that this value can be set (using the `docker` CLI to demonstrate): + +- Not specified -- `Environment.ProcessorCount` will match the total number of machine cores. +- Via `--cpus` -- `Environment.ProcessorCount` uses that (decimal) value (rounded up to the next integer). +- Via `--cpu-sets` -- `Environment.ProcessorCount` matches the count of specified CPUs. +- `DOTNET_PROCESSOR_COUNT` -- `Environment.ProcessorCount` uses this value. If other values are also specified, they are ignored. + +In the general case, there will be one heap per core. If the GC creates too many heaps, that can over-eagerly use up the memory limit, at least in part. There are controls to avoid that. ## GC Heap Hard Limit -The following configuration knobs will be exposed to enable developers to configure their applications: +The *GC Heap Hard Limit* is the maximum managed heap size. It only applies when running within a cgroup. By default, it is lower than the cgroup memory constraint (AKA "the cgroup hard limit"). + +The following configuration knobs are exposed to configure applications: -* `GCHeapHardLimit` - specifies a hard limit for the GC heap as an absolute value -* `GCHeapHardLimitPercent` - specifies a hard limit for the GC heap as a percentage of physical memory that the process is allowed to use +* `GCHeapHardLimit` - specifies a hard limit for the GC heap as an absolute value, in bytes (hex value). +* `GCHeapHardLimitPercent` - specifies a hard limit for the GC heap as a percentage of the cgroup hard limit. If both are specified, `GCHeapHardLimit` is used. -The `GCHeapHardLimit` will be calculated using the following formular if it is not specified and the process is running inside a container (or cgroup or job object) with a memory limit specified: +By default, the `GCHeapHardLimit` will be calculated using the following formula: ```console max (20mb, 75% of the memory limit on the container) @@ -31,13 +49,19 @@ The GC will more aggressive perform GCs as the GC heap grows closer to the `GCHe The GC will throw an `OutOfMemoryException` for allocations that would cause the committed heap size to exceed the `GCHeapHardLimit` memory size, even after a full compacting GC. -## GC Heap Minimum Size +## GC Heap Count Using Server GC, there are multiple GC heaps created, up to one per core. This model doesn't scale well when a small memory limit is set on a machine with many cores. The minimum _reserved_ memory size per heap: `16 MiB` -Example -- CPU unconstrained: +If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. + +Note: [.NET Framework 4.8 and 4.8.1](https://github.com/microsoft/dotnet-framework-docker/discussions/935) have the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. + +Let's look at some examples. + +### Memory constrained; CPU unconstrained ```bash docker run --rm -m 256mb mcr.microsoft.com/dotnet/samples @@ -53,7 +77,7 @@ docker run --rm -m 256mb mcr.microsoft.com/dotnet/samples `heaps = (256 * .75) / 16` `heaps = 12` -Example -- CPU constrained: +### Memory and CPU constrained ```bash docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples @@ -66,12 +90,28 @@ docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples * Server GC will create 2 GC heaps, with 16 MiB reserved memory * Only 2 cores can be used by the application -There are other scenarios, like using `--cpuset-cpus` (CPU affinity) but they all follow from these two examples. +### Memory and CPU constrained (with CPU affinity): -If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. +```bash +docker run --rm -m 256mb --cpuset-cpus 0,2,3 mcr.microsoft.com/dotnet/samples +``` -Note: [.NET Framework 4.8 and 4.8.1](https://github.com/microsoft/dotnet-framework-docker/discussions/935) have the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. +* 48 core machine +* cgroup has a 256MiB memory limit +* cgroup has 3 CPU/core limit +* 192MB `GCHeapHardLimit` +* Server GC will create 3 GC heaps, with 16 MiB reserved memory +* Only 3 cores can be used by the application -## Previous behavior +### Memory and CPU constrained (overriden by `DOTNET_PROCESSOR_COUNT`): -Previously, the **maximum GC heap size** matched the cgroup limit. +```bash +docker run --rm -m 256mb --cpus 2 -e DOTNET_PROCESSOR_COUNT=4 mcr.microsoft.com/dotnet/samples +``` + +* 48 core machine +* cgroup has a 256MiB memory limit +* cgroup has 2 CPU/core limit +* 192MB `GCHeapHardLimit` +* Server GC will create 4 GC heaps, with 16 MiB reserved memory +* Only 2 cores can be used by the application From 593dfa680bc0b66e40b7092dc8a227a5153ba8f4 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 22:19:13 -0700 Subject: [PATCH 07/13] Add GCHeapCount --- accepted/2019/support-for-memory-limits.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 0a02c1924..0935dc94b 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -53,7 +53,12 @@ The GC will throw an `OutOfMemoryException` for allocations that would cause the Using Server GC, there are multiple GC heaps created, up to one per core. This model doesn't scale well when a small memory limit is set on a machine with many cores. -The minimum _reserved_ memory size per heap: `16 MiB` +The heap count can be set two ways: + +- Manually via `DOTNET_GCHeapCount`. +- Automatically by the GC, relying on: + - Number of observed or configured cores. + - A minimum _reserved_ memory size per heap of `16 MiB`. If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. From 99192e0870f37a26be86800418f9d6ec18f5eb8a Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 22:24:12 -0700 Subject: [PATCH 08/13] Update accepted/2019/support-for-memory-limits.md Co-authored-by: Jan Kotas --- accepted/2019/support-for-memory-limits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 0935dc94b..8cebd2ec7 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -35,7 +35,7 @@ The *GC Heap Hard Limit* is the maximum managed heap size. It only applies when The following configuration knobs are exposed to configure applications: * `GCHeapHardLimit` - specifies a hard limit for the GC heap as an absolute value, in bytes (hex value). -* `GCHeapHardLimitPercent` - specifies a hard limit for the GC heap as a percentage of the cgroup hard limit. +* `GCHeapHardLimitPercent` - specifies a hard limit for the GC heap as a percentage of the cgroup hard limit (hex value). If both are specified, `GCHeapHardLimit` is used. From 9f0970c5c07e7e9f3d2a6392c6d09a10f19b35d3 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Mon, 27 Jun 2022 22:25:47 -0700 Subject: [PATCH 09/13] MiB -> MB --- accepted/2019/support-for-memory-limits.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 8cebd2ec7..abafd7e3f 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -6,7 +6,7 @@ Windows has a concept similar to cgroups called [job objects](https://docs.microsoft.com/windows/desktop/ProcThread/job-objects). .NET 6+ correctly honors job objects in the same way as cgroups. This document will focus on cgroups throughout. -It is critical to provide effective and well-defined capabilities for .NET applications within memory-limited cgroups. An application should run indefinitely given a sensible configuration for that application. It is important that .NET developers have good controls to optimize their container hosted applications. Our goal is that certain classes of .NET applications can be run with <100 MiB memory constraints. +It is critical to provide effective and well-defined capabilities for .NET applications within memory-limited cgroups. An application should run indefinitely given a sensible configuration for that application. It is important that .NET developers have good controls to optimize their container hosted applications. Our goal is that certain classes of .NET applications can be run with <100 MB memory constraints. Related: @@ -17,7 +17,7 @@ Related: cgroups control two main resources: memory and cores. Both are relevant to the .NET GC. -Memory constraints defines the maximum memory available to the cgroup. This memory is used by the guest operating system, the .NET runtime, the GC heap, and potentially other users. If a cgroup has `100 MiB` available, the app will have less than that. The cgroup will be terminated (AKA `OOMKilled`) when the memory limit is reached. +Memory constraints defines the maximum memory available to the cgroup. This memory is used by the guest operating system, the .NET runtime, the GC heap, and potentially other users. If a cgroup has `100 MB` available, the app will have less than that. The cgroup will be terminated (AKA `OOMKilled`) when the memory limit is reached. Core constraints determine how many GC heaps should be created, at maximum. The maximum heap value matches `Environment.ProcessorCount`. There are three primary ways that this value can be set (using the `docker` CLI to demonstrate): @@ -58,7 +58,7 @@ The heap count can be set two ways: - Manually via `DOTNET_GCHeapCount`. - Automatically by the GC, relying on: - Number of observed or configured cores. - - A minimum _reserved_ memory size per heap of `16 MiB`. + - A minimum _reserved_ memory size per heap of `16 MB`. If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is set, including if it differs from `--cpus`, then the GC will use the ENV value for determining the maximum number of heaps to create. @@ -73,10 +73,10 @@ docker run --rm -m 256mb mcr.microsoft.com/dotnet/samples ``` * 48 core machine -* cgroup has a 256MiB memory limit +* cgroup has a 256MB memory limit * cgroup has no CPU/core limit * 192MB `GCHeapHardLimit` -* Server GC will create 12 GC heaps, with 16 MiB reserved memory +* Server GC will create 12 GC heaps, with 16 MB reserved memory * All 48 cores can be used by the application, per [container policy](https://docs.docker.com/config/containers/resource_constraints/#cpu) `heaps = (256 * .75) / 16` @@ -89,10 +89,10 @@ docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples ``` * 48 core machine -* cgroup has a 256MiB memory limit +* cgroup has a 256MB memory limit * cgroup has 2 CPU/core limit * 192MB `GCHeapHardLimit` -* Server GC will create 2 GC heaps, with 16 MiB reserved memory +* Server GC will create 2 GC heaps, with 16 MB reserved memory * Only 2 cores can be used by the application ### Memory and CPU constrained (with CPU affinity): @@ -102,10 +102,10 @@ docker run --rm -m 256mb --cpuset-cpus 0,2,3 mcr.microsoft.com/dotnet/samples ``` * 48 core machine -* cgroup has a 256MiB memory limit +* cgroup has a 256MB memory limit * cgroup has 3 CPU/core limit * 192MB `GCHeapHardLimit` -* Server GC will create 3 GC heaps, with 16 MiB reserved memory +* Server GC will create 3 GC heaps, with 16 MB reserved memory * Only 3 cores can be used by the application ### Memory and CPU constrained (overriden by `DOTNET_PROCESSOR_COUNT`): @@ -115,8 +115,8 @@ docker run --rm -m 256mb --cpus 2 -e DOTNET_PROCESSOR_COUNT=4 mcr.microsoft.com/ ``` * 48 core machine -* cgroup has a 256MiB memory limit +* cgroup has a 256MB memory limit * cgroup has 2 CPU/core limit * 192MB `GCHeapHardLimit` -* Server GC will create 4 GC heaps, with 16 MiB reserved memory +* Server GC will create 4 GC heaps, with 16 MB reserved memory * Only 2 cores can be used by the application From 50cd55bba61a24960d18afec258527ea30b79e8c Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 28 Jun 2022 10:16:40 -0700 Subject: [PATCH 10/13] Update per feedback --- accepted/2019/support-for-memory-limits.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index abafd7e3f..2f2669f24 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -42,7 +42,7 @@ If both are specified, `GCHeapHardLimit` is used. By default, the `GCHeapHardLimit` will be calculated using the following formula: ```console -max (20mb, 75% of the memory limit on the container) +max (20 MB, 75% of the memory limit on the container) ``` The GC will more aggressive perform GCs as the GC heap grows closer to the `GCHeapHardLimit` with the goal of making more memory available so that the application can continue to safely function. The GC will avoid continuously performing full blocking GCs if they are not considered productive. @@ -73,9 +73,9 @@ docker run --rm -m 256mb mcr.microsoft.com/dotnet/samples ``` * 48 core machine -* cgroup has a 256MB memory limit +* cgroup has a 256 MB memory limit * cgroup has no CPU/core limit -* 192MB `GCHeapHardLimit` +* 192 MB `GCHeapHardLimit` * Server GC will create 12 GC heaps, with 16 MB reserved memory * All 48 cores can be used by the application, per [container policy](https://docs.docker.com/config/containers/resource_constraints/#cpu) @@ -89,9 +89,9 @@ docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples ``` * 48 core machine -* cgroup has a 256MB memory limit +* cgroup has a 256 MB memory limit * cgroup has 2 CPU/core limit -* 192MB `GCHeapHardLimit` +* 192 MB `GCHeapHardLimit` * Server GC will create 2 GC heaps, with 16 MB reserved memory * Only 2 cores can be used by the application @@ -102,9 +102,9 @@ docker run --rm -m 256mb --cpuset-cpus 0,2,3 mcr.microsoft.com/dotnet/samples ``` * 48 core machine -* cgroup has a 256MB memory limit +* cgroup has a 256 MB memory limit * cgroup has 3 CPU/core limit -* 192MB `GCHeapHardLimit` +* 192 MB `GCHeapHardLimit` * Server GC will create 3 GC heaps, with 16 MB reserved memory * Only 3 cores can be used by the application @@ -115,8 +115,8 @@ docker run --rm -m 256mb --cpus 2 -e DOTNET_PROCESSOR_COUNT=4 mcr.microsoft.com/ ``` * 48 core machine -* cgroup has a 256MB memory limit +* cgroup has a 256 MB memory limit * cgroup has 2 CPU/core limit -* 192MB `GCHeapHardLimit` +* 192 MB `GCHeapHardLimit` * Server GC will create 4 GC heaps, with 16 MB reserved memory * Only 2 cores can be used by the application From edd1bb953506030352fd18631d1c44232c679cb6 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 28 Jun 2022 11:46:08 -0700 Subject: [PATCH 11/13] Update index.md --- INDEX.md | 2 +- accepted/2019/support-for-memory-limits.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/INDEX.md b/INDEX.md index 0b64dbc5a..246917242 100644 --- a/INDEX.md +++ b/INDEX.md @@ -30,7 +30,7 @@ Use update-index to regenerate it: | 2018 | [Rolling self-contained .NET Core app deployments forward to the latest patch](accepted/2018/self-contained-roll-forward.md) | [Daniel Plaisted](https://github.com/dsplaisted) | | 2018 | [Standardized Environment Variables for CI Services](accepted/2018/build/standard-ci-env-variables.md) | [Tomáš Matoušek](https://github.com/tmat) | | 2018 | [Windows Compatibility Pack](accepted/2018/compat-pack/compat-pack.md) | [Immo Landwerth](https://github.com/terrajobst), [Wes Haggard](https://github.com/weshaggard) | -| 2019 | [.NET Core GC Support for Docker Limits](accepted/2019/support-for-memory-limits.md) | [Rich Lander](https://github.com/richlander) | +| 2019 | [.NET GC Support for Container Limits](accepted/2019/support-for-memory-limits.md) | [Rich Lander](https://github.com/richlander) | | 2019 | [Background](accepted/2019/targeting-packs-and-runtime-packs.md) | [Rich Lander](https://github.com/richlander), [Nick Guerrera](https://github.com/nguerrera) | | 2019 | [Runtime Binding Behavior](accepted/2019/runtime-binding.md) | [Rich Lander](https://github.com/richlander) | | 2019 | [System.Index and System.Range](accepted/2019/system-range/system-range.md) | [Immo Landwerth](https://github.com/terrajobst) | diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 2f2669f24..249823ada 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -1,4 +1,4 @@ -# .NET GC Support for Docker Limits +# .NET GC Support for Container Limits **Owner** [Rich Lander](https://github.com/richlander) From 92c93b11593eec5074ed717d3d62654dc387f7d6 Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 28 Jun 2022 17:48:09 -0700 Subject: [PATCH 12/13] Add runtime config doc link --- accepted/2019/support-for-memory-limits.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 249823ada..5d71a3d41 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -12,6 +12,7 @@ Related: - [implementing hard limit for GC heap dotnet/coreclr #22180](https://github.com/dotnet/coreclr/pull/22180). - [Validate container improvements with .NET 6](https://github.com/dotnet/runtime/issues/53149). +- [Runtime configuration options for garbage collection](https://docs.microsoft.com/dotnet/core/runtime-config/garbage-collector) ## Cgroup constraints @@ -64,7 +65,7 @@ If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is Note: [.NET Framework 4.8 and 4.8.1](https://github.com/microsoft/dotnet-framework-docker/discussions/935) have the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. -Let's look at some examples. +Let's look at some examples. They are also demonstrated in [Testing GC Heap Counts with Containers](https://github.com/dotnet/runtime/issues/71413). ### Memory constrained; CPU unconstrained From e900593bf8c95c04ea0cbd882f21686f1683d47c Mon Sep 17 00:00:00 2001 From: Rich Lander Date: Tue, 28 Jun 2022 23:02:30 -0700 Subject: [PATCH 13/13] Update per feedbac --- accepted/2019/support-for-memory-limits.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/accepted/2019/support-for-memory-limits.md b/accepted/2019/support-for-memory-limits.md index 5d71a3d41..804a7faf0 100644 --- a/accepted/2019/support-for-memory-limits.md +++ b/accepted/2019/support-for-memory-limits.md @@ -65,6 +65,8 @@ If [`DOTNET_PROCESSOR_COUNT`](https://github.com/dotnet/runtime/issues/48094) is Note: [.NET Framework 4.8 and 4.8.1](https://github.com/microsoft/dotnet-framework-docker/discussions/935) have the same behavior but `COMPlus_RUNNING_IN_CONTAINER` must be set. Also processor count is affected (in the same way) by `COMPlus_PROCESSOR_COUNT`. +Note: The next section talks about how many cores can be used by an application. It isn't defined in this doc, but is assumed to be per the [container runtime policy](https://docs.docker.com/config/containers/resource_constraints/#cpu). + Let's look at some examples. They are also demonstrated in [Testing GC Heap Counts with Containers](https://github.com/dotnet/runtime/issues/71413). ### Memory constrained; CPU unconstrained @@ -78,7 +80,7 @@ docker run --rm -m 256mb mcr.microsoft.com/dotnet/samples * cgroup has no CPU/core limit * 192 MB `GCHeapHardLimit` * Server GC will create 12 GC heaps, with 16 MB reserved memory -* All 48 cores can be used by the application, per [container policy](https://docs.docker.com/config/containers/resource_constraints/#cpu) +* All 48 cores can be used by the application `heaps = (256 * .75) / 16` `heaps = 12` @@ -93,8 +95,8 @@ docker run --rm -m 256mb --cpus 2 mcr.microsoft.com/dotnet/samples * cgroup has a 256 MB memory limit * cgroup has 2 CPU/core limit * 192 MB `GCHeapHardLimit` -* Server GC will create 2 GC heaps, with 16 MB reserved memory -* Only 2 cores can be used by the application +* Server GC will create 2 GC heaps +* 2 cores can be used by the application ### Memory and CPU constrained (with CPU affinity): @@ -106,8 +108,8 @@ docker run --rm -m 256mb --cpuset-cpus 0,2,3 mcr.microsoft.com/dotnet/samples * cgroup has a 256 MB memory limit * cgroup has 3 CPU/core limit * 192 MB `GCHeapHardLimit` -* Server GC will create 3 GC heaps, with 16 MB reserved memory -* Only 3 cores can be used by the application +* Server GC will create 3 GC heaps +* 3 cores can be used by the application ### Memory and CPU constrained (overriden by `DOTNET_PROCESSOR_COUNT`): @@ -119,5 +121,5 @@ docker run --rm -m 256mb --cpus 2 -e DOTNET_PROCESSOR_COUNT=4 mcr.microsoft.com/ * cgroup has a 256 MB memory limit * cgroup has 2 CPU/core limit * 192 MB `GCHeapHardLimit` -* Server GC will create 4 GC heaps, with 16 MB reserved memory -* Only 2 cores can be used by the application +* Server GC will create 4 GC heaps +* 2 cores can be used by the application