Excessive context switching in FreeBSD dotnet ThreadPool worker threads

[Partmedia]

Description

On FreeBSD, ThreadPool worker threads (identified by ".NET TP Worker") cause an excessive (as compared to Linux) number of voluntary context switches. This consumes excessive system and user CPU resources even when there's not a lot of work being done in the thread pools.

Here is a minimal example that demonstrates the problem:

using System;
using System.Threading;

namespace ThreadTest;

public class Program {
    public static void Test(object o)
    {
        Thread.Sleep(1);
    }

    public static void Main()
    {
        Console.WriteLine("Hello, World!");
        while (true)
        {
            ThreadPool.QueueUserWorkItem(new WaitCallback(Test));
            Thread.Sleep(5);
        }
    }
}

This program submits Sleep(1) jobs to the thread pool workers about 200 times per second. If you comment out the ThreadPool.QueueUserWorkItem, this program generates about ~200 voluntary context switches per second, since it sleeps for 5 ms between successive iterations of the loop. However, with the ThreadPool, the program starts generating in excess of 50000 voluntary context switches per second.

I tested the same program on Linux, and it doesn't generate nearly as many context switches.

Configuration

Tested with similar results on:

Architectures:

• FreeBSD 14.1-RELEASE-p4 GENERIC amd64 (x86-64)
• arm64

And dotnet SDKs:

• 8.0.2
• 8.0.6
• 9-rc.1

Regression?

Not Known

Data

On FreeBSD, top reports context switches when run as top -m io -H:

    PID USERNAME     VCSW  IVCSW   ... PERCENT COMMAND
   4535            26599      1    ...  0.00% ThreadTest{.NET TP Worker}
   4535            26204      0    ...  0.00% ThreadTest{.NET TP Worker}
   4535              397      0    ...  0.00% ThreadTest{ThreadTest}

Analysis

ThreadPoolWorkQueue.cs appears to use something called SpinLock which in turn uses SpinWait which alternates calls to Thread.Yield(), Thread.Sleep(0), and Thread.Sleep(1). Perhaps these things behave slightly differently on FreeBSD, and need changing on this platform?

        // These constants determine the frequency of yields versus spinning. The
        // numbers may seem fairly arbitrary, but were derived with at least some
        // thought in the design document.  I fully expect they will need to change
        // over time as we gain more experience with performance.
        internal const int YieldThreshold = 10; // When to switch over to a true yield.
        private const int Sleep0EveryHowManyYields = 5; // After how many yields should we Sleep(0)?
        internal const int DefaultSleep1Threshold = 20; // After how many yields should we Sleep(1) frequently?

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.