RichPayloadEventSource can get enabled at Verbose level

[pharring]

Repro Steps

Compile and run:

using System;
using System.Linq;
using System.Diagnostics.Tracing;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Extensibility;

namespace ConsoleApp1
{
    internal sealed class MyEventListener : EventListener
    {
        public EventSource RichPayloadEventSource;

        protected override void OnEventSourceCreated(EventSource eventSource)
        {
            if (eventSource.Name == "Microsoft-ApplicationInsights-Data")
            {
                RichPayloadEventSource = eventSource;
            }

            base.OnEventSourceCreated(eventSource);
        }
    }

    class Program
    {
        private static string GetEnabledLevels(EventSource eventSource)
            => string.Join('|',
                from EventLevel level in Enum.GetValues(typeof(EventLevel))
                where eventSource.IsEnabled(level, EventKeywords.All)
                select level);

        static void Main(string[] args)
        {
            using (var myListener = new MyEventListener())
            {
                // This has the effect of creating the RichPayloadEventSource.
                // Use an unconfigured TelemetryConfiguration so we don't get the
                // DiagnosticsTelemetryModule added. Note that when config is Disposed,
                // TelemetryConfiguration.Active is reset to null.
                using (var config = new TelemetryConfiguration())
                {
                    new TelemetryClient(config).TrackEvent("Startup");
                }

                // This has the side-effect of creating TelemetryConfiguration.Active with
                // DiagnosticsTelemetryModule added. It hooks up an EventListener for all
                // Microsoft-ApplicationInsights-* event sources, including the
                // RichPayloadEventSource. It's supposed to use EventLevel.Error.
                new TelemetryClient().TrackEvent("Hello!");

                // This prints: LogAlways|Critical|Error|Warning|Informational|Verbose
                // It's supposed to print LogAlways|Critical|Error
                Console.WriteLine(GetEnabledLevels(myListener.RichPayloadEventSource));
            }
        }
    }
}

The comments explain what's going on. The problem is that RichPayloadEventSource has been enabled with all events at all levels which makes it slow and inefficient. The events don't actually get logged because the DiagnosticListener.WriteEvent has an additional check for the level, so they get stopped there; but getting there involved allocating and serializing big payloads.

While this repro looks contrived, I don't believe this is purely hypothetical. I think we've had customers run into this.

Also note that RichPayloadEventSource is just one of the EventSources affected by this bug. Any other sources with names starting with "Microsoft-ApplicationInsights-" are affected; that includes the CoreEventSource and the Snapshot Debugger (which has an EventSource called "Microsoft-ApplicationInsights-Profiler-Session"). We've had reports of the latter getting "stuck" on.

Actual Behavior

LogAlways|Critical|Error|Warning|Informational|Verbose

Expected Behavior

LogAlways|Critical|Error

Version Info

SDK Version : 2.9.1
.NET Version : .NET Core 2.2, but it'll reproduce on all frameworks.
How Application was onboarded with SDK: Visual Studio
OS : Windows 10
Hosting Info (IIS/Azure WebApps/ etc) : N/A