Duplicate Logs in Application Insights

[aditvavikramx]

What version of .NET does your existing project use?

.NET 8

What version of .NET are you attempting to target?

.NET 8

Description

I am experiencing an issue where my logs are getting printed twice on Application Insights. Below are the details of my setup and configuration:
Logging Setup in Startup :

services.AddLogging(loggingBuilder =>
{
    loggingBuilder.SetMinimumLevel(logLevel);
    loggingBuilder.AddFilter<ApplicationInsightsLoggerProvider>("mykafkamodel", logLevel);
    loggingBuilder.AddApplicationInsights(loggingBuilder =>
    {
        loggingBuilder.IncludeScopes = true;
    });
});
services.AddApplicationInsightsTelemetryWorkerService(options: aiServiceOptions);
services.ConfigureFunctionsApplicationInsights();

Application Insights Configuration
By default, Application Insights is set to log warnings, so I reset the rules at bootup:

.ConfigureLogging(logging =>
{
    logLevel = LoadConfigurations.ApplicationInsightsLogLevel(config!);
logging.Services.Configure((Action<LoggerFilterOptions>)(options =>
    {
        var loggerFilterRule = options.Rules.FirstOrDefault(rule => rule.ProviderName == "Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider");
        if (loggerFilterRule is not null)
        {
            options.Rules.Remove(loggerFilterRule);
        }
        logging.AddFilter<ApplicationInsightsLoggerProvider>("mykafkamodel", logLevel);
    }));
});

Application Insights Service Options
var aiServiceOptions = new ApplicationInsightsServiceOptions { EnableAdaptiveSampling = false, EnableDebugLogger = false, };

Issue

Despite the above configuration, I observe that some logs are being duplicated in the trace logs. I suspect there might be an issue with my logging setup or configuration.

Request
Could someone please help identify if there is something wrong with my configuration or suggest a solution to prevent duplicate logs?

Project configuration and dependencies

I have upgraded my inprocess model to worker model dot net isolated ie dot net 8

Link to a repository that reproduces the issue

No response

[sravanreddy6745]

I am also facing same issue after migrating my function app from .NET 6 to .NET 8 LTS, seeing unwanted logs from HTTP calls

Eg : Logs while callingDefaultAzureCredential method

[s-nakagaki]

I have same issue.

Program.cs

var host = new HostBuilder()
    .ConfigureFunctionsWebApplication()
    .ConfigureServices(services =>
    {
        services.Configure<TelemetryConfiguration>(config =>
        {
            var credential = new ManagedIdentityCredential();
            config.SetAzureTokenCredential(credential);
        });
        services.AddApplicationInsightsTelemetryWorkerService();
        services.ConfigureFunctionsApplicationInsights();
    })
    .ConfigureLogging(logging =>
    {
        logging.Services.Configure<LoggerFilterOptions>(options =>
        {
            LoggerFilterRule? defaultRule = options.Rules.FirstOrDefault(rule => rule.ProviderName
                == "Microsoft.Extensions.Logging.ApplicationInsights.ApplicationInsightsLoggerProvider");
            if (defaultRule is not null)
            {
                options.Rules.Remove(defaultRule);
            }
        });
    })
    .Build();

host.json

  "logging": {
    "logLevel": {
      "default": "Warning",
      "Function": "Warning",
      "Host.Aggregator": "Information",
      "Host.Results": "Information",
      "Function.HealthCheck": "Error",
      "Function.Hoge": "Information",
      "Function.Hoge.User": "Information"
    },
    "console": {
      "isEnabled": false,
      "DisableColors": true
    },
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "maxTelemetryItemsPerSecond": 1,
        "excludedTypes": "Exception"
      },
      "enableLiveMetricsFilters": true
    }
  }

[s-nakagaki]

The stream log of application insights log at the function of azure portal is ok.
But KQL result at application insights log are duplicated.

I have another function which runs on python and it has same situation.
Is this the problem of Application Insights or Azure Monitor?

[king-11]

I am also in the same boat as @s-nakagaki my dotnet isolated function app faces this issue with open telemetry distro.

I have already opened prior issue in azure dotnet linking it here as well Azure/azure-sdk-for-net#45320

[Barsonax]

Iam seeing this as well with opentelemetry. The sdkversion is different for the logs:

[king-11]

I can ack on @Barsonax seeing duplicate logs from two different sdks as they mentioned.