System.InvalidOperationException: 'Internal connection fatal error.' #1116

figueiredorui · 2021-06-15T19:58:48Z

I'm looking for some help troubleshooting an issue that I'm having in one of my Development machines.

The following code is throwing a System.InvalidOperationException: 'Internal connection fatal error.' when opening the connection.

using System;
using Microsoft.Data.SqlClient;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
			using (var connection = new SqlConnection("Server=localhost;Database=Test02;Trusted_Connection=True;"))
			{
				connection.Open();
				Console.WriteLine("Connected successfully.");
				Console.ReadKey(true);
			}
		}
    }
}

Here is the stach trace

 at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at ConsoleApp1.Program.Main(String[] args) in Program.cs:line 12

On the same machine, using the legacy library 'System.Data.SqlClient', the following code works fine

using System.Data.SqlClient;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
			using (var connection = new SqlConnection("Server=localhost;Database=Test02;Trusted_Connection=True;"))
			{
				connection.Open();
				Console.WriteLine("Connected successfully.");
				Console.ReadKey(true);
			}
		}
    }
}

I have tried a few things like:
Repair .NET 5
Repair SQL Server 2019 Developer
Repair VS2019

Does anyone have suggestions that could help to understand the source of the issue?

Thanks

ericsampson · 2021-06-15T20:35:03Z

You could try running something like WireShark to capture the difference between the two cases at the TCP level, because there's probably not enough information in the stacktrace to help.

johnnypham · 2021-06-16T01:24:10Z

Hi @figueiredorui, what version of Microsoft.Data.SqlClient are you using? Is this a .NET Core or .NET Framework application? You can try enabling event tracing and posting the event log here.

figueiredorui · 2021-06-16T08:26:20Z

Hi @johnnypham
Using Microsoft.Data.SqlClient 3.0.0, .NET Core 5

Event Log
<prov.DbConnectionHelper.ConnectionString_Set|API> 1, 'Server=localhost;Database=Test02;Trusted_Connection=True;' <sc.AppConfigManager.FetchConfigurationSection|INFO>: Unable to load customAppContextSwitchOverrides. Default value of Ttype returns. <sc.SqlAppContextSwitchManager.ApplyContextSwitches|INFO> Entry point. <sc.SqlAppContextSwitchManager.ApplyContextSwitches|INFO> Exit point. <sc|SqlAuthenticationProviderManager|Ctor|Info>Neither SqlClientAuthenticationProviders nor SqlAuthenticationProviders configuration section found. TdsParserStateObjectFactory.CreateTdsParserStateObject | Info | AppContext switch 'Switch.Microsoft.Data.SqlClient.UseManagedNetworkingOnWindows' not enabled, native networking implementation will be used. TdsParser.Connect | SEC | Connection Object Id 4, Authentication Mode: SqlPassword TdsParser.Connect | SEC | SSPI or Active Directory Authentication Library loaded for SQL Server based integrated authentication <sc.TdsParser.Connect|SEC> Sending prelogin handshake <sc.TdsParser.SendPreLoginHandshake|INFO> ClientConnectionID 3f6323bc-78bd-418e-aae6-5204682d11ac, ActivityID 099fff25-893b-4144-921b-1c2062a11fc1:2 <sc.TdsParser.Connect|SEC> Consuming prelogin handshake <sc.SqlInternalConnectionTds.BreakConnection|RES|CPOOL> 4, Breaking connection. <sc.TdsParser.Run|ERR> Potential multi-threaded misuse of connection, unexpected TDS token found 1 <comm.ADP.TraceException|ERR|THROW> 'System.InvalidOperationException: Internal connection fatal error.' <sc.SqlInternalConnectionTds.LoginFailure|RES|CPOOL> 4

r-work · 2021-06-16T14:46:11Z

Similar issue here using 3.0 with .NET 5, version 2.1.3 works fine.

cheenamalhotra · 2021-06-16T15:24:48Z

Could you also try out the 3 preview versions of v3.0.0 to get the first version to contain this issue?

r-work · 2021-06-16T15:56:17Z

@cheenamalhotra preview 1 seems to be working. 2, 3 and stable are not working.

johnnypham · 2021-06-18T00:34:07Z

@figueiredorui @r-work can you provide as much details as you can about your client/server environment (version, platform, etc) so we can get a local repro to investigate this?

figueiredorui · 2021-06-21T17:14:43Z

I'm setting up a new development environment and installing the various tools/frameworks to find out which one makes my project fail running. So far so good ...

fraserbooth · 2021-07-05T08:50:05Z

I'm experiencing very similar with .NET Core 3.1 and Microsoft.Data.SqlClient 3.0.0:

   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n   at Microsoft.Data.SqlClient.TdsParse
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at Sites.ExecuteGet() in Sites.cs:line 33

I assume its relevant that this issue started for me when I was forced to apply the latest Windows Update (KB5003637). I'm working on Win 10 Pro with VS2019 (16.9.4).

...any thoughts on ways to investigate and resolve this?

johnnypham · 2021-07-06T20:11:51Z

.NET Core 3.1 and Microsoft.Data.SqlClient 3.0.0:
Windows Update (KB5003637). Win 10 Pro with VS2019 (16.9.4).

I wasn't able to reproduce it with this same configuration. Can you confirm if the issue is present starting with 3.0.0-preview2?

fraserbooth · 2021-07-07T07:33:25Z

Hi @johnnypham,

Thank you for your reply. I've just enabled prerelease in NuGet, and installed 3.0.0-preview2.21106.5 which is exhibiting the same behaviour.

Is there any other information I can share so that your system can reproduce the same error? One of my colleagues updated his system to the same Window Update Version and is now also getting the same error while previously everything worked fine, so we have managed to reproduced this at least twice...

All the best,

Fraser

johnnypham · 2021-07-07T17:00:01Z

Can you try 3.0.0-preview1?

fraserbooth · 2021-07-07T17:04:43Z

Unfortunately exactly the same error message...

cheenamalhotra · 2021-07-07T17:44:57Z

I assume its relevant that this issue started for me when I was forced to apply the latest Windows Update (KB5003637). I'm working on Win 10 Pro with VS2019 (16.9.4).

Hi @fraserbooth

Your issue seems different that what is being discussed here from earlier customer. In your case it seems to be related to some sort of configuration issue or an environment related problem. Could you open a new ticket with all related information:

SQL Server version
Client OS version and enabled TLS protocols
Connection string
If possible a small repro to reproduce issue by running it.

cheenamalhotra · 2021-07-09T21:35:30Z

@figueiredorui

Checking in, did you manage to create a repro to this issue?

cheenamalhotra · 2021-09-30T00:32:51Z

Closing due to inactivity and lack of repro.

pjmolina · 2022-03-17T16:02:35Z

Hi @cheenamalhotra it seems we hit the same error here:

System.InvalidOperationException: Internal connection fatal error.
      GET /api/v1/country User: '' Internal connection fatal error.
      System.InvalidOperationException: Internal connection fatal error.
         at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
         at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
         at System.Data.SqlClient.TdsParser.DrainData(TdsParserStateObject stateObj)
         at System.Data.SqlClient.TdsParser.ThrowUnsupportedCollationEncountered(TdsParserStateObject stateObj)
         at System.Data.SqlClient.TdsParser.GetCodePage(SqlCollation collation, TdsParserStateObject stateObj)
         at System.Data.SqlClient.TdsParser.TryProcessEnvChange(Int32 tokenLength, TdsParserStateObject stateObj, SqlEnvChange& sqlEnvChange)
         at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
         at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
         at System.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)
         at System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
         at System.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
         at System.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
         at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken)
         at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
         at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
         at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
         at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
         at System.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()
      --- End of stack trace from previous location ---
         at Dapper.SqlMapper.QueryAsync[T](IDbConnection cnn, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 417

Repro conditions:

Server environment: SQL Server v. 12.0.4522.0 SP1-CU13

Scenario A:
Client Environment: dotnet 6.0.2 + Dapper 2.0 on Windows 11.
Results: It works, connect to DB and makes a simple select query.

Scenario B:
Client Environment: dotnet 6.0.2 + Dapper 2.0 on Docker Linux mcr.microsoft.com/dotnet/runtime:6.0.2-alpine3.14-amd64. Results: It fails, unable to connect to DB with the initial exception stack trace.

$ uname -a
Linux 953c836e7379 5.10.60.1-microsoft-standard-WSL2 #1 SMP Wed Aug 25 23:20:18 UTC 2021 x86_64 Linux

TLS settings has been downgraded to:

MinProtocol = TLSv1.0
CipherString = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256

to discard TLS issues.

Self-signed certificate of the Sql Server machine has been incorporated to alpine image using:

COPY sql01.crt /usr/usr/local/share/ca-certificates/sql01.crt
RUN update-ca-certificates

Connection string in both scenarios:
Server=10.1.2.3;Database=db;trustServerCertificate=true;Trusted_Connection=false;Encrypt=true;User Id=user;Password=pass;

Expected behavour

To connect succesfully both on Windows and Linux Alpine docker image. Or at least, enough information to diagnose the root cause of the failure.

Actual behavour

Working on Windows and fails to connect on Linux Alpine docker image.

pjmolina · 2022-03-17T18:39:07Z

Update: fixed on my side doing the following actions:

Replaced System.Data.SqlServer for Microsoft.Data.SqlServer
Installed additional icu-libs in my docker image and disabled Globalization Invariant setting as commented here #220

RUN apk add icu-libs
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false

DavoudEshtehari · 2022-03-17T19:42:16Z

@pjmolina Thank you for the update.
Could you clarify step 1? Do you mean using Microsoft.Data.SqlClient instead of System.Data.SqlClient?

pjmolina · 2022-03-17T19:53:10Z

.

@pjmolina Thank you for the update. Could you clarify step 1? Do you mean using Microsoft.Data.SqlClient instead of System.Data.SqlClient?

Exactly: replacing the library and then updating the references across the project.

ad-akshay · 2023-06-29T07:23:29Z

pjmolina

Thank you! It works like a charm :)

vovcacik · 2024-01-18T10:43:23Z

I had the same issue with System SqlClient. After upgrading to Microsoft.Data.SqlClient it turned out the problem was with invariant globalization. Turning it off helped me with both System and Microsoft SqlClient.

 <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings> 
    <InvariantGlobalization>false</InvariantGlobalization>
  </PropertyGroup>

The moral is, you may get more useful error messages from Microsoft SqlClient than from the System one.

cheenamalhotra added the ⏳ Waiting for Customer Issues/PRs waiting for user response/action. label Jun 22, 2021

fraserbooth mentioned this issue Jul 7, 2021

System.InvalidOperationException: 'Internal connection fatal error.' #1154

Closed

cheenamalhotra closed this as completed Sep 30, 2021

DavoudEshtehari removed the ⏳ Waiting for Customer Issues/PRs waiting for user response/action. label Jun 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

System.InvalidOperationException: 'Internal connection fatal error.' #1116

System.InvalidOperationException: 'Internal connection fatal error.' #1116

figueiredorui commented Jun 15, 2021

ericsampson commented Jun 15, 2021

johnnypham commented Jun 16, 2021

figueiredorui commented Jun 16, 2021

r-work commented Jun 16, 2021 •

edited

Loading

cheenamalhotra commented Jun 16, 2021

r-work commented Jun 16, 2021

johnnypham commented Jun 18, 2021

figueiredorui commented Jun 21, 2021

fraserbooth commented Jul 5, 2021 •

edited

Loading

johnnypham commented Jul 6, 2021

fraserbooth commented Jul 7, 2021

johnnypham commented Jul 7, 2021

fraserbooth commented Jul 7, 2021

cheenamalhotra commented Jul 7, 2021

cheenamalhotra commented Jul 9, 2021

cheenamalhotra commented Sep 30, 2021

pjmolina commented Mar 17, 2022 •

edited

Loading

pjmolina commented Mar 17, 2022

DavoudEshtehari commented Mar 17, 2022

pjmolina commented Mar 17, 2022

ad-akshay commented Jun 29, 2023

vovcacik commented Jan 18, 2024

System.InvalidOperationException: 'Internal connection fatal error.' #1116

System.InvalidOperationException: 'Internal connection fatal error.' #1116

Comments

figueiredorui commented Jun 15, 2021

ericsampson commented Jun 15, 2021

johnnypham commented Jun 16, 2021

figueiredorui commented Jun 16, 2021

r-work commented Jun 16, 2021 • edited Loading

cheenamalhotra commented Jun 16, 2021

r-work commented Jun 16, 2021

johnnypham commented Jun 18, 2021

figueiredorui commented Jun 21, 2021

fraserbooth commented Jul 5, 2021 • edited Loading

johnnypham commented Jul 6, 2021

fraserbooth commented Jul 7, 2021

johnnypham commented Jul 7, 2021

fraserbooth commented Jul 7, 2021

cheenamalhotra commented Jul 7, 2021

cheenamalhotra commented Jul 9, 2021

cheenamalhotra commented Sep 30, 2021

pjmolina commented Mar 17, 2022 • edited Loading

Repro conditions:

Expected behavour

Actual behavour

pjmolina commented Mar 17, 2022

DavoudEshtehari commented Mar 17, 2022

pjmolina commented Mar 17, 2022

ad-akshay commented Jun 29, 2023

vovcacik commented Jan 18, 2024

r-work commented Jun 16, 2021 •

edited

Loading

fraserbooth commented Jul 5, 2021 •

edited

Loading

pjmolina commented Mar 17, 2022 •

edited

Loading