HttpRequestException when using SocketsHttpHandler #26629

ppekrol · 2018-06-27T10:29:33Z

Hi all,

we had to drop the SocketsHttpHandler (ravendb/ravendb@9ad7c41)
because for quite some time we started receiving random test failures on our CI (Jenkins) Agent (Hyper-V machines, 6 vCPU, 6GB RAM on WS2012R2 and WS2016 - fully updated). We tried to reproduce this on our local dev machines, but without any luck. After turning off the SocketsHttpHandler, the random failures stopped showing up.

The exception we are getting is:

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in c:\\Jenkins\\workspace\\PR_Tests_v4.1\\s\\src\\Raven.Client\\Http\\RequestExecutor.cs:line 738

We are using latest .NET Core 2.1.1 (2.1.0 was also affected).

Do you know if there are some problems in SocketsHttpHandler that could cause this? What can we do to help you solve this issue? Any pointers?

The text was updated successfully, but these errors were encountered:

ayende · 2018-06-27T10:31:12Z

Most / all of these requests were using POST operations (or other methods that have method bodies).
We suspect, but aren't sure, that this might be related to connection pooling in some manner.

stephentoub · 2018-06-27T14:37:04Z

What can we do to help you solve this issue?

A repro 😄

stephentoub · 2018-06-27T14:49:16Z

We suspect, but aren't sure, that this might be related to connection pooling in some manner.

That error message is for SocketError.OperationAborted, which likely means the socket was closed while the send or receive was in progress.

ayende · 2018-06-28T07:56:27Z

@stephentoub Is there anything here that we can look at?
To reproduce this we need to schedule > 5 runs of 4,000+ tests that takes an hour plus.
In such scenario, we typically see 1 - 3 such failures using SocketsHttpHandler.

Note that we never use SocketsHttpHandler directly, so I don't think that we improperly using it.
We got through HttpClient, a shared instance that is re-used among threads, as recommended.

The code is actually doing something like:

try
{
    return await httpHandler.SendAsync(...);
}
catch(HttpRequestException e)
{
}

So we can catch the exception and then check state / logging / something.

Anything you can recommend that we'll look at to get to the root cause?

stephentoub · 2018-06-28T14:45:48Z

Is it possible you're hitting a timeout? It shouldn't manifest this way, but it would give us a place to start looking. For example, if you change your HttpClient's Timeout to Timeout.InfiniteTimeSpan, does this still happen?

Or if it's potentially related to connection pooling, what happens if you change the SocketHttpHandler's PooledConnectionLifetime and PooledConnectionIdleTimeout to infinite... does it still repro?

ayende · 2018-06-28T15:10:06Z

It is all running locally, and we timed the error and it was about 11 ms from calling SendAsync.
We'll try with the connection lifetimes.

stephentoub · 2018-06-28T18:20:17Z

Actually, I have a theory for how this could manifest... now I just need to go look through the code for where the cause could be. Is it possible that before these errors occur for you, some other request timed out or could have otherwise been canceled (e.g. via the token you pass in)?

stephentoub · 2018-06-28T18:40:24Z

Or... are you potentially disposing of the response stream from a request while it's concurrently being read? e.g. something buggy like:

Task t = responseStream.ReadAsync(...);
responseStream.Dispose();

?

ayende · 2018-06-28T19:19:44Z

It is entirely possible that the previous request can time out, yes.
In fact, that is probably guaranteed. We have a few tests that test what happens when a request is timing out.

ayende · 2018-06-28T19:20:19Z

I don't think that I have anything like the second option, but it is possible. We might be in the middle of a read and need to abort the connection.

ayende · 2018-06-28T19:20:53Z

And... I have a repro. On my machine, this reproduce the scenario instantly.
You were correct about the timeout causing this.

using System;
using System.Net.Http;
using System.Threading.Tasks;

namespace ConsoleApp15
{
    class Program
    {
        static async Task Main(string[] args)
        {

            // http://slowwly.robertomurray.co.uk/delay/3000/url/http://www.google.co.uk


            var client = new HttpClient(new SocketsHttpHandler
            {
            })
            {
                Timeout = TimeSpan.FromMilliseconds(250)
            };

            var tasks = new[]
            {
                Run(client),
                Run(client),
                Run(client),
                Run(client),
                Run(client),
                Run(client),
                Run(client),
                Run(client),
                Run(client),
            };

            await Task.WhenAll(tasks);

            Console.WriteLine("Done");

        }

        public static async Task Run(HttpClient client)
        {
            for (int i = 0; i < 25; i++)
            {
                try
                {
                    await client.GetStringAsync("http://slowwly.robertomurray.co.uk/delay/3000/url/http://www.google.co.uk");
                    Console.WriteLine("Should timeout");
                }
                catch (Exception e)
                {
                }
                await client.GetStringAsync("http://slowwly.robertomurray.co.uk/delay/100/url/http://www.google.co.uk");
            }

        }
    }
}

stephentoub · 2018-06-28T19:24:41Z

And... I have a repro. On my machine, this reproduce the scenario instantly.

That's hitting a timeout and throwing an OperationCanceledException... that's expected.

What's not expected is getting an IOException (not wrapped in an OperationCanceledException) out of SendAsync and when nothing on the request is being canceled / timing out.

ayende · 2018-06-28T19:31:41Z

@stephentoub There are two exceptions thrown here. The first is expected, because we are hitting the timeout.
The second exception (which will actually kill the process) is thrown even though the second request shouldn't be timing out. In fact, you can replace the slowwly url in the second one entirely and still get the same error.

I think that this shows that the timeout on the request indeed poison the pool

stephentoub · 2018-06-28T19:37:54Z

The second exception (which will actually kill the process) is thrown even though the second request shouldn't be timing out.

Why shouldn't the second request time out? It takes a very long time to access and the timeout is set at 250ms. (I'm trying to access it now and it's just sitting waiting.)

ayende · 2018-06-28T19:45:05Z

Hm... looks like I was too excited.

var sp = Stopwatch.StartNew();
try
{
    await client.GetStringAsync("http://www.google.co.uk");
}
catch (Exception)
{
    Console.WriteLine(sp.ElapsedMilliseconds);
    throw;
}

Always gives > 250 ms. I assumed that this was because of the previous run, not that it is (probably) waiting there.

ayende · 2018-06-28T20:05:08Z

Okay, I think that I'm on a better track here. See the following code.
The key here is that we need to be POSTing data., I think.

I was able to reproduce the HttpRequestException when posting data and getting a timeout.
We are going to try to build a better repro for this.

ppekrol · 2018-07-02T11:32:54Z

We have some more information (still no direct repro).

We have managed to find out exactly which particular test is causing those errors. This is RavenDB_7787 - a test that is spawning a separate process with our Server and doing some basic CRUD operations on it.

The server is started with random port (we are passing http://127.82.86.78:0) and from our CI logs we can see that the address is generated correctly (Embedded URL: http://127.82.86.78:60101). What is worth noting is that we are killing this Server and whole external process in a bit rude way - using process.Kill.

After this test completes, 20-30 seconds later, few tests are failing with the exception that I wrote about in the issue. Those tests are connecting to global server that is available for whole test suite (and this server is working on a different url and port). After few (2-3) of those failures the tests are starting to work in a normal way.

It looks like something is going on after 20-30 seconds after process was killed and for some time it affects other socket connections.

This is our CI log (the 7787 turns on a switch that writes on console url's and http methods for sanity check - to know if we are hitting appropriate server). The code is available here: https://github.com/ppekrol/ravendb/commits/v4.1-temp4

FYI, skipping this test resolves the issue.

01:02:45.307     SlowTests.Tests.Sorting.AlphaNumericSorting.order_by_two_parameters_first_long_than_alphanumeric
01:02:45.579     SlowTests.Tests.Sorting.AlphaNumericSorting.basic_alphanumeric_sort
01:02:45.859     SlowTests.Tests.Sorting.AlphaNumericSorting.number_and_decimal_alphanumeric_sort
01:02:46.465     SlowTests.Tests.Sorting.AlphaNumericSorting.random_words(seed: 1144883738)
01:02:49.064     SlowTests.Issues.RavenDB_7787.TestEmbedded
01:02:50.064 Request: http://127.0.0.1:59136/databases/Round_robin_load_balancing_should_workbe797fd1-f48b-4772-9bde-6ed4d45afd71/stats?failure=check. Type: GET
01:02:54.406 Request: http://127.82.86.78:60101/cluster/topology. Type: GET
01:02:54.688 Request: http://127.82.86.78:60101/admin/databases?name=Test&replicationFactor=1. Type: PUT
01:02:56.117 Request: http://127.0.0.1:59136/databases/Round_robin_load_balancing_should_workbe797fd1-f48b-4772-9bde-6ed4d45afd71/stats?failure=check. Type: GET
01:02:58.727 Embedded URL: http://127.82.86.78:60101
01:02:58.727 Request: http://127.82.86.78:60101/topology?name=Test. Type: GET
01:02:58.727 Request: http://127.82.86.78:60101/databases/Test/bulk_docs. Type: POST
01:02:59.334 Request: http://127.82.86.78:60101/databases/Test/configuration/client. Type: GET
01:03:01.920 Request: http://127.0.0.1:59136/databases/Round_robin_load_balancing_should_workbe797fd1-f48b-4772-9bde-6ed4d45afd71/stats?failure=check. Type: GET
01:03:03.338 Request: http://127.82.86.78:60109/cluster/topology. Type: GET
01:03:03.926 Request: http://127.82.86.78:60109/admin/databases?name=Test&replicationFactor=1. Type: PUT
01:03:04.196 Embedded URL: http://127.82.86.78:60109
01:03:04.197 Request: http://127.82.86.78:60109/topology?name=Test. Type: GET
01:03:04.197 Request: http://127.82.86.78:60109/databases/Test/docs?&id=people%2F1. Type: GET
01:03:04.197 Request: http://127.82.86.78:60109/databases/Test/configuration/client. Type: GET
01:03:05.178     SlowTests.Issues.RavenDB_6414.Should_unload_db_and_send_notification_on_catastrophic_failure
01:03:05.470 Request: http://127.0.0.1:60113/cluster/topology. Type: GET
01:03:05.470 Request: http://127.0.0.1:60113/admin/databases?name=Should_unload_db_and_send_notification_on_catastrophic_failure_2534&replicationFactor=1. Type: PUT
01:03:05.470 Request: http://127.0.0.1:60113/info/tcp?tag=Supervisor. Type: GET
01:03:06.068 Request: http://127.0.0.1:60113/rachis/waitfor?index=4. Type: GET
01:03:08.060 Request: http://127.0.0.1:59136/databases/Round_robin_load_balancing_should_workbe797fd1-f48b-4772-9bde-6ed4d45afd71/stats?failure=check. Type: GET
01:03:08.060 Request: http://127.0.0.1:60113/topology?name=Should_unload_db_and_send_notification_on_catastrophic_failure_2534. Type: GET
01:03:08.060 Request: http://127.0.0.1:60113/databases/Should_unload_db_and_send_notification_on_catastrophic_failure_2534/hilo/next?tag=users&lastBatchSize=0&lastRangeAt=0001-01-01T00:00:00.0000000&identityPartsSeparator=/&lastMax=0. Type: GET
01:03:08.334 Request: http://127.0.0.1:60113/databases/Should_unload_db_and_send_notification_on_catastrophic_failure_2534/configuration/client. Type: GET
01:03:08.334 Request: http://127.0.0.1:60113/databases/Should_unload_db_and_send_notification_on_catastrophic_failure_2534/bulk_docs. Type: POST
01:03:08.334 Request: http://127.0.0.1:60113/databases/Should_unload_db_and_send_notification_on_catastrophic_failure_2534/hilo/return?tag=users&end=32&last=1. Type: PUT
01:03:08.334     SlowTests.Issues.RavenDb1962.CanExecuteLazyQueriesInAsyncSession
01:03:08.334 Request: http://127.0.0.1:58232/cluster/topology. Type: GET
01:03:08.334 Request: http://127.0.0.1:58232/admin/databases?name=CanExecuteLazyQueriesInAsyncSession_2535&replicationFactor=1. Type: PUT
01:03:08.610 Request: http://127.0.0.1:58232/rachis/waitfor?index=10760. Type: GET
01:03:08.610 Request: http://127.0.0.1:58232/topology?name=CanExecuteLazyQueriesInAsyncSession_2535. Type: GET
01:03:08.610 Request: http://127.0.0.1:58232/databases/CanExecuteLazyQueriesInAsyncSession_2535/bulk_docs. Type: POST
01:03:08.610 Request: http://127.0.0.1:58232/databases/CanExecuteLazyQueriesInAsyncSession_2535/configuration/client. Type: GET
01:03:08.610 Request: http://127.0.0.1:58232/databases/CanExecuteLazyQueriesInAsyncSession_2535/stats. Type: GET
01:03:08.610 Request: http://127.0.0.1:58232/databases/CanExecuteLazyQueriesInAsyncSession_2535/multi_get. Type: POST
01:03:08.610 Request: http://127.0.0.1:58232/admin/databases. Type: DELETE
01:03:08.610 Request: http://127.0.0.1:58232/databases/CanExecuteLazyQueriesInAsyncSession_2535/stats?failure=check. Type: GET
01:03:08.610     SlowTests.Issues.RavenDb1962.CanExecuteLazyQueriesInAsyncSession [FAIL]
01:03:08.610       Raven.Client.Exceptions.RavenException : System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request
01:03:08.610          --- End of inner exception stack trace ---
01:03:08.610          at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
01:03:08.610          at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
01:03:08.610          at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.610          --- End of inner exception stack trace ---
01:03:08.610          at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.610          at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
01:03:08.612          at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.612          at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.612          at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
01:03:08.612          at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Http\RequestExecutor.cs:line 738
01:03:08.612       Response.StatusCode - InternalServerError
01:03:08.612       Stack Trace:
01:03:08.612         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Http\RequestExecutor.cs(835,0): at Raven.Client.Http.RequestExecutor.ThrowFailedToContactAllNodes[TResult](RavenCommand`1 command, HttpRequestMessage request)
01:03:08.612         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Http\RequestExecutor.cs(756,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
01:03:08.612         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Documents\Session\AsyncDocumentSession.Lazy.cs(116,0): at Raven.Client.Documents.Session.AsyncDocumentSession.ExecuteLazyOperationsSingleStep(ResponseTimeInformation responseTimeInformation, List`1 requests, CancellationToken token)
01:03:08.612         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Documents\Session\AsyncDocumentSession.Lazy.cs(88,0): at Raven.Client.Documents.Session.AsyncDocumentSession.ExecuteAllPendingLazyOperationsAsync(CancellationToken token)
01:03:08.612         c:\Jenkins\workspace\PR_Tests_v4.1\s\test\SlowTests\Issues\RavenDb1962.cs(83,0): at SlowTests.Issues.RavenDb1962.CanExecuteLazyQueriesInAsyncSession()
01:03:08.612         --- End of stack trace from previous location where exception was thrown ---
01:03:08.612     SlowTests.Issues.RavenDb1962.CanExecuteLazyLoadsInAsyncSession_CheckSingleCall
01:03:08.612 Request: http://127.0.0.1:58232/cluster/topology. Type: GET
01:03:08.612 Request: http://127.0.0.1:58232/admin/databases?name=CanExecuteLazyLoadsInAsyncSession_CheckSingleCall_2536&replicationFactor=1. Type: PUT
01:03:08.891 Request: http://127.0.0.1:58232/rachis/waitfor?index=10764. Type: GET
01:03:08.891 Request: http://127.0.0.1:58232/topology?name=CanExecuteLazyLoadsInAsyncSession_CheckSingleCall_2536. Type: GET
01:03:08.891 Request: http://127.0.0.1:58232/databases/CanExecuteLazyLoadsInAsyncSession_CheckSingleCall_2536/bulk_docs. Type: POST
01:03:08.891 Request: http://127.0.0.1:58232/databases/CanExecuteLazyLoadsInAsyncSession_CheckSingleCall_2536/configuration/client. Type: GET
01:03:08.891 Request: http://127.0.0.1:58232/databases/CanExecuteLazyLoadsInAsyncSession_CheckSingleCall_2536/multi_get. Type: POST
01:03:08.891 Request: http://127.0.0.1:58232/admin/databases. Type: DELETE
01:03:08.891     SlowTests.Issues.RavenDb1962.CanExecuteLazyLoadsInAsyncSession_CheckSingleCall [FAIL]
01:03:08.891       Raven.Client.Exceptions.RavenException : System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request
01:03:08.891          --- End of inner exception stack trace ---
01:03:08.891          at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
01:03:08.891          at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
01:03:08.891          at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.891          --- End of inner exception stack trace ---
01:03:08.891          at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.891          at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
01:03:08.891          at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.891          at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
01:03:08.891          at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
01:03:08.891          at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Http\RequestExecutor.cs:line 738
01:03:08.891       Response.StatusCode - InternalServerError
01:03:08.891       Stack Trace:
01:03:08.891         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Http\RequestExecutor.cs(835,0): at Raven.Client.Http.RequestExecutor.ThrowFailedToContactAllNodes[TResult](RavenCommand`1 command, HttpRequestMessage request)
01:03:08.891         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Http\RequestExecutor.cs(756,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
01:03:08.891         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Documents\Session\AsyncDocumentSession.Lazy.cs(116,0): at Raven.Client.Documents.Session.AsyncDocumentSession.ExecuteLazyOperationsSingleStep(ResponseTimeInformation responseTimeInformation, List`1 requests, CancellationToken token)
01:03:08.891         c:\Jenkins\workspace\PR_Tests_v4.1\s\src\Raven.Client\Documents\Session\AsyncDocumentSession.Lazy.cs(88,0): at Raven.Client.Documents.Session.AsyncDocumentSession.ExecuteAllPendingLazyOperationsAsync(CancellationToken token)
01:03:08.891         c:\Jenkins\workspace\PR_Tests_v4.1\s\test\SlowTests\Issues\RavenDb1962.cs(52,0): at SlowTests.Issues.RavenDb1962.CanExecuteLazyLoadsInAsyncSession_CheckSingleCall()
01:03:08.891         --- End of stack trace from previous location where exception was thrown ---
01:03:08.891     SlowTests.Issues.RavenDb1962.CanExecuteLazyLoadsInAsyncSession
01:03:08.891 Request: http://127.0.0.1:58232/cluster/topology. Type: GET
01:03:08.891 Request: http://127.0.0.1:58232/admin/databases?name=CanExecuteLazyLoadsInAsyncSession_2537&replicationFactor=1. Type: PUT
01:03:09.161 Request: http://127.0.0.1:58232/rachis/waitfor?index=10767. Type: GET
01:03:09.162 Request: http://127.0.0.1:58232/topology?name=CanExecuteLazyLoadsInAsyncSession_2537. Type: GET
01:03:09.162 Request: http://127.0.0.1:58232/databases/CanExecuteLazyLoadsInAsyncSession_2537/bulk_docs. Type: POST
01:03:09.162 Request: http://127.0.0.1:58232/databases/CanExecuteLazyLoadsInAsyncSession_2537/configuration/client. Type: GET
01:03:09.162 Request: http://127.0.0.1:58232/admin/databases. Type: DELETE
01:03:09.162     SlowTests.Issues.RavenDB_11046.FacetRqlShouldSupportAliasNotation

vitalybibikov · 2018-07-10T15:31:58Z

I have this behavior on nodes in Service Fabric Cluster 1. In SF Cluster 2 it works OK
I'll try to investigate it further, still have no clues, can't reproduce it on my local machine.

ayende · 2018-07-10T15:43:23Z

We have a statistical repro. Unfortunately, it require us to run about 3,400 tests to run into this.

karelz · 2018-07-10T16:32:09Z

We can try our logging on it. Do you have steps we can "easily" replicate locally?

ayende · 2018-07-10T17:12:02Z

Get the code from this PR
ravendb/ravendb#6879

Go to test/slowtests and run dotnet xunit

This reproduce in one of five runs or so. Typically more on lower end machines.
If there is a way to turn on the logging configuration, I'll happily do so and provide the results.

karelz · 2018-07-10T17:29:30Z

Most of our logging went in AFTER 2.1 :( ... I am not even sure if it is going to cover this particular scenario ...
If we find logging useful, we can get it in 2.1.x servicing.
Is there easy way to run your test suite against master? (if you don't know, we will figure it out)

ppekrol · 2018-07-10T17:50:14Z

@karelz

I'm not sure what is needed to run from 'master', but if you want to update <RuntimeFrameworkVersion> in csprojs (all of them) then best option to do so would be to run:

.\scripts\updateFrameworkVersion.ps1 2.1.1

Also our CI is executing tests in 'Release'.

karelz · 2018-07-10T17:51:19Z

@rmkerr will take a look on our side - I asked him to try the repro in house as first step.
Just to clarify:

Was it discovered as part of .NET Core 2.0->2.1.0/2.1.1 migration? Or is it part of larger port from .NET Framework to .NET Core? (Just to scope what to look for and if it is regression against 2.0)
Did you try to run it with SocketsHttpHandler disabled on 2.1.1? (Just to confirm if it is really specific to SocketsHttpHandler or if it may be general problem in Networking) (If you didn't it is ok, we can try it too)
- Answer: It is SocketsHttpHandler specific - see ravendb/ravendb@9ad7c41

ppekrol · 2018-07-10T18:55:39Z

It started to happening all of a sudden. First we thought that it might be related to 2.0 -> 2.1 migration, but at the end we think it is related to SocketsHttpHandler and RavenDB_7787.cs only.

Check my explanation here: https://github.com/dotnet/corefx/issues/30691#issuecomment-401775334

rmkerr · 2018-07-11T17:08:45Z

I'm in the process of setting up the repro now. I think that the connection management logging in 2.1 actually isn't bad, so we can hopefully get some useful info there.

rmkerr · 2018-07-11T19:57:42Z

@ayende are there any additional steps I need to follow to get the tests working? I'm currently:

checking out the code from the linked PR
running dotnet restore in /tests/SlowTests
running dotnet xunit in /tests/SlowTests

As output I see a large number of tests skipped, and then several failures that occur every time I try to run the tests. This is running on Windows 10 RS2

Logs:

D:\ravendb\test\SlowTests>dotnet xunit Detecting target frameworks in SlowTests.csproj... Building for framework netcoreapp2.1... Sparrow -> D:\ravendb\src\Sparrow\bin\Debug\netcoreapp2.1\Sparrow.dll Sparrow -> D:\ravendb\src\Sparrow\bin\Debug\netstandard2.0\Sparrow.dll Voron -> D:\ravendb\src\Voron\bin\Debug\netstandard2.0\Voron.dll Raven.Client -> D:\ravendb\src\Raven.Client\bin\Debug\netstandard2.0\Raven.Client.dll Raven.Embedded -> D:\ravendb\src\Raven.Embedded\bin\Debug\netstandard2.0\Raven.Embedded.dll Raven.Server -> D:\ravendb\src\Raven.Server\bin\Debug\netcoreapp2.1\Raven.Server.dll Tests.Infrastructure -> D:\ravendb\test\Tests.Infrastructure\bin\Debug\netcoreapp2.1\Tests.Infrastructure.dll FastTests -> D:\ravendb\test\FastTests\bin\Debug\netcoreapp2.1\FastTests.dll Raven.TestDriver -> D:\ravendb\src\Raven.TestDriver\bin\Debug\netstandard2.0\Raven.TestDriver.dll SlowTests -> D:\ravendb\test\SlowTests\bin\Debug\netcoreapp2.1\SlowTests.dll Running .NET Core 2.1.1 tests for framework netcoreapp2.1... xUnit.net Console Runner v2.4.0-beta.1.build3958 (64-bit .NET Core 4.6.26606.02) Discovering: SlowTests Discovered: SlowTests Starting: SlowTests Max number of concurrent tests is: 2 To attach debugger to test process (x64), use proc-id: 8640. SlowTests.Issues.RavenDB_3365.index_pretty_printer_ignores_whitespaces [SKIP] RavenDB-4185 To attach debugger to test process (x64), use proc-id: 8640. Url http://127.0.0.1:50061 SlowTests.Issues.RavenDB_5256.ShouldLoadUserFromAsyncSessionTwiceInEmbeddedDocumentStore [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadUserFromSessionTwiceInShardingDocumentStore [SKIP] RavenDB-5918, RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadUserFromAsyncSessionTwiceInShardingDocumentStore [SKIP] RavenDB-5918, RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadMultipleUsersWithIncludesFromAsyncSessionInShardingDocumentStore [SKIP] RavenDB-5918, RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadMultipleUsersWithIncludesFromSessionInShardingDocumentStore [SKIP] RavenDB-5918, RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadMultipleUsersWithIncludesFromSessionInEmbeddedDocumentStore [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadMultipleUsersWithIncludesFromAsyncSessionInEmbeddedDocumentStore [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_5256.ShouldLoadUserFromSessionTwiceInEmbeddedDocumentStore [SKIP] RavenDB-6283 SlowTests.MailingList.ShardedFacets.FacetTest [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_3465.get_metadata_for_async_sharded [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_3465.get_metadata_for_sharded [SKIP] RavenDB-6283 SlowTests.Tests.Queries.CanQueryOnCustomClass.UsingConverter [SKIP] RavenDB-6263 SlowTests.MailingList.StreamingHalfWay.ShouldWork [SKIP] Missing feature: /docs/stream SlowTests.Issues.RavenDB_4904.can_create_side_by_side_index_to_replace_index_with_errors [SKIP] RavenDB-5919 SlowTests.Issues.RavenDB_4904.can_create_side_by_side_index_with_errors_to_replace_index_with_errors [SKIP] RavenDB-5919 SlowTests.Issues.RavenDB_4904.can_create_side_by_side_index_with_errors_to_replace_index [SKIP] RavenDB-5919 SlowTests.Server.Documents.SqlMigration.MySQLSchemaTest.CanFetchSchema [SKIP] Test requires MySQL database SlowTests.Issues.RavenDB_4708.CanUseSingleSyncSharded [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_4708.CanUseFirstAsyncSharded [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_4708.CanUseFirstSyncSharded [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_4708.CanUseLazilySyncShaded [SKIP] RavenDB-6283 SlowTests.MailingList.JohanNilsson.WithCustomizedTagNameAndIdentityProperty [SKIP] RavenDB-6124 SlowTests.Issues.RavenDB_3609.Test [SKIP] RavenDB-6283 SlowTests.Authentication.AuthenticationChangesTests.ChangesWithAuthentication [SKIP] RavenDB-9580: Does not work on current version .NET core. SlowTests.Issues.RavenDB_3420.bulk_insert_sharded [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_3420.bulk_insert_sharded_when_specifying_default_database [SKIP] RavenDB-6283 SlowTests.MailingList.DeserializationToObjectTests.Query_GivenDbWithComplexObjects_ShouldDeserializePropertiesToOriginalType [SKIP] RavenDB-6124 SlowTests.Bugs.Indexing.IndexingEachFieldInEachDocumentSeparately.ForIndexing [SKIP] Missing feature : RavenDB-6152 SlowTests.Bugs.Indexing.WiseShrek.Isolated [SKIP] Missing features SlowTests.Issues.RavenDB_579.OneShardPerSessionStrategy [SKIP] RavenDB-6283 SlowTests.Issues.RavenDB_579.OneShardPerSessionStrategyAsync [SKIP] RavenDB-6283 SlowTests.Cluster.ClusterModesForRequestExecutorTest.Fastst_node_should_choose_the_node_without_delay [SKIP] RavenDB-9020 SlowTests.Issues.RavenDB_6596.Run [SKIP] Culture tests are disabled. Please set 'RAVEN_ENABLE_CULTURE_TESTS' environment variable to 'true' to enable them. SlowTests.MailingList.Lindblom.Test [SKIP] RavenDB-6124 SlowTests.MailingList.Kushnir.SortOnMetadata [SKIP] RavenDB-6264 SlowTests.Issues.RavenDB_2944.CanCreateTestMapReduceIndexes [SKIP] RavenDB-6572 SlowTests.Issues.RavenDB_2944.CanCreateTestMapIndexes [SKIP] RavenDB-6572 SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.upload_archive [SKIP] Requires Amazon AWS Credentials SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.put_blob_in_folder [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.can_get_correct_error_s3 [SKIP] Requires Amazon AWS Credentials SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.upload_archive_multipart [SKIP] Requires Amazon AWS Credentials SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.put_blob [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.can_get_correct_error_glacier [SKIP] Requires Amazon AWS Credentials SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.put_object_multipart [SKIP] Requires Amazon AWS Credentials SlowTests.Server.Documents.PeriodicBackup.RavenDB_2181.put_object [SKIP] Requires Amazon AWS Credentials SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_70MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_64MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_256MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_100MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_into_folder_500MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_765MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_500MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.can_get_container_not_found [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_into_folder_765MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_into_folder_64MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_into_folder_256MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_into_folder_70MB [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.can_get_and_delete_container [SKIP] Azure Storage Emulator is not installed SlowTests.Server.Documents.PeriodicBackup.RavenDB_4163.put_blob_into_folder_100MB [SKIP] Azure Storage Emulator is not installed SlowTests.MailingList.ScriptedPatchBug.Test [SKIP] Missing feature: Tasks (operations) and their results SlowTests.Issues.RavenDB_3726.Test [SKIP] RavenDB-6283 SlowTests.Server.Documents.SqlMigration.RecursiveMigrationTest.CanEmbedOnParent(provider: MsSQL) [FAIL] System.InvalidOperationException : Use a valid connection Stack Trace: D:\ravendb\test\SlowTests\Server\Documents\ETL\SQL\SqlEtlTests.cs(73,0): at SlowTests.Server.Documents.ETL.SQL.SqlEtlTests.<>c.<.cctor>b__32_0() at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor) at System.Lazy`1.CreateValue() D:\ravendb\test\SlowTests\Server\Documents\SqlMigration\SqlAwareTestBase.cs(113,0): at SlowTests.Server.Documents.SqlMigration.SqlAwareTestBase.WithMsSqlDatabase(String& connectionString, String& databaseName, String dataSet, Boolean includeData) D:\ravendb\test\SlowTests\Server\Documents\SqlMigration\SqlAwareTestBase.cs(104,0): at SlowTests.Server.Documents.SqlMigration.SqlAwareTestBase.WithSqlDatabase(MigrationProvider provider, String& connectionString, String& schemaName, String dataSet, Boolean includeData) D:\ravendb\test\SlowTests\Server\Documents\SqlMigration\RecursiveMigrationTest.cs(119,0): at SlowTests.Server.Documents.SqlMigration.RecursiveMigrationTest.CanEmbedOnParent(MigrationProvider provider) --- End of stack trace from previous location where exception was thrown --- SlowTests.Server.Documents.SqlMigration.RecursiveMigrationTest.CanEmbedOnParent(provider: MySQL) [SKIP] Test requires MySQL database SlowTests.Voron.Bugs.RavenDB_6971.Applying_new_diff_requires_to_zero_destination_bytes_first [FAIL] Raven.Client.Exceptions.Database.DatabaseLoadTimeoutException : Raven.Client.Exceptions.Database.DatabaseLoadTimeoutException: Database Applying_new_diff_requires_to_zero_destination_bytes_first_42 after 00:01:00 is still loading, try again later. Database initialization log: [Load Database] 7/11/2018 7:41:01 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Starting database initialization [Load Database] 7/11/2018 7:41:01 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Creating db.lock file [Load Database] 7/11/2018 7:41:01 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Initializing NotificationCenter [Load Database] 7/11/2018 7:41:01 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Initializing DocumentStorage [Load Database] 7/11/2018 7:41:01 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Starting Recovery [Load Database] 7/11/2018 7:41:01 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Recovering journal 6 (upto last journal 7) [Load Database] 7/11/2018 7:46:27 PM :: Database 'Applying_new_diff_requires_to_zero_destination_bytes_first_42' : Recovering journal 7 (upto last journal 7) at Raven.Server.Routing.RouteInformation.ThrowDatabaseLoadTimeoutWithLog(StringSegment databaseName, TimeSpan timeout, String log) in D:\ravendb\src\Raven.Server\Routing\RouteInformation.cs:line 143 at Raven.Server.Routing.RouteInformation.UnlikelyWaitForDatabaseToLoad(RequestHandlerContext context, Task`1 database, DatabasesLandlord databasesLandlord, StringSegment databaseName) in D:\ravendb\src\Raven.Server\Routing\RouteInformation.cs:line 122 at Raven.Server.Routing.RouteInformation.WaitForDb(Task databaseLoading) in D:\ravendb\src\Raven.Server\Routing\RouteInformation.cs:line 162 at Raven.Server.Routing.RequestRouter.HandlePath(HttpContext context, String method, String path) in D:\ravendb\src\Raven.Server\Routing\RequestRouter.cs:line 73 at System.Threading.Tasks.ValueTask`1.get_Result() at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in D:\ravendb\src\Raven.Server\RavenServerStartup.cs:line 162 Response.StatusCode - ServiceUnavailable Stack Trace: D:\ravendb\src\Raven.Client\Http\RequestExecutor.cs(839,0): at Raven.Client.Http.RequestExecutor.ThrowFailedToContactAllNodes[TResult](RavenCommand`1 command, HttpRequestMessage request) D:\ravendb\src\Raven.Client\Http\RequestExecutor.cs(795,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) D:\ravendb\src\Raven.Client\Http\RequestExecutor.cs(828,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) D:\ravendb\src\Raven.Client\Documents\Operations\OperationExecutor.Operations.cs(22,0): at Raven.Client.Documents.Operations.OperationExecutor.SendAsync(IOperation`1 operation, SessionInfo sessionInfo, CancellationToken token) D:\ravendb\src\Raven.Client\Util\AsyncHelpers.cs(104,0): at Raven.Client.Util.AsyncHelpers.<>c__DisplayClass3_0`1.<b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- D:\ravendb\src\Raven.Client\Util\AsyncHelpers.cs(124,0): at Raven.Client.Util.AsyncHelpers.RunSync[T](Func`1 task) D:\ravendb\src\Raven.Client\Documents\Operations\OperationExecutor.Operations.cs(13,0): at Raven.Client.Documents.Operations.OperationExecutor.Send(IOperation`1 operation, SessionInfo sessionInfo) D:\ravendb\test\SlowTests\Voron\Bugs\RavenDB_6971.cs(75,0): at SlowTests.Voron.Bugs.RavenDB_6971.Applying_new_diff_requires_to_zero_destination_bytes_first() SlowTests.Server.Documents.ETL.RavenDB_8866.CanResetEtl [FAIL] Raven.Client.Exceptions.Commercial.LicenseLimitException : Raven.Client.Exceptions.Commercial.LicenseLimitException: Your current license doesn't include the RavenDB ETL feature at Raven.Server.Commercial.LicenseManager.AssertCanAddRavenEtl() in D:\ravendb\src\Raven.Server\Commercial\LicenseManager.cs:line 1269 at Raven.Server.Web.System.OngoingTasksHandler.AssertCanAddOrUpdateEtl(String databaseName, BlittableJsonReaderObject& etlConfiguration, JsonOperationContext context) in D:\ravendb\src\Raven.Server\Web\System\OngoingTasksHandler.cs:line 719 at Raven.Server.Documents.DatabaseRequestHandler.DatabaseConfigurations(Func`4 setupConfigurationFunc, String debug, RefAction beforeSetupConfiguration, Action`3 fillJson, HttpStatusCode statusCode) in D:\ravendb\src\Raven.Server\Documents\DatabaseRequestHandler.cs:line 64 at Raven.Server.Web.System.OngoingTasksHandler.AddEtl() in D:\ravendb\src\Raven.Server\Web\System\OngoingTasksHandler.cs:line 685 at Raven.Server.Routing.RequestRouter.HandlePath(HttpContext context, String method, String path) in D:\ravendb\src\Raven.Server\Routing\RequestRouter.cs:line 123 at System.Threading.Tasks.ValueTask`1.get_Result() at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in D:\ravendb\src\Raven.Server\RavenServerStartup.cs:line 162 State for A-term1: Passive=>LeaderElect at 2018-07-11T19:34:45.5363246Z because I'm the only one in the cluster, so I'm the leader Stack Trace: D:\ravendb\src\Raven.Client\Exceptions\ExceptionDispatcher.cs(117,0): at Raven.Client.Exceptions.ExceptionDispatcher.Throw(JsonOperationContext context, HttpResponseMessage response, Action`1 additionalErrorInfo) D:\ravendb\src\Raven.Client\Http\RequestExecutor.cs(1036,0): at Raven.Client.Http.RequestExecutor.HandleUnsuccessfulResponse[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, HttpRequestMessage request, HttpResponseMessage response, String url, SessionInfo sessionInfo, Boolean shouldRetry, CancellationToken token) D:\ravendb\src\Raven.Client\Http\RequestExecutor.cs(786,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) D:\ravendb\src\Raven.Client\Http\RequestExecutor.cs(828,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) D:\ravendb\src\Raven.Client\Documents\Operations\MaintenanceOperationExecutor.cs(61,0): at Raven.Client.Documents.Operations.MaintenanceOperationExecutor.SendAsync[TResult](IMaintenanceOperation`1 operation, CancellationToken token) D:\ravendb\src\Raven.Client\Util\AsyncHelpers.cs(104,0): at Raven.Client.Util.AsyncHelpers.<>c__DisplayClass3_0`1.<b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- D:\ravendb\src\Raven.Client\Util\AsyncHelpers.cs(124,0): at Raven.Client.Util.AsyncHelpers.RunSync[T](Func`1 task) D:\ravendb\src\Raven.Client\Documents\Operations\MaintenanceOperationExecutor.cs(43,0): at Raven.Client.Documents.Operations.MaintenanceOperationExecutor.Send[TResult](IMaintenanceOperation`1 operation) D:\ravendb\test\SlowTests\Server\Documents\ETL\EtlTestBase.cs(21,0): at SlowTests.Server.Documents.ETL.EtlTestBase.AddEtl[T](DocumentStore src, EtlConfiguration`1 configuration, T connectionString) D:\ravendb\test\SlowTests\Server\Documents\ETL\RavenDB_8866.cs(43,0): at SlowTests.Server.Documents.ETL.RavenDB_8866.CanResetEtl() SlowTests.Server.Documents.ETL.ConnectionStringTests.CanGetConnectionStringByName [FAIL] System.InvalidOperationException : Use a valid connection Stack Trace: D:\ravendb\test\SlowTests\Server\Documents\ETL\SQL\SqlEtlTests.cs(73,0): at SlowTests.Server.Documents.ETL.SQL.SqlEtlTests.<>c.<.cctor>b__32_0() at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode) --- End of stack trace from previous location where exception was thrown --- at System.Lazy`1.CreateValue() D:\ravendb\test\SlowTests\Server\Documents\ETL\SQL\SqlEtlTests.cs(1132,0): at SlowTests.Server.Documents.ETL.SQL.SqlEtlTests.GetConnectionString(DocumentStore store) D:\ravendb\test\SlowTests\Server\Documents\ETL\ConnectionStringTests.cs(171,0): at SlowTests.Server.Documents.ETL.ConnectionStringTests.CanGetConnectionStringByName()

ppekrol · 2018-07-11T20:21:23Z

@rmkerr

This is expected. There are over 3500 tests there and it takes around 30-40min for them to complete.

I suggest running dotnet xunit -c Release or dotnet xunit -c Release -verbose if you want to see which test is currently running.

Some of them might fail due to the missing license, or can be skipped because of missing SQL Server, but you can safely ignore that fact.

ppekrol · 2018-07-11T21:36:40Z

@rmkerr

I've pushed 1 additional commit to the branch https://github.com/ppekrol/ravendb/commits/v4.1-temp4 that disables test parallelization.

After that I've ran in test/SlowTests:

dotnet xunit -c Release -verbose > tests.txt

Following two tests have failed with the exception that we are trying to trace:

    SlowTests.Tests.Sorting.AlphaNumericSorting.random_words_using_document_query_async(seed: 1472672940)
Request: http://127.0.0.1:54525/cluster/topology. Type: GET
Request: http://127.0.0.1:54525/admin/databases?name=random_words_using_document_query_async_1557&replicationFactor=1. Type: PUT
Request: http://127.0.0.1:54525/rachis/waitfor?index=6223. Type: GET
Request: http://127.0.0.1:54525/topology?name=random_words_using_document_query_async_1557. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/next?tag=tracks&lastBatchSize=0&lastRangeAt=0001-01-01T00:00:00.0000000&identityPartsSeparator=/&lastMax=0. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/configuration/client. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/operations/next-operation-id. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/bulk_insert?id=1. Type: POST
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/next?tag=tracks&lastBatchSize=32&lastRangeAt=2018-07-11T21:13:52.7935007Z&identityPartsSeparator=/&lastMax=32. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/next?tag=tracks&lastBatchSize=64&lastRangeAt=2018-07-11T21:13:52.8037418Z&identityPartsSeparator=/&lastMax=96. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/next?tag=tracks&lastBatchSize=128&lastRangeAt=2018-07-11T21:13:52.8135658Z&identityPartsSeparator=/&lastMax=224. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/next?tag=tracks&lastBatchSize=256&lastRangeAt=2018-07-11T21:13:52.8216998Z&identityPartsSeparator=/&lastMax=480. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/next?tag=tracks&lastBatchSize=512&lastRangeAt=2018-07-11T21:13:52.8433858Z&identityPartsSeparator=/&lastMax=992. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/admin/indexes. Type: PUT
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/stats. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/stats. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/stats. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/stats. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/queries?queryHash=1369170765125549859. Type: POST
Request: http://127.0.0.1:54525/admin/databases. Type: DELETE
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/stats?failure=check. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/stats?failure=check. Type: GET
Request: http://127.0.0.1:54525/databases/random_words_using_document_query_async_1557/hilo/return?tag=tracks&end=2016&last=1024. Type: PUT
      Raven.Client.Exceptions.RavenException : System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
         at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in F:\Workspaces\HR\ravendb_2\src\Raven.Client\Http\RequestExecutor.cs:line 738
      Response.StatusCode - InternalServerError
      Stack Trace:
        F:\Workspaces\HR\ravendb_2\src\Raven.Client\Http\RequestExecutor.cs(835,0): at Raven.Client.Http.RequestExecutor.ThrowFailedToContactAllNodes[TResult](RavenCommand`1 command, HttpRequestMessage request)
        F:\Workspaces\HR\ravendb_2\src\Raven.Client\Http\RequestExecutor.cs(756,0): at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token)
        F:\Workspaces\HR\ravendb_2\src\Raven.Client\Documents\Session\AsyncDocumentQuery.cs(847,0): at Raven.Client.Documents.Session.AsyncDocumentQuery`1.ExecuteActualQueryAsync(CancellationToken token)
        F:\Workspaces\HR\ravendb_2\src\Raven.Client\Documents\Session\AsyncDocumentQuery.cs(839,0): at Raven.Client.Documents.Session.AsyncDocumentQuery`1.InitAsync(CancellationToken token)
        F:\Workspaces\HR\ravendb_2\src\Raven.Client\Documents\Session\AsyncDocumentQuery.cs(797,0): at Raven.Client.Documents.Session.AsyncDocumentQuery`1.ExecuteQueryOperation(Nullable`1 take, CancellationToken token)
        F:\Workspaces\HR\ravendb_2\test\SlowTests\Tests\Sorting\AlphaNumericSorting.cs(454,0): at SlowTests.Tests.Sorting.AlphaNumericSorting.random_words_using_document_query_async(Int32 seed)
        --- End of stack trace from previous location where exception was thrown ---

    SlowTests.Server.Documents.ModifyExistingDocument.ShouldThrowIfChangeCollection
Request: http://127.0.0.1:54525/cluster/topology. Type: GET
Request: http://127.0.0.1:54525/admin/databases?name=ShouldThrowIfChangeCollection_1623&replicationFactor=1. Type: PUT
Request: http://127.0.0.1:54525/rachis/waitfor?index=6497. Type: GET
Request: http://127.0.0.1:54525/topology?name=ShouldThrowIfChangeCollection_1623. Type: GET
Request: http://127.0.0.1:54525/databases/ShouldThrowIfChangeCollection_1623/docs?id=users%2F1. Type: PUT
Request: http://127.0.0.1:54525/databases/ShouldThrowIfChangeCollection_1623/configuration/client. Type: GET
Request: http://127.0.0.1:54525/databases/ShouldThrowIfChangeCollection_1623/docs?id=users%2F1. Type: PUT
Request: http://127.0.0.1:54525/admin/databases. Type: DELETE
      Assert.Contains() Failure
      Not found: Changing 'users/1' from 'Users' to 'UserAddresses' via update is not supported.
      Delete it and recreate the document users/1.
      In value:  System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. ---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request
         --- End of inner exception stack trace ---
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
         at Raven.Client.Http.RequestExecutor.ExecuteAsync[TResult](ServerNode chosenNode, Nullable`1 nodeIndex, JsonOperationContext context, RavenCommand`1 command, Boolean shouldRetry, SessionInfo sessionInfo, CancellationToken token) in F:\Workspaces\HR\ravendb_2\src\Raven.Client\Http\RequestExecutor.cs:line 738
      Response.StatusCode - InternalServerError
      Stack Trace:
           at SlowTests.Server.Documents.ModifyExistingDocument.ShouldThrowIfChangeCollection()
        --- End of stack trace from previous location where exception was thrown ---

Log is available here: https://drive.google.com/file/d/1syhM_yR2KwtWvzp7LGABag5_duU8cKLZ/view?usp=sharing

stephentoub · 2019-07-16T21:49:16Z

We are facing the same error. This happens with multiple threads. Works fine with single thread. Any help will be highly appreciated.

Can you share a repro?

nglpg · 2019-07-16T21:53:06Z

We have an API in .netcore 2.2 that is talking to an internal API (.netcore 2.2). Single requests are working fine but when sending multiple requests through our automation using multiple threads it fails with this error. Is this what you are looking for?

stephentoub · 2019-07-16T21:54:51Z

Is this what you are looking for?

No, I was actually looking for code that I could run to see the issue. What you describe is very unlikely to be the same underlying problem as this issue.

nglpg · 2019-07-16T21:58:13Z

I will work on it and get back to you. How do i give the code to you?

stephentoub · 2019-07-16T22:28:06Z

How do i give the code to you?

You can post it here. Hopefully you can create a standalone repro that highlights what you're seeing and is generally shareable.

ckarras · 2019-07-19T14:19:12Z

I also had this issue and was working on code to reproduce it, but I realized it can be reproduced fairly trivially by setting HttpClient.Timeout to an extremely low value and sending a HTTPS POST to a IIS Express server running the default ASP.Net Core project template. However that might be a "false positive", in my case there was really a timeout and the problem would be that the error message is misleading (it should clearly say there was a timeout instead of "either a thread exit or an application request")

Setting AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false); also caused an exception due to a timeout, it just changed the error message and stack trace.

ckarras · 2019-07-19T14:28:58Z

Correction: Not exactly the default ASP.Net Core project template, it needs to be modified to add a POST operation, and in that operation wait some delay (higher than the client's timeout) before returning a response.

nglpg · 2019-07-19T14:43:00Z

Thanks @ckarras .
@stephentoub We created a repro but the error is not happening in that but clearly we are seeing it in our API. Any ideas? I'm gonna try the timeout that ckarras has mentioned.

georgenorberg · 2019-07-23T15:02:02Z

Since I’m facing this issue I wanted to add my input to help find a solution.

@stephentoub I can add that i get the error in System.Net.Http.HttpConnection.SendAsyncCore

Some details about my specific case:
I'm currently facing this issue in an Azure Durable Functions project (dot net core v2.2) using the httpclient. I use the http client to integrate against our API.
In some cases, the durable function is running up to 5 posts in a row to the API by asynchronous function chaining and I can see the error as mentioned here in the same activity function several times but not always. I'm thinking it is most likely caused by a timeout since nothing is wrong in the API from what I can see.

If you have any questions about the implementation of this please let me know.

When I compare, I can see similarities to the previously posted logs.
From my logs:

Function 'function name (Activity)' failed with an error. Reason: System.Threading.Tasks.TaskCanceledException: The operation was canceled. 
---> System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request. 
---> System.Net.Sockets.SocketException: The I/O operation has been aborted because of either a thread exit or an application request
 --- End of inner exception stack trace ---
 at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
 at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
 at System.Net.Security.SslStreamInternal.<FillBufferAsync>g__InternalFillBufferAsync|38_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
 at System.Net.Security.SslStreamInternal.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
 at System.Net.Http.HttpConnection.FillAsync()
 at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
 at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
 --- End of inner exception stack trace ---
 at xx.DurableFunctions.Activity.ArchivePdfAfterBPMApprovedAct.Run(DurableActivityContext context, IWebApiAdapter webApiAdapter, IEmailHelper emailHelper, ILogger log) in C:\xx\PN.DF.DurableFunctions\Activity\ArchivePdfAfterBPMApprovedAct.cs:line 59
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.FunctionInvocationFilterInvoker.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 1176
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.FunctionInvocationFilterInvoker.InvokeAsync(Object instance, Object[] arguments) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 1213
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 584
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 531
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, IFunctionOutputDefinition outputDefinition, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 467
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 277
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 321
 at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsyncCore(IFunctionInstanceEx functionInstance, CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 117. IsReplay: False. State: Failed. HubName: DFTaskHubNameProdV1. AppName: PNDigitalFormsDurableFunc. SlotName: Production. ExtensionVersion: 1.8.0. SequenceNumber: 9014.

karelz · 2019-07-29T20:53:32Z

@stephentoub I see you moved it to Future, while we had it tracking for servicing 2.1/2.2 (and 3.0). Was there new info about the scenario impact? AFAIK we had 2-3 independent customers affected.

stephentoub · 2019-07-29T20:55:00Z

@karelz, no one has been able to supply a repro, and I can't repro it.

karelz · 2019-07-30T01:12:04Z

@rmkerr was able to repro it locally. He was able to minimize the repro (to running one test massively in parallel). I asked him to transfer it to you. Did it not happen? I can check my old emails to see if I was on cc.

stephentoub · 2019-07-30T02:51:45Z

@karelz, he pointed me to https://github.com/dotnet/corefx/issues/30691#issuecomment-443926134, but that's from 8 months ago, and a lot has changed since then in both the product and in the ASP.NET tests he refers to. I tried various permutations of what he suggested and never got such a failure.

stephentoub · 2019-07-30T03:00:31Z

@ppekrol, have you moved to .NET Core 3.0? If yes, are you still hitting this?

karelz · 2019-07-30T06:29:11Z

@stephentoub would it make sense to ask someone on our team to try to sync back and repro it on older 3.0 build or on 2.2? (with Max's steps)

ppekrol · 2019-07-30T13:00:31Z

@stephentoub We haven't moved to 3.0 yet. We will start migration closer to the release date, probably when RC builds with hit the shelves.

I might be wrong, but I see two separate issues here that ppl submit, 1st is probably expected (Operation has timeout out) and 2nd one (the one that we have submitted) is more generic (An error occurred while sending the request).

stephentoub · 2019-07-30T13:04:25Z

I might be wrong, but I see two separate issues here that ppl submit, 1st is probably expected (Operation has timeout out) and 2nd one (the one that we have submitted) is more generic (An error occurred while sending the request).

Yes, there are multiple reasons this could occur. As you say, most are expected error conditions; that's just the nature of networking. The one you highlighted is not expected and is much more elusive.

stephentoub · 2019-07-31T00:54:56Z

@stephentoub would it make sense to ask someone on our team to try to sync back and repro it on older 3.0 build or on 2.2?

@karelz, if you think it will be fruitful, go for it.

karelz · 2019-07-31T17:44:43Z

@scalablecory can you please help with it?

L0ndra · 2019-08-21T15:18:49Z

Have you tried to play with MaxConnectionsPerServer settings in SocketsHttpClient?

ByronAP · 2019-08-29T00:03:33Z

I am running into this too if i can get any real information i will share.

ByronAP · 2019-08-29T00:21:21Z

for me it stems from System.IO.Stream.ReadAsync

scalablecory · 2019-09-12T22:12:54Z

Unable to replicate with current code; tried pulling old versions of RavenDb and ASP.NET and didn't happen after some playing around either.

Going to close and we can open a new ticket if we find something in 3.0. Many of the replications in this thread appear to be different from what was reported so it's hard to get a solid idea of how much relevancy the issue has today.

FreedCapybara · 2020-08-28T15:17:34Z

This issue drove me crazy for a really long time. In case it helps anyone, I added some retries with Polly:

var apiExceptionPolicy = Policy
    .Handle<Refit.ApiException>() // in my case I was using Refit, but change this to HttpRequestException or whatever works for you
    .RetryAsync(3, (exception, retryCount) => Task.Delay(50));

await apiExceptionPolicy.ExecuteAsync(() => randomlyFailingCode());

This seems to have fixed it for me - I haven't had any random exceptions during local development or emails from the production server for several weeks now 👍

karelz assigned rmkerr Jul 10, 2018

karelz assigned scalablecory Jul 31, 2019

scalablecory closed this as completed Sep 12, 2019

msftgits transferred this issue from dotnet/corefx Jan 31, 2020

msftgits added this to the 5.0 milestone Jan 31, 2020

Mpdreamz mentioned this issue Apr 23, 2020

[flakey test] The I/O operation has been aborted because of either a thread exit or an application request. elastic/elasticsearch-net#4670

Closed

ghost locked as resolved and limited conversation to collaborators Dec 16, 2020

HttpRequestException when using SocketsHttpHandler #26629

HttpRequestException when using SocketsHttpHandler #26629

Comments

ppekrol commented Jun 27, 2018

ayende commented Jun 27, 2018

stephentoub commented Jun 27, 2018

stephentoub commented Jun 27, 2018

ayende commented Jun 28, 2018 • edited by karelz Loading

stephentoub commented Jun 28, 2018

ayende commented Jun 28, 2018

stephentoub commented Jun 28, 2018

stephentoub commented Jun 28, 2018

ayende commented Jun 28, 2018

ayende commented Jun 28, 2018

ayende commented Jun 28, 2018 • edited by karelz Loading

stephentoub commented Jun 28, 2018 • edited Loading

ayende commented Jun 28, 2018

stephentoub commented Jun 28, 2018 • edited Loading

ayende commented Jun 28, 2018 • edited by karelz Loading

ayende commented Jun 28, 2018

ppekrol commented Jul 2, 2018 • edited Loading

vitalybibikov commented Jul 10, 2018

ayende commented Jul 10, 2018

karelz commented Jul 10, 2018

ayende commented Jul 10, 2018

karelz commented Jul 10, 2018

ppekrol commented Jul 10, 2018

karelz commented Jul 10, 2018 • edited Loading

ppekrol commented Jul 10, 2018 • edited Loading

rmkerr commented Jul 11, 2018

rmkerr commented Jul 11, 2018 • edited Loading

ppekrol commented Jul 11, 2018

ppekrol commented Jul 11, 2018

stephentoub commented Jul 16, 2019

nglpg commented Jul 16, 2019

stephentoub commented Jul 16, 2019

nglpg commented Jul 16, 2019

stephentoub commented Jul 16, 2019

ckarras commented Jul 19, 2019 • edited Loading

ckarras commented Jul 19, 2019 • edited Loading

nglpg commented Jul 19, 2019

georgenorberg commented Jul 23, 2019 • edited Loading

karelz commented Jul 29, 2019

stephentoub commented Jul 29, 2019

karelz commented Jul 30, 2019

stephentoub commented Jul 30, 2019

stephentoub commented Jul 30, 2019

karelz commented Jul 30, 2019

ppekrol commented Jul 30, 2019

stephentoub commented Jul 30, 2019

stephentoub commented Jul 31, 2019

karelz commented Jul 31, 2019

L0ndra commented Aug 21, 2019

ByronAP commented Aug 29, 2019 • edited Loading

ByronAP commented Aug 29, 2019

scalablecory commented Sep 12, 2019

FreedCapybara commented Aug 28, 2020

ayende commented Jun 28, 2018 •

edited by karelz

Loading

ayende commented Jun 28, 2018 •

edited by karelz

Loading

stephentoub commented Jun 28, 2018 •

edited

Loading

stephentoub commented Jun 28, 2018 •

edited

Loading

ayende commented Jun 28, 2018 •

edited by karelz

Loading

ppekrol commented Jul 2, 2018 •

edited

Loading

karelz commented Jul 10, 2018 •

edited

Loading

ppekrol commented Jul 10, 2018 •

edited

Loading

rmkerr commented Jul 11, 2018 •

edited

Loading

ckarras commented Jul 19, 2019 •

edited

Loading

ckarras commented Jul 19, 2019 •

edited

Loading

georgenorberg commented Jul 23, 2019 •

edited

Loading

ByronAP commented Aug 29, 2019 •

edited

Loading