More optimizations for InFlightRequest

[jasper-d]

Based on the optimizations done by @danielwertheim I further optimized InFlightRequest for use cases w/o a timeout (i.e. when using a client provided cancellation token).

Notable changes:

• Moved InFlightRequest into a seperate file (no technical reason, just personal preference)
• Made InFlightRequest internal
• Added unit tests for InFlightRequest
• Added some optimizations to reduce resource consumption if no timeout is specified. Now, a (linked) CTS is only created if a timeout is specified.
• Further reduced allocations by avoiding closure over the TCS
• Changed the cancellation behaviour to set a NATSTimeoutException only if cancellation wasn't caused by a client provided CT. This is a potentially breaking change. Before this, a NATSTimeoutException was always set as the TCS's exception when a timeout > 0 was passed in, even when a client provided CT caused cancellation (which may happen for any reason). Hence, clients that always used a timeout may not expect/handle the TaskCanceledException that now occurs if the client provided CT cancelled the request. However, I think that's better than throwing an unrelated exception.

Benchmark results

Source: https://github.com/jasper-d/nats.net/blob/benchmarks/inflightrequest/src/Benchmarks/MicroBenchmarks/InFlightRequestBenchmark.cs
The Categories column indicates whether CancellationToken.None (DefaultToken) or a "hot" CT (ClientToken) was passed to InFlightRequest. The Timeout column refers to the timeout arguments value passed to InFlightRequest.

The suffix _Upstream indicates that the current (less optimized) version of InFlightRequest was used.

Windows

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18362
Intel Core i7-4712MQ CPU 2.30GHz (Haswell), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-UIGYTS : .NET Framework 4.8 (4.8.4121.0), X64 RyuJIT
  Job-YMFKNL : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Method	Runtime	Categories	Timeout	Mean	Error	StdDev	Median	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
InFlightRequest	.NET 4.6.2	DefaultToken	0	81.81 ns	0.171 ns	0.160 ns	81.78 ns	0.22	0.00	0.1019	-	-	321 B
InFlightRequest_Upstream	.NET 4.6.2	DefaultToken	0	369.31 ns	0.417 ns	0.370 ns	369.39 ns	1.00	0.00	0.2193	-	-	690 B
InFlightRequest_ClientToken	.NET 4.6.2	ClientToken	0	235.39 ns	0.873 ns	0.817 ns	235.30 ns	0.46	0.00	0.1197	-	-	377 B
InFlightRequest_ClientToken_Upstream	.NET 4.6.2	ClientToken	0	516.48 ns	0.474 ns	0.443 ns	516.59 ns	1.00	0.00	0.2623	-	-	826 B
InFlightRequest	.NET Core 3.1	DefaultToken	0	71.51 ns	0.145 ns	0.121 ns	71.53 ns	0.32	0.00	0.0969	-	-	304 B
InFlightRequest_Upstream	.NET Core 3.1	DefaultToken	0	226.44 ns	2.522 ns	2.236 ns	226.53 ns	1.00	0.00	0.1938	-	-	608 B
InFlightRequest_ClientToken	.NET Core 3.1	ClientToken	0	155.03 ns	3.099 ns	2.899 ns	153.63 ns	0.46	0.01	0.0968	-	-	304 B
InFlightRequest_ClientToken_Upstream	.NET Core 3.1	ClientToken	0	337.62 ns	6.852 ns	6.074 ns	335.84 ns	1.00	0.00	0.2089	-	-	656 B
InFlightRequest	.NET 4.6.2	DefaultToken	999999	822.57 ns	5.931 ns	4.953 ns	820.88 ns	1.00	0.02	0.2594	-	-	818 B
InFlightRequest_Upstream	.NET 4.6.2	DefaultToken	999999	823.71 ns	16.364 ns	16.072 ns	818.51 ns	1.00	0.00	0.2651	-	-	834 B
InFlightRequest_ClientToken	.NET 4.6.2	ClientToken	999999	1,026.69 ns	19.878 ns	23.664 ns	1,027.80 ns	1.01	0.04	0.3033	-	-	955 B
InFlightRequest_ClientToken_Upstream	.NET 4.6.2	ClientToken	999999	1,012.73 ns	19.380 ns	18.128 ns	1,005.26 ns	1.00	0.00	0.3071	-	-	971 B
InFlightRequest	.NET Core 3.1	DefaultToken	999999	440.01 ns	8.882 ns	21.109 ns	433.87 ns	0.96	0.07	0.2193	-	-	688 B
InFlightRequest_Upstream	.NET Core 3.1	DefaultToken	999999	459.33 ns	10.002 ns	21.955 ns	454.94 ns	1.00	0.00	0.2241	-	-	704 B
InFlightRequest_ClientToken	.NET Core 3.1	ClientToken	999999	579.46 ns	13.371 ns	38.579 ns	566.98 ns	1.01	0.07	0.2346	-	-	736 B
InFlightRequest_ClientToken_Upstream	.NET Core 3.1	ClientToken	999999	577.11 ns	11.588 ns	26.626 ns	571.71 ns	1.00	0.00	0.2394	-	-	752 B

Linux

BenchmarkDotNet=v0.12.0, OS=fedora 31
Intel Core i7-3612QM CPU 2.10GHz (Ivy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-KAVYNT : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Runtime=.NET Core 3.1

Method	Categories	Timeout	Mean	Error	StdDev	Ratio	Gen 0	Gen 1	Gen 2	Allocated
InFlightRequest	DefaultToken	0	114.2 ns	0.23 ns	0.20 ns	0.34	0.0968	-	-	304 B
InFlightRequest_Upstream	DefaultToken	0	338.2 ns	2.31 ns	1.93 ns	1.00	0.1936	-	-	608 B
InFlightRequest_ClientToken	ClientToken	0	219.8 ns	1.11 ns	0.99 ns	0.47	0.0968	-	-	304 B
InFlightRequest_ClientToken_Upstream	ClientToken	0	465.3 ns	1.42 ns	1.33 ns	1.00	0.2089	-	-	656 B
InFlightRequest	DefaultToken	999999	537.0 ns	4.44 ns	4.15 ns	0.92	0.2193	-	-	688 B
InFlightRequest_Upstream	DefaultToken	999999	585.1 ns	0.37 ns	0.31 ns	1.00	0.2241	-	-	704 B
InFlightRequest_ClientToken	ClientToken	999999	674.1 ns	0.67 ns	0.62 ns	0.98	0.2346	-	-	736 B
InFlightRequest_ClientToken_Upstream	ClientToken	999999	688.4 ns	5.09 ns	4.51 ns	1.00	0.2394	-	-	752 B

Coalesced CT results

Not part of this PR but used to validate the rationale. Benchmarks to investigate if coalesced CTs can provide a benefit (depends on the actual scenario). Depending on the use case, this might also help in cases like #299.
Code: https://github.com/jasper-d/nats.net/blob/benchmarks/inflightrequest/src/Benchmarks/MicroBenchmarks/CoalescedTokenBenchmark.cs
InFlightRequest uses the optimized InFlightRequest with a timeout > 0.
InFlightRequest_Upstream_Coalesced uses the current implementation with coalesced CTs.
InFlightRequest_Coalesced uses the optimized version with coalesced CTs.

Windows

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18362
Intel Core i7-4712MQ CPU 2.30GHz (Haswell), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-UIGYTS : .NET Framework 4.8 (4.8.4121.0), X64 RyuJIT
  Job-YMFKNL : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Method	Runtime	Mean	Error	StdDev	Ratio	Gen 0	Gen 1	Gen 2	Allocated
InFlightRequest	.NET 4.6.2	805.8 ns	6.38 ns	5.33 ns	1.00	0.2594	-	-	818 B
InFlightRequest_Upstream_Coalesced	.NET 4.6.2	564.4 ns	1.46 ns	1.37 ns	0.70	0.2623	-	-	826 B
InFlightRequest_Coalesced	.NET 4.6.2	290.6 ns	0.87 ns	0.82 ns	0.36	0.1197	-	-	377 B
InFlightRequest	.NET Core 3.1	417.1 ns	3.74 ns	3.50 ns	1.00	0.2193	-	-	688 B
InFlightRequest_Upstream_Coalesced	.NET Core 3.1	361.5 ns	0.93 ns	0.82 ns	0.87	0.2089	-	-	656 B
InFlightRequest_Coalesced	.NET Core 3.1	191.4 ns	0.58 ns	0.54 ns	0.46	0.0968	-	-	304 B

Linux

BenchmarkDotNet=v0.12.0, OS=fedora 31
Intel Core i7-3612QM CPU 2.10GHz (Ivy Bridge), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.101
  [Host]     : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT
  Job-FCUXKY : .NET Core 3.1.1 (CoreCLR 4.700.19.60701, CoreFX 4.700.19.60801), X64 RyuJIT

Runtime=.NET Core 3.1

Method	Mean	Error	StdDev	Ratio	Gen 0	Gen 1	Gen 2	Allocated
InFlightRequest	537.3 ns	5.43 ns	5.08 ns	1.00	0.2193	-	-	688 B
InFlightRequest_Upstream_Coalesced	551.9 ns	0.89 ns	0.84 ns	1.03	0.2089	-	-	656 B
InFlightRequest_Coalesced	285.7 ns	0.60 ns	0.47 ns	0.53	0.0968	-	-	304 B

[watfordgnf]

I have no concerns with the original TaskCancellationException coming through for a client cancellation. @ColinSullivan1 @danielwertheim any objections?

[watfordgnf]

Looks good, just one small question really.

[ColinSullivan1]

LGTM, thanks!

I'm usually resistant to breaking changes, but we're still pre 1.0, and it's the right thing to do. Expanding the exceptions thrown here shouldn't impact many users, if any at all.