-
Notifications
You must be signed in to change notification settings - Fork 647
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection Closed Prematurely #413
Comments
@bcelenk Can you try different settings for Customise it with the code below
|
@bcelenk were you able to play with the TCP backlog configuration. I was thinking whether this is something like this one here spring-cloud/spring-cloud-gateway#228 (comment)
|
Hi @violetagg Sorry about the delay, I'm on vacation. While investigating the related issues, I had seen that particular issue and changed origin to use There are various reasons to make me think like that. First of all, we are using the Second reason is that we're using our cloud provider's load balancer before our origin, which is proprietary software and couldn't fine tune these kind of backlog/connection settings. At the end of the day, a person might want to send requests to external systems using pooled What I'm pretty sure is that something is wrong with pooled Basically, this reproducer creates many channels in the pool, then restarts the origin. Which makes all of these channels in the pool invalid/useless. Then trigger another requests wave to the gateway, results with the mentioned exceptions. First of all, let's add a delay to the origin's handler method, hence
Using the Then create channels in the pool using the below command with wrk2 There shouldn't be any exceptions until now. And there are plenty of connections as the result of the Then restart the Because of the nature of the
|
@bcelenk Your last example was extremely helpful. Thanks for that. the corresponding logs
Do you think you can grab #419 and test the change? |
@violetagg Thank you for spotting that. Unfortunately, I'm still experiencing the same exception while trying the last example. I've cherry-picked #419 onto
|
Hi, I've seen some mentions of problems related to pooling - Does this fall into the same category? |
@dave-fl The problem that I described above so related to the pooling yes. |
@violetagg Will the use of netty-tcnative help here? |
@dave-fl It won't. I think the reason behind here would be the pool management itself. I tried @bcelenk 's sample as well and if the origin restarts itself, the gateway doesn't do a proper health check on the previously opened socket connections while trying the re-use them during the handshaking with the new origin. @violetagg do you have any comments on this one? if you can shed some light on this class-wise I can also take a look at it deeply. |
At any point do the connections move out of the invalid state and become useable again eg can this be worked around through retry? I’ve also read that a fixed pool can address this (not sure why). |
Hey, I added additional debugging to the 0.7.9.BUILD-SNAPSHOT.
Thanks a lot, |
Hey @violetagg,
For both origin implementations I experienced Wireshark captures are from the origin machine. |
@mulderbaba There was one use case where we missed to decrement the inactive counter (this is when the connection is created but cannot be acquired because of some exceptions in my case it was
|
@violetagg I made 2 runs with reactor-netty based gateway and origin. Using the latest snapshot of 0.7.x. First run contains all logs with debug mode. Here are the logs of gateway, logs of origin and tcpdump of origin: Second run only contains HttpClientOperations=DEBUG set. Here are the logs of gateway and tcpdump of origin: |
@violetagg Good afternoon, some possible date of solution for this problem? We are starting a project with spring webflux and we are a little concerned about the problem because we have a large volume of users, greetings and many thanks |
OS - macOS Sierra |
Please try |
Our team is also seeing the following error:
I'm not sure if this is the same issue |
@revfactory Please have in mind that this exception might not indicate a problem in Reactor Netty but that your server closes the connection on purpose (e.g. spring-cloud/spring-cloud-gateway#473 (comment)) so it is crucial for us to know your scenario. |
@violetagg - maybe my IOException is related to my docker swarm envirnoment - where I'm using IPVS load balancing - could You let me know if it could be related https://translate.google.pl/translate?sl=auto&tl=pl&u=http%3A%2F%2Fwww.anymk.com%2Fp%2Ff05294c0a456 ? I'm curnetly using 0.8.4-SNAPSHOT and have the same issue |
@violetagg looks like my problems were resolved by changing:
|
All, I committed another fix related to |
@violetagg For me after tweaking the ConnectionProvider from elastic to fixed, it fixed bunch of connection errors. Will post back if I see more issues from my side. |
Closing this one for now. If there are other use case than these fixed till now we can reopen it. |
@violetagg
I tried the new versions 0.7.12.RELEASE and run my program on Red Hat 4.8.5-16 server but result was same exceptions My environment list as follows: MacBoot Pro |
@xiaodanguoguo Try with 0.7.14.RELEASE, there are fixes for some use cases that might cause this exception to be thrown. |
@violetagg
and my gateway filter code:
Main class:
ServerHttpRequestDecorator class
I want to change the request url from the business interface number in the request body, |
@xiaodanguoguo What exactly are you trying to achieve with the code?
|
@violetagg |
@violetagg
|
@xiaodanguoguo Try to put somewhere this code as a project, it is hard to guess all the needed dependencies. Thanks. |
Hi all, |
@tareqhs Hi, Please create a new issue with a reproducible example OR scenario description. |
@violetagg I am also seeing the same issue. See: spring-cloud/spring-cloud-gateway#473 |
@tony-clarke-amdocs Are you able to test with |
Will try and get to it soon. Thanks! |
@violetagg So far this looks better. I have no been able to reproduce the problem. So whatever you did it looks better so far :) |
@tony-clarke-amdocs I backported several fixes from 0.8.x to 0.7.x |
�Hi, is this set in the container or on the host? |
Hi,
I've come across with the Connection closed prematurely problem, reproducing is trivial. Below is the stack trace.
I'm using raw cloud-gateway and webflux which are both produced by Spring Initializr/no modifications, just added a uuid to represent each user's session.
Load is generated using Gatling, 2000rps. Below is the gatling's scala file.
After 11k requests, exception occurred. This number varies between tests.
Gateway/reactor-netty's context when exception is occurred:
Origin is unaware about exception
Wireshark HTTP packages, user's first two request were delegated to origin, 3rd one wasn't forwarded.
Last http package's details
Tcp stream of the last http package(produces using right-click>follow TCP stream):
All apps are run on my personal macbook, same exception also occurs on EC2/Linux per gateway/origin with genuine clients.
Macbook specs:
macOS High Sierra 10.13.6 (17G65)
Processor 2.2 GHz Intel Core i7
Memory 16 GB 1600 MHz DDR3
Tried with 3 different jdks:
jdk1.8.0_131
jdk1.8.0_171
jdk1.8.0_181
This issue is directly addressed to reactor-netty.
The text was updated successfully, but these errors were encountered: