Cannot route to https URI: unexpected message type #228
Comments
|
Thanks. |
|
@spencergibb Repo is here: https://github.com/samtonyclarke/spring-cloud-gw-issue-228 |
samtonyclarke
changed the title
|
I'm seeing the same error "IllegalStateException: unexpected message type: DefaultHttpRequest" on Windows 7 (haven't tried it yet on Fedora yet) |
|
Having same issue with similar setup here - stress load lab - on M9. |
|
Had the same error with M9 on high load. Latest SNAPSHOT versions seem to fix it. |
|
@scheuchzer @spencergibb |
|
The reactor team is looking at things |
|
@spencergibb any update from the reactor team/issue to track on their end? |
|
There have been netty/reactor netty changes that should be available in spring boot 2.0.2.BUILD-SNAPSHOT and Finchley.BUILD-SNAPSHOT. |
Not seeing issues so far with this (abbreviated) |
|
I spoke a bit too soon. Still seeing some intermittently, but not nearly as often |
|
The exception is gone when using org.springframework.boot:spring-boot-starter-parent version 2.0.3-BUILD-SNAPSHOT this weekend (05/19/2018). A couple of hour plus runs were successful over the weekend with the project github.com/samtonyclarke/spring-cloud-gw-issue-228 |
|
I spoke to soon too. Even with 2.0.3.BUILD-SNAPSHOT, the test still fails. It can happen anywhere from a couple of minutes to a couple of hours. There is a new stacktrace: |
|
@spencergibb is this potentially a release blocker? |
|
I don't know. It's out of the gateways control. I let the reactor-netty folks know. |
|
@mkreider329 What OS are you using? The test stopped No errors on the gateway side |
|
@fitzoh any feedback? |
|
My tests were running on PCF, not sure on version info. |
|
@violetagg to I'm running on RHEL 7.3 "Linux linux03 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux" |
|
For reference, these appear to be the relevant reactor-netty issues: |
|
Did you observe any other exceptions than |
|
Yes, two exceptions were observed in the output:
Forgive the long output below, but it is the gateway output from test a run last week. Gateway output: |
|
There is a fairly simple test case to reproduce this https://github.com/samtonyclarke/spring-cloud-gw-issue-228 Just update the client DriveIt.java to make more than 200000 calls, which isn't enough to trigger the error anymore (I'd suggest at least 10000000 so it runs for several hours). https://github.com/samtonyclarke/spring-cloud-gw-issue-228/blob/master/test-driver/src/main/java/org/samtonyclarke/test/driver/DriveIt.java |
|
any update on this issue? |
|
@samtonyclarke I think that the issue is that when with the default
It is correlated with Src/Dst Port:
|
|
@violetagg Yes, the previous tests were all run on the same machine. I reran the test across three different linux boxes are the failure didn't occur. However that might just be because the timing is has changed and race condition doesn't happen any more. |
|
@samtonyclarke After investigating more the possible reasons about so many |
|
Hi, Can we close this issue? Regards, |
|
I'll close it and we can reopen if there are further issues. |
|
Hi, I've come across with the same problem, reproducing is trivial. Below is the stack trace. I'm using raw cloud-gateway and webflux which are both produced by Spring Initializr/no modifications, just added a uuid to represent each user's session. Load is generated using Gatling, 2000rps. Below is the gatling's scala file. After 11k requests, exception occurred. This number varies between tests. Gateway/reactor-netty's context when exception is occurred: Origin is unaware about exception Wireshark HTTP packages, user's first two request were delegated to origin, 3rd one wasn't forwarded. Last http package's details Tcp stream of the last http package(produces using right-click>follow TCP stream): All apps are run on my personal macbook, same exception also occurs on EC2/Linux per gateway/origin with genuine clients. Macbook specs: |
|
@bcelenk @violetagg |
|
Hi @tony-clarke-amdocs |
|
@violetagg @bcelenk |
|
@bcelenk more important is the version of reactor-netty |
|
@spencergibb from stack trace we can see it is 0.7.8.Release |
|
Yes, reactor netty version is 0.7.8, I didn't test it extensively by disabling pool since it significantly reduces throughput and unfortunately wouldn't be an option. |
|
@bcelenk thanks for the detailed explanation! @spencergibb @tony-clarke-amdocs I'm experiencing this exact problem as well, is there a way to try out |
|
Since issue title doesn't represent the "Connection closed prematurely" problem directly, and the issue itself is closed, I'm opening a new issue to reach out more people who might be able to provide a potential solution. |
|
@spencergibb @violetagg |
|
@tony-clarke-amdocs Any chance to test with Reactor Netty 0.7.11.RELEASE? |
|
@violetagg |
|
@tony-clarke-amdocs Any chance to tell us the scenario how to reproduce this or to provide some example? |
|
At a high level, take spring cloud Gateway with out of the box configuration. Then publish some API's to the gateway with https as the target URL. Then using a testing tool like Gailting send a many concurrent requests (we used 150) and run the test for 8 hours. Its difficult to reproduce in a short amount of time. We usually need about 6 to 8 hours before the first error occurs. |
|
@tony-clarke-amdocs Is your problem solved? I encountered the same problem in stress testing. |
|
Don't see that problem any more with latest netty (.7.14), but still seeing this one: #473 |
|
Using: ..mapped in bootstrap.yml: I get this: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Using GW M8 the following exception occurs when the GW tries to route to a URI over https.
I will create a github repo with a sample to recreate the problem.
The text was updated successfully, but these errors were encountered: