-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipelining causes NOAUTH errors with External Authentication on Envoy #2785
Comments
Are you putting your password in the ConfigurationOptions? because it should call the AUTH automatically. |
Yes, we are. This is really weird |
Maybe try giving each of your |
Yes all clusters are different endpoints... |
Using separate socket manager instance didn't fix the issue. |
Ok, this is very weird! To confirm, there should be no ambient state in play here, so I'm very confused. I'll ping you on Teams (myself and Nick are also MSFT) - it may be easier to get to the bottom of this directly rather than on GitHub (and obviously any fixes/knowledge-gained can still work their way back here). |
Just updating the public logs; envoy proxy is in play here; by adding RESP logging, it looks like we're seeing something very odd; outbound:
(where the inbound:
But with the weird timing complication; the other servers in the timing matrix are not behind envoy. Investigations continue. |
Updated the title and updating here with what we think is the root cause so far. Even though Envoy connections are single-threaded by nature, when external authentication (new feature, more info here) is enabled, the authentication is done asynchronously. This means that if the AUTH command is pipelined with the PING command, we will have exactly the repro we've seen. |
Emphasis: fundamentally this is an envoy bug. They must not process commands out of order (even if they buffer the response until appropriate) if the connection state is relevant, and here: the connection state is relevant. Is this logged with envoy? If not, I'm happy to log it. |
Envoy fix merged; closing - thanks for the report; between us we got to the bottom of it! |
Hello,
We have a situation where our Redis client is returning NOAUTH during the connection flow, but if we manually connect to it through redis-cli and send an
AUTH <password>
command, from that exact same machine, we get OK and can send other commands after.There is no place where we can see the actual result of the AUTH command sent by the SDK since, from what we've seen in code, it's a fire and forget message.
Any ideas how to debug this?
Important Note
We have multiple redis clusters that we're connecting to. If we put this cluster first, it works. If we put it last, it doesn't!
For example, this doesn't work:
But this does:
We're wondering if somehow the multiplexer might be using a wrong connection from a completely different instance. If there is some kind of static or thread-local state.
Some logs:
The text was updated successfully, but these errors were encountered: