-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Spurious stack traces in log when EventHubProducerClient encounters onSessionRemoteClose #41584
Comments
Thank you for your feedback. Tagging and routing to the team member best able to assist. |
We're also encountering this issue. Sounds like with the other two issues mentioned above it's pretty wide spread for apps that have a low volume period. The comment on #41535 about a V2 beta opt-in isn't very preferable for an immediate fix for production environments. As is we're looking at if a version downgrade helps the problem or if switching to the non-async client and initializing/disposing of the client periodically is a viable workaround. Having a fix for the async client would be very much preferred and I'm surprised this is made it through testing when it sounds like there are tests for low frequency clients in place. |
To clarify, the cause of Service Bus issue 41489 is unrelated though both log shows link disconnects; Service Bus 41489 was a thread bottleneck problem. The Service Bus has been running on the V2 stack engine for several months and is generally available. Event Hub is currently being integrated into the V2 stack engine and is in the testing and feature parity phase, with a beta release in progress. From our local testing, EH with V2 stack engine do not face this issue observed in lower traffic environment. Between @djarnis73, I'll clean up your comment in 41489 to prevent any confusion for others running into issue similar to that Service Bus case or this Event Hub issue. Hope that's alright with you. |
Sure, go ahead. I'm not too deep into our application or the services it uses I just happened to be the one investigating the issues or log monitoring found. So a lot of guesswork on my side. |
I'm going to try to implement a scheduled sending of keep-alive messages (once every minute) to see if that will stop the bleeding, this will of course require the receiving end being able to cope with these messages. Will revert with my findings. |
I think trying out the V2 beta would be an option for us. Where can I find the details on how to test it? |
So I'm pretty clueless when it comes to reactive programming but did some googling and was wondering if adding something like this:
would ensure that the exception would travel up the reactive stack and finally be thrown as an exception to the caller? As to be notified about the failure? In our case we are passing messages around and if we got an exception we could put the original message on a dead-letter-queue to avoid losing it. |
Hi @djarnis73, unfortunately re-throwing error from the global Reactor hook will not bubble the error to the actual send API call that application made. If we look at the first call stack in the issue description, it does not have send API call in the stack, so this error we re-throw from the hook will be signaled to thread happens to invoke the hook, likely goes to the default uncaught exception handler associated with the thread. |
The beta is available as: azure-messaging-eventhubs 5.19.0-beta.2. Setting the environment variable new EventHubClientBuilder()
.connectionString(properties.connectionString())
.configuration(new com.azure.core.util.ConfigurationBuilder()
.putProperty("com.azure.messaging.eventhubs.v2", "true")
.build())
.buildProducerClient(); |
Initial result of this looks good, no exceptions since it was deployed around 18 hours ago. But I do feel it is only a matter of time before another |
I deployed a version of our app to our test environment with V2 enabled and my manual keep alive removed. It does not seem to resolve the issue (the errors in the logs are back). So it looks like V2 has not fixed the issue. Any clues on how to investigate this any further (like enable debug logging for specific logger to ensure I have enabled V2 correctly)? |
Hi @djarnis73 - regarding logging, a reference log4j2.xml and logback.xml can be found here with DEBUG logging enabled. To verify V2 stack is enabled, we can check if there are logs with class name "com.azure.core.amqp.implementation.ReactorConnectionCache", which should be present at the start of the logs when a connection is established. To give an additional context, even with v2, we will still see the disconnect events - "onSessionRemoteClose", "the connection was closed by container…". The broker will disconnect if there is no activity, and the next send attempt by the application should force library to reconnect. Can we check
|
@anuchandy
After the first idle timeout connection closed by container we see the idle timeout exception 3 times over a period of about 1 minute immediately followed by the stacktrace related to |
@clevelandcs, thanks for getting back. |
I'm i bit hung up due to some problem regarding deployment to our test environment. But I have been analyzing number of events on producer and consumer side and they are identical, so I do not think the library is dropping events (on the producer side at least). So I think we should change the title of this issue to something in the line of "Spurious stack traces in log when EventHubProducerClient encounters onSessionRemoteClose". |
Thanks for the update @djarnis73. We couldn't replicate the NPE locally or in our testing setup, but I think the rework in this PR should resolve the potential NPE edge cases. (I'll update issue title per your latest comment) |
Ok, so I managed to deploy V2 on our test environment on Friday, so it has now been running over the weekend, what I have observed so far is:
Let me know what else I can provide (I just now managed to enable full debug logging, so I can probably provide some of those tomorrow). |
Thanks @djarnis73. Could you retrieve the logs for 20 minutes prior to and 20 minutes following one of the NPE instances? Please keep the thread name, timestamp also in the logs. |
Hi I have uploaded an extract from our datadog logs ±20 minutes around the stack trace (nothing happens 2 seconds after the exception so it is mainly stuff before). It is a csv with a timestamp column and then a json object with the full structured log data. Let me know if it is useful. |
@djarnis73, I'm looking at the logs. Could you please keep the link names in the logs so we can corelate actions on same links? (The link name is not sensitive data in case you're worried about it). Also, could you pull an additional -40 logs, i.e. total -60 minutes around the stack trace, so we can also trace a few token refresh operations. |
Hi @anuchandy , the link names are all the same, I did a simple search and replace for a single word, so you can treat all the entries as being the the same. If you need to correlate it with server side value I can share it with you. |
Thanks @djarnis73 for confirming that there is only one link, there is no need to correlate with the server traces at the moment.
|
Hi @anuchandy I have attached to extended log extract (I also added a few extra columns to try to workaround that datadog does not handle that you log the 'message' in I don't have direct access to cpu / memory info about the docker instance, but will reach out to our devops to to see if I can provide that information, I did check the metrics and cpu usage for this service is never above 1% (low volume very simple functionality). |
Hi @djarnis73, thanks for the detailed log and cleaning up it, it was very useful! I did the RCA - The Connection supports various communication pipes: channel, send link, and receive link. One specific channel, CBS, handles authentication and can face transient disconnects. We use an AmqpChannelProcessor (ACP) to automatically recreate the CBS channel if it disconnects, provided the parent Connection is healthy. If the parent Connection disconnects, the ACP is shut down to prevent it from creating new CBS channels. The issue was that when the connection disconnected, the connection initiated the ACP shutdown, but a Thread in the ACP attempted to recreate the channel before the shutdown process was fully completed. Attempt to use a disconnected connection caused the NPE. Depending on the timing, this issue may or may not occur, for instance, if the Thread in the ACP receives the shutdown signal (from connection Thread) before it retries channel creation. We're introducing a new type called RequestResponseChannelCache to replace AmqpChannelProcessor, avoiding channel creation on disposed connections. These changes have already merged and will be included in the September release happening soon. In the Event Hubs September version, the RequestResponseChannelCache will be automatically selected when the V2 stack is opted in. I'll update the thread once the new version is out so you can validate it in the test environment. thank you! |
This issue also happened in our environment. When will this release with fix be published? Thanks. |
Great news, I will test it out when it becomes available. |
Hello @yeah007, assuming no unexpected issues, the updated Event Hubs library with the fix should be available before September 26th. |
Hello @djarnis73, Sounds good. Also, if you hear from the Dev Ops about the cores allocated to the impacted Docker instance, please share it, keeping environment details is useful for future reference, even after RCA. |
Hello @anuchandy , Is the last sentence also true with the current version ? Or will the auto-reconnect only happen in the V2 ? tl;dr; can we ignore these errors ? |
Hello @s-vivien, regardless of V1 or V2, the reconnect will occur when the application calls |
Hello @djarnis73, @clevelandcs, we've just released the beta.3 with the new AMQP session cache, appreciate if you can give it a try and see experience is improved. To verify,
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-eventhubs</artifactId>
<version>5.19.0-beta.3</version>
</dependency>
Use new EventHubClientBuilder()
.connectionString(properties.connectionString())
.configuration(new com.azure.core.util.ConfigurationBuilder()
.putProperty("com.azure.messaging.eventhubs.v2", "true")
.build())
.buildProducerClient();
Use
Thanks, |
Describe the bug
We have a service to sends messages to the eventhub using the
com.azure:azure-messaging-eventhubs:5.18.6
artifact, with very low volume (less than 200 msg/day).Recently (since Aug. 17) we started seeing regular exceptions in our logs which indicates a bug in the library that leads to dropped messages.
Exception or Stack Trace
We se a batch of 4 log.errors and 1 log.warn from the library. First we get a log.error by reactor.core.publisher.Operators:
Then we get a log.warn from
com.azure.core.amqp.implementation.RequestResponseChannel
Then another log.error from
reactor.core.publisher.Operators
with messageOperator called default onErrorDropped
and stacktrace:Then another log.error from
com.azure.core.amqp.implementation.AmqpChannelProcessor
with the exception message:And finally an internal NPE, logged by
reactor.core.publisher.Operators
(which I suspect leads to the dropped message):To Reproduce
I do not have steps to reproduce, but I can see we are not the only one seeing the issue (looks related to #41535
and perhaps 41489), But I do suspect a simple app that sends test messages at a very low frequency will trigger the issue.Code Snippet
We instantiate our client very simple:
And the sending code is like:
Expected behavior
If the server disconnects the client due to inactivity, the client should notice and reconnect at next send attempt.
Setup (please complete the following information):
Additional information
I suspect an infrastructure applied change around August 17. that perhaps lowered the timeout on the server side has lead to this surfacing.
The text was updated successfully, but these errors were encountered: