Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xds: Update logic so that an error being reported when stream is closed gets propagated to subscribers #9827

Merged
merged 7 commits into from
Jan 25, 2023

Conversation

larry-safran
Copy link
Contributor

@larry-safran larry-safran commented Jan 14, 2023

Eliminated waitForReady in XdsClient's stub. (Addresses the needs of TD Client Migration team)

The AbstractAdsStream.handleRpcStreamClosed method is used to handle both rpc error and rpc completed. Previously, if AbstractAdsStream.closed == true, then all the logic in handleRpcStreamClosed was skipped. Now it is only skipped if shutdown == true.

Additionally, adds some cleanup.

Copy link
Contributor

@YifeiZhuang YifeiZhuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have question about the reason behind regarding the handleRpcStreamClosed part.
Do we see use cases that AdsStreams close improperly? Did we see handleRpcStreamClosed() called multiple times? Why we need the fix?

return;
}
Assert.fail("Expected exception for bad url not thrown");
verify(ldsResourceWatcher).onError(errorCaptor.capture());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these lines necessary after Assert.fail()? Also they seem incomplete about error verification.

Also it looks the test are not strictly testing the "narrowing down close condition" change at all.

It is a bit redundant with test case streamClosedAndRetryWithBackoff, if you wanted to test the close/retry.

The IllegalArgument may not be related to stream creation failure but about the xdstp thing?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put this in to verify that a malformed URL got processed in the expected way which was different than how a nonexistent host was handled. It had nothing to do with close or retry. Handling unknown URLs is what caused the user's problem and I noticed that both unknown and invalid url cases were missing.

Cleaned up the lines after the Assert.fail.

@larry-safran
Copy link
Contributor Author

In regards to handleRpcStreamClosed, it is used by both handleRpcCompleted() and handleRpcError(). In the case of the unknown URL, the AdsStream() hasn't ever been opened, so closed is true, but we want to propagate the error up to the subscribed observers. This change stopped the bypassing of xdsResponseHandler.handleStreamClosed(error).

@YifeiZhuang
Copy link
Contributor

In regards to handleRpcStreamClosed, it is used by both handleRpcCompleted() and handleRpcError(). In the case of the unknown URL, the AdsStream() hasn't ever been opened, so closed is true, but we want to propagate the error up to the subscribed observers. This change stopped the bypassing of xdsResponseHandler.handleStreamClosed(error).

  1. if adstream() has not been opened, the closed by default is false, isn't it?
  2. if there is no stream, how does it receive onError or onComplete?

I guess creating resourceSubscriber should never fail.
I probably missed something, can i take a look at stacktrace or debug message?

@larry-safran
Copy link
Contributor Author

I believe that you are correct that it wouldn't have mattered whether it was closed or shutdown that was checked. I definitely see handleRpcStreamClosed called after shutdown, so there needs to be some sort of check so that the rpcRetryTimer doesn't get triggered and shutdown is more descriptive of why that check is in place.

@larry-safran larry-safran added the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Jan 24, 2023
@grpc-kokoro grpc-kokoro removed the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Jan 24, 2023
@YifeiZhuang
Copy link
Contributor

There are side effects of shutdown check instead of close check: already closed stream may get retry if handleRpcStreamClosed called twice.

Also, expected behaviour of creating ResourceSubscribe is that is does not throw. If getServerInfo fails, the xdsChannel is null. Then there will be no AdsStream and the watcher will still be notified with error.

@larry-safran larry-safran merged commit 501ca8f into grpc:master Jan 25, 2023
@larry-safran larry-safran deleted the errFixXdsClient branch January 25, 2023 02:31
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants