Stream stops receiving messages after changing connection from Wifi to Cellular #337

slavabulgakov · 2018-12-03T12:01:21Z

Question Checklist

Updated grpc-swift to the latest version
I read the Contribution Guidelines
I read README
I searched for existing GitHub issues (especially, there are many resolved issues related to SSL)

Question Subject

How to restart a stream after changing connection from Wifi to Cellular?

Question Description

I use SwiftGRPC in iOS app and I have the client like:

service UpdateService {
    rpc GetUpdates (UpdatesConnection) returns (stream Update);
}

I subscribe on connectivity state of channel:

updateClient.channel.subscribe { connectivityState in
...
}

Then I make request to create stream from server:

let call = try updateClient.getUpdates(UpdatesConnection()) { result in
	switch result.statusCode {
		…
	} 
}

When I change iPhone connection from Wifi to Cellular, the stream stops receiving messages from server but the connectivityState (subscription to connectivity state) and result.statusCode keeping silent. After few minutes connectivityState gets .idle and result.statusCode gets . unavailable. When result.statusCode gets . unavailable I trying to call let call = try updateClient.getUpdates(UpdatesConnection()) again, but stream doesn't receive messages anyway.
So I've added subscription on iOS network state (Reachability) and I destroy the updateClient and recreate it when connection get changed from WIfi to Cellular. Only after destroying and recreating the updateClient the stream works properly - it receive messages.
Am I doing right? Maybe my way is wrong and there is another way to restart stream after changing connection?

The text was updated successfully, but these errors were encountered:

MrMage · 2018-12-04T12:25:42Z

SwiftGRPC is using gRPC-Core, which is using BSD sockets under the hood (I think). It might be that these never get notified of the switch to cellular or don't cause the cellular radio get turned on (there was a discussion about this once in the SwiftNIO Github repo). Unfortunately, I don't know of a good solution to that besides your workaround. A NIO-based client implementation might do better in that regard (but only on iOS 12), but is still a while out and I can't tell yet whether that would really work better.

…

On 3. Dec 2018, at 13:01, Slava Bulgakov ***@***.***> wrote: Question Checklist Updated grpc-swift to the latest version I read the Contribution Guidelines I read README I searched for existing GitHub issues (especially, there are many resolved issues related to SSL) Question Subject How to restart a stream after changing connection from Wifi to Cellular? Question Description I use SwiftGRPC in iOS app and I have the client like: service UpdateService { rpc GetUpdates (UpdatesConnection) returns (stream Update); } I subscribe on connectivity state of channel: updateClient.channel.subscribe { connectivityState in ... } Then I make request to create stream from server: let call = try updateClient.getUpdates(UpdatesConnection()) { result in switch result.statusCode { … } } When I change iPhone connection from Wifi to Cellular, the stream stops receiving messages from server but the connectivityState (subscription to connectivity state) and result.statusCode keeping silent. After few minutes connectivityState gets .idle and result.statusCode gets . unavailable. When result.statusCode gets . unavailable I trying to call let call = try updateClient.getUpdates(UpdatesConnection()) again, but stream doesn't receive messages anyway. So I've added subscription on iOS network state (Reachability) and I destroy the updateClient and recreate it when connection get changed from WIfi to Cellular. Only after destroying and recreating the updateClient the stream works properly - it receive messages. Am I doing right? Maybe my way is wrong and there is another way to restart stream after changing connection? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

Addresses grpc#198. It's currently not possible to manually shut down a gRPC channel, which means that the only way to shut down a channel is to deallocate it. This requirement is a bit fragile for cases where a consumer wants to manually shut down a channel, since it requires confidence that nothing else is holding a strong reference to the channel. This PR: - Adds a `shutdown` function to `Channel`, allowing consumers to arbitrarily shut down connections - Validates that existing calls will throw errors when attempting to read/write from a previously shut down channel - Ensures creating new calls using previously shut down channels will result in the initializer throwing (already handled by code generators) This change is increasingly relevant because consumers will need to shut down channels and restart them to help mitigate issues when switching between wifi/cellular as mentioned here: grpc#337.

rebello95 · 2019-02-26T01:00:09Z

The current issue is extensively problematic because end consumers/calls are never notified if a connection goes offline. Instead, gRPC streams/channels stay active even though they're silently offline (though not destroyed). This can result in streams appearing to hang.

I was talking to @MrMage offline earlier today about this, and there are a few ideas for how we can potentially make this a bit easier for consumers to handle (though it's not possible to solve the problem altogether as long as we're using gRPC-Core for reasons outlined above).

Build a system that identifies when state changes occur that cause disconnections within gRPC-Core that it doesn't know about (i.e., online -> offline, cellular -> wifi, etc.) and notifies a callback/observer about them
In theory, we could have the channel hook into this system and restart its underlying gRPC-Core connection. However, this brings some additional complexity:

How do we notify the consumers of active calls using this channel of the fact that the connection is being restarted?
If an outgoing message is currently in-flight, do we need to maintain state to retry it?
If new calls are created or existing calls send outgoing requests, what do we do with these while the channel is reconnecting?
If the channel fails to reconnect, how long should we wait before notifying the consumer?

Expose functionality from #1, as well as the ability to call shutdown() on a channel. This allows end consumers to observe the changes outlined above and manually shutdown/recreate channels as they see fit

None of these options are super ideal. At the moment, I'm leaning towards option #3 above since it's the least complex option and still leaves room for implementing #2 if we want to in the future. Additionally, it allows for the most customizability from the consumer perspective.

Planning to investigate this a bit more tomorrow, and I'm curious to hear other thoughts!

anjmao · 2019-02-26T07:11:53Z

@rebello95 You can to setup keep alive https://github.com/grpc/grpc-go/blob/master/examples/features/keepalive/server/main.go#L44 so server pings client and automatically closes the connection if client is offline.

MrMage · 2019-02-26T09:53:19Z

I agree that how one wants to handle this would probably be very application-specific, so letting users control the process should be advisable. We might want to provide guidance or suggestions on a "best practice" to handle this (possibly with a simple, optional class that can do this for the user), but give the user to handle this themselves in a custom fashion if desired.

Addresses grpc#198. It's currently not possible to manually shut down a gRPC channel, which means that the only way to shut down a channel is to deallocate it. This requirement is a bit fragile for cases where a consumer wants to manually shut down a channel, since it requires confidence that nothing else is holding a strong reference to the channel. This PR: - Adds a `shutdown` function to `Channel`, allowing consumers to arbitrarily shut down connections - Validates that existing calls will throw errors when attempting to read/write from a previously shut down channel - Ensures creating new calls using previously shut down channels will result in the initializer throwing (already handled by code generators) This change is increasingly relevant because consumers will need to shut down channels and restart them to help mitigate issues when switching between wifi/cellular as mentioned here: grpc#337.

Addresses grpc#198. It's currently not possible to manually shut down a gRPC channel, which means that the only way to shut down a channel is to deallocate it. This requirement is a bit fragile for cases where a consumer wants to manually shut down a channel, since it requires confidence that nothing else is holding a strong reference to the channel. This PR: - Adds a `shutdown` function to `Channel`, allowing consumers to arbitrarily shut down connections - Cancels all existing calls that are using the channel upon shutdown - Validates that existing calls will throw errors when attempting to read/write from a previously shut down channel - Ensures creating new calls using previously shut down channels will result in the initializer throwing (already handled by code generators) This change is increasingly relevant because consumers will need to shut down channels and restart them to help mitigate issues when switching between wifi/cellular as mentioned here: grpc#337.

rebello95 · 2019-02-28T00:47:30Z

Sounds good. I'll go ahead and do the following:

Add functionality for shutting down channels (in PR: Add ability to manually shut down channels #384)
Implement a system that observes these types of changes on the mobile client and notifies a callback as necessary
Write up known issues and guidance for how to avoid these on the client

rebello95 · 2019-02-28T01:53:43Z

@anjmao that also seems like a good improvement that we could expose from SwiftGRPC (documented further within the core here). It would likely aid in identifying connectivity states, but it won't fully solve the problem of channels silently disconnecting based on iOS state changes

markschmid · 2019-02-28T12:51:31Z

We've only just started looking into GRPC on mobile and have been running into this problem as well. Most often it is the case either if changing from WiFi to Cellular (or back) and/or if the (WiFi/cellular) signal gets weak. It seems that Channels can be configured e.g. with an argument keepAliveTimeout but it seems unclear if that works (and how/where notifications about it would occur).

* Add ability to manually shut down channels Addresses #198. It's currently not possible to manually shut down a gRPC channel, which means that the only way to shut down a channel is to deallocate it. This requirement is a bit fragile for cases where a consumer wants to manually shut down a channel, since it requires confidence that nothing else is holding a strong reference to the channel. This PR: - Adds a `shutdown` function to `Channel`, allowing consumers to arbitrarily shut down connections - Cancels all existing calls that are using the channel upon shutdown - Validates that existing calls will throw errors when attempting to read/write from a previously shut down channel - Ensures creating new calls using previously shut down channels will result in the initializer throwing (already handled by code generators) This change is increasingly relevant because consumers will need to shut down channels and restart them to help mitigate issues when switching between wifi/cellular as mentioned here: #337. * CR * try! -> try * support linux * CR

The SwiftGRPC implementation that is backed by [gRPC-Core](https://github.com/grpc/grpc) (and not SwiftNIO) is known to have some connectivity issues on iOS clients - namely, silently disconnecting (making it seem like active calls/connections are hanging) when switching between wifi <> cellular. The root cause of these problems is that the backing gRPC-Core doesn't get the optimizations made by iOS' networking stack when these types of changes occur, and isn't able to handle them itself. To aid in this problem, we're adding a [`ClientNetworkMonitor`](./Sources/SwiftGRPC/Core/ClientNetworkMonitor.swift) that monitors the device for events that can cause gRPC to disconnect silently. We recommend utilizing this component to call `shutdown()` (or destroy) any active `Channel` instances, and start new ones when the network is reachable. Details: - **Switching between wifi <> cellular:** Channels silently disconnect - **Network becoming unreachable:** Most times channels will time out after a few seconds, but `ClientNetworkMonitor` will notify of these changes much faster - **Switching between background <> foreground:** No known issues Original issue: grpc#337.

rebello95 · 2019-03-01T05:16:17Z

Yep @markschmid setting that would probably help with connectivity issues, but likely won't be very quick to notify you of those problems since the connection would have to time out first.

rebello95 · 2019-03-01T05:16:33Z

Implemented the ClientNetworkMonitor and added some docs here: #387

The SwiftGRPC implementation that is backed by [gRPC-Core](https://github.com/grpc/grpc) (and not SwiftNIO) is known to have some connectivity issues on iOS clients - namely, silently disconnecting (making it seem like active calls/connections are hanging) when switching between wifi <> cellular. The root cause of these problems is that the backing gRPC-Core doesn't get the optimizations made by iOS' networking stack when these types of changes occur, and isn't able to handle them itself. To aid in this problem, we're adding a [`ClientNetworkMonitor`](./Sources/SwiftGRPC/Core/ClientNetworkMonitor.swift) that monitors the device for events that can cause gRPC to disconnect silently. We recommend utilizing this component to call `shutdown()` (or destroy) any active `Channel` instances, and start new ones when the network is reachable. Details: - **Switching between wifi <> cellular:** Channels silently disconnect - **Network becoming unreachable:** Most times channels will time out after a few seconds, but `ClientNetworkMonitor` will notify of these changes much faster - **Switching between background <> foreground:** No known issues Original issue: grpc#337.

rebello95 · 2019-03-01T18:51:38Z

Also want to note that there's documentation of this behavior in the Objective-C implementation, and they took a similar approach to solving this problem (though it doesn't look like they handle the case of disconnecting when switching between 3G <> LTE): https://github.com/grpc/grpc/blob/v1.19.0/src/objective-c/NetworkTransitionBehavior.md

The SwiftGRPC implementation that is backed by [gRPC-Core(https://github.com/grpc/grpc) (and not SwiftNIO) is known to have some connectivity issues on iOS clients - namely, silently disconnecting (making it seem like active calls/connections are hanging) when switching between wifi <> cellular. The root cause of these problems is that the backing gRPC-Core doesn't get the optimizations made by iOS' networking stack when these types of changes occur, and isn't able to handle them itself. There is also documentation of this behavior in [this gRPC-Core readme](https://github.com/grpc/grpc/blob/v1.19.0/src/objective-c/NetworkTransitionBehavior.md). To aid in this problem, we're adding a [`ClientNetworkMonitor`](./Sources/SwiftGRPC/Core/ClientNetworkMonitor.swift) that monitors the device for events that can cause gRPC to disconnect silently. We recommend utilizing this component to call `shutdown()` (or destroy) any active `Channel` instances, and start new ones when the network is reachable. Details: - **Switching between wifi <> cellular:** Channels silently disconnect - **Switching between 3G <> LTE (etc.):** Channels silently disconnect - **Network becoming unreachable:** Most times channels will time out after a few seconds, but `ClientNetworkMonitor` will notify of these changes much faster - **Switching between background <> foreground:** No known issues Original issue: #337.

rebello95 · 2019-03-01T19:49:35Z

Took a long ride this morning going around San Francisco evaluating how well iOS is able to maintain a gRPC connection. Things I noticed (pretty much confirming my comments above):

Switching between wifi <> cellular causes connections to silently disconnect
Switching between 3G <> LTE causes connections to silently disconnect
gRPC does time out and shut down channels as expected in cases of poor connectivity
Backgrounding the app and foregrounding it again a few seconds later doesn't kill the connection - previously streamed responses come in upon foregrounding
Switching between cell towers seems to work fine and not cause a disconnection

The changes that cause silent disconnects were caught by the observer I added in #387, so consumers should be able to resolve these issues by utilizing that.

I also just issued a 0.8.0 release which includes these changes.

rebello95 · 2019-03-01T19:50:08Z

Going to close out this issue since it's now documented in the readme and there are workarounds on the latest release. Feel free to re-open with new concerns!

Najdan · 2019-04-03T15:14:11Z

Would update to gRPC-Core 1.13.0 dependency fix this issue?

ConnectivityMonitor is able to detect the scenario of the first issue above and actively destroy the channels. However, the second issue is not resolvable. To solve that issue the best solution is to switch to CFStream implementation which eliminates all of them.

gRPC iOS with CFStream
gRPC iOS with CFStream implementation (introduced in v1.13.0) uses Apple's networking API to make connections. It resolves the issues with TCP sockets mentioned above. Users are recommended to use this implementation rather than TCP socket implementation. The detailed behavior of streams in CFStream is not documented by Apple, but our experiments show that it accords to the expected behaviors. With CFStream implementation, an event is always received when the underlying connection is no longer viable. For more detailed information and usages of CFStream implementation, refer to the user guide.

https://github.com/grpc/grpc/blob/v1.19.0/src/objective-c/NetworkTransitionBehavior.md

rebello95 · 2019-04-03T20:11:28Z

Simply updating to 1.13.0 wouldn't solve this problem. The CFStream implementation mentioned might fix this issue, but it's currently gated by compiler conditionals upstream since it's marked as "experimental". If we wanted to utilize it today in SwiftGRPC, we could theoretically add branching logic for using either CFStream or the existing gRPC-Core implementation, but this would be a bit challenging given we have to support Carthage, Swift PM, and CocoaPods.

I do think that upgrading gRPC-Core to use this implementation when CFStream becomes the primary implementation upstream for iOS would be ideal, though!

Westacular · 2019-05-09T19:07:21Z

According to this README the CFStream implementation will be default as of 1.21.0 (which is the current in-development version)

tsabirgaliev · 2019-08-17T06:10:03Z

Any updates on this since gRPC-Core has already shipped with CFStream as default backend?

MrMage · 2019-08-19T07:04:26Z

We are still using gRPC-Core 1.19; I think CFStream is not yet enabled there.

rebello95 · 2019-08-19T16:35:11Z

Correct, we aren't using a version of gRPC-Core with CFStream as the default. If you'd like to submit a PR updating, we're happy to review!

tsabirgaliev · 2019-11-13T18:16:41Z

We did smoke tests with version 0.10.0 and confirm that connectivityObserver is called when switching networks, while it doesn’t in version 0.8.0

* Add ability to manually shut down channels Addresses grpc/grpc-swift#198. It's currently not possible to manually shut down a gRPC channel, which means that the only way to shut down a channel is to deallocate it. This requirement is a bit fragile for cases where a consumer wants to manually shut down a channel, since it requires confidence that nothing else is holding a strong reference to the channel. This PR: - Adds a `shutdown` function to `Channel`, allowing consumers to arbitrarily shut down connections - Cancels all existing calls that are using the channel upon shutdown - Validates that existing calls will throw errors when attempting to read/write from a previously shut down channel - Ensures creating new calls using previously shut down channels will result in the initializer throwing (already handled by code generators) This change is increasingly relevant because consumers will need to shut down channels and restart them to help mitigate issues when switching between wifi/cellular as mentioned here: grpc/grpc-swift#337. * CR * try! -> try * support linux * CR

The SwiftGRPC implementation that is backed by [gRPC-Core(https://github.com/grpc/grpc) (and not SwiftNIO) is known to have some connectivity issues on iOS clients - namely, silently disconnecting (making it seem like active calls/connections are hanging) when switching between wifi <> cellular. The root cause of these problems is that the backing gRPC-Core doesn't get the optimizations made by iOS' networking stack when these types of changes occur, and isn't able to handle them itself. There is also documentation of this behavior in [this gRPC-Core readme](https://github.com/grpc/grpc/blob/v1.19.0/src/objective-c/NetworkTransitionBehavior.md). To aid in this problem, we're adding a [`ClientNetworkMonitor`](./Sources/SwiftGRPC/Core/ClientNetworkMonitor.swift) that monitors the device for events that can cause gRPC to disconnect silently. We recommend utilizing this component to call `shutdown()` (or destroy) any active `Channel` instances, and start new ones when the network is reachable. Details: - **Switching between wifi <> cellular:** Channels silently disconnect - **Switching between 3G <> LTE (etc.):** Channels silently disconnect - **Network becoming unreachable:** Most times channels will time out after a few seconds, but `ClientNetworkMonitor` will notify of these changes much faster - **Switching between background <> foreground:** No known issues Original issue: grpc/grpc-swift#337.

* Add ability to manually shut down channels Addresses grpc/grpc-swift#198. It's currently not possible to manually shut down a gRPC channel, which means that the only way to shut down a channel is to deallocate it. This requirement is a bit fragile for cases where a consumer wants to manually shut down a channel, since it requires confidence that nothing else is holding a strong reference to the channel. This PR: - Adds a `shutdown` function to `Channel`, allowing consumers to arbitrarily shut down connections - Cancels all existing calls that are using the channel upon shutdown - Validates that existing calls will throw errors when attempting to read/write from a previously shut down channel - Ensures creating new calls using previously shut down channels will result in the initializer throwing (already handled by code generators) This change is increasingly relevant because consumers will need to shut down channels and restart them to help mitigate issues when switching between wifi/cellular as mentioned here: grpc/grpc-swift#337. * CR * try! -> try * support linux * CR

The SwiftGRPC implementation that is backed by [gRPC-Core(https://github.com/grpc/grpc) (and not SwiftNIO) is known to have some connectivity issues on iOS clients - namely, silently disconnecting (making it seem like active calls/connections are hanging) when switching between wifi <> cellular. The root cause of these problems is that the backing gRPC-Core doesn't get the optimizations made by iOS' networking stack when these types of changes occur, and isn't able to handle them itself. There is also documentation of this behavior in [this gRPC-Core readme](https://github.com/grpc/grpc/blob/v1.19.0/src/objective-c/NetworkTransitionBehavior.md). To aid in this problem, we're adding a [`ClientNetworkMonitor`](./Sources/SwiftGRPC/Core/ClientNetworkMonitor.swift) that monitors the device for events that can cause gRPC to disconnect silently. We recommend utilizing this component to call `shutdown()` (or destroy) any active `Channel` instances, and start new ones when the network is reachable. Details: - **Switching between wifi <> cellular:** Channels silently disconnect - **Switching between 3G <> LTE (etc.):** Channels silently disconnect - **Network becoming unreachable:** Most times channels will time out after a few seconds, but `ClientNetworkMonitor` will notify of these changes much faster - **Switching between background <> foreground:** No known issues Original issue: grpc/grpc-swift#337.

rebello95 mentioned this issue Feb 26, 2019

Add ability to manually shut down channels #384

Merged

rebello95 mentioned this issue Mar 1, 2019

Add ClientNetworkMonitor for tracking network changes #387

Merged

rebello95 closed this as completed Mar 1, 2019

a-voronov mentioned this issue Jun 18, 2019

Channel connectivityState crash after shutdown #488

Open

3 tasks

rebello95 mentioned this issue Jun 28, 2019

ios: App lifecycle & network condition handling envoyproxy/envoy-mobile#128

Closed

tsabirgaliev mentioned this issue Aug 25, 2019

Update vendored gRPC-core to v1.23.0 #563

Merged

flagoworld mentioned this issue May 5, 2020

iOS WiFi-Cellular Switch For Progressive Download superpoweredSDK/Low-Latency-Android-iOS-Linux-Windows-tvOS-macOS-Interactive-Audio-Platform#619

Closed

datmassdevs mentioned this issue Dec 26, 2020

Stream stops receiving messages after changing connection from one wifi to another #1088

Closed

ThanhDat-Vo mentioned this issue Jun 26, 2022

Flutter iOS: Connection was disconnected after changing network state between wifi <> cellular (grpc-dart - version 3.0.2) grpc/grpc-dart#564

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream stops receiving messages after changing connection from Wifi to Cellular #337

Stream stops receiving messages after changing connection from Wifi to Cellular #337

slavabulgakov commented Dec 3, 2018

MrMage commented Dec 4, 2018 via email

rebello95 commented Feb 26, 2019

anjmao commented Feb 26, 2019

MrMage commented Feb 26, 2019

rebello95 commented Feb 28, 2019

rebello95 commented Feb 28, 2019

markschmid commented Feb 28, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

Najdan commented Apr 3, 2019

rebello95 commented Apr 3, 2019

Westacular commented May 9, 2019

tsabirgaliev commented Aug 17, 2019

MrMage commented Aug 19, 2019

rebello95 commented Aug 19, 2019

tsabirgaliev commented Nov 13, 2019

Stream stops receiving messages after changing connection from Wifi to Cellular #337

Stream stops receiving messages after changing connection from Wifi to Cellular #337

Comments

slavabulgakov commented Dec 3, 2018

Question Checklist

Question Subject

Question Description

MrMage commented Dec 4, 2018 via email

rebello95 commented Feb 26, 2019

anjmao commented Feb 26, 2019

MrMage commented Feb 26, 2019

rebello95 commented Feb 28, 2019

rebello95 commented Feb 28, 2019

markschmid commented Feb 28, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

rebello95 commented Mar 1, 2019

Najdan commented Apr 3, 2019

rebello95 commented Apr 3, 2019

Westacular commented May 9, 2019

tsabirgaliev commented Aug 17, 2019

MrMage commented Aug 19, 2019

rebello95 commented Aug 19, 2019

tsabirgaliev commented Nov 13, 2019