interface/grpc/core: ListenTask gets canceled #549
Comments
ilgooz
changed the title
|
I think i've got that issue too, some execution were not executed. I looked at the logs but didn't see any logs with the stream closing but I had many logs so probably missed it. This one definitely need some investigations |
|
After some more experiments, as I remember this is not directly relavent with elapsed time or maybe it is but not too long like 40mins and it doesn't happen by itself, i think it gets triggered by a task execution. Cancellation logs will be shown depending on that. And I'm sure that this happens to me with service-webhook so it's not relevant with go-service client. |
|
I got this issue also working with only javascript services. This is the actual project I was working on https://github.com/antho1404/pegasys-hackathon but I couldn't find a way to reproduce yet |
|
What if is it because the MacBook go to sleep and close the tcp connection?
Maybe it’s because we don’t have any auto retry on the tcp connection.
… On 7 Nov 2018, at 21:31, Anthony ***@***.***> wrote:
I got this issue also working with only javascript services. This is the actual project I was working on https://github.com/antho1404/pegasys-hackathon but I couldn't find a way to reproduce yet
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@NicolasMahe yes, we should check that one as well. |
|
This is not relevant with mac going to sleep. I started a service and haven't put my mac on sleep. After one hour I tried to execute a task on that service and here I got the listen task canceled error at the bottom: |
|
For information, there is an error not returned here. If there is an error at the creation of the stream this error is not shown, maybe this is related. |
|
Long story short:
The INFO tells us an interesting thing - the error type is ErrConnClosing (https://github.com/grpc/grpc-go/blob/master/internal/transport/transport.go#L676) In such situation, client sees this as close stream. A solution, implement reconnection policy in the client, but this is the funniest part, not only in case of error, but also in case of normal connection close! func listen() {
for {
// increas sleep with every faild connection (restart after successful conneciton0)
// first try 1s, 5s, 20s
time.Sleep(1 * time.Second)
// (re)connect listen task api
stream, err := ListenTask(context.Background())
if err != nil {
// connect to listen task failed
continue
}
// buffered channel, because it might happen that ctx.Done
// will be notifed first before stream.Recv in for loop.
done := make(chan struct{}, 1)
go func(ctx context.Context) {
select {
case <-ctx.Done():
// log error from ctx.Err()
case <-done:
// do nothing
}
}(stream.Context())
for {
td, err := stream.Recv()
if err != nil {
// notify goroutine on any kind of error
done <- struct{}{}
break
}
// normal path with submit errors/results
}
// NOTE: even if listen task will be canceled without error
// it will still reconnect. So we need to create issue in grpc how to solve this (haven't seen)
// or implement this in our rpc protocol layer
}
}other solution: maybe there is an api that will allow us to distinguish normal connection close and ErrConnClosing |
|
@krhubert nice summary of the problem 👍 In the current implementation of stream in client side, there is nothing about the context (eg: I think the first implementation that needs to be done is to add support of @krhubert do we have a way to reproduce the error? even if it required to change the source of gRPC, we need to reproduce the error so we can actually test that the fix actually work. I moved the content of this message to the forum because it was not only related to this issue. With the proposed acknowledgement system, the server could also send last message (as a header or trailer) on the gRPC stream that indicates the stream will be close gracefully. Take a look at trailer: https://github.com/grpc/grpc-go/blob/master/Documentation/grpc-metadata.md#streaming-call-2 |
|
we don't have keepAlive set for underlying tcp connections so this issue might be relevant with that too. i'm not expecting this but maybe Docker's network is configured in a way to disconnect the idle connections –or might be a bug? |
|
What do you mean |
|
There is an option in net.Dialer to enable sending keep alive pings. I don't know if grpc-go client uses that or implements its own but we have an option for it. https://godoc.org/google.golang.org/grpc#WithKeepaliveParams I'll try this and provide the results. |
|
I'm testing the keepAlive, in the meanwhile, I've a few more findings:
|
|
Good news, I tested client side keep alive by waiting for 40mins. After that I executed a task on the service and there are no stream cancelation, it works! My configs were: grpc.WithKeepaliveParams(keepalive.ClientParameters{
Time: time.Minute * 2,
Timeout: time.Minute * 3,
PermitWithoutStream: true,
}),I'll try the same configs in server side. That way when can apply the configs on server side once, server sends the keep alive pings and clients doesn't need to worry about setting such a config. |
|
The way to reproduce this issue:
You will get an error on the server with The |
This happened to me with a service that uses go-service. And several times more in back with some other services that I don't remember (maybe webhook service). I'm not sure if I faced with this issue with mesg-js. So this might be relavent with go-service or can be caused from something totally else. The problem occurs a while after (<=40mins) starting the service which causes service to not react further task executions.
The text was updated successfully, but these errors were encountered: