Don’t attempt to reconnect swarm on failed join after timeout

[tonistiigi]

fixes #26646

The reproducible part of the bug was already fixed with the grpc changes in swarmkit, but this makes it more robust and makes it not rely on swarmkit timeouts.

The issue appeared because reconnecting expects state from remote hosts. There was no state because the join failed.

cc @mrjana

Signed-off-by: Tonis Tiigi [email protected]

[thaJeztah]

@tonistiigi is this only on master, or a fix for 1.12.1?

[tonistiigi]

@thaJeztah I think this can wait for v1.13

[LK4D4]

LGTM

[aaronlehmann]

Can we consider changing join behavior so that it doesn't keep retrying after the timeout? I think it's really unexpected for a node to keep trying to join a swarm after docker swarm join returned failure. Having it succeed potentially weeks or months later isn't useful behavior IMHO.

[tonistiigi]

@aaronlehmann I remember @aluzzardi saw it as an important feature. Actually, it seems that swarmkit has already started to move away from that model as for example in #26646 swarmkit doesn't try to connect until network returns but fails out quite soon so we never even reach the timeout anymore. I'm not sure if this is the case with all the possible scenarios. If it is then in Docker side we should just remove the timeout completely and swarmkit either has to join in a meaningful time or give up with an error.

[aaronlehmann]

@aluzzardi: Any thoughts?

[thaJeztah]

ping @aluzzardi PTAL!

[aluzzardi]

@aaronlehmann Well, right now we're half sync half async and we all agreed to move one way or another since right now automation is really painful.

I believe we had a chat offline a while ago where, if I remember correctly, decided to go the async route (and make the CLI look synchronous?)

[aluzzardi]

LGTM

[aaronlehmann]

LGTM