You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Switch off the network on dstack-server's host and wait a few minutes (~5) until you see these messages in server logs:
Connection to gateway 0.0.0.0 failed: ssh: connect to host 0.0.0.0 port 22: Network is unreachable
Switch the network back on.
Actual behaviour
The connection is never restarted, new errors appear in server logs when trying to restart. That means new services cannot be registered, old services don't work unless they are auth: false.
Expected behaviour
The connection is restarted soon after network is restored.
dstack version
master
Server logs
[00:24:23] INFO dstack._internal.server.services.gateways.connection:81 Connection to gateway 0.0.0.0 is down, restarting
[00:24:24] DEBUG dstack._internal.core.services.ssh.tunnel:218 SSH tunnel failed: b"Warning: Permanently added '0.0.0.0' (ED25519) to the list of known hosts.\r\nError: remote port forwarding failed for listen path /home/ubuntu/dstack/server-connections/1f76da7e-77ef-4f36-97c1-585e87d2a42b.sock\r\n"
ERROR dstack._internal.server.background.tasks.process_gateways:89 Connection to gateway 0.0.0.0 failed: Error: remote port forwarding failed for listen path /home/ubuntu/dstack/server-connections/1f76da7e-77ef-4f36-97c1-585e87d2a42b.sock
Additional information
Reason: the unix socket used for reverse forwarding remains on the gateway and prevents opening new connections with the same socket path.
Workaround: restart dstack server, it will generate a new socket path.
The text was updated successfully, but these errors were encountered:
Steps to reproduce
Actual behaviour
The connection is never restarted, new errors appear in server logs when trying to restart. That means new services cannot be registered, old services don't work unless they are
auth: false
.Expected behaviour
The connection is restarted soon after network is restored.
dstack version
master
Server logs
Additional information
Reason: the unix socket used for reverse forwarding remains on the gateway and prevents opening new connections with the same socket path.
Workaround: restart
dstack server
, it will generate a new socket path.The text was updated successfully, but these errors were encountered: