Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Error restarting gateway connections #1745

Closed
jvstme opened this issue Sep 30, 2024 · 0 comments · Fixed by #1746
Closed

[Bug]: Error restarting gateway connections #1745

jvstme opened this issue Sep 30, 2024 · 0 comments · Fixed by #1746
Assignees
Labels
bug Something isn't working gateways

Comments

@jvstme
Copy link
Collaborator

jvstme commented Sep 30, 2024

Steps to reproduce

  1. Create a gateway.
  2. Switch off the network on dstack-server's host and wait a few minutes (~5) until you see these messages in server logs:
Connection to gateway 0.0.0.0 failed: ssh: connect to host 0.0.0.0 port 22: Network is unreachable
  1. Switch the network back on.

Actual behaviour

The connection is never restarted, new errors appear in server logs when trying to restart. That means new services cannot be registered, old services don't work unless they are auth: false.

Expected behaviour

The connection is restarted soon after network is restored.

dstack version

master

Server logs

[00:24:23] INFO     dstack._internal.server.services.gateways.connection:81 Connection to gateway 0.0.0.0 is down, restarting                                                
[00:24:24] DEBUG    dstack._internal.core.services.ssh.tunnel:218 SSH tunnel failed: b"Warning: Permanently added '0.0.0.0' (ED25519) to the list of known hosts.\r\nError: remote port forwarding failed for listen path /home/ubuntu/dstack/server-connections/1f76da7e-77ef-4f36-97c1-585e87d2a42b.sock\r\n"                                                                               
           ERROR    dstack._internal.server.background.tasks.process_gateways:89 Connection to gateway 0.0.0.0 failed: Error: remote port forwarding failed for listen path /home/ubuntu/dstack/server-connections/1f76da7e-77ef-4f36-97c1-585e87d2a42b.sock

Additional information

Reason: the unix socket used for reverse forwarding remains on the gateway and prevents opening new connections with the same socket path.
Workaround: restart dstack server, it will generate a new socket path.

@jvstme jvstme added bug Something isn't working gateways labels Sep 30, 2024
@jvstme jvstme self-assigned this Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working gateways
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant