Traceback binding socket

[stevegrubb]

The code in netutil.py around line 163 calls sock.bind(sockaddr). This can raise an exception when the address family is not supported. When using getaddrinfo, you are supposed to try each address returned until it's successful or the addresses are exhausted. The code appears to be written to loop, but success should only be considered if sock.bind() doesn't cause an exception.

When starting jupyter-notebook on a system that does not have IPv6 enabled, I get the following traceback:

$ jupyter notebook
Traceback (most recent call last):
File "/usr/bin/jupyter-notebook", line 6, in
main()
File "/usr/lib/python3.6/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "", line 2, in initialize
File "/usr/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/notebook/notebookapp.py", line 1368, in initialize
self.init_webapp()
File "/usr/lib/python3.6/site-packages/notebook/notebookapp.py", line 1188, in init_webapp
self.http_server.listen(port, self.ip)
File "/usr/lib64/python3.6/site-packages/tornado/tcpserver.py", line 141, in listen
sockets = bind_sockets(port, address=address)
File "/usr/lib64/python3.6/site-packages/tornado/netutil.py", line 196, in bind_sockets
sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address

Adding a print(sockaddr) before sock.bind(sockaddr) gives the following:

('::1', 8888, 0, 0)

and

ping ::1

connect: Network is unreachable

I think the answer is wrap the sock.bind in a try/catch and continuing to iterate on error.

[bdarnell]

This sounds like a duplicate of #2199. Everything is fine with ipv6 either enabled or disabled; the problem appears to be that your environment has it partially enabled. An ipv6 address appears when resolving localhost, but it doesn't work when binding it.

Swallowing this error is generally a bad idea. It would be fine to swallow EAFNOSUPPORT, but EADDRNOTAVAIL is risky because it's raised in more situations including those where an ip address is simply mistyped.

Can you shed any more light on how your environment came to be configured in this way?

[stevegrubb]

Maybe it is a dupe. Haven't read in detail. Linux always keeps IPv6 half enabled because the IPv4 and IPv6 stack share routing table and ARP implementation and more. When its turned off, its only turning off the outer layer that you access. I don't know how getaddrinfo is deciding ::1 is a good idea. Maybe that is a separate problem. But the documentation for it says to keep iterating until successful.

Hiding the error is not exactly what I'm suggesting. You can catch the exception and output a warning to stderr and then iterate to the next address offered by getaddrinfo since the IPv4 address is next in the list. Because IPv6 can include IPv4 addresses, its offered first. But we can just iterate to the next address. If the list becomes exhausted, then a stronger message is appropriate and perhaps even exiting the application. But a traceback leaves people with more questions than actionable answers.

The current way to disable IPv6 is add the following sysctls:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

and that will do it. Thanks for looking into this issue. I appreciate it!

[bdarnell]

Linux always keeps IPv6 half enabled because the IPv4 and IPv6 stack share routing table and ARP implementation and more. When its turned off, its only turning off the outer layer that you access.

So if it doesn't save you any resources in the kernel, why would you ever half disable it? I understand that you may not have an ipv6 address on the eth0 interface, but I don't see any reason to disable ::1 on loopback.

I don't know how getaddrinfo is deciding ::1 is a good idea.

Yes, that's the question. Can you try this and let me know if the AI_ADDRCONFIG flag makes a difference? (we used to set this flag, but stopped in #593)

>>> socket.getaddrinfo('localhost', 80, socket.AF_UNSPEC, socket.SOCK_STREAM, 0, socket.AI_PASSIVE)
[(30, 1, 6, '', ('::1', 80, 0, 0)), (2, 1, 6, '', ('127.0.0.1', 80))]
>>> socket.getaddrinfo('localhost', 80, socket.AF_UNSPEC, socket.SOCK_STREAM, 0, socket.AI_PASSIVE|socket.AI_ADDRCONFIG)
[(30, 1, 6, '', ('::1', 80, 0, 0)), (2, 1, 6, '', ('127.0.0.1', 80))]

But the documentation for it says to keep iterating until successful.

Which docs? For client connections you have to iterate because you can't know whether the remote node is up, but for local connections it should know which ones will work. ("If the AI_PASSIVE flag is specified in hints.ai_flags, and node is NULL, then the returned socket addresses will be suitable for bind(2)ing a socket that will accept(2) connections.").

Hiding the error is not exactly what I'm suggesting. You can catch the exception and output a warning to stderr

That's hiding the error in my book. If we can't do what the user asked we should raise an exception, not just move on and log. What if you have multiple network interfaces (and therefore multiple addresses in the same family, as documented here)? It's better to alert the user to the misconfiguration than to try and continue.

As a workaround for all of this, you may want to explicitly bind 127.0.0.1 instead of localhost. This will unambiguously indicate that you only want to use ipv4 even though your resolver is inconsistent with your kernel configuration.

[bdarnell]

I'm taking another look at this because docker uses this configuration by default: ipv6 support is present in the kernel but disabled, and localhost resolves to both ipv4 and ipv6 because of entries in /etc/hosts (docker/for-linux#250).

I think it's worth doing something about this, but it should be narrowly targeted instead of swallowing all EADDRNOTAVAIL. I think it should probably just be for this error when the requested hostname was localhost and the address that failed is ::1. If it's anything else, the user is probably trying something deliberate and would want to know if it didn't work.