Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tuple index out of range with 3.7.1 and no IPv6 #5156

Closed
danielnelson opened this issue Oct 26, 2020 · 23 comments · Fixed by #5176
Closed

Tuple index out of range with 3.7.1 and no IPv6 #5156

danielnelson opened this issue Oct 26, 2020 · 23 comments · Fixed by #5176
Labels
bug need pull request regression Something that used to work stopped working "as before" after upgrade reproducer: present This PR or issue contains code, which reproduce the problem described or clearly understandable STR

Comments

@danielnelson
Copy link
Contributor

danielnelson commented Oct 26, 2020

🐞 Describe the bug
After upgrading to aiohttp==3.7.1, dns resolution fails with an exception.

💡 To Reproduce

  1. I believe this is due to not have IPV6 enabled in my kernel, which is custom compiled with.
    CONFIG_IPV6=n
  2. Run:
    import asyncio
    import aiohttp
    
    
    async def main():
        async with aiohttp.ClientSession() as session:
            async with session.get('http://python.org') as response:
                print(response.status)
    
    
    asyncio.run(main())
  3. Actual behavior
    $ python bug.py
    Traceback (most recent call last):
      File "bug.py", line 11, in <module>
        asyncio.run(main())
      File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
        return loop.run_until_complete(main)
      File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
        return future.result()
      File "bug.py", line 7, in main
        async with session.get('http://python.org') as response:
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 1124, in __aenter__
        self._resp = await self._coro
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 528, in _request
        req, traces=traces, timeout=real_timeout
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 541, in connect
        proto = await self._create_connection(req, traces, timeout)
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 898, in _create_connection
        _, proto = await self._create_direct_connection(req, traces, timeout)
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 1005, in _create_direct_connection
        hosts = await asyncio.shield(host_resolved)
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 871, in _resolve_host
        addrs = await self._resolver.resolve(host, port, family=self._family)
      File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/resolver.py", line 37, in resolve
        if family == socket.AF_INET6 and address[3]:  # type: ignore
    IndexError: tuple index out of range

💡 Expected behavior

Should print 200.

📋 Logs/tracebacks

📋 Your version of the Python

$ python --version
Python 3.7.9

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
Name: aiohttp
Version: 3.7.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: [email protected]
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires: multidict, async-timeout, typing-extensions, yarl, chardet, attrs
Required-by: replay-proxy
$ python -m pip show multidict
Name: multidict
Version: 4.7.6
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires:
Required-by: yarl, aiohttp
$ python -m pip show yarl
Name: yarl
Version: 1.5.1
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /home/dbn/usr/py37/lib/python3.7/site-packages
Requires: typing-extensions, multidict, idna
Required-by: aiohttp, replay-proxy

📋 Additional context

@asvetlov
Copy link
Member

I'm curious what parameters are passed into getaddrinfo() in your case and what list is returned?
Could you print them in a comment?

My initial thought is: on systems with IPv6 not installed the check if family == socket.AF_INET6 and address[3]: should fail on family == socket.AF_INET6 test; adress[3] test is not executed in this case.

@danielnelson
Copy link
Contributor Author

I added some debug statements to resolver.py:

  async def resolve(
      self, hostname: str, port: int = 0, family: int = socket.AF_INET
  ) -> List[Dict[str, Any]]:
+     print('getaddrinfo(%s, %s, type=%s, family=%s)' % (hostname, port, type, family))
      infos = await self._loop.getaddrinfo(
          hostname, port, type=socket.SOCK_STREAM, family=family
      )
+     print('infos:', infos)
getaddrinfo(python.org, 80, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('45.55.99.72', 80))]
getaddrinfo(python.org, 443, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('45.55.99.72', 443))]
getaddrinfo(www.python.org, 443, type=<class 'type'>, family=0)
infos: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00'))]
Traceback (most recent call last):
  File "bug-5156.py", line 11, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "bug-5156.py", line 7, in main
    async with session.get('http://python.org') as response:
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 1124, in __aenter__
    self._resp = await self._coro
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/client.py", line 528, in _request
    req, traces=traces, timeout=real_timeout
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 541, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 898, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 1005, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/connector.py", line 871, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/home/dbn/usr/py37/lib/python3.7/site-packages/aiohttp/resolver.py", line 39, in resolve
    if family == socket.AF_INET6 and address[3]:  # type: ignore
IndexError: tuple index out of range

For reference, here is my current DNS resolution for www.python.org

$ host www.python.org
www.python.org is an alias for dualstack.python.map.fastly.net.
dualstack.python.map.fastly.net has address 151.101.188.223
dualstack.python.map.fastly.net has IPv6 address 2a04:4e42:2d::223

@asvetlov
Copy link
Member

Interesting...
In (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') the (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') doesn't look like IPv6 at all!
Is it correct?
I mean if I do host, port = (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00') -- both host and port are broken anyway.

@danielnelson
Copy link
Contributor Author

I found this which explains the cause: https://bugs.python.org/issue16208

FWIW, I was able to process the results of getaddrinfo in C. Here is some embarrassingly bad C code:

#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

#include <arpa/inet.h>
#include <stdlib.h>


#define BUF_SIZE 500

void
print_addr(struct addrinfo *result) {
	char *addr = NULL;
	switch(result->ai_family) {
		case AF_INET: {
			struct sockaddr_in *addr_in = (struct sockaddr_in *)result->ai_addr;
			addr = malloc(INET_ADDRSTRLEN);
			inet_ntop(AF_INET, &(addr_in->sin_addr), addr, INET_ADDRSTRLEN);
			break;
		}
		case AF_INET6: {
			struct sockaddr_in6 *addr_in6 = (struct sockaddr_in6 *)result->ai_addr;
			addr = malloc(INET6_ADDRSTRLEN);
			inet_ntop(AF_INET6, &(addr_in6->sin6_addr), addr, INET6_ADDRSTRLEN);
			break;
		}
		default:
			break;
	}
	printf("IP address: %s\n", addr);
	free(addr);
}

int
main(int argc, char *argv[])
{
	struct addrinfo hints;
	struct addrinfo *result, *rp;
	int sfd, s, j;
	size_t len;
	ssize_t nread;
	char buf[BUF_SIZE];

	if (argc < 3) {
		fprintf(stderr, "Usage: %s host port msg...\n", argv[0]);
		exit(EXIT_FAILURE);
	}

	memset(&hints, 0, sizeof(struct addrinfo));
	hints.ai_family = AF_UNSPEC;    /* Allow IPv4 or IPv6 */
	hints.ai_socktype = SOCK_DGRAM; /* Datagram socket */
	hints.ai_flags = 0;
	hints.ai_protocol = 0;          /* Any protocol */

	s = getaddrinfo(argv[1], argv[2], &hints, &result);
	if (s != 0) {
		fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(s));
		exit(EXIT_FAILURE);
	}

	while (result != 0) {
		print_addr(result);
		result = result->ai_next;
	}
}
$ gcc bug.c
$ ./a.out www.python.org 443
IP address: 151.101.188.223
IP address: 2a04:4e42:2d::223

I tried to do something similar in Python, I'm not sure if I'm doing it right though:

import socket

for ai in socket.getaddrinfo('www.python.org', 443):
    print(ai)
    if ai[0] == socket.AF_INET6:
        af, socktype, proto, canonname, sa = ai
        print("AF_INET6", socket.inet_ntop(socket.AF_INET6, sa[1]))
$ python getaddrinfo.py 
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443))
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('1imagine51.101.188.223', 443))
(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('151.101.188.223', 443))
(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00'))
Traceback (most recent call last):
  File "getaddrinfo.py", line 7, in <module>
    print("AF_INET6", socket.inet_ntop(socket.AF_INET6, sa[1]))
ValueError: unknown address family 10

I'm running a Gentoo system with the -ipv6 use flag set, but I enabled it for the Python build as it seems problematic.

Perhaps a fix in aiohttp would be to check at AF_INET6 addresses are a 4 tuple and ignore if they are not.

Also, I think this is the same issue as #4557

@webknjaz
Copy link
Member

I'm running a Gentoo system with the -ipv6 use flag set, but I enabled it for the Python build as it seems problematic.

So it's enabled in CPython but disabled in the kernel? What about sysctl? Could you post your sudo sysctl -a | grep ipv6 output? Does it have something like net.ipv6.conf.lo.disable_ipv6 set? How about other interfaces? Or is this setting just missing?

@webknjaz
Copy link
Member

Also, I think this is the same issue as #4557

So it should probably be mitigated with "has ipv6" check like https://github.com/tornadoweb/tornado/pull/670/files#diff-e9701aba4f8165d329018bef200b0d98dd70c24856c159b5c29cce89ddd69e8bR56

@asvetlov
Copy link
Member

Perhaps a fix in aiohttp would be to check at AF_INET6 addresses are a 4 tuple and ignore if they are not.

From a practical point of view, I'm ok with this solution if it raises a warning before the ignoring.
From my understanding, we see a perfect example of a gray corner case that we can pass around easily.
A detection for address tuple size smells bad but looks working.

@danielnelson
Copy link
Contributor Author

So it's enabled in CPython but disabled in the kernel? What about sysctl?

Sorry, my comment was a little confusing, when I reported this issue it was disabled in Python and in the Kernel. My workaround is to enable it in Python and leave it disabled in the Kernel.

Could you post your sudo sysctl -a | grep ipv6 output?

It looks like there are no results containing ipv6.

I like the socket.has_ipv6 solution, maybe we should test for this when creating the TCPConnector and set the family to AF_UNSPEC or AF_INET. Essentially the linked tornado code.

@webknjaz
Copy link
Member

So I looked at those bugs on CPython tracker and I'm puzzled why they are "wontfix". Moreover, I think that CPython may be incorrectly processing NUL-terminated strings or something like that.

I've tried decoding that bytesting manually on a Gentoo laptop with IPv6 on and still got an error (a different error but still):

>>> socket.inet_ntop(socket.AF_INET6, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00-\x00\x00')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid length of packed IP address string

Something about it looked weird and I tried to do the packing of a known address 2a04:4e42:41::223:

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('2a04:4e42:41::223', 443, 0, 0))]
>>> socket.inet_pton(socket.AF_INET6, '2a04:4e42:2d::223')
b'*\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#'

Woah! That looks different! Let's try unpacking that correct value, just to be sure:

>>> socket.inet_ntop(socket.AF_INET6, b'*\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#')
'2a04:4e42:2d::223'

Yaaay! We've got the correct value!

Based on this, I'm pretty much convinced that the bug in CPython originates in socket.getaddrinfo() returning a malformed packed IPv6 address.

P.S. It's interesting that both representations have identical *\x04NB\x00-\x00\x00 substring ("\x01\xbb\x00\x00\x00\x00 *\x04NB\x00-\x00\x00" and " *\x04NB\x00-\x00\x00\x00\x00\x00\x00\x00\x00\x02#") so I'm convinced it has something to do with processing the memory offsets on the C level.

@webknjaz
Copy link
Member

webknjaz commented Oct 29, 2020

So I looked up the getaddrinfo(3) manpage and it seems like the fix could be as easy as adding a socket.AI_ADDRCONFIG as a flag for the getaddrinfo() call:

If hints.ai_flags includes the AI_ADDRCONFIG flag, then IPv4
addresses are returned in the list pointed to by res only if the
local system has at least one IPv4 address configured, and IPv6
addresses are returned only if the local system has at least one IPv6
address configured.
The loopback address is not considered for this
case as valid as a configured address. This flag is useful on, for
example, IPv4-only systems
, to ensure that getaddrinfo() does not
return IPv6 socket addresses
that would always fail in connect(2) or
bind(2).

@danielnelson could you please try this in your Python REPL:

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)

and also (just in case):

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)

@webknjaz
Copy link
Member

Other observations:

  1. The broken bytesting is 14 bytes long while the correct one is 16 bytes
  2. I haven't noticed anything immediately obvious in https://github.com/python/cpython/blob/master/Modules/socketmodule.c so maybe this is how the underlying C-libs behave...

@danielnelson
Copy link
Contributor Author

danielnelson commented Oct 29, 2020

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.188.223', 443))]

@asvetlov
Copy link
Member

socket.has_ipv6 is compile-time constant. I wonder how can it help.
@danielnelson do you have has_ipv6 == False on your system?

@webknjaz
Copy link
Member

@asvetlov has_ipv6 sounds like a band-aid. It'll probably help but flags​=​socket​.​AI_ADDRCONFIG is preferable as it'll make getaddrinfo() return only the address families that the network stack on the machine actually supports.

@asvetlov
Copy link
Member

On Linux, flags=0 for getaddrinfo() is considered as (AI_V4MAPPED | AI_ADDRCONFIG) according to docs: https://man7.org/linux/man-pages/man3/getaddrinfo.3.html

I read it as AI_ADDRCONFIG is passed.

Anyway, whatever we decide we need a PR that passes at least both our CI and works well on @danielnelson box.

Does anybody want to create it?

derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020
derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020
derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020
derlih added a commit to derlih/aiohttp that referenced this issue Oct 29, 2020
@danielnelson
Copy link
Contributor Author

danielnelson commented Oct 29, 2020

I believe AI_ADDRCONFIG is not passed by default because Python always passed a hints struct.

If I call s = getaddrinfo(argv[1], argv[2], NULL, &result);only IPv4 addresses are returned. But if I call with a zeroed hints struct I get all addresses families:

memset(&hints, 0, sizeof(struct addrinfo));
s = getaddrinfo(argv[1], argv[2], &hints, &result);

@webknjaz
Copy link
Member

@asvetlov It's what the underlying C API defaults to.

But no, it not the default in the CPython stdlib. Both the source (https://github.com/python/cpython/blob/dff1ad5/Modules/socketmodule.c#L6463) and the docs (https://docs.python.org/3/library/socket.html#socket.getaddrinfo) confirm that flags=0.

So I'm still convinced that setting AI_ADDRCONFIG is the best solution here.

derlih added a commit to derlih/aiohttp that referenced this issue Oct 31, 2020
derlih added a commit to derlih/aiohttp that referenced this issue Oct 31, 2020
@webknjaz webknjaz added regression Something that used to work stopped working "as before" after upgrade reproducer: present This PR or issue contains code, which reproduce the problem described or clearly understandable STR labels Oct 31, 2020
webknjaz pushed a commit that referenced this issue Nov 1, 2020
…k supports (#5176)

* Fix for #5156

* test for #5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for #5156"

This reverts commit 9d81913.

* Revert "Fix for #5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file
github-actions bot pushed a commit that referenced this issue Nov 1, 2020
…k supports (#5176)

* Fix for #5156

* test for #5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for #5156"

This reverts commit 9d81913.

* Revert "Fix for #5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file
github-actions bot pushed a commit that referenced this issue Nov 1, 2020
…k supports (#5176)

* Fix for #5156

* test for #5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for #5156"

This reverts commit 9d81913.

* Revert "Fix for #5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file
webknjaz pushed a commit that referenced this issue Nov 1, 2020
…k supports (#5176) (#5190)

* Fix for #5156

* test for #5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for #5156"

This reverts commit 9d81913.

* Revert "Fix for #5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file

Co-authored-by: Dmitry Erlikh <[email protected]>
webknjaz pushed a commit that referenced this issue Nov 1, 2020
…k supports (#5176) (#5189)

* Fix for #5156

* test for #5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for #5156"

This reverts commit 9d81913.

* Revert "Fix for #5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file

Co-authored-by: Dmitry Erlikh <[email protected]>
commonism pushed a commit to commonism/aiohttp that referenced this issue Apr 27, 2021
…k supports (aio-libs#5176)

* Fix for aio-libs#5156

* test for aio-libs#5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for aio-libs#5156"

This reverts commit 9d81913.

* Revert "Fix for aio-libs#5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file
commonism pushed a commit to commonism/aiohttp that referenced this issue Apr 27, 2021
…k supports (aio-libs#5176)

* Fix for aio-libs#5156

* test for aio-libs#5156

* add changes file

* rearrange if/else

* Revert "rearrange if/else"

This reverts commit a557e4c.

* Revert "test for aio-libs#5156"

This reverts commit 9d81913.

* Revert "Fix for aio-libs#5156"

This reverts commit 48de143.

* Add AI_ADDRCONFIG flag to loop.getaddrinfo

* update changes file
@webknjaz
Copy link
Member

A new issue #5901 just came in today reporting this problem on the recent version of the 3.7 stream (3.7.4.post0). So I'm reopening this for now. I think it may be reasonable to come up with a docker image based on Gentoo with the ipv6 USE-flag disabled globally and enabled just for CPython. We could then use that image to verify the fix and also make a regression test using it.

@webknjaz webknjaz reopened this Jul 19, 2021
@dilyanpalauzov
Copy link

As mentioned at #5901 the 3.8 branch fails on my system. Here is the output of the questions above:

import socket
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x03\x00\x00'))]

>>> socket.has_ipv6 == False
True

>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.12.223', 443)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x03\x00\x00'))]

I have libc 2.33 and kernel 4.14.217

The patch from #5901 is included in the tip of the 3.8 branch, but does not help.

@Hanaasagi
Copy link
Member

Hanaasagi commented Jul 20, 2021

Something that might be helpful. Use a self-compiled CPython with --enable-ipv6 option. And disable disable ipv6 via sysctl -w net.ipv6.conf.all.disable_ipv6=1. Then call socket.getaddrinfo with family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG, will get a socket.gaierror: [Errno -2] Name or service not known.

Python 3.8.5 (default, Jul 20 2021, 04:42:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.has_ipv6
True
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/Python-3.8.5/Lib/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('151.101.108.223', 443))]

Update:
This can reproduce with

  1. Enable ipv6 stack in Linux: sysctl -w net.ipv6.conf.all.disable_ipv6=0
  2. Compile CPython with --disable-ipv6 option
Python 3.8.5 (default, Jul 20 2021, 05:11:52)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.has_ipv6
False
>>> socket.getaddrinfo('www.python.org', 443, type=socket.SOCK_STREAM, family=socket.AF_INET6, flags=socket.AI_ADDRCONFIG)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', (10, b'\x01\xbb\x00\x00\x00\x00*\x04NB\x00\x1a\x00\x00'))]

See the CPython implemention makesockaddr fucntion:

If we don't enable ipv6 when compile CPython, this case is missing.

#ifdef ENABLE_IPV6
    case AF_INET6:
    {
        const struct sockaddr_in6 *a = (const struct sockaddr_in6 *)addr;
        PyObject *addrobj = make_ipv6_addr(a);
        PyObject *ret = NULL;
        if (addrobj) {
            ret = Py_BuildValue("OiII",
                                addrobj,
                                ntohs(a->sin6_port),
                                ntohl(a->sin6_flowinfo),
                                a->sin6_scope_id);
            Py_DECREF(addrobj);
        }
        return ret;
    }

https://github.com/python/cpython/blob/7766b96ab80b04509bbac708ee5ecf3c1c5934fc/Modules/socketmodule.c#L1237-L1252

And it will go to the default case. Just return the sa_family and raw bytes.

    default:
        /* If we don't know the address family, don't raise an
           exception -- return it as an (int, bytes) tuple. */
        return Py_BuildValue("iy#",
                             addr->sa_family,
                             addr->sa_data,
                             sizeof(addr->sa_data));

https://github.com/python/cpython/blob/7766b96ab80b04509bbac708ee5ecf3c1c5934fc/Modules/socketmodule.c#L1436-L1445

aiohttp will get a integer 10, but it doesn't mean a AddressFamily.AF_INET6. Unfortunately, it's a IntEnum, so we will mistake this for a ipv6 address.

>>> socket.AddressFamily.AF_INET6 == 10
True

My suggestion is to use socket.has_ipv6 for pre-check.

@dilyanpalauzov
Copy link

I have compiled python with ./configure --enable-loadable-sqlite-extensions --disable-ipv6 --with-system-expat --with-system-libmpdec --enable-shared.

@Hanaasagi
Copy link
Member

Link a fix: #5906.

@asvetlov
Copy link
Member

Fixed

WhyNotHugo pushed a commit to pimutils/vdirsyncer that referenced this issue Nov 2, 2021
vadim2404 added a commit to neondatabase/neon that referenced this issue Jan 31, 2023
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.7.0 to
3.7.4.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/aio-libs/aiohttp/releases">aiohttp's
releases</a>.</em></p>
<blockquote>
<h2>aiohttp 3.7.3 release</h2>
<h2>Features</h2>
<ul>
<li>Use Brotli instead of brotlipy
<code>[#3803](aio-libs/aiohttp#3803)
&lt;https://github.com/aio-libs/aiohttp/issues/3803&gt;</code>_</li>
<li>Made exceptions pickleable. Also changed the repr of some
exceptions.
<code>[#4077](aio-libs/aiohttp#4077)
&lt;https://github.com/aio-libs/aiohttp/issues/4077&gt;</code>_</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Raise a ClientResponseError instead of an AssertionError for a blank
HTTP Reason Phrase.
<code>[#3532](aio-libs/aiohttp#3532)
&lt;https://github.com/aio-libs/aiohttp/issues/3532&gt;</code>_</li>
<li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior
for patch without slash.
<code>[#3669](aio-libs/aiohttp#3669)
&lt;https://github.com/aio-libs/aiohttp/issues/3669&gt;</code>_</li>
<li>Fix overshadowing of overlapped sub-applications prefixes.
<code>[#3701](aio-libs/aiohttp#3701)
&lt;https://github.com/aio-libs/aiohttp/issues/3701&gt;</code>_</li>
<li>Make <code>BaseConnector.close()</code> a coroutine and wait until
the client closes all connections. Drop deprecated &quot;with
Connector():&quot; syntax.
<code>[#3736](aio-libs/aiohttp#3736)
&lt;https://github.com/aio-libs/aiohttp/issues/3736&gt;</code>_</li>
<li>Reset the <code>sock_read</code> timeout each time data is received
for a <code>aiohttp.client</code> response.
<code>[#3808](aio-libs/aiohttp#3808)
&lt;https://github.com/aio-libs/aiohttp/issues/3808&gt;</code>_</li>
<li>Fixed type annotation for add_view method of UrlDispatcher to accept
any subclass of View
<code>[#3880](aio-libs/aiohttp#3880)
&lt;https://github.com/aio-libs/aiohttp/issues/3880&gt;</code>_</li>
<li>Fixed querying the address families from DNS that the current host
supports.
<code>[#5156](aio-libs/aiohttp#5156)
&lt;https://github.com/aio-libs/aiohttp/issues/5156&gt;</code>_</li>
<li>Change return type of MultipartReader.<strong>aiter</strong>() and
BodyPartReader.<strong>aiter</strong>() to AsyncIterator.
<code>[#5163](aio-libs/aiohttp#5163)
&lt;https://github.com/aio-libs/aiohttp/issues/5163&gt;</code>_</li>
<li>Provide x86 Windows wheels.
<code>[#5230](aio-libs/aiohttp#5230)
&lt;https://github.com/aio-libs/aiohttp/issues/5230&gt;</code>_</li>
</ul>
<h2>Improved Documentation</h2>
<ul>
<li>Add documentation for <code>aiohttp.web.FileResponse</code>.
<code>[#3958](aio-libs/aiohttp#3958)
&lt;https://github.com/aio-libs/aiohttp/issues/3958&gt;</code>_</li>
<li>Removed deprecation warning in tracing example docs
<code>[#3964](aio-libs/aiohttp#3964)
&lt;https://github.com/aio-libs/aiohttp/issues/3964&gt;</code>_</li>
<li>Fixed wrong &quot;Usage&quot; docstring of
<code>aiohttp.client.request</code>.
<code>[#4603](aio-libs/aiohttp#4603)
&lt;https://github.com/aio-libs/aiohttp/issues/4603&gt;</code>_</li>
<li>Add aiohttp-pydantic to third party libraries
<code>[#5228](aio-libs/aiohttp#5228)
&lt;https://github.com/aio-libs/aiohttp/issues/5228&gt;</code>_</li>
</ul>
<h2>Misc</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst">aiohttp's
changelog</a>.</em></p>
<blockquote>
<h1>3.7.4 (2021-02-25)</h1>
<h2>Bugfixes</h2>
<ul>
<li>
<p><strong>(SECURITY BUG)</strong> Started preventing open redirects in
the
<code>aiohttp.web.normalize_path_middleware</code> middleware. For
more details, see
<a
href="https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg">https://github.com/aio-libs/aiohttp/security/advisories/GHSA-v6wp-4m6f-gcjg</a>.</p>
<p>Thanks to <code>Beast Glatisant
&lt;https://github.com/g147&gt;</code>__ for
finding the first instance of this issue and <code>Jelmer Vernooij
&lt;https://jelmer.uk/&gt;</code>__ for reporting and tracking it down
in aiohttp.
<code>[#5497](aio-libs/aiohttp#5497)
&lt;https://github.com/aio-libs/aiohttp/issues/5497&gt;</code>_</p>
</li>
<li>
<p>Fix interpretation difference of the pure-Python and the Cython-based
HTTP parsers construct a <code>yarl.URL</code> object for HTTP
request-target.</p>
<p>Before this fix, the Python parser would turn the URI's absolute-path
for <code>//some-path</code> into <code>/</code> while the Cython code
preserved it as
<code>//some-path</code>. Now, both do the latter.
<code>[#5498](aio-libs/aiohttp#5498)
&lt;https://github.com/aio-libs/aiohttp/issues/5498&gt;</code>_</p>
</li>
</ul>
<hr />
<h1>3.7.3 (2020-11-18)</h1>
<h2>Features</h2>
<ul>
<li>Use Brotli instead of brotlipy
<code>[#3803](aio-libs/aiohttp#3803)
&lt;https://github.com/aio-libs/aiohttp/issues/3803&gt;</code>_</li>
<li>Made exceptions pickleable. Also changed the repr of some
exceptions.
<code>[#4077](aio-libs/aiohttp#4077)
&lt;https://github.com/aio-libs/aiohttp/issues/4077&gt;</code>_</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>Raise a ClientResponseError instead of an AssertionError for a blank
HTTP Reason Phrase.
<code>[#3532](aio-libs/aiohttp#3532)
&lt;https://github.com/aio-libs/aiohttp/issues/3532&gt;</code>_</li>
<li>Fix <code>web_middlewares.normalize_path_middleware</code> behavior
for patch without slash.
<code>[#3669](aio-libs/aiohttp#3669)
&lt;https://github.com/aio-libs/aiohttp/issues/3669&gt;</code>_</li>
<li>Fix overshadowing of overlapped sub-applications prefixes.
<code>[#3701](aio-libs/aiohttp#3701)
&lt;https://github.com/aio-libs/aiohttp/issues/3701&gt;</code>_</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/0a26acc1de9e1b0244456b7881ec16ba8bb64fc3"><code>0a26acc</code></a>
Bump aiohttp to v3.7.4 for a security release</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/021c416c18392a111225bc7326063dc4a99a5138"><code>021c416</code></a>
Merge branch 'GHSA-v6wp-4m6f-gcjg' into master</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/4ed7c25b537f71c6245bb74d6b20e5867db243ab"><code>4ed7c25</code></a>
Bump chardet from 3.0.4 to 4.0.0 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5333">#5333</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/b61f0fdffc887df24244ba7bdfe8567c580240ff"><code>b61f0fd</code></a>
Fix how pure-Python HTTP parser interprets <code>//</code></li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/5c1efbc32c46820250bd25440bb7ea96cb05abe9"><code>5c1efbc</code></a>
Bump pre-commit from 2.9.2 to 2.9.3 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5322">#5322</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/007507580137efcc0a20391a0792f39b337d9c1a"><code>0075075</code></a>
Bump pygments from 2.7.2 to 2.7.3 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5318">#5318</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/5085173d947e6cc01b6daf1aa48fe7698834c569"><code>5085173</code></a>
Bump multidict from 5.0.2 to 5.1.0 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5308">#5308</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/5d1a75e68d278c641c90021409f4eb5de1810e5e"><code>5d1a75e</code></a>
Bump pre-commit from 2.9.0 to 2.9.2 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5290">#5290</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/6724d0e7a944fd7e3a710dc292d785fa8fe424fd"><code>6724d0e</code></a>
Bump pre-commit from 2.8.2 to 2.9.0 (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5273">#5273</a>)</li>
<li><a
href="https://github.com/aio-libs/aiohttp/commit/c688451ce31b914c71b11d2ac6c326b0c87e6d1f"><code>c688451</code></a>
Removed duplicate timeout parameter in ClientSession reference docs. (<a
href="https://github-redirect.dependabot.com/aio-libs/aiohttp/issues/5262">#5262</a>)
...</li>
<li>Additional commits viewable in <a
href="https://github.com/aio-libs/aiohttp/compare/v3.7.0...v3.7.4">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=aiohttp&package-manager=pip&previous-version=3.7.0&new-version=3.7.4)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the
default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as
the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as
the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/neondatabase/neon/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Vadim Kharitonov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug need pull request regression Something that used to work stopped working "as before" after upgrade reproducer: present This PR or issue contains code, which reproduce the problem described or clearly understandable STR
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants