Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot install packages with pip - timeout error #2409

Closed
1 task done
Pospes opened this issue Nov 17, 2022 · 1 comment
Closed
1 task done

Cannot install packages with pip - timeout error #2409

Pospes opened this issue Nov 17, 2022 · 1 comment
Labels
network Issues related to our CDN, users having problems connecting to PyPI

Comments

@Pospes
Copy link

Pospes commented Nov 17, 2022 •
edited
Loading

My Platform

OS is WSL: Ubuntu-22.04
(installed on W10)

Pip version: 22.3.1 (on Python 10)
CURL version: 7.81.0 (x86_64-pc-linux-gnu)

  • Doesn´t work on Ubuntu-20.04 too
  • Connecting to pypi.org via browser is possible,
  • Pip probably timeouts on TLS handshake (see the curl command)
  • HTTPS CURL from WSL to other websites sometimes work, sometimes doesn´t. For ex. curl https://windows.com does not work, curl https://seznam.cz works. ((edited after issue submission))
  • HTTPS CURL from Windows CMD to pypi.org works fine
  • also works on an Ubuntu WM on the same PC
  • adding --default-timeout=100 to the pip command doesn´t help
  • adding --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org doesn´t help
  • turning off firewall and Windows Defender doesn´t help
  • changing network doesn´t help
  • reinstalling WSL and trying pip from fresh installation doesn´t help

Fastly Debug

$ sudo python3 -m pip install -Uvvv pip
Using pip 22.3.1 from /usr/local/lib/python3.10/dist-packages/pip-22.3.1-py3.10.egg/pip (python 3.10)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-build-tracker-j40a1mry
Initialized build tracking at /tmp/pip-build-tracker-j40a1mry
Created build tracker: /tmp/pip-build-tracker-j40a1mry
Entered build tracker: /tmp/pip-build-tracker-j40a1mry
Created temporary directory: /tmp/pip-install-7ztvrwuh
Created temporary directory: /tmp/pip-ephem-wheel-cache-77sav5ko
Requirement already satisfied: pip in /usr/local/lib/python3.10/dist-packages/pip-22.3.1-py3.10.egg (22.3.1)
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
Incremented Retry for (url='/simple/pip/'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pip/
Starting new HTTPS connection (2): pypi.org:443
Incremented Retry for (url='/simple/pip/'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pip/
Starting new HTTPS connection (3): pypi.org:443
Incremented Retry for (url='/simple/pip/'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pip/
Starting new HTTPS connection (4): pypi.org:443
Incremented Retry for (url='/simple/pip/'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pip/
Starting new HTTPS connection (5): pypi.org:443
Incremented Retry for (url='/simple/pip/'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/pip/
Starting new HTTPS connection (6): pypi.org:443
Could not fetch URL https://pypi.org/simple/pip/: connection error: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")) - skipping
Skipping link: not a file: https://pypi.org/simple/pip/
Given no hashes to check 0 links for project 'pip': discarding no candidates
Created temporary directory: /tmp/pip-unpack-qeoxwuvj
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
Could not fetch URL https://pypi.org/simple/pip/: connection error: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=5)")) - skipping
Skipping link: not a file: https://pypi.org/simple/pip/
Given no hashes to check 0 links for project 'pip': discarding no candidates
WARNING: There was an error checking the latest version of pip.
See below for error
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/pip-22.3.1-py3.10.egg/pip/_internal/self_outdated_check.py", line 227, in pip_self_version_check
    upgrade_prompt = _self_version_check_logic(
  File "/usr/local/lib/python3.10/dist-packages/pip-22.3.1-py3.10.egg/pip/_internal/self_outdated_check.py", line 196, in _self_version_check_logic
    remote_version = parse_version(remote_version_str)
  File "/usr/local/lib/python3.10/dist-packages/pip-22.3.1-py3.10.egg/pip/_vendor/packaging/version.py", line 49, in parse
    return Version(version)
  File "/usr/local/lib/python3.10/dist-packages/pip-22.3.1-py3.10.egg/pip/_vendor/packaging/version.py", line 264, in __init__
    match = self._regex.search(version)
TypeError: expected string or bytes-like object
Removed build tracker: '/tmp/pip-build-tracker-j40a1mry'

DNS Resolution

$ dig pypi.org A

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> pypi.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31254
;; flags: qr rd ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;pypi.org.                      IN      A

;; ANSWER SECTION:
pypi.org.               0       IN      A       151.101.192.223
pypi.org.               0       IN      A       151.101.64.223
pypi.org.               0       IN      A       151.101.128.223
pypi.org.               0       IN      A       151.101.0.223

;; Query time: 0 msec
;; SERVER: 172.28.128.1#53(172.28.128.1) (UDP)
;; WHEN: Thu Nov 17 14:18:26 CET 2022
;; MSG SIZE  rcvd: 98

$ dig pypi.org AAAA

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> pypi.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27382
;; flags: qr rd ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;pypi.org.                      IN      AAAA

;; ANSWER SECTION:
pypi.org.               0       IN      AAAA    2a04:4e42::223
pypi.org.               0       IN      AAAA    2a04:4e42:400::223
pypi.org.               0       IN      AAAA    2a04:4e42:200::223
pypi.org.               0       IN      AAAA    2a04:4e42:600::223

;; Query time: 0 msec
;; SERVER: 172.28.128.1#53(172.28.128.1) (UDP)
;; WHEN: Thu Nov 17 14:19:13 CET 2022
;; MSG SIZE  rcvd: 146

$ dig files.pythonhosted.org A

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> files.pythonhosted.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54578
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;files.pythonhosted.org.                IN      A

;; ANSWER SECTION:
files.pythonhosted.org. 0       IN      CNAME   dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 0 IN A       146.75.117.63

;; Query time: 20 msec
;; SERVER: 172.28.128.1#53(172.28.128.1) (UDP)
;; WHEN: Thu Nov 17 14:19:27 CET 2022
;; MSG SIZE  rcvd: 158

$ dig files.pythonhosted.org AAAA

; <<>> DiG 9.18.1-1ubuntu1.2-Ubuntu <<>> files.pythonhosted.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46839
;; flags: qr rd ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;files.pythonhosted.org.                IN      AAAA

;; ANSWER SECTION:
files.pythonhosted.org. 0       IN      CNAME   dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 0 IN AAAA    2a04:4e42:8d::319

;; Query time: 20 msec
;; SERVER: 172.28.128.1#53(172.28.128.1) (UDP)
;; WHEN: Thu Nov 17 14:19:43 CET 2022
;; MSG SIZE  rcvd: 170

Traceroutes / IPv4

$ traceroute pypi.org
traceroute to pypi.org (151.101.192.223), 30 hops max, 60 byte packets
 1  MSI.mshome.net (172.28.128.1)  0.471 ms  0.375 ms  0.367 ms
 2  o2smartbox2.home (10.0.1.138)  2.069 ms  2.438 ms  2.259 ms
 3  80.70.60.254 (80.70.60.254)  6.690 ms  8.468 ms  8.434 ms
 4  * * *
 5  90-183-197-114.rcq.o2.cz (90.183.197.114)  11.318 ms  11.480 ms  11.471 ms
 6  194.228.115.116 (194.228.115.116)  10.419 ms  11.515 ms  10.349 ms
 7  * * *
 8  90.182.80.193 (90.182.80.193)  13.224 ms  9.436 ms  9.576 ms
 9  194.228.115.35 (194.228.115.35)  10.111 ms  10.106 ms 194.228.115.33 (194.228.115.33)  10.922 ms
10  194.228.115.32 (194.228.115.32)  10.989 ms 194.228.115.34 (194.228.115.34)  11.931 ms 194.228.115.32 (194.228.115.32)  10.978 ms
11  prag-b4-link.ip.twelve99.net (62.115.58.198)  11.925 ms * *
12  * * ffm-bb2-link.ip.twelve99.net (62.115.124.28)  17.059 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
$ traceroute files.pythonhosted.org
traceroute to files.pythonhosted.org (146.75.117.63), 30 hops max, 60 byte packets
 1  MSI.mshome.net (172.28.128.1)  0.429 ms  0.394 ms  0.420 ms
 2  o2smartbox2.home (10.0.1.138)  2.071 ms  2.549 ms *
 3  * * 80.70.60.254 (80.70.60.254)  6.977 ms
 4  * * *
 5  90-183-197-114.rcq.o2.cz (90.183.197.114)  10.032 ms  9.929 ms *
 6  194.228.115.116 (194.228.115.116)  9.634 ms 194.228.115.115 (194.228.115.115)  8.716 ms  9.848 ms
 7  * * *
 8  90.182.80.225 (90.182.80.225)  9.139 ms  9.396 ms  8.911 ms
 9  194.228.115.35 (194.228.115.35)  10.200 ms 194.228.115.33 (194.228.115.33)  9.587 ms  9.958 ms
10  194.228.115.34 (194.228.115.34)  10.255 ms 194.228.115.32 (194.228.115.32)  9.750 ms 194.228.115.34 (194.228.115.34)  10.441 ms
11  prag-b4-link.ip.twelve99.net (62.115.58.198)  10.504 ms *  10.119 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Traceroutes / IPv6 (If available)

$ traceroute6 pypi.org
traceroute to pypi.org (2a04:4e42::223), 30 hops max, 80 byte packets
connect: Network is unreachable
$ traceroute6 files.pythonhosted.org
traceroute to files.pythonhosted.org (2a04:4e42:8d::319), 30 hops max, 80 byte packets
connect: Network is unreachable

HTTPS Requests / IPv4

$ curl -vvv -I --ipv4 https://pypi.org/pypi/pip/json
*   Trying 151.101.192.223:443...
* Connected to pypi.org (151.101.192.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL connection timeout
* Closing connection 0
curl: (28) SSL connection timeout
$ curl -vvv -I --ipv4 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz
*   Trying 146.75.117.63:443...
* Connected to files.pythonhosted.org (146.75.117.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to files.pythonhosted.org:443
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to files.pythonhosted.org:443

HTTPS Requests / IPv6 (If available)

$ curl -vvv -I --ipv6 https://pypi.org/pypi/pip/json
*   Trying 2a04:4e42::223:443...
* Immediate connect fail for 2a04:4e42::223: Network is unreachable
*   Trying 2a04:4e42:400::223:443...
* Immediate connect fail for 2a04:4e42:400::223: Network is unreachable
*   Trying 2a04:4e42:200::223:443...
* Immediate connect fail for 2a04:4e42:200::223: Network is unreachable
*   Trying 2a04:4e42:600::223:443...
* Immediate connect fail for 2a04:4e42:600::223: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server
$ curl -vvv -I --ipv6 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz
*   Trying 2a04:4e42:8d::319:443...
* Immediate connect fail for 2a04:4e42:8d::319: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server

TLS Debug / IPv4

$ echo -n | openssl s_client -4 -connect pypi.org:443
CONNECTED(00000003)

and stays open afterwards...

$ echo -n | openssl s_client -4 -connect files.pythonhosted.org:443
CONNECTED(00000003)

and stays open afterwards...

TLS Debug / IPv6 (If available)

$ echo -n | openssl s_client -6 -connect pypi.org:443
40073E1EC37F0000:error:80000065:system library:BIO_connect:Network is unreachable:../crypto/bio/bio_sock2.c:125:calling connect()
40073E1EC37F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
40073E1EC37F0000:error:80000065:system library:BIO_connect:Network is unreachable:../crypto/bio/bio_sock2.c:125:calling connect()
40073E1EC37F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
40073E1EC37F0000:error:80000065:system library:BIO_connect:Network is unreachable:../crypto/bio/bio_sock2.c:125:calling connect()
40073E1EC37F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
40073E1EC37F0000:error:80000065:system library:BIO_connect:Network is unreachable:../crypto/bio/bio_sock2.c:125:calling connect()
40073E1EC37F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
connect:errno=101
$ echo -n | openssl s_client -6 -connect files.pythonhosted.org:443
406769D4717F0000:error:80000065:system library:BIO_connect:Network is unreachable:../crypto/bio/bio_sock2.c:125:calling connect()
406769D4717F0000:error:10000067:BIO routines:BIO_connect:connect error:../crypto/bio/bio_sock2.c:127:
connect:errno=101

Code of Conduct

  • I agree to follow the PSF Code of Conduct
@Pospes Pospes added the network Issues related to our CDN, users having problems connecting to PyPI label Nov 17, 2022
@Pospes
Copy link
Author

Pospes commented Nov 17, 2022

[SOLVED]

For any future reference, the trouble in reaching https://pypi.org with pip and curl was the MTU size. Some servers were still reachable via HTTPS probably because they use the newer TCPv1.3. However, many other with just TCPv1.2 (pypi.org as well) were not reachable as the TLS handshake timeouted. Pip now works normally.

Changing the MTU size:

  1. PowerShell: netsh interface ipv4 show subinterface
  2. WSL2: ip link list
  3. WSL2: sudo ip link set dev eth0 mtu 1400

The solution can be found here:
microsoft/WSL#4698
https://sbulav.github.io/wsl/wsl2-vm-tls-handshake-failed/

@Pospes Pospes closed this as completed Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
network Issues related to our CDN, users having problems connecting to PyPI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Pospes