urllib.request.getproxies() misparses Windows registry proxy settings

[benrg]

BPO	42627
Nosy	@pfmoore, @tjguk, @zware, @zooba, @corona10, @chrisxiao, @NateScarlet, @CrazyBoyFeng, @keuin
PRs	bpo-42627: Fix wrong parsing of Windows registry proxy settings #26307

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-12.20:27:52.875>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = 'urllib.request.getproxies() misparses Windows registry proxy settings'
updated_at = <Date 2021-11-20.14:38:47.145>
user = 'https://bugs.python.org/benrg'

bugs.python.org fields:

activity = <Date 2021-11-20.14:38:47.145>
actor = 'CrazyBoyFeng'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2020-12-12.20:27:52.875>
creator = 'benrg'
dependencies = []
files = []
hgrepos = []
issue_num = 42627
keywords = ['patch']
message_count = 10.0
messages = ['382921', '387468', '388149', '388152', '393927', '394668', '395276', '401215', '401995', '406658']
nosy_count = 11.0
nosy_names = ['paul.moore', 'tim.golden', 'benrg', 'zach.ware', 'steve.dower', 'corona10', 'chrisxiao', 'kotori', 'NateScarlet', 'CrazyBoyFeng', 'keuin']
pr_nums = ['26307']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42627'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[benrg]

If HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyServer contains the string http=host:123;https=host:456;ftp=host:789, then getproxies_registry() should return

{'http': 'http://host:123', 'https': 'http://host:456', 'ftp': 'http://host:789'}

for consistency with WinInet and Chromium, but it actually returns

{'http': 'http://host:123', 'https': 'https://host:456', 'ftp': 'ftp://host:789'}

This bug has existed for a very long time (since Python 2.0.1 if not earlier), but it was exposed recently when urllib3 added support for HTTPS-in-HTTPS proxies in version 1.26. Before that, an https prefix on the HTTPS proxy url was silently treated as http, accidentally resulting in the correct behavior.

There are additional bugs in the treatment of single-proxy strings (the case when the string contains no = character).

The Chromium code for parsing the ProxyServer string can be found here: https://source.chromium.org/chromium/chromium/src/+/refs/tags/89.0.4353.1:net/proxy_resolution/proxy_config.cc;l=86

Below is my attempt at modifying the code from getproxies_registry to approximately match Chromium's behavior. I could turn this into a patch, but I'd like feedback on the corner cases first.

    if '=' not in proxyServer and ';' not in proxyServer:
        # Use one setting for all protocols.
        # Chromium treats this as a separate category, and some software
        # uses the ALL_PROXY environment variable for a similar purpose,
        # so arguably this should be 'all={}'.format(proxyServer),
        # but this is more backward compatible.
        proxyServer = 'http={0};https={0};ftp={0}'.format(proxyServer)

    for p in proxyServer.split(';'):
        # Chromium and WinInet are inconsistent in their treatment of
        # invalid strings with the wrong number of = characters. It
        # probably doesn't matter.
        protocol, addresses = p.split('=', 1)
        protocol = protocol.strip()

        # Chromium supports more than one proxy per protocol. I don't
        # know how many clients support the same, but handling it is at
        # least no worse than leaving the commas uninterpreted.
        for address in addresses.split(','):
            if protocol in {'http', 'https', 'ftp', 'socks'}:
                # See if address has a type:// prefix
                if not re.match('(?:[^/:]+)://', address):
                    if protocol == 'socks':
                        # Chromium notes that the correct protocol here
                        # is SOCKS4, but "socks://" is interpreted
                        # as SOCKS5 elsewhere. I don't know whether
                        # prepending socks4:// here would break code.
                        address = 'socks://' + address
                    else:
                        address = 'http://' + address

            # A string like 'http=foo;http=bar' will produce a
            # comma-separated list, while previously 'bar' would
            # override 'foo'. That could potentially break something.
            if protocol not in proxies:
                proxies[protocol] = address
            else:
                proxies[protocol] += ',' + address

[kotori]

I came across this issue as well.
I checked Microsoft documentations and it seems InternetGetProxyInfo in WinInet is deprecated, while WinHttpGetIEProxyConfigForCurrentUser in WinHTTP will return the exact same string as what it stored registery.

https://docs.microsoft.com/en-US/troubleshoot/windows-client/networking/configure-client-proxy-server-settings-by-registry-file
Also from this documentation, a proxy server could have "http://" prefix, so I guess it could also support "https://" prefix if a user set a https proxy.

[CrazyBoyFeng]

We know Windows reslove:
http=host:port;https=host:port;ftp=host:port;socks=host:port
as:
http=http://host:port;https=http://host:port;ftp=http://host:port;socks=socks://host:port
means:
Using HTTP type proxy for HTTP, HTTPS and FTP requests, but Socks4/4a type proxy for the other TCP requests.
We notice that socks are different from the others.

Now I want to know what if Windows slove this:
http=socks://host:port;ftp=https://host:port;socks=https://host:port
Does it mean using Socks4/4a type proxy for HTTP requests, HTTPS type proxy for FTP requests, and HTTP type proxy for the other TCP requests?
Or just invalid settings?

[CrazyBoyFeng]

I make some black box tests with the HTTPS proxy.
Set system proxy http=https://host:port and start a HTTPS proxy, then IE, Edge (Chromium) and benrg's code (using urllib3) work fine while fetching http://website. Then I shutdown the HTTPS proxy and start a HTTP proxy on the same port, I get SSLError. That's it.

benrg should add your patch as soon as possible.
Too few people actually use HTTPS proxy. Even if there is a bug with HTTPS proxy, it will not be too late to fix it when it is discovered.

[chrisxiao]

I found this problem too.
expecting for fixing

[CrazyBoyFeng]

I removed the multi-proxies-per-protocol support from the PR I submitted. The reason is that:

1. This code reads the proxy settings from the Windows registry. Multi-proxies-per-protocol cannot be resolved by Windows system.
2. Using a comma-separated string for the multi-proxies-per-protocol support, is not backward compatible. Not only Windows, but Linux is also one proxy per protocol.

If you want to implement the multi-proxies-per-protocol support, then I think you should open another issue and make changes to all OS platforms.

[CrazyBoyFeng]

We should have no problem with how to parse HTTP proxy and HTTPS proxy. But I recently discovered an additional problem when using requests, that is, how to parse the SOCKS proxy correctly.

I investigated the parsing of some commonly used software:
curl will try to read the all_proxy variable first, if not, then read the two system environment variables http_proxy and https_proxy.
Only these two environment variables related to proxy exist in most Linux distributions.
The following formats are valid for curl:

export http_proxy=socks4h://127.0.0.1:5050
export https_proxy=socks5://1.1.1.1:4040

The h in socks4h means doing DNS query on the proxy server.

requests also follows this format.
If you want to use the socks proxy in requests[socks], you need to specify the proxies parameter:

proxies['http']='socks4://127.0.0.1:6060'
proxies['https']='socks5h://127.0.0.1:7070'

Since it is customary to resolve SOCKS proxies in this way, I think CPython can consider doing the same with Windows registry.

In previous versions, CPython parsed the SOCKS proxy in the Windows registry socks=localhost:8080 as:

proxies['socks']='socks://localhost:8080'

I think it can be changed to

proxies['http']=proxies['http'] or'socks4://localhost:8080'
proxies['https']=proxies['http'] or'socks4://localhost:8080'
proxies['socks']='socks://localhost:8080'

The use of or is to prevent the existing HTTP and HTTPS proxy settings from being overwritten.
I tried it on my Win10 PC. When I set up HTTP and SOCKS proxies at the same time, IE and Edge will use HTTP proxy.
The reason using socks4 instead of socks4h or socks5 is that, when I change system proxy setting to a remote SOCKS proxy, I found that IE and Edge do not query DNS from the proxy but from local DNS polluted by the ISP. And if the running socks proxy version is SOCKS5, it will output errors of unsupported socks version 0x4. So Windows only supports the SOCKS proxy socks4://.
It's a pity that we don't know the code of Windows, and I can't prove it with the codes.
proxies['socks'] is reserved for backward compatibility.

If you have any comments or suggestions, please let me know.

[keuin]

The fix is available as a pull request on GitHub for months (#26307). However, it seems that this pull request needs an approval from one maintainer before running any test. Could anyone help this out?

[zooba]

I think the PR is basically ready, unfortunately it's stuck behind a CI issue we only just fixed, and needs to merge from main before it'll clear.

Posting here once CI is green will get attention faster than on GitHub.

[CrazyBoyFeng]

Sorry I didn't see this comment before.

Can it be merged now?

[SekiBetu]

I think the PR is basically ready, unfortunately it's stuck behind a CI issue we only just fixed, and needs to merge from main before it'll clear.

Posting here once CI is green will get attention faster than on GitHub.

any updates?

[zooba]

I retriggered the CI to see if it passes now.

[TTimo]

Hello,

I am curious if this will also address https://stackoverflow.com/questions/65931275/python-requests-module-does-it-use-system-level-on-windows-proxy-settings or if it's yet another problem?

[zooba]

@TTimo From a quick glance, I think it'll deal with part of the second half of that question. It should be able to pick up manually customised proxies, but I don't think it'll get automatically configured ones. And anything that's bypassing urllib here won't get any benefit, naturally.