macOS client doesn't work through SOCKS5 Proxy

[Charles-CongFu]

Describe the bug

I'm using dropbox in China, where the service is blocked by national firewall. Dropbox client can work with self created SOCKS5 proxy on rented server. However Maestral client can't make connection the same way, even if I enable SOCKS5-Proxy in WLAN settings or route the client through Proxifier.

With VPN enabled

VPN disabled, only SOCKS5 proxy.
With conventional VPN Maestral can work fine though, which is a bit strange.

To Reproduce

Expected behaviour

Is there any workaround? I tried to convert socks5 to https and route Maestral through https proxy in Proxifier. Sadly it still won't work.
System:

• Maestral version: 1.4.8
• Python version: 3.9.6
• OS: macOS 11.5.2
• Desktop environment:
• PyQt version (for Linux GUI):

Additional context

[Charles-CongFu]

I found that I forgot to mosaic email address. And it seems impossible to delete the issue. So I guess I'll leave it open then.

[samschott]

You can edit your post to delete the screenshot!

[samschott]

Alternatively, I can delete the issue completely, if you prefer.

[Charles-CongFu]

Thanks for the advice. Screenshots have been replaced.

[samschott]

Regarding the actual issue, I'm not sure how SOCKS5 proxies work exactly. Since #434, proxy settings are retrieved from environment variables and handled by the requests module. From the requests documentation here:

Requests relies on the proxy configuration defined by standard environment variables http_proxy, https_proxy, no_proxy and curl_ca_bundle. Uppercase variants of these variables are also supported.

Could you check if the environment variable https_proxy is set in your case? I am not sure if macOS will set it for you when you configure a proxy in System Preferences. If it does not, I'll need to look into alternative methods of proxy discovery.

[Charles-CongFu]

It sounds kinda dumb but I really don't know how to check it(noob in coding). But I do find out that me using privoxy to forward HTTP to SOCKS5 proxy is only useful for other devices that connect to the same network as my Mac, but not for Mac apps in this every same device. Tried localhost:8118(8118 is default port selection in privoxy), 0.0.0.0:8118, 127.0.0.1:8118, and local IP of my Mac:8118. None of them worked.

I guess I'll have to dig deeper into Privoxy manuals, or resort to other apps for HTTP proxy in this local machine. Maestral sync should work afterwards.

[samschott]

It sounds kinda dumb but I really don't know how to check it(noob in coding).

Not dumb at all! On macOS, you can check this by opening the Terminal app and typing the command (and hitting enter):

echo $https_proxy

I suspect that this will just return an empty line, which means that the proxy configuration is not stored in the environment variable but somewhere else.

If it's not too much trouble, you can also try the following: Again in the Terminal app, type python3 to start Python. Then run the following two commands:

import urllib.request
print(urllib.request.getproxies())

This time, I hope that the correct proxy configuration should be shown, namely:

{'https': '127.0.0.1:8118'}

If yes, I'll know how to fix it...

[Charles-CongFu]

Thanks for detailed instructions. I do get an empty line after the first command. And in Python3, what I have after those 2 commands is 127.0.0.1:1083, which is the original SOCKS5 proxy that I deploy. Unfortunately I didn't take screenshot, cuz I thought what you meant by "correct proxy configuration"was to type echo $https_proxy again, so I just close the terminal and open a new one. It should be: {'socks5': '127.0.0.1:1083'} if I remember it correctly.

Edit:
Sry I forgot to mention that I use "alias socks5='export all_proxy=socks5://127.0.0.1:1083'" in .zshrc file for homebrew to upgrade faster through SOCKS5 proxy in Terminal. That's why I got that result. With it disabled, what I get is {}.

But I can do a workaround to get similar result that's expected from you.
Since I've already used privoxy, I can set another alias to let terminal use HTTP proxy.

alias http='export all_proxy=http://0.0.0.0:8118'
alias unproxy='unset all_proxy'

And it seems to be working based on the following screenshot:

With HTTP proxy enabled the IP address is identified to the server I rented; After disabling it, my local IP address is shown.

[samschott]

Ok, thanks for the detailed infos.

It appears that the situation is more complicated than I thought. Python's request library will retrieve the proxy settings from the all_proxy=socks5://127.0.0.1:1083 environment variable which you have already defined for homebrew. If and only if it cannot find any proxy settings in environment variables, it will query proxy settings directly from System Preferences. I suspect that this will be the case for Maestral, since it is a standalone program which is not started from the Terminal. I'm therefore a bit confused why it doesn't just work out of the box.

How and where did you configure the SOCKS5 proxy? Could you also check the return value from the following Python command:

import urllib.request
print(urllib.request.getproxies_macosx_sysconf())

This will bypass all environment settings and directly get the proxy configuration from System Preferences. What does this return in your case?

[Charles-CongFu]

It seems that proxy configuration is blank from system preferences.

Edit: My SOCKS proxy is implemented on rented server. It's functionalized through a java "file", which is located in download folder.

It can be run with/without GUI. Normally I run "sudo java -jar xxx.jar" in terminal. Unfortunately I don't know which environment it's running in.

[hronro]

The reliability of urllib.request.getproxies() in retrieving proxy information from the System Configuration for macOS and Windows Systems Registry for Windows seems questionable. Below are my settings as displayed in the macOS System Configuration panel:

This is what I did using urllib.request.getproxies():

I also found some similar issues on Windows:

https://stackoverflow.com/questions/65931275/python-requests-module-does-it-use-system-level-on-windows-proxy-settings
python/cpython#86793

A workaround on macOS is to use scutil --proxy to get the proxy information:

[samschott]

Indeed, it looks like urllib.request.getproxies() ignores SOCKS proxies on macOS. See here for how proxies are read from the system configuration:

https://github.com/python/cpython/blob/f5c38382f9c40f0017cef086896a8160e313ac9e/Modules/_scproxy.c#L234

In particular, kSCPropNetProxiesSOCKSEnable and kSCPropNetProxiesSOCKSProxy are never checked. That being said, requests also requires PySocks to be installed to support SOCKS proxies. This is not currently bundled in the Maestral macOS app either (will be easy to include though).

I'm a bit reluctant to introduce custom support for SOCKS proxies in Maestral, IMO this should be handled by using an external library, or ideally by allowing Python's urllib to detect them.

[samschott]

I've filed python/cpython#104180 upstream and will create a PR to fix this in urllib.

[samschott]

The fix from python/cpython#104181 included in Python 3.12 and the latest macOS app bundle v1.9.1 is based on Python 3.12.