Root certificates on Windows 7?

[chupocro]

Hi,

let me first congratulate and thank you for this awesome program!! I've found it when I was looking for a solution on how to continue using Eudora email client even after gmail started forcing OAuth2. There are thousands of people still using Eudora since 1st half of the 90s because no other email client can replace it.

It is true Eudora could still be used by enabling App Password but that option forces enabling 2-Step Verification and 2-Step Verification forces using phone number which is not the case with OAuth2.

I have the very same problem as described in Issue #14 but this time when using Windows 7. The authorisation window works well when running Email OAuth2 Proxy in Windows 10 but I can't make it working with Windows 7 running in Virtualbox.

I have several Windows 7 virtual machines already running for various purposes and that's why I would like to use one of those VMs for running Email OAuth2 Proxy too.

After adding traceback.print_exc() as you suggested in Issue #14, the error is:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

I tried:

pip install python-certifi-win32

and

pip install --upgrade certifi

but still the same. I am sure that problem could be solved but I couldn't find the solution.

BTW, when I tried to install the requirement for the 1st time there was an error:

Ignoring pyobjc-framework-Cocoa: markers 'sys_platform == "darwin"' don't match
your environment
Ignoring pyobjc-framework-SystemConfiguration: markers 'sys_platform == "darwin"
' don't match your environment
Ignoring pyoslog: markers 'sys_platform == "darwin"' don't match your environmen
t
ERROR: Double requirement given: pywebview>=3.5 (from -r requirements.txt (line
14)) (already in pywebview (from -r requirements.txt (line 5)), name='pywebview'
)
WARNING: You are using pip version 19.2.3, however version 22.1.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' comm
and.

and none of the packages were installed. However, everything worked well after upgrading pip.

ps
It would be even more awesome if you could add the POP protocol as you mentioned under "Potential improvements". I tested gmail + OAuth2 + POP using Thunderbird and it works well. There is a problem when switching the existing Eudora account from POP to IMAP - in that case all messages in the inbox are duplicated (the originals already fetched via POP and the the "new" headers fetched via IMAP). The ideal solution would be if it would be still possible to continue using POP.

[simonrob]

Thanks for this report, and the kind words.

I don't have a copy of Windows 7 but tried a virtual machine and Python 3.8.10. I can't replicate the certificate issue, however. To help isolate the problem with your Python installation you could try replacing the proxy's two uses of ssl.create_default_context() with ssl._create_unverified_context(). If the proxy works with this change then this is the same problem as #14, as you suspect, and I'd suggest searching for a way to install updated SSL certificates for Python on Windows 7. (Note that as suggested in the function name the proxy edit will disable certificate verification, which is not a good idea as a long-term solution.)

Thanks also for reporting the double requirement error. This is an issue with older versions of pip that is no-longer the case, but in 0add215 I've updated requirements.txt so that this doesn't happen any more even with these versions.

POP support is relatively easy to add given the existing proxy structure – see the new POP branch. I haven't tested any of this except for the login part, but there's no reason why it shouldn't just work. You can configure POP servers as shown in the examples below. Please let me know how you get on.

[POP-1995]
server_address = outlook.office365.com
server_port = 995

[POP-1996]
server_address = pop.gmail.com
server_port = 995

One thing to note regarding POP is that some providers require it to be enabled explicitly. For example, Gmail responds eyJzdGF0dXMiOiI0MDAiLCJzY2hlbWVzIjoiQmVhcmVyIiwic2NvcGUiOiJodHRwczovL21haWwuZ29vZ2xlLmNvbS8ifQ== which, decoded, is {"status":"400","schemes":"Bearer","scope":"https://mail.google.com/"} when POP is not enabled. It's easy to enable POP in the Gmail settings to resolve this, but just bear in mind that any error message may not actually be accurate or helpful.

[chupocro]

Thank you very much for your reply and especially for the POP branch!! :-) I will test it as soon as I manage to get the main branch working with Windows 7.

I replaced ssl.create_default_context() with ssl._create_unverified_context() and in that case the authorisation window did open without error. I then reverted the code back to the original and was trying various things to get everything working without disabling certificate verification and after a while the authorisation window "suddenly" showed and there wasn't SSL error.

The only changes that I did were:

pip uninstall python-certifi-win32
pip uninstall certifi

and then:

pip install certifi

After that the SSL error disappeared and the authorisation window opened as it should but after I entered the email address there was a note:

Couldn't sign you in
This browser or app may not be secure.
Try using a different browser. If you're already using a supported browser, you can try again to sign in.

And there was a "Try again" button.

Exactly the same happens when using either ssl.create_default_context() or ssl._create_unverified_context().

When the authorisation window opened the email address was not already filled (I am not sure if email address should be filled automatically but it was filled automatically when I was testing authorisation of Thunderbird only without the proxy).

The output in the terminal when the authorisation window opened was:

Authorisation request received for [email protected] (interactive mode)
[pywebview] MSHTML is deprecated. See https://pywebview.flowrl.com/guide/renderer.html#web-engine on details how to use Edge Chromium

Is it possible the "browser may not be secure" error is because the browser window is using some (outdated) Windows 7 components which should be updated? The Windows 7 ISO I am using with Virtualbox is from year 2016.

[simonrob]

Thanks for following up – I'm glad to hear the certificate issue has been resolved.

For the remaining issue it is definitely worth trying the proxy's external authorisation mode. Use the --external-auth parameter or select this option from the authorisation submenu, and when authorisation requests are raised you'll be given a link to open externally instead of doing this within the proxy's popup web browser. You can log in and authorise access using this link from outside of the virtual machine (or in a more modern browser on the VM itself), which I suspect will fix the issue. (Following the pywebview guide to specify the web engine would also probably resolve this, but MSHTML is the only browser engine that is pretty much guaranteed to exist on Windows.)

Re: pre-filling the email address, it looks like Google's OAuth screen offers support for this via a login_hint parameter that Thunderbird is using. Other providers also seem to offer this, so I will add it to a future version of the proxy.

[chupocro]

Yes, using the external authorisation mode worked well :-))

When the window containing the submit form opened I could't select the link to copy/paste it into the browser outside of VM, the only way to copy the link was to click it and after it opened in the browser inside the VM copy it from the address bar of the browser. I couldn't copy the link from the authorisation window even using right click (there wasn't a context menu).

After I pasted the link into the browser outside of VM the link to be pasted into the submit form didn't show. After clicking Continue there was an animation and there was only "localhost" at the bottom of Firefox window but the link which was to be used for autorisation wasn't visible or accessible. The only way to copy the link for the authorisation was to open Developer Tools and to copy the URL from the GET request the browser was trying to send and only then I could paste the URL into the submit form.

IMAP access using Thunderbird worked well, the messages from the Inbox, Outbox, Sent and Bin showed correctly and I could send the test email without problems. However, as soon as I tried to send the 2nd email there was another request for the authorisation - I didn't expect the token would expire in just a minutes.

After completing the autorisation one more time IMAP is now working as it should.

Then I wanted to test the POP access too - I downloaded the POP branch, renamed emailproxy.py to emailproxy_pop.py and pasted the file into the same folder where I was testing IMAP. Then I edited emailproxy.config and added:

[POP-1996]
server_address = pop.gmail.com
server_port = 995

and then I created another POP account in the Thunderbird but the error reported by Thunderbird was:

Checking password...
Unable to log in at server. Probably wrong configuration, username of password.

and there wasn't any output from the emailproxy in the terminal in the same time when Thunderbird was trying to log in - as if emailproxy didn't receive any request from the Thunderbird. On the other hand, IMAP access is still working well.

Maybe I missed something :-/ When creating the POP account I configured SMTP for the outgoing server exactly the same as for the IMAP account and for the incomming server I configured POP instead of IMAP, localhost, port 1996, connection security: none, authentication method: Normal password and for the password I used the same password as for the IMAP account.

If I understand correctly how everything works I think there isn't need for removing already working authorisation and generating another. I think If IMAP is working well then POP should work with the same access credentials. And I think there should be some output in the terminal when Thunderbird is trying to access the POP server even if authorisation was missing.

I thought the problem was because there were two POP accounts configured in Thunderbird - one for accessing gmail without the proxy and the other one for accessing it using the proxy - but nothing changed even after I removed the original POP account for accessing the gmail without the proxy.

Seems as for some reason the POP request never arrives towards the proxy or if it arrives it doesn't trigger any action.

[simonrob]

Great - I'm pleased the external authorisation mode worked here.

The copy/paste problem is an issue others have encountered, but not something I can really solve (the proxy relies on pywebview for this). Glad you were able to work around the previous issue, though.

Similarly with the authorisation link – browsers have an annoying habit of not committing updates to the address bar unless a response is received from the server or there is a clear failure (try entering a non-existent website in the address bar, then cancel navigation – often what you've entered is simply deleted). This is also not really something the proxy can solve directly, though it does try to work around this in the default authorisation mode by looking for the navigation request rather than its result. The proxy's --local-server-auth option is the only way to definitively resolve this, but it is aimed at non-GUI use and brings other issues with redirect_uri port conflicts that it is best not to have to deal with.

The repeated login request is strange – this shouldn't happen unless you're using different client passwords for the same IMAP/SMTP/POP account. I'll clarify this in the documentation, but given that this is not a typical situation I presume this wasn't what you did. Tokens normally last on the order of months rather than minutes. Let me know if this happens again and you can find a way to replicate it reliably.

I would avoid renaming files and just use a separate directory for the POP branch for now. The emailproxy.config file is portable, and can just be copied into that directory. From a quick check it looks like Thunderbird is using the AUTH PLAIN method that I hadn't added – ef46726 adds this to the POP branch. Let me know how you get on – as before this is not tested (and still needs additions such as handling the CAPA command and documenting POP account scopes), but it should work enough for you to test.

[chupocro]

Thank you very much! I will test the program and report how it woks in a few days when I return home.

[simonrob]

It's worth updating to 617c123 which improves POP support and also adds the login_hint parameter. Let me know how you get on?

[simonrob]

To follow up again, 4d8224c adds a button to copy the link when in external authorisation mode. While it's not possible to fix the right-click issue, this should help avoid having to click the link to be able to use it externally.

[simonrob]

I'd like to merge this branch if it is working well for POP access. Could you let me know?

[chupocro]

Today I'll return home and I'll report you how the program works until tomorrow.

[chupocro]

Yes!! :-)) POP is now working too :-)) I tested proxy with Thunderbird and I'll test it with Eudora during the day. There will be many happy Eudora users if it works and I will then tell others at Eudora newsgroup about your program.

login_hint works well, the email address was automatically filled as it should and URL copy button worked well too.

One thing I noticed is in the original emailproxy.config file the line terminators are 0x0a and by Windows 7 Notepad everything is displayed in single line. However, as soon as the authorization is completed and the emailproxy.config has been updated with the credentials line terminators change to 0x0d 0x0a and file content is displayed correctly even with Notepad. Just an observation - some more recent text editor has to be used for the very first edit on Windows.

One more thing I noticed - when registering a Google API desktop app client there was an option to upload the logo image but that image never shows. There is "pencil&ruller" logo under Third-party apps with account access in Gmail > Security instead of the image I uploaded when registering the desktop app access.

I'll reply as soon as I test if everything works with Eudora as well.

One (strange) question: Would it be possible to port Email OAuth 2.0 Proxy to Python 2.7? I am asking because I have one Windows XP computer running 24/7 since 2006. and I'd like to run the proxy on that computer if possible. I have many specific programs (PCB design, CNC control, ...) because of which I am (on this computer) still using not only Window XP but Windows XP SP2 :-))

If that will not be possible then I'll try to run Email OAuth 2.0 Proxy on Linux with Raspberry Pi or Orange Pi PC.

[chupocro]

In the meanwhile I tested POP access with Eudora and it works well :-)

However, proxy didn't work with Eudora + IMAP. The messages reported by Eudora were:

Fetching mailbox list from server: "localhost"...

and then:

The IMAP command has failed.
Reason: Could not connect to "localhost"
Cause: connection refused (10061)

I enabled debug mode in Email OAuth 2.0 Proxy but nothing showed in the log file.

[simonrob]

Excellent – I'm glad POP support is working well. Your line ending change suggestion is a good one, too. The API client logo image is not something the proxy has anything to do with, though – I'm not sure why Google is not displaying your logo here.

I had a (very) quick look at what it would take to support Python 2.7. The changes are mostly superficial (module name differences; function signature changes, etc), but it is still a nontrivial task to be able to support both v2 and v3. I'll bear this in mind for potential future version. Please feel free to open an issue as a way to track this feature request.

I'm not sure why IMAP didn't work with Eudora, but would need a debug log to be able to trace this. A connection refused message suggests that either the port is incorrect or there is perhaps a similar SSL issue to the one you encountered before.

[chupocro]

I'm not sure why IMAP didn't work with Eudora, but would need a debug log to be able to trace this. A connection refused message suggests that either the port is incorrect or there is perhaps a similar SSL issue to the one you encountered before.

I couldn't find any mention of event 10061 or any error connected to Eudora or Email OAuth 2.0 Proxy in the Windows Event Viewer. And here is what was logged in emailproxy.txt after checking email with Eudora + IMAP:

2022-07-10 08:25:04,162: Initialising Email OAuth 2.0 Proxy from config file C:\Users\xxxx\Desktop\email-oauth2-proxy-pop/emailproxy.config
2022-07-10 08:25:04,255: Starting IMAP server at localhost:1995 (unsecured) proxying imap.gmail.com:993 (SSL/TLS)
2022-07-10 08:25:04,271: Starting SMTP server at localhost:1465 (unsecured) proxying smtp.gmail.com:465 (SSL/TLS)
2022-07-10 08:25:04,271: Starting POP server at localhost:1996 (unsecured) proxying pop.gmail.com:995 (SSL/TLS)
2022-07-10 08:25:04,271: Initialised Email OAuth 2.0 Proxy - listening for authentication requests

On the other hand, there are more than 100 lines of log messages when using Thunderbird where POP and IMAP access work in the same time.

Here is what the log looks like when checking email with Eudora + POP which works well:

2022-07-10 09:04:28,195: Accepting new connection to POP server at localhost:1996 (unsecured) proxying pop.gmail.com:995 (SSL/TLS) via ('127.0.0.1', 49252)
2022-07-10 09:04:28,320: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995) --> [ Client connected ]
2022-07-10 09:04:28,414: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995)     <-- b'+OK Gpop ready for requests from [[ IP removed ]] [[ characters removed ]]\r\n'
2022-07-10 09:04:28,414: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995) <-- b'+OK Gpop ready for requests from [[ IP removed ]] [[ characters removed ]]\r\n'
2022-07-10 09:04:28,414: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995) --> b'USER [[ email address removed ]]\r\n'
2022-07-10 09:04:28,414: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995) <-- b'+OK\r\n'
2022-07-10 09:04:28,414: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995) --> b'PASS [[ Credentials removed from proxy log ]]\r\n'
2022-07-10 09:04:28,414: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995)     --> b'AUTH XOAUTH2\r\n'
2022-07-10 09:04:28,477: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995)     <-- b'+ \r\n'
2022-07-10 09:04:28,617: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995)     --> b'[[ Credentials removed from proxy log ]]'
2022-07-10 09:04:28,758: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]])     <-- b'+OK Welcome.\r\n'
2022-07-10 09:04:28,758: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) [ Successfully authenticated POP connection - removing proxy ]
2022-07-10 09:04:28,758: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) <-- b'+OK Welcome.\r\n'
2022-07-10 09:04:28,758: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) --> b'STAT\r\n'
2022-07-10 09:04:28,836: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) <-- b'+OK 0 0\r\n'
2022-07-10 09:04:28,836: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) --> b'QUIT\r\n'
2022-07-10 09:04:28,914: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) <-- b'+OK Farewell.\r\n'
2022-07-10 09:04:28,914: POP (localhost:1996; 127.0.0.1:49252->pop.gmail.com:995; [[ email address removed ]]) <-- [ Server disconnected ]

Maybe I should use TCPView and/or Wireshark to check what exaxtly happens when Eudora is trying to access the email account via IMAP.

[simonrob]

Debug mode should always produce lots of output like your POP example. The IMAP output here is just showing startup of the proxy, and it is not receiving any connection attempts. I think having a look at the raw connection as you suggest is a good idea.

[chupocro]

I captured the traffic with Wireshark and the problem was Eudora was, despite my settings, still using the default IMAP port 143. After I configured port 143 in emailproxy.config even IMAP started to work as it should :-)

I still didn't find a way to change the IMAP port but everything works even with the default IMAP port which is below 1024 and besides I've always been using POP.

Eudora is something as Vim among editors or Microplanet Gravity among newsreaders. It has very unusual way of changing some settings - to change the port esoteric.epi file has to be moved from extrastuff folder into the main folder and only then many new options show but even more options could be changed only by editing a few .ini files. The settings for POP and SMTP port did work well but for some reason the setting for IMAP port is ignored :-/

Thank you again for making Email OAuth 2.0 Proxy and for adding support for the POP protocol! I will notify users at comp.mail.eudora.ms-windows, many of them still have all their emails ever sent and received in Eudora which has quite a few unique features not any other email client has (it can handle dozens of mailboxes with thousands of emails without lag, I remember someone having more than 100 mailboxes). Some of them even have additional 2nd or 3rd monitor for Eudora only.

I am really happy Eudora could still be used with your program :-) I've been using it for 25 years since dial-up times and 14400 bps modems.

Best regards and greetings from Hrvatska

[simonrob]

Excellent - I'm glad the proxy has helped here. Thanks for the kind words too!