unsafe legacy renegotiation disabled error in specific website on Python 3.12

[ilotoki0804]

Issue Description

I've encountered a puzzling issue related to the requests library when using Python 3.12. Specifically, I'm receiving an "unsafe legacy renegotiation disabled" error when trying to access a specific website. This problem appears to be unique to Python 3.12, as I did not encounter it when using Python 3.11.4.

I've encountered a puzzling issue related to the requests library when using Python 3.12. Specifically, I'm receiving an "unsafe legacy renegotiation disabled" error when trying to access a specific website, bufftoon.plaync.com/series/1001216. While I initially suspected that the issue might be related to the website itself, further investigation suggests that it's not the root cause. The error message indicates a problem with renegotiating SSL connections, which may be caused by Python 3.12, the requests library, OpenSSL, or other system-specific elements.

Reproduction Steps

I've tested this in both Jupyter and a standard Python file, and the outcomes were the same. While accessing the website, you might need to use a VPN or Proxy to bypass any restrictions related to an abroad IP address. However, I found that the website produced consistent results when accessed through Github Codespace, hosted in Singapore, it appears that such limitations may not apply.

To reproduce the issue, you can use the following Python code:

import requests

res = requests.get('https://bufftoon.plaync.com/series/1001216')
print(res)

Expected Result

The expected result when running the code should be:

<Response [200]>

Actual Result

However, when executing the same code on Python 3.12.0rc3, I encountered the following error:

---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
...
SSLError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1000)

During handling of the above exception, another exception occurred:

SSLError                                  Traceback (most recent call last)
...
SSLError: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1000)

The above exception was the direct cause of the following exception:

MaxRetryError                             Traceback (most recent call last)
...
MaxRetryError: HTTPSConnectionPool(host='bufftoon.plaync.com', port=443): Max retries exceeded with url: /series/1001216 (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

SSLError                                  Traceback (most recent call last)
...
SSLError: HTTPSConnectionPool(host='bufftoon.plaync.com', port=443): Max retries exceeded with url: /series/1001216 (Caused by SSLError(SSLError(1, '[SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:1000)')))

Additional Information

I conducted tests on both Python 3.11.4 and Python 3.12.0rc3, and the outcomes were as follows:

Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]:

<Response [200]>

Python 3.12.0rc3 (tags/v3.12.0rc3:b973ab3, Sep 19 2023, 09:39:54) [MSC v.1935 64 bit (AMD64)]:

The SSL error as described above.

System Information

Here's the system information for Python 3.11.4 and Python 3.12:

Python 3.11.4:

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.2.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.4"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "1010115f"
  },
  "urllib3": {
    "version": "2.0.5"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Python 3.12.0rc3:

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.2.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.12.0rc3"
  },
  "platform": {
    "release": "11",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "300000a0"
  },
  "urllib3": {
    "version": "2.0.5"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Please help investigate this issue as it appears to be specific to Python 3.12 and is causing disruptions in accessing this website. Thank you.

[sigmavirus24]

That exact error is coming from the ssl module. It's not an issue with requests. I would determine what needs to be done to an SSLContext to allow you to connect and then plumb that through requests via a custom adapter for that site only.

[ilotoki0804]

@sigmavirus24 Thanks for your response. I understand that the error is originating from the ssl module and not specifically from requests. However, I'm still puzzled about a few aspects:

1. What's causing this issue in Python 3.12 but not in Python 3.11.4? Are there specific changes or updates in Python 3.12 that might be triggering this SSL error?

2. Why is a custom adapter necessary to access the website while standard web browsers have no trouble?

If this isn't considered an error, could you provide any recommendations for potential solutions involving a custom adapter?

[sigmavirus24]

We use the standard cipher suites from the stdlib/OpenSSL. There may be a change there causing you problems. You could use a tool to compare them like Wireshark but that isn't something we will help you with.

Browsers don't use OpenSSL. They're also usually far more permissive to avoid complaints like this. https://stackoverflow.com/a/71646353 Should help you

[sigmavirus24]

See also urllib3/urllib3#2653

[ilotoki0804]

@sigmavirus24 Thank you. I will use the adapter you recommended.

[ghost]

For complete implementation in code -> https://gist.github.com/FluffyDietEngine/94c0137445555a418ac9f332edfa6f4b