OpenSSL 3 performance regression on Windows

[boxothing]

Bug report

Bug description:

After updating my Python from version 3.11.3 to 3.11.5 on Windows, I noticed unusually high CPU usage when running Python scripts. I ran simple python scripts with requests, and it was more evident when I started running multiple instances.

With Python version 3.11.3, my CPU usage is extremely low even when I have 100's of python instances running concurrently. (< 10%)

I tried other version of Python releases and found out this behavior is apparent for Python versions 3.11.5 and above.

3.11.3 - Normal
3.11.4 - Normal
3.11.5 - High CPU usage
3.12.0rc1 - High CPU usage
3.12.0rc2 - High CPU usage

Tested on both Widows 10 and 11.

Here's an example python code used.

from concurrent.futures import ProcessPoolExecutor
import time
import requests

def worker(name):
    while True:
        try:
            res = requests.get("https://httpbin.org/get")

            if res:
                print(f"Worker {name}: {res.status_code}")
        except:
            pass

        time.sleep(3)

def start_worker(workers):
    executor = ProcessPoolExecutor(max_workers=workers)

    for n in range(workers):
        executor.submit(worker, n)
        time.sleep(1)

    executor.shutdown(wait=False)

if __name__ == '__main__':
    print("Starting workers ...")
    start_worker(20)

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Windows 10, 11

[ericvsmith]

Can you reproduce this without using requests? Maybe with urllib.request?

[boxothing]

Can you reproduce this without using requests? Maybe with urllib.request?

It did not occur to me because nothing else was changed when I upgraded Python version.
I could not reproduce the same issue using urllib.request, or urllib3.

Does this mean this is requests specific issue? It's odd that it only happens with Python version 3.11.5 and above.

Instead of calling requests.get(), creating a Session() object outside loop seems to alleviate this issue.

s = requests.Session()
while True:
    res = s.get("https://httpbin.org/get")
    time.sleep(3)

[neonene]

OpenSSL 3.0.9 seems to have this issue on Windows, which has been used since e2d7366.

Repro without requests extension:

import time
import threading
import ssl
import certifi
cert = certifi.where()

def worker():
    while True:
        ssl.create_default_context(cafile=cert)
        time.sleep(1)

thrs = [threading.Thread(target=worker, daemon=True) for i in range(10)]
for thr in thrs:
    thr.start()
    thr.join(0.1)
input(':')

[ericvsmith]

@neonene : on which version of python and which platform did you run this?

[neonene]

I ran OP's script and mine on Windows 10/11, and my bisection reached:
e2d7366 (main)
6de9cff (3.12.0b4+)
4a5e8d9 (3.11.4+)

requests extension creates SSLContext object on every https-access via urllib3.
I suspect that current OpenSSL has some overhead.

[kinshukdua]

The performance hit from using OpenSSL 3.X has been known since 2021. It was first mentioned in this open issue openssl/openssl#17064. There's not much we can do, since downgrading to version 1.1.1 is only a temporary fix. Since version 1.1.1 is no longer supported.

[rseuchter-vw]

With one of my scripts that is doing lots of HTTP requests through requests (across multiple processes) I noticed a performance degradation of roughly factor 5 while the CPU usage skyrockets.

I concluded the move to OpenSSL 3.0 introduced these regressions. I found openssl/openssl#17627 which links multiple performance degradations. There, openssl/openssl#17627 (comment) suggests that OpenSSL 3.1 brings some improvements.

Are there plans to move to OpenSSL 3.1+ in Python 3.11, 3.12, or higher?

[zware]

We haven't really discussed an update to 3.1 as far as I know, but I would not expect it. We prefer to avoid feature updates in our dependencies during the lifetime of a given Python release, so we'll probably stick to LTS OpenSSL releases (especially for 3.11 and 3.12), which 3.1 is not. I have not been able to quickly find whether 3.2 (currently at alpha 2) will be LTS or not.

All that said, it should be possible to build and use your own 3.1 if you really need it. Have a look at the prepare_ssl.bat script to get started down that road.

[rseuchter-vw]

I don't have the necessary infrastructure to build my own binaries. So for now I'll have to live with it.

However, after reading through #95031 I moved to a session based approach when using requests. This has brought the script's throughput close to what it was before. However, I think cpu consumption is still higher and it has driven up the script's complexity.

[zware]

Thanks for the pointer to #95031. I think we can close this as a duplicate and consolidate discussion there.

Duplicate of #95031.