Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression: ImportError for HeaderWriteError in long-running process post-Python update for CVE-2024-6923 #124170

Open
julian-klode opened this issue Sep 17, 2024 · 8 comments
Open

Regression: ImportError for HeaderWriteError in long-running process post-Python update for CVE-2024-6923 #124170

julian-klode opened this issue Sep 17, 2024 · 8 comments
Labels
topic-email type-bug An unexpected behavior, bug, or error

Comments

@julian-klode
Copy link

julian-klode commented Sep 17, 2024 •
edited by github-actions bot
Loading

Bug report

Bug description:

Pull request #122233 introduced a new class HeaderWriteError in commit 0976339 and imports that from email.generator.

This breaks running applications that have imported other parts of email before the update, and then try to import the generator past the update.

Now this is a bit silly, but it is what email.message.Message.as_string() does, it imports email.generator inside the function - which may happen at any point of the program run-time rather than at startup.

For example, the following pseudo-code will fail, assuming it has not generated another email earlier or manually imported the email.generator module.

import email.message
<do something for a long time, such as wait for a web form, Python is being upgraded here>
msg = <prepare a message>
msg.as_string()

A particular instance of the issue is the unattended-upgrades package in Ubuntu and Debian, which will install the security update and then may send an email and fail there due to the ImportError, see https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940.

I'm wondering if it's feasible to add a workaround to the stable branches:

Cchange the email.generator module import:

from email.errors import HeaderWriteError

to graciously support the previous version email.errors:

try:
    from email.errors import HeaderWriteError
except ImportError:
    from email.errors import MessageError as HeaderWriteError

This is a safe change, existing applications, where the import fails can't be having except HeaderWriteError statements anyway.

Thanks.

CPython versions tested on:

3.12

Operating systems tested on:

No response
@julian-klode julian-klode added the type-bug An unexpected behavior, bug, or error label Sep 17, 2024
@julian-klode
Copy link
Author

I think the real fix is to not lazily import modules like this in the standard library, but I assume there's a reason email.generators is lazily imported?

@jimkoeleman
Copy link

Since yesterday I have the same error on my servers, using Python 3.8. It happens to crash my unicorn workers resulting in bad gateway errors for end users. Until I manually restart my servers. Happened yesterday and also today in the morning.

@muzaffar-omer
Copy link

It happened to one of our servers as well. Running:

  • Django 4.2.16
  • gunicorn: 22.0.0
  • Python 3.8.10

We upgraded recently to Django 4.2.16, commit django/django@bf4888d might be related.

Error 
gunicorn[1014232]: [18/Sep/2024 10:58:11][common.services.pandora_service._user_access_token:187][Root=1-66ea9623-5aee4eae0266bcd41a3a1159][DEBUG] Token retrieved in : 9.248 ms
gunicorn[1182738]: [2024-09-18 08:58:11 +0000] [1182738] [ERROR] Exception in worker process
gunicorn[1182738]: Traceback (most recent call last):
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
gunicorn[1182738]:     worker.init_process()
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 147, in init_process
gunicorn[1182738]:     super().init_process()
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 134, in init_process
gunicorn[1182738]:     self.load_wsgi()
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
gunicorn[1182738]:     self.wsgi = self.app.wsgi()
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
gunicorn[1182738]:     self.callable = self.load()
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
gunicorn[1182738]:     return self.load_wsgiapp()
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
gunicorn[1182738]:     return util.import_app(self.app_uri)
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/gunicorn/util.py", line 371, in import_app
gunicorn[1182738]:     mod = importlib.import_module(module)
gunicorn[1182738]:   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
gunicorn[1182738]:     return _bootstrap._gcd_import(name[level:], package, level)
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
gunicorn[1182738]:   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
gunicorn[1182738]:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
gunicorn[1182738]:   File "/opt/cloud/current/pix4uav/pix4d/__init__.py", line 3, in <module>
gunicorn[1182738]:     from .celery import app as celery_app  # noqa
gunicorn[1182738]:   File "/opt/cloud/current/pix4uav/pix4d/celery.py", line 16, in <module>
gunicorn[1182738]:     from .settings.sentry import before_send
gunicorn[1182738]:   File "/opt/cloud/current/pix4uav/pix4d/settings/sentry.py", line 5, in <module>
gunicorn[1182738]:     from sentry_sdk.integrations.django import DjangoIntegration
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 61, in <module>
gunicorn[1182738]:     from sentry_sdk.integrations.django.middleware import patch_django_middlewares
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/middleware.py", line 38, in <module>
gunicorn[1182738]:     from .asgi import _asgi_middleware_mixin_factory
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/sentry_sdk/integrations/django/asgi.py", line 13, in <module>
gunicorn[1182738]:     from django.core.handlers.wsgi import WSGIRequest
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 5, in <module>
gunicorn[1182738]:     from django.core.handlers import base
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 12, in <module>
gunicorn[1182738]:     from django.utils.log import log_response
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/django/utils/log.py", line 6, in <module>
gunicorn[1182738]:     from django.core import mail
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/django/core/mail/__init__.py", line 10, in <module>
gunicorn[1182738]:     from django.core.mail.message import (
gunicorn[1182738]:   File "/opt/cloud/releases/20240913T095509/.venv/lib/python3.8/site-packages/django/core/mail/message.py", line 4, in <module>
gunicorn[1182738]:     from email import generator, message_from_string
gunicorn[1182738]:   File "/usr/lib/python3.8/email/generator.py", line 17, in <module>
gunicorn[1182738]:     from email.errors import HeaderWriteError
gunicorn[1182738]: ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.8/email/errors.py)
gunicorn[1182738]: [2024-09-18 08:58:11 +0000] [1182738] [INFO] Worker exiting (pid: 1182738)
gunicorn[1014225]: [2024-09-18 08:58:11 +0000] [1014225] [ERROR] Worker (pid:1182738) exited with code 3
gunicorn[1014228]: [2024-09-18 10:58:12 +0200] [1014228] [INFO] Worker exiting (pid: 1014228)
gunicorn[1014235]: [2024-09-18 10:58:12 +0200] [1014235] [INFO] Worker exiting (pid: 1014235)
gunicorn[1014232]: [2024-09-18 10:58:12 +0200] [1014232] [INFO] Worker exiting (pid: 1014232)
gunicorn[1014229]: [2024-09-18 10:58:12 +0200] [1014229] [INFO] Worker exiting (pid: 1014229)
gunicorn[1014225]: [2024-09-18 08:58:17 +0000] [1014225] [ERROR] Shutting down: Master
gunicorn[1014225]: [2024-09-18 08:58:17 +0000] [1014225] [ERROR] Reason: Worker failed to boot.
systemd[1]: gunicorn.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
systemd[1]: gunicorn.service: Failed with result 'exit-code'.

@jimkoeleman
Copy link

jimkoeleman commented Sep 18, 2024 •
edited
Loading

We run on Django 3.2.9 (..i know), got the following error log. Still no solution found for the problem.

[2024-09-18 03:41:11 +0200] [17792] [INFO] Booting worker with pid: 17792
[2024-09-18 04:31:45 +0200] [18105] [INFO] Booting worker with pid: 18105
[2024-09-18 06:27:00 +0200] [20686] [INFO] Booting worker with pid: 20686
[2024-09-18 06:27:01 +0200] [20686] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/www/crftwrkbn/crftr/crftr/wsgi.py", line 13, in <module>
    from django.core.wsgi import get_wsgi_application
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/django/core/wsgi.py", line 2, in <module>
    from django.core.handlers.wsgi import WSGIHandler
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 5, in <module>
    from django.core.handlers import base
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 12, in <module>
    from django.utils.log import log_response
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/django/utils/log.py", line 6, in <module>
    from django.core import mail
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/django/core/mail/__init__.py", line 9, in <module>
    from django.core.mail.message import (
  File "/var/www/crftwrkbn/.venv/lib/python3.8/site-packages/django/core/mail/message.py", line 2, in <module>
    from email import (
  File "/usr/lib/python3.8/email/generator.py", line 17, in <module>
    from email.errors import HeaderWriteError
ImportError: cannot import name 'HeaderWriteError' from 'email.errors' (/usr/lib/python3.8/email/errors.py)
[2024-09-18 06:27:01 +0200] [20686] [INFO] Worker exiting (pid: 20686)
[2024-09-18 06:27:04 +0200] [542] [INFO] Shutting down: Master
[2024-09-18 06:27:04 +0200] [542] [INFO] Reason: Worker failed to boot.

@JelleZijlstra
Copy link
Member

It sounds like those of you who were getting this error were upgrading Python on the file system without restarting your processes. If so, you should be able to get around the issue by restarting all Python processes to have the new code.

@jimkoeleman
Copy link

Thanks for your reply. Hmm this could mean that the server restart solved the problem. In our setup with 4 web servers it could hypothetically be that yesterday server-1 and server-2 were automatically updated, and server-3 and server-4 today so that it should not happen anymore after today. Let's wait until tomorrow morning to see if this assumption is correct.

@DanielRuf
Copy link
Contributor

The question is, is a manual restart needed because unattended-upgrades did not do some restart and this error state will persist until the restart is done?

What are the possible options for current servers? So far I understand:

  • restart python (what is the cleanest way to do that so that current python3 processes like /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers and /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal are correctly started again?)
  • do a full system restart
  • wait for the patch

Does the error happen again on systewms which are still running but everything was upgraded by unattended-upgrades?

@frewper
Copy link

frewper commented Oct 22, 2024

how can one disable this automatic update ?

mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: reload the "email.errors" package to workaround (LP: 2080940)
d95158a 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Thanks to Julian Klode for his excellent analysis.
@mvo5 mvo5 mentioned this issue Nov 10, 2024
Merged
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: reload the "email.errors" package to workaround (LP: 2080940)
2c565f8 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Thanks to Julian Klode for his excellent analysis.
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: reload the "email.errors" package to workaround (LP: 2080940)
613d2fc 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules. So instead import email as late as
possible.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Thanks to Julian Klode for his excellent analysis.
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: import "email" package late to workaround (LP: 2080940)
c8f846d 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules. So instead import email as late as
possible.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Thanks to Julian Klode for his excellent analysis.
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: import "email" package late to workaround (LP: 2080940)
9fa4d29 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules. So instead import email as late as
possible.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Thanks to Julian Klode for his excellent analysis.
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: import "email.genators" package early to workaround
b2a85ab 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules. So instead import email.generators early
so that it is available in memory and there is no mismatch.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Thanks to Julian Klode for his excellent analysis.

Closes: LP:2080940
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: import "email.genators" package early as workaround
b908b17 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules. So instead import email.generators early
so that it is available in memory and there is no mismatch.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Unfortunately while this commit fixes the autopkgtest failure that
is now triggered by this the real issue that the users see is not
fixed because the "old" unattended-upgrades will perform the upgrade
and it will upgrade both the problematic python and the fixed u-u
but because u-u does not re-exec the "old" u-u still runs and the
fixed code from u-u will not be used.

Thanks to Julian Klode for his excellent analysis.
mvo5 added a commit to mvo5/unattended-upgrades that referenced this issue Nov 10, 2024
@mvo5
u-u: import "email.genators" package early as workaround
ca9c9e0 
This commit works around an issue when python itself is upgraded
by u-u and the content of email.message now needs a newer email.errors
but u-u already (implicitely) loaded that module so there is a
old version in sys.modules. So instead import email.generators early
so that it is available in memory and there is no mismatch.

See python/cpython#124170 and
https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2080940
for details.

Unfortunately while this commit fixes the autopkgtest failure that
is now triggered by this the real issue that the users see is not
fixed because the "old" unattended-upgrades will perform the upgrade
and it will upgrade both the problematic python and the fixed u-u
but because u-u does not re-exec the "old" u-u still runs and the
fixed code from u-u will not be used.

Thanks to Julian Klode for his excellent analysis.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic-email type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@DanielRuf @JelleZijlstra @muzaffar-omer @julian-klode @terryjreedy @frewper @jimkoeleman