-
-
Notifications
You must be signed in to change notification settings - Fork 30.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[3.7] gh-95778: CVE-2020-10735: Prevent DoS by very large int() #96504
Conversation
This is based off of psrt/CVE-2020-10735-3.8backport branch at cd54fc3.
This is required for the 3.7 tree to pass on modern macOS.
I don't know why, but macOS in 3.7 CI is failing to build the zlib module these days so it's exposing this test that didn't have the proper `@requires_zlib` annotation. Getting it to build with zlib and other things that are now wrongly "missing" in the 3.7 CI setup would be nice, but probably involves invasive backporting of parts of python@b29d0a5 by a macOS domain expert. Not worth it.
This test also appears to require changes to Lib/ctypes/macholib/dyld.py to work in the existing macOS CI config. I'm just skipping it, backporting that would be a feature. Not going to happen in 3.7. There may be a way to configure macOS CI to use an older macOS and toolchain instead as an alternate option. Someone else can figure that out if so. This branch only lives for another 9 months per https://peps.python.org/pep-0537/
Thanks Ned!
the 3.8 branch got rid of this line already. it blocks seeing the actual error while testing a doc build!
Kinda looks like the Windows issue is the
Then we see further down that
Not sure why |
Per mdickinson@'s comment on the main branch PR.
…#96537) Converting a large enough `int` to a decimal string raises `ValueError` as expected. However, the raise comes _after_ the quadratic-time base-conversion algorithm has run to completion. For effective DOS prevention, we need some kind of check before entering the quadratic-time loop. Oops! =) The quick fix: essentially we catch _most_ values that exceed the threshold up front. Those that slip through will still be on the small side (read: sufficiently fast), and will get caught by the existing check so that the limit remains exact. The justification for the current check. The C code check is: ```c max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10 ``` In GitHub markdown math-speak, writing $M$ for `max_str_digits`, $L$ for `PyLong_SHIFT` and $s$ for `size_a`, that check is: $$\left\lfloor\frac{M}{3L}\right\rfloor \le \left\lfloor\frac{s - 11}{10}\right\rfloor$$ From this it follows that $$\frac{M}{3L} < \frac{s-1}{10}$$ hence that $$\frac{L(s-1)}{M} > \frac{10}{3} > \log_2(10).$$ So $$2^{L(s-1)} > 10^M.$$ But our input integer $a$ satisfies $|a| \ge 2^{L(s-1)}$, so $|a|$ is larger than $10^M$. This shows that we don't accidentally capture anything _below_ the intended limit in the check. <!-- gh-issue-number: pythongh-95778 --> * Issue: pythongh-95778 <!-- /gh-issue-number --> Co-authored-by: Gregory P. Smith [Google LLC] <[email protected]>
"12 successful checks" is a little thin, no? |
In fact, it looks like the latest |
I don't believe it does. There is no history of success for those ASAN and USAN builds on the 3.8 and 3.7 branches. |
We looked into this. 3.7 and 3.8 are typically not running address and undefined behavior sanitizers on buildbots. The label-driven trigger on PRs is overly broad and runs configurations that aren't used by us for the oldest branches in regular testing. At least 3.8 fails ASAN and UBSAN without GH-96503 applied as well. I'll leave it to @ned-deily to land this to his branch but I'm landing 3.8. |
@ypankovych, don't leave unsolicited approving reviews without comments, those are not actionable for us. |
@ambv sorry, that was more of a stamp |
I get it, @ypankovych, your intentions were good. But we get many random stamps on open PRs and when they don't even have a comment saying why that person is confident the PR is good, we can't really do anything with that. |
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
pythongh-95778: CVE-2020-10735: Prevent DoS by very large int() (pythonGH-96504) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca94) pythongh-95778: Mention sys.set_int_max_str_digits() in error message (python#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc) pythongh-96848: Fix -X int_max_str_digits option parsing (python#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 4135166)
Integer to and from text conversions via CPython's bignum
int
type is not safe against denial of service attacks due to malicious input. Very large input strings with hundred thousands of digits can consume several CPU seconds.This PR comes fresh from a pile of work done in our private PSRT security response team repo.
This backports #96499 aka 511ca94
Signed-off-by: Christian Heimes [Red Hat] [email protected]
Tons-of-polishing-up-by: Gregory P. Smith [Google] [email protected]
Reviews via the private PSRT repo via many others (see the NEWS entry in the PR).
I wrote up a one pager for the release managers.
Additional review is wise on this 3.7 PR as the backport was more complicated due to internal Python config and startup code cleanup in 3.8 and onwards.