Use After Free in initContext(_lsprof.c)

[kcatss]

Crash report

What happened?

Version
Python 3.14.0a0 (heads/main:34f5ae69fe, Jun 9 2024, 21:27:54) [GCC 11.4.0]
bisect from commit 2158977

Root Cause

The call_timer function can execute arbitrary code from pObj, which is initialized by the user. If the code calls _lsprof_type_Profiler_post1.disable() in Python, it will hit profiler_disable in C. Then, the self (ProfileContext) will be freed. Consequently, after call_timer returns, self->t0 will cause a use-after-free error.

static void
initContext(ProfilerObject *pObj, ProfilerContext *self, ProfilerEntry *entry)
{
    self->ctxEntry = entry;
    self->subt = 0;
    self->previous = pObj->currentProfilerContext;
    pObj->currentProfilerContext = self;
    ++entry->recursionLevel;
    if ((pObj->flags & POF_SUBCALLS) && self->previous) {
        /* find or create an entry for me in my caller's entry */
        ProfilerEntry *caller = self->previous->ctxEntry;
        ProfilerSubEntry *subentry = getSubEntry(pObj, caller, entry);
        if (subentry == NULL)
            subentry = newSubEntry(pObj, caller, entry);
        if (subentry)
            ++subentry->recursionLevel;
    }
    self->t0 = call_timer(pObj); // <-- execute arbitrary code
}

POC

import _lsprof
class evil():
    def __call__(self):
        _lsprof_type_Profiler_post1.disable()
        return True
_lsprof_type_Profiler_post1 = _lsprof.Profiler(evil())
_lsprof_type_Profiler_post1.enable()

print ("dummy")

ASAN

asan

=================================================================
==21434==ERROR: AddressSanitizer: heap-use-after-free on address 0x6060000626d0 at pc 0x7f2a7eaa4f22 bp 0x7ffe7a876d30 sp 0x7ffe7a876d20
WRITE of size 8 at 0x6060000626d0 thread T0
    #0 0x7f2a7eaa4f21 in initContext Modules/_lsprof.c:310
    #1 0x7f2a7eaa684f in ptrace_enter_call Modules/_lsprof.c:379
    #2 0x7f2a7eaa8098 in ccall_callback Modules/_lsprof.c:653
    #3 0x563c6a7ce4da in cfunction_vectorcall_FASTCALL Objects/methodobject.c:425
    #4 0x563c6ab10cb2 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #5 0x563c6ab10cb2 in call_one_instrument Python/instrumentation.c:907
    #6 0x563c6ab1258b in call_instrumentation_vector Python/instrumentation.c:1095
    #7 0x563c6ab1663d in _Py_call_instrumentation_2args Python/instrumentation.c:1150
    #8 0x563c6aa2b5c5 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:3229
    #9 0x563c6aa4ca7b in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #10 0x563c6aa4ca7b in _PyEval_Vector Python/ceval.c:1819
    #11 0x563c6aa4cc9c in PyEval_EvalCode Python/ceval.c:599
    #12 0x563c6ab64c51 in run_eval_code_obj Python/pythonrun.c:1292
    #13 0x563c6ab67b96 in run_mod Python/pythonrun.c:1377
    #14 0x563c6ab68976 in pyrun_file Python/pythonrun.c:1210
    #15 0x563c6ab6ae55 in _PyRun_SimpleFileObject Python/pythonrun.c:459
    #16 0x563c6ab6b349 in _PyRun_AnyFileObject Python/pythonrun.c:77
    #17 0x563c6abcc718 in pymain_run_file_obj Modules/main.c:357
    #18 0x563c6abcefea in pymain_run_file Modules/main.c:376
    #19 0x563c6abcfbfb in pymain_run_python Modules/main.c:639
    #20 0x563c6abcfd8b in Py_RunMain Modules/main.c:718
    #21 0x563c6abcff72 in pymain_main Modules/main.c:748
    #22 0x563c6abd02ea in Py_BytesMain Modules/main.c:772
    #23 0x563c6a539b15 in main Programs/python.c:15
    #24 0x7f2a81d33d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
    #25 0x7f2a81d33e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
    #26 0x563c6a539a44 in _start (/home/kcats/cpython/python+0x282a44)

0x6060000626d0 is located 16 bytes inside of 56-byte region [0x6060000626c0,0x6060000626f8)
freed by thread T0 here:
    #0 0x7f2a820ce537 in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:127
    #1 0x563c6a7e49c5 in _PyMem_RawFree Objects/obmalloc.c:90
    #2 0x563c6a7e6c2f in _PyMem_DebugRawFree Objects/obmalloc.c:2754
    #3 0x563c6a7e755d in _PyMem_DebugFree Objects/obmalloc.c:2891
    #4 0x563c6a81abad in PyMem_Free Objects/obmalloc.c:1010
    #5 0x7f2a7eaa4c21 in flush_unmatched Modules/_lsprof.c:766
    #6 0x7f2a7eaa6d4f in profiler_disable Modules/_lsprof.c:815
    #7 0x563c6a70183f in method_vectorcall_NOARGS Objects/descrobject.c:447
    #8 0x563c6a6d7bb9 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #9 0x563c6a6d7d14 in PyObject_Vectorcall Objects/call.c:327
    #10 0x563c6aa148c4 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:813
    #11 0x563c6aa4ca7b in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #12 0x563c6aa4ca7b in _PyEval_Vector Python/ceval.c:1819
    #13 0x563c6a6d720d in _PyFunction_Vectorcall Objects/call.c:413
    #14 0x563c6a6dbe79 in _PyObject_VectorcallDictTstate Objects/call.c:135
    #15 0x563c6a6dc351 in _PyObject_Call_Prepend Objects/call.c:504
    #16 0x563c6a876cc8 in slot_tp_call Objects/typeobject.c:9668
    #17 0x563c6a6d75e7 in _PyObject_MakeTpCall Objects/call.c:242
    #18 0x7f2a7eaa452d in _PyObject_VectorcallTstate Include/internal/pycore_call.h:166
    #19 0x7f2a7eaa452d in _PyObject_CallNoArgs Include/internal/pycore_call.h:184
    #20 0x7f2a7eaa452d in CallExternalTimer Modules/_lsprof.c:90
    #21 0x7f2a7eaa4e1c in call_timer Modules/_lsprof.c:121
    #22 0x7f2a7eaa4e1c in initContext Modules/_lsprof.c:310
    #23 0x7f2a7eaa684f in ptrace_enter_call Modules/_lsprof.c:379
    #24 0x7f2a7eaa8098 in ccall_callback Modules/_lsprof.c:653
    #25 0x563c6a7ce4da in cfunction_vectorcall_FASTCALL Objects/methodobject.c:425
    #26 0x563c6ab10cb2 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #27 0x563c6ab10cb2 in call_one_instrument Python/instrumentation.c:907
    #28 0x563c6ab1258b in call_instrumentation_vector Python/instrumentation.c:1095
    #29 0x563c6ab1663d in _Py_call_instrumentation_2args Python/instrumentation.c:1150
    #30 0x563c6aa2b5c5 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:3229
    #31 0x563c6aa4ca7b in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #32 0x563c6aa4ca7b in _PyEval_Vector Python/ceval.c:1819
    #33 0x563c6aa4cc9c in PyEval_EvalCode Python/ceval.c:599
    #34 0x563c6ab64c51 in run_eval_code_obj Python/pythonrun.c:1292
    #35 0x563c6ab67b96 in run_mod Python/pythonrun.c:1377

previously allocated by thread T0 here:
    #0 0x7f2a820ce887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x563c6a7e556c in _PyMem_RawMalloc Objects/obmalloc.c:62
    #2 0x563c6a7e489f in _PyMem_DebugRawAlloc Objects/obmalloc.c:2686
    #3 0x563c6a7e4907 in _PyMem_DebugRawMalloc Objects/obmalloc.c:2719
    #4 0x563c6a7e759f in _PyMem_DebugMalloc Objects/obmalloc.c:2876
    #5 0x563c6a81aa69 in PyMem_Malloc Objects/obmalloc.c:981
    #6 0x7f2a7eaa6892 in ptrace_enter_call Modules/_lsprof.c:373
    #7 0x7f2a7eaa8098 in ccall_callback Modules/_lsprof.c:653
    #8 0x563c6a7ce4da in cfunction_vectorcall_FASTCALL Objects/methodobject.c:425
    #9 0x563c6ab10cb2 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #10 0x563c6ab10cb2 in call_one_instrument Python/instrumentation.c:907
    #11 0x563c6ab1258b in call_instrumentation_vector Python/instrumentation.c:1095
    #12 0x563c6ab1663d in _Py_call_instrumentation_2args Python/instrumentation.c:1150
    #13 0x563c6aa2b5c5 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:3229
    #14 0x563c6aa4ca7b in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #15 0x563c6aa4ca7b in _PyEval_Vector Python/ceval.c:1819
    #16 0x563c6aa4cc9c in PyEval_EvalCode Python/ceval.c:599
    #17 0x563c6ab64c51 in run_eval_code_obj Python/pythonrun.c:1292
    #18 0x563c6ab67b96 in run_mod Python/pythonrun.c:1377
    #19 0x563c6ab68976 in pyrun_file Python/pythonrun.c:1210
    #20 0x563c6ab6ae55 in _PyRun_SimpleFileObject Python/pythonrun.c:459
    #21 0x563c6ab6b349 in _PyRun_AnyFileObject Python/pythonrun.c:77
    #22 0x563c6abcc718 in pymain_run_file_obj Modules/main.c:357
    #23 0x563c6abcefea in pymain_run_file Modules/main.c:376
    #24 0x563c6abcfbfb in pymain_run_python Modules/main.c:639
    #25 0x563c6abcfd8b in Py_RunMain Modules/main.c:718
    #26 0x563c6abcff72 in pymain_main Modules/main.c:748
    #27 0x563c6abd02ea in Py_BytesMain Modules/main.c:772
    #28 0x563c6a539b15 in main Programs/python.c:15
    #29 0x7f2a81d33d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: heap-use-after-free Modules/_lsprof.c:310 in initContext
Shadow bytes around the buggy address:
  0x0c0c80004480: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0c80004490: 00 00 00 00 00 00 00 fa fa fa fa fa fd fd fd fd
  0x0c0c800044a0: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
  0x0c0c800044b0: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0c800044c0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
=>0x0c0c800044d0: fd fd fd fa fa fa fa fa fd fd[fd]fd fd fd fd fa
  0x0c0c800044e0: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c0c800044f0: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x0c0c80004500: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
  0x0c0c80004510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c80004520: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==21434==ABORTING

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a0 (heads/main:34f5ae69fe, Jun 9 2024, 21:27:54) [GCC 11.4.0]

Linked PRs

• gh-120289: Do not free memory in disable() to prevent use-after-free #120297
• [3.13] gh-120289: Disallow disable() and clear() in external timer to prevent use-after-free (GH-120297) #121984
• [3.12] gh-120289: Disallow disable() and clear() in external timer to prevent use-after-free (GH-120297) #121989
• gh-120289: Add external timer in traverse of _lsprof.Profiler #121998
• [3.13] gh-120289: Add external timer in traverse of _lsprof.Profiler (GH-121998) #122000
• [3.12] gh-120289: Add external timer in traverse of _lsprof.Profiler (GH-121998) #122001

[Eclips4]

I cannot reproduce it using your example on current main branch, debug build, macOS.

[Zheaoli]

I cannot reproduce it using your example on current main branch, debug build, macOS.

The same, I cannot reproduce the issue with the poc code for O3 or debug build in Linux with main branch

[gaogaotiantian]

Confirmed on main. @Eclips4 and @Zheaoli , did you use --with-address-sanitizer?

So this is a very rare case, or rather an explicit case to crash the profiler. I can't think of any real use case where stopping a profiler inside the timer function makes sense. It's a very interesting hack, but not that meaningful in the real world.

We can do some heuristics to check this rare case, but the profiler itself is kind of performance sensitive. I don't want to make it any slower in most real-world cases. So I'll make a PR to detect if an external timer changes the profiler context and if so, write a warning (unraisable exception). This way the user would know that they are doing something crazy.

There's no way for us to propagate the exception from there because it returns a time. This has no effect in normal cases where an internal timer is used. I think this is a good middle ground to address this issue.

I believe even with the evil code, in most case this won't crash because the memory is not really "freed". You need ASAN to report this. So having an extra exception look reasonable to me.

The alternative is to do a PyErr_Occured check every time we uses call_timer() before we assign it to anything. We can protect it with the externel timer check but still some overhead. I'm not the security expert and from my perspective it seems very difficult to utilize this problem.

[gaogaotiantian]

Oh we actually do have an ASAN check. Maybe I'll have to make it work with ASAN.

[gaogaotiantian]

After some investigation on this. I don't believe there's a quick and clean fix to check this. I think the best way is not to free the memory of profile context when the profiler is disabled. Only free when the profiler itself is released. I made the PR and it fixed the issue.

[gaogaotiantian]

Fixed in main and 3.12 and 3.13.