PyBuffer::get leads to memory corruption under PyPy

[oconnor663]

I've put up a repository containing a minimal repro of this bug: https://github.com/oconnor663/pyo3_pybuffer_repro. You can run the repro.sh script there to execute all the repro steps below.

I'm on x86-64 Arch Linux, with CPython 3.9.6, PyPy 7.3.5, and Rust 1.53.0. Here is a minimal Rust extension demonstrating this bug:

Cargo.toml:

[package]
name = "repro"
version = "0.1.0"
edition = "2018"

[lib]
name = "repro"
crate-type = ["cdylib"]

[dependencies.pyo3]
version = "0.14.1"
features = ["extension-module"]

src/lib.rs:

use pyo3::buffer::PyBuffer;
use pyo3::prelude::*;

#[pyfunction]
fn buffer_len(obj: &PyAny) -> PyResult<usize> {
    let buffer = PyBuffer::<u8>::get(obj)?;              // <-- THIS IS THE INTERESTING LINE
    Ok(buffer.item_size() * buffer.item_count())
}

#[pymodule]
fn repro(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(buffer_len, m)?)?;
    Ok(())
}

test.py:

from repro import buffer_len

assert buffer_len(b"foo") == 3

I create a virtualenv -p pypy3, and inside that virtualenv I build and install with extension module with maturin develop. (Note that this relies on the fix at PyO3/maturin#596, which only just shipped in v0.11.2.) Finally I execute test.py over and over in a loop something like this in Bash (where python refers to the virtualenv Python interpreter):

i=1
while python ./test.py ; do
    echo "run number $i"
    i=$(($i + 1))
done

Here's an example run of repro.sh (which executes all the steps above) on my box:

❯ ./repro.sh
created virtual environment PyPy3.7.10.final.0-64 in 144ms
  creator PyPy3Posix(dest=/tmp/tmp.n8Evydf1LW, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/jacko/.local/share/virtualenv)
    added seed packages: pip==21.1.2, setuptools==57.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
🔗 Found pyo3 bindings
🐍 Found PyPy 3.7 at python
   Compiling pyo3 v0.14.1
   Compiling repro v0.1.0 (/home/jacko/tmp/repro)
    Finished dev [unoptimized + debuginfo] target(s) in 3.63s
run number 1
run number 2
run number 3
malloc_consolidate(): invalid chunk size
./repro.sh: line 14: 125301 Aborted                 (core dumped) python ./test.py

Note that the assert in test.py did pass a few times before the crash in this case. On some runs it crashes immediately for me, and on others it gets through a handful of executions before crashing like it did here.

If I replace virtualenv -p pypy3 with virtualenv -p /usr/bin/python (CPython 3.9.6), test.py always succeeds and that while loop runs forever.

[messense]

I can reproduce this on x86_64 macOS with pypy3.7

❯ ./repro.sh
run number 1
python(98724,0x11530fe00) malloc: Incorrect checksum for freed object 0x7ff6e6604138: probably modified after being freed.
Corrupt value: 0x1
python(98724,0x11530fe00) malloc: *** set a breakpoint in malloc_error_break to debug
./repro.sh: line 9: 98724 Abort trap: 6           python ./test.py

lldb backtrace

(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x00007fff203ba30e libsystem_malloc.dylib`tiny_free_list_remove_ptr + 383
    frame #1: 0x00007fff203b9748 libsystem_malloc.dylib`tiny_free_no_lock + 1018
    frame #2: 0x00007fff203b91f9 libsystem_malloc.dylib`free_tiny + 442
    frame #3: 0x00000001010441f0 libpypy3-c.dylib`___lldb_unnamed_symbol30423$$libpypy3-c.dylib + 624
    frame #4: 0x00000001004eb9ea libpypy3-c.dylib`___lldb_unnamed_symbol9141$$libpypy3-c.dylib + 170
    frame #5: 0x0000000100163b45 libpypy3-c.dylib`___lldb_unnamed_symbol430$$libpypy3-c.dylib + 101
    frame #6: 0x000000010031af9c libpypy3-c.dylib`___lldb_unnamed_symbol7003$$libpypy3-c.dylib + 380
    frame #7: 0x00000001003aa6e6 libpypy3-c.dylib`___lldb_unnamed_symbol7848$$libpypy3-c.dylib + 70
    frame #8: 0x00000001003a1ae3 libpypy3-c.dylib`___lldb_unnamed_symbol7819$$libpypy3-c.dylib + 355
    frame #9: 0x0000000100384f75 libpypy3-c.dylib`___lldb_unnamed_symbol7565$$libpypy3-c.dylib + 53
    frame #10: 0x000000010093f3f6 libpypy3-c.dylib`___lldb_unnamed_symbol19277$$libpypy3-c.dylib + 70
    frame #11: 0x0000000100c845c0 libpypy3-c.dylib`___lldb_unnamed_symbol24263$$libpypy3-c.dylib + 144
    frame #12: 0x0000000100352d2f libpypy3-c.dylib`___lldb_unnamed_symbol7288$$libpypy3-c.dylib + 1023
    frame #13: 0x00000001010440a2 libpypy3-c.dylib`___lldb_unnamed_symbol30423$$libpypy3-c.dylib + 290
    frame #14: 0x00000001003ab866 libpypy3-c.dylib`___lldb_unnamed_symbol7850$$libpypy3-c.dylib + 1078
    frame #15: 0x00000001003a205b libpypy3-c.dylib`___lldb_unnamed_symbol7819$$libpypy3-c.dylib + 1755
    frame #16: 0x0000000100384f75 libpypy3-c.dylib`___lldb_unnamed_symbol7565$$libpypy3-c.dylib + 53
    frame #17: 0x000000010093f3f6 libpypy3-c.dylib`___lldb_unnamed_symbol19277$$libpypy3-c.dylib + 70
    frame #18: 0x0000000100c845c0 libpypy3-c.dylib`___lldb_unnamed_symbol24263$$libpypy3-c.dylib + 144
    frame #19: 0x0000000100352d2f libpypy3-c.dylib`___lldb_unnamed_symbol7288$$libpypy3-c.dylib + 1023
    frame #20: 0x00000001010440a2 libpypy3-c.dylib`___lldb_unnamed_symbol30423$$libpypy3-c.dylib + 290
    frame #21: 0x00000001003ab866 libpypy3-c.dylib`___lldb_unnamed_symbol7850$$libpypy3-c.dylib + 1078
    frame #22: 0x00000001003a205b libpypy3-c.dylib`___lldb_unnamed_symbol7819$$libpypy3-c.dylib + 1755
    frame #23: 0x0000000100384f75 libpypy3-c.dylib`___lldb_unnamed_symbol7565$$libpypy3-c.dylib + 53
    frame #24: 0x000000010093f3f6 libpypy3-c.dylib`___lldb_unnamed_symbol19277$$libpypy3-c.dylib + 70
    frame #25: 0x0000000100c845c0 libpypy3-c.dylib`___lldb_unnamed_symbol24263$$libpypy3-c.dylib + 144
    frame #26: 0x0000000100352d2f libpypy3-c.dylib`___lldb_unnamed_symbol7288$$libpypy3-c.dylib + 1023
    frame #27: 0x00000001010440a2 libpypy3-c.dylib`___lldb_unnamed_symbol30423$$libpypy3-c.dylib + 290
    frame #28: 0x00000001003ab866 libpypy3-c.dylib`___lldb_unnamed_symbol7850$$libpypy3-c.dylib + 1078
    frame #29: 0x00000001003a205b libpypy3-c.dylib`___lldb_unnamed_symbol7819$$libpypy3-c.dylib + 1755
    frame #30: 0x0000000100384f75 libpypy3-c.dylib`___lldb_unnamed_symbol7565$$libpypy3-c.dylib + 53
    frame #31: 0x000000010093f3f6 libpypy3-c.dylib`___lldb_unnamed_symbol19277$$libpypy3-c.dylib + 70
    frame #32: 0x0000000100c845c0 libpypy3-c.dylib`___lldb_unnamed_symbol24263$$libpypy3-c.dylib + 144
    frame #33: 0x0000000100352d2f libpypy3-c.dylib`___lldb_unnamed_symbol7288$$libpypy3-c.dylib + 1023
    frame #34: 0x00000001010440a2 libpypy3-c.dylib`___lldb_unnamed_symbol30423$$libpypy3-c.dylib + 290
    frame #35: 0x000000010030b8ad libpypy3-c.dylib`___lldb_unnamed_symbol6947$$libpypy3-c.dylib + 125
    frame #36: 0x000000010030625b libpypy3-c.dylib`___lldb_unnamed_symbol6929$$libpypy3-c.dylib + 4715
    frame #37: 0x0000000101154176 libpypy3-c.dylib`___lldb_unnamed_symbol33050$$libpypy3-c.dylib + 70
    frame #38: 0x0000000100003f34 python`start + 52

[davidhewitt]

Nice repro, that's really useful. I think I've found the issue, see #1737