PyBytes::new leaks memory

[little-dude]

Hi,

First, thank you so much for pyo3 and for the great documentation.

I think I encountered a memory leak with PyBytes::new which I'm using extensively to pass numpy and pandas dataframes from Rust to Python (and vice-versa).

🌍 Environment

• Your operating system and version: Linux 5.4.23 (NixOS)
• Your python version: 3.7.6
• How did you install python (e.g. apt or pyenv)? Did you use a virtualenv?: I use a virtualenv
• Your rust version (rustc --version): rustc 1.43.0-nightly (75cf41afb 2020-03-04)
• Are you using the latest pyo3 version? Have you tried using latest master (replace version = "0.x.y" with git = "https://github.com/PyO3/pyo3")?. I'm using 0.9.0.alpha-1.

💥 Reproducing

I generated a 1MB file with dd if=/dev/zero of=1MB count=1024 bs=1024. Then cargo run the following main.rs:

// main.rs
use pyo3::{
    types::{PyBytes, PyModule},
    Python,
};
use std::{fs::File, io::Read};

fn main() {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let mut file = File::open("1MB").unwrap();
    let mut bytes = vec![];
    file.read_to_end(&mut bytes).unwrap();
    for _ in 0..4000 {
        PyBytes::new(py, &bytes[..]);
    }
}

The RAM usage should increase by 4G when this program runs. I'm not sure when the memory should be deallocated: is it when the PyBytes is dropped? Or should the Python runtime garbage-collect it, since it's not being used?

Here is a diagram of the memory usage, generated with gperftools (I zipped it because github doesn't support svg files)

heap.zip

[davidhewitt]

This is expected at the moment, see #311

I hope to improve the situation when we rethink things slightly as part of #679 .

[kngwyu]

Thank you for the nice perf, @little-dude !
As David says, it's intended behavior now and you need to release GILGuard to decrease reference counts of objects.

[little-dude]

Oh perfect, thank you @davidhewitt and @kngwyu, releasing the GILGuard works indeed.

And thanks again for your amazing work on pyo3!