Python (shutdown) leaks

[jiridanek]

Brought forward from DISPATCH-1962 Make LSan suppressions more targeted and specific (within reason)

= Python leaks

Leaks in CPython itself are known and there is ongoing effort to fight them: https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and https://www.python.org/dev/peps/pep-3121

Here's valgrind suppression file from somebody who actually investigated the Python leaks and identified the harmless ones: https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp

= Router leaks

One example of a hidden leak in dispatch, which is revealed by making Python suppressions more targetted:

9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
9:     #0 0x7f78a3606e8f in __interceptor_malloc (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
9:     #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
9:     #2 0x7f78a2d657da in qdr_core_subscribe ../src/router_core/route_tables.c:149
9:     #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
9:     #4 0x7f78a2353a6c in type_call (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)

The problem is in

class Agent:
    ...
    def activate(self, address):
        ...
        self.io = IoAdapter(self.receive, address, 'L', '0', TREATMENT_ANYCAST_CLOSEST)

IoAdapter refers to Agent (through the bound method reference self.receive) and Agent refers to IoAdapter (through property self.io). Since IoAdapter is implemented in C and does not implement support for Python's cyclic GC, there is no way to break the cycle, collect the garbage, and run destructors.

Heap dump in attachment of the original Jira. The bound method is at the top of the picture. (Ignore the Mock objects, I was trying to simplify the picture while not getting crashes due to too much meddling).

== Random observations

It is possible to build special Debug build of Python, which has tools to detect leaks, asserts to prevent negative refcounts, etc. https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label

Use the following to detect python leaks (instead of valgrind)
https://docs.python.org/3/library/tracemalloc.html

Use https://pypi.org/project/objgraph (with graphviz) to view heap object trees. The following renders the picture as a png under /tmp and prints the path to stdout.

int ret = PyRun_SimpleString("import objgraph; objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
PyErr_PrintEx(0);
assert(ret == 0);

== Tools

=== CPyChecker

• https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
• https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
• https://nedbatchelder.com/blog/201502/cpychecker.html

=== Pungi

• http://www.cse.psu.edu/~gxt29/papers/refcount.pdf

[jiridanek]

Another resource is NumPy, https://numpy.org/devdocs/dev/development_advanced_debugging.html