Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Memory Profiler #31534

Closed
wants to merge 5 commits into from
Closed

[WIP] Memory Profiler #31534

wants to merge 5 commits into from

Commits on Apr 2, 2019

  1. Propagate incomplete backtrace return code through rec_backtrace*() 

    Also adds `bt_overflow` flag instead of spitting out messages in the
middle of profiling, to be used by client profiling code.

This change allows for better checking of stack frames that could be
incomplete due to insufficient backtrace buffer space.  Realistically, a
single truncated stack trace in the case of a sampling time profiler is
unlikely to create large problems.  However when taking backtraces for
things such as a memory profiler, it is critical that all backtraces be
accurate, and so we allow client code to be somewhat stricter here.
    @staticfloat
    staticfloat committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    397d182 View commit details
    Browse the repository at this point in the history

  2. Add _reformat_bt() method that takes a single array of backtraces

    @staticfloat
    staticfloat committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    a74dabb View commit details
    Browse the repository at this point in the history

  3. Standardize backtrace buffers as uintptr_t

    @staticfloat
    staticfloat committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    5d9f3fa View commit details
    Browse the repository at this point in the history

  4. Instrument GC with memory profiler implementation 

    This adds C support for a memory profiler within the GC, tracking
locations of allocations, deallocations, etc...  It operates in a
similar manner as the time profiler with single large buffers setup
beforehand through an initialization function, reducing the need for
expensive allocations while the program being measured is running.

The memory profiler instruments the GC in all locations that the GC
statistics themselves are being modified (e.g. `gc_num.allocd` and
`gc_num.freed`) by introducing new helper functions
`jl_gc_count_{allocd,freed,reallocd}()`.  Those utility functions call
the `jl_memprofile_track_{de,}alloc()` method to register an address,
a size and a tag with the memory profiler.  We also track type
information as this can be critically helpful when debugging, and to do
so without breaking API guarantees we insert methods to set the type of
a chunk of memory after allocating it where necessary.

The tagging system allows the memory profiler to disambiguate, at
profile time, between e.g. pooled allocations and the "big" allocator.
It also allows the memory allocator to support tracking multiple "memory
domains", e.g. a GPU support package could manually call
`jl_memprofile_track_alloc()` any time a chunk of memory is allocated on
the GPU so as to use the same system.  By default, all values are
tracked, however one can set a `memprof_tag_filter` value to track only
the values you are most interested in.  (E.g. only CPU domain big
allocations)

To disambiguate the memory and time profilers, we split them out into
separate modules.
    @staticfloat
    staticfloat committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    e38ab23 View commit details
    Browse the repository at this point in the history

  5. Update stdlib/Profile/src/memory.jl 

    Co-Authored-By: staticfloat <[email protected]>
    @ararslan @staticfloat
    ararslan and staticfloat committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    a32f2a6 View commit details
    Browse the repository at this point in the history