OS X symbolization still does not work

[ezyang]

What version of pprof are you using?

fe62a0baab87ba3abca12f4a621532bf67c9a7d2

What operating system and processor architecture are you using?

OS X Mojave

What did you do?

$ which addr2line
/usr/local/bin/addr2line
$ cat test.cpp
#include <vector>
#include <iostream>

int main() {
  std::vector<int> x(10000000, 2);
  for (int i = 0; i < 100000; i++) {
    x[i] *= x[(i+200) % 1000];
  }
  x.resize(1000000000);
  std::cerr << x[123] << "\n";
}

$ clang++ test.cpp -g3 -fno-pie -ltcmalloc
$ HEAPPROFILE=/tmp/hh ./a.out
Starting tracking the heap
Dumping heap profile to /tmp/hh.0001.heap (3852 MB allocated cumulatively, 3852 MB currently in use)
4
Dumping heap profile to /tmp/hh.0002.heap (Exiting, 0 bytes in use)
$ pprof a.out /tmp/hh.0001.heap
Using local file a.out.
Using local file /tmp/hh.0001.heap.
Welcome to pprof!  For help, type 'help'.
(pprof) top
Total: 3852.8 MB
  3852.8 100.0% 100.0%   3852.8 100.0% 0x00007fff64e72c05

What did you expect to see?

Expected address to be symbolized

What did you see instead?

Not symbolized

[kalyanac]

What is the version of addr2line / binutils?
If it is less than 2.21, addr2line doesnt support the "-a" flag.

[ezyang]

addr2line 2.32 Excerpts from Kalyana Chadalavada's message of 2019-07-18 04:24:34 -0700:

…

What is the version of addr2line / binutils? If it is less than 2.21, addr2line doesnt support the "-a" flag.

[aalexand]

We'll do our best to look at this some time in the future, but not sure how soon this is going to be. Your best bet is to debug this a bit yourself, as you have the source code.

[ezyang]

My real plan is to just stop trying to run this on OS X :) Excerpts from Alexey Alexandrov's message of 2019-07-19 15:48:03 -0700:

…

We'll do our best to look at this some time in the future, but not sure how soon this is going to be. Your best bet is to debug this a bit yourself, as you have the source code.

[kalyanac]

@ezyang Can you try again and let us know if you are still having this issue?

[alk]

Hi. Notably, I see some symbols whenever I run without addr2line available in PATH in osx. It fails to deal with inlinings and it fails to perform C++ demangings, but it kinda works.

I.e. when running this on gperftools cpu profiler output (which uses legacy, e.g. non-protobuf format) of it's malloc_bench:

alk@hair gperftools % pprof --text pro 
Local symbolization failed for libsystem_pthread.dylib: error reading fat Mach-O file /usr/lib/system/libsystem_pthread.dylib: architecture not found in /usr/lib/system/libsystem_pthread.dylib: arm64
Local symbolization failed for libsystem_kernel.dylib: error reading fat Mach-O file /usr/lib/system/libsystem_kernel.dylib: architecture not found in /usr/lib/system/libsystem_kernel.dylib: arm64
Local symbolization failed for libsystem_malloc.dylib: stat /usr/lib/system/libsystem_malloc.dylib: no such file or directory
Local symbolization failed for libsystem_c.dylib: stat /usr/lib/system/libsystem_c.dylib: no such file or directory
Local symbolization failed for libdyld.dylib: stat /usr/lib/system/libdyld.dylib: no such file or directory
Local symbolization failed for libc++abi.dylib: stat /usr/lib/libc++abi.dylib: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: liboah.dylib
Type: cpu
Showing nodes accounting for 10.20s, 98.84% of 10.32s total
Dropped 10 nodes (cum <= 0.05s)
      flat  flat%   sum%        cum   cum%
     2.89s 28.00% 28.00%      6.64s 64.34%  [libsystem_malloc.dylib]
     1.86s 18.02% 46.03%      1.86s 18.02%  [libsystem_pthread.dylib]
     0.84s  8.14% 54.17%      1.85s 17.93%  _tc_malloc
     0.70s  6.78% 60.95%      3.55s 34.40%  [libc++abi.dylib]
     0.69s  6.69% 67.64%      1.75s 16.96%  _tc_free
     0.59s  5.72% 73.35%      1.39s 13.47%  __ZN12_GLOBAL__N_17mz_sizeEP14_malloc_zone_tPKv
     0.43s  4.17% 77.52%      0.43s  4.17%  __ZN15MallocExtension8instanceEv
     0.43s  4.17% 81.69%      0.50s  4.84%  __ZNSt3__124uniform_int_distributionIlEclINS_26linear_congruential_engineIjLj48271ELj0ELj2147483647EEEEElRT_RKNS1_10param_typeE
     0.37s  3.59% 85.27%      0.37s  3.59%  __ZN22TCMallocImplementation12GetOwnershipEPKv
     0.25s  2.42% 87.69%      0.25s  2.42%  __ZN8tcmalloc15CentralFreeList8PopulateEv
     0.24s  2.33% 90.02%      0.24s  2.33%  __ZN22TCMallocImplementation16GetAllocatedSizeEPKv
     0.18s  1.74% 91.76%      0.21s  2.03%  __ZN8tcmalloc15CentralFreeList14ReleaseToSpansEPv
     0.17s  1.65% 93.41%      2.07s 20.06%  __Z29perftools_pthread_getspecificm
     0.15s  1.45% 94.86%      1.78s 17.25%  __Z24randomize_one_size_classm
     0.15s  1.45% 96.32%      8.51s 82.46%  __ZL25bench_fastpath_throughputlm
     0.12s  1.16% 97.48%      0.12s  1.16%  _measure_once.cold.2
     0.07s  0.68% 98.16%      0.07s  0.68%  __ZNSt3__125__independent_bits_engineINS_26linear_congruential_engineIjLj48271ELj0ELj2147483647EEEyE6__evalENS_17integral_constantIbLb1EEE
     0.03s  0.29% 98.45%      0.27s  2.62%  __ZN8tcmalloc15CentralFreeList11InsertRangeEPvS1_i
     0.02s  0.19% 98.64%      0.29s  2.81%  __ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEji
     0.01s 0.097% 98.74%      0.27s  2.62%  __ZN8tcmalloc15CentralFreeList11RemoveRangeEPPvS2_i
     0.01s 0.097% 98.84%      0.26s  2.52%  __ZN8tcmalloc15CentralFreeList21FetchFromOneSpansSafeEiPPvS2_
         0     0% 98.84%      0.13s  1.26%  <unknown>
         0     0% 98.84%     10.32s   100%  [libdyld.dylib]
         0     0% 98.84%      1.79s 17.34%  __Z22randomize_size_classesv
         0     0% 98.84%      0.26s  2.52%  __ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEj
         0     0% 98.84%      0.27s  2.62%  __ZN8tcmalloc11ThreadCache21FetchFromCentralCacheEjiPFPvmE
         0     0% 98.84%     10.32s   100%  _main
         0     0% 98.84%      8.51s 82.46%  _measure_once
         0     0% 98.84%      8.53s 82.66%  _report_benchmark

But whenever I give it addr2line (which is whatever brew install gave me; version 2.40) I only see library names. (Just updating ticket; I am not really osex person and don't intend to try fixing the issue)