Resolving framework symbols

[tmds]

As described in https://github.com/dotnet/runtime/blob/master/docs/project/linux-performance-tracing.md, to do performance tracing and resolve framework symbols, you should

fetch EXACTLY the right version of crossgen for the runtime you happen to be using

For source-build, the simplest may be to include crossgen?
This could be included in the runtime package, or maybe as part of a debug symbols package.

In our source-build .NET Core 3.1 we include pdb files for Microsoft.NETCore.App dlls.
I'm not sure what tools can use these pdb files? and how they relate to needing, or not needing crossgen?
Also, we don't seem to have pdb files for Microsoft.AspNetCore.App, so we may be missing debug info for ASP.NET framework?

cc @brianrob @noahfalk @omajid @dagood @crummel @dseefeld

[dagood]

(Context I grabbed from the doc to get myself up to speed:) This is about perf tracing the shared framework (runtime) we crossgen during source-build. Not apps that are crossgen'd as part of runtime-dependent or self-contained publish.

include crossgen?

The clear way forward here is #1215. There's no obvious place to stick it based on the Microsoft build's current behavior. (Not that it can't be unique to source-build... but would be nice not to have to invent something.)

Also, we don't seem to have pdb files for Microsoft.AspNetCore.App

Filed #1565 for this.

---

I think it would be useful to get this sentence clearer from that doc:

Framework symbols need to be manually generated at the time the trace is collected.

Why "need"? Is there something about perf tracing where the data doesn't exist, e.g. it's app-specific in some way? Or is it just because the symbols aren't generally accessible, so we can provide PDBs instead if we've managed to keep track of them?

(I think I'm basically stating the same question here. 😛)

[noahfalk]

I think it would be useful to get this sentence clearer from that doc:

Framework symbols need to be manually generated at the time the trace is collected.

Why "need"? Is there something about perf tracing where the data doesn't exist, ...

When the runtime is first deployed onto a machine the symbolic data that maps RVAs to method names for R2R code is present inside the R2R images, however the profiling tools don't understand how to directly parse the R2R format to extract the data. The tools do understand a textual format called perfmap so the crossgen tool is used to read the RVA->method name map from the R2R format and transcode it to the textual perfmap format. If you skip this step then the IPs for code loaded from R2R images will not be resolvable in the trace.

[tmds]

Also, we don't seem to have pdb files for Microsoft.AspNetCore.App
Filed #1565 for this.

Thanks @dagood !

When the runtime is first deployed onto a machine the symbolic data that maps RVAs to method names for R2R code is present inside the R2R images, however the profiling tools don't understand how to directly parse the R2R format to extract the data.

@noahfalk thank you for chiming in. I wonder what is the proper way forward here:

• include crossgen with source-build as part of Support building sdks that can build self-contained / ReadyToRun / NativeAOT applications without using pre-builts from nuget.org #1215, and let profiling tools use it.
• have another way of extracting method names from R2R images that doesn't require crossgen.
• run crossgen during source build to obtain the perfmap data, and package that up with .NET Core (e.g. in debug symbol package).

[noahfalk]

The first option there, including crossgen with source-build, is the one I'd suggest exploring first. Its possible further research would discover a problem, but two things that make it appealing to me:

• Regardless of profiling, developers may want to use crossgen to precompile their own app. If crossgen is already being provided for one purpose it is usually low-cost to acquire/use it for a 2nd purpose as well.
• When customers are deploying their own applications using R2R binaries they compiled then they will want perfmap data for those binaries when profiling. If crossgen is available as a tool on the machine then they can get perfmap files regardless of how their app was deployed. If crossgen is not available then additional onus falls on the customer to pre-create the perfmap at build time and figure out how it would be deployed when it is needed.

Also to cover all the bases I guess I should ask, is it reasonable that perf could understand enough of the R2R binary format that we didn't need to transcode it? For customers I suspect it would be ideal if no pre-processing was needed, but I don't know if this is something the maintainers of perf would consider?

[tmds]

Also to cover all the bases I guess I should ask, is it reasonable that perf could understand enough of the R2R binary format that we didn't need to transcode it? For customers I suspect it would be ideal if no pre-processing was needed, but I don't know if this is something the maintainers of perf would consider?

I'm not a perf contributor, based on input I got, this is a valid option.

[noahfalk]

Thanks @tdms!

@davidwr @jkotas @MichalStrehovsky - What would you guys think about having an external tool (perf) understand enough of the R2R format that it could decode symbolic information directly? The upside is that it makes .NET feel more platform native, the downside is that it increases our compat burden. I don't know how much churn you were anticipating the format to have in the future?
cc @sdmaclea

[jkotas]

I do not think that the format is complete and stable enough to do that.

[jkotas]

I think that the right options for this are:

• Option 1: Generate the R2R images with platform native envelope (e.g. .so) and then acquire the platform native symbols for these as any other platform native symbols.
• Option 2: Generate the symbols at runtime using perf jitdump or something similar.

[brianrob]

I agree with @jkotas that the options he specifies are good choices. FWIW, I intend to update perfcollect to implement option # 2.

[noahfalk]

Thanks all!

Generate the R2R images with platform native envelope (e.g. .so) and then acquire the platform native symbols for these as any other platform native symbols.

Is this a task that has pre-existing priority around it, or we'd need to drive it based on the demands of the perf scenario? I recalled hearing desire to use platform native formats over the years, but I hadn't gotten the impression it was something where active progress was occuring (or would be likely to occur in the near future).

FWIW, I intend to update perfcollect to implement option # 2.

My primary concern (if I am remembering correctly) is that it does a flushed file write on every R2R method load which probably distorts startup a fair amount. Am I more worried about it than I should be or any thoughts on how we ameliorate it?

[sdmaclea]

flushed file write on every R2R method load

The flushing was removed to improve perf. dotnet/runtime#229

[jkotas]

Generate the R2R images with platform native envelope (e.g. .so) and then acquire the platform native symbols for these as any other platform native symbols.

Is this a task that has pre-existing priority around it, or we'd need to drive it based on the demands of the perf scenario?

This is not scheduled at the moment, but I can imagine it getting prioritized at some point for one or more of: performance, diagnostics, security, be Linux native.