Consolidate symbols into a single artifact

[MichaelSimons]

Currently in source-build, there is a symbols tarball produced for each product repo. From a consumer perspective, this seems undesirable. My hypothesis is that having all symbols packagef together in a single tarball would be more convenient.

[MichaelSimons]

@dotnet/distro-maintainers, please comment on how you would like the symbols packaged.

[tmds]

Currently in source-build, there is a symbols tarball produced for each product repo. From a consumer perspective, this seems undesirable. My hypothesis is that having all symbols packagef together in a single tarball would be more convenient.

+1

In #3547 (which was closed last week), I requested symbols to be included in the SDK tarball. The rationale was that maintainers shouldn't have to figure out where these files need to be placed.

Having a separate symbols tarball which can be extracted on top of the SDK tarball also meets that goal.

[NikolaMilosavljevic]

Currently in source-build, there is a symbols tarball produced for each product repo. From a consumer perspective, this seems undesirable. My hypothesis is that having all symbols packagef together in a single tarball would be more convenient.

+1

In #3547 (which was closed last week), I requested symbols to be included in the SDK tarball. The rationale was that maintainers shouldn't have to figure out where these files need to be placed.

Having a separate symbols tarball which can be extracted on top of the SDK tarball also meets that goal.

Both sdk and installer repos are doing a fair bit of "layout" work, to adjust SDK composition, move files around, and even create copies where needed. Additionally, when assembling the SDK, contributing packages would bring additional copies of some of the same binaries. As a result, SDK naturally contains a complex layout that is very different from binary (or obj) output of each repo's build.

We can only harvest symbols for each repo individually, and that was done with dotnet/arcade#14015. Repo-specific symbols packages are preserving the repo's output hierarchy, so that we can support the case of multiple different copies of the file with the same name.

It is a non-trivial task to assemble an SDK tarball that would have symbols for each binary placed right next to the binary.

Is this a goal for distros? I was under the impression that symbols are generally published to some distro server, to make them available for debug scenarios.

By consolidating symbols in a single archive, we'd enable distros to publish the contents using less steps.

[omajid]

symbols are generally published to some distro server, to make them available for debug scenarios

This is what happens, but not exactly. More often, the symbols are packaged up into distro-style-packages that users can then install to get access to the full debug symbols (managed, and unmanaged). I don't think we even have debug servers that can provide managed debug symbols (unmanaged debug symbol servers are a recent thing on Linux).

It is a non-trivial task to assemble an SDK tarball that would have symbols for each binary placed right next to the binary.

As a distro maintainer, I primarily care that symbols are available in a location where the tools that use the symbols can pick them up. Does anyone know how tools like vscode handles scenarios like this? Does the pdb file need to be placed next to the dll, or can we put all symbols in a single directory on disk and have vscode be happy with that too? If tools are happy with all symbols in a single central I don't think we need to create more work for ourselves to mirror the SDK layout.

[dviererbe]

I primarily care that symbols are available in a location where the tools that use the symbols can pick them up. Does anyone know how tools like vscode handles scenarios like this? Does the pdb file need to be placed next to the dll, or can we put all symbols in a single directory on disk and have vscode be happy with that too?

Good point! As far as I know, the vscode-csharp extension searches by default next to the dll where the pdb was originally dropped to, but you can configure a directory where the extension should search in (see: debugger-launchjson.md#symbol-options)

[NikolaMilosavljevic]

I primarily care that symbols are available in a location where the tools that use the symbols can pick them up. Does anyone know how tools like vscode handles scenarios like this? Does the pdb file need to be placed next to the dll, or can we put all symbols in a single directory on disk and have vscode be happy with that too?

Good point! As far as I know, the vscode-csharp extension searches by default next to the dll where the pdb was originally dropped to, but you can configure a directory where the extension should search in (see: debugger-launchjson.md#symbol-options)

Unfortunately VSCode only checks for symbols in the actual directory provided in searchPaths - it does not check the sub-directories. This means that a flat, non-hierarchical collection of symbols is needed. I'm going to look into what this means for some of the biggest repos like runtime.

[NikolaMilosavljevic]

Having a separate symbols tarball which can be extracted on top of the SDK tarball also meets that goal.

I'm looking into this option. We'd still preserve all symbols, as there are a lot of packages (i.e. various nugets) that are not included in SDK.

[NikolaMilosavljevic]

Fixed with dotnet/installer#17454

We now have two symbols packages: one with all symbols and one with SDK symbols in folder hierarchy that matches SDK binaries. SDK symbols package can be extracted on top of extracted SDK package contents to enable easy debugging.

Package names:

dotnet-sdk-symbols-8.0.100-rtm.23477.1-fedora.38-x64.tar.gz
dotnet-symbols-8.0.100-rtm.23477.1-fedora.38-x64.tar.gz

[tmds]

dotnet-symbols-8.0.100-rtm.23477.1-fedora.38-x64.tar.gz

The package name is actually dotnet-sdk-symbols-8.0.100-rtm.23477.1-fedora.38-x64.tar.gz.

It would have been nice to avoid the prefix match with the sdk.
dotnet-sdk-* now matches two tarballs.

cc @omajid

[MichaelSimons]

[Triage] Reopening as we agree that it is convenient for everyone if a wildcard search pattern can be utilitized.

[NikolaMilosavljevic]

Fixed with dotnet/installer#17521

New symbols package names are:

dotnet-symbols-sdk-8.0.100-rtm.23477.1-fedora.38-x64.tar.gz
dotnet-symbols-all-8.0.100-rtm.23477.1-fedora.38-x64.tar.gz