Tooling for work with Profile-Guided Optimization (PGO) profiles

[zamazan4ik]

Hi!

For PGO profiles it's important to understand some insights about the profiles: covered functions, counters for each function, compare PGO profiles between each other, and calculate PGO profile overlap. In the GraalVM documentation, I cannot find information about that. Are there any PGO profile-related tools for GraalVM like it's done in other PGO ecosystems? I mean llvm-profdata for LLVM and gcov-tool for GCC.

Thanks in advance.

[axel22]

We will soon have LCOV dumps that will show which methods were executed (based on PGO data).

counters for each function, compare PGO profiles between each other, and calculate PGO profile overlap.

For this, it would be useful to understand what the use-case specifically is.
The profiles in our iprof files contain a mapping from method:BCI pairs to values, so the raw data there is a bit low-level.

[zamazan4ik]

For this, it would be useful to understand what the use-case specifically is.

Our use cases are the following:

• We want to track how PGO profiles differ between our different workloads. If the difference between two profiles are larger than X% (X we decide manually for each application) - we perform two distinct builds of our application with two different PGO profiles and deploy them simultaneously. In this case, we get the most optimized PGO builds for different workloads. As far as I know, Google uses the same approach for their in-house PGO pipeline.
• We want to track how PGO profiles differ between each GraalVM upgrade or change some compiler options. We already met some issues in this area in LLVM (link). From my tests, enabling/disabling LTO or even changing from -O2 to -O3 completely invalidated the PGO profile. We want to have a tool to track it by PGO profile metrics.

I hope my use cases are clear. If you have more questions - feel free to ask :)

[axel22]

Thanks for clarifying and providing more input!

We want to track how PGO profiles differ between our different workloads. If the difference between two profiles are larger than X% (X we decide manually for each application) - we perform two distinct builds of our application with two different PGO profiles and deploy them simultaneously. In this case, we get the most optimized PGO builds for different workloads. As far as I know, Google uses the same approach for their in-house PGO pipeline.

The use-cases make sense.

It would be easy to implement a utility that diffs two profiles like this, but we would need to precisely define what the difference metric is (i.e. exactly how to calculate X%).

[zamazan4ik]

It would be easy to implement a utility that diffs two profiles like this, but we would need to precisely define what the difference metric is (i.e. exactly how to calculate X%).

As a reference, I could suggest looking at https://llvm.org/docs/CommandGuide/llvm-profdata.html#profdata-overlap implementation. This tool is open-source and, hopefully, you can get some helpful insights from it.

[zamazan4ik]

Another question goes about merging multiple PGO profiles. Is it possible to do it somehow? If yes, does it support weighted merge? The complete use case is described in the same issue for the Go compiler: golang/go#64487

[axel22]

There is an option for the native-image-configure tool that allows merging multiple iprof files.

cc @jovanstevanovic Can you provide more input?

[jovanstevanovic]

@zamazan4ik Yes, you can merge profiles on Native Image as well with native-image-configure tool:

> native-image-configure help
.....
merge-pgo-profiles        combine multiple profile files into a single one.
    --input-dir=<path>
                          reads a profiles (*.iprof) stored in <path> directory (not subdirectories).
    --output-file=<path>
                          specifies a file to which the merged profiles are written.
    --input-file=<path>
                          reads a profile (*.iprof) from given <path>.
    --print-dropped-calling-contexts
                          print a list of all calling-contexts that are dropped during a profile merge.

[zamazan4ik]

Would be great if you put this information directly into the GraalVM PGO documentation. Because right now it's not clear that such a possibility is available for the users (at least it wasn't clear for me :))

[jovanstevanovic]

Sure, that is definitely something that we should do.

[jovanstevanovic]

@zamazan4ik About weighted merge, we do not have that implemented so far, but we plan to do it in the near future.