Add Benchmarking CI

[christiangnrd]

Using a JuliaGPU branch instead of from my fork. See #419 for the start of this branch.

Original text:
Copying over the benchmarks from CUDA.jl.

I'm not sure if I converted them properly. Some function seem to be missing Metal implementations (like reverse).

The final (and biggest) problem is how inconsistent these results have been. Simply rerunning the benchmarks on the same code gives some huge performance differences. There is always at least one benchmark that is >20% slower or faster. The benchmarks seem to be way more consistent on the runners. Only a few seem to have big variance.

See #418 (comment)

Todo:

• Implement actual CI benchmark solution from GemmKernels.jl
• Ensure Benchmarks are correct

[christiangnrd]

The culprit was whitespace... This is (one of the many reasons) why Julia > Python.

[christiangnrd]

@maleadt Do you know how I can set up the buildkite token on the Github Actions side?

[maleadt]

I can add a token. Which permissions do you need?

[maleadt]

Found https://github.com/actions-marketplace-validations/EnricoMi_download-buildkite-artifact-action?tab=readme-ov-file#configuration; I've added a token

[christiangnrd]

Do we want to save the minimum, the median, or the mean?

For testing now I'll do median since that's what https://github.com/LuxDL/LuxLib.jl/pull/128/files is doing.

[christiangnrd]

This is mostly ready. I assume benchmarks will be posted to new PRs after this is merged to master?

A few uncertainties:

• Which summary statistic do we want to report?
• Most benchmarks are working, but I'd like someone with a better understanding of lower-level gpu programming to make sure that I converted them from CUDA to Metal properly.
• volumerhs isn't running, can/should that be fixed or is it not relevant for Metal?
• Do we want to run the benchmarks even if CI fails like in CUDA.jl or should I get rid of those lines?

[maleadt]

I guess we'll only see this in action on subsequent PRs?

[maleadt]

Why do you need macOS 15 for the benchmarks?

[christiangnrd]

I initially wanted to see the performance impact of logging. Seems like now it's only enabled when actually used so might not be worth it.

[christiangnrd]

I guess we'll only see this in action on subsequent PRs?

Looking at the Lux PR I based this off of it seems like it.