Warning about benchmarktools

[evelyne-ringoot]

Related to #683, proposing to add a warning regarding the use of benchmarktools

[luraess]

Thanks for the addition. It is indeed valuable to have some notes about potential limitations using e.g. BenchmarkTools with AMDGPU. One could however relax slightly the formulation as BenchmarkTools may slow down code in some cases but this may not be generalisable to all cases. Also, it would be valuable to have a self contained example and potentially a direct comparison good vs bad perf such that a user could directly run the suggested code snippet. This could and should be further adapted depending on the outcome of investigating #683 .

[vchuravy]

In addition, the same performance issues are observed when using benchmarktools, which relies on copies.

I am surprised by this statement. BenchmarkTools does not introduce copies

[evelyne-ringoot]

In addition, the same performance issues are observed when using benchmarktools, which relies on copies.

I am surprised by this statement. BenchmarkTools does not introduce copies

My bad, I think I might've put something on benchmarktools that is not actually its fault. I thought I saw weird performance results with benchmarktoold before, but I cannot manage to reproduce them right now so closing this for now

[luraess]

In some case, in the "past", I also had weird or failing AMDGPU test when using BenchmarkTools which forced me to things in a more manual fashion. I did no longer bump into such failing configs recently though...