Benchmarking with interpolated type gives different results #124
Comments
jrevels
changed the title
|
This behavior is intentional, but it seems to be too poorly documented in BenchmarkTools, because a lot of people run into this. In the former example, you're asking for the runtime of a function For example, the former might be considerably slower due to the compiler's choice of type argument specialization, i.e. |
|
Thanks for your answer. I understand the problem now, but was not aware of it when writing the random benchmarks. I had intended to measure the the second form ( |
This fixes part of JuliaCI#124. E.g. When wanting to benchmark `rand(Int)` and `rand(UInt)`, the following loop won't measure what is expected: `for T=(Int, UInt); @benchmarkable rand($T); end` In order to benchmark the equivalent of `@benchmarkable rand(Int); @benchmarkable rand(UInt)`, we "value-fy" the types with `Val`.
This fixes part of JuliaCI#124. E.g. When wanting to benchmark `rand(Int)` and `rand(UInt)`, the following loop won't measure what is expected: `for T=(Int, UInt); @benchmarkable rand($T); end` In order to benchmark the equivalent of `@benchmarkable rand(Int); @benchmarkable rand(UInt)`, we "value-fy" the types with `Val`.
This fixes part of JuliaCI#124. E.g. When wanting to benchmark `rand(Int)` and `rand(UInt)`, the following loop won't measure what is expected: `for T=(Int, UInt); @benchmarkable rand($T); end` In order to benchmark the equivalent of `@benchmarkable rand(Int); @benchmarkable rand(UInt)`, we "value-fy" the types with `Val`.
This fixes part of JuliaCI#124. E.g. When wanting to benchmark `rand(Int)` and `rand(UInt)`, the following loop won't measure what is expected: `for T=(Int, UInt); @benchmarkable rand($T); end` In order to benchmark the equivalent of `@benchmarkable rand(Int); @benchmarkable rand(UInt)`, we "value-fy" the types with `Val`.
This fixes part of JuliaCI#124. E.g. When wanting to benchmark `rand(Int)` and `rand(UInt)`, the following loop won't measure what is expected: `for T=(Int, UInt); @benchmarkable rand($T); end` In order to benchmark the equivalent of `@benchmarkable rand(Int); @benchmarkable rand(UInt)`, we "value-fy" the types with `Val`.
This fixes part of #124. E.g. When wanting to benchmark `rand(Int)` and `rand(UInt)`, the following loop won't measure what is expected: `for T=(Int, UInt); @benchmarkable rand($T); end` In order to benchmark the equivalent of `@benchmarkable rand(Int); @benchmarkable rand(UInt)`, we "value-fy" the types with `Val`.
Sorry for the unclear title. The problem can be summarized as follows:
T = UInt; run(tune!(@benchmarkable rand($T)))gives a very over-estimated time compared torun(tune!(@benchmarkable rand(UInt)). While preparing a PR against julia/master, the RandomBenchmarks showed a lot of regressions because of this (in this case,Tis set in a loop), even though the performance is not degraded when running individual benchmarks using the second form (i.e. usingUIntdirectly). I tried solving this by using some incantation ofeval, with no success.My last try was something like(edit: it doesn't)T=UInt; RD=RandomDevice(); g[...] = eval(@benchmarkable rand(Expr(:$, RD), $T)(hereRDmust not be interpolated byeval, only by@benchmarkable). I'm not sure whether this works as intended, but it's ugly, sowanted to discuss this problem here before working more on this.The text was updated successfully, but these errors were encountered: