judge does not work when the commits have different dependency versions

[shashi]

I'm trying to run a benchmark against master vs PR where the PR bumps a dependency version one breaking level up. But it fails to checkout the older version on the master branch. I find it confusing that it says " └─restricted to versions 0.15.2 by an explicit requirement — no versions left" for the dependency, but then I never quite understand Pkg errors.

Here is the output: https://github.com/JuliaSymbolics/Symbolics.jl/pull/365/checks?check_run_id=3495273188#step:7:268

I also tried having using Pkg; Pkg.resolve() at the top of benchmark.jl, but get the same error.

[palday]

what about

judge(Symbolics, 
      BenchmarkConfig(;id="origin/HEAD", juliacmd=`julia -O3 -e'using Pkg; Pkg.update()'`);
      verbose=false)

?