[examples workflow:] Why instantiate only and not update

[theogf]

ApproximateGPs.jl/docs/make.jl

Line 10 in 5e475e3

let script = "using Pkg; Pkg.activate(ARGS[1]); Pkg.instantiate()"

Can someone remind the reasoning of only instantiating and not updating the environment when running the examples? This is very tedious to do manually...

[devmotion]

To make them reproducible, otherwise different versions than the ones specified in the Manifest.toml would be used and it would be unclear what version exactly was used to run these examples.

[devmotion]

One could remove the Manifest.toml files and only add a Project.toml with a compat section (to avoid that new breaking releases are picked up automatically) but additionally also print the package versions that were used to run the examples at the end of each example (e.g., printing the output of ] st --manifest or ] st; possibly the Julia version as well). It would make it more difficult to reproduce the results though since one would have to install the same package versions manually instead of instantiating a given project environment.

[st--]

I agree, I also find it really annoying to have to update all the Manifests one by one each time I change dependencies. I'm still longing for a Makefile/make.jl that automates this step. But also I'd be in favour of just removing them (and relying on compat entries instead to avoid random breakage). @devmotion Why would this make it more difficult to reproduce the results? The Pkg resolver should work the same on CI and locally, so it should give the same results. If it ever becomes an actual issue we could also revert to including the Manifests in the repo. But so far I've mainly lost a bunch of time on having them included, and not seen any benefit yet.

[st--]

Project.toml with a compat section (to avoid that new breaking releases are picked up automatically)

that's already in place actually, no?

[devmotion]

Why would this make it more difficult to reproduce the results? The Pkg resolver should work the same on CI and locally, so it should give the same results.

It works the same (well, apart from os and architecture specific JLLs) but it doesn't work the same if you rerun it some hours/days/... later even if you use exactly the same Julia version. Even if you pin all direct dependencies with equality constraints, the resolver can and will install different versions of the indirect dependencies since it just installs the latest compatible version of these. Eg, even if you pin KernelFunctions a new unbreaking release of Distances can change the results or even accidentally break your example (as we saw recently).

TL;DR: A Project.toml file only makes the examples not reproducible - if you don't pin both direct and indirect dependencies or install them manually (if they are only printed). In both cases it seems easier to work with a Manifest.toml file that ensures reproducibility.

[devmotion]

So I think either one does not care if users can rerun and reproduce the examples or tries to solve the tooling problem. Eg, AFAIK one could adapt some of the existing Manifest update github actions and run them, manually and/or automatically.

[st--]

@theogf is this resolved by #97 now?