Debugging functionality

[itsdfish]

Develop a debug mode or seperate set of functions that gives the user detailed access to chains and error messages.

[itsdfish]

Hi Rob-

After giving this some thought, I think the best approach might be to simply allow chains to be collected. More detailed debugging, such as evaluating proposals and gradients etc., would require a different solution that is beyond the scope of this project. The solution might be sampler specific or perhaps there might be a way to develop a package that provides a common debugging interface to access and analyze the internal workings of the sampler.

What I have in mind for this project is something like the following

#Initialize NamedTuple to collect chains
chains = initChains(samplers)
#set savechain = true (default false)
options = (Nsamples=2000,Nadapt=1000,delta=.8,Nd=Nd,savechain=true)

#perform the benchmark
results = benchmark(samplers,GaussianGen,Nreps,chains;options...)

chains is a NamedTuple that separates chains by sampler:

chains
  (CmdStanNUTS = Chains[], AHMCNUTS = Chains[], DHMCNUTS = Chains[])

You can also use a single sampler rather than a tuple. Would this be useful?

[goedman]

A good approach which covers 80%.

For in-depth debugging purposes, I wonder if we should combine the sampler chains. Looking at what you send to Tamas is a good example. When we ask for help any additional layer might be more of a hindrance than help. I can certainly live with separate sampler scripts in each Example directory for that purpose which are not normally part of the MCMCBenchmarks workflow..

Once debugged, the scripts could import the models from the Models directory.

[itsdfish]

Just to clarify, are you suggesting that we use separate model files for each sampler? For example, CmdStanGaussian.jl would contain only the Stan code for the Gaussian model? I think this would be a good idea when asking for help from others. I agree that the other code is useful for benchmarking but is a hindrance when we want a MWE.

[itsdfish]

P.S. I think Turing might have some native debugging features, but they are not documented.

[goedman]

Yes, exactly, only when helpful and/or needed we could leave script files like CmdStanGaussian.jl behind.

[itsdfish]

On second thought, I might need more clarification. There might be some points of confusion. Do you want to have separate files for each sampler configuration in the Models folder? For example, Models/Gaussian/CmdGaussian.jl, Models/Gaussian/AHMCGaussian.jl would be some of the files. So the files would be integrated following development (or else we could define a function to include all).

Or do you want separate scripts in the Examples folder, e.g. Examples/Gaussian/CmdStan_Gaussian_Example.jl etc. ? The later case is quite simple because benchmark and pbenchmark accepts a sampler object or a tuple of sampler objects. Sorry for the confusion.

[goedman]

The latter, simple one. In the Example folder we could e.g. have Examples/Gaussian/CmdStan_Gaussian_Example.jl and Examples/Gaussian/DHMC_Gaussian_Example.jl in addition to the benchmark version (Gaussian_Example.jl)

Once all is fine, we could use:

using MCMCBenchmarks: CmdStanGaussian

in above CmdStan_Gaussian_Example.jl script to import the Stan model, etc.

I choose to call the scripts something like Gaussian_CmdStan.jl or LBA_dHMC.jl but that is a detail.

Does that help?

[itsdfish]

Yes. I think I understand now. Thanks. I'll add this today or tomorrow.

[itsdfish]

Hi Rob-

I explored a few alternative configurations, but I did not find anything that did not seem to overcomplicate things. For the purpose of development, I have been creating a temp.jl file that calls relevant functions. For example:

using MCMCBenchmarks

#Load all of the relevant functions 
include("../../Models/Gaussian/Gaussian_Models.jl")

ProjDir = @__DIR__
cd(ProjDir)

samplers=(
  CmdStanNUTS(CmdStanConfig,ProjDir),
  AHMCNUTS(AHMCGaussian,AHMCconfig),
  DHMCNUTS(sampleDHMC,2000)
  )
  
data = GaussianGen(;Nd=10)

#Select a sampler for testing. 
chain = runSampler(samplers[1],data)

If I change the model definition, I re-run include("../../Models/Gaussian/Gaussian_Models.jl"). If I want to run a single sampler through the entire benchmark, I execute something like this:

options = (Nsamples=2000,Nadapt=1000,delta=.8,Nd=Nd)

#perform the benchmark with first sampler (CmdStan) 
results = benchmark(samplers[1],GaussianGen,Nreps;options...)

This seemed to work for development. One of the problems that I encountered with

using MCMCBenchmarks: CmdStanGaussian

is that I had to import all of the models (LBA, Gaussian etc.) into the MCMCBenchmarks module and import the variables I need into Main. This seems to slow down loading even further. Feel free to change things or use multiple files if that helps your workflow.

I'll go ahead and close this issue because I pushed the changes for the original functionality. If you want to look at the chains for development/debugging, you can pass an optional NamedTuple:

#Initialize NamedTuple to collect chains
chains = initChains(samplers)
#set savechain = true (default false)
options = (Nsamples=2000,Nadapt=1000,delta=.8,Nd=Nd,savechain=true)

#perform the benchmark
results = benchmark(samplers,GaussianGen,Nreps,chains;options...)