Speed up Stan compilation

[itsdfish]

You may have noticed that compiling a Stan model for each processor is slow in the parallel version. Each model is compiled one at a time. Rather than making this a parallel process, I was wondering whether the compiled executable and .hpp files can be copied like we already do with the Stan model file? One complication I noticed is that the .hpp file uses the model name. If this is the only complication, it might be easy to do some sort of find and replace. Are there other problems aside from that?

[goedman]

I believe there are 2 conditions:

1. The contents of the .stan file needs to be identical with the model string, and
2. ‘make’ needs to think all is ok, thus time stamps.

Not behind my laptop right now to check that.

[goedman]

Of course this is all on a single machine as we don’t store ‘tmp’ on github.

[itsdfish]

Thanks Rob. If that turns out to be laborious or problematic in some way, we can keep it as is.

[goedman]

Once back in CO I would like to look into some changes to CmdStan that would have helped out for our purposes and minimize compilations.

One other thing we could do is to write all results go to a dir that is not on Github, like tmp, but do store selected versions and time stamped results. That way, if we run a quick test, it does not override the results from longer runs that we would like to keep. Will require some manual intervention (or/and a new release of the package). But once registered, that is simple.

[itsdfish]

Sounds like a plan. It would be nice if we could speed that up or have the ability to write to the hard drive like rstan.

I did some benchmarking of the load times. It turns out that a significant amount of time is spent loading Julia packages and compiling. On my system, executing the code from the top of the script to @everywhere Turing.turnprogress(false) requires 118 seconds, nearly two minutes. I hope some progress is made on load times. By contrast, compiling Stan takes about 46 seconds. While that is long, it turns out not to be the biggest problem. There might not be much we can do to speed it up.

[goedman]

Yes, I am also very concerned about these very long startup times. I expect a lot of the discussions around PackageCompiler (and maybe containers) are all at least partially a consequence of this.

Just filed the registration issue.

[itsdfish]

I'll go ahead and close this since we should run multiple chains to have valid rhat estimates.