switch to incremental compilation

[ssagaert]

Only compile changed code instead of all code of a file when running that file. This will greatly improve startup time.

Even when AOT compilation becomes available this will also be useful for that scenario.

[JeffBezanson]

This issue is too vague. Is it about the startup time of the julia prompt, or the performance of loading packages (#4373), or repeatedly includeing files during development? When including a file, I'm not sure it's reasonable or necessary to attempt a fine-grained diff to avoid reloading parts of it. What granularity do you have in mind? Do you have a proposal for how this could work? Maybe an example?

See also #8745, #265

[ssagaert]

it's about doing "julia mycode.jl". Mycode.jl is one module (rather large, hundreds to thousands of lines) which uses around 5 libraries via "using".

I'm not sure where most of the compilation time is spent: in compiling mycode.jl or in compiling the libraries it uses but when you change only one line or a few lines you're in for several seconds of compilation time. That's too slow for RAD.

I'm not sure if it's even technically possible but ideally incremental compilation could be done at a submodule level like function or type scope. If that's not possible than only recompile changed files (since a module can be comprised of multiple files and hence file scope <= module scope).

In the end I'l like to get to the same compilation time of something like java + eclipse where when you hit save (ctrl +s) and the recompilation is almost instantenously.

[timholy]

Once packages have been loaded and compiled, they don't need recompilation. So it's all in your module.

Juno provides some support for updating functions in previously-defined modules.

[nalimilan]

@timholy But he appears to be restarting julia at every run, in which case all non-Base modules are recompiled.

[timholy]

Oh, I see---thanks @nalimilan. @ssagaert, you'll be much happier if you use the REPL for your development. You can just reload your module. (Or use Juno, of course.) Or, help contribute to getting #8745 merged.

I'm going to close this as not being immediately actionable, and better covered through open pull requests and issues.

[JeffBezanson]

Yes, in the case of running julia mycode.jl where mycode.jl uses various packages, I think the best solution is to have the packages precompiled.

[ericjang]

I'm just a Julia user and not a dev, so this is easy for me to say (pardon my audacity):

In other interpreted environments (Python, LuaJIT, Node), it's quite a common task to launch a script from a bash shell, or launch processes from another language in some kind of batch-job scheduler.

Telling the user to use the REPL for development is a suitable workaround, but I don't feel like that is a good solution in the long term. It reminds me of JVM start time with Java programs, and these overheads are noticeable. One could launch a JVM/Julia interpreter process and pipe in inputs via STDIN, but this is a lot harder than calling subprocess.check_output(['julia','myscript.jl'])

Are there ongoing efforts, or at least plans, for static compilation of modules? I can't seem to find the relevant issues in the tracker. I may or may not have the relevant expertise but maybe I can give it a shot if I know more about the designs for such a mechanism.

[yuyichao]

@ericjang #8745

[malmaud]

Long story short, you can check out the master branch now and type Base.compile(:packagename) to statically compile a package.