reinit Main by clearing everything in it after building the sysimg

[vtjnash]

Continuing the discussion started in #5687.

We can't compile the sysimg twice because of old crap left in the Main module. This implements a empty!(::Module) functionality so that Main is left in an almost pristine state after redefining Base.

[vtjnash]

See also #1195 and #2385 which this almost addresses.

[timholy]

Today I gave a demonstration to my lab of some Julia code I've been writing; overall it went well, but everyone was surprised that it took 52 seconds just to load the code before I could start the demo (and then slow performance for the next ~15 seconds while various bits got compiled). Such slow startup times don't help debugging, either.

I'd love to vouch for this PR, but it's out of my league (heck, I don't even understand why the scheduler was related to precompiled packages). Anyone else feel up to it?

[StefanKarpinski]

The issue was that things had to happen in this order:

1. load sysimg
2. save precompiled image
3. start scheduler
4. load packages

In order to precompile packages, one wants to move 2 after 4 but 2 had to come before 3 and 4 had to come after 3. The solution was to remove 3 altogether, allowing 2 and 4 to be arbitrarily ordered.

[vtjnash]

I think the only person who can evaluate this is Jeff. Unfortunately, he can be a hard man to contact.

[timholy]

Well, I know he's been working hard on scheduler-related issues, and perhaps the dust hasn't even settled from that yet. I may be able to chip away at those load times a bit by some code-partitioning.

@StefanKarpinski, thanks for the explanation.

[timholy]

Rather than packing everything into the julia binary, is it at all practical to go with a more modular approach: have precompile("Gtk") write a binary file to ~/.julia/Gtk/deps that gets loaded (when available) by using Gtk?

[vtjnash]

That is much much harder. You can pack stuff in now -- just need a base sys0.ji image to build from which doesn't have the extra stuff

[timholy]

Good point, I'll do that locally.

[JeffBezanson]

I don't think mutating the module is a good way to do this. We probably want to do what we're doing with Base, just one level up. There could be a clear() function that basically re-runs the snippet in init.c that sets up Main. This is pretty close to what many people expect clear() to do, but it's hard to think of a name that does justice to the level of violence inflicted by this function.

[quinnj]

murder(), slaughter(), completely_wipe_out()....... :)

[StefanKarpinski]

EXTERMINATE!!!()

[staticfloat]

@StefanKarpinski I was scrolling down to write that suggestion, only to see that another great mind had already thought alike. ;)

[vtjnash]

Nice use of ! In the method name, Stefan

I tried that first Jeff, but ran into a lot of problems since it requires changing the binding of almost everything so that Base and Core gets reparented in the new Main.

[Keno]

How about Base.(( symbol(char(0x1f4a3)) ))(m::Module) (github won't allow me to type the actual character)

[StefanKarpinski]

There's something hilarious about providing this functionality but making people type unicode to use it.

[JeffBezanson]

I didn't think the parent module of Core mattered much --- what happens if we just switch jl_core_module->parent ? Base shouldn't need to be reparented; we can make a new Main module and put the new Base in the new Main.

[vtjnash]

It wasn't likely the Core module itself that caused issues. And I'm not saying it is completely impossible, but it seemed more fragile that way, so I switched approaches. Perhaps it was just that I tried to also replace Core, but it seemed that the jl_binding_t references from locations like Base.Main, were just easier to resolve if Base.Main didn't suddenly try to move while pretending not to move.

Perhaps instead we could architect:

module Main
    module Core end
    module Base end
    module This_Is_All_New
        module Main
            module Core end
            module Base end
            module MoreNewStuff_WhichCantSeeMain end
        end
    end
end
Main = Main.This_Is_All_New

[JeffBezanson]

I'm not sure what "pretending not to move" means. I was thinking

module Main
    module Core end
    module Base end
end

old_main = Main

module Main
    module Core
    module Base
        # new Base defined here
    end
end

[vtjnash]

I'm not sure what old_main = Main means there either. Are you suggesting always constructing a new Main module if the user passes the --build flag? Would it be better to just disallow passing -J with --build (and eliminate sys0.ji)?

[JeffBezanson]

old_main = Main is not really necessary there, it's just keeping a reference to the old Main.

I don't think we can eliminate the 2-stage bootstrap; it could have bad performance implications.

[JeffBezanson]

Superseded by #5821

[vtjnash]

Thanks