Support run-time interpreter configuration?

[tkf]

(I'm asking this before seriously reading PyCall.jl since @stevengj or someone else may/can answer it with a simple No. 😁)

Using PyJulia with Julia 1.0 is now very hard especially in Conda and Ubuntu JuliaPy/pyjulia#185. PyJulia needs the "hack" using Base.DEPOT_PATH JuliaPy/pyjulia#173 or waits for julia core support JuliaLang/julia#28518.

If PyCall.jl can support run-time configuration of Python interpreter, it will be much easier since PyJulia can just reuse the same precompilation cache. If it is possible, it is much more appealing to me than implementing a bulky Base.DEPOT_PATH hack which would be obsolete once julia core provides more straightforward support.

In PyCall.jl, I see there are many const and macro that depends on the Python interpreter. But is run-time simple if branch that costly compared to Python interpreter? Also isn't it possible to embed (a reference to) such information in (say) PyObject so that no branching would happen after initialization?

Or are there other reasons than rum-time speed? Reducing JIT time or something? (Just guessing from #167) Would parametrizing types with Python major version and precompileing for both versions solve it? (Or simply drop Python 2 support?)

[stevengj]

The main reason I switched to build-time configuration of libpython was to reduce load time. I used to identify libpython at runtime, and it was way slower to do using PyCall.

[tkf]

Thanks. Reading #167, it looks like the reduction of the time for using PyCall is mainly coming from precompiling the functions used in __init__. Is it correct? If so, wouldn't it be solved if you precompile all the versions you need?

I'm guessing there are two ways to do it: (1) parametrizing the types and functions by Python versions (and also static or not?); (2) creating sub-modules (e.g., PyCall.Py2.Static, PyCall.Py3.Dynamic) which does include("startup.jl") etc. to statically generate what is required.

[stevengj]

It seems like it would be better to just set up a separate depot or similar for pyjulia.

[tkf]

Yeah, I think you are right. My suggestions would be solved by julia core in a much more general way if it implements package options. Implementing it manually in PyCall.jl seems to be too much work for a premature optimization.