WIP: Integrate IPython setup on windows

[davidanthoff]

I tested this on a brand new windows system with nothing but julia installed:

Pkg.clone("https://github.com/davidanthoff/IJulia.jl.git","IJulia")
Pkg.checkout("IJulia", "integratedipysetup")
Pkg.build("IJulia")

using IJulia
notebook()

works on that system.

This PR lacks proper error handling etc, consider it more a proof of concept.

[vtjnash]

I think this approach makes a lot of sense. I was working last night on doing something very similar to this through WinRPM. I'm still working through some of the details (on how to organize the files).

I think you may want to also use virtualenv (I believe it'll make upgrading easier, since you can more easily just delete the virtualenv folder and leave the rest of the python installation -- but since this is a local private python installation anyways, that may not matter).

[StefanKarpinski]

These should start with / – this makes them absolute relative to the directory the .gitignore file is in.

[davidanthoff]

My take on the private vs user-installed python is this:

I think for IJulia it should always be a private install of python and ipython that should be tightly controlled by IJulia's build.jl, i.e. no one should install anything into that private python, it should just be hidden from users, i.e. users don't even have to know that it is there. From a users points of view it really is an implementation detail that IJulia happens to depend on python. The major benefit of that approach is that it should be much more robust, and hopefully issues like #86 will be a thing of the past.

For things like PyCall and PyPlot etc I think we should never provide a python installation, i.e. I think those should always rely on a user-installed python, with whatever packages the user wants to have etc.

From my point of view users that want to use IJulia should not even need to know that python is involved, but if someone wants to use a bridge to python it seems quite logical that he/she has to get a python first.

The main question is if that setup actually works: a private python for IJulia in combination with a user-installed python for PyCall. I think that should work, but we should make sure I'm right about that.

[vtjnash]

If calling using PyCall inside of the IPython REPL gets the "system" python (eg for integration with PyPlot ) I think the situation you describe is ideal (with a private ijulia python install)

[davidanthoff]

I tried this for PyPlot, and it seems to work: IPython notebook based on the private install, but then it uses my anaconda python for pyplot when I use pyplot from a notebook. I haven't used pycall, maybe someone more familiar with it could try that out?

[davidanthoff]

I also played around with some code that makes the separation between IPython and IJulia even more strong: instead of storing the julia profile in the default .ipython folder, we could use the --ipython-dir option and store the julia profile in a separate location, probably ideally somewhere in the IJulia package folder. I did not include that code in this PR because it would probably make sense to do that for all platforms, not just windows, if we decide to go that route.

What does get a bit more complicated is the command to launch IJulia. With the new option to use notebook() from within Julia that complexity could be completely hidden from the user, though.

[davidanthoff]

Actually, the question of how to start IJulia is a bit more complicated in any case if we were to go with a private python install. When a system python is used, the entry point for users is the system's ipython, which might be on the PATH etc. With the private python install this gets tricky. We certainly don't want to put the private python on the PATH. I see two options:

1. Have notebook() from within julia be the only user entry point. Easy and fool proof...
2. Create a ijulia.bat maybe in deps\scripts that starts the private ipython install with the correct parameters (either --profile=julia or --ipython-dir=<juliaprofiledir>) and then add only that deps\scripts directory to the PATH. That way the only thing pulled into the general path would be the ijulia.bat command.

Somehow options 2 strikes me as a bit of a hack, or rather it seems that if that is the goal, really there should be a standard julia way to add commands to the PATH, like the scripts directory in python. But that seems a whole separate discussion... So maybe just stick with option 1... Not sure though.

[Carreau]

IMHO Don't over-engineer that now. With IPython kernel spec and the IPython/Jupyter split you will soon not need to specify a profile dir anymore.

Envoyé de mon iPhone

Le 7 août 2014 à 20:09, David Anthoff [email protected] a écrit :

Actually, the question of how to start IJulia is a bit more complicated in any case if we were to go with a private python install. When a system python is used, the entry point for users is the system's ipython, which might be on the PATH etc. With the private python install this gets tricky. We certainly don't want to put the private python on the PATH. I see two options:

1. Have notebook() from within julia be the only user entry point. Easy and fool proof...
2. Create a ijulia.bat maybe in deps\scripts that starts the private ipython install with the correct parameters (either --profile=julia or --ipython-dir=) and then add only that deps\scripts directory to the PATH. That way the only thing pulled into the general path would be the ijulia.bat command.

Somehow options 2 strikes me as a bit of a hack, or rather it seems that if that is the goal, really there should be a standard julia way to add commands to the PATH, like the scripts directory in python. But that seems a whole separate discussion... So maybe just stick with option 1... Not sure though.

—
Reply to this email directly or view it on GitHub.

[StefanKarpinski]

I was going to mention Jupyter as well. My understanding is that in the future a single instance of Jupyter will be able to open Julia, Python, R, etc. notebooks. With that design it makes more sense not to separate the configurations.

[davidanthoff]

Hm, just looked at the whole kernel spec story and that seems to move in a very different direction than this PR... In particular, it looks like a whole lot of work to make one central IPython/Jupyter install work with different language kernels. It looks to me like that will do nothing to solve the problem we are trying to solve in this PR: to make Pkg.add("IJulia") be all users have to do to get a fully working IJulia.

I think from an IJulia point of view a system where there is as little integration into a system IPython/Jupyter as possible is much preferred because it requires less setup steps from users to get running, and there are way fewer integration points that can go wrong... I guess I just would prefer that from a user's point of view IJulia is a self contained package with no apparent dependencies on anything else on my system. It might happen to share code with IPython under the hood, but is there really a need for users to know about that?

[Carreau]

Not saying the contrary, I'm just saying the the separate profile stuff is too complicated as in a few month you will not want to start Ipython/Jupyter with a different profile.

The IJulia installing IPython/Jupyter if not present make a lot of sens.

Envoyé de mon iPhone

Le 7 août 2014 à 21:05, David Anthoff [email protected] a écrit :

Hm, just looked at the whole kernel spec story and that seems to move in a very different direction than this PR... In particular, it looks like a whole lot of work to make one central IPython/Jupyter install work with different language kernels. It looks to me like that will do nothing to solve the problem we are trying to solve in this PR: to make Pkg.add("IJulia") be all users have to do to get a fully working IJulia.

I think from an IJulia point of view a system where there is as little integration into a system IPython/Jupyter as possible is much preferred because it requires less setup steps from users to get running, and there are way fewer integration points that can go wrong... I guess I just would prefer that from a user's point of view IJulia is a self contained package with no apparent dependencies on anything else on my system. It might happen to share code with IPython under the hood, but is there really a need for users to know about that?

—
Reply to this email directly or view it on GitHub.

[davidanthoff]

The PR here is a VERY different model from IJulia "installing IPython/Jupyter if not present". The PR here will ALWAYS install a private python and ipython, hidden from everything else on the system, only for IJulia use. If there is an IPython already on the system, this PR will completely ignore it and have no integration with that. The python/ipython installed by this PR is not meant to be the new "default system/user ipython". It is a very different model from where Jupyter seems to be heading...

[davidanthoff]

Will the new Jupyter still have an option like --ipython-dir so that one can use it without it ever interacting with the normal locations for kernel registration?

I think maybe a good model for the Jupyter time frame would be this:

• IJulia installs a private python/ipython with a private kernel spec in the IJulia profile folder, i.e. everything is completely separate from any other Jupyter setup on the system. Calling notebook() from within julia will run that IJulia instance. It will have no support for other kernels etc, it is just a plain IJulia notebook, where users don't even know that this is based on IPython, that there might be other languages etc.
• IF build.jl detects a system wide jupyter installation, it will also create a kernel spec in the normal jupyter kernel registration places, so that julia appears in the selection of kernels if one starts jupyter etc.

[stevengj]

@davidanthoff, that makes sense to me.

[davidanthoff]

I need some help (@StefanKarpinski maybe?). This line

run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR="$pyinstalldir"`)

doesn't work when pyinstalldir contains spaces. What I want is that the following line gets run:

msiexec /passive /quiet /a somename.msi TARGETDIR="C:\path with space\py"

Instead, it looks like the whole TARGETDIR="C:\path with space\py" thing gets quotes, like this:

msiexec /passive /quiet /a somename.msi 'TARGETDIR=C:\path with space\py'

I read the running external programs section but couldn't figure out how to achieve that.

[StefanKarpinski]

In UNIX shell quoting, those are the same thing. I'm not super clear on how things get executed on Windows, but if it's through a system call, then it's likely similar. @Keno, @vtjnash?

[davidanthoff]

It seems msiexec.exe does treat them differently... People trying to call msiexec.exe from Powershell seem to have the same problem...

[Keno]

Try

run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=$(string('"',pyinstalldir,'"'))`)

[StefanKarpinski]

Ah, so msiexec needs the quotes in the argument? In that case quoting the quotes out to work:

julia> pythonmsifilename = "somename.msi"
"somename.msi"

julia> pyinstalldir = "C:\\path with space\\py"
"C:\\path with space\\py"

julia> `msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=\"$pyinstalldir\"`
`msiexec /passive /quiet /a somename.msi 'TARGETDIR="C:\path with space\py"'`

[davidanthoff]

Tried all of these, no luck. What I can get is this:

`msiexec /passive /quiet /a 'downloads\python-2.7.8.msi' 'TARGETDIR="C:\Users\anthoff\.julia\v0.3\IJulia\deps\usr\python 27"'`

The problem probably is that there are still single quotes around the whole TARGETDIR stuff, i.e. it should be like this instead:

`msiexec /passive /quiet /a 'downloads\python-2.7.8.msi' TARGETDIR="C:\Users\anthoff\.julia\v0.3\IJulia\deps\usr\python 27"`

But not really sure because I don't actually know what these single ' actually mean/do.

[Keno]

The single quotes don't get passed to the operating system, they just denote which arguments are a unit.

[Keno]

Alright, next attempt. Try this:

run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=$(split(string('"',pyinstalldir,'"'),' ')`)

[vtjnash]

The single quotes don't get passed to the operating system, they just denote which arguments are a unit.

except that this is windows, so they do get passed out to the operating system (although they will turn into double quotes " and other quotes will be properly escaped)

[davidanthoff]

@Keno, tried that (had to add another )), didn't work.

Given what @vtjnash just wrote, we need to get rid of the surrounding quotes around argument units for this to work.

[Keno]

@davidanthoff I made a mistake in that invocation, hold on

[vtjnash]

libuv quotes the arguments under the assumption that the recipient program will use the correct windows API function to unquote them. perhaps the authors of msiexec are unfamiliar with windows? anyways, there's a flag we should expose in the libuv api that lets you turn this off

[Keno]

@davidanthoff
Alright, last one before we should look at that flag:

args = split(string('"',pyinstalldir,'"'),' ')
run(`msiexec /passive /quiet /a $pythonmsifilename TARGETDIR=$(args[1]) $(args[2:end])`)

[davidanthoff]

@tkelman This is 32 bit Python, and that should work with both 32 and 64 bit Julia.

[davidanthoff]

@tkelman I tried the download_cmd command from BinDeps, but I can't get it to work. I'm using

download_cmd("https://sourceforge.net/projects/minimalportablepython/files/python-3.4.2.zip", "python.zip")

But that doesn't download anything. Any pointers what I should be doing instead?

[tkelman]

BinDeps.download_cmd should return a command object - most likely curl -o python.zip -L https://sourceforge.net/projects/minimalportablepython/files/python-3.4.2.zip with the current version of BinDeps, so you'll need to call run(BinDeps.download_cmd(...)) on it.

[tkelman]

It definitely looks like the download from Base, which uses URLDownloadToFileW on Windows, just isn't following redirects. Might be worth raising an issue with base Julia to figure out how to make it do that.

[davidanthoff]

Alright, it now downloads the python binaries from sourceforge, thanks for the help! I think this is ready for testing by others.

[tkelman]

stylistically I think @windows_only begin would look better here

[davidanthoff]

Done

[tkelman]

It looks like this works on 32 bit Julia, I'll try 64 bit later. Whatever is happening between Successfully installed pip-6.0.8 setuptools-12.1 and the first [ProfileCreate] really badly needs some feedback to the user that it's doing something though, it takes a while and looks completely dead. Is -qqq the absolute quietest setting for pip? Can we change that to something that isn't completely silent?

And this probably doesn't need to download, extract, and set up python again if you call Pkg.build("IJulia") a second time and the desired version has already been downloaded, extracted, and set up properly... Would be good if there were an early success detection, "looks like things are already set up and working here" with perhaps a tag file or something that you could manually delete if you needed to force a complete reinstall.

[tkelman]

do we need this separately? don't the latest versions of python have ensurepip included?

[davidanthoff]

Good point, I now use ensurepip

[davidanthoff]

Is -qqq the absolute quietest setting for pip? Can we change that to something that isn't completely silent?

Done

And this probably doesn't need to download, extract, and set up python again if you call Pkg.build("IJulia") a second time and the desired version has already been downloaded, extracted, and set up properly... Would be good if there were an early success detection, "looks like things are already set up and working here" with perhaps a tag file or something that you could manually delete if you needed to force a complete reinstall.

I added support for this. Essentially if a previous build.jl call was successful, then the next call will only do upgrade commands for pip and ipython for the existing private python installation. I still recreate the private ipython profile that lives in usr/.ijulia on every build.jl call. This is mainly to avoid another update mechanism we might need, namely one where the profile is updated.

[tkelman]

could these be rm(downloadsdir, recursive=true) instead of cmd /C RD ?

[davidanthoff]

Ha, I didn't find the recursive option when I originally looked. Now changed.

[tkelman]

This is looking really good. I'll make a point to test this in combination with Pkg.checkout("BinDeps") since I just merged that PR that makes the default download_cmd use powershell on Windows for better results when behind proxies.

[davidanthoff]

One question is still whether we should hardcode the IPython version that is installed or just always get the latest. Right now it is hard coded. I'm worried that if we just get the latest version instead things might break. E.g., as soon as 3.0 is released, it would be used, and who knows whether we might need to adjust things. So I would vote to keep things as they are with a hardcoded version, but would like to hear other input as well. It does mean we would have to keep this line updated.

[tkelman]

Markupsafe and tornado are invoking MSVC when they install on my machine. I hope they would continue to function correctly when MSVC is not installed...

[davidanthoff]

I've tested this on a fresh Windows XP install throughout that certainly has no MSVC installed, and things seem to work. For tornado I know that it will try to compile things for speed, but if that fails for any reason it will still work. I assume (hope) Markupsafe follows a similar strategy.

[tkelman]

Cool, I guess if the performance would suffer noticeably from not having the C extensions compiled, we can bug (or ask nicely, whatever) the developers of those packages to put up compiled binary wheels and that would resolve the performance difference for users who don't have MSVC installed?

[davidanthoff]

Yes, I think that would work. I tried to test on XP again yesterday, but the julia ZMQ package failed to build. I'm traveling this week, but will try again on Fri.

[tkelman]

the julia ZMQ package failed to build

Presumably that's the same underlying issue as JuliaInterop/ZMQ.jl#69 and should go away after a restart of Julia. I wish people would write down error messages...

[tkelman]

I think all this might need is a rebase and an eventual squash? The ZMQ install issue is fixed on nightlies, no restarting Julia necessary there.

[davidanthoff]

There was a lot of change to build.jl on master this week so I very much doubt I can just merge master right now, will probably require a bit of work to sort that out.

[stevengj]

Closing in this since the Conda alternative has been merged.

[davidanthoff]

Excellent news!