fix for "python -c 'import nest'" readline issues with Anaconda Python

[espenhgn]

importing nest with Anaconda Python may be broken for some due to some mismatch between the system and Anaconda-provided readline library files, such that importing nest in a running python interpreter would work, while "python -c 'import nest'" would not. Importing readline before pynestkernel takes care of that.

[apeyser]

Does the pynest code assume that readline gets imported? If not, we shouldn't create a forced dependency on readline. Why not use python -c 'import readline, nest'?

[JanneM]

It should perhaps depend on whether you use --with-readline during
configuration or not.

2016-01-19 20:45 GMT+09:00 Alexander Peyser [email protected]:

Does the pynest code assume that readline gets imported? If not, we
shouldn't create a forced dependency on readline. Why not use python -c
'import readline, nest'?

—
Reply to this email directly or view it on GitHub
#209 (comment).

Jan Morén
http://sites.google.com/site/janmoren/
http://janneinosaka.blogspot.com

[apeyser]

@JanneM : right. configure.ac currently has a --with/out-readline option, and on bgq it disables readline entirely. Doing a sed over an import file would seem overkill, as well as having a 'import nest_with_readline' option. There's /GNUreadline in systemdict, but that's available after the pynestkernel is imported, and anything more is overkill.

@espenhgn : so I go back to asking, why not just do 'import readline, nest' manually for the Anaconda Python case? Or using your own anaconda-python script that does that?

side note: Anaconda Python -- what an awful, awful name. Why name something intentionally ambiguously?

[espenhgn]

@apeyser I don't think the average Nest user would quickly understand that importing readline before nest would fix that ImportError. Note that import nest in a py-script started from the command line crashes similarly, and why should people have to adapt their codes to run with a particular python environment? It's not like the init.py file do not already contain several workarounds for other issues already.

Another less neat alternative could be to put the pynestkernel import within a try-except ImportError statement.

[apeyser]

@espenhgn Does this issue appear only with Anaconda Python? Do we know what's going on here?

[espenhgn]

@apeyser Tested only with anaconda so far, could occur with e.g., Enthought. @tammoippen tried figuring out what libreadline is used when, but the conclusion was not clear to me. Traceback:

espen@espen-VirtualBox:~$ python -c "import nest"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/espen/.local/nest/lib/python2.7/site-packages/nest/__init__.py", line 52, in <module>
    from . import pynestkernel as _kernel
ImportError: /home/espen/anaconda2/bin/../lib/libreadline.so.6: undefined symbol: PC
´´´

[apeyser]

@espenhgn
The following issue seems similar: ContinuumIO/anaconda-issues#152
So anaconda has ongoing issues with library linking because (I think) it tries to replace dynamically linked libraries with it's own, that then have secondary issues with links.

Anaconda seems to be a "bad thing". Who is using this kludge and for what? What does this add over pip and virtualenv's?

[espenhgn]

@apeyser I'm not gonna survey who's using anaconda on their own machines, but at least in our lab everyone using Python on the Blaustein cluster is. This import issue do not occur there though. The main advantage I guess is that users won't have to deal with all dependencies e.g., for pip-installing scipy etc.

[apdavison]

@apeyser The main advantage of Anaconda is that it provides pre-built binaries for many Python packages. This saves a lot of time, compared to pip, when installing large packages such as Numpy, Scipy, Matplotlib. It is also a huge gain on OS X and Windows, where compiling such packages is sometimes tricky. (This advantage may disappear if/when Python wheels take off).

[apeyser]

@espenhgn @apdavison

Thanks for the info. So the problem here is essentially that people are putting pre-built binaries from one set-up on to their local distribution and there will be inevitable clashes in the dynamic libraries.

So the solutions are:

• We try to hack around this by adding imports to get the correct linking order
• We complain to anaconda to improve their system to handle linking issues
• We push it back on to the users to properly compile nest using the right link options to fit anaconda
• Are there anaconda recipes or package specs that could add the right link options for users to build nest under anaconda?

[apeyser]

I think the final solution to this, is that this is a problem of improper linking using anaconda. The build is picking up a different environment than the run.
The solution is thus to build under anaconda, using an anaconda recipe, which would also allow the inclusion of releases in anaconda.
So, someone using anaconda could put together a proper build and share it...

If I'm missing something, please let me know. Otherwise, I'll close the issue until someone fulfills request #210, which imho is the correct solution.

[espenhgn]

@tammoippen suggested to import readline only if 'linux' in sys.platform and 'Anaconda' in sys.version.

[apeyser]

Much less ugly... I wonder whether there should be a Makefile 'platform hacks' module created that gets loaded here...

but I wonder, if NEST in Anaconda is so popular, why hasn't there been a volunteer to make a linux anaconda recipe?

[apdavison]

I think Anaconda is more popular with OS X and Windows users; it is not needed so much on Linux distros as they have fully-integrated package managers, and most open source tools are developed primarily for Linux and so build more reliably.

[jougs]

During the NEST Core meeting on 15th of February 2016, we decided to merge this as we can't expect Anaconda to fix the problem in a general way and a Anaconda recipe for NEST is not in sight.