loading EasyBuild module results in Python loading system modules

[dominikborkowski]

I noticed this issue while trying to install a piece of software that listed 'numpy' as its dependency. When 'EasyBuild' module is loaded, Python falls back onto the numpy provided by the OS, rather than the one from either numpy easybuild, or any of the python installs created by EasyBuild. Order of loading the modules does not matter. Presumably this affects other modules than just numpy, for example GLib compilation fails, because of the pyxml OS package that comes with SLES11 SP3.

Here's an example of how this manifests itself:

dom@sfxlogin3:~> which python
/usr/bin/python
dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/usr/lib64/python2.6/site-packages/numpy/core/include

dom@sfxlogin3:~> module load Python/2.7.9-goolf-1.7.20
dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/apps/easybuild/software/shadowfax-sandy_bridge/Python/2.7.9-goolf-1.7.20/lib/python2.7/site-packages/numpy/core/include

dom@sfxlogin3:~> module load EasyBuild
dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/usr/lib64/python2.6/site-packages/numpy/core/include

Now the reverse load:

dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/usr/lib64/python2.6/site-packages/numpy/core/include

dom@sfxlogin3:~> module load EasyBuild
dom@sfxlogin3:~> which python
/usr/bin/python
dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/usr/lib64/python2.6/site-packages/numpy/core/include

dom@sfxlogin3:~> module load Python/2.7.9-goolf-1.7.20
dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/usr/lib64/python2.6/site-packages/numpy/core/include

dom@sfxlogin3:~> module load numpy/1.8.2-goolf-1.7.20-Python-2.7.9
dom@sfxlogin3:~> python -c "import numpy; print(numpy.get_include())"
/usr/lib64/python2.6/site-packages/numpy/core/include

[wpoely86]

This is an issues due to setuptools.pth (or maybe easy-install.pth) that adds the system python folder.

[boegel]

To be more specific: if the EasyBuild module was installed with the setuptools available on the OS, the setuptools.pth includes the path to where Python packages are installed on the OS.

This (apparently) results in the OS Python package location being included in the Python search path whenever the EasyBuild module is loaded, which is clearly wrong...

Two options:

• remove the setuptools.pth after installing the EasyBuild module
• installing our own setuptools in the EasyBuild installation prefix (which should avoid having the OS Python pkg location being used in setuptools.pth

This problem is not specific to the EasyBuild module, but any Python tool that is installed using the system Python (see the allow_system_deps = [('Python', SYS_PYTHON_VERSION)] line in the EasyBuild easyconfig files).

[dominikborkowski]

Verified, and sure enough, by removing
/usr/lib64/python2.6/site-packages

from:
EasyBuild/2.3.0/lib64/python2.6/site-packages/setuptools.pth
EasyBuild/2.3.0/lib64/python2.6/site-packages/easy-install.pth

I'm able to compile numpy-1.9.2 (before it was segfaulting) or compile graph-tool. I'll test it with GLib, but I'm already guessing it will compile just fine.

Moving forward, is this something we should address on the system, or something that EasyBuild could fix?

Thanks!

[boegel]

@dominikborkowski: you had /usr in both .pth files?

We should try and figure out a fix in EasyBuild for this, there's nothing wrong on your system (unless you consider having setuptools installed system-wide wrong ;-)).

[dominikborkowski]

Indeed, that path appeared in both of those files. Interestingly enough, it shows up only under EasyBuild-2.3.0, while EasyBuild 2.2.0 didn't have it. Presumably difference there being the fact that EasyBuild 2.2.0 was the vanilla install from bootstrap, while 2.3.0 was installed via EasyBuild 2.2.0

• 2.2.0:

cat EasyBuild/2.2.0/lib64/python2.6/site-packages/easy-install.pth
import sys; sys.__plen = len(sys.path)
./vsc_base-2.2.3-py2.6.egg
./easybuild_framework-2.2.0-py2.6.egg
./easybuild_easyblocks-2.2.0-py2.6.egg
./easybuild_easyconfigs-2.2.0-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

> cat EasyBuild/2.2.0/lib64/python2.6/site-packages/setuptools.pth
/tmp/tmpkNhSgi/lib64/python2.6/site-packages/distribute-0.6.34-py2.6.egg

• 2.3.0 (original values, before implementing this workaround)

> cat EasyBuild/2.3.0/lib64/python2.6/site-packages/setuptools.pth
/usr/lib64/python2.6/site-packages

> cat EasyBuild/2.3.0/lib64/python2.6/site-packages/easy-install.pth
import sys; sys.__plen = len(sys.path)
./vsc_base-2.2.4-py2.6.egg
/usr/lib64/python2.6/site-packages
./easybuild_framework-2.3.0-py2.6.egg
./easybuild_easyblocks-2.3.0-py2.6.egg
./easybuild_easyconfigs-2.3.0-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

[dominikborkowski]

In case if this information may be useful to others. This is on a SLES 11 SP3 system with these python packages installed:

> rpm -qa python*
python-gobject2-2.20.0-1.1.56
python-gtk-2.16.0-1.1.251
python-xml-2.6.9-0.33.1
python-gnome-menus-2.28.0.1-1.1.213
python-numeric-24.2-195.17
python-argparse-1.2.1-2.5.65
python-pip-1.4.1-2.2
python-smbios-2.2.27-4.4.83.sles11
python-notify-0.1.1-5.1.126
python-setuptools-0.6c8-10.19.6.1
python-satsolver-0.44.5-0.5.148
python-2.6.9-0.33.1
python-curses-2.6.9-0.33.1
python-tk-2.6.9-0.33.1
python-orbit-2.14.3-137.26
python-base-2.6.9-0.33.1
python-gtksourceview-2.4.0-1.32
python-numpy-1.3.0-1.1.42
python-cairo-1.6.4-1.26
python-devel-2.6.9-0.33.1
python-gnome-2.22.0-67.32
python-gnome-32bit-2.22.0-67.32
python-utmp-0.7-4.2

[dominikborkowski]

Another data point, which may or may not be useful in figuring out why EasyBuild//lib/python2.6/site-packages/setuptools.pth points to system python's site-packages:

When installed via bootstrap, this is not an issue. Only when EasyBuild is used to install another EasyBuild (newer or older) the resulting EasyBuild has the 'broken' setuptools.pth. More interesting, this happens also when the source EasyBuild has the workaround in place (removed content of setuptools.pth). I've tried this on various combinations using EasyBuild 2.3.0, 2.2.0, 2.1.1, 2.0.0.

[tvandera]

I can confirm this on a Ubuntu 14.04 with an Easybuild 2.3.0 which was installed using EasyBuild 2.1.1.
Patching the .pth files 'fixed' the problem

[pescobar]

I also found another issue related to this. I have nose library in the extensions list in the python easyconfig:

    ('nose', '1.1.2', {
        'source_urls': ['http://pypi.python.org/packages/source/n/nose/'],
    }),

when installing Python easybuild finds the nose library which is installed in the system and doesn't install it to the easybuild's install dir. See the library is loaded from /usr/lib/python2.6

== 2015-12-31 10:17:15,143 runpy.run DEBUG cmd "python -c "import nose"" exited with exitcode 0 and output:
/usr/lib/python2.6/site-packages/nose/proxy.py:93: SyntaxWarning: assertion is always true, perhaps remove parentheses?
  assert (test is self.test

and then when installing matplotlib which is also in the extensions list matplotlib doesn't find nose library and tries to download it again. If I have no internet access it crashes, if I have internet access it will download nose library so I cannot reproduce the same installation.

Processing dependencies for matplotlib==1.3.1
Searching for nose
Reading http://pypi.python.org/simple/nose/
Download error on http://pypi.python.org/simple/nose/: [Errno 111] Connection refused -- Some packages may not be found!
Reading http://pypi.python.org/simple/nose/
Download error on http://pypi.python.org/simple/nose/: [Errno 111] Connection refused -- Some packages may not be found!
Couldn't find index page for 'nose' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading http://pypi.python.org/simple/
Download error on http://pypi.python.org/simple/: [Errno 111] Connection refused -- Some packages may not be found!
No local packages or download links found for nose
error: Could not find suitable distribution for Requirement.parse('nose')
 (at easybuild/tools/run.py:397 in parse_cmd_output)
== 2015-12-31 10:23:11,423 runpy.easyblock WARNING build failed (first 300 chars): cmd " python setup.py install --prefix=/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10 " exited with exitcode 1 and output:

[pescobar]

I have just hit the same problem again. As easybuild can do "import pytz" the library is not installed by easybuild:

== 2015-12-31 12:06:15,577 runpy.run DEBUG run_cmd: running cmd python -c "import pytz" (in /scicore/home/scicore/escobar/easybuild/python-fix-generic-softstack)
== 2015-12-31 12:06:15,727 runpy.run DEBUG cmd "python -c "import pytz"" exited with exitcode 0 and output:

Looking at the log file I get no clue that the library being loaded comes from the system libraries, but I can manually check that this library is only available in the system libs folder:

[escobar@usi120 python-fix-generic-softstack]$ ls /usr/lib/python2.6/site-packages/pytz
__init__.py  __init__.pyc  __init__.pyo  reference.py  reference.pyc  reference.pyo  tzfile.py  tzfile.pyc  tzfile.pyo  tzinfo.py  tzinfo.pyc  tzinfo.pyo

[escobar@usi120 python-fix-generic-softstack]$ ls ~/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pytz
ls: cannot access /scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pytz: No such file or directory

@boegel to address this in the future I would suggest that when building with --debug easybuild could print in the log file the full path to the library being loaded

[boegel]

@pescobar: I'm looking into options to prevent this from happening, but it's not easy...

Can you share what this prints, after loading the Python module?

python -c "import sys; print sys.path"
python -S -c "import sys; print sys.path"

[pescobar]

[escobar@usi120 python-fix-generic-softstack]$ which python
~/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/bin/python

[escobar@usi120 python-fix-generic-softstack]$ python -c "import sys; print sys.path"
['', '/scicore/soft/apps/EasyBuild/2.5.0/lib/python2.6/site-packages/vsc_base-2.2.4-py2.6.egg', '/scicore/soft/apps/EasyBuild/2.5.0/lib/python2.6/site-packages/easybuild_framework-2.5.0-py2.6.egg', '/scicore/soft/apps/EasyBuild/2.5.0/lib/python2.6/site-packages/easybuild_easyblocks-2.5.0-py2.6.egg', '/scicore/soft/apps/EasyBuild/2.5.0/lib/python2.6/site-packages/easybuild_easyconfigs-2.5.0-py2.6.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pip-7.1.2-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/six-1.10.0-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/argcomplete-1.0.0-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/argh-0.26.1-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/certifi-2015.11.20.1-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/ecdsa-0.13-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/python_ldap-2.4.22-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/distribute-0.6.34-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/liac_arff-1.1-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/paramiko-1.12.0-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/matplotlib-1.3.1-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/nose-1.3.6-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pyparsing-2.0.7-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/backports_abc-0.4-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/gffutils-0.8.6.1-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/simplejson-3.8.1-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pyfaidx-0.4.6-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/ez_setup-0.9-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/bx_python-0.7.3-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pandas-0.15.2-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/DendroPy-3.12.0-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/requests-2.2.1-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pybedtools-0.6.4-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pymongo-2.6.3-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pysam-0.8.1-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/lxml-3.3.5-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/HTSeq-0.6.1-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/leveldb-0.193-py2.7-linux-x86_64.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/pyfasta-0.5.2-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/protobuf-2.5.0-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/sphinx_rtd_theme-0.1.9-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/alabaster-0.7.6-py2.7.egg', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages/Babel-2.1.1-py2.7.egg', '/scicore/soft/apps/EasyBuild/2.5.0/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python27.zip', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/plat-linux2', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/lib-tk', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/lib-old', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/lib-dynload', '/scicore/home/scicore/escobar/.local/lib/python2.7/site-packages', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages']

[escobar@usi120 python-fix-generic-softstack]$ python -S -c "import sys; print sys.path"
['', '/scicore/soft/apps/EasyBuild/2.5.0/lib/python2.6/site-packages', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python27.zip', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/plat-linux2', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/lib-tk', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/lib-old', '/scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/lib-dynload']

[pescobar]

using python -S /usr/lib/python2.6/ is no longer in the sys.path but the folder /scicore/home/scicore/escobar/.local/easybuild/software/Python/2.7.5-goolf-1.4.10/lib/python2.7/site-packages isn't there neither and it should be, isn't it?

[boegel]

Yes, I didn't anticipate that...

Using python -S is only useful if we're installing Python packages using the system Python (e.g., when EasyBuild itself is being installed). When we have our own Python installation in $EASYBUILD_INSTALLPATH, python -S results in all Python packages that were installed together with it are no longer found either...

@pescobar: can you check what you have in the .pth files in your EasyBuild installation? See comments above. I suspect the setuptools.pth file in there results in adding the system Python lib path to the Python search path.
We should be able to avoid this by using python -S when installing EasyBuild itself (or, by simply removing the setuptools.pth file after installing EasyBuild).

[boegel]

Candidate fix in easybuilders/easybuild-easyblocks#786.

[boegel]

This should be fixed by a combination of #1569 and easybuilders/easybuild-easyblocks#813 .

I'm closing this since I'm pretty confident that this issue is fixed with these changes, but don't hesitate to report back here in case the problem persists in some way with the upcoming EasyBuild v2.6.0