gcc fails when non-admin user tries to pip install a repo from within a notebook

[zaneselvans]

I sent a non-admin user a nbgitpuller link to our new tutorial notebook repo, to run on the Catalyst Cooperative pilot hub, the first cell of which installs the snappy compression library (using conda) and then our PUDL package. It also upgrades pip because I had previously gotten an error about the resolver not taking into account all installed packages, and wondered if a new version would address the issue. It appears that the pip upgrade worked, but the pip install of pudl from our repository resulted in a compiler error on the user's machine, pasted below.

The error message I previously got from pip was:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sat-stac 0.4.0 requires python-dateutil~=2.7.5, but you have python-dateutil 2.8.1 which is incompatible.

Testing now with fresh restarts of the server, I see that I get that error regardless of whether I upgrade pip, so I've removed that line from the first cell.

I haven't experienced the behavior below when running the same cell in the same notebook. Maybe it's because I'm an admin? Not sure what else would be different with a fresh slate after a new login.

notebook cell contents

!conda install --yes --quiet python-snappy
!pip install --upgrade pip
!pip install --quiet git+https://github.com/catalyst-cooperative/pudl.git@jupyterhub-beta
!cp ~/shared/shared-pudl.yml ~/.pudl.yml

resulting outputs

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /srv/conda/envs/notebook

  added / updated specs:
    - python-snappy


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    openssl-1.1.1i             |       h7f98852_0         2.1 MB  conda-forge
    python-snappy-0.5.4        |   py38hd7b2518_3          31 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         2.2 MB

The following NEW packages will be INSTALLED:

  python-snappy      conda-forge/linux-64::python-snappy-0.5.4-py38hd7b2518_3

The following packages will be UPDATED:

  openssl                                 1.1.1h-h516909a_0 --> 1.1.1i-h7f98852_0


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Requirement already satisfied: pip in /srv/conda/envs/notebook/lib/python3.8/site-packages (20.3.1)
Collecting pip
  Downloading pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 4.6 MB/s eta 0:00:01
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.1
    Uninstalling pip-20.3.1:
      Successfully uninstalled pip-20.3.1
Successfully installed pip-20.3.3
  ERROR: Command errored out with exit status 1:
   command: /srv/conda/envs/notebook/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-3ri5ch8z
       cwd: /tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/
  Complete output (20 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/bitarray
  copying bitarray/__init__.py -> build/lib.linux-x86_64-3.8/bitarray
  copying bitarray/test_util.py -> build/lib.linux-x86_64-3.8/bitarray
  copying bitarray/util.py -> build/lib.linux-x86_64-3.8/bitarray
  copying bitarray/test_bitarray.py -> build/lib.linux-x86_64-3.8/bitarray
  running build_ext
  building 'bitarray._bitarray' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/bitarray
  /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC -I/srv/conda/envs/notebook/include/python3.8 -c bitarray/_bitarray.c -o build/temp.linux-x86_64-3.8/bitarray/_bitarray.o
  x86_64-conda_cos6-linux-gnu-gcc -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include build/temp.linux-x86_64-3.8/bitarray/_bitarray.o -o build/lib.linux-x86_64-3.8/bitarray/_bitarray.cpython-38-x86_64-linux-gnu.so
  /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
  /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
  collect2: error: ld returned 1 exit status
  error: command 'x86_64-conda_cos6-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for bitarray
    ERROR: Command errored out with exit status 1:
     command: /srv/conda/envs/notebook/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-gwj9ihe8/install-record.txt --single-version-externally-managed --compile --install-headers /srv/conda/envs/notebook/include/python3.8/bitarray
         cwd: /tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/
    Complete output (20 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/bitarray
    copying bitarray/__init__.py -> build/lib.linux-x86_64-3.8/bitarray
    copying bitarray/test_util.py -> build/lib.linux-x86_64-3.8/bitarray
    copying bitarray/util.py -> build/lib.linux-x86_64-3.8/bitarray
    copying bitarray/test_bitarray.py -> build/lib.linux-x86_64-3.8/bitarray
    running build_ext
    building 'bitarray._bitarray' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/bitarray
    /srv/conda/envs/notebook/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include -fPIC -I/srv/conda/envs/notebook/include/python3.8 -c bitarray/_bitarray.c -o build/temp.linux-x86_64-3.8/bitarray/_bitarray.o
    x86_64-conda_cos6-linux-gnu-gcc -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/srv/conda/envs/notebook/lib -Wl,-rpath-link,/srv/conda/envs/notebook/lib -L/srv/conda/envs/notebook/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /srv/conda/envs/notebook/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /srv/conda/envs/notebook/include build/temp.linux-x86_64-3.8/bitarray/_bitarray.o -o build/lib.linux-x86_64-3.8/bitarray/_bitarray.cpython-38-x86_64-linux-gnu.so
    /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
    /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-conda_cos6-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /srv/conda/envs/notebook/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dtyq92j4/bitarray_bc8688d45a1144eb88a537bff35c0079/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-gwj9ihe8/install-record.txt --single-version-externally-managed --compile --install-headers /srv/conda/envs/notebook/include/python3.8/bitarray Check the logs for full command output.

[zaneselvans]

I had another non-admin user try to run this notebook cell, and she didn't have the gcc failure, but did get the pip resolver warning, which unfortunately did then prevent the rest of the notebook from running so.... wtf is up with pip? Maybe this is only happening in this context because python-snappy has been installed with conda first, and that's resulting in the incompatible version of dateutils? Ugh. It feels like trying to debug this interim software install setup may be a waste of time if what we really want to do is pull an image from Docker hub and use that as the environment. But also it's frustrating that we can't try things out with some users first.

[choldgraf]

hmmmm I could see issues popping up with pip's new dependency solver if it is finding versions of things that were installed by a different tool...I wonder if @pradyunsg has run into this issue before?

[yuvipanda]

This was also resolved by #147. Catalyst Coop was accidentally on the pangeo image, rather than our own. The commands you provided worked for me now! Once we switch to your own docker image, these problems should go away.

Re: admin / non-admin, I'm not sure why that would be :|

[yuvipanda]

@zaneselvans can you try it out and let me know if it works out ok?

[zaneselvans]

Hey @yuvipanda yes, it's working now! I'm still confused why the in-notebook software installs would have continued working for other users, but not for a new one -- maybe I'm misunderstanding how / when the Docker image is deployed.