Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MPI error #68

Open
rabernat opened this issue Oct 16, 2019 · 3 comments
Open

MPI error #68

rabernat opened this issue Oct 16, 2019 · 3 comments

Comments

@rabernat
Copy link

I tried to run the curvilinear example from the docs on a relatively fresh conda environment on mac os.

When building the grid weights, I hit the same issue that came up in pydata/xarray#2281 (comment).

Fatal error in MPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(572)..............: 
MPID_Init(224).....................: channel initialization failed
MPIDI_CH3_Init(105)................: 
MPID_nem_init(324).................: 
MPID_nem_tcp_init(178).............: 
MPID_nem_tcp_get_business_card(425): 
MPID_nem_tcp_init(384).............: gethostbyname failed, Ryans-MacBook-Pro.local (errno 1)
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=3191567
:
system msg for write_line failure : Bad file descriptor

causing my kernel to die.

I know this is an ESMF/ESMPy issue, but I thought I would raise it here first.

I installed everything from conda-forge.

$ conda list
# packages in environment at /Users/rpa/miniconda3/envs/pangeo:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   pypi_0    pypi
alembic                   1.2.1                    pypi_0    pypi
appdirs                   1.4.3                      py_1    conda-forge
appnope                   0.1.0                 py36_1000    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
asn1crypto                0.24.0                py36_1003    conda-forge
atomicwrites              1.3.0                    pypi_0    pypi
attrs                     19.1.0                     py_0    conda-forge
babel                     2.7.0                    pypi_0    pypi
backcall                  0.1.0                      py_0    conda-forge
basemap                   1.2.1            py36hd103db2_1    conda-forge
binutils-meta             1.0.4                         0    conda-forge
bleach                    3.1.0                      py_0    conda-forge
blinker                   1.4                        py_1    conda-forge
bokeh                     1.3.4                    py36_0    conda-forge
boost-cpp                 1.70.0               h75728bb_2    conda-forge
boto3                     1.9.229                    py_0    conda-forge
botocore                  1.12.229                   py_0    conda-forge
bottleneck                1.2.1           py36h917ab60_1001    conda-forge
bqplot                    0.11.8                     py_0    conda-forge
bzip2                     1.0.8                h01d97ff_1    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
cachetools                2.1.0                      py_0    conda-forge
cairo                     1.16.0            he1c11cd_1002    conda-forge
cartopy                   0.17.0          py36hd103db2_1006    conda-forge
cctools                   921                  h5ba7a2e_4    conda-forge
certifi                   2019.6.16                py36_1    conda-forge
cffi                      1.12.3           py36hccf1714_0    conda-forge
cfgrib                    0.9.7.1                    py_0    conda-forge
cfitsio                   3.470                h84d2f63_2    conda-forge
cftime                    1.0.3.4         py36h1e5eb4f_1001    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
clang                     9.0.0                h28b9765_0    conda-forge
click                     7.0                        py_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                      py_0    conda-forge
cloudpickle               1.2.2                      py_0    conda-forge
colorcet                  2.0.1                      py_0    conda-forge
commonmark                0.9.0                    pypi_0    pypi
configobj                 5.0.6                      py_0    conda-forge
cryptography              2.7              py36h212c5bf_0    conda-forge
curl                      7.65.3               h22ea746_0    conda-forge
cycler                    0.10.0                     py_1    conda-forge
cython                    0.29.13          py36h6de7cb9_0    conda-forge
cytoolz                   0.10.0           py36h01d97ff_0    conda-forge
dask                      2.4.0                      py_0    conda-forge
dask-core                 2.4.0                      py_0    conda-forge
datashader                0.7.0                      py_1    conda-forge
datashape                 0.5.4                      py_1    conda-forge
dbus                      1.13.6               h2f22bb5_0    conda-forge
decorator                 4.4.0                      py_0    conda-forge
defusedxml                0.5.0                      py_1    conda-forge
distributed               2.4.0                      py_0    conda-forge
doctr                     1.8.0                    pypi_0    pypi
docutils                  0.15.2                   py36_0    conda-forge
eccodes                   2.13.1               h43bd019_1    conda-forge
entrypoints               0.3                   py36_1000    conda-forge
esmf                      7.1.0             h963e782_1008    conda-forge
esmpy                     7.1.0            py36h24bf2e0_3    conda-forge
expat                     2.2.5             h6de7cb9_1003    conda-forge
fasteners                 0.14.1                     py_3    conda-forge
fiona                     1.8.6            py36h7eb7563_3    conda-forge
fontconfig                2.13.1            h6b1039f_1001    conda-forge
fortran-compiler          1.0.4                he991be0_0    conda-forge
freetype                  2.10.0               h24853df_1    conda-forge
freexl                    1.0.5             h1de35cc_1002    conda-forge
fsspec                    0.4.5                      py_0    conda-forge
future                    0.17.1                   pypi_0    pypi
fuzzywuzzy                0.17.0                   pypi_0    pypi
gcsfs                     0.3.0                      py_1    conda-forge
gdal                      2.4.1            py36h39889d8_8    conda-forge
geopandas                 0.5.1                      py_0    conda-forge
geos                      3.7.2                h6de7cb9_2    conda-forge
geotiff                   1.4.3             h113155d_1001    conda-forge
geoviews                  1.6.3                      py_0    conda-forge
geoviews-core             1.6.3                      py_0    conda-forge
gettext                   0.19.8.1          h46ab8bc_1002    conda-forge
gfortran_impl_osx-64      7.3.0                hf4212f2_2    conda-forge
gfortran_osx-64           7.3.0                h22b1bf0_5    conda-forge
giflib                    5.1.7                h01d97ff_1    conda-forge
glib                      2.58.3            h9d45998_1002    conda-forge
gmp                       6.1.2             h0a44026_1000    conda-forge
google-auth               1.6.3                      py_0    conda-forge
google-auth-oauthlib      0.4.0                      py_0    conda-forge
gsw                       3.3.1            py36h01d97ff_0    conda-forge
h5py                      2.10.0          nompi_py36h6248fd5_100    conda-forge
hdf4                      4.2.13            hf3c6af0_1002    conda-forge
hdf5                      1.10.5          nompi_h15a436c_1103    conda-forge
heapdict                  1.0.0                 py36_1000    conda-forge
holoviews                 1.12.5                     py_0    pyviz
icu                       64.2                 h6de7cb9_1    conda-forge
idna                      2.8                   py36_1000    conda-forge
imageio                   2.5.0                    py36_0    conda-forge
imagesize                 1.1.0                    pypi_0    pypi
importlib-metadata        0.23                     pypi_0    pypi
intake                    0.5.3                      py_0    conda-forge
intake-xarray             0.3.1                      py_0    conda-forge
ipykernel                 5.1.2            py36h5ca1d4c_0    conda-forge
ipython                   7.8.0            py36h5ca1d4c_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.5.1                      py_0    conda-forge
isl                       0.19                          0    conda-forge
jasper                    1.900.1           h636a363_1006    conda-forge
jedi                      0.15.1                   py36_0    conda-forge
jinja2                    2.10.1                     py_0    conda-forge
jmespath                  0.9.4                      py_0    conda-forge
joblib                    0.13.2                     py_0    conda-forge
jpeg                      9c                h1de35cc_1001    conda-forge
json-c                    0.13.1            h1de35cc_1001    conda-forge
json5                     0.8.5                      py_0    conda-forge
jsonschema                3.0.2                    py36_0    conda-forge
jupyter                   1.0.0                    pypi_0    pypi
jupyter-book              0.5.2                    pypi_0    pypi
jupyter-contrib-core      0.3.3                    pypi_0    pypi
jupyter-contrib-nbextensions 0.5.1                    pypi_0    pypi
jupyter-highlight-selected-word 0.2.0                    pypi_0    pypi
jupyter-latex-envs        1.4.6                    pypi_0    pypi
jupyter-nbextensions-configurator 0.4.1                    pypi_0    pypi
jupyter_client            5.3.1                      py_0    conda-forge
jupyter_console           6.0.0                      py_0    conda-forge
jupyter_core              4.4.0                      py_0    conda-forge
jupyterlab                1.1.3                      py_0    conda-forge
jupyterlab_server         1.0.6                      py_0    conda-forge
kealib                    1.4.10            h6659575_1005    conda-forge
kiwisolver                1.1.0            py36h770b8ee_0    conda-forge
krb5                      1.16.3            hcfa6398_1001    conda-forge
latexcodec                1.0.7                    pypi_0    pypi
ld64                      409.12               h3c32e8a_4    conda-forge
libaec                    1.0.4                h0a44026_0    conda-forge
libblas                   3.8.0               12_openblas    conda-forge
libcblas                  3.8.0               12_openblas    conda-forge
libcurl                   7.65.3               h16faf7d_0    conda-forge
libcxx                    8.0.1                         0    conda-forge
libcxxabi                 8.0.1                         0    conda-forge
libdap4                   3.20.4               habf5908_0    conda-forge
libedit                   3.1.20170329      hcfe32e1_1001    conda-forge
libffi                    3.2.1             h6de7cb9_1006    conda-forge
libgdal                   2.4.1                h75d216d_8    conda-forge
libgfortran               4.0.0                         2    conda-forge
libiconv                  1.15              h01d97ff_1005    conda-forge
libkml                    1.3.0             hed7d534_1010    conda-forge
liblapack                 3.8.0               12_openblas    conda-forge
libllvm9                  9.0.0                h770b8ee_2    conda-forge
libnetcdf                 4.6.2             h1a02027_1003    conda-forge
libopenblas               0.3.7                h4bb4525_1    conda-forge
libpng                    1.6.37               h2573ce8_0    conda-forge
libpq                     11.5                 h756f0eb_1    conda-forge
libsodium                 1.0.17               h01d97ff_0    conda-forge
libspatialindex           1.9.0                h6de7cb9_1    conda-forge
libspatialite             4.3.0a            hb3e03d6_1030    conda-forge
libssh2                   1.8.2                hcdc9a53_2    conda-forge
libtiff                   4.0.10            hd08fb8f_1003    conda-forge
libuv                     1.30.1               h01d97ff_0    conda-forge
libxml2                   2.9.9                h12c6b28_5    conda-forge
llvm-openmp               8.0.1                h770b8ee_0    conda-forge
llvmlite                  0.29.0           py36h3fea490_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lxml                      4.4.1                    pypi_0    pypi
lz4-c                     1.8.3             h6de7cb9_1001    conda-forge
mako                      1.1.0                    pypi_0    pypi
markdown                  2.6.11                   pypi_0    pypi
markupsafe                1.1.1            py36h1de35cc_0    conda-forge
matplotlib                3.1.1                    py36_1    conda-forge
matplotlib-base           3.1.1            py36h3a684a6_1    conda-forge
mistune                   0.8.4           py36h1de35cc_1000    conda-forge
monotonic                 1.5                        py_0    conda-forge
more-itertools            7.2.0                    pypi_0    pypi
mpc                       1.1.0             h3aece09_1006    conda-forge
mpfr                      4.0.2                h44b798e_0    conda-forge
mpi                       1.0                       mpich    conda-forge
mpich                     3.3.1                hc856adb_1    conda-forge
msgpack-numpy             0.4.4.3                    py_0    conda-forge
msgpack-python            0.6.1            py36h04f5b5a_0    conda-forge
multipledispatch          0.6.0                      py_0    conda-forge
munch                     2.3.2                      py_0    conda-forge
nbclean                   0.3.2                    pypi_0    pypi
nbconvert                 5.4.1                    pypi_0    pypi
nbformat                  4.4.0                      py_1    conda-forge
nbgrader                  0.6.0                    pypi_0    pypi
nbsphinx                  0.4.2                    pypi_0    pypi
ncurses                   6.1               h0a44026_1002    conda-forge
netcdf-fortran            4.4.5             h1993a31_1004    conda-forge
netcdf4                   1.5.1.2          py36h876e625_1    conda-forge
networkx                  2.3                        py_0    conda-forge
nodejs                    12.8.1               hec2bf70_0    conda-forge
notebook                  6.0.1                    py36_0    conda-forge
numba                     0.45.1           py36h86efe34_0    conda-forge
numcodecs                 0.6.3            py36h0a44026_0    conda-forge
numpy                     1.17.2           py36h6b0580a_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.1                hdc36067_1    conda-forge
openssl                   1.1.1c               h01d97ff_0    conda-forge
oset                      0.1.3                    pypi_0    pypi
owslib                    0.18.0                     py_0    conda-forge
packaging                 19.0                       py_0    conda-forge
pandas                    0.25.1           py36h86efe34_0    conda-forge
pandoc                    2.7.3                         0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
panel                     0.6.2                      py_0    pyviz
param                     1.9.1                      py_0    conda-forge
parso                     0.5.1                      py_0    conda-forge
partd                     1.0.0                      py_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.41              h0a44026_1003    conda-forge
pexpect                   4.7.0                    py36_0    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pillow                    6.1.0            py36h75ffe9a_1    conda-forge
pip                       19.2.3                   py36_0    conda-forge
pixman                    0.38.0            h01d97ff_1003    conda-forge
pluggy                    0.13.0                   pypi_0    pypi
poppler                   0.67.0               hd5eb092_7    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                11.5                 h25afefd_1    conda-forge
proj4                     5.2.0             h6de7cb9_1006    conda-forge
prometheus_client         0.7.1                      py_0    conda-forge
prompt_toolkit            2.0.9                      py_0    conda-forge
psutil                    5.6.3            py36h01d97ff_0    conda-forge
ptyprocess                0.6.0                   py_1001    conda-forge
py                        1.8.0                    pypi_0    pypi
pyasn1                    0.4.6                      py_0    conda-forge
pyasn1-modules            0.2.6                      py_0    conda-forge
pybtex                    0.22.2                   pypi_0    pypi
pybtex-docutils           0.2.1                    pypi_0    pypi
pycparser                 2.19                     py36_1    conda-forge
pyct                      0.4.6                      py_0    conda-forge
pyct-core                 0.4.6                      py_0    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pygments                  2.4.2                      py_0    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pykdtree                  1.3.1           py36h917ab60_1002    conda-forge
pyopenssl                 19.0.0                   py36_0    conda-forge
pyparsing                 2.4.2                      py_0    conda-forge
pyproj                    1.9.6           py36h01d97ff_1002    conda-forge
pyqt                      5.9.2            py36h2a560b1_4    conda-forge
pyresample                1.13.0           py36h86efe34_0    conda-forge
pyrsistent                0.15.4           py36h01d97ff_0    conda-forge
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.0                    py36_0    conda-forge
pytest                    5.1.2                    pypi_0    pypi
python                    3.6.7             h94afb7f_1005    conda-forge
python-dateutil           2.8.0                      py_0    conda-forge
python-editor             1.0.4                    pypi_0    pypi
python-snappy             0.5.4            py36h1e06ddd_0    conda-forge
pytz                      2019.2                     py_0    conda-forge
pyviz_comms               0.7.2                      py_0    conda-forge
pywavelets                1.0.3            py36h1e5eb4f_1    conda-forge
pyyaml                    5.1.2            py36h01d97ff_0    conda-forge
pyzmq                     18.1.0           py36hee98d25_0    conda-forge
qt                        5.9.7                h8cf7e54_3    conda-forge
qtconsole                 4.5.5                      py_0    conda-forge
readline                  8.0                  hcfe32e1_0    conda-forge
recommonmark              0.6.0                    pypi_0    pypi
requests                  2.22.0                   py36_1    conda-forge
requests-oauthlib         1.2.0                      py_0    conda-forge
rise                      5.5.1                    py36_0    conda-forge
rsa                       3.4.2                      py_1    conda-forge
rtree                     0.8.3           py36h666c49c_1002    conda-forge
ruamel.yaml               0.16.5           py36h01d97ff_1    conda-forge
ruamel.yaml.clib          0.1.2            py36h01d97ff_0    conda-forge
s3fs                      0.3.4                      py_0    conda-forge
s3transfer                0.2.1                    py36_0    conda-forge
scikit-image              0.15.0           py36h86efe34_2    conda-forge
scikit-learn              0.21.3           py36hd4ffd6c_0    conda-forge
scipy                     1.3.1            py36h7e0e109_2    conda-forge
seaborn                   0.9.0                      py_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                41.2.0                   py36_0    conda-forge
shapely                   1.6.4           py36h0567c5e_1006    conda-forge
sip                       4.19.8          py36h0a44026_1000    conda-forge
six                       1.12.0                py36_1000    conda-forge
snappy                    1.1.7             h6de7cb9_1002    conda-forge
snowballstemmer           1.9.1                    pypi_0    pypi
sortedcontainers          2.1.0                      py_0    conda-forge
sphinx                    1.6.5                    pypi_0    pypi
sphinx-bootstrap-theme    0.7.1                    pypi_0    pypi
sphinx-copybutton         0.2.5                    pypi_0    pypi
sphinx-jinja              1.1.0                    pypi_0    pypi
sphinx-markdown-tables    0.0.9                    pypi_0    pypi
sphinx-nbexamples         0.4.0                    pypi_0    pypi
sphinx-pangeo-theme       0.1.0                    pypi_0    pypi
sphinxcontrib-bibtex      0.4.2                    pypi_0    pypi
sphinxcontrib-srclinks    0.2.4                    pypi_0    pypi
sphinxcontrib-websupport  1.1.2                    pypi_0    pypi
sqlalchemy                1.3.8                    pypi_0    pypi
sqlite                    3.29.0               hb7d70f7_1    conda-forge
statsmodels               0.10.1           py36heacc8b8_0    conda-forge
tapi                      1000.10.8            h770b8ee_3    conda-forge
tblib                     1.4.0                      py_0    conda-forge
terminado                 0.8.2                    py36_0    conda-forge
testpath                  0.4.2                   py_1001    conda-forge
tk                        8.6.9             h2573ce8_1003    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py36h01d97ff_0    conda-forge
tqdm                      4.35.0                     py_0    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
traittypes                0.2.1                      py_1    conda-forge
tzcode                    2019a             h01d97ff_1002    conda-forge
urllib3                   1.25.3                   py36_0    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.33.6                   py36_0    conda-forge
widgetsnbextension        3.5.1                    py36_0    conda-forge
xarray                    0.12.3+71.gf7c12a9a           dev_0    <develop>
xcape                     0+untagged.15.gfe33fdd.dirty           dev_0    <develop>
xerces-c                  3.2.2             hbda6038_1004    conda-forge
xesmf                     0.2.1                      py_0    conda-forge
xhistogram                0.1.1                     dev_0    <develop>
xz                        5.2.4             h1de35cc_1001    conda-forge
yaml                      0.1.7             h1de35cc_1001    conda-forge
zarr                      2.3.2                    py36_0    conda-forge
zeromq                    4.3.2                h6de7cb9_2    conda-forge
zict                      1.0.0                      py_0    conda-forge
zipp                      0.6.0                    pypi_0    pypi
zlib                      1.2.11            h01d97ff_1006    conda-forge
zstd                      1.4.0                ha9f0a20_0    conda-forge
@JiaweiZhuang
Copy link
Owner

JiaweiZhuang commented Oct 16, 2019 •
edited
Loading

This error can happen on HPC clusters with pre-installed, incompatible ESMF/NetCDF/MPI, and can be solved by unset LD_LIBRARY_PATH or module purge before conda install (#55 (comment)) But on Mac OS there shouldn't be such an error, unless you have some MPI installed...

@rabernat
Copy link
Author

Thanks for the quick reply!

But on Mac OS there shouldn't be such an error, unless you have some MPI installed...

Nothing special here, just a pretty vanilla macbook with a fresh conda environment.

These dependency problems are very hard / annoying to debug.

@JiaweiZhuang
Copy link
Owner

People do get the same error with Mac OS + Conda + MPI and there seems to be a solution: conda-forge/fenics-feedstock#44 (comment)

@mickaellalande mickaellalande mentioned this issue Mar 2, 2020
Closed
aulemahal pushed a commit to Ouranosinc/xESMF that referenced this issue Jan 26, 2021
@raphaeldussin
Merge pull request JiaweiZhuang#68 from raphaeldussin/test_ocean
4a8cf60 
start adding testing for commonly used ocean models
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rabernat @JiaweiZhuang