PROJ.4 data directory not found

[djhoese]

New travis environments are failing with messages like:

  File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/pyresample/geometry.py", line 858, in __init__
    proj = Proj(**proj_dict)
  File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/pyproj/__init__.py", line 322, in __init__
    self.crs = CRS.from_user_input(projparams if projparams is not None else kwargs)
  File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/pyproj/crs.py", line 218, in from_user_input
    return cls(**value)
  File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/pyproj/crs.py", line 140, in __init__
    super(CRS, self).__init__(projstring)
  File "pyproj/_crs.pyx", line 307, in pyproj._crs._CRS.__init__
  File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/pyproj/datadir.py", line 62, in get_data_dir
    "Valid PROJ.4 data directory not found."
pyproj.exceptions.DataDirError: Valid PROJ.4 data directory not found.Either set the path using the environmental variable PROJ_LIB or with `pyproj.datadir.set_data_dir`.

See https://travis-ci.org/pytroll/pytroll-schedule/jobs/505465801#L1073 for details.

If this isn't something simple that we're doing dumb, I will try to find more information tomorrow.

[ocefpaf]

If this isn't something simple that we're doing dumb, I will try to find more information tomorrow.

@snowman2 should we bring the datadir patch back or is this something we can fix upstream? (Sorry, I'm not really following the upstream development as I should lately.)

[snowman2]

What version of pyproj?

[snowman2]

What version of proj? How was it installed?

[djhoese]

@snowman2 The failing travis job is linked above. It is installed with conda (with conda-forge packages obviously) and installs pyproj 2.0.1.

Edit: We are using astropy's ci-helpers but that is just a wrapper around conda to map environment variable lists of packages to conda commands.

[snowman2]

Also, version of proj? What is PROJ_LIB?

[djhoese]

As listed in the linked to travis job above:

proj4-6.0.0                |       h765ec9c_0         9.2 MB  conda-forge

[snowman2]

Apologies for the short questions. I was rushing out the door and attempting to assist you on my personal time with my phone. So, I appreciate you adding the information requested directly in the issue to help me to help you debug this.

I think the most telling piece will be what you have PROJ_LIB set to and what is inside the directory. If you could share that, I would appreciate it.

[snowman2]

should we bring the datadir patch back

The datadir code has changed quite a bit (see this file), so if there is an issue with the code itself, it will likely need to be an upstream fix.

[djhoese]

I am not setting PROJ_LIB. I was having trouble getting pyproj 2.0+ installed last night on my local system, but on travis we are just installing dependencies and running code that uses pyproj. Sorry I don't have more details right now. @adybbroe was the one who originally ran in to these issues.

[snowman2]

PROJ_LIB should be set by activating the conda environment.
For me, it is:

(pyproj)$ env | grep PROJ
PROJ_LIB=/home/snowal/miniconda/envs/pyproj/share/proj

[snowman2]

So, pyproj=2.* only works with PROJ version 6.0.0. So, that may be part if the issues you have. If you have a need for an earlier version of PROJ in conda, there are wheels for pyproj you can install with pip that should not conflict with older versions of PROJ.

[djhoese]

I'm confused. We are using pyproj 2+ and don't care about the version of PROJ C but the one being installed by conda is 6.0.0.

[snowman2]

Everything works fine on my end:

$ conda create -n proj python=3.6 pyproj=2.0.1
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.5.11
  latest version: 4.6.8

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /home/snowal/miniconda/envs/proj

  added / updated specs: 
    - pyproj=2.0.1
    - python=3.6


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pyproj-2.0.1               |   py36h14c3975_0         157 KB  conda-forge

The following NEW packages will be INSTALLED:

    ca-certificates: 2019.3.9-hecc5488_0  conda-forge
    certifi:         2019.3.9-py36_0      conda-forge
    libffi:          3.2.1-hf484d3e_1005  conda-forge
    libgcc-ng:       7.3.0-hdf63c60_0     conda-forge
    libstdcxx-ng:    7.3.0-hdf63c60_0     conda-forge
    ncurses:         6.1-hf484d3e_1002    conda-forge
    openssl:         1.1.1b-h14c3975_1    conda-forge
    pip:             19.0.3-py36_0        conda-forge
    proj4:           6.0.0-h765ec9c_0     conda-forge
    pyproj:          2.0.1-py36h14c3975_0 conda-forge
    python:          3.6.7-h381d211_1004  conda-forge
    readline:        7.0-hf8c457e_1001    conda-forge
    setuptools:      40.8.0-py36_0        conda-forge
    sqlite:          3.26.0-h67949de_1001 conda-forge
    tk:              8.6.9-h84994c4_1000  conda-forge
    wheel:           0.33.1-py36_0        conda-forge
    xz:              5.2.4-h14c3975_1001  conda-forge
    zlib:            1.2.11-h14c3975_1004 conda-forge


Downloading and Extracting Packages
pyproj-2.0.1         | 157 KB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate proj
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) snowal@snowal-lx2:~$ co proj
(proj) snowal@snowal-lx2:~$ conda list | grep proj
# packages in environment at /home/snowal/miniconda/envs/proj:
proj4                     6.0.0                h765ec9c_0    conda-forge
pyproj                    2.0.1            py36h14c3975_0    conda-forge
(proj) snowal@snowal-lx2:~$ env | grep PROJ
PROJ_LIB=/home/snowal/miniconda/envs/proj/share/proj
_CONDA_SET_PROJ_LIB=/home/snowal/miniconda/envs/pyproj/share/proj
(proj) snowal@snowal-lx2:~$ python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyproj
>>> pyproj.CRS("epsg:4326")
<CRS: epsg:4326>
Name: WGS 84
Ellipsoid:
- semi_major_metre: 6378137.00
- semi_minor_metre: 6356752.31
- inverse_flattening: 298.26
Area of Use:
- name: World
- bounds: (-180.0, -90.0, 180.0, 90.0)
Prime Meridian:
- longitude: 0.0000
- unit_name: degree
- unit_conversion_factor: 0.01745329
Axis Info:
- Geodetic latitude[Lat] (north) EPSG:9122 (degree)
- Geodetic longitude[Lon] (east) EPSG:9122 (degree)

>>> 

[snowman2]

I bet it has to do with PROJ_LIB not being set in your pipeline.

[djhoese]

Hm I'm having trouble installing pyproj 2.0+ with rasterio on OSX (it only wants to do pyproj 1.9.6). I'm guessing they can't be installed together right now due to gdal rebuilds being needed?

Edit: Oops 'rasterio' isn't needed for these tests.

[snowman2]

If you are installing it with GDAL, then yes, that is likely the issue. That is what I was referring to for using pip wheels. So, you can try pip install pyproj and see if it works for you.

[djhoese]

Just mentioned in my edit, I was reading the dependencies wrong. I don't need rasterio.

[djhoese]

I have an idea. I think ci-helpers is creating the environment, then installing pyproj, and continuing on. I'm guessing (@ocefpaf correct me if I'm wrong) that conda doesn't re-run the necessary activations on install (setting PROJ_LIB)?

[djhoese]

That seems to have helped (and deactivate and activate after creating the environment): https://travis-ci.org/pytroll/pytroll-schedule/builds/505823076?utm_source=github_status&utm_medium=notification

[ocefpaf]

conda doesn't re-run the necessary activations on install (setting PROJ_LIB)?

Recent versions of conda do that. Not sure that version started to support this.

[djhoese]

Looks like the most recent version of miniconda uses 4.5.12 but on my system I see at least 4.6.8 is available. I'll see if I can ask the ci-helpers people to add a conda update to their commands.

[djhoese]

If I had to guess I'd say it's this version (4.6.3): https://github.com/conda/conda/blob/master/CHANGELOG.md#463-2019-02-07

[ocefpaf]

Yep, Activate environments prior to running post-link scripts (#8229) looks like it.

[djhoese]

I've made a pull request to fix the ci-helpers issue of not using a recent version of conda (so not re-activating the env vars), but @snowman2 does conda-forge's gdal/libgdal/rasterio need a rebuild to work with the current build of pyproj/proj?

Edit: I meant to say, I am still unable to get pyproj 2.0+ on a conda-forge environment (with no PyPI wheels) on OSX.

[ocefpaf]

conda-forge's gdal/libgdal/rasterio need a rebuild to work with the current build of pyproj/proj?

It does. conda-forge global pinning is proj4 pinning is 5.2.0, we are only waiting on cartopy to move proj4 to 6.0.0.

[ocefpaf]

@djhoese can we close this one?

[djhoese]

The changes to ci-helpers have kind of fallen through, but regardless I think this issue has been fixed as best as it can (new version of conda). @ocefpaf does cartopy work with the new proj4 yet?

[ocefpaf]

@ocefpaf does cartopy work with the new proj4 yet?

No but there is a PR for that SciTools/cartopy#1289