ImportError DLL load failed only on Windows + GDAL >= 2.2 + Python 3.6

[Aztorius]

Issue:
After creating a fresh dedicated environment for GDAL 2.3 with Python 3.6, osgeo import is not working (only on Windows).
Downgrading to GDAL 2.1 + Python 3.6 works (using conda-forge with label broken).
Downgrading to GDAL 2.3 + Python 2.7 works.
But GDAL>=2.2 + Python 3.6 doesn't work on Windows (but works on Linux).

To reproduce (only on Windows) :

$ conda create -n GDAL python=3.6 gdal
$ activate GDAL
$ python
>>> import osgeo

Same error as #175

I tried reinstalling conda from scratch on two different Windows machine (Win 7 and Win Server 2008 R2).

Environment (conda list):

$ conda list
# packages in environment at D:\Utilisateurs\william.bonnaventure\AppData\Local\Continuum\miniconda3\envs\GDAL:
#
# Name                    Version                   Build  Channel
ca-certificates           2018.4.16                     0    conda-forge
certifi                   2018.4.16                py36_0    conda-forge
curl                      7.61.0               h7602738_0  
expat                     2.2.5                he025d50_1    conda-forge
freexl                    1.0.5                hd288d7e_1    conda-forge
gdal                      2.3.1            py36h10bbe5c_0    conda-forge
geos                      3.6.2                he025d50_2    conda-forge
geotiff                   1.4.2                h2080a26_2    conda-forge
hdf4                      4.2.13               h712560f_2  
hdf5                      1.10.2               hbc3b0e7_1    conda-forge
icc_rt                    2017.0.4             h97af966_0  
intel-openmp              2018.0.3                      0  
jpeg                      9c                   hfa6e2cd_0    conda-forge
kealib                    1.4.9                h4380d0d_2    conda-forge
libcurl                   7.61.0               h7602738_0  
libgdal                   2.3.1                h5df42a0_0    conda-forge
libiconv                  1.15                 hfa6e2cd_1    conda-forge
libnetcdf                 4.6.1                h183b1a9_3    conda-forge
libpng                    1.6.35               h7602738_0    conda-forge
libpq                     9.6.6                h14f23c8_0  
libspatialite             4.3.0a              h582a839_21    conda-forge
libssh2                   1.8.0                hc4dcbb0_2    conda-forge
libtiff                   4.0.9                hf1753bf_1    conda-forge
libxml2                   2.9.8                haffccfe_2    conda-forge
mkl                       2018.0.3                      1  
mkl_fft                   1.0.4                    py36_0    conda-forge
mkl_random                1.0.1                    py36_0    conda-forge
numpy                     1.14.3           py36h9fa60d3_2  
numpy-base                1.14.3           py36h5c71026_0  
openjpeg                  2.3.0                h25a6d84_3    conda-forge
openssl                   1.0.2o               hfa6e2cd_1    conda-forge
pip                       18.0                     py36_0    conda-forge
proj4                     4.9.3                hcf24537_7  
python                    3.6.6                he025d50_0    conda-forge
setuptools                40.0.0                   py36_0    conda-forge
sqlite                    3.24.0               hb652765_0    conda-forge
vc                        14.1                 h0510ff6_3  
vs2015_runtime            15.5.2                        3  
wheel                     0.31.1                   py36_0    conda-forge
wincertstore              0.2                      py36_1    conda-forge
xerces-c                  3.2.0                h44c76bb_2  
xz                        5.2.4                h2fa13f4_0    conda-forge
zlib                      1.2.11               h2fa13f4_3    conda-forge

Details about conda and system ( conda info ):

$ conda info
     active environment : GDAL
    active env location : D:\Utilisateurs\william.bonnaventure\AppData\Local\Continuum\miniconda3\envs\GDAL
            shell level : 2
       user config file : D:\Utilisateurs\william.bonnaventure\.condarc
 populated config files : D:\Utilisateurs\william.bonnaventure\.condarc
          conda version : 4.5.8
    conda-build version : not installed
         python version : 3.6.5.final.0
       base environment : D:\Utilisateurs\william.bonnaventure\AppData\Local\Continuum\miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : D:\Utilisateurs\william.bonnaventure\AppData\Local\Continuum\miniconda3\pkgs
                          D:\Utilisateurs\william.bonnaventure\AppData\Local\conda\conda\pkgs
       envs directories : D:\Utilisateurs\william.bonnaventure\AppData\Local\Continuum\miniconda3\envs
                          D:\Utilisateurs\william.bonnaventure\AppData\Local\conda\conda\envs
                          D:\Utilisateurs\william.bonnaventure\.conda\envs
               platform : win-64
             user-agent : conda/4.5.8 requests/2.18.4 CPython/3.6.5 Windows/7 Windows/6.1.7601
          administrator : False
             netrc file : None
           offline mode : False

[ocefpaf]

@gillins I don't have access to a Windows machine at the moment but my guess is that, based on the number of packages coming from defaults there, we either have an unsolvable env and something needs to be rebuild, or conda is not respecting channel preference and is returning a broken env.

I really wish that channel preference would always be respect, that would make debugging issues like these much easier...

[gillins]

@ocefpaf I can reproduce this, but have no idea where to start debugging. Any suggestions? It is very strange it prefers not to use conda-forge numpy. I'll try and have a proper look tomorrow.

[ocefpaf]

It is very strange it prefers not to use conda-forge numpy.

Yeah, that sucks. I have to add conda-forge::numpy everywhere to get conda-forge's numpy instead of defaults. (Ping @kalefranz, this is another pain point on how channel preference works.)

@ocefpaf I can reproduce this, but have no idea where to start debugging. Any suggestions?

The way I debugged this was to force all packages from conda-forge, see if the situation is unsolvable in that scenario, and update the packages that need to be re-built with the new pinning.

One guess is libnetcdf. Note that we are still building gdal with old libnetcdf on Windows b/c I can get conda-forge/libnetcdf-feedstock#49 to work.

[kalefranz]

Just released (to canary yesterday, defaults Wednesday) conda 4.5.9 that includes
conda/conda#7601 for testing. Open to switching the default value of that config feature switch in either 4.6.0 or a later 4.5, based on community input.

We also need to get our metadata patching coordinated for both vc and blas. Both the PR and metadarapatching are independent solutions that tackle the same problem, not that we shouldn’t push for both.

[ocefpaf]

Just released (to canary yesterday, defaults Wednesday) conda 4.5.9 that includes
conda/conda#7601 for testing. Open to switching the default value of that config feature switch in either 4.6.0 or a later 4.5, based on community input.

Thanks! I'll try to test them soon and get back to you.

We also need to get our metadata patching coordinated for both vc and blas. Both the PR and metadata patching are independent solutions that tackle the same problem, not that we shouldn’t push for both.

As someone that does not know conda internals how hard would it be to solve from the preferred channel only, and then get the packages that do not exists on the preferred channel from the ones below? Would that be too slow? Lead to impossible solutions? If that works though the issues with features would be "solved," right?

Just some background of where this is coming from. I'm used to the concept of my Linux distro that has a 0-99 level of repository preference (horrible numbers, I know.), that allows me to fine tune how much I want from "latest packages" vs "repository preference." Ideally conda would need only 3 levels IMO:

• 0 solve/get all packages from the preferred channel except for the packages that do not exist there;
• 1 the current behavior;
• 2 get the latest packages from all channels no matter the preference order.

None of those options would ensure stability, specially 2, but 0 would make debugging broken envs solution much easier and the others will give users choices in case they know what they are doing.

BTW, I do know that 0 is done by many people when they have the luxury of having a single channel and block all others, but that cut them from getting packages from 3rd patry channel easily without some manual labor of listing them with 3r_party_channel::package_I_want in their envs.

[jacobadams-cc]

FWIW, I'm getting a similar behavior when trying to run the gdal command line tools after updating a year-old setup—a system error stating "libtiff.dll was not found" pops up.

When building a pure clone of the libgdal feedstock from source, the test script loads everything from conda-forge, with vc at version 14-0 and vs2015_runtime at 14.0.25420-0. However, conda install --use-local libgdal in a fresh, empty environment pulls in the packages from both default and conda-forge as noted. Here, vc is 14.1 and vs2015_runtime 15.5.2.

I've found adding "vs2015_runtime 14.*" to the pinned file in conda-meta allows a normal conda install gdal that will at least import osgeo and run gdalinfo --version properly.

[gillins]

If I force numpy from conda-forge then everything else gets pulled in from conda-forge also. I can't work out why defaults numpy would be preferred given that is an earlier version (1.14.3 vs 1.15.0)....

I hadn't seen that issue with libnetcdf - I'll have a look when I get a moment, but I don't think this is the problem here.

[gillins]

I note that the same problem (pulling in defaults numpy) currently also happens on Linux (but not the import error).

[gboeing]

@ocefpaf I believe this is the root cause of conda-forge/fiona-feedstock#89

[gboeing]

Running:

conda create -n GDAL gdal

resulted in a broken environment and the same error the others have described above. But running:

conda create --override-channels -c conda-forge -n GDAL python=3 gdal

resulted in a working environment and solved this error. Seems to be a channel-mixing problem with defaults sneaking in, unless explicitly overridden. FWIW here's my conda info:

     active environment : GDAL
    active env location : c:\Anaconda\envs\GDAL
            shell level : 1
       user config file : C:\Users\Geoff\.condarc
 populated config files : C:\Users\Geoff\.condarc
          conda version : 4.5.9
    conda-build version : not installed
         python version : 3.6.5.final.0
       base environment : c:\Anaconda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/win-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/win-64
                          https://repo.anaconda.com/pkgs/pro/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : c:\Anaconda\pkgs
                          C:\Users\Geoff\AppData\Local\conda\conda\pkgs
       envs directories : c:\Anaconda\envs
                          C:\Users\Geoff\AppData\Local\conda\conda\envs
                          C:\Users\Geoff\.conda\envs
               platform : win-64
             user-agent : conda/4.5.9 requests/2.18.4 CPython/3.6.5 Windows/10 Windows/10.0.17134
          administrator : False
             netrc file : None
           offline mode : False

[ocefpaf]

Seems to be a channel-mixing problem with defaults sneaking in, unless explicitly overridden

Ping @kalefranz and @msarahan these are getting more and more common. A solvable env with conda-forge on top is not enough to avoid pulling packages from defaults and resulting in a broken env :-(

I did not get the time to test #219 (comment) yet. As soon as I do I'll report back here. (Hard to get my hands on a Windows machine to test that.)

[MorningGlory747]

Hello, I'm having a similar issue on my Windows 10 in the way that I also receive the ImportError DLL when trying to import GDAL 2.2.2 from either Spyder 3.3.0 (from Anaconda 5.1.0) or Pycharm 2018.2 (which uses conda as the interpreter).

However, I can get GDAL to work from the Anaconda prompt when doing:
activate GDAL
python
import osgeo (from the python console)

I must note that when I installed GDAL from conda-forge and also everytime I activate GDAL, I receive the notice that the filepath 'GDAL_DRIVER_PATH=' cannot be found, but I don't know how to solve that issue.

Any help would be appreciated :D

[hans-permana]

I have observed the same issue and figured out that the culprit is the vs2015_runtime. gdal (and subsequently fiona) seems to require dll files that are available on v14.x but not on v15.x.

The workaround is to downgrade your vs2015_runtime to v14.x by running this command conda install -c conda-forge vs2015_runtime=14.

A long term solution would be to include a requirement to vs2015_runtime 14.x on gdal conda recipe (only for windows build).

[ocefpaf]

@hans-permana that is just a symptom of conda/conda#7626 and not a real solution. Once conda fixes that bug everything should be OK on Windows again.

Closing this b/c it is not actionable by the team here. We need to wait for a real fix, meanwhile people have to use those workarounds.