[NetCDF] provide a template build script

[visr]

This is more of a builder request than a build. But it's a start, hoping to get the ball rolling on this one.

@Alexander-Barth attempted this earlier but ran into #329. That issue is closed now, but it seems HDF5 is still the problem:

checking for library containing H5Fflush... no
configure: error: Can't find or link to the hdf5 library. Use --disable-netcdf-4, or see config.log for errors.

Even though I see the HDF5 header files under prefix/include.

I also found this patch being applied to a NetCDF fork, there is a chance we need that as well: https://github.com/JuliaPackaging/Yggdrasil/pull/366/files#diff-d492fa4a88a52553c127373c22f25bb3

Links

https://www.unidata.ucar.edu/software/netcdf/docs/getting_and_building_netcdf.html
https://github.com/JuliaGeo/NetCDF.jl
https://github.com/Alexander-Barth/NCDatasets.jl

[giordano]

Quoting from the CMake file:

# Option to Enable HDF5
#
# The HDF5 cmake variables differ between platform (linux/osx and Windows),
# as well as between HDF5 versions.  As a result, this section is a bit convoluted.
#
# Note that the behavior seems much more stable across HDF5 versions under linux,
# so we do not have to do as much version-based tweaking.

😳 😩 😭

[giordano]

I have the feeling that we need to properly build HDF5 in order for it to be really usable as a dependency...

[simonbyrne]

could we do the same thing we do for HDF5 for now? Conda is so unreliable.

[visr]

See #567 for the current status of HDF5. There is also some discussion on the HDF Forum on cross compilation, more people want to get it working, but it seems quite an effort.

[giordano]

I think Simon was suggesting to repackage NetCDF from Anacoda, exactly what we're doing for HDF5 now. If we want NetCDF provided through BinaryBuilder that might be the easiest way. I wouldn't hold my breath for #567, even though I plan to come back to it at some point.

[visr]

Ah ok, I misunderstood. Yes that would probably already be a nice improvement. Not sure how that works with the zlib and curl dependencies, and if that would cause issues.

[giordano]

I had a look to anaconda, but I'm not sure it provides what we need. I could only find:

• netcdf4 which looks like Python-only wrapper, no library as far as I could tell
• netcdf-fortran which provides the Fortran library, but I assume you're looking for the C library

Am I missing something? Do you know from where we could get the pre-built libraries?

[visr]

I think you want the libnetcdf package:
https://anaconda.org/anaconda/libnetcdf
https://github.com/conda-forge/libnetcdf-feedstock

[giordano]

The library provided by conda requires HDF4 😑

[giordano]

And HDF4 from Conda uses libgpeg9 that we don't have. This is turning into a rabbit hole

[visr]

Hmm HDF4 is optional though. I'm not sure how often it is still used. Wasn't aware that the conda build includes it.

[giordano]

I'm trying to build HDF4 from source, with the hope that it cross-compiles. It works for x86_64-linux-gnu, need to try with the other platforms

[simonbyrne]

The library provided by conda requires HDF4 😑

Can we just disable that? From what I understand that is just a legacy format.

[simonbyrne]

Oh, maybe not: https://www.unidata.ucar.edu/software/netcdf/docs/file_format_specifications.html#hdf4_sd_format?
but there it is possible to disable it: https://github.com/Unidata/netcdf-c/blob/master/.travis.yml#L15

[giordano]

Can we just disable that? From what I understand that is just a legacy format.

The problem is that the libnetcdf provided by Conda explicitly links to libmfhdf.so

[giordano]

Ok, I give up. With this builder I can compile HDF4 for x86_64-linux-gnu and macOS (didn't try the other platforms), but when building for Windows I hit this code in the configure script:

# ======================================================================
## Checks for system services
## ======================================================================
## Copy NetCDF header files.
#
## FIXME: This is code stolen^Wborrowed from the old HDF4 configure.
## These header files should probably be generated by autoconf instead
## of being copied depending upon the host.
case "$host" in
  *-linux*)             BAR="linux"       ;;
  *-freebsd*)           BAR="fbsd"        ;;
  *-ibm-aix*)           BAR="aix"         ;;
  sparc64-*-solaris2*)  BAR="solaris64"   ;;
  *-*-solaris2*)        BAR="solaris"     ;;
  *-apple*)             BAR="apple"       ;;
  *-pc-cygwin*)         BAR="linux"       ;;
  *)                    echo "*** unknown host $host!"; exit 1 ;;
esac
src_files=""
src_files="`echo $src_files | sed -e s/FOO/${BAR}/g`"

But in this case host is *-w64-mingw32, so I enter the "unknown host" branch...

Frankly, spending time to (maybe!) cross-compile HDF4 for Windows doesn't seem worth the effort since we don't even want it in the end, it would just be a way to reuse the conda libraries. I should probably look again at cross-compiling HDF5 one day...

[visr]

Thanks for trying, I really appreciate it.

[Alexander-Barth]

Here is some work in progress:
#1090

armv7l-linux-gnueabihf fails but the other platforms do work (but without hdf4)

[visr]

Nice, closing this now that #1090 is merged. Thanks for making this work, @Alexander-Barth.