GDAL, GEOS and PROJ versions

[Robinlovelace]

This isn't a bug per se, but a question: are there any plans for updating the GDAL and PROJ versions shipped in rocker/geospatial? Both are evolving, as outlined here: r-spatial/sf#988

I think documentation on this may also be useful. One resource that I've found useful, that may be of use/interest for updating GDAL versions, is this (although it builds on Ubuntu not Debian, should be similar): https://github.com/GeographicaGS/Docker-GDAL2

[cboettig]

Thanks for the ping, great question!

So when debian:buster (aka debian:10) comes out this summer the latest tags (probably starting with R ~3.6 release given the timing?) will start to build on that instead of stretch, but it looks like buster's gdal is only at 2.4 and buster's libproj is at 5.2.

So, it probably makes sense to build these from source with specific versions. That should keep the Dockerfiles reproducible while giving us the more recent versions (we had to do this on debian:jessie / debian:8 for gdal anyway because the packages version was way too old).

Would you be interested in sending a PR adding these? @edzer's ubuntu-based recipe should probably work out of the box.

I think that would make a lot of sense and provide a good model going forward, so we could bump the versions more frequently (e.g. with the R minor releases). timing could be good if we do this now we could include it on the R-3.5.3 images (starting tomorrow!)

[cboettig]

@Robinlovelace I put the build-from-source recipes for Proj 6.0.0, GEOS 3.7.0, and GDAL 2.4.0 onto the devel Dockerfile; provided DockerHub doesn't time out with all that compiling it should be built as rocker/geospatial:devel (i.e. on R-devel) so we can test it out.

Looks like @edzer still has GDAL at 2.4 instead of 2.5 in his recipe; and maybe the R package isn't ready for 2.5 yet anyway? Note that the above recipe is building from current CRAN version of R packages (via snapshot) and not the GitHub version of sf and friends; and that likely to create some issues. (for instance, looks like CRAN versions of rgdal and sf aren't finding the proj_api.h now? So I guess we may need to creep these version bumps up in step with the CRAN releases of the R packages that bind them. Any advice on that front would be appreciated!

[edzer]

FYI: GDAL 2.5 hasn't been released yet, and will require larger changes from both rgdal and sf. Dev versions of sf, lwgeom and rgdal now all compile against GDAL 2.4 with PROJ 6.

[cboettig]

Thanks @edzer , that's just what I needed. So once the dev versions of sf, lwgeom and rgdal hit CRAN, we'll go ahead and bump rocker/geospatial:latest and rocker/geospatial:devel to build from source with GDAL 2.5 and PROJ 6. (Which will leave 3.5.2 and earlier tags on the older versions, and probably appear in 3.5.3 and later).

Any opinions on GEOS version? Looks like stretch is at 3.5.1 and the move to buster will bring us up to 3.7.1 if we keep installing it from apt-get.

[edzer]

GEOS updates should be fine, nothing drastic coming up.

[Robinlovelace]

Aha, I didn't check for a dev branch, great to see it has one. That pretty much answers my question, and will probably be useful for testing. I would say that yes it would be useful for R packages to be at least the latest CRAN versions in the rocker images devel.

Just tested what I think was the devel image and it didn't seem to have up-to-date versions of either GDAL or sf:

docker run -d -p 8788:8787 -v $(pwd):/home/rstudio/data -e USERID=$UID -e PASSWORD= rocker/geospatial:devel
docker exec -it agitated_pare bash
root@1e9ffe634e35:/# gdal-config --version
2.1.2
root@1e9ffe634e35:/# R

R Under development (unstable) (2019-02-02 r76043) -- "Unsuffered Consequences"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> packageVersion("sf")
[1] ‘0.7.2’

I'm not 100% sure I was using the image represented by that DOCKERFILE you linked to, but will be great to have a more bleeding-edge geospatial image, especially now the main geospatial image is pinned to a MRAN release and therefore (seemingly in my experience) ultra stable although not always up-to-date.

[cboettig]

@Robinlovelace Sorry, just a few clarifications:

Yes, latest and devel build nightly and pull the latest versions of CRAN packages. They still are pinned to MRAN though, so install.packages() by default is gonna install packages from the date the nightly was built. You have to docker pull again to get the new packages (or unpin the MRAN and run update.packages() locally -- though since that involves recompiling it can be slower than pulling a fresh image anyhow).

yeah, devel failed to build because I tried bumping it up to PROJ 6, and as Edzer just confirmed, CRAN versions of sf & rgdal fail to install, so I probably need to keep that at PROV 4 until those hit CRAN. which is to say geospatial:devel won't really be a bleeding-edge image, since I think shipping libraries like proj 6 on geospatial:devel before the CRAN versions of said packages can support them is asking for trouble.

Note that the devel tag just denotes that is has R-devel; in all other ways that tag is meant to be pretty much identical to latest, with R packages coming from CRAN. (This is handy when releasing stuff to CRAN and confirming it works on R-devel, since this matches CRAN's meaning as well.)

[Robinlovelace]

Many thanks for all the clarifications Carl. One follow-on question: how do you unpin MRAN? I had a brief look online and couldn't find a way to do that.

All the rest makes sense and I realise that the geospatial is not designed to be a testbed for developers, but a stable environment for users. And as it's all open source and well-documented, there's nothing stopping anyone from modifying Dockerfiles and putting in pull requests.

I've got the clarifications I was after, and the plan to update GEOS (and PROJ/GDAL) when they are supported on CRAN versions sounds good to me.

[cboettig]

Just do options(repos=c(CRAN="https://cran.rstudio.com")).

Probably want to add that to your .Rprofile to make it persistent; the MRAN repo is pinned in the system .Rprofile in $R_HOME/etc/Rprofile.site I believe.

Thanks again for highlighting this thread, and hopefully we'll see the dev versions hit CRAN so we can bump proj up, and then can bump things again once GEOS 2.5 is released and the packages support it.

[colman-humphrey]

Hey all - really great work with these images and packages.

I'm hoping to use the "nearest" functionality in sf. Works great on my local machine with GEOS 3.6.1. If I'm understanding correctly, once debian:buster makes its way to the r-ver image, it can bubble up to this image?

[cboettig]

@colman-humphrey correct, but meantime did you try rocker/geospatial:devel ? I think you can edit our devel Dockerfile to bump to GEOS 2.5 and it should build..

[colman-humphrey]

Hey @cboettig thanks - went ahead and commented out the ENV GEOS_VERSION 3.7.0 line, and the install lines below it. Working great, thanks!

[josiekre]

Looks like there's commits recently on r-ver in the dev version with debian:buster. Does anyone know what the release timeline looks like for it?

Could really use the GeoJSONSeq driver introduced in GDAL 2.4 in a production environment.

[cboettig]

Thanks for the message. Yup, for new R release, 3.6.2, we'll be on debian:buster for the whole r-ver stack, including geospatial. should be prepped soon I'm just a bit slow due to holidays

[josiekre]

So fast to answer-- thanks! I'll keep my eye out for it.

[cboettig]

@josiekre just a note that rocker/geospatial is now at R 3.6.2, Linking to GEOS 3.7.1, GDAL 2.4.0, PROJ 5.2.0. (as of 10 days ago, but I've been away). Also just an update re @Robinlovelace 's comments above: based on community feedback the latest tag will now point to a standard CRAN mirror (https://cran.rstudio.com) by default and not pin to a rolling MRAN date. Earlier versions (3.6.1 and earlier) still pin to MRAN as always. cheers!

[josiekre]

Thanks @cboettig. I pulled it down the other day and am testing. Does the tagged version (3.6.2) point to a MRAN instead of CRAN? Or does it also install by default from CRAN?

[cboettig]

Should be CRAN also (latest is just an alias for the most recent tag, 3.6.2)

On Thu, Jan 2, 2020 at 8:12 PM Josie Kressner ***@***.***> wrote: Thanks @cboettig <https://github.com/cboettig>. I pulled it down the other day and am testing. Does the tagged version (3.6.2) point to a MRAN instead of CRAN? Or does it also install by default from CRAN? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20?email_source=notifications&email_token=AABWK6XORSBR36AQ45AYBKTQ323LHA5CNFSM4G444DH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIAHNQA#issuecomment-570455744>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABWK6VRYWVYSU4MIH3WURDQ323LHANCNFSM4G444DHQ> .

-- --- Carl Boettiger http://carlboettiger.info/

[Robinlovelace]

based on community feedback the latest tag will now point to a standard CRAN

Very happy about this change. We should update the geocompr docker images in response to this. Heads-up @Nowosad I'm up for giving this a bash next week but happy if you get there first (know you've done a load of dockery things of late in support of efforts to adapt to new versions of GDAL, PROJ and GEOS).

[josiekre]

Darn it-- that actually is bad for reproducible docker containers over time with fixed library versions no matter when you build it. We can't have a production environment depending on a link to CRAN.

I'm sure there is as a way to change this in a Dockerfile built from 3.6.2. Suggestions?

[Robinlovelace]

Good point @josiekre, is there a tag that still points to the pinned MRAN repo @cboettig ? I think devel/latest being CRAN and production or similar being MRAN could provide good options for all users.

[cboettig]

@josiekre Note that prior to the change, the MRAN date was being updated to the build date each night when the image was rebuilt (which also occurred automatically on the hub), so the image didn't really end up version stable until the next release (since we have always opted to pin MRAN date to the last day that the image is current, rather than the first day, as that better matches CRAN's rolling releases model for a user who keeps their system up to date). Once the next version of R is released, 3.6.2 will be frozen (just as before). Or use 3.6.1 which is already frozen. (but until the next release comes out, pointing at rocker/geospatial:3.6.2 is identical to pointing at rocker/geospatial:latest).

However, if you're building the images locally you can still always set the default repo to MRAN, overriding the latest CRAN mirror in the geocompr Dockerfile with whatever your preferred date is.

[josiekre]

Good to know @cboettig. We were using an older tag for so long that we never realized this.

I will add something like this to our Dockerfile to lock it down.