Implement 'clip and ship' services

[gubuntu]

Implement 'clip and ship' services for a selected layer or map (multiple layers) where the user chooses an AOI and gets a zip download containing all the data + metadata for the clip area.

• must support Implement 'clip and ship' services QGIS#5
• must address and close Data export based on extent GeoNode/geonode#889

Further technical requirements

• client to construct WFS or WCS requests against GeoNode API, which will proxy these to GeoServer or QGIS backend. i.e. must be backend independent

Proposed Interaction

• Add this to the download functionality in the GeoNode layer view
• the default download AOI is currently the entire layer extent. This should remain the case.
• If the user zooms and pans to something different than the full layer extent, then the default download AOI is the intersection of the layer extent and the view extent
• future enhancements (out of scope now):
• allow the user to draw a rectangle
• allow the user to draw a polygon (in which case we need to pass the BB and AOI/mask)
• allow the user to select a polygon from another layer as the clip AOI
• allow the user the select the extent of another layer as the AOI
• allow the user to upload a polygon to use as the AOI

cc @cchristelis @lucernae

Standards compliance

According to QGIS docs, 2.14 supports WFS 1.0.0 and WCS 1 1.1.1 so our client must construct compliant requests.

Look on the OGC website for specifications matching these protocol versions if you need a reference for constructing requests.

In future it can be extended to query the backend and respond with requests that match supported version. e.g. Geoserver supports more recent versions of WFS and WCS and future QGIS versions might also.

child of #457

[meomancer]

hi @gubuntu
for clipping work, is it using wfs and wcs?
could you give example or documentation for how to do clipping?

or could you give advice how to clip?

[gubuntu]

The 'proposed interaction' in the OP describes how to clip from a UX point of view. Use the extents and user choices to construct WFS (vector) or WCS (raster) requests against the GeoNode OWS base url, which will be proxied to the Geoserver or QGIS backend.

Ensure you comply with the specification version that the backend supports (e.g. WCS 1.1 vs 1.3)

A resource I prepped for raster WCS clipping a while ago (only accessible to Kartoza staff; links contained in it might be out of date): https://docs.google.com/a/kartoza.com/document/d/10DBcHC4tUm1gmd-u38LkhOeWDlejm21ZfdcKTWh5yqU/edit?usp=sharing

WFS vector clipping follow similar principles.

[dimasciput]

For the front end, is something like this will be enough?

[gubuntu]

looking good. In your mockup do you click 'download clip' and then draw a polygon?

See the OP - the first step is to get the clip working by sending the view extent and using default resolution and format for the response.

Sending a polygon (AOI) is more complex and remember in that case that the returned raster will still be a rectangle, but with cells outside the AOI set to nodata.

[gubuntu]

Google Leaflet support for WFS and WCS, there seem to be a few projects out there. So you don't have to construct the requests from first principles... It might work getting Leaflet to construct the requests, although we want to download the result, not render it in the map...

[Gustry]

And we want to clip the result with an irregular polygon after the WCS on the server.

[gubuntu]

Is it a specific requirement to support irregular polygon clipping?

WCS 1.1.1 supports only bounding box clip.

So as @Gustry says, we need to intersect the rectangular image and clip with a polygon. some possibilities:

• gdal
• PyQGIS
• WPS
• rasterio

[Gustry]

We talked about:

• gdal: partly included in geodjango, so available whatever the backend (geoserver or qgis server). We might need an access to the full gdal API, not only the one provided in geodjango
• pyqgis: we try to make it work with geoserver too, so this one is not best
• WPS: We talked about this one. It will work on both geoserver and qgis (only the name of the remote algorithm and parameters will be different)

[gubuntu]

A suggestion: we pass the custom polygon as an extension to the WCS 1.1 request in the same way as WCS 2. For now we intercept that and remove it from the request behind the proxy so that the custom process can use it, but later when QGIS supports WCS 2 then we won't need to change the request much?

[meomancer]

hi @gubuntu
i tried WCS, on qgis-server
like this:

http://192.168.100.10/qgis-server/ogc/?CRS=EPSG%3A4326&RESX=0.5&RESY=0.5&SERVICE=WCS&FORMAT=GEOTIFF&REQUEST=getcoverage&MAP=%2Fusr%2Fsrc%2Fapp%2Fgeonode%2Fqgis_layer%2Ftanzania_nsunga_earthquake_1_LJ3K4j4.qgs&VERSION=1.0.0&BBOX=1.1644706071806057%2C35.738525390625%2C-3.228271011252635%2C27.498779296875&COVERAGE=geonode:mdg10adjv4&VERSION=1.0.0%3C

but COVERAGE=geonode:mdg10adjv4 is not found.

<ServiceExceptionReport xmlns="http://www.opengis.net/ogc" version="1.3.0">
<ServiceException code="RequestNotWellFormed">
The layer for the COVERAGE 'geonode:mdg10adjv4' is not found
</ServiceException>
</ServiceExceptionReport>

Is the request correct? (using like docs)

if it is correct, where can i get the coverage layer? in the frontend, how do we use this coverage parameter?

[Gustry]

Did you create a new project with the OTF plugin latest version?
Can you check the GetCapabilities if you see your layer?

[meomancer]

hi @Gustry
yes, it is using latest otf-plugin (checked to container that create-project filter is removed) and i create project using that
the layer is shown

how do i check GetCapabilities? because i get GetCapabilities is not supported yet.

[meomancer]

@Gustry
UPDATE:
i can see GetCapabilities

http://192.168.100.10/qgis-server/ogc/?CRS=EPSG%3A4326&RESX=0.5&RESY=0.5&SERVICE=WCS&FORMAT=GEOTIFF&REQUEST=GetCapabilities&LAYERS=tanzania_nsunga_earthquake_1&MAP=%2Fusr%2Fsrc%2Fapp%2Fgeonode%2Fqgis_layer%2Ftanzania_nsunga_earthquake_1_LJ3K4j4.qgs&VERSION=1.0.0&BBOX=1.1644706071806057%2C35.738525390625%2C-3.228271011252635%2C27.498779296875&VERSION=1.0.0%3C

is the coverage that i use is from CoverageOfferingBrief.name? which is tanzania_nsunga_earthquake_1?

i tried using that coverage, but got 500, internal server error

http://192.168.100.10/qgis-server/ogc/?CRS=EPSG%3A4326&RESX=0.5&RESY=0.5&SERVICE=WCS&FORMAT=GEOTIFF&REQUEST=GetCoverage&MAP=%2Fusr%2Fsrc%2Fapp%2Fgeonode%2Fqgis_layer%2Ftanzania_nsunga_earthquake_1_LJ3K4j4.qgs&LAYERS=tanzania_nsunga_earthquake_1&VERSION=1.0.0&BBOX=1.1644706071806057%2C35.738525390625%2C-3.228271011252635%2C27.498779296875&COVERAGE=tanzania_nsunga_earthquake_1&VERSION=1.0.0%3C

[gubuntu]

@meomancer @dimasciput is this up to date? Where is all the latest Haiti clip'n ship activity happening?

[Gustry]

Sorry @meomancer Do you still have the issue? I missed your answer

[meomancer]

hi @gubuntu
here is the latest PR for ship and clip
meomancer#1
this PR is finshed, but mostly we convert all of code to haiti repo, in here:
HaitiData/GeoNode#1

@Gustry
We already overcome the issue. We are using gdal to ship and clip instead of using the qgis-server. But at the moment, it is available just for raster.

[gubuntu]

test at http://88.99.166.60:33300

[gubuntu]

@meomancer can you do a PR into this repo as well? cc @cchristelis

So this just supports clip 'n ship with QGIS server backend (not geoserver)?

[meomancer]

hi @gubuntu
i think the PR already there : meomancer#1
the clip and ship just use gdal, so it is support for qgis server and geoserver, unless gdal is installed

[gubuntu]

@meomancer cc @cchristelis please wrap this up asap

[gubuntu]

ping @meomancer cc @cchristelis

[gubuntu]

@dimasciput will try to merge this as a back-end-agnostic GeoNode tool into KBIMS, then we should automatically get it here

[dimasciput]

@gubuntu
I don't know if I can do that, might need help from @lucernae , because currently we use default geonode structure from https://github.com/geosolutions-it/geonode-generic