integrate CLMS HRL VPP

[jdries]

https://land.copernicus.eu/pan-european/biophysical-parameters/high-resolution-vegetation-phenology-and-productivity
problem here is that there's one collection, with multiple producttypes, and each type has a single band
this should become one collection with multiple bands

[jdries]

@JohanKJSchreurs This is a good candidate to test our new STAC api.
This collection already exists in opensearch catalog as well, mostly a matter of porting metadata to STAC.

[jdries]

Collection id: copernicus_r_3035_x_m_hrvpp-vpp_p_2017-now_v01_openeo

Example collection metadata that we would also target:
https://collections.eurodatacube.com/stac/vegetation-phenology-and-productivity-parameters-season-1.json

Python code to get the product metadata:
https://github.com/eea/clms-hrvpp-tools-python/blob/main/HRVPP_opensearch_demo/HRVPP%20catalogue%20and%20download%20demo.ipynb

Opensearch collections:
https://phenology.hrvpp2.vgt.vito.be/collections

[jdries]

Example url for getting S3 links:
https://phenology.hrvpp2.vgt.vito.be/products.geojson?collection=copernicus_r_3035_x_m_hrvpp-vpp_p_2017-now_v01&accessedFrom=S3

[JohanKJSchreurs]

We are implementing this in the stac-catalog-builder project.

The issue linked below is the main one that contains a breakdown of the parts/features we need for this integration
VitoTAP/stac-catalog-builder#16

[JohanKJSchreurs]

The implementation in the stac-catalog-builder is complete: VitoTAP/stac-catalog-builder#16

Some small improvements can still be done, to update the collections & items in the STAC API with improved information, but those should be separate GH issues.

At present, three of the VPP collections have been converted and uploaded now to the development environment of the terra-stac-api at VITO.
The largest collection has many products (6.5 million) and we will need to download that in several pieces because the process runs too long.

[JohanKJSchreurs]

Fourth collection has also been uploaded to the STAC API.

Overview:

collection	download + conversion	upload to dev STAC API	number of products	number of STAC Items
copernicus_r_3035_x_m_hrvpp-st_p_2017-now_v01	done	done	388_008	194_004
copernicus_r_3035_x_m_hrvpp-vpp_p_2017-now_v01	done	done	150_849	10_778
copernicus_r_utm-wgs84_10_m_hrvpp-st_p_2017-now_v01	done	done	470_066	235_058
copernicus_r_utm-wgs84_10_m_hrvpp-vi_p_2017-now_v01	did not finish (process got killed)	TO DO	6_564_209	unkown at present
copernicus_r_utm-wgs84_10_m_hrvpp-vpp_p_2017-now_v01	done	done	182_784	13_056

[JohanKJSchreurs]

How we can solve the long download of the large collection:

We can add some options to the command to tell it what time slice to download.
Right now it tries to download the entire collection, that is tot say the entire period.

We already divide that entire period into smaller time slots in order to limit the number of products in each query to a reasonable number. So if we add options for a start and end date then we could do a partial download.
That way we can test just upload and test the collection with a more limited set of STAC items

Furthermore, with same additional work we could download the whole collection in several parts, and in each run save out the STAC items for just those slices.
We can already upload upload the STAC items in several parts or sets.
At present, the collection.json file would be created and overwritten every time on disk when you do a partial download, but the collection file would still have the same data anyway. (It does not link to its STAC items in this case because there are far too many items for a static STAC collection). However with a little extra work we could split up the command so one command downloads/creates the "empty" collection and another command downloads/creates the STAC items.

[VictorVerhaert]

Update:
Trying out some of the suggestions johan gave to build de last collection. So far I always encounter an error stating that a request is to large.

I have been trying to load the uploaded stac collections in openeo, without luck.
Even with the isoformat fix on openeo dev I still encounter an IllegalArgument exception:
j-240321bb64b14251b296ace46577d0b4. @bossie could you have another look at this?

[VictorVerhaert]

It seems that the eo:bands are missing from the stac items, which is needed for openeo.

[JeroenVerstraelen]

• STAC API does not work with openeo yet
  • eo:bands are missing

Will take 2 weeks of debugging.

We already have 4/5 collections but does require someone to help testing of STAC API and integration with openEO.

[VictorVerhaert]

I'll keep editing this column as bugs are found.

I ran some different load_stac's from different sources, as I noticed the errors I encounter often differ.
tests ran on https://openeo.vito.be/:

Stac api	collection	job id	result	stac request url
https://stac.terrascope.be	terrascope-s2-toc-v2	j-240325bbab144c388897826ebd12a00f	java.lang.IllegalArgumentException: requirement failed: Server doesn't support ranged byte reads	https://stac.terrascope.be/search?limit=20&bbox=5.0%2C51.2%2C5.01%2C51.21&datetime=2017-06-01T00%3A00%3A00Z%2F2017-07-29T23%3A59%3A59.999000Z&collections=terrascope-s2-toc-v2
https://stac.terrascope.be	terrascope-s2-ndvi-v2	j-2403251d212041a69f875afcb6476848	error without error log in the editor
https://stac-openeo-dev.vgt.vito.be	copernicus_r_utm-wgs84_10_m_hrvpp-st_p_2017-now_v01	j-240325eb608247938a2ef79efbf66f0d	java.lang.IllegalArgumentException: requirement failed	https://stac-openeo-dev.vgt.vito.be/search?limit=20&bbox=5.0%2C51.2%2C5.01%2C51.21&datetime=2017-06-01T00%3A00%3A00Z%2F2017-07-29T23%3A59%3A59.999000Z&collections=copernicus_r_utm-wgs84_10_m_hrvpp-st_p_2017-now_v01
https://stac-openeo-dev.vgt.vito.be	TEST_Landsat_three-annual_NDWI_v1	j-240325604db14055b31b93ab58894870	OpenEOApiException(status_code=400, code='NoDataAvailable', message='There is no data available for the given extents.', id='no-request')	https://stac-openeo-dev.vgt.vito.be/search?limit=20&bbox=5.0%2C51.2%2C5.01%2C51.21&datetime=2000-06-01T00%3A00%3A00Z%2F2000-07-29T23%3A59%3A59.999000Z&collections=TEST_Landsat_three-annual_NDWI_v1

comments:

• the https://stac.openeo.vito.be/api.html source contains no collections and cannot be tested
• (third line) the copernicus hrvpp-st_p collection shows no eo:bands property for its assets. I have yet to discover why these were not included in the upload. Only the name property is used from eo:bands, so a quickfix might be to fall back on another name field on openeo's side
• the TEST_landsat... collection is the only one containing eo:bands information on the stac-openeo-dev source

[bossie]

FYI, this error when querying https://stac.terrascope.be:

java.lang.IllegalArgumentException: requirement failed: Server doesn't support ranged byte reads

is because the underlying assets require authentication (if you click an asset's href in your browser, it will redirect you to a login page; it's this response that doesn't have an Accept-Ranges: bytes header, hence the error).

[VictorVerhaert]

Update: S3 seems to work on CDSE
This leaves the following TODO's for the integration:

• mount buckets on terrascope (optional, should the collections be required there)
• test full collection on CDSE (WIP)
• rebuild al the collections with s3 links and upload to https://stac-openeo.vgt.vito.be/api.html#/ (production stac api, currently empty)
• adjust stac-builder to build largest collection without OOM (WIP)

[VictorVerhaert]

The following collection are now available on https://stac.openeo.vito.be/ and should work on CDSE-staging

• copernicus_r_3035_x_m_hrvpp-vpp_p_2017-now_v01
• copernicus_r_utm-wgs84_10_m_hrvpp-vpp_p_2017-now_v01
• copernicus_r_utm-wgs84_10_m_hrvpp-st_p_2017-now_v01
• copernicus_r_3035_x_m_hrvpp-st_p_2017-now_v01

I am still trying to build copernicus_r_utm-wgs84_10_m_hrvpp-vi_p_2017-now_v01 but making good progress on this by speeding up the pipeline with thread and process pools, with memory cleanup.

[JeroenVerstraelen]

@bossie Define collections that are stac based (layercatalog). So load_collection has to call load_stac.

[VictorVerhaert]

important note:
The way the stac_api works, only items of which the datetime property (often equal to the start_datetime) lies within the temporal_extent of load_stac are loaded.
For yearly assets, this means that the 1st of january needs to be included in the temporal_extent for it to be loaded.

[bossie]

Note: the assets are in the HRVPP bucket on S3 endpoint http://data.cloudferro.com (not externally accessible).

[bossie]

@VictorVerhaert copernicus_r_utm-wgs84_10_m_hrvpp-vpp_p_2017-now_v01 can be reingested as filtering by property (in this case: "season") should work without adverse side-effects (in this case: empty results).

I did notice that proj:epsg and proj:bbox are both in 4326, whereas the actual assets are in UTM so this might be something that you want to include e.g. https://stac.openeo.vito.be/search?limit=20&bbox=5.0%2C51.2%2C5.01%2C51.21&datetime=2017-07-01T00%3A00%3A00Z%2F2018-07-30T23%3A59%3A59.999000Z&collections=copernicus_r_utm-wgs84_10_m_hrvpp-vpp_p_2017-now_v01&fields=%2Bproperties

For some reason loading these collections seems to take a really long time, much longer than I remember. 🤔 Maybe something similar to #250?

[bossie]

TODO: incorporate property filters defined in creo_layercatalog.json to support collections per season.

[VictorVerhaert]

copernicus_r_utm-wgs84_10_m_hrvpp-vpp_p_2017-now_v01 has been reuploaded with the season property. @bossie

[bossie]

Confirmed: works (for "s1" and "s2"):

data_cube = (connection
             .load_collection(collection_id, bands=["SPROD", "TPROD", "QFLAG"], properties={"season": lambda s: s == "s1"})
             .filter_temporal(["2017-07-01", "2018-07-31"])
             .filter_bbox([5.00, 51.20, 5.01, 51.21])
             .save_result("GTiff"))

[jdries]

commited collection config for seasonal collections

[bossie]

Still needs work wrt/ bands order defined in creo_layercatalog.json.

Adapt related test:

openeo-geopyspark-driver/tests/test_load_collection.py

Line 495 in 286aff2

bands=["SPROD", "TPROD"]) # TODO: remove other bands from layercatalog.json, then drop this bands argument