Temporal extent is null for vectorcube STAC items #852

VincentVerelst · 2024-08-30T10:45:05Z

When downloading vectorcube assets like CSV or Parquet from OpenEO, the associated STAC items' temporal interval is null. Here is an example (ran on Terrascope backend):
item-example.json

This makes it so that libraries like pystac don't consider this a valid STAC items and will throw errors.

bossie · 2024-09-06T07:50:33Z

From the item spec re: datetime:

null is allowed, but requires start_datetime and end_datetime from common metadata to be set.

bossie · 2024-09-06T07:53:26Z

@VincentVerelst just wondering: are you able to open this timeseries.parquet file? I'm getting this error:

shaded.org.apache.avro.SchemaParseException: Illegal character in: S1-SIGMA0-VV

It's probably the two dashes it's complaining about.

bossie · 2024-09-06T11:23:25Z

no temporal bounds e.g. load_stac/temporal_extent anywhere in the process graph,
⬇️
no temporal_extent source constraints in the dry run,
⬇️
start_datetime and end_datetime are both None for these results,
⬇️
datetime is None as well.
💥

This means that whether or not the result is a vector cube is irrelevant.

@VincentVerelst this can be worked around by passing an arbitrarily large temporal_extent to one of those load_stac processes e.g. ["1970-01-01", "2070-01-01"]

Open-EO/openeo-geopyspark-driver#852

VincentVerelst · 2024-09-10T11:36:19Z

@VincentVerelst just wondering: are you able to open this timeseries.parquet file? I'm getting this error:

shaded.org.apache.avro.SchemaParseException: Illegal character in: S1-SIGMA0-VV

It's probably the two dashes it's complaining about.

What are you using to open the file? For me it works using GeoPandas (which itself uses PyArrow), but I think it's because it ignores the schema validation.

bossie · 2024-09-11T07:30:04Z

I was using the Java Parquet CLI; works with GeoPandas though 👍

Open-EO/openeo-geopyspark-driver#852

#852

Open-EO/openeo-geopyspark-driver#852

#852

Open-EO/openeo-geopyspark-driver#852

#852

:vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_List_iterator< unsigned long> > >, std::_Select1st<std::pair<unsigned long const, std::pair<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char , std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_List_iterator<unsigned long> > > >, std::less<unsigne d long>, std::allocator<std::pair<unsigned long const, std::pair<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_ traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_List_iterator<unsigned long> > > > >::find(unsigned long const&)+ 0x11 /core.366) #852

Open-EO/openeo-geopyspark-driver#852

#852

Open-EO/openeo-geopyspark-driver#852

* be explicit about which metadata Open-EO/openeo-geopyspark-driver#852 * expose start/end_datetime from asset metadata Open-EO/openeo-geopyspark-driver#852 * update CHANGELOG Open-EO/openeo-geopyspark-driver#852 * update version and CHANGELOG Open-EO/openeo-geopyspark-driver#852

#852

#852 #861

bossie · 2024-09-16T07:57:54Z

Much like e.g. GeoTiff Items provide a datetime property in their asset metadata regardless of the temporal_extent source constraints in the process graph, this will populate start_datetime and end_datetime properties.

* propagate temporal extent from load_stac #852 * fix tests #852 * add test #852 * quick fix for new test crashing the JVM when running all tests :vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_List_iterator< unsigned long> > >, std::_Select1st<std::pair<unsigned long const, std::pair<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char , std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_List_iterator<unsigned long> > > >, std::less<unsigne d long>, std::allocator<std::pair<unsigned long const, std::pair<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_ traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_List_iterator<unsigned long> > > > >::find(unsigned long const&)+ 0x11 /core.366) #852 * reference fixed test data #852 * debug KeyError: start_datetime #852 * reference openeo_driver with fix #852 * simplify filter_temporal #852 * DRY: move GDALWarp.deinit() elsewhere #852 * merge_cubes: merge temporal extents #852 #861

#852

Traceback (most recent call last): File "batch_job.py", line 530, in start_main main(sys.argv) File "batch_job.py", line 206, in main run_driver() File "batch_job.py", line 167, in run_driver run_job( File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/utils.py", line 56, in memory_logging_wrapper return function(*args, **kwargs) File "batch_job.py", line 285, in run_job result = ProcessGraphDeserializer.evaluate(process_graph, env=env, do_dry_run=tracer) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 428, in evaluate result = convert_node(top_level_node, env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in apply_process args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in <dictcomp> args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 485, in convert_node return [convert_node(x, env=env) for x in processGraph] File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 485, in <listcomp> return [convert_node(x, env=env) for x in processGraph] File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in apply_process args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in <dictcomp> args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 471, in convert_node return convert_node(processGraph['node'], env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1676, in apply_process the_mask = convert_node(mask_node, env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 471, in convert_node return convert_node(processGraph['node'], env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in apply_process args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in <dictcomp> args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 471, in convert_node return convert_node(processGraph['node'], env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in apply_process args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in <dictcomp> args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 471, in convert_node return convert_node(processGraph['node'], env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in apply_process args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1695, in <dictcomp> args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 471, in convert_node return convert_node(processGraph['node'], env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 453, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1729, in apply_process return process_function(args=ProcessArgs(args, process_id=process_id), env=env) File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1333, in filter_temporal return cube.filter_temporal(start=extent[0], end=extent[1]) File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/geopysparkdatacube.py", line 77, in run return func(*args, **kwargs) File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/geopysparkdatacube.py", line 154, in filter_temporal metadata=self.metadata.filter_temporal(start, end) File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/geopysparkcubemetadata.py", line 78, in filter_temporal raise ValueError(start, end) ValueError: ('2018-08-14', '2018-08-14') eu-cdse/openeo-cdse-infra#261 #852

bossie · 2024-09-17T14:03:10Z

Fixed on https://openeo-dev.vito.be; the workaround is no longer necessary @VincentVerelst.

JeroenVerstraelen assigned bossie Sep 2, 2024

bossie added the bug label Sep 6, 2024

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Sep 9, 2024

default start/end_datetime for STAC Items

360ba2d

Open-EO/openeo-geopyspark-driver#852

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Sep 11, 2024

be explicit about which metadata

c285fc4

Open-EO/openeo-geopyspark-driver#852

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Sep 12, 2024

expose start/end_datetime from asset metadata

ab4b22e

Open-EO/openeo-geopyspark-driver#852

bossie added a commit that referenced this issue Sep 12, 2024

propagate temporal extent from load_stac

465b233

#852

bossie mentioned this issue Sep 12, 2024

expose start/end_datetime from asset metadata Open-EO/openeo-python-driver#311

Merged

bossie linked a pull request Sep 12, 2024 that will close this issue

expose start/end_datetime from asset metadata Open-EO/openeo-python-driver#311

Merged

bossie mentioned this issue Sep 12, 2024

propagate temporal extent from load_stac #864

Merged

bossie linked a pull request Sep 12, 2024 that will close this issue

propagate temporal extent from load_stac #864

Merged

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Sep 12, 2024

update CHANGELOG

4727c5e

Open-EO/openeo-geopyspark-driver#852

bossie added a commit that referenced this issue Sep 12, 2024

fix tests

1f251f0

#852

bossie mentioned this issue Sep 12, 2024

merge_cubes: The merge_cubes function doesn't merge the temporal range of the previous cube. #861

Closed

bossie added a commit to Open-EO/openeo-geopyspark-driver-testdata that referenced this issue Sep 13, 2024

add test files

92d2aa5

Open-EO/openeo-geopyspark-driver#852

bossie added a commit that referenced this issue Sep 13, 2024

add test

07305d9

#852

bossie added a commit to Open-EO/openeo-geopyspark-driver-testdata that referenced this issue Sep 13, 2024

fix test files

ffd293c

Open-EO/openeo-geopyspark-driver#852

bossie added a commit that referenced this issue Sep 13, 2024

reference fixed test data

35d0e5c

#852

bossie added a commit that referenced this issue Sep 13, 2024

debug KeyError: start_datetime

da53335

#852

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Sep 13, 2024

update version and CHANGELOG

55685a5

Open-EO/openeo-geopyspark-driver#852

bossie closed this as completed in Open-EO/openeo-python-driver#311 Sep 13, 2024

bossie added a commit that referenced this issue Sep 13, 2024

reference openeo_driver with fix

caef851

#852

bossie added a commit that referenced this issue Sep 13, 2024

simplify filter_temporal

56ef03f

#852

bossie added a commit that referenced this issue Sep 16, 2024

DRY: move GDALWarp.deinit() elsewhere

a47292b

#852

bossie added a commit that referenced this issue Sep 16, 2024

merge_cubes: merge temporal extents

d195916

#852 #861

bossie mentioned this issue Sep 16, 2024

default start/end_datetime for STAC Items Open-EO/openeo-python-driver#312

Closed

bossie linked a pull request Sep 16, 2024 that will close this issue

default start/end_datetime for STAC Items Open-EO/openeo-python-driver#312

Closed

bossie added a commit that referenced this issue Sep 16, 2024

cleanup

72df128

#852

bossie mentioned this issue Sep 20, 2024

aggregate_temporal_period after load_stac gives unexpected results #855

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Temporal extent is null for vectorcube STAC items #852

Temporal extent is null for vectorcube STAC items #852

VincentVerelst commented Aug 30, 2024

bossie commented Sep 6, 2024

bossie commented Sep 6, 2024

bossie commented Sep 6, 2024

VincentVerelst commented Sep 10, 2024

bossie commented Sep 11, 2024

bossie commented Sep 16, 2024

bossie commented Sep 17, 2024

Temporal extent is null for vectorcube STAC items #852

Temporal extent is null for vectorcube STAC items #852

Comments

VincentVerelst commented Aug 30, 2024

bossie commented Sep 6, 2024

bossie commented Sep 6, 2024

bossie commented Sep 6, 2024

VincentVerelst commented Sep 10, 2024

bossie commented Sep 11, 2024

bossie commented Sep 16, 2024

bossie commented Sep 17, 2024