export_workspace: support file copy #676

jdries · 2024-02-06T13:02:17Z

Assume that the workspace is mounted as POSIX directory, then in 'export_workspace' copy the relevant STAC and data files to the right directory.
Workspace metadata can be defined in config file.

bossie · 2024-02-20T11:04:10Z

Needs more work.

Open-EO/openeo-geopyspark-driver#676

I wonder why this worked before. Traceback (most recent call last): File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/flask/app.py", line 870, in full_dispatch_request rv = self.dispatch_request() File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/flask/app.py", line 855, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/users/auth.py", line 88, in decorated return f(*args, **kwargs) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/views.py", line 655, in result result = backend_implementation.processing.evaluate(process_graph=process_graph, env=env) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 301, in evaluate return evaluate(process_graph=process_graph, env=env) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 377, in evaluate result = convert_node(result_node, env=env) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 402, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1563, in apply_process args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1563, in <dictcomp> args = {name: convert_node(expr, env=env) for (name, expr) in sorted(args.items())} File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 416, in convert_node return convert_node(processGraph['node'], env=env) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 402, in convert_node process_result = apply_process(process_id=process_id, args=processGraph.get('arguments', {}), File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1595, in apply_process return process_function(args=ProcessArgs(args, process_id=process_id), env=env) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 2204, in load_stac return env.backend_implementation.load_stac(url=url, load_params=load_params, env=env) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/openeogeotrellis/backend.py", line 1004, in load_stac opensearch_client.add_feature( File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( File "/usr/local/spark/python/pyspark/errors/exceptions/captured.py", line 169, in deco return f(*a, **kw) File "/var/lib/jenkins/workspace/_openeo-geopyspark-driver_PR-687/venv38/lib64/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o24174.add_feature. : java.lang.NullPointerException at java.base/java.net.URI$Parser.parse(URI.java:3104) at java.base/java.net.URI.<init>(URI.java:600) at org.openeo.geotrellis.file.FixedFeaturesOpenSearchClient.$anonfun$add_feature$1(FixedFeaturesOpenSearchClient.scala:28) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286) at scala.collection.Iterator.foreach(Iterator.scala:943) at scala.collection.Iterator.foreach$(Iterator.scala:943) at scala.collection.AbstractIterator.foreach(Iterator.scala:1431) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.TraversableLike.map(TraversableLike.scala:286) at scala.collection.TraversableLike.map$(TraversableLike.scala:279) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.openeo.geotrellis.file.FixedFeaturesOpenSearchClient.add_feature(FixedFeaturesOpenSearchClient.scala:26) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:829)

bossie · 2024-02-29T09:36:56Z

Added disk workspace vdboschj-public-workspace on Terrascope to test:

workspaces = {
    "vdboschj-public-workspace": DiskWorkspace(root_directory=Path("/data/users/Public/vdboschj/workspace"))
}

In this case, it was an integration test with a "discard_result" process. Traceback (most recent call last): File "batch_job.py", line 1339, in <module> main(sys.argv) File "batch_job.py", line 1014, in main run_driver() File "batch_job.py", line 985, in run_driver run_job( File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/utils.py", line 56, in memory_logging_wrapper return function(*args, **kwargs) File "batch_job.py", line 1189, in run_job _export_workspace(result, result_metadata, stac_metadata_dir=job_dir) File "batch_job.py", line 1217, in _export_workspace asset_paths = [Path(asset["href"]) for asset in result_metadata["assets"].values()] KeyError: 'assets'

Traceback (most recent call last): File "batch_job.py", line 1345, in <module> main(sys.argv) File "batch_job.py", line 1014, in main run_driver() File "batch_job.py", line 985, in run_driver run_job( File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/utils.py", line 56, in memory_logging_wrapper return function(*args, **kwargs) File "batch_job.py", line 1189, in run_job _export_workspace(result, result_metadata, stac_metadata_dir=job_dir) File "batch_job.py", line 1224, in _export_workspace stac_paths = _write_exported_stac_collection(stac_metadata_dir, result_metadata) File "batch_job.py", line 1259, in _write_exported_stac_collection item_files = [write_stac_item_file(asset_id, asset) for asset_id, asset in result_metadata["assets"].items()] KeyError: 'assets'

bossie · 2024-03-06T13:59:01Z

This process graph in a batch job will copy the batch job's output assets to subdirectory test_export_workspace/ of test workspace vdboschj-public-workspace *:

{
  "loadcollection1": {
    "process_id": "load_collection",
    "arguments": {
      "id": "PROBAV_L3_S10_TOC_333M",
      "temporal_extent": [
        "2017-11-21",
        "2017-11-22"
      ],
      "spatial_extent": {
        "west": 3.5385,
        "south": 51.3548,
        "east": 3.9869,
        "north": 51.625
      },
      "bands": [
        "NDVI"
      ]
    }
  },
  "saveresult1": {
    "process_id": "save_result",
    "arguments": {
      "data": {
        "from_node": "loadcollection1"
      },
      "format": "GTiff"
    }
  },
  "exportworkspace1": {
    "process_id": "export_workspace",
    "arguments": {
      "data": {
        "from_node": "saveresult1"
      },
      "workspace": "vdboschj-public-workspace",
      "merge": "test_export_workspace"
    },
    "result": true
  }
}

* maps to directory /data/users/Public/vdboschj/workspace/test_export_workspace

jdries assigned soxofaan and bossie Feb 8, 2024

bossie added a commit that referenced this issue Feb 19, 2024

simplify batch job results logic #676

f90b5ab

bossie linked a pull request Feb 20, 2024 that will close this issue

simplify batch job results logic #676 #685

Merged

bossie added a commit that referenced this issue Feb 20, 2024

add test #676

d0956f4

bossie closed this as completed in #685 Feb 20, 2024

bossie added a commit that referenced this issue Feb 20, 2024

simplify batch job results logic #676

bbc095f

bossie reopened this Feb 20, 2024

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 21, 2024

reference process description

2dd0e37

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 21, 2024

support export_workspace

5f4cc9b

Open-EO/openeo-geopyspark-driver#676

bossie mentioned this issue Feb 21, 2024

676 export workspace support file copy Open-EO/openeo-python-driver#262

Merged

bossie linked a pull request Feb 21, 2024 that will close this issue

676 export workspace support file copy Open-EO/openeo-python-driver#262

Merged

bossie added a commit that referenced this issue Feb 21, 2024

export assets to disk #676

9324d9d

bossie mentioned this issue Feb 21, 2024

export assets to disk #676 #687

Merged

bossie linked a pull request Feb 21, 2024 that will close this issue

export assets to disk #676 #687

Merged

bossie added a commit that referenced this issue Feb 22, 2024

export load_stac'able STAC Collection #676

837f23f

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 23, 2024

work around save_result not returning a SaveResult

b0a1a22

Open-EO/openeo-geopyspark-driver#676

bossie added a commit that referenced this issue Feb 23, 2024

test asset files can be downloaded #676

08c2dfc

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 26, 2024

revert submodule update in favor of copy

b1f9c83

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 26, 2024

test export_workspace process

bf08f85

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 27, 2024

test SaveResult changes

6eb3e44

Open-EO/openeo-geopyspark-driver#676

tcassaert pushed a commit that referenced this issue Feb 27, 2024

simplify batch job results logic #676

4aaf55d

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 27, 2024

more SaveResult tests

5baa423

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 27, 2024

test DiskWorkspace

93142bb

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 27, 2024

test DiskWorkspace actually supports "."

d7edd20

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 27, 2024

refactor: introduce WorkspaceRepository

0d023bf

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 28, 2024

refactor: use dataclass iso/ dict

fd6f913

Open-EO/openeo-geopyspark-driver#676

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 28, 2024

update version and CHANGELOG

98011b9

Open-EO/openeo-geopyspark-driver#676

bossie closed this as completed in Open-EO/openeo-python-driver#262 Feb 28, 2024

bossie added a commit to Open-EO/openeo-python-driver that referenced this issue Feb 28, 2024

support export_workspace to disk

94b9cab

Open-EO/openeo-geopyspark-driver#676

bossie added a commit that referenced this issue Feb 28, 2024

reference export_workspace capable Python driver #676

ce75f53

bossie added a commit that referenced this issue Feb 28, 2024

adapt CHANGELOG #676

cdc1be0

bossie added a commit that referenced this issue Feb 29, 2024

support export_workspace in batch jobs #676

763947c

bossie mentioned this issue Mar 6, 2024

support export_workspace to static STAC #606

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

export_workspace: support file copy #676

export_workspace: support file copy #676

jdries commented Feb 6, 2024 •

edited

Loading

bossie commented Feb 20, 2024

bossie commented Feb 29, 2024 •

edited

Loading

bossie commented Mar 6, 2024

export_workspace: support file copy #676

export_workspace: support file copy #676

Comments

jdries commented Feb 6, 2024 • edited Loading

bossie commented Feb 20, 2024

bossie commented Feb 29, 2024 • edited Loading

bossie commented Mar 6, 2024

jdries commented Feb 6, 2024 •

edited

Loading

bossie commented Feb 29, 2024 •

edited

Loading