allow empty datacubes #649

jdries · 2024-01-23T15:29:05Z

We have checks like this one:
https://github.com/Open-EO/openeo-geotrellis-extensions/blob/68143219e6882dffd0d05a46aaf29e530c0b93a5/openeo-geotrellis/src/main/scala/org/openeo/geotrellis/layers/FileLayerProvider.scala#L974
To avoid empty datacubes. There are however use cases where this is not desirable, because the workflow can deal with the absence of data.

Stack trace in sentinelhub:

rg.openeo.geotrellissentinelhub.package$NoSuchFeaturesException: no features found for criteria:
collection ID "sentinel-1-grd"
1 polygon(s)
[2021-04-12T00:00:00+00:00, 2022-09-18T00:00:00+00:00)
metadata properties {sat:orbit_state={eq=ASCENDING}, polarization={eq=DV}}
at org.openeo.geotrellissentinelhub.PyramidFactory.datacube_seq(PyramidFactory.scala:414)

Open-EO/openeo-geopyspark-driver#649

#649

jdries · 2024-01-24T06:53:28Z

load_collection -> featureflags -> allow_empty_cube

soxofaan · 2024-01-24T11:52:47Z

@m-mohr do you think it makes sense to start a discussion about the concept of an empty datacube on the level of openeo-api or openeo-processes ?

To roughly sketch the use case we are trying to address here:
user that loads S1 ASCENDING and DESCENDING separately and after that wants to pick the cube with most observations (e.g. with an if based on count of temporal labels). Sometimes it can happen that either of ASC or DESC has no observations at all for the spatio-temporal extent, which at the moment raises something like an "NoDataAvailable" error, failing the whole job. Some kind of empty cube (with zero observations or temporal labels), could be a workaround.

jdries · 2024-01-24T11:59:26Z

Update from the user: on our CDSE backend, the usecase actually does work. It indeed generates an empty datacube, but the processes used by that case did not crash on it.

One aspect that the spec could help with: what should we generate when we write an empty datacube to a file? Only stac metadata and no asset, or instead error the job? (We'll now error in most cases.)

m-mohr · 2024-01-24T18:03:52Z

Generally openEO doesn't disallow empty data cubes, we just felt it's more user-friendly to report no data in load_collection directly.

soxofaan · 2024-06-03T14:14:50Z

Another use case (related to #785):

user wants to download a merged cube of Sentinel2 and Sentinel1 bands.
if any of the source has no data in the requested spatiotemporal extent (possibly combined with cloud filtering), it is fine to just assign no-data values to the corresponding bands

This currently is not possible because because the NoDataAvailable is thrown too eagerly from load_collection

EmileSonneveld · 2024-06-04T10:13:15Z

@Pratichhya , can you provide a minimal case that gives this error?

EmileSonneveld · 2024-06-05T12:30:24Z

Apearantly the graphs here are a good example already: #785 (comment)

JeroenVerstraelen · 2024-06-17T14:05:15Z

Investigate if running this on CDSE could be a good workaround. Contact Dennis. @HansVRP
Adjust sentinelhub collection to return an empty datacube. Empty RDD with dummy metadata. This already works on CDSE. @bossie

Job option would be best. Then we don't need to adhere to the specification.

bossie · 2024-06-19T12:08:55Z

This already works on CDSE. @bossie

i.e. the SENTINEL1_GRD collection.

HansVRP · 2024-06-19T15:01:03Z

Based on a discussion with Dennis, we are in favor of running it on CDSE.

Yet, as the customer was attracted through Terrascope and has Terrascope credits we will need to evaluate how to handle it.

@Pratichhya Could we test cropsar1D on CDSE for these particular fields?

bossie · 2024-06-20T12:04:07Z

As discussed, the allow_empty_cube feature flag was already implemented so ideally this is just a matter of passing it as a parameter to the CropSAR process: TBC.

bossie · 2024-06-20T12:18:34Z

@Pratichhya, @HansVRP, locally I've managed to run the process graph with the 6 polygons succesfully:

{
  "process_graph": {
    "CropSAR1": {
      "process_id": "CropSAR",
      "arguments": {
        "biopar_type": "FCOVER",
        "date": [
          "2023-01-01",
          "2023-02-01"
        ],
        "minimal_input_checks": true,
        "polygon": {
          "type": "FeatureCollection",
          "features": [
            {
              "id": "0",
              "type": "Feature",
              "properties": {
                "*Field_Area": "60.7104355585474",
                "Area_": "60.7104355585474",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-06-",
                "ENCLOSED_AREA": "60.607",
                "Errors": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.22650350455383,
                      44.279303122758215
                    ],
                    [
                      -89.2265108460764,
                      44.27925677020906
                    ],
                    [
                      -89.22653215200468,
                      44.27921495497037
                    ],
                    [
                      -89.226565336766,
                      44.27918177020906
                    ],
                    [
                      -89.22660715200468,
                      44.27916046428077
                    ],
                    [
                      -89.22665350455384,
                      44.27915312275822
                    ],
                    [
                      -89.226699857103,
                      44.27916046428077
                    ],
                    [
                      -89.22674167234167,
                      44.27918177020906
                    ],
                    [
                      -89.226774857103,
                      44.27921495497037
                    ],
                    [
                      -89.22679616303128,
                      44.27925677020906
                    ],
                    [
                      -89.22680350455384,
                      44.279303122758215
                    ],
                    [
                      -89.22679616303128,
                      44.27934947530737
                    ],
                    [
                      -89.226774857103,
                      44.27939129054606
                    ],
                    [
                      -89.22674167234167,
                      44.27942447530737
                    ],
                    [
                      -89.226699857103,
                      44.27944578123566
                    ],
                    [
                      -89.22665350455384,
                      44.27945312275821
                    ],
                    [
                      -89.22660715200468,
                      44.27944578123566
                    ],
                    [
                      -89.226565336766,
                      44.27942447530737
                    ],
                    [
                      -89.22653215200468,
                      44.27939129054606
                    ],
                    [
                      -89.2265108460764,
                      44.27934947530737
                    ],
                    [
                      -89.22650350455383,
                      44.279303122758215
                    ]
                  ]
                ]
              }
            },
            {
              "id": "1",
              "type": "Feature",
              "properties": {
                "*Field_Area": "35.0716392460504",
                "Area_": "35.0716392460504",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-33-",
                "ENCLOSED_AREA": "35.012",
                "Errors": "41f0280f-651e-4f73-9cba-b0e4fc822970",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "41f0280f-651e-4f73-9cba-b0e4fc822970",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.20597244614544,
                      44.31106799892245
                    ],
                    [
                      -89.205979787668,
                      44.31102164637329
                    ],
                    [
                      -89.20600109359629,
                      44.3109798311346
                    ],
                    [
                      -89.2060342783576,
                      44.31094664637329
                    ],
                    [
                      -89.20607609359628,
                      44.310925340445
                    ],
                    [
                      -89.20612244614544,
                      44.31091799892245
                    ],
                    [
                      -89.2061687986946,
                      44.310925340445
                    ],
                    [
                      -89.20621061393328,
                      44.31094664637329
                    ],
                    [
                      -89.2062437986946,
                      44.3109798311346
                    ],
                    [
                      -89.20626510462289,
                      44.31102164637329
                    ],
                    [
                      -89.20627244614545,
                      44.31106799892245
                    ],
                    [
                      -89.20626510462289,
                      44.3111143514716
                    ],
                    [
                      -89.2062437986946,
                      44.31115616671029
                    ],
                    [
                      -89.20621061393328,
                      44.311189351471604
                    ],
                    [
                      -89.2061687986946,
                      44.31121065739989
                    ],
                    [
                      -89.20612244614544,
                      44.311217998922444
                    ],
                    [
                      -89.20607609359628,
                      44.31121065739989
                    ],
                    [
                      -89.2060342783576,
                      44.311189351471604
                    ],
                    [
                      -89.20600109359629,
                      44.31115616671029
                    ],
                    [
                      -89.205979787668,
                      44.3111143514716
                    ],
                    [
                      -89.20597244614544,
                      44.31106799892245
                    ]
                  ]
                ]
              }
            },
            {
              "id": "2",
              "type": "Feature",
              "properties": {
                "*Field_Area": "162.580630362381",
                "Area_": "162.580630362381",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-08-",
                "ENCLOSED_AREA": "162.31",
                "Errors": "6045134c-f7ed-4f16-aa16-98f02750d2ff",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "6045134c-f7ed-4f16-aa16-98f02750d2ff",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.17208056415653,
                      44.20295170761009
                    ],
                    [
                      -89.17208790567909,
                      44.202905355060935
                    ],
                    [
                      -89.17210921160738,
                      44.202863539822246
                    ],
                    [
                      -89.1721423963687,
                      44.20283035506093
                    ],
                    [
                      -89.17218421160737,
                      44.20280904913265
                    ],
                    [
                      -89.17223056415654,
                      44.20280170761009
                    ],
                    [
                      -89.1722769167057,
                      44.20280904913265
                    ],
                    [
                      -89.17231873194437,
                      44.20283035506093
                    ],
                    [
                      -89.1723519167057,
                      44.202863539822246
                    ],
                    [
                      -89.17237322263398,
                      44.202905355060935
                    ],
                    [
                      -89.17238056415654,
                      44.20295170761009
                    ],
                    [
                      -89.17237322263398,
                      44.202998060159246
                    ],
                    [
                      -89.1723519167057,
                      44.203039875397934
                    ],
                    [
                      -89.17231873194437,
                      44.20307306015925
                    ],
                    [
                      -89.1722769167057,
                      44.203094366087534
                    ],
                    [
                      -89.17223056415654,
                      44.20310170761009
                    ],
                    [
                      -89.17218421160737,
                      44.203094366087534
                    ],
                    [
                      -89.1721423963687,
                      44.20307306015925
                    ],
                    [
                      -89.17210921160738,
                      44.203039875397934
                    ],
                    [
                      -89.17208790567909,
                      44.202998060159246
                    ],
                    [
                      -89.17208056415653,
                      44.20295170761009
                    ]
                  ]
                ]
              }
            },
            {
              "id": "3",
              "type": "Feature",
              "properties": {
                "*Field_Area": "27.4171938026853",
                "Area_": "27.4171938026853",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-16-",
                "ENCLOSED_AREA": "27.371",
                "Errors": "75f86ffe-f88f-4445-9264-5addd09817ac",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "75f86ffe-f88f-4445-9264-5addd09817ac",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.21124489034247,
                      44.29969997229918
                    ],
                    [
                      -89.21125223186503,
                      44.299653619750025
                    ],
                    [
                      -89.21127353779332,
                      44.299611804511336
                    ],
                    [
                      -89.21130672255464,
                      44.29957861975002
                    ],
                    [
                      -89.21134853779331,
                      44.29955731382174
                    ],
                    [
                      -89.21139489034248,
                      44.29954997229918
                    ],
                    [
                      -89.21144124289164,
                      44.29955731382174
                    ],
                    [
                      -89.21148305813031,
                      44.29957861975002
                    ],
                    [
                      -89.21151624289163,
                      44.299611804511336
                    ],
                    [
                      -89.21153754881992,
                      44.299653619750025
                    ],
                    [
                      -89.21154489034248,
                      44.29969997229918
                    ],
                    [
                      -89.21153754881992,
                      44.299746324848336
                    ],
                    [
                      -89.21151624289163,
                      44.299788140087024
                    ],
                    [
                      -89.21148305813031,
                      44.29982132484834
                    ],
                    [
                      -89.21144124289164,
                      44.299842630776624
                    ],
                    [
                      -89.21139489034248,
                      44.29984997229918
                    ],
                    [
                      -89.21134853779331,
                      44.299842630776624
                    ],
                    [
                      -89.21130672255464,
                      44.29982132484834
                    ],
                    [
                      -89.21127353779332,
                      44.299788140087024
                    ],
                    [
                      -89.21125223186503,
                      44.299746324848336
                    ],
                    [
                      -89.21124489034247,
                      44.29969997229918
                    ]
                  ]
                ]
              }
            },
            {
              "id": "4",
              "type": "Feature",
              "properties": {
                "*Field_Area": "35.1545679670981",
                "Area_": "35.1545679670981",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-15-",
                "ENCLOSED_AREA": "35.095",
                "Errors": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.2112191943176,
                      44.29602472823923
                    ],
                    [
                      -89.21122653584015,
                      44.295978375690076
                    ],
                    [
                      -89.21124784176844,
                      44.29593656045139
                    ],
                    [
                      -89.21128102652976,
                      44.295903375690074
                    ],
                    [
                      -89.21132284176844,
                      44.29588206976179
                    ],
                    [
                      -89.2113691943176,
                      44.295874728239234
                    ],
                    [
                      -89.21141554686676,
                      44.29588206976179
                    ],
                    [
                      -89.21145736210543,
                      44.295903375690074
                    ],
                    [
                      -89.21149054686676,
                      44.29593656045139
                    ],
                    [
                      -89.21151185279504,
                      44.295978375690076
                    ],
                    [
                      -89.2115191943176,
                      44.29602472823923
                    ],
                    [
                      -89.21151185279504,
                      44.29607108078839
                    ],
                    [
                      -89.21149054686676,
                      44.296112896027076
                    ],
                    [
                      -89.21145736210543,
                      44.29614608078839
                    ],
                    [
                      -89.21141554686676,
                      44.296167386716675
                    ],
                    [
                      -89.2113691943176,
                      44.29617472823923
                    ],
                    [
                      -89.21132284176844,
                      44.296167386716675
                    ],
                    [
                      -89.21128102652976,
                      44.29614608078839
                    ],
                    [
                      -89.21124784176844,
                      44.296112896027076
                    ],
                    [
                      -89.21122653584015,
                      44.29607108078839
                    ],
                    [
                      -89.2112191943176,
                      44.29602472823923
                    ]
                  ]
                ]
              }
            },
            {
              "id": "5",
              "type": "Feature",
              "properties": {
                "*Field_Area": "151.089663418471",
                "Area_": "151.089663418471",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-04-",
                "ENCLOSED_AREA": "150.83",
                "Errors": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.21943779014806,
                      44.279709888570515
                    ],
                    [
                      -89.21944513167062,
                      44.27966353602136
                    ],
                    [
                      -89.21946643759891,
                      44.27962172078267
                    ],
                    [
                      -89.21949962236023,
                      44.27958853602136
                    ],
                    [
                      -89.2195414375989,
                      44.27956723009307
                    ],
                    [
                      -89.21958779014807,
                      44.27955988857052
                    ],
                    [
                      -89.21963414269723,
                      44.27956723009307
                    ],
                    [
                      -89.2196759579359,
                      44.27958853602136
                    ],
                    [
                      -89.21970914269723,
                      44.27962172078267
                    ],
                    [
                      -89.21973044862551,
                      44.27966353602136
                    ],
                    [
                      -89.21973779014807,
                      44.279709888570515
                    ],
                    [
                      -89.21973044862551,
                      44.27975624111967
                    ],
                    [
                      -89.21970914269723,
                      44.27979805635836
                    ],
                    [
                      -89.2196759579359,
                      44.27983124111967
                    ],
                    [
                      -89.21963414269723,
                      44.27985254704796
                    ],
                    [
                      -89.21958779014807,
                      44.27985988857051
                    ],
                    [
                      -89.2195414375989,
                      44.27985254704796
                    ],
                    [
                      -89.21949962236023,
                      44.27983124111967
                    ],
                    [
                      -89.21946643759891,
                      44.27979805635836
                    ],
                    [
                      -89.21944513167062,
                      44.27975624111967
                    ],
                    [
                      -89.21943779014806,
                      44.279709888570515
                    ]
                  ]
                ]
              }
            }
          ]
        }
      },
      "namespace": "vito"
    },
    "saveresult1": {
      "process_id": "save_result",
      "arguments": {
        "data": {
          "from_node": "CropSAR1"
        },
        "format": "JSON",
        "options": {}
      },
      "result": true
    }
  }
}

Does this look like a plausible result?

{
  "Field_0_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_1_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_2_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_3_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_4_cropSAR": {
    "2023-01-01": null,
    "2023-01-02": null,
    "2023-01-03": null,
    "2023-01-04": null,
    "2023-01-05": null,
    "2023-01-06": null,
    "2023-01-07": null,
    "2023-01-08": null,
    "2023-01-09": null,
    "2023-01-10": null,
    "2023-01-11": null,
    "2023-01-12": null,
    "2023-01-13": null,
    "2023-01-14": null,
    "2023-01-15": null,
    "2023-01-16": null,
    "2023-01-17": null,
    "2023-01-18": null,
    "2023-01-19": null,
    "2023-01-20": null,
    "2023-01-21": null,
    "2023-01-22": null,
    "2023-01-23": null,
    "2023-01-24": null,
    "2023-01-25": null,
    "2023-01-26": null,
    "2023-01-27": null,
    "2023-01-28": null,
    "2023-01-29": null,
    "2023-01-30": null,
    "2023-01-31": null,
    "2023-02-01": null
  },
  "Field_5_cropSAR": {
    "2023-01-01": 0.5519999861717224,
    "2023-01-02": 0.5609999895095825,
    "2023-01-03": 0.5680000185966492,
    "2023-01-04": 0.574999988079071,
    "2023-01-05": 0.5830000042915344,
    "2023-01-06": 0.5899999737739563,
    "2023-01-07": 0.5960000157356262,
    "2023-01-08": 0.6029999852180481,
    "2023-01-09": 0.609000027179718,
    "2023-01-10": 0.6140000224113464,
    "2023-01-11": 0.6190000176429749,
    "2023-01-12": 0.6240000128746033,
    "2023-01-13": 0.628000020980835,
    "2023-01-14": 0.6330000162124634,
    "2023-01-15": 0.6370000243186951,
    "2023-01-16": 0.6399999856948853,
    "2023-01-17": 0.6430000066757202,
    "2023-01-18": 0.6460000276565552,
    "2023-01-19": 0.6489999890327454,
    "2023-01-20": 0.6510000228881836,
    "2023-01-21": 0.652999997138977,
    "2023-01-22": 0.656000018119812,
    "2023-01-23": 0.6579999923706055,
    "2023-01-24": 0.6600000262260437,
    "2023-01-25": 0.6620000004768372,
    "2023-01-26": 0.6639999747276306,
    "2023-01-27": 0.6660000085830688,
    "2023-01-28": 0.6669999957084656,
    "2023-01-29": 0.6690000295639038,
    "2023-01-30": 0.6710000038146973,
    "2023-01-31": 0.671999990940094,
    "2023-02-01": 0.6729999780654907
  }
}

HansVRP · 2024-06-20T12:25:08Z

cropSAR might actually give an output eventhough it receives an empty input cube. But it also might need a dedicated feature flag.

I guess we should be able to tie this flag in with the flag for allowing empty datacubes?

bossie · 2024-06-20T13:47:52Z

Yes, that is what this PR is about: https://git.vito.be/projects/APPL/repos/nextland/pull-requests/5/overview

This is an example process graph with the feature enabled:

{
  "process_graph": {
    "CropSAR1": {
      "process_id": "CropSAR",
      "arguments": {
        "biopar_type": "FCOVER",
        "date": [
          "2023-01-01",
          "2023-02-01"
        ],
        "minimal_input_checks": true,
        "allow_empty_cubes": true,
        "polygon": {
          "type": "FeatureCollection",
          "features": [
            {
              "id": "0",
              "type": "Feature",
              "properties": {
                "*Field_Area": "60.7104355585474",
                "Area_": "60.7104355585474",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-06-",
                "ENCLOSED_AREA": "60.607",
                "Errors": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "079c5aa6-4aaa-4ba9-8349-2a778046dbd6",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.22650350455383,
                      44.279303122758215
                    ],
                    [
                      -89.2265108460764,
                      44.27925677020906
                    ],
                    [
                      -89.22653215200468,
                      44.27921495497037
                    ],
                    [
                      -89.226565336766,
                      44.27918177020906
                    ],
                    [
                      -89.22660715200468,
                      44.27916046428077
                    ],
                    [
                      -89.22665350455384,
                      44.27915312275822
                    ],
                    [
                      -89.226699857103,
                      44.27916046428077
                    ],
                    [
                      -89.22674167234167,
                      44.27918177020906
                    ],
                    [
                      -89.226774857103,
                      44.27921495497037
                    ],
                    [
                      -89.22679616303128,
                      44.27925677020906
                    ],
                    [
                      -89.22680350455384,
                      44.279303122758215
                    ],
                    [
                      -89.22679616303128,
                      44.27934947530737
                    ],
                    [
                      -89.226774857103,
                      44.27939129054606
                    ],
                    [
                      -89.22674167234167,
                      44.27942447530737
                    ],
                    [
                      -89.226699857103,
                      44.27944578123566
                    ],
                    [
                      -89.22665350455384,
                      44.27945312275821
                    ],
                    [
                      -89.22660715200468,
                      44.27944578123566
                    ],
                    [
                      -89.226565336766,
                      44.27942447530737
                    ],
                    [
                      -89.22653215200468,
                      44.27939129054606
                    ],
                    [
                      -89.2265108460764,
                      44.27934947530737
                    ],
                    [
                      -89.22650350455383,
                      44.279303122758215
                    ]
                  ]
                ]
              }
            },
            {
              "id": "1",
              "type": "Feature",
              "properties": {
                "*Field_Area": "35.0716392460504",
                "Area_": "35.0716392460504",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-33-",
                "ENCLOSED_AREA": "35.012",
                "Errors": "41f0280f-651e-4f73-9cba-b0e4fc822970",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "41f0280f-651e-4f73-9cba-b0e4fc822970",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.20597244614544,
                      44.31106799892245
                    ],
                    [
                      -89.205979787668,
                      44.31102164637329
                    ],
                    [
                      -89.20600109359629,
                      44.3109798311346
                    ],
                    [
                      -89.2060342783576,
                      44.31094664637329
                    ],
                    [
                      -89.20607609359628,
                      44.310925340445
                    ],
                    [
                      -89.20612244614544,
                      44.31091799892245
                    ],
                    [
                      -89.2061687986946,
                      44.310925340445
                    ],
                    [
                      -89.20621061393328,
                      44.31094664637329
                    ],
                    [
                      -89.2062437986946,
                      44.3109798311346
                    ],
                    [
                      -89.20626510462289,
                      44.31102164637329
                    ],
                    [
                      -89.20627244614545,
                      44.31106799892245
                    ],
                    [
                      -89.20626510462289,
                      44.3111143514716
                    ],
                    [
                      -89.2062437986946,
                      44.31115616671029
                    ],
                    [
                      -89.20621061393328,
                      44.311189351471604
                    ],
                    [
                      -89.2061687986946,
                      44.31121065739989
                    ],
                    [
                      -89.20612244614544,
                      44.311217998922444
                    ],
                    [
                      -89.20607609359628,
                      44.31121065739989
                    ],
                    [
                      -89.2060342783576,
                      44.311189351471604
                    ],
                    [
                      -89.20600109359629,
                      44.31115616671029
                    ],
                    [
                      -89.205979787668,
                      44.3111143514716
                    ],
                    [
                      -89.20597244614544,
                      44.31106799892245
                    ]
                  ]
                ]
              }
            },
            {
              "id": "2",
              "type": "Feature",
              "properties": {
                "*Field_Area": "162.580630362381",
                "Area_": "162.580630362381",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-08-",
                "ENCLOSED_AREA": "162.31",
                "Errors": "6045134c-f7ed-4f16-aa16-98f02750d2ff",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "6045134c-f7ed-4f16-aa16-98f02750d2ff",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.17208056415653,
                      44.20295170761009
                    ],
                    [
                      -89.17208790567909,
                      44.202905355060935
                    ],
                    [
                      -89.17210921160738,
                      44.202863539822246
                    ],
                    [
                      -89.1721423963687,
                      44.20283035506093
                    ],
                    [
                      -89.17218421160737,
                      44.20280904913265
                    ],
                    [
                      -89.17223056415654,
                      44.20280170761009
                    ],
                    [
                      -89.1722769167057,
                      44.20280904913265
                    ],
                    [
                      -89.17231873194437,
                      44.20283035506093
                    ],
                    [
                      -89.1723519167057,
                      44.202863539822246
                    ],
                    [
                      -89.17237322263398,
                      44.202905355060935
                    ],
                    [
                      -89.17238056415654,
                      44.20295170761009
                    ],
                    [
                      -89.17237322263398,
                      44.202998060159246
                    ],
                    [
                      -89.1723519167057,
                      44.203039875397934
                    ],
                    [
                      -89.17231873194437,
                      44.20307306015925
                    ],
                    [
                      -89.1722769167057,
                      44.203094366087534
                    ],
                    [
                      -89.17223056415654,
                      44.20310170761009
                    ],
                    [
                      -89.17218421160737,
                      44.203094366087534
                    ],
                    [
                      -89.1721423963687,
                      44.20307306015925
                    ],
                    [
                      -89.17210921160738,
                      44.203039875397934
                    ],
                    [
                      -89.17208790567909,
                      44.202998060159246
                    ],
                    [
                      -89.17208056415653,
                      44.20295170761009
                    ]
                  ]
                ]
              }
            },
            {
              "id": "3",
              "type": "Feature",
              "properties": {
                "*Field_Area": "27.4171938026853",
                "Area_": "27.4171938026853",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-16-",
                "ENCLOSED_AREA": "27.371",
                "Errors": "75f86ffe-f88f-4445-9264-5addd09817ac",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "75f86ffe-f88f-4445-9264-5addd09817ac",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.21124489034247,
                      44.29969997229918
                    ],
                    [
                      -89.21125223186503,
                      44.299653619750025
                    ],
                    [
                      -89.21127353779332,
                      44.299611804511336
                    ],
                    [
                      -89.21130672255464,
                      44.29957861975002
                    ],
                    [
                      -89.21134853779331,
                      44.29955731382174
                    ],
                    [
                      -89.21139489034248,
                      44.29954997229918
                    ],
                    [
                      -89.21144124289164,
                      44.29955731382174
                    ],
                    [
                      -89.21148305813031,
                      44.29957861975002
                    ],
                    [
                      -89.21151624289163,
                      44.299611804511336
                    ],
                    [
                      -89.21153754881992,
                      44.299653619750025
                    ],
                    [
                      -89.21154489034248,
                      44.29969997229918
                    ],
                    [
                      -89.21153754881992,
                      44.299746324848336
                    ],
                    [
                      -89.21151624289163,
                      44.299788140087024
                    ],
                    [
                      -89.21148305813031,
                      44.29982132484834
                    ],
                    [
                      -89.21144124289164,
                      44.299842630776624
                    ],
                    [
                      -89.21139489034248,
                      44.29984997229918
                    ],
                    [
                      -89.21134853779331,
                      44.299842630776624
                    ],
                    [
                      -89.21130672255464,
                      44.29982132484834
                    ],
                    [
                      -89.21127353779332,
                      44.299788140087024
                    ],
                    [
                      -89.21125223186503,
                      44.299746324848336
                    ],
                    [
                      -89.21124489034247,
                      44.29969997229918
                    ]
                  ]
                ]
              }
            },
            {
              "id": "4",
              "type": "Feature",
              "properties": {
                "*Field_Area": "35.1545679670981",
                "Area_": "35.1545679670981",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-15-",
                "ENCLOSED_AREA": "35.095",
                "Errors": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "8d4a8002-08d0-4da4-a7ce-42441b2e6b74",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.2112191943176,
                      44.29602472823923
                    ],
                    [
                      -89.21122653584015,
                      44.295978375690076
                    ],
                    [
                      -89.21124784176844,
                      44.29593656045139
                    ],
                    [
                      -89.21128102652976,
                      44.295903375690074
                    ],
                    [
                      -89.21132284176844,
                      44.29588206976179
                    ],
                    [
                      -89.2113691943176,
                      44.295874728239234
                    ],
                    [
                      -89.21141554686676,
                      44.29588206976179
                    ],
                    [
                      -89.21145736210543,
                      44.295903375690074
                    ],
                    [
                      -89.21149054686676,
                      44.29593656045139
                    ],
                    [
                      -89.21151185279504,
                      44.295978375690076
                    ],
                    [
                      -89.2115191943176,
                      44.29602472823923
                    ],
                    [
                      -89.21151185279504,
                      44.29607108078839
                    ],
                    [
                      -89.21149054686676,
                      44.296112896027076
                    ],
                    [
                      -89.21145736210543,
                      44.29614608078839
                    ],
                    [
                      -89.21141554686676,
                      44.296167386716675
                    ],
                    [
                      -89.2113691943176,
                      44.29617472823923
                    ],
                    [
                      -89.21132284176844,
                      44.296167386716675
                    ],
                    [
                      -89.21128102652976,
                      44.29614608078839
                    ],
                    [
                      -89.21124784176844,
                      44.296112896027076
                    ],
                    [
                      -89.21122653584015,
                      44.29607108078839
                    ],
                    [
                      -89.2112191943176,
                      44.29602472823923
                    ]
                  ]
                ]
              }
            },
            {
              "id": "5",
              "type": "Feature",
              "properties": {
                "*Field_Area": "151.089663418471",
                "Area_": "151.089663418471",
                "AssetType_": "Crop Year 2023",
                "Descriptor": "W-04-",
                "ENCLOSED_AREA": "150.83",
                "Errors": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6",
                "GROUP_AREA": null,
                "Grower": "Heartland Farms",
                "IGuid_": "b49ffd22-3c4d-49d6-bc43-aec0cf5818d6",
                "INVALID_TYPE": null,
                "INVALID_X": null,
                "INVALID_Y": null,
                "ISLAND_AREA": null,
                "LENGTH": null,
                "WIDTH": null,
                "h3index": "83275cfffffffff",
                "tile": "16TCQ"
              },
              "geometry": {
                "type": "Polygon",
                "coordinates": [
                  [
                    [
                      -89.21943779014806,
                      44.279709888570515
                    ],
                    [
                      -89.21944513167062,
                      44.27966353602136
                    ],
                    [
                      -89.21946643759891,
                      44.27962172078267
                    ],
                    [
                      -89.21949962236023,
                      44.27958853602136
                    ],
                    [
                      -89.2195414375989,
                      44.27956723009307
                    ],
                    [
                      -89.21958779014807,
                      44.27955988857052
                    ],
                    [
                      -89.21963414269723,
                      44.27956723009307
                    ],
                    [
                      -89.2196759579359,
                      44.27958853602136
                    ],
                    [
                      -89.21970914269723,
                      44.27962172078267
                    ],
                    [
                      -89.21973044862551,
                      44.27966353602136
                    ],
                    [
                      -89.21973779014807,
                      44.279709888570515
                    ],
                    [
                      -89.21973044862551,
                      44.27975624111967
                    ],
                    [
                      -89.21970914269723,
                      44.27979805635836
                    ],
                    [
                      -89.2196759579359,
                      44.27983124111967
                    ],
                    [
                      -89.21963414269723,
                      44.27985254704796
                    ],
                    [
                      -89.21958779014807,
                      44.27985988857051
                    ],
                    [
                      -89.2195414375989,
                      44.27985254704796
                    ],
                    [
                      -89.21949962236023,
                      44.27983124111967
                    ],
                    [
                      -89.21946643759891,
                      44.27979805635836
                    ],
                    [
                      -89.21944513167062,
                      44.27975624111967
                    ],
                    [
                      -89.21943779014806,
                      44.279709888570515
                    ]
                  ]
                ]
              }
            }
          ]
        }
      },
      "namespace": "vito"
    },
    "saveresult1": {
      "process_id": "save_result",
      "arguments": {
        "data": {
          "from_node": "CropSAR1"
        },
        "format": "JSON",
        "options": {}
      },
      "result": true
    }
  }
}

Does this approach (a parameter of the CropSAR process) support your use case with regards to being able to turn the flag on and off?

Instead of a parameter, a job_option is also an... option but requires a bit of extra work.

bossie · 2024-06-20T13:49:21Z

These is the result for the 60 polygon process graph: CropSAR_60_polygons.json

Pratichhya · 2024-06-20T13:55:17Z

These is the result for the 60 polygon process graph: CropSAR_60_polygons.json

Are these for the previously failed polygons of j-24051737210441be92441beedd8ab5ce (60 polygons)?

bossie · 2024-06-20T14:01:25Z

Are these for the previously failed polygons of j-24051737210441be92441beedd8ab5ce (60 polygons)?

That is right.

#649

Pratichhya · 2024-06-24T15:27:24Z

Hi @bossie, Was the result achieved when tested in staging?

bossie · 2024-06-25T05:36:55Z

No, I ran this locally, it's not on dev or staging yet. I'm taking a different approach and the plan is to get this out shortly.

Would you consider the results I attached correct?

Pratichhya · 2024-06-25T06:03:20Z

No, I ran this locally, it's not on dev or staging yet. I'm taking a different approach, and the plan is to get this out shortly.

ahh ok. Thank you for confirming

Would you consider the results I attached correct?

It looks similar to the expected result. However, for those with null, I was wondering if these didn't have any of Sentinel 1 and Sentinel 2. Also, I simply wanted to check if they still remain null when tested for the entire year.

KeyError: "Your key: allow_empty_cubes was not in the whitelist [vault_token, sentinel_hub_client_alias, max_soft_errors_ratio, dependencies, pyramid_levels, require_bounds, correlation_id, user]. This error needs to be fixed in the backend."

bossie · 2024-06-26T14:11:10Z

Abandoned the PR in the nextland repo in favor of a job option:

{
  "allow_empty_cubes": true
}

bossie · 2024-06-27T09:26:26Z

@Pratichhya this is available on staging.

jdries self-assigned this Jan 23, 2024

jdries added a commit to Open-EO/openeo-geotrellis-extensions that referenced this issue Jan 23, 2024

add option to allow for an empty datacube to be returned

c791325

Open-EO/openeo-geopyspark-driver#649

jdries added a commit that referenced this issue Jan 24, 2024

pass on feature flag to allow empty cubes

9ff51b7

#649

soxofaan mentioned this issue Jun 3, 2024

catch multipolygon errors #785

Closed

JeroenVerstraelen assigned EmileSonneveld and unassigned jdries Jun 4, 2024

JeroenVerstraelen assigned bossie and unassigned EmileSonneveld Jun 17, 2024

bossie added a commit that referenced this issue Jun 21, 2024

support allow_empty_cubes job_option

93adb22

#649

bossie mentioned this issue Jun 21, 2024

support allow_empty_cubes job_option #811

Merged

bossie linked a pull request Jun 21, 2024 that will close this issue

support allow_empty_cubes job_option #811

Merged

bossie added a commit that referenced this issue Jun 21, 2024

document allow_empty_cubes job_option

a6d2cba

#649

bossie added a commit that referenced this issue Jun 25, 2024

support allow_empty_cubes job_option #649

26fb8bd

bossie closed this as completed in #811 Jun 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow empty datacubes #649

allow empty datacubes #649

jdries commented Jan 23, 2024 •

edited

Loading

jdries commented Jan 24, 2024

soxofaan commented Jan 24, 2024

jdries commented Jan 24, 2024

m-mohr commented Jan 24, 2024

soxofaan commented Jun 3, 2024

EmileSonneveld commented Jun 4, 2024

EmileSonneveld commented Jun 5, 2024

JeroenVerstraelen commented Jun 17, 2024 •

edited

Loading

bossie commented Jun 19, 2024

HansVRP commented Jun 19, 2024

bossie commented Jun 20, 2024

bossie commented Jun 20, 2024

HansVRP commented Jun 20, 2024

bossie commented Jun 20, 2024

bossie commented Jun 20, 2024

Pratichhya commented Jun 20, 2024

bossie commented Jun 20, 2024

Pratichhya commented Jun 24, 2024

bossie commented Jun 25, 2024

Pratichhya commented Jun 25, 2024

bossie commented Jun 26, 2024

bossie commented Jun 27, 2024

allow empty datacubes #649

allow empty datacubes #649

Comments

jdries commented Jan 23, 2024 • edited Loading

jdries commented Jan 24, 2024

soxofaan commented Jan 24, 2024

jdries commented Jan 24, 2024

m-mohr commented Jan 24, 2024

soxofaan commented Jun 3, 2024

EmileSonneveld commented Jun 4, 2024

EmileSonneveld commented Jun 5, 2024

JeroenVerstraelen commented Jun 17, 2024 • edited Loading

bossie commented Jun 19, 2024

HansVRP commented Jun 19, 2024

bossie commented Jun 20, 2024

bossie commented Jun 20, 2024

HansVRP commented Jun 20, 2024

bossie commented Jun 20, 2024

bossie commented Jun 20, 2024

Pratichhya commented Jun 20, 2024

bossie commented Jun 20, 2024

Pratichhya commented Jun 24, 2024

bossie commented Jun 25, 2024

Pratichhya commented Jun 25, 2024

bossie commented Jun 26, 2024

bossie commented Jun 27, 2024

jdries commented Jan 23, 2024 •

edited

Loading

JeroenVerstraelen commented Jun 17, 2024 •

edited

Loading