Rebuild stac scripts #158
Merged
Rebuild stac scripts #158
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tiles + Moved the asset definitions to worldcereal's source folder as they are accessef from multiple points
1 task
VincentVerelst
approved these changes
Sep 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implemented scripts to build the STAC catalogue from the extracted patches. The procedure is done in three steps
scripts/stac/build_paths.pyParses a folder of S1/S2 patches (for example/vitodata/worldcereal_data/EXTRACTIONS/SENTINEL2/) and creates a file of a pickled list of paths to parallelize & parse in the next script.scripts/stac/cataloguer_builder.pyUsing the Spark cluster to read the extracted patches and build catalogues, then each separate catalogue is pickled in a different output file in a specified output folder.scripts/stac/split_catalogues.pyMerges all the catalogues built by the previous script together and separates them by UTM as required by OpenEO. Separated by GFMAP's functionalityopeneo_gfmap.utils.split_stac.split_collection_by_epsg. The output folder will contain each STAC fully written as json.In addition, I moved constants previously defined in the extraction scripts such as Asset Definitions in worldcereal's source folder. As from know on multiple scripts will use those values.
Finally, I disabled the creation/management of STAC assets from GFMAPJobManager in the extraction scripts.
To ease up the execution of the catalogue_builder.py script, I used the package
mepsydeveloped by Daniele which allows to run Spark Jobs in our cluster with less hassle. I didn't add this dependency in the package as it is not publicly installable but instead accessible in our private git server: https://git.vito.be/projects/TAP-VEGTEAM/repos/mepsy/browseWe could make this more broadly available by not using mepsy, but then not many have a spark cluster for themselves and each one works on different configuration. Let me know what you think...