Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDSE: set smaller tile size for processing #311

Closed
jdries opened this issue Jul 24, 2024 · 2 comments
Closed

CDSE: set smaller tile size for processing #311

jdries opened this issue Jul 24, 2024 · 2 comments
Assignees

Comments

@jdries
Copy link
Contributor

jdries commented Jul 24, 2024

We notice that there's some benefits to processing with a smaller tile size.
We previously tried to increase it, but this was because data loading was faster for large tiles.
For the rest of the processing, we see that smaller tiles result in fewer memory issues, so perhaps we can now use a 64px tilesize as default if possible, or else derive it from apply_neighborhood parameters.

@jdries
Copy link
Contributor Author

jdries commented Jul 26, 2024

We can do something rather general in layercatalog.py:

elif(get_backend_config().default_reading_strategy == "load_per_product"):
            datacubeParams.setLoadPerProduct(True)
            if "tilesize" not in feature_flags:
                #when doing load_per_product, tilesize does not affect read_performance, and smaller chunks are better for memory usage
                getattr(datacubeParams, "tileSize_$eq")(128)

not committing this now, as it requires some followup. Maybe even better is to make this 'default' chunk size a parameter in the backend config or have it as a job option rather than custom feature flag.

@jdries jdries self-assigned this Jul 26, 2024
jdries added a commit to Open-EO/openeo-geopyspark-driver that referenced this issue Aug 5, 2024
@jdries
Copy link
Contributor Author

jdries commented Aug 14, 2024

configured new default of 128 on dev/staging

@jdries jdries closed this as completed Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant