Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory issues with percentile calculation #2

Open
DamienIrving opened this issue Oct 21, 2022 · 1 comment
Open

Memory issues with percentile calculation #2

DamienIrving opened this issue Oct 21, 2022 · 1 comment
Assignees
@DamienIrving
Labels
blocker Issues that are blocking further progress

Comments

@DamienIrving
Copy link
Member

When using xclim or icclim to calculate indices that involve calculating a threshold along the entire time axis (a percentile-based threshold), memory usage can explode for large multi-dimensional (i.e. high lat/lon resolution) arrays.

A solution to this problem (i.e. appropriate file chunking and/or use of dask) needs to be found before we can commit to using xclim or icclim for the calculation of climate indices.
@DamienIrving DamienIrving self-assigned this Oct 21, 2022
@DamienIrving DamienIrving added the blocker Issues that are blocking further progress label Oct 21, 2022
@DamienIrving
Copy link
Member Author

Update (cc @ngben @chunhsusu): The latest version of icclim (v6.0.0) is much more efficient memory-wise, to the point where setting up a dask cluster isn't necessary for most/all problems. I've updated my run_icclim.py script at the following repo and the README has details: https://github.com/AusClimateService/indices

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DamienIrving DamienIrving

Labels
blocker Issues that are blocking further progress
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@DamienIrving