Optimized datashader aggregation of NdOverlays

[philippjfr]

This PR provides major optimizations when using the datashader operations to aggregate multiple objects in an NdOverlay using the count, sum, and mean operations. Each Element is aggregated separately and the individual aggregates are summed. A small complication is that NaNs have to be replaced by zeros and masked at the end. mean is supported by dividing sum and count aggregates. This avoids the large memory and performance overhead of concatenating multiple dataframes together. I'm still working on adding an optimization for count_cat but it should also be fairly straightforward.

[jbednar]

Excellent, thanks! I'm too tired to try to parse the code, but is it using the ability of datashader to compute multiple aggregations in a single pass?

[philippjfr]

Excellent, thanks! I'm too tired to try to parse the code, but is it using the ability of datashader to compute multiple aggregations in a single pass?

No, even so it's still faster, which is perhaps a bit surprising. I'll do some more profiling tomorrow.

[philippjfr]

I was wrong it's slightly slower, but including the concatenation step it still wins out massively both on performance and memory load.

[philippjfr]

Means it's perhaps still worth optimizing get_agg_data directly to ensure it only has to be done once. Particularly worth testing how well concatenating multiple dask dataframes performs.

[philippjfr]

count_cat now optimized as well.

[philippjfr]

Here are some benchmarks, the data here are 12 curves of increasing length where 1 minute is equivalent to 60000*60 samples. The four conditions are comparing line aggregation of multiple curves either by summing the aggregates (the new approach) or by aggregating over concatenated curves separated by NaNs.

You can see that the new approach is generally slightly slower than aggregating over already concatenated lines, but it scales much better when using dask.

[jlstevens]

@philippjfr Thanks for fixing the warning!

Is it now ready to merge or is there something else you wish to do first?

[philippjfr]

Yes, this is ready to merge now. Further optimizations can come in later PRs.

[jlstevens]

Great! Merging.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.