Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix nondefault compression handling #358

Merged

Conversation

peytondmurray
Copy link
Collaborator

@peytondmurray peytondmurray commented Jul 30, 2024

This PR fixes compression handling for nondefault compression types, closes #321 and partially addresses #331. Additionally if recreate_dataset is called, we now log a warning message if compression is enabled for the dataset; this is because h5py stores compression settings in Dataset._filters, and we have no way of distinguishing these from other filters that may be present in the file.

In practice I think this is an unusual use case, but we now emit a warning if this does happen.

If an invalid compression setting is requested in write_dataset, a more informative error message is displayed with a listing of all detected compression settings.

@peytondmurray peytondmurray marked this pull request as ready for review July 30, 2024 05:21
@ArvidJB
Copy link
Collaborator

ArvidJB commented Jul 31, 2024

Looks good!

Can you add a test for blosc compression via pytables like in #321 (comment) ? pytables should be an optional dependency for testing only.

@peytondmurray
Copy link
Collaborator Author

peytondmurray commented Aug 5, 2024

I parametrized the tests I added to try with both hdf5plugin and pytables, but pytables appears to be incompatible with numpy>=2 (PyTables/PyTables#1172). I've made the tests skip if the user has both pytables and numpy>2 installed. Right now if you want to use blosc compression with the latest numpy, it seems like hdf5plugin is the way to go.

@peytondmurray peytondmurray merged commit 71c05c7 into deshaw:master Aug 5, 2024
7 checks passed
@peytondmurray peytondmurray deleted the fix-compression-handling-321 branch August 5, 2024 22:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Handle non-default compression (PyInf#12601)
2 participants