Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storing non ndarray types as blocks #895

Open
CagtayFabry opened this issue Dec 10, 2020 · 2 comments
Open

storing non ndarray types as blocks #895

CagtayFabry opened this issue Dec 10, 2020 · 2 comments

Comments

@CagtayFabry
Copy link
Contributor

Is there a way to store any other types besides ndarray as a block in an ASDF file?

I am looking for a simple way to serialize (and tag) bytes, bytearray, BytesIO or similar without going through numpy

Could this be done in an extension library or would this need changes in asdf?

@eslavich
Copy link
Contributor

eslavich commented Apr 2, 2021

Currently the code requires that anything written to a block be wrapped in ndarray, but I think it's possible to wrap bytes in ndarray without copying the memory:

In [21]: some_bytes = b'a' * 1024

In [22]: array = np.frombuffer(some_bytes, dtype=np.uint8)

In [23]: array.flags
Out[23]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

Now that array can be assigned to a block using the BlockManager.find_or_create_block_for_array method.

On the way back I don't know how to retrieve the bytes except to call ndarray.tobytes, which does make a copy.

@CagtayFabry
Copy link
Contributor Author

yes that is a possible workaround we used before

However I was looking for a possibility to explicitly differentiate from ndarray but in this case I will probably just wrap the numpy calls in a custom tag/object

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants