feat(python): design an image extension type

[rok]

See #1199.

[changhiskhan]

1. is it possible to have a super type interface? e.g., for higher level applications, they may only care about showing an image, or getting a numpy array, and not necessarily care whether the underlying image is represented as string uri, tensor, or bytes.
2. can we add some methods to convert from one extension type to another?

(Totally fine if the answer is we'll address it later, let's focus on delivering the immediate use case ofc).

[jrabary]

Having the ability to decode 16 bits images would be great (for example depth map saved in a png file)

[wjones127]

Could we get docstrings for each of these methods? Preferably with doctest-verified examples.

[rok]

@wjones127 I assume we want to stay in arrow as much as possible, but encoder/encoder and TF bridge use numpy here.
Do you have an idea how to avoid numpy at the arrow <-> numpy bridge?

[wjones127]

I assume we want to stay in arrow as much as possible, but encoder/encoder and TF bridge use numpy here.
Do you have an idea how to avoid numpy at the arrow <-> numpy bridge?

I think there are two concerns:

1. It would be nice to not require numpy as a dependency.
2. We should make sure if we aren't having to make data copies unnecessarily.

If this is all internal, I think it's fine to accept this cost for now. We might just have to wait for the ecosystem to get more Arrow native before we can drop numpy as dependency for this.

[rok]

If this is all internal, I think it's fine to accept this cost for now. We might just have to wait for the ecosystem to get more Arrow native before we can drop numpy as dependency for this.

Indeed. Let's keep an eye out for more arrow native ways to move images to tensors while we wait.

Could we get docstrings for each of these methods? Preferably with doctest-verified examples.

Added.

[rok]

1. is it possible to have a super type interface? e.g., for higher level applications, they may only care about showing an image, or getting a numpy array, and not necessarily care whether the underlying image is represented as string uri, tensor, or bytes.

I think we should add this now. I'd only propose we do it as a separate PR to manage scope here.

2. can we add some methods to convert from one extension type to another?

We now have from_uris, read_uris, image_to_tensor, to_tf and to_encoded. Are we missing something we really need at this point? I suppose we are missing pytorch but I'm not sure it's being requested.

[rok]

@changhiskhan

1. is it possible to have a super type interface? e.g., for higher level applications, they may only care about showing an image, or getting a numpy array, and not necessarily care whether the underlying image is represented as string uri, tensor, or bytes.

I think we should add this now. I'd only propose we do it as a separate PR to manage scope here.

2. can we add some methods to convert from one extension type to another?

We now have from_uris, read_uris, image_to_tensor, to_tf and to_encoded. Are we missing something we really need at this point? I suppose we are missing pytorch but I'm not sure it's being requested.

[wjones127]

I think most important change is that we add a roundtrip test to Lance.

[westonpace]

Very cool, only a few thoughts

[westonpace]

Hmm...if I'm a user I might wonder what the shape of the resulting array would be. It looks like, for both tensorflow and pillow, you get a 3d array [height, width, channel]. Do you think we should document and/or mandate this?

[rok]

The annoying part here is that PNG comes with an extra transparency channel etc. Since we're offering a custom decoder option it'd be awkward to enforce output shapes here.
I'm not sure what a good user experience here. Maybe we could get some user feedback?

[rok]

FixedShapeTensorType can't be subclassed at the moment, so we need to consider other options for FixedShapeImageTensor. Here are some ideas:

• Use vanilla FixedShapeTensorType and hence FixedShapeTensorArray. Downside is that we can't add methods to these, but we could instead provide a utility module for images. This makes image types (uri, encoded, tensor) less uniform, but we're probably adding a universal image class on top of them anyway.
• Wait for arrow to add a way to subclass canonical extensions to add ImageTensor type.
• Create our own extension class that is a copy of FixedShapeTensorType. This would work immediately and make API nicer. We would miss some methods FixedShapeTensorArray implements in C++, but we could still use those with on the fly casting.

In related news VariableShapeTensorType proposal will go to vote soon.

[rok]

@wjones127 can we merge this?