Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add generic tools for image resampling #10

Open
EiffL opened this issue Jun 20, 2021 · 3 comments
Open

Add generic tools for image resampling #10

EiffL opened this issue Jun 20, 2021 · 3 comments
Labels
enhancement New feature or request

Comments

@EiffL
Copy link
Member

EiffL commented Jun 20, 2021

The most fundamental operation we need, besides the FFT, is the ability to do image resampling, this is crucial to many tasks from convolution to shearing.

Right now, we are relying on existing TF tools. Namely, for the x_interpolant in our convolution tools we are using the built-in tf.image.resize, this has a lot of limitations, and in particular makes weird assumptions about pixel centers, which can induce shifts. We also use the tensorflow addons image resample tool, but this one is limited to linear interpolation.

So... we want a generic tool that can resample an image using linear, cubic, quintic and lanczos kernels, and this needs to be differentiable with respect to all the inputs, including the resampling grid.

@EiffL EiffL added the enhancement New feature or request label Jun 20, 2021
@EiffL
Copy link
Member Author

EiffL commented Jun 20, 2021

Hummmm did a bunch of experimentation... I don't know if I can come up with an efficient implementation at the TF level for different interpolation kerrnels.
Here is a notebook with some experiments for cubic interpolation, but it's super slow: https://colab.research.google.com/drive/14Jr01GgaVwkmS5ZV6RC4pjmslokgDqK_?usp=sharing

One alternative.... would be to modify the tfa implementation to support additional interpolation kernels.... would require a tiny bit of effort ^^"

@EiffL
Copy link
Member Author

EiffL commented Jun 20, 2021

After further thought, shouldn't be impossible to do to add a higher order interpolation kernel given that pretty much all utilities for that are already in place in TFA. So, I'm opening a fork:
https://github.com/DifferentiableUniverseInitiative/addons

And will be extending the resampler ( https://github.com/DifferentiableUniverseInitiative/addons/blob/master/tensorflow_addons/custom_ops/image/cc/kernels/resampler_ops.cc ) to support additional interpolation methods

@EiffL
Copy link
Member Author

EiffL commented Jun 20, 2021

good news, cubic interpolant implemented, and 2 orders of magnitude in reduction of residuals.

Before:
image

After:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant